HTTP for Backend Engineers: A Comprehensive Guide

1. HTTP (Hypertext Transfer Protocol)

HTTP is an application-layer protocol designed for distributed, collaborative, hypermedia information systems. It’s the foundation of data communication for the World Wide Web.

Key Characteristics

• Client-Server Model: HTTP follows a request-response pattern where clients (browsers, mobile apps) send requests and servers respond with data
• Stateless Protocol: Each request is independent; the server doesn’t retain information about previous requests
• Text-Based: HTTP messages are human-readable text (though the body can be binary)
• Port: Default port is 80

HTTP Request Structure

GET /api/users HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0
Accept: application/json

HTTP Response Structure

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 348

{"users": [...]}

---

2. HTTPS (HTTP Secure)

HTTPS is HTTP with encryption. It uses TLS (Transport Layer Security) or its predecessor SSL (Secure Sockets Layer) to encrypt data between client and server.

Why HTTPS Matters

• Encryption: Protects data from eavesdropping and man-in-the-middle attacks
• Authentication: Verifies that users are communicating with the intended server
• Data Integrity: Ensures data hasn’t been tampered with during transmission
• SEO Benefits: Search engines favor HTTPS sites
• Trust: Browsers mark HTTP sites as “Not Secure”

How HTTPS Works

1. TLS Handshake: Client and server establish a secure connection
2. Certificate Verification: Server presents its SSL/TLS certificate
3. Key Exchange: Symmetric encryption keys are exchanged
4. Encrypted Communication: All data is encrypted using agreed-upon keys

Key Differences: HTTP vs HTTPS

Feature	HTTP	HTTPS
Port	80	443
Security	No encryption	TLS/SSL encryption
Certificate	Not required	SSL certificate required
Speed	Slightly faster	Minimal overhead with HTTP/2
SEO	Lower ranking	Higher ranking

---

3. HTTP Headers

HTTP headers allow the client and server to pass additional information with requests and responses. Headers are case-insensitive key-value pairs.

Common Request Headers

Host

Specifies the domain name of the server

Host: api.example.com

User-Agent

Identifies the client application

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)

Accept

Specifies media types the client can process

Accept: application/json, text/html

Authorization

Contains credentials for authenticating the client

Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

Content-Type

Indicates the media type of the request body

Content-Type: application/json

Cookie

Contains stored HTTP cookies

Cookie: sessionId=abc123; userId=456

Common Response Headers

Content-Type

Indicates the media type of the response body

Content-Type: application/json; charset=utf-8

Content-Length

Size of the response body in bytes

Content-Length: 1234

Set-Cookie

Sends cookies from server to client

Set-Cookie: sessionId=abc123; HttpOnly; Secure; SameSite=Strict

Cache-Control

Directives for caching mechanisms

Cache-Control: public, max-age=3600

Access-Control-Allow-Origin (CORS)

Specifies which origins can access the resource

Access-Control-Allow-Origin: https://example.com

Location

Used in redirects to specify the new URL

Location: https://example.com/new-page

Custom Headers

Backend engineers often create custom headers (conventionally prefixed with X-)

X-Request-ID: 550e8400-e29b-41d4-a716-446655440000
X-Rate-Limit-Remaining: 99

---

4. HTTP Methods

HTTP methods (also called verbs) indicate the desired action to be performed on a resource.

GET

Retrieves data from the server. Should be safe and idempotent.

GET /api/users/123 HTTP/1.1
Host: example.com

Characteristics:

• No request body
• Can be cached
• Remains in browser history
• Can be bookmarked

POST

Submits data to create a new resource. Not idempotent.

POST /api/users HTTP/1.1
Host: example.com
Content-Type: application/json

{
  "name": "John Doe",
  "email": "john@example.com"
}

Characteristics:

• Has a request body
• Not cached by default
• Not stored in browser history

PUT

Updates or replaces an entire resource. Idempotent.

PUT /api/users/123 HTTP/1.1
Host: example.com
Content-Type: application/json

{
  "name": "John Smith",
  "email": "john.smith@example.com"
}

PATCH

Partially updates a resource. Not necessarily idempotent.

PATCH /api/users/123 HTTP/1.1
Host: example.com
Content-Type: application/json

{
  "email": "newemail@example.com"
}

DELETE

Removes a resource. Idempotent.

DELETE /api/users/123 HTTP/1.1
Host: example.com

HEAD

Similar to GET but returns only headers, no body. Useful for checking if a resource exists.

HEAD /api/users/123 HTTP/1.1
Host: example.com

OPTIONS

Returns allowed HTTP methods for a resource. Used in CORS preflight requests.

OPTIONS /api/users HTTP/1.1
Host: example.com

Method Comparison Table

Method	Safe	Idempotent	Cacheable	Has Body
GET	✅	✅	✅	❌
POST	❌	❌	⚠️	✅
PUT	❌	✅	❌	✅
PATCH	❌	❌	❌	✅
DELETE	❌	✅	❌	❌
HEAD	✅	✅	✅	❌
OPTIONS	✅	✅	❌	❌

---

5. HTTP Status Codes

Status codes indicate the result of the HTTP request. They’re grouped into five classes.

1xx - Informational

Request received, continuing process.

• 100 Continue: Server has received request headers, client should send body
• 101 Switching Protocols: Server is switching protocols as requested

2xx - Success

Request was successfully received, understood, and accepted.

• 200 OK: Standard successful response
• 201 Created: Resource successfully created (typically after POST)
• 202 Accepted: Request accepted but processing not complete
• 204 No Content: Success but no content to return (common with DELETE)
• 206 Partial Content: Partial resource delivered (used in range requests)

3xx - Redirection

Further action needed to complete the request.

• 301 Moved Permanently: Resource permanently moved to new URL
• 302 Found: Temporary redirect
• 304 Not Modified: Resource hasn’t changed (used with caching)
• 307 Temporary Redirect: Like 302 but method must not change
• 308 Permanent Redirect: Like 301 but method must not change

4xx - Client Errors

Request contains bad syntax or cannot be fulfilled.

• 400 Bad Request: Malformed request syntax
• 401 Unauthorized: Authentication required or failed
• 403 Forbidden: Server understood but refuses to authorize
• 404 Not Found: Resource doesn’t exist
• 405 Method Not Allowed: HTTP method not supported for resource
• 409 Conflict: Request conflicts with current state (e.g., duplicate resource)
• 422 Unprocessable Entity: Validation errors
• 429 Too Many Requests: Rate limit exceeded

5xx - Server Errors

Server failed to fulfill a valid request.

• 500 Internal Server Error: Generic server error
• 501 Not Implemented: Server doesn’t support the functionality
• 502 Bad Gateway: Invalid response from upstream server
• 503 Service Unavailable: Server temporarily unavailable (maintenance, overload)
• 504 Gateway Timeout: Upstream server didn’t respond in time

Best Practices for Backend Engineers

// Example: Proper status code usage in Express.js
app.post('/api/users', async (req, res) => {
  try {
    const user = await createUser(req.body);
    res.status(201).json(user); // 201 Created
  } catch (error) {
    if (error.name === 'ValidationError') {
      res.status(422).json({ error: error.message }); // 422 Unprocessable Entity
    } else if (error.name === 'DuplicateError') {
      res.status(409).json({ error: 'User already exists' }); // 409 Conflict
    } else {
      res.status(500).json({ error: 'Internal server error' }); // 500 Server Error
    }
  }
});

---

6. HTTP Caching

Caching stores copies of resources to reduce latency, network traffic, and server load.

Cache-Control Header

The primary header for controlling caching behavior.

Common Directives

public: Response can be cached by any cache

Cache-Control: public, max-age=3600

private: Response is for a single user, shouldn’t be cached by shared caches

Cache-Control: private, max-age=3600

no-cache: Cache must revalidate with server before using cached copy

Cache-Control: no-cache

no-store: Must not cache the response at all

Cache-Control: no-store

max-age: Maximum time (in seconds) resource is considered fresh

Cache-Control: max-age=86400

s-maxage: Like max-age but only for shared caches (CDNs)

Cache-Control: public, s-maxage=3600, max-age=600

must-revalidate: Cache must verify stale resources with origin server

Cache-Control: max-age=3600, must-revalidate

ETag (Entity Tag)

A unique identifier for a specific version of a resource.

Server Response:

HTTP/1.1 200 OK
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"
Cache-Control: max-age=3600

Client Request (Conditional):

GET /api/users/123 HTTP/1.1
If-None-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

Server Response (Not Modified):

HTTP/1.1 304 Not Modified
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

Last-Modified / If-Modified-Since

Time-based conditional requests.

Server Response:

HTTP/1.1 200 OK
Last-Modified: Wed, 01 Jan 2026 12:00:00 GMT
Cache-Control: max-age=3600

Client Request:

GET /api/users/123 HTTP/1.1
If-Modified-Since: Wed, 01 Jan 2026 12:00:00 GMT

Caching Strategies

1. Immutable Assets (CSS, JS, Images with hash)

Cache-Control: public, max-age=31536000, immutable

2. API Responses (Frequently changing)

Cache-Control: private, max-age=60, must-revalidate

3. Static Content (Rarely changing)

Cache-Control: public, max-age=86400

4. Sensitive Data (Never cache)

Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache

Backend Implementation Example

// Express.js caching middleware
app.get('/api/users/:id', async (req, res) => {
  const user = await getUser(req.params.id);
  const etag = generateETag(user);
  
  // Check if client has current version
  if (req.headers['if-none-match'] === etag) {
    return res.status(304).end(); // Not Modified
  }
  
  res.set({
    'ETag': etag,
    'Cache-Control': 'private, max-age=300' // 5 minutes
  });
  
  res.json(user);
});

---

7. Preflight Requests

Preflight requests are automatic OPTIONS requests sent by browsers before certain cross-origin requests to check if the actual request is safe to send.

When Preflight Occurs

Browsers send preflight requests for:

1. Non-simple methods: PUT, DELETE, PATCH, etc.
2. Custom headers: Authorization, X-Custom-Header, etc.
3. Non-simple Content-Type: application/json, application/xml, etc.

Simple Requests (No Preflight)

These don’t trigger preflight:

• Methods: GET, HEAD, POST
• Headers: Accept, Accept-Language, Content-Language
• Content-Type: application/x-www-form-urlencoded, multipart/form-data, text/plain

Preflight Flow

1. Browser Sends OPTIONS Request

OPTIONS /api/users HTTP/1.1
Host: api.example.com
Origin: https://frontend.example.com
Access-Control-Request-Method: DELETE
Access-Control-Request-Headers: Authorization, Content-Type

2. Server Responds with Permissions

HTTP/1.1 204 No Content
Access-Control-Allow-Origin: https://frontend.example.com
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Authorization, Content-Type
Access-Control-Max-Age: 86400

3. Browser Sends Actual Request

DELETE /api/users/123 HTTP/1.1
Host: api.example.com
Origin: https://frontend.example.com
Authorization: Bearer token123

Backend Implementation

// Express.js preflight handling
app.options('/api/*', (req, res) => {
  res.set({
    'Access-Control-Allow-Origin': 'https://frontend.example.com',
    'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, PATCH, OPTIONS',
    'Access-Control-Allow-Headers': 'Authorization, Content-Type, X-Request-ID',
    'Access-Control-Max-Age': '86400' // 24 hours
  });
  res.status(204).end();
});

Optimizing Preflight Requests

Cache preflight responses:

Access-Control-Max-Age: 86400

This tells the browser to cache the preflight response for 24 hours, reducing unnecessary OPTIONS requests.

---

8. CORS (Cross-Origin Resource Sharing)

CORS is a security mechanism that allows servers to specify which origins can access their resources. It’s enforced by browsers to prevent malicious websites from making unauthorized requests.

Same-Origin Policy

Browsers block requests to different origins by default. Same origin means:

• Same protocol (http/https)
• Same domain (example.com)
• Same port (80, 443, 3000, etc.)

Examples:

URL 1	URL 2	Same Origin?
https://example.com/page1	https://example.com/page2	✅ Yes
https://example.com	http://example.com	❌ No (protocol)
https://example.com	https://api.example.com	❌ No (subdomain)
https://example.com:443	https://example.com:3000	❌ No (port)

CORS Headers

Access-Control-Allow-Origin

Specifies which origins can access the resource.

Access-Control-Allow-Origin: https://frontend.example.com

Allow all origins (use cautiously):

Access-Control-Allow-Origin: *

Access-Control-Allow-Methods

Specifies allowed HTTP methods.

Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS

Access-Control-Allow-Headers

Specifies allowed request headers.

Access-Control-Allow-Headers: Authorization, Content-Type, X-Request-ID

Access-Control-Allow-Credentials

Allows cookies and authentication headers.

Access-Control-Allow-Credentials: true

Note: When using credentials, Access-Control-Allow-Origin cannot be *.

Access-Control-Expose-Headers

Specifies which response headers are accessible to JavaScript.

Access-Control-Expose-Headers: X-Total-Count, X-Page-Number

Access-Control-Max-Age

Specifies how long preflight results can be cached.

Access-Control-Max-Age: 86400

Backend Implementation Examples

Express.js with CORS Middleware

const cors = require('cors');

// Simple CORS - Allow all origins
app.use(cors());

// Configured CORS
app.use(cors({
  origin: 'https://frontend.example.com',
  methods: ['GET', 'POST', 'PUT', 'DELETE'],
  allowedHeaders: ['Authorization', 'Content-Type'],
  credentials: true,
  maxAge: 86400
}));

// Dynamic origin validation
app.use(cors({
  origin: function(origin, callback) {
    const allowedOrigins = [
      'https://frontend.example.com',
      'https://app.example.com'
    ];
    
    if (!origin || allowedOrigins.includes(origin)) {
      callback(null, true);
    } else {
      callback(new Error('Not allowed by CORS'));
    }
  },
  credentials: true
}));

Manual CORS Headers

app.use((req, res, next) => {
  res.set({
    'Access-Control-Allow-Origin': 'https://frontend.example.com',
    'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, OPTIONS',
    'Access-Control-Allow-Headers': 'Authorization, Content-Type',
    'Access-Control-Allow-Credentials': 'true'
  });
  
  // Handle preflight
  if (req.method === 'OPTIONS') {
    return res.status(204).end();
  }
  
  next();
});

Common CORS Errors

Error: “No ‘Access-Control-Allow-Origin’ header is present”

• Solution: Add the header to server responses

Error: “The ‘Access-Control-Allow-Origin’ header contains multiple values”

• Solution: Return only one origin value, not multiple

Error: “Credentials flag is true, but Access-Control-Allow-Credentials is not”

• Solution: Add Access-Control-Allow-Credentials: true header

---

9. Idempotent vs Non-Idempotent Requests

Idempotency means that making the same request multiple times produces the same result as making it once. This is crucial for reliability and retry logic.

Idempotent Methods

GET

Reading data doesn’t change server state.

GET /api/users/123

Result: Same user data returned every time (assuming no external changes).

PUT

Replacing a resource with the same data produces the same result.

PUT /api/users/123
Content-Type: application/json

{
  "name": "John Doe",
  "email": "john@example.com"
}

Result: User 123 has the same data whether you send this once or 100 times.

DELETE

Deleting a resource multiple times has the same effect.

DELETE /api/users/123

Result: First request deletes the user, subsequent requests return 404 (but the end state is the same - user doesn’t exist).

PATCH (Sometimes)

Can be idempotent depending on implementation.

Idempotent PATCH:

PATCH /api/users/123
Content-Type: application/json

{
  "email": "newemail@example.com"
}

Non-Idempotent PATCH:

PATCH /api/users/123
Content-Type: application/json

{
  "loginCount": { "$increment": 1 }
}

Non-Idempotent Methods

POST

Creating resources typically generates new IDs each time.

POST /api/users
Content-Type: application/json

{
  "name": "John Doe",
  "email": "john@example.com"
}

Result: Each request creates a new user with a different ID.

Why Idempotency Matters

1. Retry Safety

Network failures are common. Idempotent requests can be safely retried.

// Safe to retry GET, PUT, DELETE
async function fetchUserWithRetry(userId) {
  let attempts = 0;
  const maxAttempts = 3;
  
  while (attempts < maxAttempts) {
    try {
      return await fetch(`/api/users/${userId}`);
    } catch (error) {
      attempts++;
      if (attempts >= maxAttempts) throw error;
      await sleep(1000 * attempts); // Exponential backoff
    }
  }
}

2. Distributed Systems

In microservices, duplicate messages can occur. Idempotency prevents issues.

3. User Experience

Prevents duplicate charges, duplicate orders, etc.

Implementing Idempotency for POST

Use idempotency keys to make POST requests idempotent.

Client Request:

POST /api/orders
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
Content-Type: application/json

{
  "items": [...],
  "total": 99.99
}

Backend Implementation:

const processedKeys = new Map(); // In production, use Redis

app.post('/api/orders', async (req, res) => {
  const idempotencyKey = req.headers['idempotency-key'];
  
  if (!idempotencyKey) {
    return res.status(400).json({ error: 'Idempotency-Key required' });
  }
  
  // Check if we've already processed this request
  const cached = await redis.get(`idempotency:${idempotencyKey}`);
  if (cached) {
    return res.status(200).json(JSON.parse(cached));
  }
  
  // Process the order
  const order = await createOrder(req.body);
  
  // Cache the result for 24 hours
  await redis.setex(
    `idempotency:${idempotencyKey}`,
    86400,
    JSON.stringify(order)
  );
  
  res.status(201).json(order);
});

Idempotency Comparison Table

Method	Idempotent	Safe	Example Use Case
GET	✅	✅	Fetch user profile
POST	❌	❌	Create new order
PUT	✅	❌	Update entire user profile
PATCH	⚠️	❌	Update user email (idempotent) / Increment counter (not)
DELETE	✅	❌	Delete user account
HEAD	✅	✅	Check if resource exists
OPTIONS	✅	✅	Check allowed methods

---

10. HTTP Compression

HTTP compression reduces the size of response bodies, improving transfer speed and reducing bandwidth usage.

Common Compression Algorithms

1. Gzip

Most widely supported, good compression ratio.

Content-Encoding: gzip

2. Brotli

Better compression than gzip, supported by modern browsers.

Content-Encoding: br

3. Deflate

Older algorithm, less common.

Content-Encoding: deflate

How Compression Works

1. Client Advertises Support

GET /api/users HTTP/1.1
Host: example.com
Accept-Encoding: gzip, deflate, br

2. Server Compresses Response

HTTP/1.1 200 OK
Content-Type: application/json
Content-Encoding: gzip
Content-Length: 1234
Vary: Accept-Encoding

[compressed binary data]

3. Client Decompresses

Browser automatically decompresses the response.

What to Compress

✅ Compress:

• Text-based formats: HTML, CSS, JavaScript, JSON, XML, SVG
• Large responses (> 1KB)

❌ Don’t Compress:

• Already compressed formats: JPEG, PNG, MP4, ZIP, GZIP files
• Small responses (< 1KB) - compression overhead isn’t worth it
• Streaming responses in some cases

Backend Implementation

Express.js with Compression Middleware

const compression = require('compression');

// Basic compression
app.use(compression());

// Configured compression
app.use(compression({
  // Only compress responses larger than 1KB
  threshold: 1024,
  
  // Compression level (0-9, higher = better compression but slower)
  level: 6,
  
  // Filter function to decide what to compress
  filter: (req, res) => {
    // Don't compress if client doesn't support it
    if (req.headers['x-no-compression']) {
      return false;
    }
    
    // Use compression filter function
    return compression.filter(req, res);
  }
}));

Manual Compression

const zlib = require('zlib');

app.get('/api/users', async (req, res) => {
  const users = await getUsers();
  const json = JSON.stringify(users);
  
  // Check if client accepts gzip
  const acceptEncoding = req.headers['accept-encoding'] || '';
  
  if (acceptEncoding.includes('gzip')) {
    zlib.gzip(json, (err, compressed) => {
      if (err) {
        return res.status(500).send('Compression error');
      }
      
      res.set({
        'Content-Encoding': 'gzip',
        'Content-Type': 'application/json',
        'Vary': 'Accept-Encoding'
      });
      
      res.send(compressed);
    });
  } else {
    res.json(users);
  }
});

Compression Performance

Example: JSON API response

Format	Size	Reduction
Uncompressed	100 KB	-
Gzip	15 KB	85%
Brotli	12 KB	88%

Best Practices

1. Use Vary Header

Tell caches that compressed and uncompressed versions are different.

Vary: Accept-Encoding

2. Pre-compress Static Assets

For production builds, pre-compress files during build time.

# Generate .gz files
gzip -k -9 dist/*.js dist/*.css

# Generate .br files
brotli -k -9 dist/*.js dist/*.css

3. Configure Nginx

gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml;
gzip_comp_level 6;

# Brotli (requires module)
brotli on;
brotli_types text/plain text/css application/json application/javascript text/xml application/xml;
brotli_comp_level 6;

4. Monitor Compression Ratio

app.use((req, res, next) => {
  const originalSend = res.send;
  
  res.send = function(data) {
    const originalSize = Buffer.byteLength(data);
    console.log(`Original size: ${originalSize} bytes`);
    
    originalSend.call(this, data);
    
    const compressedSize = res.get('Content-Length');
    if (compressedSize) {
      const ratio = ((1 - compressedSize / originalSize) * 100).toFixed(2);
      console.log(`Compressed size: ${compressedSize} bytes (${ratio}% reduction)`);
    }
  };
  
  next();
});

Security Considerations

BREACH Attack: Compression can leak information in HTTPS responses. Mitigate by:

• Adding random padding to responses
• Disabling compression for sensitive data
• Using CSRF tokens

// Disable compression for sensitive endpoints
app.get('/api/sensitive-data', (req, res, next) => {
  res.set('X-No-Compression', '1');
  next();
}, getSensitiveData);

---

Conclusion

Understanding HTTP deeply is fundamental for backend engineering. These concepts form the foundation for:

• Building RESTful APIs: Proper use of methods, status codes, and headers
• Performance Optimization: Caching and compression strategies
• Security: HTTPS, CORS, and secure header practices
• Reliability: Idempotency and proper error handling
• Scalability: Efficient use of HTTP features

Key Takeaways

1. Always use HTTPS in production for security
2. Choose appropriate HTTP methods and status codes
3. Implement caching strategies to reduce server load
4. Handle CORS properly for cross-origin requests
5. Make operations idempotent where possible for reliability
6. Enable compression to reduce bandwidth and improve performance
7. Understand preflight requests to optimize cross-origin communication

Further Learning

• HTTP/2 and HTTP/3: Modern protocol versions with multiplexing and improved performance
• WebSockets: Full-duplex communication over HTTP
• Server-Sent Events (SSE): Server push over HTTP
• GraphQL: Alternative to REST APIs
• gRPC: High-performance RPC framework

---

Additional Resources

• MDN Web Docs - HTTP
• RFC 7231 - HTTP/1.1 Semantics
• HTTP Status Codes
• Can I Use - Browser Compatibility