Requests - HTTP Requests Headers

Understanding and Using HTTP Headers with Requests

HTTP headers are a crucial part of HTTP requests and responses, carrying metadata about the message, such as content type, authentication tokens, caching directives, and more. This chapter explores how to inspect response headers and send custom request headers using the requests library.

• Inspecting Response Headers
• Sending Custom Request Headers
• Common Use Cases for Headers

Inspecting Response Headers

When you make a request, the server sends back a response that includes headers. These headers provide valuable information about the response itself. The requests library makes these headers easily accessible through the headers attribute of the Response object.

The response.headers attribute is a dictionary-like object (specifically, a CaseInsensitiveDict) that allows you to access header values. Header names are case-insensitive.

Example: Viewing All Response Headers

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')

print("All Response Headers:")
for key, value in response.headers.items():
    print(f"{key}: {value}")

Output (example - truncated for brevity)

All Response Headers:
Date: Mon, 01 Jul 2024 10:00:00 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 83
Connection: keep-alive
Cache-Control: public, max-age=14400
Etag: W/"53-hf75nVaR4L/XeNTPTY/042RfiEY"
...

Example: Accessing a Specific Header

You can access a specific header value by its name. Using the .get() method is safer as it returns None if the header is not present, avoiding a KeyError.

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')

content_type = response.headers.get('Content-Type')
print(f"Content-Type: {content_type}")

# Accessing a non-existent header with get()
non_existent_header = response.headers.get('X-Custom-Header')
print(f"X-Custom-Header: {non_existent_header}")

# Accessing using dictionary-style access (case-insensitive)
server_header = response.headers['server'] # Assuming 'server' header exists
print(f"Server: {server_header}")

Output (example)

Content-Type: application/json; charset=utf-8
X-Custom-Header: None
Server: cloudflare

Sending Custom Request Headers

Sometimes, you need to send custom headers with your request. This might be for authentication (e.g., API keys), specifying the desired content type of the response (Accept header), or providing other metadata. You can pass a dictionary of headers to the headers parameter of any request method (get, post, etc.).

Example: Sending Custom Headers

import requests

url = 'https://httpbin.org/headers' # This endpoint echoes request headers
custom_headers = {
    'X-API-Key': 'your_secret_api_key_here',
    'User-Agent': 'MyAwesomeApp/1.0',
    'Accept': 'application/json'
}

response = requests.get(url, headers=custom_headers)

print(f"Status Code: {response.status_code}")
# The response from httpbin.org/headers will contain the headers we sent.
print("Echoed Headers from Server:")
print(response.json()['headers'])

Output (example)

Status Code: 200
Echoed Headers from Server:
{
  "Accept": "application/json",
  "Accept-Encoding": "gzip, deflate, br, zstd",
  "Host": "httpbin.org",
  "User-Agent": "MyAwesomeApp/1.0",
  "X-Amzn-Trace-Id": "Root=...",
  "X-Api-Key": "your_secret_api_key_here"
}

Note that requests adds certain headers by default (like Host, Accept-Encoding, and a default User-Agent if not overridden). Your custom headers will be added to or will override these defaults.

Common Use Cases for Headers

• Authorization: Sending credentials, like API keys or bearer tokens (e.g., Authorization: Bearer <token>).
• Content-Type: (For requests like POST/PUT) Specifies the media type of the request body (e.g., application/json, application/x-www-form-urlencoded). requests often sets this automatically when using data or json parameters.
• Accept: Informs the server about the types of content the client can understand in the response (e.g., Accept: application/json).
• User-Agent: Identifies the client software making the request. It’s good practice to set a descriptive User-Agent for your application.
• If-None-Match / If-Modified-Since: Used for conditional requests to make applications more efficient by avoiding re-downloading data that hasn’t changed (related to caching and ETags).

Understanding and utilizing HTTP headers effectively is key to interacting with web services and APIs robustly and efficiently.

Further Resources

For a comprehensive list of HTTP headers and their meanings, consult the MDN Web Docs: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers