Handling Response for HTTP Requests

Requests - Handling HTTP Response Data

When you make a request using the requests library, the server sends back an HTTP response. The Response object returned by requests contains a wealth of information, including the response body, status code, headers, and more. This chapter explores how to access and work with this data.

• Accessing Response Content (Text, Bytes, JSON)
• Checking Status Codes
• Inspecting Headers
• Handling Encodings
• Working with Raw and Binary Responses

Accessing Response Content

Let’s make a request to a public API to demonstrate response handling:

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')

Text Content (response.text)

The response.text attribute provides the response body as a Unicode string. requests automatically decodes the content based on the HTTP headers (e.g., Content-Type with a charset).

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print("Response Text:")
print(response.text)

Expected Output (Text)

Response Text:
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

This is similar to viewing the ‘source’ of a webpage in a browser if the content were HTML, or the raw JSON string if it’s a JSON API.

Byte Content (response.content)

The response.content attribute provides the response body as raw bytes. This is useful for non-textual data like images, PDFs, or when you need to handle the decoding yourself.

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print("\nResponse Content (Bytes):")
print(response.content)

Expected Output (Bytes)

Response Content (Bytes):
b'{\n  "userId": 1,\n  "id": 1,\n  "title": "delectus aut autem",\n  "completed": false\n}'

Notice the b'' prefix, indicating a byte string. This is the raw data before any decoding based on character sets.

JSON Response (response.json())

If the response content is JSON, you can use the response.json() method to parse it directly into a Python dictionary or list.

import requests
import json # For potential JSONDecodeError

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
try:
    data = response.json()
    print("\nJSON Data:")
    print(data)
    print(f"Type of parsed data: {type(data)}")
    print(f"Title from JSON: {data['title']}")
except json.JSONDecodeError:
    print("\nFailed to decode JSON.")

Expected Output (JSON)

JSON Data:
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}
Type of parsed data: <class 'dict'>
Title from JSON: delectus aut autem

If the response is not valid JSON, response.json() will raise a json.JSONDecodeError (or requests.exceptions.JSONDecodeError in newer versions of requests, which subclasses the former).

Checking Status Codes

The response.status_code attribute gives you the HTTP status code of the response. It’s essential for checking if the request was successful (e.g., 200 OK) or encountered an error (e.g., 404 Not Found, 500 Internal Server Error).

import requests

response_ok = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print(f"Status for /todos/1: {response_ok.status_code}")

response_not_found = requests.get('https://jsonplaceholder.typicode.com/nonexistent')
print(f"Status for /nonexistent: {response_not_found.status_code}")

# Best practice: Raise an exception for bad status codes (4xx or 5xx)
try:
    response_not_found.raise_for_status()
except requests.exceptions.HTTPError as e:
    print(f"HTTP error occurred for /nonexistent: {e}")

Expected Output (Status Codes)

Status for /todos/1: 200
Status for /nonexistent: 404
HTTP error occurred for /nonexistent: 404 Client Error: Not Found for url: https://jsonplaceholder.typicode.com/nonexistent

response.raise_for_status() is a convenient way to assert that a request was successful.

Inspecting Headers

HTTP headers provide metadata about the response. You can access them via response.headers. It’s a case-insensitive dictionary-like object.

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print("\nResponse Headers:")
for key, value in response.headers.items():
    print(f"  {key}: {value}")

print(f"\nContent-Type Header: {response.headers['Content-Type']}")
# Access is case-insensitive
print(f"content-type Header (lowercase key): {response.headers['content-type']}")

Expected Output (Headers - truncated)

Response Headers:
  Date: ...
  Content-Type: application/json; charset=utf-8
  Content-Length: ...
  Connection: keep-alive
  ...

Content-Type Header: application/json; charset=utf-8
content-type Header (lowercase key): application/json; charset=utf-8

Handling Encodings

requests attempts to guess the encoding of the response content based on headers. You can inspect this with response.encoding.

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print(f"\nGuessed Encoding: {response.encoding}")

# If requests guesses incorrectly, or if no charset is specified,
# you can set the encoding manually before accessing .text
# response.encoding = 'ISO-8859-1' # Example
# print(f"Manually set Encoding: {response.encoding}")
# print(response.text) # .text will now use the new encoding

Expected Output (Encoding)

Guessed Encoding: utf-8

Raw and Binary Responses

Raw Response (response.raw)

For low-level access to the response stream, you can use response.raw. This requires setting stream=True in your request. response.raw is a file-like object (specifically, an urllib3.response.HTTPResponse instance).

import requests

# Example: Downloading a small image in chunks
# Using a placeholder for an image URL, replace with a real one if needed
# For demonstration, we'll use a text endpoint with streaming
response = requests.get('https://jsonplaceholder.typicode.com/todos/1', stream=True)

print(f"\nRaw response object: {response.raw}")

# You can read data from the raw stream
# It's good practice to release the connection when done with raw
with response:
    try:
        # Read a small chunk of the raw response
        raw_chunk = response.raw.read(50) # Read up to 50 bytes
        print(f"First 50 bytes of raw response: {raw_chunk}")
        # In a real scenario, you might write this to a file in chunks
        # for chunk in response.iter_content(chunk_size=8192):
        #     if chunk: # filter out keep-alive new chunks
        #         # process_chunk (e.g. f.write(chunk))
        #         pass
    except Exception as e:
        print(f"Error reading raw stream: {e}")

Expected Output (Raw Response - bytes will match the JSON content)

Raw response object: <urllib3.response.HTTPResponse object at 0x...>
First 50 bytes of raw response: b'{\n  "userId": 1,\n  "id": 1,\n  "title": "delectus aut au'

Streaming (stream=True) is particularly useful for downloading large files or handling streaming APIs where you don’t want to load the entire response into memory at once.

Binary Response (Revisited: response.content)

As mentioned earlier, response.content gives you the response body in bytes. This is the most direct way to get binary data, such as images, audio files, or any non-text content. It’s suitable when the entire content can fit comfortably in memory.

import requests

# Hypothetical image URL
# response = requests.get('https://via.placeholder.com/150.png')
# with open('image.png', 'wb') as f:
#    f.write(response.content)
# print("\nImage downloaded as image.png (using response.content)")

# For this example, let's re-show .content with the JSON API:
response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print("\nBinary content (response.content):")
print(response.content)

Expected Output (Binary Content)

Binary content (response.content):
b'{\n  "userId": 1,\n  "id": 1,\n  "title": "delectus aut autem",\n  "completed": false\n}'

Understanding how to handle different aspects of the HTTP response is key to building robust applications that interact with web services. For more details, refer to the Requests Quickstart and Advanced Usage sections in the official documentation.