Requests - Working with Requests

Requests - Making HTTP Requests

The primary purpose of the Requests library is to simplify the process of making HTTP requests to web servers and handling their responses. This chapter covers the fundamentals: making basic GET requests, passing URL parameters, and inspecting the various parts of the response.

Making a Simple GET Request

The GET method is the most common HTTP method, used to retrieve data from a specified resource. To make a GET request, first import the requests module:

import requests

Then, use the requests.get() function, providing the URL of the resource you want to access. This function returns a Response object, which contains the server’s response to your request.

import requests

# A common public API for examples
url = 'https://jsonplaceholder.typicode.com/todos/1'

try:
    response = requests.get(url)
    # The 'response' object itself gives some info when printed
    print(f'Response Object: {response}')
except requests.exceptions.RequestException as e:
    print(f'Request failed: {e}')

Expected Output:

Response Object: <Response [200]>

The output <Response [200]> indicates that the request was successful, and the server responded with an HTTP status code of 200 (OK).

Inspecting the Response

The Response object is rich with information about the server’s reply. Key attributes and methods include:

• Status Code: response.status_code (integer) provides the HTTP status code. The response.ok boolean property is True if status_code is less than 400 (i.e., not an error).
• Headers: response.headers is a case-insensitive dictionary-like object containing the response headers (e.g., Content-Type, Date, Server).
• Text Content: response.text returns the response body as a string. Requests attempts to decode it using an appropriate encoding (often found in headers or defaulting to UTF-8). You can inspect or override the encoding with response.encoding.
• Byte Content: response.content returns the response body as raw bytes. This is suitable for non-textual content like images, audio, or other binary files.
• JSON Content: If the response content type is JSON (e.g., application/json), response.json() will parse it into a Python dictionary or list. It raises a requests.exceptions.JSONDecodeError if the content is not valid JSON.

It’s crucial to handle potential errors. The response.raise_for_status() method is a convenient way to raise an requests.exceptions.HTTPError if the request resulted in an unsuccessful status code (4xx for client errors or 5xx for server errors).

import requests

url = 'https://jsonplaceholder.typicode.com/todos/1'

try:
    response = requests.get(url, timeout=5) # Added a timeout
    response.raise_for_status()  # Check for HTTP errors (4xx or 5xx)

    print(f'Status Code: {response.status_code}')
    print(f'Is response OK? {response.ok}')
    print(f'Content-Type Header: {response.headers.get("Content-Type")}')

    # Get JSON data since this API returns JSON
    todo_item = response.json()
    print('\nParsed JSON Data:')
    print(todo_item)
    print(f"Item Title: {todo_item.get('title')}")

    # Example of accessing raw text (though json() is better for JSON data)
    # print('\nRaw Text Data (first 100 chars):')
    # print(response.text[:100] + '...' if len(response.text) > 100 else response.text)

except requests.exceptions.HTTPError as http_err:
    print(f'HTTP error occurred: {http_err} - Status Code: {http_err.response.status_code}')
except requests.exceptions.JSONDecodeError as json_err:
    print(f'JSON decode error: {json_err}')
except requests.exceptions.RequestException as e:
    print(f'An error occurred: {e}')

Expected Output:

Status Code: 200
Is response OK? True
Content-Type Header: application/json; charset=utf-8

Parsed JSON Data:
{
  'userId': 1,
  'id': 1,
  'title': 'delectus aut autem',
  'completed': False
}
Item Title: delectus aut autem

Passing URL Parameters (Query String)

Often, you need to send data to a server as part of the URL’s query string (e.g., example.com/search?query=python&page=1). Requests makes this easy by allowing you to pass a dictionary of parameters to the params argument of the requests.get() method.

import requests

# Using httpbin.org/get, which reflects the request's arguments
base_url = 'https://httpbin.org/get'

query_parameters = {'name': 'Alice', 'age': 30, 'city': 'Wonderland'}

try:
    response = requests.get(base_url, params=query_parameters, timeout=5)
    response.raise_for_status()

    print(f'Constructed Request URL: {response.url}')
    print('\nServer saw these arguments (from response JSON):')
    # httpbin.org/get returns a JSON where 'args' contains the query parameters
    print(response.json().get('args'))

except requests.exceptions.RequestException as e:
    print(f'An error occurred: {e}')

Expected Output:

Constructed Request URL: https://httpbin.org/get?name=Alice&age=30&city=Wonderland

Server saw these arguments (from response JSON):
{'name': 'Alice', 'age': '30', 'city': 'Wonderland'}
(Note: Values in 'args' are strings as they are parsed from the URL)

Requests automatically URL-encodes the parameters and constructs the full query string.

Beyond GET: Other HTTP Methods (e.g., POST)

Requests fully supports other HTTP methods like POST, PUT, DELETE, HEAD, and OPTIONS. For instance, POST requests are commonly used to send data to a server to create or update a resource.

You can send data in the request body. For form-encoded data, use the data parameter with a dictionary. For JSON payloads, it’s even simpler with the json parameter:

import requests

# httpbin.org/post reflects the POSTed data
url_post = 'https://httpbin.org/post'

# Sending JSON data
json_payload = {'username': 'testuser', 'email': 'test@example.com', 'active': True}

try:
    response = requests.post(url_post, json=json_payload, timeout=5)
    response.raise_for_status()

    print('JSON POST Response Details:')
    print(f'Status Code: {response.status_code}')

    # httpbin.org/post returns a JSON that includes the parsed JSON payload under 'json'
    returned_data = response.json()
    print(f'Sent JSON payload as received by server: {returned_data.get("json")}')
    print(f'Request Content-Type header: {returned_data.get("headers", {}).get("Content-Type")}')

except requests.exceptions.RequestException as e:
    print(f'An error occurred during POST: {e}')

Expected Output:

JSON POST Response Details:
Status Code: 200
Sent JSON payload as received by server: {'username': 'testuser', 'email': 'test@example.com', 'active': True}
Request Content-Type header: application/json

When you use the json parameter, Requests automatically serializes your Python dictionary to a JSON string and sets the Content-Type header to application/json.

Specifying Timeouts

Network operations can sometimes be slow or hang indefinitely. To prevent your program from waiting forever, always specify a timeout for your requests. The timeout parameter in Requests can be a single float (representing seconds for both connect and read timeouts combined) or a tuple (connect_timeout, read_timeout).

import requests

# httpbin.org/delay/N endpoint delays the response by N seconds
url_delayed = 'https://httpbin.org/delay/3'

try:
    # Timeout set to 2 seconds, which is less than the server's 3-second delay
    print('Attempting request with 2s timeout (expected to time out)...')
    response = requests.get(url_delayed, timeout=2)
    print('Request successful (this should not be reached).')
except requests.exceptions.Timeout:
    print('The request timed out as expected after 2 seconds.')
except requests.exceptions.RequestException as e:
    print(f'An unexpected error occurred: {e}')

try:
    # Timeout set to 5 seconds, which is more than the server's 3-second delay
    print('\nAttempting request with 5s timeout (expected to succeed)...')
    response = requests.get(url_delayed, timeout=5)
    response.raise_for_status()
    print(f'Request successful within 5 seconds. Status: {response.status_code}')
except requests.exceptions.Timeout:
    print('The request timed out (this should not happen with a 5s timeout for a 3s delay).')
except requests.exceptions.RequestException as e:
    print(f'An error occurred: {e}')

Expected Output:

Attempting request with 2s timeout (expected to time out)...
The request timed out as expected after 2 seconds.

Attempting request with 5s timeout (expected to succeed)...
Request successful within 5 seconds. Status: 200

Setting appropriate timeouts is a best practice for writing robust network applications.