Requests - How Http Requests Work?

How HTTP Requests Work with Python’s requests Library

The Hypertext Transfer Protocol (HTTP) is the foundation of data communication for the World Wide Web. When you browse a website, your browser sends HTTP requests to a server, and the server replies with HTTP responses. Python’s requests library is a powerful yet simple tool that allows you to send these HTTP requests programmatically, making it easy to interact with web services and APIs.

This interaction follows a client-server model. Your Python script, using the requests library, acts as the client. It constructs an HTTP request and sends it to a web server. The server processes this request and sends back an HTTP response, which might contain the data you asked for (like a webpage or JSON data), or information about whether your request was successful.

The Anatomy of an HTTP Request

An HTTP request has several key components:

1. The URL (Uniform Resource Locator)

This is the address of the resource you want to interact with on the web. For example: https://api.example.com/data. It specifies the protocol (e.g., https or http), the hostname (e.g., api.example.com), and the path to the resource (e.g., /data).

2. The HTTP Method

This defines the action to be performed on the resource. Common methods include:

GET: Retrieves data from the server. For example, fetching a webpage or user information.

POST: Sends data to the server to create a new resource. For example, submitting a form or creating a new user.

PUT: Sends data to the server to update an existing resource or create it if it doesn’t exist. For example, updating a user’s profile.

DELETE: Removes a resource from the server. For example, deleting a user’s account.

3. Headers

These provide metadata about the request, such as the type of content being sent (Content-Type) or authentication tokens (Authorization). The requests library often handles common headers automatically, but you can customize them if needed.

4. Body (Payload)

This contains the actual data being sent to the server, typically used with POST or PUT requests. This could be form data, JSON, XML, or other formats.

Making Requests with Python’s requests Library

First, ensure you have the requests library installed (pip install requests) and then import it into your Python script:

import requests

GET Requests: Fetching Data

GET requests are used to retrieve information. Here’s a basic example:

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')
print(response.text) # Print the response content as text

To send data as query parameters in the URL (e.g., https://api.example.com/users?id=9&username=Delphine), use the params argument with a dictionary:

payload = {'id': '9', 'username': 'Delphine'}
response = requests.get('https://jsonplaceholder.typicode.com/users', params=payload)
# The actual URL requested will be: https://jsonplaceholder.typicode.com/users?id=9&username=Delphine
print(response.url) # You can inspect the URL that was actually requested

The requests library automatically URL-encodes the params dictionary and appends it to the URL.

POST Requests: Sending Data

POST requests are used to send data to a server, often to create a new resource.

To send data as if it’s from an HTML form (form-encoded, application/x-www-form-urlencoded), use the data parameter:

form_payload = {'title': 'My New Post', 'body': 'This is the content.', 'userId': 1}
response = requests.post('https://jsonplaceholder.typicode.com/posts', data=form_payload)
print(response.status_code) # Check the status code (e.g., 201 for created)
print(response.json())      # See the server's response (often the created object)

For sending JSON data, which is common for modern APIs, use the json parameter. requests will automatically serialize your Python dictionary to a JSON string and set the Content-Type header to application/json:

json_payload = {'title': 'Another Post', 'body': 'JSON content here.', 'userId': 5}
response = requests.post('https://jsonplaceholder.typicode.com/posts', json=json_payload)
print(response.status_code)
print(response.json())

PUT Requests: Updating Data

PUT requests are typically used to update an existing resource at a specific URL. If the resource doesn’t exist, PUT might create it, depending on the server’s implementation.

update_payload = {'id': 1, 'title': 'Updated Title', 'body': 'Updated body content.', 'userId': 1}
# The resource ID (e.g., 1) is often part of the URL for PUT requests.
response = requests.put('https://jsonplaceholder.typicode.com/posts/1', json=update_payload)
print(response.status_code)
print(response.json())

DELETE Requests: Removing Data

DELETE requests are used to remove a resource identified by a URL.

# The resource ID (e.g., 1) is part of the URL.
response = requests.delete('https://jsonplaceholder.typicode.com/posts/1')
print(response.status_code) # Should be 200 or 204 if successful (server dependent)
# DELETE requests often return an empty body or a success message.

Understanding the Response

After making a request, the response object returned by requests contains the server’s response. Here are some important attributes:

Status Codes

The response.status_code attribute provides the HTTP status code, indicating the outcome of the request. Common codes include:

200 OK: The request was successful.

201 Created: The request was successful, and a new resource was created (common for POST/PUT).

204 No Content: The request was successful, but there’s no content to return (common for DELETE).

400 Bad Request: The server could not understand the request due to invalid syntax.

401 Unauthorized: Authentication is required and has failed or has not yet been provided.

403 Forbidden: The server understood the request, but refuses to authorize it.

404 Not Found: The requested resource could not be found on the server.

500 Internal Server Error: The server encountered an unexpected condition that prevented it from fulfilling the request.

It’s good practice to check the status code. You can use response.raise_for_status() which will raise an HTTPError if the request returned an unsuccessful status code (4xx or 5xx).

For a comprehensive list, refer to the MDN Web Docs on HTTP status codes.

Response Content

You can access the body of the response in various ways:

response.text: The content of the response, in Unicode. requests attempts to decode it from the server’s specified encoding.

response.content: The content of the response, in bytes. Useful for non-textual responses like images or files.

response.json(): If the response content is JSON, this method will parse it into a Python dictionary or list. It will raise a JSONDecodeError if the content is not valid JSON.

import requests

response = requests.get('https://jsonplaceholder.typicode.com/todos/1')

if response.status_code == 200:
    try:
        data = response.json()
        print(f"Title: {data['title']}")
    except requests.exceptions.JSONDecodeError:
        print("Response was not in JSON format")
else:
    print(f"Request failed with status code: {response.status_code}")

Response Headers

response.headers is a dictionary-like object containing the response headers from the server (e.g., Content-Type, Date, Server). Headers are case-insensitive.

HTTPS and Security

requests handles HTTPS (HTTP Secure) connections seamlessly. When you make a request to an https:// URL, requests automatically verifies the server’s SSL certificate, ensuring your communication is encrypted and secure. This is a critical feature for protecting sensitive data.

Key Takeaways

HTTP is the protocol for client-server communication on the web, defining methods like GET, POST, PUT, and DELETE.

Python’s requests library simplifies making these HTTP requests.

Use params for GET request query strings, data for form-encoded POST/PUT bodies, and json for JSON-encoded POST/PUT bodies.

Always check response.status_code (or use response.raise_for_status()) and process response content using .text, .content, or .json() as appropriate.

This foundational understanding of HTTP requests and the requests library will empower you to interact with a vast array of web services, APIs, and retrieve diverse data from the internet.