Requests - Working with Cookies

Requests - Working with Cookies

HTTP cookies are small pieces of data that servers send to a user’s web browser. The browser stores these cookies and sends them back to the same server with subsequent requests. Cookies are widely used for session management (e.g., keeping users logged in), personalization, and tracking. The Requests library simplifies cookie handling in Python.

Receiving and Inspecting Cookies

When a server includes Set-Cookie headers in its response, Requests automatically processes them. These cookies are stored in the response.cookies attribute, which is a requests.cookies.RequestsCookieJar object. This object behaves much like a dictionary.

Let’s use httpbin.org, a helpful service for testing HTTP requests, to see how Requests handles cookies set by a server.

import requests

# This endpoint sets a cookie named 'my_cookie_name' with value 'my_cookie_value'
url_set_cookie = 'https://httpbin.org/cookies/set/my_cookie_name/my_cookie_value'

response = requests.get(url_set_cookie)

print('Cookies received from the server after initial set:')
if response.cookies:
    for cookie_name, cookie_value in response.cookies.items():
        print(f'- Name: {cookie_name}, Value: {cookie_value}')

    # Access a specific cookie by name
    if 'my_cookie_name' in response.cookies:
        print(f"Value of 'my_cookie_name': {response.cookies['my_cookie_name']}")
else:
    print('No cookies were set or visible on the final response object.')

# Note: httpbin.org/cookies/set/... redirects. The cookie is set on the initial response,
# which becomes part of response.history. The final response (after redirect) might not show it directly
# in its .cookies attribute if the redirect is to a different path not sending cookies.
# However, if using a Session, the cookie would be persisted and sent on subsequent requests to the domain.

# Let's verify by immediately asking httpbin what cookies it received (it won't see the one above without a session)
check_url = 'https://httpbin.org/cookies'
check_response_no_session = requests.get(check_url)
print(f'\nCookies seen by {check_url} without session: {check_response_no_session.json()}')

Expected Output (varies slightly based on httpbin behavior):

Cookies received from the server after initial set:
- Name: my_cookie_name, Value: my_cookie_value
Value of 'my_cookie_name': my_cookie_value

Cookies seen by https://httpbin.org/cookies without session: {'cookies': {}}

The response.cookies object contains cookies from the final response in a redirect chain. If cookies are set on an earlier response during redirects, they might be in response.history[i].cookies. Session objects (covered later) make managing this transparent.

Sending Custom Cookies

You can send your own cookies to the server by passing a dictionary to the cookies parameter of a request function (like requests.get() or requests.post()).

import requests

url_check_cookies = 'https://httpbin.org/cookies'
custom_cookies_to_send = {'user_preference': 'dark_theme', 'language': 'en-US'}

response = requests.get(url_check_cookies, cookies=custom_cookies_to_send)

print('Server received the following custom cookies:')
print(response.json())  # httpbin.org/cookies returns a JSON of cookies it received

Expected Output:

Server received the following custom cookies:
{
  'cookies': {
    'language': 'en-US',
    'user_preference': 'dark_theme'
  }
}

Persistent Cookies with Session Objects

For scenarios requiring cookie persistence across multiple requests (e.g., maintaining a login session, tracking preferences), requests.Session objects are the ideal solution. A Session object automatically manages cookies: any cookies set by the server are stored in the session and sent with subsequent requests to the same domain.

import requests

# Create a Session object. It's good practice to use it as a context manager.
with requests.Session() as session:
    # First, let the server set a cookie within this session
    # This endpoint will set 'session_cookie_key' to 'session_cookie_value'
    session.get('https://httpbin.org/cookies/set/session_cookie_key/session_cookie_value')
    print("A session cookie should now be set internally by the session object.")

    # Now, make another request to an endpoint that shows received cookies.
    # The session will automatically send the cookie it received earlier.
    response_with_session_cookie = session.get('https://httpbin.org/cookies')

    print('\nCookies sent by the session and received by httpbin.org/cookies:')
    received_cookies_data = response_with_session_cookie.json()
    print(received_cookies_data)

    if 'session_cookie_key' in received_cookies_data.get('cookies', {}):
        value = received_cookies_data['cookies']['session_cookie_key']
        print(f"Successfully received 'session_cookie_key' with value: {value}")

Expected Output:

A session cookie should now be set internally by the session object.

Cookies sent by the session and received by httpbin.org/cookies:
{
  'cookies': {
    'session_cookie_key': 'session_cookie_value'
  }
}
Successfully received 'session_cookie_key' with value: session_cookie_value

Using Session objects is the recommended way to handle stateful interactions like logins or multi-step processes that rely on cookies.

Inspecting Cookies in Your Browser (For Context)

While not directly related to the Requests library, understanding how to view cookies in a web browser can be helpful. You can typically inspect cookies using your browser’s developer tools (often accessible by pressing F12). Look for tabs like ‘Application’ (in Chrome/Edge), ‘Storage’ (in Firefox), or check individual requests in the ‘Network’ tab to see Set-Cookie (response) and Cookie (request) headers.

Mastering cookie handling with Requests is essential for effective web scraping, API interaction, and automating web-based tasks.