Requests - Proxy

Requests: Routing Traffic Through Proxies

A proxy server acts as an intermediary for requests from clients seeking resources from other servers. Using a proxy can be necessary for various reasons, such as accessing resources within a restricted network, enhancing security, anonymity, or for development and testing purposes. The requests library provides straightforward support for routing your HTTP(S) requests through a proxy.

How Proxies Work

When you use a proxy with requests:

1. Your client (Python script using requests) sends the request to the proxy server.
2. The proxy server forwards your request to the target server (e.g., httpbin.org).
3. The target server sends the response back to the proxy server.
4. The proxy server relays this response back to your client.

Configuring Proxies in requests

You can specify proxies by passing a dictionary to the proxies parameter in your request methods (e.g., requests.get(), requests.post()). The dictionary should map URL schemes (like ‘http’ or ‘https’) to the URL of the proxy server.

Example: Basic HTTP and HTTPS Proxy

To use a proxy, you need its address and port. For this example, let’s assume you have an HTTP proxy at http://myproxy.example.com:8080 and an HTTPS proxy (which might itself be an HTTP proxy capable of tunneling HTTPS traffic) at http://secureproxy.example.com:8888.

import requests

# Replace with your actual proxy addresses and ports
# If your HTTPS proxy is also an HTTP proxy, the scheme is 'http' for its address
proxies_config = {
    'http': 'http://myproxy.example.com:8080',  # For HTTP requests
    'https': 'http://secureproxy.example.com:8888' # For HTTPS requests
}

# Test with an HTTP URL
try:
    print("Attempting HTTP request via proxy...")
    response_http = requests.get('http://httpbin.org/ip', proxies=proxies_config)
    print(f"HTTP Request Status: {response_http.status_code}")
    print(f"Origin IP (should be proxy's IP): {response_http.json()['origin']}")
except requests.exceptions.ProxyError as e:
    print(f"HTTP Proxy Error: {e}. Ensure your proxy is running and accessible.")
except requests.exceptions.RequestException as e:
    print(f"HTTP Request Error: {e}")

print("\n---\n")

# Test with an HTTPS URL
try:
    print("Attempting HTTPS request via proxy...")
    response_https = requests.get('https://httpbin.org/ip', proxies=proxies_config)
    print(f"HTTPS Request Status: {response_https.status_code}")
    print(f"Origin IP (should be proxy's IP): {response_https.json()['origin']}")
except requests.exceptions.ProxyError as e:
    print(f"HTTPS Proxy Error: {e}. Ensure your proxy is running and accessible.")
except requests.exceptions.RequestException as e:
    print(f"HTTPS Request Error: {e}")

Note: For these examples to work, you must have actual proxy servers running at the specified addresses and ports, and they must be configured to allow connections from your machine. If myproxy.example.com is not a real proxy, these requests will fail.

Output (if proxies are working and httpbin.org/ip is reached)

Attempting HTTP request via proxy...
HTTP Request Status: 200
Origin IP (should be proxy's IP): <IP_ADDRESS_OF_HTTP_PROXY>

---

Attempting HTTPS request via proxy...
HTTPS Request Status: 200
Origin IP (should be proxy's IP): <IP_ADDRESS_OF_HTTPS_PROXY>

The httpbin.org/ip endpoint returns the IP address from which it received the request. If the proxy is working correctly, this IP should be that of your proxy server, not your local machine.

Proxies Requiring Authentication

If your proxy server requires a username and password, include them in the proxy URL:

proxies_auth = {
    'http': 'http://user:password@myproxy.example.com:8080',
    'https': 'http://user:password@secureproxy.example.com:8888'
}

# response = requests.get('http://httpbin.org/ip', proxies=proxies_auth)

Remember to URL-encode any special characters in your username or password if necessary.

SOCKS Proxies

requests itself does not directly support SOCKS proxies. However, you can add SOCKS proxy support by installing an extra package like PySocks:

pip install pysocks

Once installed, you can specify SOCKS proxies using socks5://, socks5h:// (for remote DNS resolution), socks4://, or socks4a:// schemes:

proxies_socks = {
    'http': 'socks5h://user:password@my-socks-proxy.example.com:1080',
    'https': 'socks5h://user:password@my-socks-proxy.example.com:1080'
}

# response = requests.get('https://httpbin.org/ip', proxies=proxies_socks)

Environment Variables for Proxies

requests will automatically try to use proxy settings from standard environment variables if the proxies argument is not provided in the request. These variables are:

• HTTP_PROXY: For HTTP traffic.
• HTTPS_PROXY: For HTTPS traffic.
• NO_PROXY: A comma-separated list of hostnames that should bypass the proxy.

For example, in your shell:

# On Linux/macOS
export HTTP_PROXY="http://myproxy.example.com:8080"
export HTTPS_PROXY="http://secureproxy.example.com:8888"

# On Windows (Command Prompt)
set HTTP_PROXY=http://myproxy.example.com:8080
set HTTPS_PROXY=http://secureproxy.example.com:8888

If these environment variables are set, requests will use them by default without needing the proxies parameter in your code. This is often preferred for system-wide or user-specific proxy configurations.

Common Issues and Troubleshooting

• Proxy Not Running/Reachable: Ensure the proxy server is operational and accessible from your machine. Network firewalls might block connections.
• Authentication Errors: Double-check credentials if your proxy requires authentication.
• HTTPS Inspection (Man-in-the-Middle): Corporate proxies sometimes intercept and decrypt HTTPS traffic. This might require installing a custom root certificate trusted by your system or requests. See the requests documentation on SSL verification for more details (using the verify parameter).
• Incorrect Proxy Scheme: Ensure you are using the correct scheme (e.g., http://, https://, socks5://) for your proxy server type and address.

Using proxies effectively allows requests to operate in diverse network environments and for specialized tasks. Always ensure your proxy usage complies with relevant terms of service and ethical guidelines.