HTTP URL Encoding

HTTP - URL Encoding (Percent-Encoding)

URLs are designed to be transmitted over the internet, which historically meant using the ASCII character set. While modern Internationalized Resource Identifiers (IRIs) support a wider range of characters, the process of converting an IRI to a URI often involves encoding characters that are not in the “unreserved” set or have special meaning within a URI. This encoding process is called percent-encoding.

Percent-encoding ensures that URLs are uniformly interpreted across different systems. Unsafe or special characters are replaced with a ’%’ symbol followed by two hexadecimal digits representing the character’s byte value. For characters outside the US-ASCII range (e.g., accented letters, Chinese characters), they are typically first encoded into a sequence of bytes using UTF-8, and then each of those bytes is percent-encoded.

Key Concepts:

• Reserved Characters: Characters like ’:’, ’/’, ’?’, ’#’, ’[’, ’]’, ’@’, ’!’, ’$’, ’&’, ''', ’(’, ’)’, ’*’, ’+’, ’,’, ’;’, ’=’ have special meanings in a URI. If these characters are part of data (e.g., a query parameter value) and not used for their special purpose, they MUST be percent-encoded.
• Unreserved Characters: Characters like A-Z, a-z, 0-9, ’-’, ’_’, ’.’, ’~’ do not need to be encoded.
• Space Character: A space is typically encoded as ‘%20’. In the query string part of a URL, it can also be encoded as ’+’ (plus sign), though ‘%20’ is universally accepted in all parts of a URI.
• Non-ASCII Characters: Convert to UTF-8 bytes, then percent-encode each byte. For example, ‘é’ (UTF-8: 0xC3 0xA9) becomes ‘%C3%A9’.

Example in JavaScript:

Most programming languages provide built-in functions for URL encoding. In JavaScript, encodeURIComponent() is commonly used for encoding query parameter values:

let paramValue = "search query with spaces & symbols!";
let encodedValue = encodeURIComponent(paramValue);
console.log(encodedValue);
// Output: search%20query%20with%20spaces%20%26%20symbols!

Commonly Encoded Characters:

The following table shows some common characters and their percent-encoded representation. Note that characters not listed but falling into ‘reserved’ or non-ASCII categories follow the same encoding rules.

Character Name	Symbol	Percent-Encoded
Control Characters (ASCII < 32)	Varies (e.g., newline)	Encode with %xx (e.g., %0A for newline)
Space	’ ‘	%20 (or + in query string)
Percent	’%‘	%25
Ampersand	’&‘	%26 (especially in query parameters)
Plus	’+‘	%2B (if ’+’ itself is data)
Forward Slash	’/‘	%2F (if part of a path segment, not a delimiter)
Question Mark	’?’	%3F (if part of data, not query delimiter)
Hash / Pound	’#‘	%23 (if part of data, not fragment delimiter)
Equals	’=‘	%3D (especially in query parameters)
Non-ASCII (e.g., ’€‘)	Euro sign	%E2%82%AC (UTF-8 bytes C2 82 AC, then percent-encoded)

For a comprehensive understanding, refer to RFC 3986, which defines URIs.