Sed Strings

Sed - Manipulating Strings

Substitute Command (s)

The substitute command, s, is one of Sed’s most powerful features, used for finding and replacing text. Its basic syntax is:

[address1[,address2]]s/pattern/replacement/[flags]

Where:

address1, address2: Optional line addresses or patterns to restrict the command’s scope.
s: The substitute command.
/: A delimiter. While / is common, any character (except backslash or newline) can be used, e.g., s|pattern|replacement| or s#pattern#replacement#. This is useful if pattern or replacement contains slashes.
pattern: A regular expression to search for.
replacement: The string to replace the matched pattern with. Special characters like & (the entire matched pattern) and \1 through \9 (captured groups) can be used here.
flags: Optional modifiers that change the substitution behavior.

Common Flags:

g: Global. Replace all occurrences of pattern on the line, not just the first.
N (a number): Replace only the Nth occurrence.
p: Print. If a substitution occurs, print the (modified) line. Often used with -n to print only changed lines.
w file: Write. If a substitution occurs, write the (modified) line to file.
i or I (GNU Sed extension): Case-insensitive matching for pattern.

Let’s use a sample books.txt file:

A Storm of Swords, George R. R. Martin, 1216
The Two Towers, J. R. R. Tolkien, 352
The Alchemist, Paulo Coelho, 197
The Fellowship of the Ring, J. R. R. Tolkien, 432
The Pilgrimage, Paulo Coelho, 288
A Game of Thrones, George R. R. Martin, 864

Example 1: Replace the first comma with a pipe.

[user]$ sed 's/,/ | /' books.txt

Output:

1) A Storm of Swords | George R. R. Martin, 1216
2) The Two Towers | J. R. R. Tolkien, 352
3) The Alchemist | Paulo Coelho, 197
4) The Fellowship of the Ring | J. R. R. Tolkien, 432
5) The Pilgrimage | Paulo Coelho, 288
6) A Game of Thrones | George R. R. Martin, 864

Notice only the first comma on each line is replaced. To replace all commas, use the g flag:

[user]$ sed 's/,/ | /g' books.txt

Output:

1) A Storm of Swords | George R. R. Martin | 1216
2) The Two Towers | J. R. R. Tolkien | 352
3) The Alchemist | Paulo Coelho | 197
4) The Fellowship of the Ring | J. R. R. Tolkien | 432
5) The Pilgrimage | Paulo Coelho | 288
6) A Game of Thrones | George R. R. Martin | 864

Example 2: Replace commas only on lines containing “The Pilgrimage”.

[user]$ sed '/The Pilgrimage/s/,/ | /g' books.txt

Output:

1) A Storm of Swords, George R. R. Martin, 1216
2) The Two Towers, J. R. R. Tolkien, 352
3) The Alchemist, Paulo Coelho, 197
4) The Fellowship of the Ring, J. R. R. Tolkien, 432
5) The Pilgrimage | Paulo Coelho | 288
6) A Game of Thrones, George R. R. Martin, 864

Example 3: Replace only the second comma with a pipe.

[user]$ sed 's/,/ | /2' books.txt

Output:

1) A Storm of Swords, George R. R. Martin | 1216
2) The Two Towers, J. R. R. Tolkien | 352
3) The Alchemist, Paulo Coelho | 197
4) The Fellowship of the Ring, J. R. R. Tolkien | 432
5) The Pilgrimage, Paulo Coelho | 288
6) A Game of Thrones, George R. R. Martin | 864

Example 4: Print only changed lines (using -n and p flag).

[user]$ sed -n 's/Paulo Coelho/PAULO COELHO/p' books.txt

Output:

3) The Alchemist, PAULO COELHO, 197
5) The Pilgrimage, PAULO COELHO, 288

Example 5: Write changed lines to a file (using w flag).

[user]$ sed 's/Paulo Coelho/PAULO COELHO/w changed_authors.txt' books.txt > /dev/null # suppress normal output
[user]$ cat changed_authors.txt

Output from cat changed_authors.txt:

3) The Alchemist, PAULO COELHO, 197
5) The Pilgrimage, PAULO COELHO, 288

Note: The original command sed -n 's/.../w file' would print nothing to stdout. If you want both default output and writing to file for changed lines, use sed 's/.../&/w file'. The example above explicitly redirects main output.

Example 6: Case-insensitive substitution (GNU Sed i flag).

[user]$ sed -n 's/pAuLo CoElHo/PAULO COELHO/pi' books.txt

Output:

3) The Alchemist, PAULO COELHO, 197
5) The Pilgrimage, PAULO COELHO, 288

Example 7: Using different delimiters to replace paths.

If you need to replace /usr/local/bin with /opt/bin, using / as a delimiter requires extensive escaping:

[user]$ echo "Path is /usr/local/bin" | sed 's/\/usr\/local\/bin/\/opt\/bin/'

Output:

Path is /opt/bin

Using an alternative delimiter like |, #, or ! makes it much more readable:

[user]$ echo "Path is /usr/local/bin" | sed 's|/usr/local/bin|/opt/bin|'

Output:

Path is /opt/bin

Creating Substrings with Capturing Groups

Sed allows you to capture parts of the matched pattern and reuse them in the replacement string. This is done using capturing groups.

In Basic Regular Expressions (BRE, Sed’s default), groups are defined with \( and \). The captured text can be referenced using \1 for the first group, \2 for the second, and so on.

If using Extended Regular Expressions (sed -E or sed -r), groups are ( and ), and still referenced by \1, \2, etc.

Consider the input string: “Three One Two”. We want to rearrange it to “One Two Three”.

[user]$ echo "Three One Two" | sed 's|\(\w\+\) \(\w\+\) \(\w\+\)|\2 \3 \1|'

Explanation:

\(\w\+\): This is a capturing group.
- \w\+: Matches one or more word characters (GNU Sed extension; [[:alnum:]_]+ is more portable for the “word character” concept). Escaped \+ for BRE.
The pattern \(\w\+\) \(\w\+\) \(\w\+\) captures three words separated by spaces.
- \1 captures “Three”
- \2 captures “One”
- \3 captures “Two”
\2 \3 \1: The replacement string reorders these captured groups.

Output:

One Two Three

Using ERE, the command is cleaner:

[user]$ echo "Three One Two" | sed -E 's|(\w+) (\w+) (\w+)|\2 \3 \1|'

Example with comma-separated values:

[user]$ echo "Smith,John,123 Main St" | sed -E 's|([^,]+),([^,]+),(.+)|\2 \1, \3|'

Output (rearranges to Firstname Lastname, Address):

John Smith, 123 Main St

Here, [^,]+ matches one or more characters that are not a comma.

String Case Modification in Replacement (GNU Sed Extensions)

GNU Sed provides special sequences in the replacement string to change the case of matched text or captured groups.

\L: Converts the rest of the replacement (or up to \E) to lowercase.
\l: Converts the next character to lowercase.
\U: Converts the rest of the replacement (or up to \E) to uppercase.
\u: Converts the next character to uppercase.
\E: Stops the case conversion started by \L or \U.

Example 1: Using \L

[user]$ sed -n 's/Paulo/PA\LULO COELHO/p' books.txt

Output (PA followed by ulo in lowercase):

3) The Alchemist, PAulo COELHO, 197
5) The Pilgrimage, PAulo COELHO, 288

Example 2: Using \u

[user]$ sed -n 's/Paulo/p\uaul\uo/p' books.txt

Output (p, then A, then ul, then O):

3) The Alchemist, pAulO Coelho, 197
5) The Pilgrimage, pAulO Coelho, 288

Example 3: Using \U

[user]$ sed -n 's/Paulo/\Upaulo/p' books.txt

Output (converts ‘paulo’ to ‘PAULO’):

3) The Alchemist, PAULO Coelho, 197
5) The Pilgrimage, PAULO Coelho, 288

Example 4: Using \U and \E

[user]$ sed -n 's/Paulo Coelho/Title: \U&\E - Author./p' books.txt # & refers to the whole match 'Paulo Coelho'

Output (converts ‘Paulo Coelho’ to uppercase within the replacement):

3) The Alchemist, Title: PAULO COELHO - Author., 197
5) The Pilgrimage, Title: PAULO COELHO - Author., 288

Practical Application: Converting CSV to a formatted string.

Input data.csv: name,age,city Alice,30,New York Bob,24,Los Angeles

[user]$ sed -E '1d; s/([^,]+),([^,]+),(.+)/Name: \U\1\E, Age: \2, City: \u\3/' data.csv

Explanation:

1d: Deletes the header line.
s/([^,]+),([^,]+),(.+)/.../: Captures three comma-separated fields.
- \1 is name, \2 is age, \3 is city.
Name: \U\1\E: Prints “Name: ” followed by the name in uppercase.
Age: \2: Prints “Age: ” followed by the age.
City: \u\3: Prints “City: ” followed by the city with its first letter capitalized.

Output:

Name: ALICE, Age: 30, City: New York
Name: BOB, Age: 24, City: Los Angeles