Sed Useful Recipes

Practical sed Recipes for Common Tasks

sed is a versatile tool, and often there are multiple ways to solve a text manipulation problem. This section provides a collection of ‘recipes’ – common tasks that can be accomplished with sed, often simulating functionality found in other standard Unix utilities. These examples showcase sed’s power and flexibility. For some examples, we’ll use a generic input.txt or data.txt. Assume books.txt refers to the single-line-per-entry format unless specified.

Simulating cat (Concatenate and Display Files)

Method 1: Using default behavior (no commands).

$ sed '' books.txt

Method 2: Explicitly printing each line.

$ sed -n 'p' books.txt

Both commands will display the entire content of books.txt. The -n option in the second example suppresses automatic printing, so p (print) command is needed.

Removing Empty Lines

To delete all empty lines from a file:

$ echo -e "Line #1\n\n\nLine #2" | sed '/^$/d'

Output:

Line #1
Line #2

Explanation: ^$ is a regular expression matching an empty line (start of line immediately followed by end of line). The d command deletes such lines from the pattern space.

Alternatively, to print only non-empty lines:

$ echo -e "Line #1\n\n\nLine #2" | sed -n '/^$/!p'

Output:

Line #1
Line #2

Explanation: !p means print the line if it does not match the pattern /^$/.

Removing Single-Line Comments (e.g., from C++ code)

Consider a simple C++ file hello.cpp:

#include <iostream>
using namespace std;

int main(void)
{
   // Displays message on stdout.
   cout << "Hello, World !!!" << endl;
   return 0; // Return success.
}

To remove // comments:

$ sed 's|//.*||' hello.cpp

Output:

#include <iostream>
using namespace std;

int main(void)
{
   cout << "Hello, World !!!" << endl;
   return 0;
}

Explanation: s|//.*|| substitutes the pattern //.* (double slash followed by any characters to the end of the line) with an empty string. Note: This is a simple approach and might not correctly handle // within strings or block comments in complex code. For robust parsing, dedicated linters or parsers are better.

Adding Comments Before Specific Lines

Suppose script.sh contains shell commands. To comment out lines 3 to 5 by prepending #:

$ sed '3,5s/^/#/' script.sh

Example script.sh output after transformation (original content not shown):

#!/bin/bash
pwd
#hostname
#uname -a
#ls
who

Explanation: 3,5 is an address range selecting lines 3 through 5. s/^/#/ substitutes the beginning of the line (^) with #.

Simulating wc -l (Count Lines)

$ sed -n '$=' books.txt

Output (assuming books.txt has 6 lines):

6

Explanation: $ addresses the last line. = prints its line number. -n suppresses default line printing.

Simulating head (Display First N Lines)

To display the first 10 lines of long_file.txt:

$ sed '10q' long_file.txt

Explanation: The q command tells sed to quit after processing line 10. This is often more efficient than sed -n '1,10p' for large files.

Simulating tail -1 (Display Last Line)

$ sed -n '$p' books.txt

Output (last line of books.txt):

6) A Game of Thrones, George R. R. Martin, 864

Explanation: $ addresses the last line, p prints it, and -n suppresses other output.

Simulating dos2unix (Convert CRLF to LF)

DOS/Windows uses Carriage Return + Line Feed (\r\n) for newlines, while Unix uses Line Feed (\n). To convert:

$ sed 's/\r$//' dos_file.txt > unix_file.txt

Explanation: s/\r$// removes the carriage return (\r) if it’s at the end of the line ($). Note: Some sed versions might require typing ^M (Ctrl+V, Ctrl+M in many terminals) instead of \r if \r is not recognized as a special escape.

Simulating unix2dos (Convert LF to CRLF)

$ sed 's/$/\r/' unix_file.txt > dos_file.txt

Explanation: s/$/\r/ replaces the end of the line ($) with a carriage return (\r). sed automatically adds the line feed (\n) when printing.

Simulating cat -E (Show $ at End of Lines)

$ sed 's/$/$/' input.txt

Explanation: s/$/$/ replaces the end of the line (matched by $) with a literal $ character.

Simulating nl (Number Lines)

$ sed '=' input.txt | sed 'N; s/\n/\t/'

Example Output (for input.txt with 2 lines “Line A” and “Line B”)

1  Line A
2  Line B

Explanation: The first sed '=' input.txt prints the line number, then a newline, then the line content. The second sed command processes this output: N appends the next line (the actual content line) to the pattern space (which holds the line number). s/\n/\t/ replaces the newline between the number and content with a tab.

Simulating cp (Copy File)

$ sed 'w new_copy.txt' original.txt

If you only want to write and not print to stdout:

$ sed -n 'w new_copy.txt' original.txt

Explanation: The w new_copy.txt command writes the pattern space to new_copy.txt for every line processed.

Simulating expand (Convert Tabs to Spaces)

To convert each tab to 8 spaces (common default for expand):

$ sed 's/\t/        /g' file_with_tabs.txt

Explanation: s/\t/ /g replaces every tab (\t) with eight spaces, globally (g) on each line.

Simulating tee (Output to Stdout and File)

$ sed 'p; w output_file.txt' input.txt

If default printing is suppressed by -n (e.g. if other commands are used):

$ sed -n 'p; w output_file.txt' input.txt

Explanation: p prints the current pattern space to standard output. w output_file.txt writes the pattern space to output_file.txt. If using sed without -n, the p might cause double printing unless the script logic is designed carefully. sed 'w output_file.txt' input.txt will also work, printing to stdout by default and writing to file.

Simulating cat -s (Squeeze Multiple Blank Lines to One)

To reduce sequences of two or more empty lines down to a single empty line:

$ sed '/^$/{N; /\n$/D;}' input_with_blanks.txt

Example: Input A\n\n\nB becomes A\n\nB.

Explanation: This is a common idiom. If a line is empty (/^$/), N appends the next line. If the pattern space now consists of just two newlines (\n$, actually pattern is original_empty_line\nnext_line, so if next_line is also empty, PS looks like \n), D deletes up to the first newline and restarts the cycle. This effectively handles pairs of blank lines. For a perfect cat -s emulation that handles all edge cases and any number of blank lines correctly, more complex scripts or multiple passes might be needed in POSIX sed, or one might rely on GNU sed extensions.

Simulating grep (Print Lines Matching Pattern)

$ sed -n '/pattern/p' input.txt

Explanation: -n suppresses default output. /pattern/p prints lines that match pattern.

Simulating grep -v (Print Lines NOT Matching Pattern)

$ sed -n '/pattern/!p' input.txt

Explanation: /pattern/!p prints lines that do NOT (!) match pattern.

Simulating tr (Translate Characters)

To convert all uppercase ABC to lowercase abc:

$ echo "ABC TEST" | sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/'

Output:

abc test

Explanation: The y/source_chars/dest_chars/ command transliterates characters. Each character in source_chars is replaced by the character at the corresponding position in dest_chars.