Unix / Linux - Pipes & Filters

Unix / Linux - Pipes and Filters

One of the most powerful concepts in the Unix philosophy is combining simple, specialized tools to perform complex tasks. Pipes and filters are central to this approach.

A pipe, represented by the vertical bar character (|), connects the standard output (stdout) of one command to the standard input (stdin) of the next command. This allows data to flow sequentially through a chain of commands.

A filter is a command designed to take input from stdin, process it in some way (e.g., select lines, sort data, transform text), and write the result to stdout. Many standard Unix utilities can act as filters.

command1 | command2 | command3

In this example, the output of command1 becomes the input for command2, and the output of command2 becomes the input for command3. The final output appears on the terminal (or can be redirected).

The grep Command (Filter)

grep (Global Regular Expression Print) is a fundamental filter used to search for lines containing a specific pattern within input text (from stdin or files).

Syntax:

$ grep [options] pattern [file(s)]
# Or using stdin:
$ command | grep [options] pattern

The pattern can be a simple string or a more complex regular expression.

Example: List only the directories in the current path using ls -l and grep.

# ls -l lists files and directories
# grep '^d' filters for lines starting with 'd' (directories)
$ ls -l | grep '^d'
drwxr-xr-x 2 user group 4096 Jul 25 15:30 documents
drwxr-xr-x 4 user group 4096 Jul 25 15:35 project
drwxr-xr-x 2 user group 4096 Jul 25 15:28 tmp

Common grep options:

Option	Description
-i	Ignore case distinctions in the pattern.
-v	Invert match: Select lines that do not match the pattern.
-n	Prefix each matching line with its line number.
-c	Print only a count of matching lines.
-l	Print only the names of files containing matching lines.
-w	Match whole words only.
-r or -R	Recursively search directories.
-E	Use extended regular expressions (ERE).
-F	Treat pattern as a fixed string (no regex), faster.

Example: Find lines containing ‘error’ (case-insensitive) in log files:

$ grep -i 'error' /var/log/app.log /var/log/syslog

Example: Count how many times user ‘alice’ is logged in (using who command output):

$ who | grep '^alice ' | wc -l

The sort Command (Filter)

The sort command arranges lines of text alphabetically or numerically.

# Contents of colors.txt:
# blue
# red
# green
# yellow

$ sort colors.txt
blue
green
red
yellow

Common sort options:

Option	Description
-r	Reverse the order of the sort.
-n	Numeric sort (treats values as numbers).
-k field[,field]	Sort based on a specific key field. Fields are whitespace-separated by default. -k 2 sorts on the second field, -k 3n sorts numerically on the third field.
-t char	Use char as the field separator (for use with -k).
-u	Unique: Output only the first of an equal run of lines.
-f	Fold lowercase to uppercase for comparison (case-insensitive sort).
-h	Human numeric sort (e.g., sorts 2K before 1M).

Example: List files, sort by size (5th field, numerically), largest first.

# ls -l output: permissions links owner group size date name
# We want to sort on the 5th field (size) numerically (-n) and reverse (-r)
$ ls -l | sort -k 5nr
-rw------- 1 user group 1048576 Jul 25 16:00 large_file.dat
-rw-r--r-- 1 user group   51200 Jul 25 15:55 medium_data.bin
drwxr-xr-x 5 user group    4096 Jul 25 15:35 project
-rw-r--r-- 1 user group    1024 Jul 25 16:01 small_config.txt
...

Example: Sort a CSV file based on the 2nd field (comma-separated).

$ sort -t ',' -k 2 data.csv

The uniq Command (Filter)

The uniq command filters out adjacent duplicate lines from a sorted input stream. It’s crucial that the input is sorted first.

# Contents of items.txt:
# apple
# banana
# apple
# orange
# banana

# Sort first, then find unique items
$ sort items.txt | uniq
apple
banana
orange

Common uniq options:

• -c: Prefix lines by the number of occurrences.
• -d: Only print duplicate lines (one for each group).
• -u: Only print unique lines (those that don’t repeat).

Example: Count occurrences of each item.

$ sort items.txt | uniq -c
      2 apple
      2 banana
      1 orange

Paging Output: less

When the output of a command or pipeline is too long to fit on the screen, pipe it through a pager like less. less allows you to scroll up and down, search, and view the output one screen at a time.

less is generally preferred over the older more command due to its ability to scroll backwards easily.

# View long process list page by page
$ ps aux | less

# List all files, sort by size, view page by page
$ ls -l /usr/bin | sort -k 5nr | less

Inside less, use arrow keys, PageUp/Down to navigate, / to search forward, ? to search backward, and q to quit.

Combining Filters

The real power comes from chaining multiple filters together.

Example: Find the top 5 most frequently occurring IP addresses in a web server log file (access.log). Assume IP is the first field.

# 1. Extract the first field (IP address) using cut
# 2. Sort the IPs alphabetically
# 3. Count adjacent duplicates (using uniq -c)
# 4. Sort the counts numerically, largest first (sort -nr)
# 5. Take the top 5 lines (head -n 5)

$ cut -d ' ' -f 1 access.log | sort | uniq -c | sort -nr | head -n 5
  1532 192.168.1.101
   987 10.0.0.5
   750 172.16.30.2
   610 192.168.1.150
   405 10.0.0.8

Other useful filters include cut (extract sections from lines), tr (translate or delete characters), sed (stream editor for text transformations), and awk (powerful pattern scanning and processing language). Exploring these tools unlocks advanced text processing capabilities.