Learning SED

Chapter 9 — The y Command and Other Commands

Completing the Command Set

The previous chapters covered SED's core commands — s, d, p, q, a, i, c, the hold space commands, the multi-line trio, and branching. This chapter covers the remaining commands that round out the full SED toolkit. Some are used daily; others are specialist tools for specific tasks.

The y Command — Transliterate

The y command performs character-by-character translation across the entire pattern space. It works like the standalone tr utility but is integrated into SED so it can be combined with addresses and other commands in a single pass.

The syntax mirrors s but uses source and destination character lists instead of a pattern and replacement:

y/source-chars/dest-chars/

Every character in the pattern space that appears in source-chars is replaced by the character at the same position in dest-chars. The two lists must be the same length. Like s, the delimiter can be changed to any character.

Case conversion

# Convert lowercase a-z to uppercase A-Z
sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' file.txt

# Convert uppercase to lowercase
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' file.txt

# GNU sed shorthand using character classes in s// is often cleaner for case
sed 's/.*/\U&/' file.txt   # uppercase (GNU sed \U extension)
sed 's/.*/\L&/' file.txt   # lowercase (GNU sed \L extension)

ROT13 encoding

# Classic ROT13 — rotate letters by 13 positions (encode and decode are identical)
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz/NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm/' file.txt

Transliterating digits and special characters

# Replace digits 0-9 with their word equivalents using a proxy
# (y can only map single chars — for multi-char replacements use s//)

# Swap pairs of characters (simple cipher)
sed 'y/aeiouAEIOU/eioauEIOAU/' file.txt

# Replace path separators — Windows to Unix
sed 'y/\\/\//' winpaths.txt
# Backslash → forward slash (each needs escaping)

# Translate specific punctuation
sed 'y/;:,/|||/' data.txt
# Replace semicolons, colons, and commas all with pipe characters

y vs s — knowing which to use

The r and R Commands — Reading Files

The r command reads the entire contents of a named file and inserts them into the output stream after the addressed line. It does not affect the pattern space — the file content is sent directly to output after the current line's normal print completes. If the file does not exist, r silently does nothing (no error).

r filename       # note: there must be exactly one space between r and the filename

Inserting a file after a pattern match

# Insert a standard footer after the last line of every file
sed '$r footer.txt' document.txt

# Insert a licence header after line 1 of a script
# (use a after line 1 for before — r always appends after)
sed '1r licence_header.txt' script.sh

# Insert a file after every line matching a pattern
sed '/^## Summary/r summary_boilerplate.txt' report.md

Template substitution — the killer use case for r

The most powerful use of r is template filling: replace a placeholder line with the contents of another file. Combine r (to insert the file after the placeholder) with d (to delete the placeholder itself):

# template.html contains the line: ##CONTENT_PLACEHOLDER##
# Replace it with the contents of content.html
sed '/##CONTENT_PLACEHOLDER##/ {
  r content.html
  d
}' template.html

# Same result, more compact:
sed '/PLACEHOLDER/ { r insert.txt
d }' template.txt

The R command — reading one line at a time

R (GNU sed only) reads a single line from a file each time it fires, advancing an internal pointer. This lets you interleave lines from two files:

# Interleave every line of file.txt with lines from extra.txt
sed 'R extra.txt' file.txt
# Output: line1 of file.txt
#         line1 of extra.txt
#         line2 of file.txt
#         line2 of extra.txt  ...

# Substitute a placeholder with one line from a data file each time it appears
sed '/TEMPLATE_LINE/ {
  R data.txt
  d
}' template.txt

The w and W Commands — Writing Files

The w command writes the pattern space to a named file. SED creates (or truncates) the output file when the script starts, then appends to it each time w fires during processing. The file is ready to use as soon as SED finishes.

w filename       # one space between w and filename

Splitting a file by content

# Route error lines to errors.log and everything else to normal.log
sed -n '/ERROR/  w errors.log
         /ERROR/! w normal.log' app.log

# Split a CSV into separate files by first field value
# (limited — sed can only write to fixed filenames, not dynamic ones)
sed -n '/^alice,/ w alice.csv
/^bob,/   w bob.csv
/^carol,/ w carol.csv' users.csv

Writing a subset of lines to a file

# Extract all lines containing IP addresses to a separate file
sed -n '/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/ w ips.txt' logfile.txt

# Write changed lines to a log while also applying changes in-place
sed -i.bak 's/old_api_url/new_api_url/w changes.log' config.js
# -i.bak edits in place with backup; w flag on s// logs changed lines

Using w as a side-effect alongside normal output

# Print all lines normally, but ALSO write matching lines to a separate file
sed '/ERROR/ w error_summary.txt' app.log
# All lines go to stdout as normal; ERROR lines are additionally written to the file

The W command — writing only the first line

# After N has joined lines, write only the first line of the pair to a file
sed -n '$!N; /^key:/ W keys_only.txt; P; D' data.txt

The e Command — Execute

The e command (GNU sed only) has two distinct forms with different behaviours:

Using e as a standalone command

# Insert the current date before every line of the file
sed 'e date "+%Y-%m-%d"' file.txt

# Run a command and inject its output before matching lines
sed '/^hostname:/ e hostname' server.conf
# Inserts the actual hostname before each "hostname:" line

Using the e flag on s

# For each filename in a list, replace it with its actual file size
sed 's/.*/stat --format="%s bytes: &" &/e' filelist.txt
# s builds a stat command from each filename
# e flag executes that command and replaces the line with its output

# Evaluate each line as a shell arithmetic expression
sed 's/.*/echo $((&))/e' expressions.txt
# Input: "2 + 2"
# s wraps it: "echo $((2 + 2))"
# e executes it → output: "4"

The z Command — Zap the Pattern Space

The z command (GNU sed only) clears the pattern space — sets it to an empty string — without starting a new cycle. This is the key difference from d:

# Clear the pattern space but append the hold space onto the now-empty line
sed '/HEADER/ { z; G }' file.txt
# z empties the pattern space; G appends the hold space
# Net result: the HEADER line's content is replaced with whatever is in hold space
# (different from c because z + G lets you build the replacement dynamically)

# Use z to reset accumulated content mid-script
sed -n '/START/ { H }
/END/   { H; g; /important/p; z; h }
/START/! { /END/! H }' file.txt
# z resets the pattern space after printing a block
# h then saves the now-empty pattern space back to hold, resetting the accumulator

Practical: z as a conditional line suppressor

# Clear lines that fail a validation check (output blank line as placeholder)
sed '/^[0-9]\+$/ !z' numbers.txt
# Lines that are NOT pure numbers: z clears them (outputs blank line)
# Lines that ARE pure numbers: printed unchanged

# Then pipe to remove the blank lines if you don't want them:
sed '/^[0-9]\+$/ !z' numbers.txt | sed '/^$/d'
# (equivalent to: sed -n '/^[0-9]\+$/p' numbers.txt)

The = Command — Print Line Number

Already introduced in Chapter 4, = prints the current line number to stdout followed by a newline. It is included here for the complete command reference.

# Count lines in a file (print only the last line number)
sed -n '$=' file.txt

# Print line numbers alongside matching lines (grep -n equivalent)
sed -n '/ERROR/ { =; p }' logfile.txt

# Add line numbers to every line's output (cat -n equivalent)
sed '=' file.txt | paste - -
# = prints number on its own line; paste merges pairs of lines

Combining These Commands — Practical Recipes

Build a simple template engine

# template.conf:
#   ServerName ##HOSTNAME##
#   ##SSL_CONFIG##
#   DocumentRoot /var/www/##SITENAME##

sed -e 's/##HOSTNAME##/www.example.com/g' \
    -e 's/##SITENAME##/mysite/g' \
    -e '/##SSL_CONFIG##/ { r ssl_block.conf
                         d }' \
    template.conf

Audit log: apply changes and record what changed

# Edit in place, write a change log simultaneously
sed -i.bak 's/password=.*/password=REDACTED/w redacted.log' config.ini
# -i.bak: edit in place with backup
# w flag: write lines where substitution succeeded to redacted.log

ROT13 encode only comment lines in a script

sed '/^#/ y/ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz/NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm/' script.sh
# y only runs on lines matching /^#/ — code lines are untouched

Split output into matched and unmatched files

# Write every line to one of two output files, suppress stdout
sed -n '/^[0-9]/ w numeric.txt
       /^[^0-9]/ w alpha.txt' mixed.txt

Interleave a file with generated line numbers

# Number each line with = and merge onto same line as content
sed '=' file.txt | sed 'N; s/\n/\t/'
# First sed: = prints number on its own line before each content line
# Second sed: N joins pairs; s replaces the joining newline with a tab

Evaluate shell expressions embedded in a config file

# A config file contains lines like: path=$(pwd)/subdir
# Evaluate only those lines
sed '/\$(/s/.*/echo "&"/e' config.txt
# s wraps matched lines in echo "..."
# e executes: echo evaluates the $() expansion
# WARNING: only use on trusted files

Generate a numbered HTML list from plain text

# Wrap each line in <li> tags and add ol wrappers
sed -e '1i\<ol>' \
    -e 's/.*\/  <li>&<\/li>/' \
    -e '$a\<\/ol>' \
    items.txt

Complete SED Command Reference

Now that all commands have been covered, here is the full command set in one place:

Quick Reference — Chapter 9