Learning SED

Chapter 2 — The Substitute Command

The Most Important Command in SED

If you only ever learn one SED command, make it s — the substitute command. It is by far the most commonly used, and mastering it covers the vast majority of real-world SED use cases. This chapter is a complete guide to everything s can do.

The full syntax of the substitute command is:

s/pattern/replacement/flags

Each part has a precise role:

• s — the command itself (substitute)
• / — delimiter; separates the three parts (can be changed — see below)
• pattern — a regular expression to search for in the pattern space
• replacement — the text to substitute in place of the matched portion
• flags — optional modifiers controlling how the substitution behaves

SED applies the pattern against the entire pattern space (the current line). When it finds a match, it replaces the matched text with the replacement. Everything outside the matched portion is left exactly as it was.

Substitution Flags

Flags appear after the final delimiter and modify how the substitution operates. Multiple flags can be combined.

The g flag — Global replacement

Without g, SED replaces only the first match on each line. The g flag replaces all matches:

# Without g — only the first "the" is replaced
echo "the cat and the dog and the bird" | sed 's/the/a/'
a cat and the dog and the bird

# With g — all occurrences replaced
echo "the cat and the dog and the bird" | sed 's/the/a/g'
a cat and a dog and a bird

The i flag — Case-insensitive matching (GNU sed)

Makes the pattern match regardless of case:

echo "Error ERROR error eRrOr" | sed 's/error/fault/gi'
fault fault fault fault

The Nth occurrence flag — Replace a specific occurrence

A numeric flag tells SED which occurrence to replace. 2 means the second match, 3 the third, and so on:

echo "aa bb aa cc aa dd" | sed 's/aa/XX/2'
aa bb XX cc aa dd

# Combine with g to replace from the Nth occurrence onwards
echo "aa bb aa cc aa dd" | sed 's/aa/XX/2g'
aa bb XX cc XX dd

The p flag — Print if substitution was made

Prints the pattern space if and only if the substitution succeeded. Most useful with -n to show only lines that were actually changed:

# Show only lines where a substitution was made
sed -n 's/foo/bar/p' file.txt

# Practical: show which config lines contain "debug" and what they changed to
sed -n 's/debug=true/debug=false/p' app.conf

The w flag — Write matching lines to a file

Writes the pattern space to a named file whenever the substitution succeeds:

# Apply substitution AND write changed lines to changes.log
sed 's/ERROR/FAULT/w changes.log' logfile.txt

The e flag — Execute replacement as a shell command (GNU sed)

After substitution, the resulting pattern space is executed as a shell command and the output replaces the line. Powerful but use carefully:

# Get the file size of each filename listed in a file
sed 's/.*/du -sh & 2>\/dev\/null/' filelist.txt | sed -e 's/.*/&/'e

Flag summary table

Changing the Delimiter

The / character is SED's default delimiter, but it is just a convention. SED accepts any character immediately after s as the delimiter. This matters most when your pattern or replacement contains forward slashes — otherwise you would need to escape every one with a backslash.

Compare these two equivalent commands:

# Using / as delimiter — slashes in the path must be escaped
sed 's/\/etc\/nginx\/nginx.conf/\/etc\/nginx\/nginx.conf.bak/' file.txt

# Using | as delimiter — much more readable
sed 's|/etc/nginx/nginx.conf|/etc/nginx/nginx.conf.bak|' file.txt

# Using # as delimiter — common convention for path substitutions
sed 's#/usr/local/bin#/usr/bin#g' scripts.txt

# Using , as delimiter — works with URLs too
sed 's,https://old-domain.com,https://new-domain.com,g' links.html

Special Replacement Sequences

The replacement string is not plain text — it has its own set of special sequences that expand to useful values at substitution time.

& — The entire matched text

& in the replacement expands to whatever the pattern matched. This lets you wrap, prefix, or suffix the matched text without re-typing it:

# Wrap every number in brackets
echo "port 8080 and timeout 30" | sed 's/[0-9][0-9]*/[&]/g'
port [8080] and timeout [30]

# Quote every word
echo "alpha beta gamma" | sed 's/[a-z]*/"\0&"/g'

# Add a prefix to every line matching a pattern
sed 's/^ERROR.*/[ALERT] &/' logfile.txt
# Before: ERROR connection refused
# After:  [ALERT] ERROR connection refused

\1, \2, ... \9 — Capture group backreferences

When your pattern contains groups enclosed in \( and \) (in basic regex, BRE), each group's matched text is captured. \1 in the replacement refers to the first group, \2 to the second, and so on. This is one of the most powerful features of the substitute command.

# Swap the order of first and last name (BRE groups)
echo "Smith, John" | sed 's/\([^,]*\), \(.*\)/\2 \1/'
John Smith

# With -E (ERE) the groups are written without backslashes
echo "Smith, John" | sed -E 's/([^,]*), (.*)/\2 \1/'
John Smith

# Reformat a date from YYYY-MM-DD to DD/MM/YYYY
echo "2024-07-15" | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'
15/07/2024

# Extract the domain from a URL
echo "https://www.example.com/path/page" | sed -E 's|https?://([^/]+).*|\1|'
www.example.com

Case modifiers (GNU sed only)

GNU sed supports special escape sequences in the replacement to change the case of matched text:

# Capitalise the first letter of every word
echo "hello world from sed" | sed 's/\b\([a-z]\)/\u\1/g'
Hello World From Sed

# Convert matched text to uppercase
echo "warning: disk space low" | sed 's/warning/\U&/'
WARNING: disk space low

# Convert entire line to lowercase
echo "SYSTEM ALERT: DISK FULL" | sed 's/.*/\L&/'
system alert: disk full

Escaping in Patterns and Replacements

Some characters have special meaning in SED patterns or replacements and must be escaped with a backslash when you want them treated literally:

# Replace a literal dot (not "any character")
echo "version 1.0.5" | sed 's/1\.0\.5/2\.0\.0/'
version 2.0.0

# Insert a literal ampersand into the replacement
echo "Tom Jerry" | sed 's/ / \& /'
Tom & Jerry

# Replace a literal backslash
echo 'C:\Users\emuba' | sed 's/\\/\//g'
C:/Users/emuba

Inserting Newlines

You can insert a literal newline into the replacement string. In GNU sed, use \n in the replacement:

# Split a comma-separated pair onto two lines
echo "name:John,age:30" | sed 's/,/\n/g'
name:John
age:30

# Add a blank line before every section heading
sed 's/^\[.*\]/\n&/' config.ini

To insert a newline in a portable (POSIX) way, use a backslash followed by a literal newline inside the script:

# POSIX-portable newline in replacement (note the literal newline after \)
sed 's/,/\
/g' file.txt

Addresses with Substitution

By default the substitute command runs on every line. You can restrict it to specific lines by prefixing an address — a line number or a pattern. Addresses are covered fully in Chapter 3, but here is a preview because they are so commonly combined with substitution:

# Only substitute on line 5
sed '5s/old/new/' file.txt

# Only substitute on lines 10 through 20
sed '10,20s/old/new/g' file.txt

# Only substitute on lines containing "SECTION"
sed '/SECTION/s/old/new/' file.txt

# Only substitute on lines NOT containing "SECTION" (! negates the address)
sed '/SECTION/!s/old/new/' file.txt

# Between a start pattern and an end pattern
sed '/START/,/END/s/foo/bar/g' file.txt

Practical Recipes

Here are real-world substitution patterns you will use regularly:

Trim leading and trailing whitespace

# Trim leading whitespace (spaces and tabs)
sed 's/^[[:space:]]*//' file.txt

# Trim trailing whitespace
sed 's/[[:space:]]*$//' file.txt

# Trim both in one pass
sed 's/^[[:space:]]*//; s/[[:space:]]*$//' file.txt

Comment out a line in a config file

# Add a # to the beginning of a line containing "ServerName"
sed 's/^\(ServerName\)/#\1/' httpd.conf

# Uncomment a line (remove leading #)
sed 's/^#\(ServerName\)/\1/' httpd.conf

Update a version number

# Replace semantic version (flexible — matches any x.y.z)
sed -E 's/version = "[0-9]+\.[0-9]+\.[0-9]+"/version = "2.1.0"/' pyproject.toml

# Increment the patch version (extract groups, add 1 — requires awk for arithmetic)
sed -E 's/(version = ")([0-9]+\.[0-9]+\.)([0-9]+)(")/\1\2PATCH\4/' file.txt

Remove HTML tags

# Strip all HTML/XML tags from a file
sed 's/<[^>]*>//g' page.html

# Remove a specific tag and its closing tag
sed 's/<b>//g; s/<\/b>//g' page.html

Reformat log timestamps

# Change 2024-07-15 12:30:00 to 15/07/2024 12:30:00
sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/g' app.log

Mask sensitive data

# Replace credit card numbers (16 digits) with asterisks
sed -E 's/[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}/****-****-****-****/g' data.txt

# Mask everything after "password=" keeping the key name
sed -E 's/(password=).*/\1[REDACTED]/' config.txt

Normalise line endings

# Convert Windows CRLF (\r\n) to Unix LF (\n)
sed 's/\r//' windows_file.txt

# Same with -i to fix the file in place
sed -i 's/\r//' windows_file.txt

Replace across multiple files at once

# Update an API endpoint URL across all .js files in a project
sed -i 's|https://api.old-domain.com|https://api.new-domain.com|g' $(find src/ -name "*.js")

# Safer: preview changes first before -i (dry run using diff)
sed 's|https://api.old-domain.com|https://api.new-domain.com|g' app.js | diff app.js -

BRE vs ERE — A Quick Reminder

The substitute command uses Basic Regular Expressions (BRE) by default. With the -E flag it uses Extended Regular Expressions (ERE). The practical difference is how you write grouping, alternation, and quantifiers:

Common Mistakes

Greedy matching going too far

# Trying to remove the content inside <b> tags on a single line
echo "<b>bold</b> and <b>more bold</b>" | sed 's/<b>.*<\/b>//'
# Removes EVERYTHING between first <b> and last </b> — greedy!
 and   # probably not what you wanted

# Fix: use [^<]* instead of .* to stop at the first tag character
echo "<b>bold</b> and <b>more bold</b>" | sed 's/<b>[^<]*<\/b>//g'
 and   # hmm — still empty; let's see the full output below

echo "before <b>bold</b> middle <b>more</b> after" | sed 's/<b>[^<]*<\/b>//g'
before  middle  after

Forgetting that . matches anything — including dots

# Trying to match a literal IP-style string
echo "192.168.1.1 and 192X168Y1Z1" | sed 's/192.168.1.1/REDACTED/g'
# Both match because . means "any character"!
REDACTED and REDACTED

# Fix: escape the dots
echo "192.168.1.1 and 192X168Y1Z1" | sed 's/192\.168\.1\.1/REDACTED/g'
REDACTED and 192X168Y1Z1

Applying -i without testing first

# ALWAYS test without -i first, then add it when satisfied
# Step 1: preview — does the output look right?
sed 's/old_value/new_value/g' important.conf

# Step 2: only then edit in place with a backup
sed -i.bak 's/old_value/new_value/g' important.conf

Quick Reference — Chapter 2