Learning SED

Chapter 10 — Regular Expressions in SED

Regex in the SED Context

Regular expressions appear in two places in SED: in address patterns (/regex/command) and in the substitute command (s/regex/replacement/). Everything you know about regex applies in both places, but SED has its own flavour — with specific quirks, two distinct modes (BRE and ERE), and some useful extensions that are not found in other tools.

This chapter covers the full regex feature set available in SED: what works in both modes, what requires -E, what is GNU-specific, and the common traps that catch even experienced users.

BRE vs ERE — The Two Modes

SED operates in one of two regular expression modes:

• BRE — Basic Regular Expressions — the default. Older, more verbose syntax where many metacharacters must be backslash-escaped to activate their special meaning.
• ERE — Extended Regular Expressions — activated with the -E flag (or -r on older GNU sed). Cleaner, more intuitive syntax familiar from grep -E, awk, and most modern regex flavours.

The table below shows every place where the syntax differs:

Anchors

Anchors match positions in the line rather than characters. SED supports four anchors:

# Match lines that start with a digit
sed -n '/^[0-9]/p' file.txt

# Match blank lines (start immediately followed by end)
sed '/^$/d' file.txt

# Match lines that end with a semicolon
sed -n '/;$/p' source.js

# Replace the word "cat" but not "category" or "concatenate"
sed 's/\bcat\b/dog/g' file.txt

# Remove trailing whitespace (match any whitespace at end of line)
sed 's/[[:space:]]*$//' file.txt

The Dot — Matching Any Character

. matches any single character except a newline. This "except newline" rule is the default in SED and most regex flavours. In a multi-line pattern space (after N), the embedded \n is not matched by ..

# . matches any single character (not newline)
echo "cat bat hat sat" | sed 's/.at/dog/g'
dog dog dog dog

# Common mistake: using . when you mean a literal dot
echo "192.168.1.1 and 192X168X1X1" | sed 's/192.168/NETWORK/g'
NETWORK.1.1 and NETWORKX1X1   # both matched! . = "any char"

# Fix: escape the dot
echo "192.168.1.1 and 192X168X1X1" | sed 's/192\.168/NETWORK/g'
NETWORK.1.1 and 192X168X1X1   # only the real IP matched

# .* — match everything (greedy)
echo "<b>bold</b> and <b>more</b>" | sed 's/<.*>//'
 and <b>more</b>   # .* is greedy — matches as much as possible

Quantifiers

Quantifiers control how many times the preceding element must match:

Greedy matching — the default and its consequences

All quantifiers in SED are greedy — they match as much as possible while still allowing the overall pattern to succeed. This is the source of many surprising match results:

# Greedy: .* matches as much as possible
echo "start middle end" | sed 's/s.*e/X/'
X nd   # matched from "s" all the way to the last "e"

# Lazy matching does not exist in SED (no .*? syntax)
# Workaround: use a negated character class to stop at the first delimiter
echo "<b>bold</b> and <i>italic</i>" | sed 's/<[^>]*>//g'
bold and italic   # [^>]* means "anything that is not >"

Practical quantifier examples

# Match one or more digits
sed -E 's/[0-9]+/NUM/g' file.txt

# Match an optional minus sign before digits
sed -E 's/-?[0-9]+/NUM/g' file.txt

# Match exactly 4 digits (year)
sed -E 's/\b[0-9]{4}\b/YEAR/g' file.txt

# Match 2 to 4 lowercase letters
sed -E 's/\b[a-z]{2,4}\b/WORD/g' file.txt

# Match an HTTP or HTTPS URL
sed -E 's|https?://[^[:space:]]+|URL|g' file.txt

Character Classes

A character class [...] matches any single character from the listed set. A negated class [^...] matches any character not in the set.

Literal character classes

# Match any vowel
sed 's/[aeiouAEIOU]/*/g' file.txt

# Match any hex digit
sed -n '/^[0-9A-Fa-f]\+$/p' file.txt

# Match anything that is not a digit
sed 's/[^0-9]//g' file.txt   # strip all non-digits

Special characters inside [...]

Most regex metacharacters lose their special meaning inside a character class. But a few need care:

POSIX character classes — the portable way

POSIX defines named character classes that work correctly across different locales and character sets. These are written inside an extra pair of brackets within the class: [[:name:]].

# Trim leading whitespace using POSIX class (portable across locales)
sed 's/^[[:space:]]*//' file.txt

# Remove all punctuation from a line
sed 's/[[:punct:]]//g' file.txt

# Keep only alphanumeric characters and spaces
sed 's/[^[:alnum:][:space:]]//g' file.txt

# Remove control characters (e.g. stray \r, bell chars)
sed 's/[[:cntrl:]]//g' file.txt

GNU sed Extensions

GNU sed adds several escape sequences not in POSIX BRE or ERE. These are available in both BRE and ERE mode (unless noted):

# Match word boundaries using \b (GNU sed)
sed 's/\bthe\b/THE/g' file.txt

# Strip all whitespace characters (including tabs)
sed 's/\s//g' file.txt

# Match a tab character explicitly
sed 's/\t/  /g' file.txt   # replace tabs with two spaces

# Match word characters (letters, digits, underscore)
sed 's/\w\+/TOKEN/g' file.txt

Grouping and Backreferences

Parentheses group parts of a pattern, and the text matched by each group can be referenced in the replacement string as \1, \2, etc. This is one of SED's most powerful features.

# Capture groups in BRE (backslash required around parentheses)
echo "2024-07-15" | sed 's/\([0-9]\{4\}\)-\([0-9]\{2\}\)-\([0-9]\{2\}\)/\3\/\2\/\1/'
15/07/2024

# Same thing in ERE — much cleaner
echo "2024-07-15" | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'
15/07/2024

# Use backreferences to match repeated content (BRE)
echo "the the cat cat" | sed 's/\(\b\w\+\b\) \1/\1/g'
the cat   # removes duplicated consecutive words

# Wrap numbers in brackets
echo "port 8080 and timeout 30" | sed -E 's/([0-9]+)/[\1]/g'
port [8080] and timeout [30]

# Swap two comma-separated fields
echo "Smith,John" | sed -E 's/([^,]+),([^,]+)/\2 \1/'
John Smith

Backreferences in patterns (not just replacements)

Backreferences can also appear in the pattern itself to match repeated content:

# Delete lines where the same word appears twice consecutively
sed '/\(\b[a-z]\+\b\).*\1/d' file.txt

# Delete adjacent duplicate lines (hold space approach — but this also works)
sed '/^\(.*\)$/ { N; /^\(.*\)\n\1$/d }' file.txt

Alternation

Alternation matches one of several alternatives. In ERE the pipe character | separates alternatives; in BRE you need \| (GNU extension — not POSIX).

# Match "colour" or "color" — ERE
sed -E 's/colou?r/colour/g' file.txt   # using ? is cleaner here
sed -E 's/colour|color/colour/g' file.txt

# Match any of several log levels
sed -En '/(ERROR|WARN|FATAL)/p' logfile.txt

# Delete lines containing any of three patterns
sed -E '/DEBUG|TRACE|VERBOSE/d' logfile.txt

# BRE equivalent (GNU extension — not portable)
sed '/DEBUG\|TRACE\|VERBOSE/d' logfile.txt

Common Regex Traps in SED

Trap 1: Unescaped dot matching too broadly

# Intended: match IP 192.168.0.1
sed 's/192.168.0.1/REDACTED/' file.txt
# Problem: matches 192X168Y0Z1 too (. = any char)

# Fix: escape the dots
sed 's/192\.168\.0\.1/REDACTED/' file.txt

Trap 2: Greedy .* swallowing too much

# Intended: remove each HTML tag individually
echo "<b>bold</b> and <i>italic</i>" | sed 's/<.*>//g'
    # wrong — .* consumed everything from <b> to </i>

# Fix: use [^>]* to stop at the first >
echo "<b>bold</b> and <i>italic</i>" | sed 's/<[^>]*>//g'
bold and italic

Trap 3: Forgetting BRE vs ERE parenthesis rules

# BRE: ( is literal, \( starts a group
echo "foo(bar)" | sed 's/(bar)/[bar]/'
foo[bar]   # ( and ) are literal in BRE — matched the literal parentheses

echo "foobar" | sed 's/\(bar\)/[\1]/'
foo[bar]   # \( \) are grouping in BRE — captured "bar"

# ERE: ( is grouping, \( is a literal parenthesis
echo "foobar" | sed -E 's/(bar)/[\1]/'
foo[bar]   # ( ) are grouping in ERE

echo "foo(bar)" | sed -E 's/\(bar\)/[bar]/'
foo[bar]   # \( \) are literal in ERE

Trap 4: Character range ambiguity in different locales

# [a-z] matches differently depending on LC_COLLATE
# In a C/POSIX locale: only a-z ASCII lowercase
# In some UTF-8 locales: may include accented characters between a and z

# Safe: use POSIX class instead
sed 's/[[:lower:]]/*/g' file.txt   # always matches lowercase letters for current locale

# Or force POSIX locale for predictable ASCII behaviour
LC_ALL=C sed 's/[a-z]/*/g' file.txt

Trap 5: * matching zero occurrences

# * means ZERO or more — it can match the empty string
echo "abc" | sed 's/x*/X/g'
XaXbXcX   # x* matches the empty string before and after every character!

# Fix: use + (one or more) instead of * when you need at least one match
echo "abc" | sed -E 's/x+/X/g'
abc   # no match — there are no x's

Trap 6: Forgetting that SED regex is not PCRE

# These PCRE features do NOT exist in SED:
# (?:...)   — non-capturing group
# (?=...)   — lookahead
# (?<!...)  — lookbehind
# .*?       — lazy quantifier
# \d \w \s  — only in GNU sed, not POSIX
# For complex PCRE needs, use: grep -P, perl, or python

Practical Regex Recipes

Validate and extract patterns

# Print only valid IPv4 addresses
sed -En '/^([0-9]{1,3}\.){3}[0-9]{1,3}$/p' file.txt

# Print only lines that look like email addresses
sed -En '/^[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}$/p' file.txt

# Extract all hex colour codes from a CSS file
sed -En 's/.*#([0-9A-Fa-f]{3,6}).*/\1/p' styles.css

Transformations using groups

# Reformat US date MM/DD/YYYY to ISO YYYY-MM-DD
sed -E 's|([0-9]{2})/([0-9]{2})/([0-9]{4})|\3-\1-\2|g' file.txt

# Convert snake_case to camelCase
sed -E 's/_([a-z])/\u\1/g' file.txt
# \u makes next char uppercase (GNU sed replacement extension)

# Add thousands separators to numbers (1234567 → 1,234,567)
sed ':a; s/\([0-9]\)\([0-9]\{3\}\)\b/\1,\2/; ta' file.txt

Defensive patterns

# Match a whole word safely (not a substring)
sed 's/\bword\b/replacement/g' file.txt

# Match from start of line to first colon (non-greedy via negated class)
sed 's/^[^:]*/KEY/' file.txt   # replace everything before first :

# Match a quoted string (single quotes, no embedded quotes)
sed -E "s/'[^']*'/STRING/g" file.txt

Quick Reference — Chapter 10