Learning SED

Chapter 11 — SED Scripts and Real-World Recipes

From One-Liners to Script Files

So far every example has used the -e flag or a single quoted string. For serious work — multi-step transformations, reusable utilities, version-controlled text processing tools — you write a SED script file and invoke it with sed -f scriptfile.sed input.txt.

A script file is plain text. Each line is a SED command, exactly as you would write after -e, but without the quoting and with full freedom to use whitespace, blank lines, and comments.

Invoking a script file

# Basic usage
sed -f normalise_config.sed input.conf

# In-place edit
sed -i -f normalise_config.sed *.conf

# Combine a script file with an extra -e command
sed -f normalise_config.sed -e 's/localhost/10.0.0.1/g' input.conf

# ERE mode with a script file
sed -E -f myscript.sed input.txt

# Self-executable script (shebang must be first line)
chmod +x normalise_config.sed
./normalise_config.sed input.conf

Structuring Complex Scripts

Using braces for clarity

# Group related commands under an address — visually clear in script files
/^[[:space:]]*\[/ {
    # Lines that look like INI section headers
    s/[[:space:]]//g         # strip all spaces
    s/\[/[/              # rewrite opening bracket (noop here but explicit)
    y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/  # uppercase
}

# Process lines NOT in the header range
/^[[:space:]]*\[/! {
    s/^[[:space:]]*//       # trim leading whitespace
    s/[[:space:]]*$//       # trim trailing whitespace
}

Labels and branching for loops in scripts

# Script: collapse repeated blank lines into a single blank line
# Algorithm: join current line to next with N, check for blank+blank,
# loop until no longer two consecutive blanks.

/^$/ {
    N
    /^\n$/b blankloop   # if still blank after join, keep looping
    P
    D
}
b                      # non-blank line: fall through to default print

:blankloop
N
/^\n$/b blankloop
P
D

Log File Processing

Filter and extract from logs

# Show only ERROR and FATAL lines (suppress everything else)
sed -n '/\(ERROR\|FATAL\)/p' app.log

# Same in ERE
sed -En '/(ERROR|FATAL)/p' app.log

# Delete DEBUG and TRACE lines in-place (quiet the noise)
sed -i '/\(DEBUG\|TRACE\)/d' app.log

# Anonymise IP addresses (replace last two octets)
sed -E 's/([0-9]{1,3}\.[0-9]{1,3})\.[0-9]{1,3}\.[0-9]{1,3}/\1.x.x/g' access.log

# Extract just the timestamp column (first field before space)
sed 's/[[:space:]].*//' app.log

# Add a prefix to every log line for pipeline tagging
sed 's/^/[APP] /' app.log

Reformat Apache / Nginx combined log format

# Combined log format:
# IP - - [timestamp] "METHOD /path HTTP/1.1" status bytes
# Extract: IP, status, path
sed -E 's/^([^ ]+) [^ ]+ [^ ]+ \[[^]]+\] "[^ ]+ ([^ ]+)[^"]*" ([0-9]+).*/\1 \3 \2/' access.log

# Count each unique status code (pipe to sort | uniq -c for frequency)
sed -E 's/^.* "[^"]*" ([0-9]{3}) .*/\1/' access.log | sort | uniq -c

# Keep only 4xx and 5xx errors
sed -En '/\" [45][0-9]{2} /p' access.log

Multi-line log entry processing

# Java stack traces span multiple lines — collect them
# Strategy: when a line starts with "at " or "Caused by:", append it
# to the previous line so the full trace stays together

# Simple version: join lines beginning with whitespace (indented continuations)
sed '/^[[:space:]]/{H;d}; /^[[:space:]]/!{x;s/\n/ /g;p}' -n app.log

Config File Manipulation

INI / key=value files

# Set a specific key to a new value (key=value format)
sed 's/^\(max_connections\s*=\s*\).*/\1100/' db.conf

# Same with ERE — cleaner
sed -E 's/^(max_connections\s*=\s*).*/\1100/' db.conf

# Comment out a specific key (prepend #)
sed 's/^\(listen_address\s*=\)/#\1/' db.conf

# Uncomment a specific key (remove leading #)
sed 's/^#\(listen_address\s*=\)/\1/' db.conf

# Delete a key entirely
sed '/^old_deprecated_key\s*=/d' db.conf

# Append a new key after an existing one
sed '/^max_connections/a new_param = 50' db.conf

# Insert a key before an existing one
sed '/^\[database\]/i # Database section begins here' db.conf

Work safely within a specific section only

# Only modify keys inside the [database] section
# Strategy: range from section header to the next section header (or EOF)
sed -E '/^\[database\]/,/^\[/ s/^(host\s*=\s*).*/\1newhost/' app.conf

# Caution: the closing /^\[/ also matches the NEXT section header line itself
# To avoid modifying that line, negate it:
sed -E '/^\[database\]/,/^\[/ { /^\[database\]/! { /^\[/! s/^(host\s*=\s*).*/\1newhost/ } }' app.conf

XML / HTML attribute updates

# Update a version attribute in a single-line XML tag
sed -E 's/(version=")[^"]*/\11.2.3/' pom.xml

# Replace a URL inside quotes (uses | as delimiter to avoid escaping /)
sed -E 's|href="[^"]*"|href="https://new.example.com"|g' index.html

# Remove a specific HTML attribute entirely
sed -E 's/ style="[^"]*"//g' index.html

CSV and Tabular Data

# Extract the third comma-separated field
sed -E 's/^([^,]*,){2}([^,]*).*/\2/' data.csv

# Delete the first field (up to and including the first comma)
sed 's/[^,]*,//' data.csv

# Replace comma delimiter with pipe
sed 's/,/|/g' data.csv

# Add a header line if the file lacks one
sed '1i name,age,city' data.csv

# Skip the header line and process only data rows
sed '1d; s/,/ /g' data.csv

# Surround every field with double quotes (simple CSV → quoted CSV)
sed -E 's/([^,]+)/"\1"/g' data.csv

# Trim spaces around comma separators
sed 's/[[:space:]]*,[[:space:]]*/,/g' data.csv

Source Code Transformations

Rename a function or variable across a codebase

# Safe rename: only match whole identifiers (not substrings)
sed -i 's/\bgetUser\b/fetchUser/g' src/*.js

# Rename across all .py files recursively (using find + xargs)
find . -name '*.py' | xargs sed -i 's/\bold_func\b/new_func/g'

Add / remove language constructs

# Add a semicolon to every non-blank, non-comment line (JS minification prep)
sed '/^[[:space:]]*$/d; /^[[:space:]]*\/\//d; s/$/;/' code.js

# Remove C-style single-line comments (// to end of line)
sed 's|//.*$||' code.c

# Add "export " prefix to every function declaration
sed '/^function /s/^/export /' module.js

# Indent every line by 4 spaces (e.g. when inserting into another file)
sed 's/^/    /' snippet.py

# Remove all indentation (dedent)
sed 's/^[[:space:]]*//' snippet.py

# Convert tabs to 4 spaces
sed 's/\t/    /g' code.py

# Remove trailing whitespace (common pre-commit hook fix)
sed -i 's/[[:space:]]*$//' *.py

Version number bumping

# Bump patch version X.Y.Z → X.Y.Z+1 is tricky without arithmetic
# But you can replace a known version string precisely:
sed 's/version = "1.2.3"/version = "1.2.4"/' pyproject.toml

# Replace any version string matching the pattern (capture major.minor, replace patch)
# Note: SED cannot do arithmetic — only string replacement
sed -E 's/(version = ")([0-9]+\.[0-9]+\.)[0-9]+/\11.2.4/' pyproject.toml

Text Formatting and Document Processing

Line spacing

# Double-space a file (add a blank line after every line)
sed 'G' file.txt

# Remove all blank lines
sed '/^[[:space:]]*$/d' file.txt

# Remove consecutive blank lines (leave at most one blank line)
sed '/./,/^$/!d' file.txt   # classical one-liner

# Number every line
sed '=' file.txt | sed 'N; s/\n/\t/'

# Number only non-blank lines
sed '/./=' file.txt | sed '/./N; s/\n/\t/'

Reverse a file (tac equivalent)

# Pure SED tac: accumulate all lines in hold space, then print reversed
sed -n '1!G; h; $p' file.txt
# 1!G  — on every line except the first: prepend hold space to pattern space
# h    — copy pattern space to hold space
# $p   — on the last line: print (now contains all lines reversed)

Prepend and append boilerplate

# Insert a copyright header at the very top of a file
sed -i '1i # Copyright 2024 Example Corp. All rights reserved.' *.py

# Append a footer line at the very end
sed -i '$a # End of file' file.txt

# Replace lines 1-3 (existing header) with a new header
sed -i '1,3c\# New header line 1\n# New header line 2' file.txt

# Wrap entire file content in XML tags
sed '1i <document>; $a </document>' content.txt

Markdown and plain-text utilities

# Convert Markdown headings to HTML
sed -E 's/^### (.+)/<h3>\1<\/h3>/;
         s/^## (.+)/<h2>\1<\/h2>/;
         s/^# (.+)/<h1>\1<\/h1>/' doc.md

# Convert *word* to <em>word</em>
sed -E 's/\*([^*]+)\*/<em>\1<\/em>/g' doc.md

# Remove all Markdown emphasis markers (strip * and _)
sed 's/[*_]//g' doc.md

# Centre-align text within 72 characters (approximate)
sed 's/^/                                    /; s/^\(.\{36\}\)\(.*\)/\1\2/' title.txt

Data Masking and Security

# Mask credit card numbers (keep last 4 digits)
sed -E 's/\b([0-9]{4}[- ]?){3}([0-9]{4})\b/****-****-****-\2/g' report.txt

# Redact email addresses
sed -E 's/[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}/[REDACTED]/g' data.txt

# Mask all but the first and last character of a password field
sed -E 's/(password=")([^"])(.*)(.")(\s*")/\1\2***\4\5/g' config.xml

# Remove any line containing an API key pattern
sed -i '/api[_-]key\s*[:=]/Id' config.txt   # I flag = case-insensitive (GNU)

# Scrub phone numbers (various formats)
sed -E 's/(\+?1[-. ]?)?(\(?\d{3}\)?[-. ]?)(\d{3}[-. ]?\d{4})/[PHONE]/g' data.txt

The Essential SED One-Liner Library

These are the most referenced SED one-liners — memorise or bookmark them:

sed '/^$/d'

sed 's/[[:space:]]*$//'

sed 's/^[[:space:]]*//'

sed 's/^[[:space:]]*//; s/[[:space:]]*$//'

sed -n '5,10p'

sed -n '$p'

sed 'G'

sed '=' | sed 'N;s/\n/\t/'

sed -n '1!G;h;$p'

sed -n '/START/,/END/p'

sed '/START/,/END/d'

sed '$!N;/^\(.*\)\n\1$/!P;D'

sed -i.bak 's/foo/bar/g' file

sed 's/^/PREFIX: /'

sed 's/\r$//'

sed 's/foo/bar/2'  # 2nd only

Building Reusable SED Libraries

For teams or personal toolkits, it pays to organise frequently used SED scripts as self-documenting files:

#!/usr/bin/sed -Ef
# ──────────────────────────────────────────────────────
# sanitise_csv.sed
# Purpose : Normalise a messy CSV before import
# Usage   : sed -Ef sanitise_csv.sed input.csv > output.csv
# Requires: GNU sed (uses \s, -E, -f together)
# ──────────────────────────────────────────────────────

# 1. Remove UTF-8 BOM if present (first line, first three bytes = EF BB BF)
1s/^\xef\xbb\xbf//

# 2. Normalise Windows CRLF to LF
s/\r$//

# 3. Trim whitespace around each comma
s/[[:space:]]*,[[:space:]]*/,/g

# 4. Remove trailing comma if present
s/,$//

# 5. Remove completely empty lines
/^$/d

Quick Reference — Chapter 11