Learning SED

Chapter 12 — SED in Shell Scripts and Pipelines

SED as a Pipeline Stage

SED's greatest strength is that it reads from stdin and writes to stdout — it fits naturally anywhere in a Unix pipeline. Every tool in the pipeline handles one concern, and SED handles the text transformation step.

# Typical pipeline: filter → transform → aggregate
grep "ERROR" app.log \
  | sed -E 's/.*\[([0-9:T-]+)\].*/\1/' \
  | sort | uniq -c | sort -rn

# SED reading from a command's output via process substitution
sed 's/foo/bar/g' <(curl -s "https://example.com/data.txt")

# SED writing to a variable via command substitution
CLEANED=$(sed 's/[[:space:]]*$//' <<< "  hello world  ")
echo "'$CLEANED'"
'  hello world'

Using SED Inside Shell Functions

Wrapping SED in shell functions creates readable, reusable utilities. The function handles argument validation and quoting; SED handles the transformation.

Basic wrapper pattern

# Function: trim whitespace from both ends of a string
trim() {
    sed 's/^[[:space:]]*//; s/[[:space:]]*$//' <<< "$1"
}

# Usage
trim "   hello world   "
hello world

# Capture the result
RESULT=$(trim "  data  ")

Functions that process files or stdin

# Function: strip all comments and blank lines from a config file
strip_config() {
    local file="${1:--}"   # default to stdin if no arg
    sed '/^[[:space:]]*#/d; /^[[:space:]]*$/d' "$file"
}

# Use on a file
strip_config /etc/ssh/sshd_config

# Use on stdin via pipeline
cat app.conf | strip_config

# ──────────────────────────────────────────────────────
# Function: replace a key=value in a file (safe in-place)
set_config_key() {
    local key="$1" value="$2" file="$3"
    if [[ -z "$key" || -z "$file" ]]; then
        echo "Usage: set_config_key KEY VALUE FILE" >&2
        return 1
    fi
    # Escape the value so it's safe inside the sed replacement
    local escaped_value
    escaped_value=$(sed 's/[\/&]/\\&/g' <<< "$value")
    sed -i "s|^\($key\s*=\s*\).*|\1$escaped_value|" "$file"
}

# Usage
set_config_key "max_connections" "200" db.conf
set_config_key "host" "db.example.com" app.conf

Passing shell variables into SED expressions

# Use double quotes to expand shell variables in the sed expression
OLD="localhost"
NEW="10.0.0.5"
sed "s/$OLD/$NEW/g" config.txt

# Problem: if variables contain / you need to escape or change delimiter
OLD_URL="http://old.example.com/api"
NEW_URL="https://new.example.com/v2"
sed "s|$OLD_URL|$NEW_URL|g" config.txt   # use | as delimiter

# Safer: escape the variable content before inserting into the pattern
escape_sed_pattern() {
    # Escapes special BRE characters in a string for use in a sed pattern
    sed 's/[]\/$*.^[]/\\&/g' <<< "$1"
}
escape_sed_replacement() {
    # Escapes special replacement characters (& and \)
    sed 's/[\/&]/\\&/g' <<< "$1"
}

# Safe substitution using the escape helpers
PATTERN=$(escape_sed_pattern "$OLD_URL")
REPLACE=$(escape_sed_replacement "$NEW_URL")
sed "s|$PATTERN|$REPLACE|g" config.txt

Safe In-Place Editing Patterns

In-place editing with -i is powerful but destructive if something goes wrong. In scripts you should always protect against failures.

# Always take a backup with -i.bak before modifying in scripts
sed -i.bak 's/foo/bar/g' important.conf

# Verify success, restore on failure
if ! sed -i.bak 's/foo/bar/g' important.conf; then
    echo "sed failed — restoring backup" >&2
    mv important.conf.bak important.conf
    exit 1
fi

# Atomic edit pattern: write to temp file, then move (POSIX mv is atomic)
atomic_sed_edit() {
    local expr="$1" file="$2"
    local tmp
    tmp=$(mktemp)
    if sed "$expr" "$file" > "$tmp"; then
        mv "$tmp" "$file"
    else
        rm -f "$tmp"
        echo "ERROR: sed edit failed on $file" >&2
        return 1
    fi
}

# Usage
atomic_sed_edit 's/version=1\.0/version=2.0/' app.conf

Error Handling and Exit Codes

# SED exit codes
# 0 — success (even if no substitutions were made)
# 1 — error (bad syntax, file not found)
# 2 — usage error (GNU sed)

# Check if sed succeeded
if sed -n '/pattern/p' file.txt > output.txt; then
    echo "Processing complete"
else
    echo "sed encountered an error" >&2
    exit 1
fi

# Note: sed returns 0 even if no lines matched — it is not grep
# To detect whether a substitution actually happened, use the w flag + wc -l
COUNT=$(sed -n 's/foo/bar/w /dev/stdout' file.txt | wc -l)
echo "$COUNT lines were changed"

# Validate a sed expression before using it on real data
if echo "" | sed -n "$EXPR" 2>/dev/null; then
    echo "Expression is valid"
else
    echo "Invalid sed expression: $EXPR" >&2
    exit 1
fi

Performance on Large Files

SED is fast — it processes line by line with minimal memory overhead. But there are patterns that hurt performance, and strategies to avoid them.

What slows SED down

Practical performance patterns

# BAD: three separate sed processes
sed 's/foo/bar/g' data.txt | sed 's/baz/qux/g' | sed '/^#/d'

# GOOD: one process, three expressions
sed 's/foo/bar/g; s/baz/qux/g; /^#/d' data.txt

# BEST for large files: pre-filter then transform
grep -F "foo" huge.log | sed 's/foo/bar/g'

# Use q/Q to stop early when you only need the first match
sed -n '/pattern/{p;q}' huge.log   # stop at first match — don't read the rest

# Process only a known range to avoid scanning the whole file
sed -n '1000,2000p' huge.log   # print lines 1000-2000 then stop

# For enormous files: use LC_ALL=C to avoid multibyte locale overhead
LC_ALL=C sed 's/foo/bar/g' 10gb.log

SED with find, xargs and Loops

# Edit all .conf files in a directory tree in-place
find /etc -name '*.conf' -exec sed -i.bak 's/old/new/g' {} \;

# Faster: use xargs to batch files (fewer sed invocations)
find /etc -name '*.conf' | xargs sed -i.bak 's/old/new/g'

# Handle filenames with spaces safely
find . -name '*.txt' -print0 | xargs -0 sed -i 's/foo/bar/g'

# Shell loop with sed — useful when you need per-file logic
for file in src/*.js; do
    if grep -q "TODO" "$file"; then
        echo "Processing: $file"
        sed -i 's/TODO/FIXME/g' "$file"
    fi
done

# Edit files modified in the last 24 hours
find . -name '*.py' -mtime -1 | xargs -0r sed -i 's/\t/    /g'

Combining SED with Other Tools

Knowing when to hand off to a different tool is as important as knowing how to use SED. Here is a practical guide to the most common partnerships:

Classic SED + awk pipeline

# SED strips noise, awk does the field work and arithmetic
grep "RESPONSE_TIME" app.log \
  | sed -E 's/.*RESPONSE_TIME=([0-9.]+)ms.*/\1/' \
  | awk '{ sum+=$1; n++ } END { printf "avg: %.2fms\n", sum/n }'

# SED normalises the format, cut extracts the column
sed 's/  */ /g' data.txt | cut -d ' ' -f3

# sed + sort + uniq for a frequency count
sed -E 's/.*status=([0-9]{3}).*/\1/' app.log | sort | uniq -c | sort -rn

Portability Checklist for Scripts

If your script must run on both Linux (GNU sed) and macOS (BSD sed), test against this checklist:

# Portable in-place edit that works on both GNU and BSD sed
if sed --version 2>/dev/null | grep -q 'GNU'; then
    # GNU sed: -i accepts no argument (or empty string)
    sed -i 's/foo/bar/g' file.txt
else
    # BSD sed: -i requires an explicit extension argument (even if empty)
    sed -i '' 's/foo/bar/g' file.txt
fi

When to Stop Using SED

SED is the right tool for fast, line-oriented text transformations. Reach for something else when:

• The data has structure (JSON, YAML, XML, real CSV) — use a parser that understands the format. SED treats everything as flat text.
• You need arithmetic — SED cannot add, subtract, or compare numbers. Use awk or bc.
• The regex would need lookahead/lookbehind — SED has no PCRE. Use grep -P, perl, or python.
• The script exceeds ~20 lines — a SED script of 30+ lines is a maintenance burden. Rewrite in Python or awk where logic can be clearly expressed.
• You need error handling, conditionals, or data structures — SED has none of these in any meaningful way. Use a real scripting language.
• The transformation depends on values from other lines (e.g. database-style joins, lookups) — SED's hold space is a single buffer; use awk or a scripting language.

Quick Reference — Chapter 12

Course Complete — Full Chapter Index