Learning SED

Chapter 7 — Multi-line Commands

Crossing the Line Boundary

Every technique covered so far has processed one line at a time. Each cycle reads a single line into the pattern space, the commands run, the line is printed, and the pattern space is cleared. The line boundary is the fundamental unit of SED processing.

But real text does not always break conveniently at line boundaries. An XML tag might open on one line and close on another. A function call might be split across two lines. An error message might span a header line and a detail line. To process these structures, SED provides three commands that deliberately cross the line boundary — N, P, and D. Together they form a powerful trio for multi-line work.

N — Appending the Next Line

The lowercase n command (Chapter 4) replaces the pattern space with the next line. The uppercase N is different: it appends the next line to whatever is already in the pattern space, inserting a literal newline character between them.

# After N fires, the pattern space contains TWO lines joined by \n
printf 'line one\nline two\nline three\n' | sed 'N; s/\n/ /'
line one line two
line three
# N joins line 1 and line 2 into the pattern space
# s/\n/ / replaces the embedded newline with a space → "line one line two"
# Line three is processed alone (no pair to join with)

Key behaviours of N

• The pattern space now contains an embedded \n — you can match across it with \n in a regex
• The line counter is advanced (the next line has been consumed)
• If there is no next line (N fires on the last line), GNU sed prints the pattern space and exits — it does not error
• You can call N repeatedly in a loop to accumulate more than two lines

Matching across a line boundary

# Find "foo" on one line followed by "bar" on the next, join them
sed '/foo/ { N; /bar/ s/foo\nbar/foobar/ }' file.txt

# Remove a line break between any line ending with a comma and the next
sed ':a; /,$/ { N; s/,\n/,/; ta }' data.txt
# :a is a label; ta branches back to :a if a substitution succeeded
# This loops, consuming continuation lines until no trailing comma remains

# Delete a pattern that spans two lines
sed '/^BEGIN$/ { N; /^BEGIN$\nEND$/ d }' file.txt

Substituting across line boundaries

# Replace a newline between two specific patterns with a space
sed '/^Subject:/ { N; s/\n[[:space:]]/ / }' email.txt
# Joins folded email headers (RFC 2822 header continuation)

# Collapse a multi-line function call onto one line (lines ending with ,)
sed -E ':a; N; s/,\n[[:space:]]*/,/; ta' source.js

P — Printing the First Line

Once the pattern space holds multiple lines (thanks to N or hold space operations), you often need to output only the first line — not the entire accumulated content. That is what P does. It prints everything up to the first embedded \n.

# P vs p on a two-line pattern space
printf 'a\nb\nc\n' | sed -n 'N; P'
a
c
# N joins a+b into pattern space; P prints only "a"
# cycle ends (no default print due to -n)
# Next cycle reads "c" alone (b was consumed by N); P on single line prints "c"

Compare the three print behaviours side by side:

D — Deleting the First Line and Restarting

D is the most complex of the three. It does two things atomically:

1. Deletes the first line of the pattern space (everything up to and including the first \n)
2. Restarts the script from the beginning — without reading a new line from input

This restart behaviour is what makes D powerful. After deleting the first line, the remainder of the pattern space (the second line and beyond) is still there, and the full script runs again on that remaining content. D creates an implicit loop.

# Practical effect: D is like d but keeps the second line "in play"
printf 'a\nb\nc\n' | sed 'N; P; D'
a
b
c
# This is a sliding-window loop — prints each line once:
# Cycle 1: N → PS="a\nb"; P → prints "a"; D → PS="b", restart script
# Script restart: N → PS="b\nc"; P → prints "b"; D → PS="c", restart
# Script restart: N on last line → PS="c" (no more input); P → prints "c"; D → empty, done

The N-P-D Loop — A Sliding Window

The combination N; P; D is a fundamental SED idiom. It creates a sliding window of two lines: at any moment, the pattern space holds the current line and the next. After each step, the window slides forward by one. This gives every SED command the ability to look at two consecutive lines simultaneously.

Step-by-Step Traces

Trace: Remove duplicate consecutive lines

Script: sed '$!N; /^\(.*\)\n\1$/!P; D'

Practical Recipes

Join lines that end with a backslash (shell/Makefile continuation)

# Collapse continuation lines ending in \ into one logical line
sed ':a; /\\$/ { N; s/\\\n//; ta }' Makefile
# :a   — label for looping
# /\\$/ — if current (accumulated) line ends with a backslash
# N    — append the next line
# s/\\\n// — remove the backslash-newline join point
# ta   — if substitution succeeded, loop back to :a

Join folded email/HTTP headers

# RFC 2822: header continuation lines start with a space or tab
sed ':a; N; s/\n[[:space:]]/ /; ta; P; D' headers.txt
# Accumulate lines, join continuation whitespace, slide the window forward

Remove blank lines between paragraphs (collapse multiple blanks into one)

# Replace two or more consecutive blank lines with a single blank line
sed '/^$/ { N; /^\n$/ d }' file.txt

# More robust version using the N-P-D loop:
sed ':a; /^$/ { $!N; /^\n/{ s/^\n//; ta } }' file.txt

Delete a line and the line immediately after it

# Delete any line matching "REMOVE" together with the line that follows
sed '/REMOVE/ { N; d }' file.txt

Delete a line and the line immediately before it

# Using the sliding window: delete the window when the second line matches
sed -n '$!N; /\nREMOVE$/!P; D' file.txt
# P only fires when the second line does NOT match — suppressing both lines when it does

Wrap long lines at word boundaries

# Split any line longer than 72 characters at the last space before position 72
sed -E ':a; s/^(.{72}[^ ]*) (.)/\1\n\2/; ta' longlines.txt
# Matches 72+ chars followed by a space and next char, inserts a newline at the break
# ta loops until no more splits needed on this line

Convert Unix line endings back to Windows (add \r before \n)

# GNU sed: insert carriage return before every newline
sed 's/$/\r/' unix.txt > windows.txt

Find and remove a pattern that spans exactly two lines

# Delete any two-line block where line 1 = "START" and line 2 = "END"
sed '/^START$/ { N; /^START\nEND$/ d }' file.txt

Substitute across a line boundary — replace split tag

# An opening XML tag is sometimes split across two lines in generated output
# <longTag
#   attr="value">
# Join and normalise it:
sed '/<[A-Za-z]/ { :a; />/!{ N; ta }; s/\n[[:space:]]*/  /g }' file.xml

Print paragraphs containing a keyword

# A paragraph is a block of non-blank lines separated by blank lines
# Accumulate each paragraph, print if it contains "keyword"
sed -n '/^$/ { x; /keyword/p; d }
         H' file.txt
# H accumulates lines into hold space
# On blank line: x swaps hold to pattern, check for keyword, print or discard

Split a CSV line at every comma onto separate lines

# Turn "a,b,c,d" into four separate lines
sed 's/,/\n/g' data.csv

# More controlled: only split the first field off
sed 's/,/\n/' data.csv

n vs N — The Critical Difference

It is easy to confuse lowercase n and uppercase N. The distinction is essential:

# n — move forward: prints "a", then processes "b"
printf 'a\nb\n' | sed 'n; s/b/B/'
a
B

# N — join: pattern space = "a\nb", then substitute across the boundary
printf 'a\nb\n' | sed 'N; s/a\nb/a+b/'
a+b

P vs p and D vs d

Similarly, it is worth keeping the uppercase/lowercase pairs clearly separate in your mind:

Common Mistakes

N on the last line terminating unexpectedly

# If N fires on the last line and there is no next line to read:
# GNU sed: prints pattern space and exits normally
# BSD/POSIX sed: exits immediately without printing

# Safe pattern — skip N on the last line:
sed '$!N; s/\n/ /' file.txt
# $! = "not the last line" — N only fires when there is a next line to read

Forgetting that D restarts the script

# MISTAKE: assuming D just removes the first line and continues normally
sed 'N; D; s/foo/bar/' file.txt
# s/foo/bar/ never runs! D causes the script to restart from the top
# The substitution is unreachable

Infinite loop with D and no exit condition

# DANGER: D restarts the script — if the restarted script fires N again
# immediately, it will grow the pattern space forever until input is exhausted
# Always ensure your D-based loops have a termination condition
# Good pattern: condition on content before D, not just N; ... D blindly

Quick Reference — Chapter 7