Capstone Project

Chapter 8 · Final Chapter · Capstone
Real-World One-Liners and Scripts
Log analysis, CSV processing, and combining AWK with grep and sed in realistic command-line pipelines

Every previous chapter introduced one mechanism at a time. This final chapter is entirely practical — genuine one-liners for tasks that come up constantly, and a look at how AWK fits alongside grep and sed (introduced back in Chapter 1) in real combined pipelines rather than in isolation.

Log Analysis

$ awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head -10
Top 10 most frequent IP addresses in an Apache/nginx access log. Extract field 1 (IP), count occurrences with sort+uniq -c, sort descending, take the top 10. AWK does the extraction; the rest is a classic Unix pipeline.
$ awk '$9 >= 500' access.log
Lines where the HTTP status code (field 9 in standard Apache log format) is 500 or higher — every server error, instantly filtered from a huge log file.
$ awk '{ sizes[$9] += $10 } END { for (s in sizes) print s, sizes[s] }' access.log
Total bytes transferred (field 10), grouped by status code (field 9) — combines Chapter 6's grouping pattern directly on real log data.

CSV Processing

$ awk -F',' 'NR > 1 { sum += $3 } END { print "Total:", sum }' sales.csv
NR > 1 skips the header row (Chapter 3's comparison pattern) — a near-universal first line in any CSV-processing AWK command.
$ awk -F',' '{ print $2 }' data.csv | sort -u
Every distinct value in column 2, alphabetically — using the external sort -u rather than Chapter 6's !seen[] idiom, when sorted order matters more than first-seen order.
$ awk -F',' '{ print $1","$3","$2 }' data.csv
Reorder CSV columns — print fields back out in a different order than they appeared, with explicit comma separators since the output isn't using OFS automatically here.

System Administration

$ ps aux | awk '{ print $4, $11 }' | sort -rn | head -5
Top 5 processes by memory percentage (field 4) alongside their command (field 11) — real-time system inspection using a tool already on every Linux/Mac system.
$ df -h | awk '$5+0 > 80 { print $1, $5 }'
Disk partitions over 80% full. $5+0 forces the percentage string ("85%") into a number for comparison — the +0 trick strips the % since AWK's numeric conversion stops at the first non-numeric character.

Combining grep, AWK, and SED in One Pipeline

Chapter 1 introduced the principle: each tool does the part it's genuinely best at. A realistic combined pipeline looks like this:

grep narrow down lines awk extract/compute fields sed final text cleanup output
Each tool handles exactly the part it's built for — none of the three try to do everything alone
$ grep "ERROR" app.log | awk -F'|' '{ print $2, $4 }' | sed 's/^[ \t]*//'
grep narrows to error lines only; awk extracts just the timestamp and message fields from a pipe-delimited format; sed strips any leading whitespace from the final output. Each command stays simple because it only does one job.
$ awk '/ERROR/,/RESOLVED/' app.log | grep -v "DEBUG" | wc -l
AWK's range pattern (Chapter 3) extracts an incident's full log section; grep -v filters out noisy debug lines from within that section; wc -l counts what's left — a genuinely useful incident-analysis one-liner built from three simple pieces.
If a "one-liner" is getting hard to read, it's time for a script file
Everything in this chapter fits comfortably on one line, but a real AWK script with several BEGIN/END blocks, multiple patterns, and real logic (Chapter 7) is more maintainable in a saved .awk file, run with awk -f script.awk (Chapter 1) — readability matters more than cleverness once a script is going to be reused or maintained by anyone, including future you.

Chapter 8 Quick Reference — and Course Wrap-Up

  • Log analysis: field extraction + sort/uniq -c for frequency counts; status-code filtering with comparison patterns
  • CSV: NR > 1 to skip headers; sort -u or !seen[] for unique values
  • System admin: ps/df piped into AWK for quick numeric filtering and sorting on live system data
  • +0 trick: forces a string like "85%" into a number, stopping at the first non-numeric character
  • Combined pipelines: grep narrows, AWK extracts/computes, sed does final text cleanup — each tool does one job
  • Move to a script file once a one-liner starts being hard to read at a glance
  • Course recap: intro & pattern-action (Ch1) → fields/records (Ch2) → patterns/BEGIN/END (Ch3) → functions (Ch4) → printf (Ch5) → arrays (Ch6) → control flow (Ch7) → real-world pipelines (Ch8)