Capstone Container

Binary File Formats in Practice

Chapter 10 · Capstone: A Tiny Container Format

Nine chapters of taking other people's formats apart end here with building one. A container format is a file that holds other files: think of any archive you have opened. Yours will be small enough to read in one sitting, yet it will use every idea in the course: a magic number and version, a fixed header read with struct, length-prefixed records, a flags field, a checksum, optional compression, and the defensive reader from Binary File Formats in Practice 9. The finished program is about a hundred lines, and you will test it with round trips, hand-damaged files, a decompression bomb and a fuzzer.

What was run

Every snippet and solution in this chapter was executed, and the output shown in the comments is what Python printed. Everything is done in memory (apart from one exercise that uses a temporary folder which is deleted afterwards). The snippets build on one another: run them in order in one session. The format is invented for this course, so it is not compatible with any real tool.

Step 1: Design Before You Code

Writing a format down before writing any code is the habit that separates a format from an accident. The decisions are few, and each one answers a question from an earlier chapter. We call the format TBX (tiny box), version 1. All integers are little-endian, as in Binary File Formats in Practice 2.

Part	Fields	struct format
Header (8 bytes)	magic `TBX1`, version, flags, entry count	`<4sBBH`
Entry header (13 bytes)	name length, stored size, original size, CRC-32 of the original data	`<BIII`
Entry body	the name in UTF-8, then the stored data	raw bytes

TBX file
├── header          8 bytes     magic "TBX1", version 1, flags, count
└── entry x count
    ├── entry header   13 bytes     name_len, stored, original, crc32
    ├── name           name_len bytes   UTF-8, no "/" or "\", not empty
    └── data           stored bytes     raw, or zlib if flag bit 0 is set
    

The design choices, and why:

Magic and version up front. The reader can reject a stranger's file in the first eight bytes (Binary File Formats in Practice 4). The version lets a later format change shape without confusing old readers.
Flags with a rule for unknown bits. Bit 0 means "data is zlib-compressed". A reader that meets any other bit set must refuse, because it does not know what that bit promises (Binary File Formats in Practice 7).
Stored size and original size. Both are written so that the reader can cap decompression at the declared original size, and so it can verify the claim afterwards.
CRC of the original data. It catches accidental damage after decompression. Remember from Binary File Formats in Practice 9 that it detects accidents, not tampering.
Counts instead of an end marker. The entry count in the header lets the reader reject an impossible count before looping.
Limits written next to the format. At most 1,000,000 bytes per file, 100 entries and 500,000 bytes per entry, with names of 1 to 255 bytes. These are choices, not laws; the point is that they exist.

Step 2: The Writer

The writer is the easy half, but it still validates: a writer that can produce a file its own reader rejects is a bug. Names are checked by one shared function, used by both halves, so the rule lives in one place.

import struct, zlib

MAGIC = b"TBX1"
VERSION = 1
HEADER = struct.Struct("<4sBBH")    # magic, version, flags, entry count
ENTRY = struct.Struct("<BIII")      # name length, stored size, original size, crc32
FLAG_ZLIB = 0x01
KNOWN_FLAGS = FLAG_ZLIB

MAX_FILE = 1_000_000
MAX_ENTRIES = 100
MAX_ORIGINAL = 500_000

class FormatError(ValueError):
    """The input is not a valid TBX file, or cannot be written as one."""

def check_name(name):
    if not name or "/" in name or "\\" in name or name in (".", ".."):
        raise FormatError(f"bad entry name {name!r}")
    if len(name.encode("utf-8")) > 255:
        raise FormatError("entry name too long")

def write_tbx(entries, compress=False):
    if len(entries) > MAX_ENTRIES:
        raise FormatError("too many entries")
    seen, body = set(), []
    for name, data in entries:
        check_name(name)
        if name in seen:
            raise FormatError(f"duplicate name {name!r}")
        seen.add(name)
        if len(data) > MAX_ORIGINAL:
            raise FormatError(f"{name!r} is over {MAX_ORIGINAL} bytes")
        stored = zlib.compress(data, 9) if compress else data
        raw_name = name.encode("utf-8")
        body.append(ENTRY.pack(len(raw_name), len(stored), len(data), zlib.crc32(data)))
        body.append(raw_name)
        body.append(stored)
    flags = FLAG_ZLIB if compress else 0
    return HEADER.pack(MAGIC, VERSION, flags, len(entries)) + b"".join(body)

files = [("hello.txt", b"Hello, world!\n"), ("zeros.bin", bytes(300)), ("empty", b"")]
blob = write_tbx(files)
print(len(blob), blob[:8].hex(" "))
small = write_tbx(files, compress=True)
print(len(small), small[:8].hex(" "))

# 384 54 42 58 31 01 00 03 00
# 113 54 42 58 31 01 01 03 00
    

The uncompressed file is 384 bytes: the 8-byte header, three 13-byte entry headers, the names (9 + 9 + 5), and 14 + 300 + 0 bytes of data. The compressed one is 113 bytes because the 300 zero bytes shrink to almost nothing. Byte 4 is the version (01), and byte 5 is the flags: 00 in the first file, 01 in the second.

Step 3: The Reader

The reader is where the chapter on robust parsing pays off. It is built on the checked Reader class, converts every kind of failure into FormatError, and checks each claim before acting on it: file size, magic, version, flag bits, entry count (against a limit and against the bytes that could hold it), name encoding and content, duplicate names, declared sizes, the real decompressed size, the checksum, and finally the trailing bytes.

class Reader:
    def __init__(self, buf):
        self.buf = bytes(buf)
        self.pos = 0

    def remaining(self):
        return len(self.buf) - self.pos

    def take(self, n):
        if n < 0 or n > self.remaining():
            raise FormatError(f"need {n} bytes at offset {self.pos}, {self.remaining()} left")
        chunk = self.buf[self.pos:self.pos + n]
        self.pos += n
        return chunk

    def unpack(self, st):
        return st.unpack(self.take(st.size))

def inflate(data, expected):
    d = zlib.decompressobj()
    try:
        out = d.decompress(data, expected + 1)
    except zlib.error as e:
        raise FormatError(f"corrupt zlib data: {e}") from e
    if len(out) != expected or not d.eof or d.unused_data:
        raise FormatError(f"zlib data does not match the declared size {expected}")
    return out

def read_tbx(buf):
    if len(buf) > MAX_FILE:
        raise FormatError(f"file over {MAX_FILE} bytes")
    r = Reader(buf)
    magic, version, flags, count = r.unpack(HEADER)
    if magic != MAGIC:
        raise FormatError(f"bad magic {magic!r}")
    if version != VERSION:
        raise FormatError(f"unsupported version {version}")
    if flags & ~KNOWN_FLAGS:
        raise FormatError(f"unknown flag bits {flags & ~KNOWN_FLAGS:#04x}")
    if count > MAX_ENTRIES:
        raise FormatError(f"entry count {count} over limit {MAX_ENTRIES}")
    if count * (ENTRY.size + 1) > r.remaining():
        raise FormatError(f"entry count {count} cannot fit in {r.remaining()} bytes")
    out, seen = [], set()
    for _ in range(count):
        start = r.pos
        name_len, stored, original, crc = r.unpack(ENTRY)
        try:
            name = r.take(name_len).decode("utf-8")
        except UnicodeDecodeError as e:
            raise FormatError(f"entry at {start}: name is not UTF-8") from e
        check_name(name)
        if name in seen:
            raise FormatError(f"duplicate name {name!r}")
        seen.add(name)
        if original > MAX_ORIGINAL:
            raise FormatError(f"{name!r}: declared size {original} over limit")
        data = r.take(stored)
        if flags & FLAG_ZLIB:
            data = inflate(data, original)
        elif stored != original:
            raise FormatError(f"{name!r}: stored size {stored} != original size {original}")
        if zlib.crc32(data) != crc:
            raise FormatError(f"{name!r}: checksum mismatch")
        out.append((name, data))
    if r.remaining():
        raise FormatError(f"{r.remaining()} trailing bytes at offset {r.pos}")
    return out

print(read_tbx(blob) == files, read_tbx(small) == files)
print(read_tbx(write_tbx([])))
for name, data in read_tbx(small):
    print(f"{name:10} {len(data):4} bytes")

# True True
# []
# hello.txt    14 bytes
# zeros.bin   300 bytes
# empty         0 bytes
    

Read read_tbx once top to bottom and notice that it is a list of questions, in an order that never lets a later step trust something an earlier step has not checked. In inflate, requesting expected + 1 bytes lets a single call prove that the stream would produce more than promised, without holding the excess; unused_data catches bytes left after the stream ends. The empty archive is legal and round-trips to an empty list.

Step 4: Round Trips

The first test of any reader and writer pair is the round trip: write something, read it back, and expect the same thing. Doing it with random contents, of random sizes, both compressed and not, catches the cases you did not think of. The random generator is seeded so the test gives the same answer every time.

import os, random

def roundtrip(entries, compress):
    return read_tbx(write_tbx(entries, compress)) == entries

random.seed(27)
trials = 0
for _ in range(200):
    entries = [(f"f{i}.dat", os.urandom(random.randint(0, 200)) if random.random() < 0.5
                else bytes(random.randint(0, 2000)))
               for i in range(random.randint(0, 8))]
    for compress in (False, True):
        assert roundtrip(entries, compress)
        trials += 1
print(trials, "round trips ok")
print(roundtrip([("é.txt", "héllo".encode())], True))

# 400 round trips ok
# True
    

The accented name shows that names travel as UTF-8, and that the length field counts bytes rather than characters: é is one character but two bytes.

Step 5: Hand-Made Bad Files

A round trip only proves the happy path. Now damage a known-good file on purpose, one field at a time, and read the error message: each one should name the problem and, where it can, the offset. patched overwrites a field at a known position, using the layout from the design table (the first entry header starts at offset 8).

def bad(label, buf):
    try:
        read_tbx(buf)
        print(label, "-> ACCEPTED")
    except FormatError as e:
        print(label, "->", e)

def patched(buf, offset, fmt, value):
    return buf[:offset] + struct.pack(fmt, value) + buf[offset + struct.calcsize(fmt):]

bad("empty file", b"")
bad("bad magic", b"XXXX" + blob[4:])
bad("version 2", patched(blob, 4, "<B", 2))
bad("unknown flag", patched(blob, 5, "<B", 0x82))
bad("huge count", patched(blob, 6, "<H", 60000))
bad("count too big for data", patched(blob, 6, "<H", 90))
bad("cut in the middle", blob[:-5])
bad("trailing bytes", blob + b"\x00")
bad("flipped data bit", blob[:30] + bytes([blob[30] ^ 1]) + blob[31:])
bad("size mismatch", patched(blob, 13, "<I", 15))
one = ENTRY.pack(1, 1, 1, zlib.crc32(b"x")) + b"ax"
bad("duplicate names", HEADER.pack(MAGIC, VERSION, 0, 2) + one + one)
bad("zlib cut", small[:-3])

# empty file -> need 8 bytes at offset 0, 0 left
# bad magic -> bad magic b'XXXX'
# version 2 -> unsupported version 2
# unknown flag -> unknown flag bits 0x82
# huge count -> entry count 60000 over limit 100
# count too big for data -> entry count 90 cannot fit in 376 bytes
# cut in the middle -> need 5 bytes at offset 379, 0 left
# trailing bytes -> 1 trailing bytes at offset 384
# flipped data bit -> 'hello.txt': checksum mismatch
# size mismatch -> 'hello.txt': stored size 14 != original size 15
# duplicate names -> duplicate name 'a'
# zlib cut -> need 8 bytes at offset 105, 5 left
    

Twelve different damages, twelve different messages, and no exception that is not a FormatError. Writing these cases is also the quickest way to find a missing check: each time you can imagine a field lying, add a line here and see what the reader says.

Step 6: A Bomb and a Fuzzer

The last two tests are the heavy ones. First a decompression bomb: a 48 KB stream that inflates to 50 million bytes, hidden in an entry that claims a modest size. Then the fuzzer from the previous chapter, with one small change: it records a label for each case, so a crash could be traced to the exact damage that caused it.

bomb = zlib.compress(bytes(50_000_000), 9)
print(len(bomb), "byte bomb")
liar = HEADER.pack(MAGIC, VERSION, FLAG_ZLIB, 1) + ENTRY.pack(1, len(bomb), 1000, 0) + b"b" + bomb
bad("bomb, small claim", liar)
huge = HEADER.pack(MAGIC, VERSION, FLAG_ZLIB, 1) + ENTRY.pack(1, len(bomb), 50_000_000, 0) + b"b" + bomb
bad("bomb, honest claim", huge)

# 48610 byte bomb
# bomb, small claim -> zlib data does not match the declared size 1000
# bomb, honest claim -> 'b': declared size 50000000 over limit
    

The first bomb lies about its size and is stopped by the capped decompressor; the second tells the truth and is stopped by the per-entry limit before any decompression starts. (The compressed size, 48,610 bytes, is what this machine's zlib produced and may differ slightly with another version (check).)

def fuzz(parser, seed):
    cases = [("cut at %d" % i, seed[:i]) for i in range(len(seed))]
    for i in range(len(seed)):
        for val in (0x00, 0xFF, seed[i] ^ 0x80):
            if val != seed[i]:
                cases.append((f"byte {i} -> {val:#04x}", seed[:i] + bytes([val]) + seed[i + 1:]))
    ok = rejected = 0
    crashes = []
    for label, case in cases:
        try:
            parser(case)
            ok += 1
        except FormatError:
            rejected += 1
        except Exception as e:
            crashes.append((label, type(e).__name__))
    return len(cases), ok, rejected, crashes

for name, seed in (("plain", blob), ("zlib", small)):
    total, ok, rejected, crashes = fuzz(read_tbx, seed)
    print(name, len(seed), "bytes:", total, "cases,", ok, "accepted,", rejected, "rejected,", len(crashes), "crashes")

# plain 384 bytes: 1212 cases, 23 accepted, 1189 rejected, 0 crashes
# zlib 113 bytes: 422 cases, 27 accepted, 395 rejected, 0 crashes
    

Zero crashes in 1,634 damaged files: every one is either refused with FormatError or accepted. The accepted ones deserve a look, because accepting damaged input is what the fuzzer is really hunting. Here all 23 accepted cases in the uncompressed file turn out to be changes to a byte of an entry name (offsets 21 to 29, 57 to 65 and 379 to 383), which produce a different but still valid name. That is a real finding about the design, not a bug in the code: the CRC covers the data, and nothing covers the names. Exercise 2 closes that gap.

Design lesson

A fuzzer tests your parser against your own idea of "valid". When it reports something accepted, ask whether the format should have noticed. Often the answer is a field nobody is protecting.

What You Would Add Next

A real format keeps growing, and the design above leaves room to do it safely:

New flags. A reader that knows only bit 0 refuses a file using bit 1, instead of misreading it. Old files stay valid.
A new version. Change the version byte when the layout changes, as Exercise 2 does with a whole-file checksum. A reader supports the versions it knows and rejects the rest by name.
An index. Reading one file currently means walking every entry. A table of offsets at the end (like many real archives, which put the index at the end so files can be appended) allows direct access, at the price of the offset checks from Binary File Formats in Practice 9.
Tagged extra data. A type-length-value area per entry, as in Binary File Formats in Practice 8, so that new metadata (timestamps, comments) can be added and unknown tags skipped.
Streaming. The reader here takes the whole file as bytes. For huge archives you would read from a file object, keeping the same checks on a bounded buffer.

The Whole Course in One Table

Each chapter contributed something to the format you have just built:

Chapter	Used in TBX as
Binary File Formats in Practice 1	The hex dump (`hex(" ")`) used to look at the output, and binary reading and writing
Binary File Formats in Practice 2	Little-endian integers throughout
Binary File Formats in Practice 3	The two `struct.Struct` objects and their sizes
Binary File Formats in Practice 4	The magic number and version at the start
Binary File Formats in Practice 5	CRC-32 with zlib, and compressed data
Binary File Formats in Practice 6	Fixed header followed by sized chunks of data
Binary File Formats in Practice 7	The flags byte and its unknown-bits rule
Binary File Formats in Practice 8	Length-prefixed names and data
Binary File Formats in Practice 9	The checked reader, limits, bomb protection, fuzzing

Challenges

Exercise 1

Connect the format to the disk. Write pack_dir(path, compress=False), which archives the regular files of one folder (skipping sub-folders, in sorted order), and unpack_to(buf, dest), which validates the whole archive first, refuses to overwrite any existing file, and only then writes the files. Use a temporary folder: pack a folder with two files and a sub-folder, unpack it elsewhere and compare, try to unpack twice, and show that a hand-built archive containing the name ../evil.txt is rejected and writes nothing.

📄 View solution

Exercise 2

Close the gap the fuzzer found. Define version 2 of TBX: the same layout with the version byte set to 2 and a 4-byte little-endian CRC-32 of everything before it added at the end. Write write_v2 and a read_any that accepts both versions (checking the whole-file checksum first for version 2). Fuzz both versions with the same harness and show that version 1 accepts some damaged files while version 2 accepts none, and show what a one-bit change to a name does in each.

📄 View solution

Exercise 3

Write a reusable selftest(seed, rounds, mutations) for the format: seeded random archives round-tripped both compressed and not, then thousands of randomly damaged copies (one to four byte changes, sometimes a truncation), counting round trips, accepted and rejected files and any crash by exception name. Show that the same seed gives the same report. Finish by reading a 50-million-byte bomb under tracemalloc and showing the peak memory stays small.

📄 View solution

Quick Reference

Design first: magic, version, flags with an unknown-bits rule, explicit sizes, a checksum, and written-down limits
Header: struct.Struct("<4sBBH"); entry header: struct.Struct("<BIII")
Writer: validate names, sizes and counts so it never produces what the reader refuses
Reader: one gate (take), one error type (FormatError), check every claim before using it, reject trailing bytes
Compression: decompressobj().decompress(data, expected + 1), then compare length, eof and unused_data
Testing: seeded round trips, hand-damaged files, a bomb, and a truncate-and-mutate fuzzer
A fuzzer's accepted cases: read them; they point at fields the format forgot to protect
Evolving: new flag bits and a new version byte, never a silent change of meaning

Course complete

You can now read a hex dump, decode integers of any width and byte order, describe a layout with struct, recognise files by their signatures, walk the chunks of real formats, pull apart flag bytes and nested records, defend a parser against hostile input, and design a format of your own. The next time you meet an unfamiliar binary file, the method is the same: find the signature, find the header, learn what each length and flag claims, and verify every claim before you trust it.