Magic Numbers and Headers

Binary File Formats in Practice

Chapter 4 · Magic Numbers and Headers

A file name is only a label. Rename a text file to photo.png and nothing about its bytes changes. Programs that need to know what a file really is look at its first few bytes, where most binary formats put a fixed signature, often called a magic number. After the signature comes the header: the fields that describe everything that follows. This chapter uses the struct tools from Binary File Formats in Practice 3 to read both.

What was run

Every snippet and solution in this chapter was executed, and the output shown in the comments is what Python printed.

Magic Numbers

A magic number is a short, fixed byte sequence at a known position, nearly always offset 0. It is chosen to be unlikely to appear at the start of anything else. Some are readable text, some deliberately are not:

Format	Signature (hex)	As text
PNG	`89 50 4e 47 0d 0a 1a 0a`	`.PNG` then CR LF, Ctrl-Z, LF
JPEG	`ff d8 ff`	not printable
GIF	`47 49 46 38 37 61` or `...39 61`	`GIF87a` / `GIF89a`
PDF	`25 50 44 46 2d`	`%PDF-`
ZIP	`50 4b 03 04`	`PK` then two control bytes
gzip	`1f 8b`	not printable

These values are widely documented but worth confirming against each format's specification before you rely on them for anything important (check). The PNG signature is a good example of careful design: the high-bit first byte stops it being mistaken for text, the letters name the format, and the CR LF, Ctrl-Z and LF bytes make common text-mode file transfers that mangle line endings show up as a broken signature.

Identifying a File by Its Signature

The technique is a table of signatures and a loop with bytes.startswith. Order matters when one signature could begin another, so put the longest and most specific first:

SIGNATURES = [
    (b"\x89PNG\r\n\x1a\n", "PNG image"),
    (b"\xff\xd8\xff", "JPEG image"),
    (b"GIF87a", "GIF image"),
    (b"GIF89a", "GIF image"),
    (b"%PDF-", "PDF document"),
    (b"PK\x03\x04", "ZIP archive"),
    (b"\x1f\x8b", "gzip data"),
]

def identify(head):
    for magic, name in SIGNATURES:
        if head.startswith(magic):
            return name
    return "unknown"

print(identify(b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR"))   # PNG image
print(identify(b"%PDF-1.7\n%..."))                   # PDF document
print(identify(b"hello world"))                      # unknown
print(identify(b""))                                 # unknown
    

With a real file you only need to read as many bytes as your longest signature. Sixteen is plenty here, and reading a few bytes is far cheaper than reading the whole file. A file shorter than the signature simply fails to match; startswith never raises an error for short input.

import os, tempfile
path = os.path.join(tempfile.gettempdir(), "demo_sig.bin")
with open(path, "wb") as f:
    f.write(b"PK\x03\x04" + bytes(20))
with open(path, "rb") as f:
    head = f.read(16)
print(len(head), identify(head))     # 16 ZIP archive
os.remove(path)

print(b"\x89PNG\r\n\x1a\n".hex(" "))   # 89 50 4e 47 0d 0a 1a 0a
print(b"GIF89a"[:3], b"%PDF-1.7"[5:8])  # b'GIF' b'1.7'
    

A signature is evidence, not proof

A matching signature only says the file starts like that format. A truncated or corrupt file can still match, and a hostile file can be built to. Identify first, then validate the rest of the structure as you parse it (more on this in Binary File Formats in Practice 9).

What a Header Holds

Right after the signature a header usually carries a handful of fixed-size fields. The common ones are:

Version: which revision of the format wrote the file.
Flags: optional features switched on or off (Binary File Formats in Practice 7 takes these apart bit by bit).
Counts or sizes: how many records follow, or how long the data is.
Offsets: where to find the next section, so a reader can jump straight to it.

A header is a fixed-layout record, so a struct.Struct describes it exactly. Here is a small invented format: a 4-byte tag, a major and minor version byte, a 16-bit flags value and a 32-bit record count, 12 bytes in all. The parser checks each assumption before trusting the next:

import struct

HEADER = struct.Struct("<4sBBHI")
MAGIC = b"DEMO"

def build(major, minor, flags, count):
    return HEADER.pack(MAGIC, major, minor, flags, count)

def parse_header(raw):
    if len(raw) < HEADER.size:
        raise ValueError(f"header needs {HEADER.size} bytes, got {len(raw)}")
    magic, major, minor, flags, count = HEADER.unpack_from(raw)
    if magic != MAGIC:
        raise ValueError(f"bad magic {magic!r}")
    if major != 1:
        raise ValueError(f"unsupported major version {major}")
    return {"major": major, "minor": minor, "flags": flags, "count": count}

print(HEADER.size)                    # 12
h = build(1, 3, 5, 42)
print(h.hex(" "))                   # 44 45 4d 4f 01 03 05 00 2a 00 00 00
print(parse_header(h))                # {'major': 1, 'minor': 3, 'flags': 5, 'count': 42}

for bad in (b"DEMO", b"XXXX" + h[4:], build(2, 0, 0, 0)):
    try:
        parse_header(bad)
    except ValueError as e:
        print("ValueError:", e)
        # ValueError: header needs 12 bytes, got 4
        # ValueError: bad magic b'XXXX'
        # ValueError: unsupported major version 2
    

The order of the checks is deliberate: enough bytes, then the right signature, then a version this code understands. Only after all three does the count mean anything.

Versions

Formats change. A version field lets a reader decide, before it misreads anything, whether it can handle a file. A common convention splits the version in two:

Major changes when the layout changes in a way old readers cannot cope with. A reader refuses a major version it does not know.
Minor changes when something is added in a way old readers can safely ignore. A reader accepts any minor version of a major it knows.

That is why parse_header above rejected major 2 but happily accepted minor 3. Refusing clearly is much better than reading garbage and carrying on.

Header Length: Room to Grow

A fixed header size is a trap: add one field and every old reader breaks. Many formats instead store the header's own length, so a reader takes the fields it understands and uses the length to jump to the payload. Fields added later simply sit in the part it skips:

V1 = struct.Struct("<4sBBH")

def build2(extra=b""):
    hl = V1.size + 2 + len(extra)
    return V1.pack(b"DEMO", 1, 0, 0) + struct.pack("<H", hl) + extra

def read_header(raw):
    magic, major, minor, flags = V1.unpack_from(raw)
    (hl,) = struct.unpack_from("<H", raw, V1.size)
    return hl, raw[hl:]

a = build2() + b"PAYLOAD"
b = build2(b"\x01\x02\x03\x04") + b"PAYLOAD"
print(read_header(a), read_header(b))   # (10, b'PAYLOAD') (14, b'PAYLOAD')
print(a.hex(" "))   # 44 45 4d 4f 01 00 00 00 0a 00 50 41 59 4c 4f 41 44
print(b.hex(" "))   # 44 45 4d 4f 01 00 00 00 0e 00 01 02 03 04 50 41 59 4c 4f 41 44
    

Both files are read by the same code even though the second has four extra header bytes. In real files, treat a stored length as a claim to be checked: it must be at least the size you need and no larger than the data you actually have.

Design choice	Benefit	Cost
Magic number	Identify the format without trusting the name	A few bytes of every file
Major / minor version	Readers can refuse or accept safely	Everyone must follow the convention
Header length field	New fields do not break old readers	One more value to validate
Offset to data	Jump straight to a section	Offsets can point outside the file

Challenges

Exercise 1

Write identify_file(path) using the signature table from this chapter. It reads at most 16 bytes and returns the format name, "unknown", or "empty file". In a temporary folder create a PNG-like file, a PDF-like file, a GIF-like file, a plain text file, a text file named e.png, an empty file, a real ZIP made with zipfile and a real gzip file made with gzip. Print the result for each, and confirm the folder is gone afterwards.

📄 View solution

Exercise 2

Using the 12-byte "<4sBBHI" header, write write_demo(path, major, minor, flags, records) (records are unsigned 16-bit values written after the header) and read_demo(path), which returns ((major, minor), flags, values). Reject a truncated header, a wrong magic, a major version other than 1, and a body shorter than the count claims. Show two good files (minor versions 0 and 7) and all four failures, then delete the temporary file.

📄 View solution

Exercise 3

Design a header of "<4sBBH" followed by a 16-bit header_length (10 bytes minimum). Write write_file(extra, payload) that sets the length correctly and read_file(raw) that returns the flags, the header length and the payload. Show one file with no extra header bytes and one with a 4-byte extra field, read both with the same function, and reject lengths that are too small or larger than the data.

📄 View solution

Quick Reference

Signature check: read the first bytes, then head.startswith(magic)
Read only what you need: f.read(16) is enough for the table in this chapter
Header as a record: a module-level struct.Struct("<...") with .size and unpack_from
Check order: enough bytes, then magic, then version, then trust counts and lengths
Versions: refuse an unknown major version; accept any minor of a known major
Header length field: read what you know, skip to the stored length, and validate it first
Remember: a matching signature is evidence, not proof

Coming next

Binary File Formats in Practice 5 reads a real format end to end: the PNG signature, its chunk structure, the length fields, and how a CRC from zlib proves each chunk arrived intact.