The struct Module

Binary File Formats in Practice

Chapter 3 · The struct Module

Reading a header with int.from_bytes means slicing the bytes by hand for every field. A real header has a dozen of them. The standard struct module describes the whole layout in one short format string and converts the entire group in a single call.

What was run

Every snippet and solution in this chapter was executed, and the output shown in the comments is what Python printed.

Format Strings

A format string is an optional byte-order character followed by one code per field. struct.pack(fmt, *values) builds bytes and struct.unpack(fmt, raw) returns a tuple, always a tuple, even for one field. Here H is an unsigned 16-bit integer and I an unsigned 32-bit one:

import struct

print(struct.pack("<HI", 1, 2).hex(" "))   # 01 00 02 00 00 00
print(struct.pack(">HI", 1, 2).hex(" "))   # 00 01 00 00 00 02
print(struct.unpack("<HI", b"\x01\x00\x02\x00\x00\x00"))  # (1, 2)
print(struct.calcsize("<HI"))                  # 6
    

The first character picks the byte order, and as in Binary File Formats in Practice 2 it should always be stated for file work: < little-endian, > big-endian. struct.calcsize(fmt) tells you how many bytes a format occupies, which is the number you pass to f.read().

Code	Meaning	Size	Python type
b / B	signed / unsigned byte	1	int
h / H	signed / unsigned short	2	int
i / I	signed / unsigned int	4	int
q / Q	signed / unsigned long long	8	int
f / d	float / double	4 / 8	float
?	boolean	1	bool
4s	fixed-length byte string (4 bytes)	4	bytes
x	one padding byte, no value	1	none

A count before a code repeats it, so "<3H" is three unsigned shorts. The exception is s, where the count is the length of one byte string rather than a repeat:

print(struct.unpack("<3H", b"\x01\x00\x02\x00\x03\x00"))   # (1, 2, 3)
print(struct.pack("<4s", b"ab"))                      # b'ab\x00\x00'
print(struct.pack("<4s", b"abcdef"))                  # b'abcd'
print(struct.pack("<?", True), struct.unpack("<?", b"\x05"))  # b'\x01' (True,)
    

Notice that a short byte string is padded with zero bytes and a long one is silently cut. Neither raises an error, so check lengths yourself when truncation would matter.

A Header in One Call

Suppose a file starts with a 4-byte tag, a 16-bit version, a 16-bit flags value and a 32-bit count, all little-endian. One format describes it, and unpacking gives named variables directly:

data = struct.pack("<4sHHI", b"DEMO", 2, 7, 1000)
print(data.hex(" "))                 # 44 45 4d 4f 02 00 07 00 e8 03 00 00

magic, ver, flags, count = struct.unpack("<4sHHI", data)
print(magic, ver, flags, count)       # b'DEMO' 2 7 1000
    

Compare that with four slices and four calls to int.from_bytes. The format string also doubles as documentation: it is the layout, written once.

Alignment and Padding

The byte-order character has a second job. With @ (the default when you give none) or =, struct uses your machine's native rules, and @ also inserts padding so each field starts at a multiple of its own size, as a C compiler would. The characters <, >, ! and = use standard sizes; of those, only = keeps native byte order, and none of them pad.

print(struct.calcsize("@bi"), struct.calcsize("<bi"))   # 8 5
print(struct.pack("@bi", 1, 2).hex(" "))      # 01 00 00 00 02 00 00 00
print(struct.pack("<bi", 1, 2).hex(" "))      # 01 02 00 00 00
print(struct.calcsize("@bxxxi"))             # 8
    

The native results above are from one machine; sizes and padding bytes depend on the platform's rules (check on yours). That is exactly the danger: a file written with @ is not portable. Padding is also not exclusive to @ in the file itself: some formats contain padding bytes on purpose, and for those you spell them out with x.

Prefix every file format string

Start with < or > (or ! for network order, which is big-endian). Then sizes are fixed, there is no hidden padding, and the code behaves the same on every machine.

Offsets, Buffers and Struct Objects

Files are rarely just a header. unpack_from reads at an offset without slicing, pack_into writes into an existing bytearray, and a struct.Struct object compiles a format once and gives you the same methods plus .size:

buf = b"\xaa\xbb" + data + b"\xcc"
print(struct.unpack_from("<4sHHI", buf, 2))   # (b'DEMO', 2, 7, 1000)
print(struct.unpack_from("<H", buf, 6))        # (2,)

ba = bytearray(12)
struct.pack_into("<HH", ba, 4, 5, 6)
print(ba.hex(" "))                       # 00 00 00 00 05 00 06 00 00 00 00 00

rec = struct.Struct("<HhI")
print(rec.size, rec.format)             # 8 <HhI
blob = rec.pack(1, -1, 70000) + rec.pack(2, -2, 80000)
for t in rec.iter_unpack(blob):
    print(t)
    # (1, -1, 70000)
    # (2, -2, 80000)
    

iter_unpack is the natural tool for a table of equal-sized records: it steps through the buffer one record at a time. Because every record is the same size, record number n sits at offset n times rec.size.

When the Bytes Do Not Fit

struct is strict. A buffer of the wrong length, a value out of range, or a missing value all raise struct.error. Treat it as useful information about the file or your code, not an annoyance:

try:
    struct.unpack("<I", b"\x01\x02")
except struct.error as e:
    print("struct.error:", e)   # struct.error: unpack requires a buffer of 4 bytes

try:
    struct.pack("<B", 256)
except struct.error as e:
    print("struct.error:", e)   # struct.error: 'B' format requires 0 <= number <= 255

try:
    struct.pack("<HH", 1)
except struct.error as e:
    print("struct.error:", e)   # struct.error: pack expected 2 items for packing (got 1)
    

The exact wording of these messages can differ between Python versions (check).

Task	Chapter 2 tools	struct
One number	`int.from_bytes(raw, "little")`	`struct.unpack("<I", raw)[0]`
Whole header	Slice and convert each field	One format string, one call
Odd sizes (3 bytes)	Works directly	No code for it; use `int.from_bytes`
Table of records	Loop with slicing	`Struct.iter_unpack`

Challenges

Exercise 1

For each of the formats hi, bhb, bbh, qb and bq, print struct.calcsize with the @ prefix and with the < prefix. Then print the hex of struct.pack("@bi", 1, 2) and struct.pack("<bi", 1, 2), and the size of "@bxxxi". Explain every difference between the two prefixes in terms of padding.

📄 View solution

Exercise 2

Using a struct.Struct("<HhI") record, write the four records (1, -5, 1000), (2, 300, 70000), (3, -32768, 4294967295) and (4, 0, 0) to a temporary file, read it back, and print the file length against four times the record size. Check that iter_unpack returns the original list, then use unpack_from to fetch the third record directly by computing its offset. Delete the file afterwards.

📄 View solution

Exercise 3

Define a record "<8sH" holding an 8-byte ASCII name and a 16-bit value. Write pack_record(name, value) and unpack_record(raw) so that the name comes back without its zero padding. Show the packed hex for "temp", round-trip an exactly 8-character name, show what happens to an 11-character name, and catch the error raised when only 5 bytes are supplied.

📄 View solution

Quick Reference

Build / parse: struct.pack(fmt, ...), struct.unpack(fmt, raw) (always a tuple)
Size: struct.calcsize(fmt) or Struct(fmt).size
Prefix: < little, > big, ! network; avoid bare or @ formats for files
Codes: B H I Q unsigned, b h i q signed, f d floats, ? bool, Ns bytes, x pad
At an offset: unpack_from(fmt, buf, offset), pack_into(fmt, buf, offset, ...)
Tables: Struct.iter_unpack(buf); record n starts at n times the size
Errors: struct.error for wrong length, out-of-range or missing values

Coming next

Binary File Formats in Practice 4 puts these tools to work on real files: magic numbers that identify a format, how headers are laid out, and how version fields let a reader decide whether it understands what it has been given.