Bit Fields and Flags
Binary File Formats in Practice
Chapter 7 · Bit Fields and Flags
Every field so far has been a whole number of bytes. Real formats are thriftier: a single byte often carries several yes/no answers, or a small number squeezed in beside them, because a header that is read millions of times is worth keeping short. This chapter opens up the byte. It covers the bitwise operators, the standard recipes for testing, setting, clearing and extracting bits, and then reads two real packed fields. It builds on Binary File Formats in Practice 2 and Binary File Formats in Practice 4.
Bits Are Numbered From the Right
A byte has eight bits. Bit 0 is the least significant, the rightmost one, worth 1; bit 7 is the leftmost, worth 128. Format documents nearly always number bits this way, so "bit 3" means the value 8 (1 << 3). Do not confuse this with byte order from Binary File Formats in Practice 2: endianness is about the order of whole bytes in a larger number, while bit numbering inside one byte is fixed by place value.
The Operators
| Operator | Name | Effect on each bit |
|---|---|---|
a & b | AND | 1 only where both are 1: keeps bits |
a | b | OR | 1 where either is 1: turns bits on |
a ^ b | XOR | 1 where they differ: flips bits |
~a | NOT | Flips every bit |
a << n | Shift left | Moves bits up n places (multiplies by 2n) |
a >> n | Shift right | Moves bits down n places, dropping the low ones |
bin() drops leading zeros, which is why 0b00001111 masked with 178 prints as 0b10. The value is the same; only the display is shorter. Use format(n, "08b") when you want all eight digits.
A Trap: NOT in Python
In C, ~ on a byte stays within a byte. Python integers have no fixed width, so ~b is defined as -b - 1, a negative number. To get the byte you meant, mask it back down:
This matters mostly when clearing a bit: flags & ~MASK works in Python without the extra & 0xFF, because a negative number behaves as if it had endless leading 1 bits and the AND with a small positive number gives the right small answer. It is only when you store or print a bare ~x that the negative number surprises you.
Flags: Test, Set, Clear, Toggle
A flags byte gives each bit its own meaning. Name each bit as a one-bit mask, then four recipes cover everything:
| Goal | Recipe | Why it works |
|---|---|---|
| Test | f & MASK | Non-zero only if that bit is on |
| Set | f | MASK | OR can only turn bits on |
| Clear | f & ~MASK | The inverted mask keeps everything except that bit |
| Toggle | f ^ MASK | XOR with 1 flips, XOR with 0 leaves alone |
Fields Wider Than One Bit
A bit field may be several bits wide: a 2-bit version number, a 4-bit length. To read one, shift it down to bit 0, then mask to its width. A mask of width w is (1 << w) - 1: width 2 gives 0b11, width 4 gives 0b1111. To write one, clear its old bits and OR in the new value shifted up:
The new >> width check refuses a value too big for the field. Without it, the extra bit would silently spill into the neighbouring field, which is the classic bug with packed data. Reading is forgiving, since the mask trims anything extra; writing needs the guard.
A Real Flags Byte: gzip
The gzip file format starts with a 10-byte header whose fourth byte, FLG, is a flags byte. Per the gzip specification (RFC 1952) (check): bit 0 is a text hint, bit 1 a header checksum, bit 2 an extra-data field, bit 3 a stored file name and bit 4 a comment; bits 5 to 7 are reserved and must be zero. Python's gzip module will make a sample for us in memory:
1f 8b is the gzip magic number from Binary File Formats in Practice 4, 08 the compression method (deflate) and the next 08 is FLG: only bit 3 is on, so a file name follows the header. The last two bytes of the header (02 and ff) are extra flags and an operating-system code that this chapter does not need (check what they mean in the specification before relying on them).
Naming Bits With IntFlag
Hand-written masks are error-prone. The standard library's enum.IntFlag gives every bit a name and still behaves as an integer, so it reads and writes straight into a struct:
The text shown when you print an IntFlag differs between Python versions (check), so this chapter prints plain integers. Note too that older versions of IntFlag accept unknown bits silently, while newer ones may raise or keep them depending on the class's boundary setting (check), so a format reader should still test the reserved bits itself.
A Packed 16-Bit Field: the DOS Date
Some formats pack a whole date into two bytes. The classic MS-DOS layout, still used by the ZIP format for file times (check), gives the year 7 bits, the month 4 and the day 5:
| Bits | Width | Field |
|---|---|---|
| 15 to 9 | 7 | Year, counted from 1980 |
| 8 to 5 | 4 | Month, 1 to 12 |
| 4 to 0 | 5 | Day, 1 to 31 |
The whole date is one 16-bit integer, then stored little-endian like any other: bit packing decides the value, and the earlier chapters' byte-order rules decide how that value sits in the file. Reading is the reverse: struct.unpack("<H", ...) first, then shifts and masks. Because the year has only 7 bits, the format cannot represent a date after 2107 (1980 + 127).
| Idea | Byte-sized fields | Bit fields |
|---|---|---|
| Smallest unit | 1 byte, read with struct | 1 bit, read with shifts and masks |
| Order question | Endianness (Binary File Formats in Practice 2) | Bit 0 is the least significant; fixed |
| Writing risk | Wrong size or sign | A value too big for its width spills into a neighbour |
| Validation | Check lengths | Check reserved bits are zero |
Reserved Bits
Formats usually mark some bits "reserved, must be zero". A strict reader rejects a byte with those bits set; a lenient one ignores them. The choice has consequences: rejecting means an old reader refuses files from a newer writer that started using the bit, while ignoring means a corrupted byte can slip through unnoticed. This course's readers are strict, and say so in an error message that includes the offending bits.
Challenges
Define four flags (READONLY, HIDDEN, COMPRESSED, ENCRYPTED) in bits 0 to 3 of a byte, with bits 4 to 7 reserved. Write decode_flags(byte) returning the list of names that are set, and encode_flags(names) returning the byte. Reject a byte with reserved bits set (including the bits in the message) and an unknown flag name. Show an encode and decode, the empty case, both failures, and prove that decoding then encoding returns the original for every value from 0 to 15.
Write pack_date/unpack_date and pack_time/unpack_time for the DOS layouts (time: hour in bits 15 to 11, minute in bits 10 to 5, seconds divided by 2 in bits 4 to 0). Reject out-of-range values. Show 2026-11-02 and 14:30:45 packed as hex and as little-endian bytes, show that the seconds lose their odd digit on a round trip, show three rejections, and round-trip a spread of dates.
Write read_header(raw) for the gzip header: check the magic number, the method (8) and the reserved bits of FLG, return the list of set flag names and the file name when the name flag is set (a zero-terminated string after the 10-byte header). Refuse the extra-data flag as not handled. Build files with gzip.GzipFile with and without a name, show both results, show five failures (magic, short, reserved, unterminated name, extra flag), and confirm gzip.decompress still works on the named file.
Quick Reference
- Bit numbering: bit 0 is the least significant (value 1); bit n has value
1 << n - Test / set / clear / toggle:
f & M,f | M,f & ~M,f ^ M - Read a field:
(v >> shift) & ((1 << width) - 1) - Write a field: clear with
v & ~mask, then OR innew << shift; reject values that do not fit - NOT:
~xis negative in Python; mask with& 0xFFto get a byte - Multi-byte fields: unpack the integer with
structfirst, then shift and mask - Reserved bits: decide on purpose whether to reject or ignore them
- IntFlag: names for bits; print integers to avoid version differences