Bytes and Hex

Binary File Formats in Practice 🏆

Chapter 1 · Bytes and Hex

Images, audio, archives and documents are all binary files: sequences of bytes with an agreed layout. Once you can read those bytes yourself, a file format stops being a mystery and becomes a specification you can follow. This course builds that skill step by step, starting here with the raw material. All examples use Python 3 with only the standard library, so there is nothing to install.

What was run

Every snippet in this chapter was executed, and the output shown in the comments is what Python printed. Run them yourself to see the same results.

Bytes Are Small Integers

A byte is a number from 0 to 255. Python's bytes type is an immutable sequence of them. A literal looks like a string with a b in front, and bytes that are not printable characters are shown as \x followed by two hex digits.

data = b"Hi!\x00\xff"
print(data)                          # b'Hi!\x00\xff'
print(len(data))                     # 5
print(data[0], data[4])              # 72 255
print(data[0:2])                     # b'Hi'
print(type(data[0]), type(data[0:1]))  # <class 'int'> <class 'bytes'>
print(list(data))                    # [72, 105, 33, 0, 255]
    

Notice the difference between indexing and slicing: data[0] is an int (72), while data[0:1] is a one-byte bytes object. Forgetting this is the most common early mistake, for example comparing data[0] == b"H", which is never true.

Type	Holds	Mutable?	Example
str	characters (text)	no	`"Hi"`
bytes	bytes, 0 to 255	no	`b"Hi"`
bytearray	bytes, 0 to 255	yes	`bytearray(b"Hi")`

Making Bytes

Besides literals, you can build bytes from a list of integers, from hex text, or by encoding a string. Values outside 0 to 255 are rejected, and a bytes object cannot be changed in place.

print(data.hex())                    # 48692100ff
print(bytes.fromhex("48 69 21 00 ff"))  # b'Hi!\x00\xff'
print(bytes([72, 105, 33]))          # b'Hi!'
print("é".encode("utf-8"))            # b'\xc3\xa9'

try:
    data[0] = 65
except TypeError as e:
    print("TypeError:", e)          # TypeError: 'bytes' object does not support item assignment

try:
    bytes([256])
except ValueError as e:
    print("ValueError:", e)         # ValueError: bytes must be in range(0, 256)
    

When you need to change bytes, use a bytearray. It behaves like bytes but allows assignment, append and similar edits. Convert back with bytes(...) when you are done.

buf = bytearray(data)
buf[0] = 74                      # 74 is the code for "J"
buf.append(10)
print(buf)                       # bytearray(b'Ji!\x00\xff\n')
print(bytes(buf))                # b'Ji!\x00\xff\n'
    

Reading Hex and Binary

Hexadecimal (base 16) is the standard way to write bytes because one byte is exactly two hex digits, from 00 to ff. Python accepts 0x and 0b prefixes and has formatting helpers for the other direction.

print(0x48, 0xff, 0b1010)          # 72 255 10
print(hex(255), bin(10))            # 0xff 0b1010
print(f"{255:08b}", f"{10:02x}")       # 11111111 0a
    

Files in Binary Mode

Open a file with "rb" or "wb" and Python gives you bytes with no text decoding and no newline translation. read(n) returns the next n bytes, tell() reports your position, and seek() moves it. Binary mode matters: opening a binary file in text mode can corrupt data or raise decoding errors.

import os, tempfile

path = os.path.join(tempfile.gettempdir(), "demo.bin")
with open(path, "wb") as f:
    n = f.write(bytes(range(0, 40, 3)))
print(n)                         # 14

with open(path, "rb") as f:
    head = f.read(4)
    rest = f.read()
print(head, rest)                # b'\x00\x03\x06\t' b"\x0c\x0f\x12\x15\x18\x1b\x1e!$'"
print(os.path.getsize(path))     # 14

with open(path, "rb") as f:
    f.seek(-3, 2)               # 2 means "relative to the end"
    print(f.read(), f.tell())    # b"!$'" 14
os.remove(path)
    

The odd-looking output is just repr choosing its quoting. Byte 9 is a tab, shown as \t, and bytes 33, 36 and 39 are the printable characters !, $ and '. Because one of them is an apostrophe, Python switched the whole display to double quotes.

Build a Hex Dump

A hex dump shows a file the way a format specification describes it: an offset, the bytes in hex, and a text column where printable characters appear and everything else is a dot. This small function is the tool you will reach for throughout the course.

def hexdump(data, width=16):
    lines = []
    for offset in range(0, len(data), width):
        chunk = data[offset:offset + width]
        hexpart = " ".join(f"{b:02x}" for b in chunk)
        text = "".join(chr(b) if 32 <= b < 127 else "." for b in chunk)
        lines.append(f"{offset:08x}  {hexpart:<{width * 3 - 1}}  |{text}|")
    return "\n".join(lines)

sample = b"Binary files: bytes, not text.\n\x00\x01\x02\xff\xfe"
print(hexdump(sample))
    

The output, exactly as printed:

00000000  42 69 6e 61 72 79 20 66 69 6c 65 73 3a 20 62 79  |Binary files: by|
00000010  74 65 73 2c 20 6e 6f 74 20 74 65 78 74 2e 0a 00  |tes, not text...|
00000020  01 02 ff fe                                      |....|
    

The first column is the offset in hex, so the second row starts at byte 16 (0x10) and the third at byte 32 (0x20). In the second row the final 2e is a real full stop, while the 0a (a newline) and 00 after it are not printable, so they show as two more dots, giving ... after "text". The padding in the last row (:<{width * 3 - 1}) keeps the text column lined up when the final chunk is short.

A habit worth forming

Before writing any parsing code, look at the file with a hex dump. Open in "rb", remember that indexing gives an int and slicing gives bytes, and use a bytearray only when you must edit in place.

Challenges

Exercise 1

Take the string "Café". Encode it to UTF-8 and print the bytes, print the hex with a space between bytes (look at the optional separator argument of hex()), print the length of the string and the length of the bytes, and confirm that decoding gives back the original string. Predict the two lengths first.

📄 View solution

Exercise 2

Write every byte value from 0 to 255 once into a temporary file, then read it back in binary mode. Print the length, how many bytes are 128 or greater, and the first four bytes. Then reopen the file, use seek to jump to five bytes before the end, and print those five bytes as spaced hex. Delete the file afterwards.

📄 View solution

Exercise 3

Write find_all(data, pattern) that returns a list of every offset where pattern occurs in data, including overlapping matches. Use bytes.find in a loop. Test it with b"abcabcab" and b"ab", with b"aaaa" and b"aa", and with a pattern that is not present.

📄 View solution

Quick Reference

bytes is immutable, bytearray is mutable; both hold values 0 to 255
Index gives an int; slice gives bytes
Convert: .hex() and bytes.fromhex(); str.encode() and bytes.decode()
Hex and binary: 0x.., 0b.., f"{n:02x}", f"{n:08b}"
Files: "rb" and "wb"; read(n), tell(), seek(offset, whence)
Hex dump your data before you try to parse it

Coming next

Binary File Formats in Practice 2 turns groups of bytes into numbers: how int.from_bytes works, what endianness means, and how signed values are stored.