Compiling and Iterating

Regular Expressions in Practice

Chapter 6 · Compiling and Iterating

So far each call has handed re a pattern string and let it do the work. This chapter covers two tools that make larger jobs tidier: compiled patterns, which let you name a pattern once and reuse it, and iteration, which lets you walk through matches one at a time and see exactly where each one sits. Together they make it possible to write a small scanner: a loop that reads text from left to right and labels each piece.

What was run
As in the earlier chapters, every snippet was executed and the comments show what Python printed.

Compiling a pattern

re.compile(pattern, flags) returns a pattern object. It has the same operations as the module (search, match, fullmatch, findall, finditer, sub, split), but the pattern and flags are already fixed, so you do not pass them again.

import re pat = re.compile(r"\d{4}-\d{2}-\d{2}") print(pat.findall("on 2026-11-02 and 2027-01-15")) # ['2026-11-02', '2027-01-15'] print(pat.search("due 2026-11-02")) # <re.Match object; span=(4, 14), match='2026-11-02'> print(pat.sub("<date>", "on 2026-11-02 and 2027-01-15")) # on <date> and <date> print(pat.pattern) # \d{4}-\d{2}-\d{2}

Why bother?

  • A name for the idea. date_re or order_re says what the pattern is for, and one definition near the top of a file is easier to maintain than the same string pasted in three places.
  • Flags travel with it. re.compile(r"...", re.I | re.X) is written once; later calls need no flags=.
  • Speed, a little. The module functions compile the pattern behind the scenes and keep a small internal cache, so calling re.search repeatedly is not wildly slow. Compiling yourself skips that lookup and avoids any chance of recompiling when the cache is busy. Treat speed as a minor benefit and clarity as the main one.

A pattern object also reports facts about itself, and compiling an already-compiled pattern simply returns the same object.

p2 = re.compile(r"(?P<h>\d+):(?P<m>\d+)", re.I) print(p2.groups, p2.groupindex) # 2 {'h': 1, 'm': 2} words = re.compile(r"[a-z]+", re.I) print(words.findall("Hello, World")) # ['Hello', 'World'] print(re.compile(words) is words) # True

Where to put the compile

Compile at module level (or once before a loop), not inside a function that is called many times. The compiled object is plain data and can be shared freely.

Match objects and positions

findall returns strings, which is convenient but throws away where each match was. A match object keeps that. m.start(), m.end() and m.span() give positions in the original string, and m.group(n) or m["name"] gives the text.

finditer

finditer does what findall does, but yields a match object for each non-overlapping match, lazily, one at a time.

text = "Order 66 shipped; order 7 pending; ORDER 105 done" for m in re.finditer(r"order (\d+)", text, re.I): print(m.group(1), m.start(), m.end(), m.span()) # 66 0 8 (0, 8) # 7 18 25 (18, 25) # 105 35 44 (35, 44) print(re.findall(r"order (\d+)", text, re.I)) # ['66', '7', '105']

Use finditer when you need positions (to highlight, to slice the original text, to report line and column), when you want several groups by name, or when the text is large and you would rather not build one big list. Use findall when the strings are all you need.

pairs = re.compile(r"(?P<key>\w+)=(?P<val>\w+)") for m in pairs.finditer("a=1 b=22 c=x"): print(m.groupdict(), m.span("val")) # {'key': 'a', 'val': '1'} (2, 3) # {'key': 'b', 'val': '22'} (6, 8) # {'key': 'c', 'val': 'x'} (11, 12) d = {m["key"]: m["val"] for m in pairs.finditer("a=1 b=22 c=x")} print(d) # {'a': '1', 'b': '22', 'c': 'x'}

The iterator is single-use: once it has been walked, create a new one to go again. Empty matches are reported too, and the engine steps past them so the loop always ends:

print([m.span() for m in re.finditer(r"\d*", "a12")]) # [(0, 0), (1, 3), (3, 3)]

Starting part-way: pos and endpos

A compiled pattern's search, match, fullmatch, findall and finditer accept optional pos and endpos arguments that restrict where the engine looks, without slicing the string. Positions in the results still refer to the whole string.

p = re.compile(r"\d+") s = "ab12cd345" print(p.search(s, 3)) # <re.Match object; span=(3, 4), match='2'> print(p.match(s, 2)) # <re.Match object; span=(2, 4), match='12'> print(p.match(s, 0)) # None print(p.findall(s, 0, 4)) # ['12']

Two details are worth remembering. First, p.search(s, 3) starts looking at index 3, which is in the middle of 12, so it finds only '2'. Second, match(s, pos) insists that the match begin exactly at pos, which is the key to the scanner loop below. A caution: ^ does not match at pos unless pos is really the start of the string (or of a line, with re.M):

print(re.compile(r"^\d+").search(s, 2)) # None

So when scanning with pos, leave ^ out and let match do the anchoring.

A scanner with named groups

A tokenizer splits text into labelled pieces: numbers, names, operators. With regex the recipe is one alternation of named groups, one group per kind of token. The match object's lastgroup attribute gives the name of the group that matched, so you never need to test which group is not None.

TOKEN = re.compile(r""" (?P<NUM>\d+(?:\.\d+)?) | (?P<NAME>[A-Za-z_]\w*) | (?P<OP>[-+*/=()]) | (?P<SKIP>\s+) | (?P<BAD>.) """, re.VERBOSE) def tokens(src): for m in TOKEN.finditer(src): kind = m.lastgroup if kind == "SKIP": continue if kind == "BAD": raise ValueError(f"unexpected {m.group()!r} at {m.start()}") yield kind, m.group() print(list(tokens("rate = (3.5 + x1) * 2"))) # [('NAME', 'rate'), ('OP', '='), ('OP', '('), ('NUM', '3.5'), ('OP', '+'), # ('NAME', 'x1'), ('OP', ')'), ('OP', '*'), ('NUM', '2')] try: list(tokens("a $ b")) except ValueError as e: print("error:", e) # error: unexpected '$' at 2

Several design points are packed into that example:

  • Order matters. Alternatives are tried left to right, so put more specific kinds first. The final BAD group is a catch-all, so anything unrecognised surfaces as an error with a position instead of being silently skipped.
  • Full coverage. Because SKIP and BAD together cover every character, the matches tile the whole input with no gaps. Without a catch-all, finditer would skip unrecognised characters without a word.
  • It is a generator. yield hands tokens out lazily, so a caller can stop early.

A manual loop with match(string, pos)

finditer is allowed to skip over text that nothing matches. When you need the opposite, a hard guarantee that every character is accounted for, move a position forward yourself and use match at that position. If it fails, you know exactly where the input went wrong.

NUM = re.compile(r"\d+") WS = re.compile(r"\s*") def parse_ints(s): out, pos = [], 0 while pos < len(s): pos = WS.match(s, pos).end() if pos >= len(s): break m = NUM.match(s, pos) if not m: raise ValueError(f"bad input at {pos}: {s[pos]!r}") out.append(int(m.group())) pos = WS.match(s, m.end()).end() if pos < len(s): if s[pos] != ",": raise ValueError(f"expected comma at {pos}") pos += 1 return out print(parse_ints("1, 22 ,333")) # [1, 22, 333] try: parse_ints("1, x") except ValueError as e: print("error:", e) # error: bad input at 3: 'x'

Here WS can match an empty string, so WS.match(s, pos) always succeeds and simply returns the position after any spaces. The loop is the scanner: each pass consumes one piece, and pos only ever moves forward.

NeedReach for
Just the matched stringsfindall
Positions, named groups, or lazy processingfinditer
Labelled pieces covering all the textNamed groups with a catch-all, plus lastgroup
A guarantee about every character, with exact error positionsA while loop using match(s, pos)
Same pattern used in several placesre.compile once, near the top
lastgroup and lastindex
m.lastgroup is the name of the last matched capturing group, and m.lastindex is its number. With a top-level alternation of groups, that is the group that matched. If the matching group has no name, lastgroup is None but lastindex still works. Nested groups can make "last" surprising, so keep scanner groups flat and use (?:...) for any grouping inside them, as NUM does above.

Challenges

Exercise 1

Given "GET /a 200 12ms; POST /b 404 8ms; GET /c 200 130ms", compile one pattern with named groups method, path, status and ms. Use finditer to print each method, path and the position where its record starts, then print the total of the ms values.

📄 View solution
Exercise 2

Write a scanner for "go 12 steps, then 7 more!" with named groups INT, WORD, SP (spaces) and a catch-all X for anything else. Print the list of (kind, text) pairs without the spaces, and use collections.Counter on lastgroup to count how many tokens of each kind there were.

📄 View solution
Exercise 3

Split say 'hello world' to "big wide" world into words, keeping a quoted phrase together (single or double quotes) and removing the quote marks. Use one compiled pattern with an alternation and finditer; expected result: ['say', 'hello world', 'to', 'big wide', 'world']. Hint: a match has several groups, only one of which is not None.

📄 View solution

Quick Reference

  • re.compile(p, flags): returns a pattern object with the same methods as the module; compile once, reuse, and give it a meaningful name
  • finditer: lazy iterator of match objects; use start(), end(), span(), group(), groupdict()
  • findall vs finditer: strings only versus full match objects with positions
  • pos / endpos: limit the search on a compiled pattern; results still index the whole string; ^ ignores pos
  • match(s, pos): must match exactly at pos; the basis of a manual scanner loop
  • Tokenizer recipe: flat named groups in one alternation, a catch-all last, and m.lastgroup to label each piece
Coming next
Regular Expressions in Practice 7 applies everything so far to validating real-world data such as dates, phone numbers and email addresses, and looks honestly at where regular expressions stop being the right tool.