Pitfalls

Regular Expressions in Practice

Chapter 9 · Performance and Pitfalls

Most patterns are fast and most mistakes are quiet: the pattern runs, returns something, and the something is wrong. This chapter collects the problems that cause the most trouble in practice. First comes the rare but severe one, a pattern that takes minutes to fail. Then come the everyday traps, and finally the question of when a plain string method is the better tool.

What was run

Every snippet was executed and the comments show what Python printed. Timings differ from machine to machine, so the examples compare timings with a test rather than printing raw numbers.

Backtracking in one paragraph

When a pattern has a choice to make (a quantifier that can take more or fewer characters, or an alternation), the engine picks one option and remembers the others. If the rest of the pattern fails, it goes back and tries the next option. That is backtracking, and it is what makes patterns flexible. It is harmless when there are few choices. It becomes a problem when the number of ways to split the text grows very fast.

Catastrophic backtracking

The classic cause is a quantifier inside a quantifier, where both can match the same characters. In ^(a+)+$ the inner a+ and the outer + can share a run of letters in many different ways. If the text then fails at the very end (a trailing b), the engine tries every one of those ways before giving up. Each extra character roughly doubles the work.

import re, time

def took(pattern, text):
    start = time.perf_counter()
    pattern.search(text)
    return time.perf_counter() - start

risky = re.compile(r"^(a+)+$")
t18 = took(risky, "a" * 18 + "b")
t22 = took(risky, "a" * 22 + "b")
print("four more characters, at least 5x slower:", t22 > 5 * t18)

safe = re.compile(r"^a+$")
print("safe pattern, 100000 characters, under 0.1s:", took(safe, "a" * 100000 + "b") < 0.1)
# four more characters, at least 5x slower: True
# safe pattern, 100000 characters, under 0.1s: True
    

Four extra characters made the risky pattern many times slower, and the growth continues: a few dozen characters would take longer than you would wait. The safe pattern handles a hundred thousand characters with no trouble. Both patterns accept exactly the same strings. Only the amount of work differs.

This matters most when the text comes from someone else, such as a web form. A user who submits a long string that nearly matches can tie up your program. This is sometimes called a regular expression denial of service.

How to avoid it

Do not put a repeat around something that already repeats, unless the pieces cannot match the same text. (a+)+ is the warning sign.
Be careful with alternations whose branches overlap inside a repeat, such as (a|aa)+.
Prefer a specific class over .*. A pattern that can only match the right characters has fewer choices to explore.
Test with a long near-miss, not only with matching text. The slow case is the one that fails at the end.
Limit the length of untrusted input before matching it.

# Removing the inner repeat gives the same meaning with no nested choice
print(re.search(r"^(?:a+)$", "a" * 22 + "b"))
print(re.search(r"^(a|aa)+$", "a" * 3) is not None)
# None
# True
    

Pitfall: greedy matching across a boundary

Chapter 1 introduced greedy and lazy quantifiers. It is worth repeating because it is the most common wrong answer. A greedy .* runs to the last possible end, so it can swallow several items at once. There are two fixes, and the second is usually better:

html = "<b>one</b> and <b>two</b>"
print(re.findall(r"<b>.*</b>", html))
print(re.findall(r"<b>.*?</b>", html))
print(re.findall(r"<b>([^<]*)</b>", html))
# ['<b>one</b> and <b>two</b>']
# ['<b>one</b>', '<b>two</b>']
# ['one', 'two']
    

The lazy form stops at the first closing tag. The negated class [^<]* says what the text is allowed to contain, and so it also gives the engine nothing to backtrack over. A class that excludes the closing delimiter is both clearer and faster.

Pitfall: match is not search

re.match only tries the start of the string. Reaching for it when you meant search gives None and no error message.

print(re.match(r"\d+", "ab12"))
print(re.search(r"\d+", "ab12").group())
# None
# 12
    

Pitfall: the dollar sign and the newline

$ matches at the end of the string and just before a final newline. So a value that looks validated can still carry a newline. For validation, use fullmatch, or \Z, which matches only at the true end.

print(re.search(r"^\d+$", "42\n"))
print(re.fullmatch(r"\d+", "42\n"))
print(re.search(r"^\d+\Z", "42\n"))
# <re.Match object; span=(0, 2), match='42'>
# None
# None
    

Pitfall: \d is wider than 0 to 9

For text (str) patterns, \d matches digits from other writing systems too, which may not be what a validator wants. Use [0-9] or the re.ASCII flag when you mean the ordinary digits.

arabic = "\u0664\u0662"
print(bool(re.fullmatch(r"\d+", arabic)))
print(bool(re.fullmatch(r"[0-9]+", arabic)))
print(bool(re.fullmatch(r"\d+", arabic, re.A)))
# True
# False
# False
    

Pitfall: alternation takes the first branch that works

Alternation is tried left to right and the first branch that lets the whole pattern succeed wins, even if a later branch would match more. Put longer alternatives first.

print(re.match(r"cat|category", "category").group())
print(re.match(r"category|cat", "category").group())
# cat
# category
    

Pitfall: user text inside a pattern

If a pattern is built from text you did not write, characters such as ., + and ( will be read as pattern syntax. re.escape turns them into literals. Without it, the search can match the wrong text, or fail to compile.

user = "a.b"
print(re.findall(user, "a.b axb"))
print(re.findall(re.escape(user), "a.b axb"))
print(re.escape("1.5+2"))
# ['a.b', 'axb']
# ['a.b']
# 1\.5\+2
    

When not to use a pattern

A pattern is a tool for text with shape. If you are looking for fixed text, the built-in string methods are shorter, faster and cannot be broken by special characters.

name = "report_2026.txt"
print(name.endswith(".txt"), name.startswith("report"), "2026" in name)
print(name.replace("_", "-"), name.split("."))
# True True True
# report-2026.txt ['report_2026', 'txt']
    

Situation	Better tool	Why
Fixed text present or absent	`in`, `startswith`, `endswith`	No syntax to get wrong
Replace one fixed piece of text	`str.replace`	Clear and fast
Split on one fixed separator	`str.split`	No pattern needed
Text with variable shape (digits, optional parts)	`re`	This is what patterns are for
Nested or recursive structure (HTML, JSON, source code)	A real parser	A pattern cannot track nesting reliably
Untrusted long input	A length limit, then a simple pattern	Bounds the worst case

Practical habits

Test with long near-misses, not only with matching text.
Avoid a repeat of something that itself repeats.
Prefer a negated class to .* when you know the closing character.
Use fullmatch for validation and search for finding.
Escape any text that comes from outside with re.escape.
Ask first whether a string method will do.

Challenges

Exercise 1

The pattern ^(\d+)+$ accepts strings made only of digits, but it is dangerous on a long string that ends in a non-digit. Write a safe pattern with the same meaning, and show that it returns quickly (under 0.1 seconds) on 5000 ones followed by x. Also check that the old and new patterns agree on the short inputs "", "7", "77", "7a", "a7" and "12 3". Do not run the old pattern on the long input.

📄 View solution

Exercise 2

Given Hello big and small world, end, first show what .* finds with findall and explain the problem. Then write a pattern using a negated class that returns just the three words inside the tags.

📄 View solution

Exercise 3

Write count_literal(needle, text) that counts how many times a piece of text occurs, using re.findall and re.escape. Test it on the needles 1.5, (a+b) and a+b against Price 1.5, price 1x5, code (a+b) and (a+b) again, 1.5 end, and print each count beside text.count(needle). Then show how many matches the unescaped pattern 1.5 finds, and say which tool you would really use here.

📄 View solution

Quick Reference

Catastrophic backtracking: a repeat inside a repeat over the same characters, such as (a+)+; work roughly doubles per extra character on failure
Greedy overreach: use .*? or, better, a negated class like [^<]*
match vs search: match only tries the start of the string
$ and newline: $ allows a final newline; use fullmatch or \Z
\d: matches other scripts' digits; use [0-9] or re.A
Alternation: first working branch wins; put longer branches first
re.escape: makes outside text literal inside a pattern
Not a pattern job: fixed text (string methods) and nested structure (a parser)

Coming next

Regular Expressions in Practice 10 is the capstone: a small set of reusable text helpers that combine what the whole course has covered.