Alternation and Flags

Regular Expressions in Practice

Chapter 3 · Alternation and Flags

Two tools finish the basics. Alternation lets one pattern say "this or that". Flags change how the whole engine reads a pattern: ignoring case, letting ^ and $ work on every line, letting the dot cross line breaks, and allowing you to lay a long pattern out with spaces and comments.

What was run

As in the earlier chapters, every snippet was executed and the comments show what Python printed.

Alternation with the Pipe

The pipe | means "match the left side, or else the right side". It has the lowest precedence of any operator, so it splits the whole pattern unless you fence it in with a group.

import re

s = "A cat, a dog and a catalog"
print(re.findall(r"cat|dog|bird", s))            # ['cat', 'dog', 'cat']
print(re.findall(r"\b(?:cat|dog|bird)\b", s))    # ['cat', 'dog']
    

The first pattern also found the cat inside catalog. Wrapping the choice in (?:...) and adding \b word boundaries on both sides restricts it to whole words.

The pipe splits the whole pattern

Without parentheses, each side of the pipe is a complete alternative, including any anchors or text beside it.

print(re.search(r"^a|bc", "xbc").group())      # bc   (means: starts with a, OR contains bc)
print(re.search(r"^(?:a|b)c", "bc") is not None)  # True (starts with a or b, then c)
print(re.findall(r"gr(?:a|e)y", "gray grey graey"))   # ['gray', 'grey']
print(re.findall(r"gr[ae]y|colou?r", "gray color colour"))
# ['gray', 'color', 'colour']
    

When the choice is a single character, a class such as [ae] is shorter and faster than (?:a|e). Use the pipe when the alternatives are longer than one character.

Order matters

The engine tries alternatives left to right and takes the first one that lets the whole pattern succeed. It does not look for the longest. Put longer alternatives first when one is a prefix of another.

print(re.search(r"cat|catalog", "catalog").group())   # cat
print(re.search(r"catalog|cat", "catalog").group())   # catalog
    

Flags

A flag is passed as a third argument (re.findall(pattern, text, re.I)) or, later, to re.compile. Combine several with the bitwise-or operator |. Each has a long name and a one-letter alias.

Long name	Short	Effect
`re.IGNORECASE`	`re.I`	letters match in either case
`re.MULTILINE`	`re.M`	`^` and `$` match at every line, not just the string ends
`re.DOTALL`	`re.S`	`.` also matches the newline character
`re.VERBOSE`	`re.X`	whitespace and `#` comments in the pattern are ignored

IGNORECASE

print(re.findall(r"hello", "Hello HELLO hello"))                 # ['hello']
print(re.findall(r"hello", "Hello HELLO hello", re.IGNORECASE))  # ['Hello', 'HELLO', 'hello']
print(re.findall(r"(?i)hello", "Hello HELLO hello"))            # ['Hello', 'HELLO', 'hello']
    

The last line shows an inline flag: (?i) at the start of the pattern switches the flag on without an extra argument. The letters are i, m, s and x. Inline flags are handy when the pattern is stored somewhere that cannot carry a separate flags value.

MULTILINE

By default ^ matches only at the very start of the string and $ only at the very end (or just before a final newline). With re.M they match at the start and end of each line.

t = "one\ntwo\nthree"
print(re.findall(r"^\w+$", t))                # []
print(re.findall(r"^\w+$", t, re.MULTILINE))  # ['one', 'two', 'three']
print(re.findall(r"^t\w+", t, re.M))          # ['two', 'three']
    

The first call finds nothing because ^\w+$ would need the whole string to be one word. Note that \w never matches a newline, so a match can never leak from one line into the next here.

DOTALL

The dot normally stops at a newline. re.S lets it cross, which matters when a single item spans several lines.

print(re.search(r"one.two", t))                       # None
print(repr(re.search(r"one.two", t, re.S).group()))     # 'one\ntwo'
print(re.findall(r".+", t))                          # ['one', 'two', 'three']
print(re.findall(r".+", t, re.S))                    # ['one\ntwo\nthree']

src = "<b>x\ny</b> <b>z</b>"
print(re.findall(r"<b>(.*?)</b>", src))             # ['z']
print(re.findall(r"<b>(.*?)</b>", src, re.S))       # ['x\ny', 'z']
    

Be careful not to mix up the two line flags. MULTILINE changes the anchors; DOTALL changes the dot. They are independent and are often used together.

Combining flags

log = "ERROR disk\nwarn cpu\nInfo ok\nerror net"

print(re.findall(r"^(?:error|warn)\b.*$", log))
# []
print(re.findall(r"^(?:error|warn)\b.*$", log, re.I | re.M))
# ['ERROR disk', 'warn cpu', 'error net']
    

VERBOSE

A long pattern on one line is hard to read. re.X tells the engine to ignore whitespace and everything from a # to the end of the line, so you can lay a pattern out in parts and explain each part. Use a triple-quoted raw string.

pat = re.compile(r"""
    (?P<area>\d{3})   # three-digit area code
    [-.\s]?           # optional separator
    (?P<num>\d{4})    # four-digit number
""", re.VERBOSE)

print(pat.search("call 555-0199").groupdict())   # {'area': '555', 'num': '0199'}
    

The price of verbose mode is that spaces and # in the pattern are now ignored, so to match them literally you must escape them (\ , \#) or put them in a class ([ ], [#]).

print(re.search(r"a b", "a b").group())             # a b
print(re.search(r"a b", "a b", re.X))              # None  (the space was ignored, so it wants "ab")
print(re.search(r"a[ ]b", "a b", re.X).group())     # a b
print(re.search(r"a#b", "a#b", re.X).group())      # a   ("#b" was treated as a comment)
    

Choosing flags

Reach for re.I when the data's case is unreliable, re.M when you process text line by line as one string, re.S when one item crosses lines, and re.X for any pattern longer than a single short line.

Challenges

Exercise 1

Given s = "The Cat sat; a DOG barked; cats and dogs; concat", find every whole word that is cat or dog, with an optional plural s, ignoring case. Print the list and its length. Then run the naive cat|dog with re.I and explain the difference.

📄 View solution

Exercise 2

Given a four-line log string (each line: a date, a level in mixed case, a message), use finditer with named groups date, level and msg, and the flags needed, to print a groupdict() for only the error and warn lines. Then show what findall returns for the same pattern when you leave out the flags, and say why.

📄 View solution

Exercise 3

Write a verbose pattern with comments that matches clock times such as 9:05, 14:30:59 and 9:05 PM: named groups h, m, optional s (non-capturing colon group) and optional ap, with leading and trailing space allowed and case ignored. Test it on "9:05", "14:30:59", "9:05 PM", "9:5" and "25:61x", and say why the last one does not match.

📄 View solution

Quick Reference

Alternation: a|b has the lowest precedence; fence it with (?: ... )
Order: alternatives are tried left to right; put longer ones first
Whole words: \b(?:cat|dog)\b
re.I: ignore case; inline (?i)
re.M: ^ and $ work per line
re.S: . also matches a newline
re.X: ignore pattern whitespace and # comments; escape or bracket literal spaces
Combine: re.I | re.M

Coming next

Regular Expressions in Practice 4 covers substituting and splitting: re.sub with backreferences and functions, and re.split.