Substituting and Splitting

So far patterns have only found text. This chapter uses them to change it. re.sub replaces every match with something else, and re.split cuts a string wherever a pattern matches. Both are far more flexible than the plain string methods str.replace and str.split.

re.sub

The call is re.sub(pattern, replacement, string, count=0, flags=0). It returns a new string; the original is never changed. A count of 0 means "replace all".

re.subn works the same way but returns a tuple of the new string and the number of replacements, which is useful when you need to know whether anything changed.

Backreferences in the replacement

The replacement string can reuse what the pattern captured. Write \1, \2 for numbered groups, or \g<name> for named ones. This is how you reorder text.

The last line shows why \g<1> exists: \10 would be read as group ten, not group one followed by a zero. Use the \g<...> form whenever a digit follows the reference.

Backreferences inside the pattern

A group can also be referred to within the pattern itself, to demand a repeat of whatever it matched.

Here (\w)\1 matches a doubled letter. In the pattern, \1 means "the same character group 1 just matched"; in the replacement, it means "insert group 1".

A function as the replacement

When plain reordering is not enough, pass a function. re.sub calls it once per match with the match object and substitutes whatever string it returns. The replacement function can do arithmetic, look things up, or format.

The function must return a string, which is why the first example wraps the number in str(). Inside the function, m.group(), m[1] and m['n'] all work as in earlier chapters.

Escapes in the replacement string

The replacement string is itself interpreted: a backslash starts an escape. A literal backslash in the output must be doubled. When literal text containing special characters is used as the pattern, protect it with re.escape.

A function replacement avoids the escape rules altogether, because its return value is used as-is. If a substitution inserts user-supplied text, return it from a function.

Empty matches

A pattern that can match nothing replaces at every position between characters, because an empty match exists everywhere.

re.split

re.split(pattern, string, maxsplit=0, flags=0) returns a list of the pieces between matches. Unlike str.split, the separator can be a pattern.

Two things to notice. Consecutive separators produce an empty string between them ('a,b,,c'). And a separator at the very end leaves an empty string after it, as in the last line; filter these out with [p for p in parts if p] when you do not want them.

Keeping the separators

If the split pattern contains a capturing group, the text it matched is included in the result between the pieces.

Splitting at a position

Lookaround (covered in the next chapter) can split without consuming anything. As a preview, this cuts after sentence-ending punctuation but keeps it attached. The second example splits on blank lines.

Challenges