Parsing Logs

Log files are where patterns earn their keep. Each line is a small record written by a program, so the lines share a layout, but the layout is plain text and there is no ready-made parser. In this chapter you will turn raw lines into dictionaries, cope with lines that do not fit, and finish with a short report. It pulls together named groups, re.VERBOSE, fullmatch and the iteration tools from the earlier chapters.

Read the line first

Start by looking at the structure, not by writing a pattern. A line in our example log looks like this:

There are five parts: a date, a time, a level in capitals, a source in square brackets, and then a free-text message that may hold key=value pairs. The first four parts are rigid; the message is loose. That suggests two steps: one pattern for the fixed frame, and a second, simpler one for the pairs inside the message.

The frame pattern

Named groups make the result self-describing, and re.VERBOSE lets each part carry a label. Note that in verbose mode the literal spaces are written as \s, because plain spaces in the pattern are ignored.

groupdict() returns exactly the dictionary we wanted. The last group uses .* deliberately: it is the one place where "anything" is the right answer, and because it is at the end and the match is anchored by fullmatch, it cannot swallow the earlier fields.

The key=value pattern

For the message, findall with two groups returns a list of pairs, and dict() turns that list straight into a dictionary:

A message with no pairs simply gives an empty list, and so an empty dictionary. No special case is needed. This is the benefit of using findall rather than search here: "nothing found" is a normal, harmless result.

Never assume every line is good

Real logs contain blank lines, truncated lines and lines from some other program. A parser that raises an error on the first odd line is useless on a big file. The habit is to return None for a line that does not match, and let the caller decide what to do. Here is the complete parser, with the bad lines collected rather than hidden:

Two details are worth noticing. splitlines() removes the line endings, so the pattern never has to think about \n. And rec.pop("msg") removes the raw message as it is used, so each record holds the structured fields only.

From records to a report

Once each line is a dictionary, the regular expressions are finished. Everything else is ordinary Python, and collections.Counter does most of the counting:

Values from kv_re are always strings, so numbers such as ms are converted with int() at the point of use. Converting too early, before you know a field is present and numeric, is a common source of errors.

Practical habits

• Look at real lines first and note which parts are rigid and which are loose.
• Use two simple patterns (a frame and the details) rather than one that does everything.
• Use named groups so the code reads like the log format.
• Keep the bad lines and report how many there were. Silent skipping hides format changes.
• Parse once into dictionaries, then do all counting and filtering on the structured data.
• Large files: loop over the open file object line by line rather than reading everything into one string.

Challenges