Log parsing – grep, cat, zgrep and zcat

More on log parsing, I’m taking notes on how to read log files and get the information that I need. On Linux environment, these tools are perfect: grep, cat, zgrep and zcat.

Extracting patterns with grep

We can extract information from a text file using grep. Example, we can extract lines of log file containing patterns like GET /checkout/* where status code is 500.

grep -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.log

Depending on the Apache log format, above will extract lines whose request is /checkout/* and status code is 500 where it may support HTTP/1.0 or HTTP/1.1. However, that would extract the whole line. To only extract the matching pattern, use the -o option.

grep -o -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.log

And to save the matching patterns to a file, simply redirect the output to file.

grep -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.log > checkout-errors.txt

Using cat

cat is usually used to output contents of a file. This is a small but very useful Linux utility. For example, we can combine multiple log files (uncompressed) into a single log file.

cat /path/to/log-files/*.log > /combined/log-file.log

Compressed counterpart

grep and cat have their compressed file counterpart. For grep, there’s zgrep.

zgrep -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.gz > checkout-errors.txt

For cat, there’s zcat.

zcat /path/to/log-files/*.gz > /combined/log-file.log

I’ve done so many combination last week that I don’t remember them all and not able to include in this post. Happy log parsing.

Leave a reply

Your email address will not be published. Required fields are marked *