More on log parsing, I’m taking notes on how to read log files and get the information that I need. On Linux environment, these tools are perfect: grep, cat, zgrep and zcat.
Extracting patterns with grep
We can extract information from a text file using grep. Example, we can extract lines of log file containing patterns like GET /checkout/* where status code is 500.
grep -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.log
Depending on the Apache log format, above will extract lines whose request is /checkout/* and status code is 500 where it may support HTTP/1.0 or HTTP/1.1. However, that would extract the whole line. To only extract the matching pattern, use the -o option.
grep -o -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.log
And to save the matching patterns to a file, simply redirect the output to file.
grep -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.log > checkout-errors.txt
Using cat
cat is usually used to output contents of a file. This is a small but very useful Linux utility. For example, we can combine multiple log files (uncompressed) into a single log file.
cat /path/to/log-files/*.log > /combined/log-file.log
Compressed counterpart
grep and cat have their compressed file counterpart. For grep, there’s zgrep.
zgrep -E -e 'GET /checkout/.* HTTP/1\.(0|1)" 500' some-log-file.gz > checkout-errors.txt
For cat, there’s zcat.
zcat /path/to/log-files/*.gz > /combined/log-file.log
I’ve done so many combination last week that I don’t remember them all and not able to include in this post. Happy log parsing.