Log Parsing Cheat Sheet
Hey all! So, Doug Arcidino shared with some of our team today this cool cheat sheet that Thomas Roccia @fr0gger_ created. Thomas posted on Twitter two infographics (seen below). I converted the images to text so it’s easier to copy/paste from.
| GREP | GREP allows you to search patterns in files. ZGREP for GZIP files. $grep <pattern> file.log | -n: Number of lines that matches -i: Case insensitive -v: Invert matches -E: Extended regex -c: Count number of matches -I: Find filenames that matches the pattern |
| NGREP | NGREP is used for analyzing network packets. $ngrep -I file.pcap | -d: Specify network interface -i: Case insensitive -x: Print in alternate hexdump -t: Print timestamp -I: Read pcap file |
| CUT | The CUT command is used to parse fields from delimited logs. $cut -d -f 2 file.log | -d: Use the field delimiter -f: The field numbers -c: Specifies characters position |
| SED | SED (Stream Editor) is used to replace strings in a file. $sed s/regex/replace/g | s: Search g: Replace d: Delete W: Append to file -e: Execute command -n: Suppress output |
| SORT | SORT is used to sort a file. $sort foo.txt | -o: Output to file -r: Reverse order -n: Numerical sort -k: Sort by column. -c: Check if ordered -u: Sort and remove -f: Ignore case -h: Human sort |
| UNIQ | UNIQ is used to extract uniq occurrences. $uniq foo.txt | -c: Count the number of duplicates -d: Print duplicates -i: Case insensitive |
| DIFF | DIFF is used to display differences in files by comparing line by line. $diff foo.log bar.log | How +o read output? a: Add c: Change d: Delete #: Line numbers <: File 1 <: File 2 |
| AWK | AWK is a programming language use to manipulate data. $awk {print $2} foo.log | Print first column with separator $awk -F: ‘{print $I}’ /etc/passwd Extract uniq value from two files: $awk ‘FNR==NR {a[$0]++; next} |($0 in a)’ fl.txt f2.txt |
| HEAD | HEAD ¡s used to display the first 10 lines of a file by default. $head file.log | -n: Number of lines to display -c: Number of bytes to display |
| TAIL | TAIL is used to display the last 10 lines of a file by default. $tail file.log | -n: Number of lines to display -f: Wait for additional data -F: Same as -f even if file is rotated |
| CSVCUT | LESS is used to visualize the content of a file, faster than MORE. ZLESS for compressed files, $less file.log | space: Display next page /: Search n: Next g: Beginning of the file G: Start of the file +F: Like tail -f |
| JQ | COMM is used to select or reject lines common to two files. $comm foo.log bar.log | Three columns as output: Column 1: lines only in file 1 Column 2: lines only in file 2 Column 3: lines in both files -1, -2, -3 Suppress columns output |
| CSVCUT | CSVCUT is used to parse CSV files. $csvcut -c 3 data.csv | -n: Print columns name -c: Extract the specified column -C: Extract all columns except specified one -x: Delete empty rows |
| JQ | JQ is used to parse JSON files. $jq foojson | jq . f.json: Pretty print jq ‘.[]’ f.json: Output elements from arrays jq ‘.[0]’.<keyname>’ f.json |
| TR | TR is used to replace a character in a file. $tr “;” “,” < foo.txt | -d: Delete character -s: Compress characters to a single one Lower to upper every character: tr “[:lower:]” “[:upper:]” < foo.txt |
| CCZE | CCZE is used to color logs. $ccze < foo.log | -h: Output in html -C: Convert Unix timestamp -l: List available plugins -p: Load specified plugin |
