Log Parser

by mike on February 28, 2011

The idea behind a log parser is to create a loop that will check all of your required logs to return information about specific needs that you may have as an administrator.    For example, with this script you may want to search a specific log for one or two text strings that are very important to the function of your system.

#!/bin/bash

script=$(basename $0)_errors
log1=/var/log/messages
log2=/var/log/secure
log3=/var/log/dmesg


mydate=$(date +%b\ %d)

for log in $log{1,2,3}
do
if [ -e $log ] && [ -s $log ]
then
echo
echo BEGIN $log
grep -E "$mydate" $log | grep -E 'Device|fail'  2> $script
echo END $log
echo
fi
done

The script variable allows you to set up an error log that you can refer back to if you needed to.
script=$(basename $0)_errors

The current way this is set up it creates the error file in the same directory that the script is in.  If you wanted to alter that function you could create the path.

script=/var/log/$(basename $0)_errors

The advantage of providing your scripts with numbers is that you can list as many logs as you want to employ and then loop through them with a for loop and use braces to list all of the logs.

log1=/var/log/messages
log2=/var/log/secure
log3=/var/log/dmesg

The mydate variable allows for the checking of your logs for the current day.  You can change this to whatever you want but since there are other options for logs, and since you probably are looking to solve these types of problems as quick as possible, you will be wanting the current day.

mydate=$(date +%b\ %d)

The formatting is provided by this structure and it just helps you visualize the errors easier by providing some separation.

echo
echo BEGIN $log

echo END $log
echo

You end up with nicer output.  The sections “BEGIN” and “END” help as well.

BEGIN /var/log/messages
Aug 31 01:21:43 mail kernel: pnp: Device 00:05 does not support disabling.
Aug 31 01:21:53 mail smartd[2046]: Device: /dev/hda, opened
Aug 31 01:21:53 mail smartd[2046]: Device: /dev/hda, not found in smartd database.
Aug 31 01:21:53 mail smartd[2046]: Device: /dev/hda, lacks SMART capability
Aug 31 01:21:53 mail smartd[2046]: Device: /dev/hda, to proceed anyway, use '-T permissive' Directive.
Aug 31 01:21:53 mail smartd[2046]: Device: /dev/hdc, opened
Aug 31 01:21:53 mail smartd[2046]: Device: /dev/hdc, packet devices [this device CD/DVD] not SMART capable
END /var/log/messages

BEGIN /var/log/secure
Aug 31 01:21:48 mail sshd[1872]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
END /var/log/secure

BEGIN /var/log/dmesg
END /var/log/dmesg

The heart of the script is the use of grep to look for not only today’s logs but also to use filters that will search for specific problems.  Here you can see the filters “Device” and “fail” are separated by the pipe symbol.  You could add as many filters as you wanted here.

grep -E "$mydate" $log | grep -E 'Device|fail'  2> $script

Another point to look at is that because of the use of a variable and the filters you must use the “-E” for the ability to the extended regular expressions.

There are also several tests that are performed in order to help in management.  First the “-e” is testing to verify that the log actually exists AND that the log is not an empty file “-s”.  This will cut down on errors.

if [ -e $log ] && [ -s $log ]

{ 2 comments }

Pétur Ingi Egilsson March 1, 2011 at 7:00 am

Informative, thank you.

John McKown March 1, 2011 at 2:07 pm

I do not understand the statement:

… Another point to look at is that because of the use of a variable … you must use the “-E” for the ability to the extended regular expressions.

I don’t understand why the -E is needed in
grep -E “$mydate” $log

the contents of $mydate are a 3 letter month, space, and 2 digit day. That is not a regular expression. The $mydate is expanded by the shell and not grep (as I’m sure most already know), so I don’t know why -E is needed. I’d use -F instead. Why invoke the regex engine for no purpose? Granted, not a big deal, why why waste CPU power. Especially if this could be in a virtualized environment where the physical machine is shared with multiple virtual machines.

Comments on this entry are closed.

Previous post:

Next post: