Monitoring log events, part I (the swatch way)

Monitoring log events, part I (the swatch way)


I wanted to build something simple to monitor certain log events (in a flat file). This time I actually wanted to 'pipe' each line containing string foo from a certain legacy log file to a database. Here I will just demonstrate passing the whole row (containing string foo) to a shell script though. From there on you can write whatever script suits your needs.
In case you want to stream massive amount of logs to a remote server I would recommend syslog, fluentd or logstash to do the job and maybe enhanced with redis, ELK stack, etc.

Choosing the software

There are two nice software packages available for almost any Linux distro:
  • sec (Simple Event Correlator)
  • swatch (Simple Log Watcher)
In my opinion you need to do more typing with swatch (like daemon script to start with as sec comes with builtin one) and sec has way more options and use cases.
Sec also seems to be more popular but for some strange reason I went ahead with swatch.
UPDATE: I made a post how to implement this with sec. I would recommend using sec if you have a possibility to choose over one another.

Let's get started with swatch!

Install swatch
On CentOS, RHEL, etc:
sudo yum install epel-release && yum install swatch
On Ubuntu, Debian, etc
sudo apt-get install swatch 
Configuration

Lets create a dedicated directory for the configuration & shell script files.
sudo mkdir /etc/swatch
cd /etc/swatch
sudo nano log.conf
This is my configuration file. The file to be monitored will be defined later...
#
# Teemu Otala - 2015/02
# Monitor log for string foo (case insensitive)
#
# word foo anywhere in a log line (case insensitive search (see the i in the end))
watchfor /foo/i
# execute a shell script with and pass the whole line as parameter
        exec "/etc/swatch/log.sh '$_'"
Please notice few very important things here:
  • full path to shell script - relative won't do
  • the full row is stored in $_ (not in $0 or $* as the documentation claims!)
  • the $_ needs to be wrapped in apostrophe ' - if there are any spaces in the variable/row then it would be chopped to $1, $2, $3 and so on on the receiving end (BTW $0 would be the script name)

Next to the shell script
sudo nano log.sh 
This is my shell script - just quick'n'dirty hack to demonstrate how to handle the variable
#!/bin/bash
# Shell script for log.conf
#
message=$(hostname)
message+=" ## "
message+=$1
echo "$message" >> /var/log/test2.log
And make the script executable
chmod +x log.sh
So there it is!

Testing it

This will do it for the testing (temporarily starting the process with --tail-file=[file_to_monitor])
sudo swatch --config-file=/etc/swatch/log.conf --tail-file=/var/log/test.log
Do on a second terminal window
sudo echo "lorem ipsum foo bar me too" >> /var/log/test.log
And at the same time on third terminal window (please test Terminator if you fancy multiple terminals, (sudo yum install epel-release && yum install terminator or sudo apt-get install terminator))
sudo tailf  /var/log/test2.log
Voila! It should be working

Some additional ideas

You could do within the shell script these (the mysql example is not secure)
  • curl --data-urlencode "foo=$1" http://foobar.com
  • mysql -uuser -ppassword -e 'insert into table (column) values ($1)' -h foobar.com
  • of course any executable file/language will do - php-cli, perl, python - not only bash script
  • if your log row has some common delimiter then pass $_ without apostrophe and you get the row ready chopped - you could then send it easy as json with curl! Some good examples here

Daemonizing it

No need to re-invent the wheel - here is a working one (init script):

Copy it to
sudo nano /etc/init.d/swatch
Just update these ones:
  • CHECK_LOG="/var/log/test.log" 
  • SWATCH_CONF="/etc/swatch/log.conf"
And do
sudo chmod a+x /etc/init.d/swatch
sudo service swatch start
Please remember to do chkconfig swatch on in case you want the fun to last over the next reboot :-)


Comments

Post a Comment