Wouldn’t it be nice if you had a web interface to the logs on your central log server? Well, dream no more because this one is a reality thanks to LogAnalyzer (aka phplogcon). Let's take a look at how to setup both it and its suggested syslog variant, rsyslog, on a central log server. We will approach this from the perspectives of both Gentoo where rsyslog is not the default (nor is anything else) and Ubuntu 10.04 LTS which is already using rsyslog as the defacto logger. It is also worth mentioning that, according to the rsyslog site, Fedora also now defaults to rsyslog.

The Prep:

Before we dive into setting up any of this we need to do a little prep work. If you are going to be looking at these logs in a web browser then it might be good if the time stamps you are seeing reflect the appropriate timezone. For this I shall borrow from the Gentoo Handbook as it is fitting for both distros:

7.a. Timezone
You first need to select your timezone so that your system knows where it is located. Look for your timezone in /usr/share/zoneinfo, then copy it to /etc/localtime. Please avoid the /usr/share/zoneinfo/Etc/GMT* timezones as their names do not indicate the expected zones. For instance, GMT-8 is in fact GMT+8.

Suppose you are in the eastern timezone:

~# cp /usr/share/zoneinfo/EST5EDT /etc/localtime

Time:

If you are going to be compiling all of your logs in one place then it also becomes rather important that everyone involved know the correct time. To achieve this you can use an NTP client.

Ubuntu 10.04 LTS

Using your favorite text editor, create the file /etc/cron.daily/ntpdate and insert the following:

ntpdate ntp.ubuntu.com

Save the file then

~# chmod 755 /etc/cron.daily/ntpdate

Now let's run it to get our time corrected:

~# /etc/cron.daily/ntpdate

Lastly, verify...

~# date
You should see your current time here.

Gentoo

In Gentoo this is taken care of by net-misc/ntp.

~ # emerge -a ntp
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild  N    ] net-misc/ntp-4.2.4_p7-r1 
      USE="ssl -caps -debug -ipv6 -openntpd 
           -parse-clocks (-selinux) -vim-syntax -zeroconf"

Would you like to merge these packages? [Yes/No] y

~ # cp /usr/share/ntp/ntp.conf /etc/ntp.conf 
~ # rc-update add ntp-client default
 * ntp-client added to runlevel default
~ # rc-update add ntpd default
 * ntpd added to runlevel default
~ # /etc/init.d/ntp-client start
 * Caching service dependencies ...                [ ok ]
 * Setting clock via the NTP client 'ntpdate' ...  [ ok ]
~ # /etc/init.d/ntpd start
 * Starting ntpd ...                               [ ok ]
~ # date
You should see your current time here.

The Logging:

Now that we have our clocks synchronized let's move on to the meat and potatoes of this article. There are some prerequisite software packages that need to be setup here: Apache, MySQL, PHP... aka the LAMP stack. I am not going to go into detail on these but I do to note a couple of things:

• On Gentoo, I suggest adding the following to /etc/portage/package.keywords. If you are not on a 64-bit system then replace ~amd64 with ~x86.
  • dev-db/mysql ~amd64
  • virtual/mysql ~amd64
  • dev-lang/php ~amd64
• I also suggest the mysql and php use flags be set in /etc/make.conf
• On Ubuntu, just run tasksel and pick LAMP from there.

  

• After installing the LAMP stack, don’t forget to manually restart Apache.
• On both distros I would suggest also setting up phpMyAdmin to simplify managing MySQL.

Rsyslog on Gentoo

Gentoo may not have been setup with rsyslog initially so let's get that running. Before you emerge it though you need to add relp as a use flag.

~ # emerge -a rsyslog
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild  N    ] dev-libs/librelp-0.1.1  USE="-debug" 
[ebuild  N    ] app-admin/rsyslog-3.22.0 
      USE="mysql relp zlib -dbi -debug -gnutls -kerberos -postgres -snmp"

Would you like to merge these packages? [Yes/No] y

After installing rsyslog run its script to create the database and tables in MySQL:
~ # mysql 

Verify it with these commands:
~ # mysql -p -e "show databases;"

You should see a database named Syslog.
~ # mysql -p -e "show tables;" Syslog

You should see two tables in this one.

Now, create a user that rsyslog can use to manipulate the Syslog database:
~ # mysql -p -e "grant all on Syslog.* to rsyslog@'localhost' identified by 'password';"

We now need to make a couple of tweeks to /etc/rsyslog.conf.  Add this just below the comments at the top:
# Logging to MySQL stuff
$ModLoad ommysql
*.*       :ommysql:localhost,Syslog,rsyslog,password

Near the bottom, change what is there to look like this:
# ######### Receiving Messages from Remote Hosts ##########
# TCP Syslog Server:
# provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imtcp.so  # load module
$InputTCPServerRun 514 # start a TCP syslog server at standard port 514

# UDP Syslog Server:
$ModLoad imudp.so  # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514

Now, let's fire up our new logger and stop the old one.
~ # rc-update del syslog-ng 	// or whatever logger you used.
 * 'syslog-ng' removed from the following runlevels: default
~ # rc-update add rsyslog default
 * rsyslog added to runlevel default
~ # /etc/init.d/rsyslog start 
 * Starting rsyslogd ...                                             [ ok ]

It is a little rough to read if your terminal is not wide enough, but you can verify that the logs are going to the database with
~# mysql -p -e "SELECT * FROM Syslog.SystemEvents;"

Now that rsyslog is running we can get rid of the old logger... 
~ # emerge -aC syslog-ng

Rsyslog on Ubuntu 10.04 LTS
Ubuntu now comes with rsyslog as its defacto logger but we need to add a little bit of additional functionality to it.  Namely, we need to add MySQL output support and add in the Reliable Event Logging Protocol (relp):
~# apt-get install rsyslog-mysql rsyslog-relp
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  dbconfig-common librelp0
The following NEW packages will be installed:
  dbconfig-common librelp0 rsyslog-mysql rsyslog-relp
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 677kB of archives.
After this operation, 2,335kB of additional disk space will be used.
Do you want to continue [Y/n]? y

During this install’s process you will be prompted to make the tables that are needed in MySQL:


Do this.  You will then be asked for your MySQL root password followed by being asked to create a password for rsyslog to use.  This is the password that rsyslog will use in its config files.
Now, just like on Gentoo, we need to make a couple of tweeks to the config files of rsyslog.  The biggest difference here is that Ubuntu takes advantage of the fact that rsyslog can use multiple config files that are merged into one “config.”  You still have /etc/rsyslog.conf but you also have a directory named /etc/rsyslog.d/ that contains additional configs.  In there you will now see one named mysql.conf that contains the needed info to dump our logs into the database.  To turn on accepting remote logs though we still have to uncoment a couple of lines in /etc/rsyslog.conf
# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Now, let's apply our changes:
~# service rsyslog restart

As I mentioned in the Gentoo section, it is a little rough to read if your terminal is not wide enough, but you can verify that the logs are going to the database with
~# mysql -p -e "SELECT * FROM Syslog.SystemEvents;"

RELP: Reliable Event Logging Protocol
t this point we have configured our servers to accept messages over TCP & UDP on the standard port of 514, which is what is needed to be compatible with the majority of network devices that can shoot logs to a central server.  Now, let's take it a step farther and add support to the config files for a third protocol: RELP, which is implemented via librelp.  This is supposed to be a much more reliable method of sending syslog data over the network than even TCP.  More info about the unreliability of tcp and the creation of RELP can be found on the developer’s blog.  Rsyslog has native support for it via the imrelp & omrelp modules.
Now, on Gentoo edit /etc/rsyslog.conf again and add the following to the bottom of the file:
# RELP Syslog Server:
$ModLoad imrelp.so # provide RELP syslog reception
$InputRELPServerRun 20514 # start a RELP syslog server at port 20514

And on Ubuntu, use your favorite editor and create /etc/rsyslog.d/relp.conf and enter the following in it:
$ModLoad imrelp
$InputRELPServerRun 20514

Adding reliability to your logging systems
One of the many helpful articles at rsyslog.com explains how to log massive amounts of syslog data to a database by setting up buffering on your server so that messages are less likely to be lost if a system or link is under very high load.  To do this you first need to make a working directory that it can store queue files in if needed.
~ # mkdir -p /var/rsyslog/work
Now we need to add the following to /etc/rsyslog.conf or /etc/rsyslog.d/mysql.conf
# Buffering stuff:
$WorkDirectory /var/rsyslog/work # default location for work (spool) files
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName dbq    # set file name, also enables disk mode
$ActionResumeRetryCount -1  # infinite retries on insert failure

And now we need to restart rsyslog:
Gentoo:
~ # /etc/init.d/rsyslog restart

Ubuntu:
~ # service rsyslog restart

The Viewing:
To view the info that we are now dumping into MySQL via the web we need to setup LogAnalyzer.  Step one of this is to download the software from http://loganalyzer.adiscon.com/downloads.  As of this writing, the newest version is v3.0.1.
~ # wget http://download.adiscon.com/loganalyzer/loganalyzer-3.0.1.tar.gz
~ # tar -xzf loganalyzer-*.*.*.tar.gz
~ # cd loganalyzer-3.0.1
~/loganalyzer-3.0.1# mkdir /var/www/logs
~/loganalyzer-3.0.1# cp -R src/* /var/www/logs/
~/loganalyzer-3.0.1# cp contrib/* /var/www/logs/
~/loganalyzer-3.0.1# cd /var/www/logs/
/var/www/logs# chmod +x configure.sh secure.sh
/var/www/logs# ./configure.sh

To enable the authentication part of LogAnalyzer we need to make an empty database for users to be stored in and grant privileges on it. 
/var/www/logs# mysql -p
mysql> create database LogAnalyzerUsers;
mysql> show databases;
mysql> grant all on LogAnalyzerUsers.* to LAUser@'localhost' identified by 'password';
mysql> quit

*** Parts of the following section are direct quotes from http://loganalyzer.adiscon.com/doc/install.html:

Now, go to http://yourlogserver/logs/ and you will be pointed to the installation script which will guide you through the process of setting up LogAnalyzer.
Basic Configuration
You can set several basic options here.

• Number of syslog messages per page = 50 (default)
• This is the number of syslog messages displayed on each page. You can increase the value (makes LogAnalyzer slower) or decrease the value (makes it faster).
• Message character limit for the main view = 80 (default)
• Set the number of characters per message which will be shown in the last column of the main view. Full messages can be reviewed by hovering the mouse over it.
  • Many folks prefer to use a setting of "0", which means complete messages will be displayed
• Show message details popup (default yes) = yes (default). Note that many people find the popups intrusive and prefer to disable them. Use "no" in this case.
• During the setup you will also be prompted to enable the user database. Do so and enter in the information that is requested.
• A couple of pages later you will be prompted for the main (admin) user.
• The defaults on Step 7 demonstrate that it is possible to use this without the database backend. We need to change this to match our setup though.
• Name the source something logical seeing as it is going to be the compiled logs from all your servers.
• Source Type = MYSQL Native
• Select View = Syslog Fields
• Table type = MonitorWare
• Database Host = localhost
• Database Name = Syslog
• Database Tablename = SystemEvents
• Database User = rsyslog
• Enable Row Counting = no

Once you finish up, log into your new site and have a look at what has been being logged on your server so far.

Your other boxes

Linux and Solaris

Setting up your other boxes is fairly straight forward. For this example, assume the server you have been setting up is named myserv. If you don’t want to edit your hosts file then either use the FQDN or the IP in place of myserv. If you are willing to run rsyslog on them then all you have to do is install librelp and then add the following to your rsysnc configuration file:

$ModLoad omrelp
# forward messages to the remote server "myserv" on
# port 20514
*.* :omrelp:myserv:20514;RSYSLOG_ForwardFormat

Details of this can be found on the omrelp page. If you are running a traditional version of syslog then you will be sending your messages via UDP. To do so add the following to your syslog.conf file:

*.* @myserv

This works on most Linux hosts along with Solaris. Just remember to restart your logger so that it reads the changes to the config file.

Network Devices

If you want to point your network devices to this server then just remember that it is listening on port 514 for both TCP and UDP connections.

OS X

OS X is already running syslog so it is basically the same as configuring Linux... just add the following to /etc/syslog.conf:

*.* @myserv

Now restart syslog with

launchctl stop com.apple.syslogd 
launchctl start com.apple.syslogd

Windows

If you want your Windows 2000 - Server 2008 machines to send logs to your new central log server then check out eventlog-to-syslog. Just download it, unzip it, copy evtsys.exe to C:\Windows\system32\ and execute

C:\Windows\system32>evtsys.exe -i -h ip.address.of.myserv

I tested the x64 version of this in a Windows 7 Enterprize virtual machine and it worked great.

Even more sources

You can also send logs from Apache, IIS, ISA, Squid Proxy Servers, Lotus Notes, Exchange, and others to your central log server. Take a look at the Snare Agents from Intersect Alliance. They are available from both their site and SourceForge.

Happy logging :)