Transferring Conserver Logs to Elasticsearch

If your organization manages Linux, AIX, HP-UX or Solaris servers in-house, chances are your system administrators at least occasionally need low-level access to those devices. Typically, administrators use some kind of serial console—for example, traditional serial port, Serial-over-LAN or Intelligent Platform Management Interface (IPMI). Managing and auditing console access is not trivial, so many organizations rely on the Conserver application to create session logs when accessing these servers via the serial console. These logs can be useful for various reasons—for example, maintenance or troubleshooting (to review why something crashed), security (to find out who did what—connecting user names to actual users) or compliance (to provide detailed session logs).

This article covers the following:

  • How to parse and process serial console logs using syslog-ng Open Source Edition (Balabit).

  • How to send the logs to Elasticsearch (Elastic), so you get a complete, searchable audit trail of the console access.

  • How to integrate the console logs into a real-time monitoring system using Riemann.

Conserver

Conserver is a wonderful piece of software that lets you manage your infrastructure's serial consoles, whether they be old-style hardware serial ports or state-of-the-art Serial-over-LAN (SOL) baseboards. Its distributed design permits a decentralized user experience using a secure, TLS-encrypted protocol. The straightforward workflow consists of the user connecting to any conserver master node, which then forwards the traffic to the node that manages the console you want to access. As all masters share the same configuration file, it is very straightforward to redistribute consoles among servers automatically (provided they are virtual SOL devices, like IPMI) using configuration management (for example, using the Puppet module we developed at CC-IN2P3).

So where is the catch? As far as we are concerned at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3), the logging mechanism could be greatly improved, because conserver's design is quite ancient now. For example, it does not support logging to syslog. From a user perspective, logging is awesome, as you can use a keystroke to access the logs of the console, and the console logs contain the complete session. From an architecture perspective though, things are not so great, as every conserver master stores the logs of the consoles it manages locally in a file.

syslog-ng

This is where syslog-ng Open Source Edition comes into play. The idea is to transport the logfiles of the conserver masters to our favorite event store back ends, which are Riemann and Elasticsearch. They provide powerful real-time stream processing and long-term indexed storage capabilities, respectively. In addition, with syslog-ng, you simply can send the logs to Riemann and Elasticsearch directly; there is no need for any additional agents (like Logstash). To see how this system works before going into configuration details, watch this video.

The video shows what the user does in the console (in the top-right section of the screen), its effect on the real-time Riemann-dash dashboard (bottom-right) and the near-real-time Elasticsearch front end (Kibana|Elastic, on the left).

As you can see, the user activities and events of the session are transported to the back ends, including useful metadata like conserver.is_attached: true. This tells you whether or not someone was attached to the console (which is obviously the case in this example).

Requirements

To create the system shown in the demo video, you need the following software:

  • syslog-ng Open Source Edition 3.7.2 or newer.

  • conserver (tested with 8.2.1).

  • Riemann (tested with 0.2.10).

  • Elasticsearch (tested with 1.6.0).

Note that this article does not cover how to install, configure (in general) and get the above software working. You can find plenty of related tutorials on-line. If you need help with these tasks, check the documentation, mailing lists or on-line forums for the software you need help with. The following sections of this article explain how to configure the components of this infrastructure for the specific needs of our scenario as an example.

Configuring Conserver

The conserver configuration in our setup is nothing special. It creates a unified logfile, which will serve as the glue between conserver and syslog-ng. You can activate the unified logfile using either of the following methods:

1) Run the conserver executable with the -U /var/log/console.log flag.

2) Use the following configuration block in conserver.cf:


<config * { unifiedlog * /var/log/console.log; }

You also can set the server's general logfile (where conserver stores the global messages that are unrelated to individual consoles)—for example, to /var/log/conserver.log.

Both /var/log/conserver.log and /var/log/console.log will be inputs for syslog-ng. You might want to take special care of the log rotation of these files. As you are sending them to Elasticsearch, there is no need to keep them for too long.

______________________

Fabien Wernli (faxm0dem) has been administering Linux clusters at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3) for 10+ years.