Secure Logging Over a Network
Undoubtedly, one of the most important tasks for keeping a networked system safe is monitoring its activity. Most system programs today communicate with syslogd noting important events (such as an su request). Also, the kernel can note hardware failures and other things on that level. Of course, when monitoring for possible break-ins or trying to track down the path an intruder used to violate a system, the logs can be a precious resource. However, it is evident that an intruder could manipulate system logs if he were able to get the necessary permissions, thus completely fooling any attempt to track down the intrusion, even to the extreme point of making it hard to spot.
A first step toward solving the problem of keeping the system logs on the system itself was made many years ago when the ability to send logging messages over the network to another host was added to syslogd. Sending the needed log parts to another host, which is quite secure (a dedicated machine, for example), was actually a half-solution to the problem. Using a trivial protocol like UDP to transmit data over networks made it very easy to spoof, making it possible for the intruder to add malicious messages or even try to create a denial-of-service attack. Again, all data is sent through the network in clear form, making it simple to sniff, so an intruder could get more details about the system before attempting a break-in.
A solution we developed to solve this problem is having the standard syslogd facility use the secure shell (ssh) package to forward the logs, using authentication and encryption, to another computer. This way, the system we want to keep logged won't require special changes, and the system that will maintain the logs will not require another running syslogd. It will require sshd (secure shell dæmon) running with some normal system tools on a normal user account.
We will present two technically similar but conceptually different methods, with some pros and cons, of performing this important task. The first could be compared to a “push” technology; that is, the machine that wants to log its data on an outside machine will actually force data to go on the other side. The second method will use a “pull” technology, since the remote client will try to retrieve the logs from the desired machine.
Since our solutions use the standard syslog facilities to grab messages, you need a syslog dæmon that supports logging output to named pipes. Most recent syslog dæmons have this (check the man page for support, or try it directly). It is very important that you have a syslogd that opens pipes in non-blocking mode.
Older syslogs used to open the pipes in blocking mode, so if the named pipe was full (in our context, for example, if the connection had been broken for some time), then the whole syslog process stopped while waiting for the pipe to have some space available. This, of course, could have dramatic consequences, since all the logging, not just the remote one, would have been compromised. As a practical test to see whether your syslogd is handling pipes correctly, configure the whole system, then try writing in the pipe until it reaches the maximum dimension, usually 4KB. When the connection is down, check whether syslog is still logging something or if it seems to be frozen. Anyway, we suggest getting the latest version you can find (which should correct other bugs as well) and installing it on your system (at the moment, the latest version is 1.3-31). The more advanced syslog-ng package should work as well with our solution. Apart from this, you will need a Perl interpreter, the tail command from the textutils package, and of course, the secure shell package.
The theory behind this “push” solution is quite straightforward. A Perl script will run in the background, and will connect (using ssh) to the host where we wish to keep the copy of the logs. The Perl script will then monitor a named pipe we will create, and send anything it will get from the pipe over the secure connection to the other side, where we will put everything in a file. Instructing syslog to send the desired logging events to our named pipe will make the work complete.
Let's give a brief technical description of the source code (see Listing1). Once started, the script will open a local log (defined in $local_log) to report failures (this can be disabled by undefining the variable), then it will try to connect to the remote host (defined in $host). The connection subroutine opens the ssh command to the remote side, specifying the user name to be used, whether to use compression and to establish a quiet connection. As you can see, ssh will ask the remote end to execute a cat of a given file, so everything will be stored there for further use. Once connected, the script will open the local named pipe we created, then forward everything to the other side, where the cat command will store it. Of course, if the secure connection should die for some reason (technically, if the print on the file handle should fail), the script will try to establish the connection again.
As a last feature, the script will send a timestamp each defined time interval to the other end, which could be useful for tracking down when the last message actually came from the host if something serious has happened. In fact, the script can detect that the connection went down each time it sends something over the network. By sending a timestamp to the FIFO (i.e., first-in/first-out pipe), it will automatically trigger the check that the connection is alive at that moment.