Detecting Suspect Traffic

With the release of the much-awaited Linux 2.4.0 kernel early this year, GNU/Linux received a major boost toward entering the realm of enterprise-class operating systems. Several areas of the kernel have been improved since the 2.2.x series, not the least of which is the firewall code. Netfilter in the 2.4.x kernel series replaces the old ipchains firewall code in the 2.2.x series and incorporates many features that are required in an enterprise-class firewall, such as statefullness, DoS protection/rate limiting, network address translation (NAT), MAC address filtering and, last but not least, TCP packet filtering and logging based on arbitrary combinations of TCP flags. By contrast, ipchains (in addition to not being stateful, etc.) have the lamentable limitation of being able to distinguish between only two types of TCP packets: those that set and those that do not set the SYN flag. Netfilter's ability to distinguish arbitrary TCP flag combinations allows advanced port scans, such as those that are easily leveraged against a machine via nmap, to be detected. First we need a bit of background on nmap so that we can illustrate both how advanced port scans work and how to detect them.

nmap is the world's best-known port scanner and incorporates many advanced port scanning techniques that can be used to detect open ports on a target machine and/or network. In addition, nmap incorporates operating system fingerprinting and TCP sequence number prediction (among other things) to give better reconnaissance information than can be gleaned by determining which ports are open on a target machine. Three particularly interesting types of TCP scanning modes that nmap supports are FIN, NULL and XMAS scans. In normal TCP traffic, a FIN packet (or packet that has the FIN flag set) can be sent by either end of a TCP connection indicating that the session is finished or, more precisely, that there is no more data to be sent. A FIN scan works on the principle that an orphaned FIN packet (i.e., one that is not already part of an existing TCP session) sent to an open TCP port should not be acknowledged. If such a packet is sent to a closed port, however, a normal unfiltered TCP stack is supposed to respond with an RST packet. Hence, to scan a host with FIN packets, nmap simply sends a single FIN packet to each of the target ports and waits for RST packets to trickle back. Ports that do not respond with an RST are open (or filtered by a firewall). Both the NULL and XMAS scans function similarly, but instead of setting only the FIN flag, the XMAS scan also sets the URG and PSH flags, whereas the NULL scan utilizes TCP packets that have no flag set whatsoever.

The basic principle in setting up a secure firewall is to have a default deny stance in which all traffic not explicitly allowed by the firewall is denied or rejected. In addition, the firewall should be configured to log any unauthorized packets to a log file for analysis by the system administrator or by automated logfile watchers such as psad.

For more information on getting iptables to run on your system, I highly recommend Rusty Russell's “Netfilter HOWTO”, available at netfilter.samba.org.

# iptables -L
Chain INPUT (policy ACCEPT)
target  prot source   destination
ACCEPT  all  anywhere anywhere state RELATED,ESTABLISHED
ACCEPT  TCP  anywhere anywhere TCP dpt:ssh flags:FIN, SYN,RST,PSH,ACK,URG/SYN
ACCEPT  TCP  anywhere anywhere TCP dpt:www flags:FIN, SYN,RST,PSH,ACK,URG/SYN
LOG     TCP  anywhere anywhere LOG level warning prefix `DENY '
DROP    TCP  anywhere anywhere

Does such an iptables policy stops port scans? First, any TCP packet (regardless of TCP flags) sent to a destination port other than 80 or 22 will be logged by rule number four and then dropped by rule number five. This takes care of any noisy scans of any type against any of the 65535 TCP ports other than 80 and 20. What about scans that test whether 80 and 22 are open? The answer is that it depends upon the type of scan. Rules three and four accept packets that have the SYN flag set and all other flags cleared, so any scan that makes use of just SYN packets will succeed. Also, any scans that make use of normal TCP connect() system calls, as a normal web browser or SSH client would use, will succeed since rule number one allows three-way TCP handshakes to complete.

nmap supports both of these scanning techniques with the -sS (half-open or SYN scan) and -sT (TCP connect() scan) command-line switches. Any other scanning technique that is not based on legitimate TCP traffic to these ports will be blocked and logged. Such scans include the FIN, XMAS and NULL scans mentioned in the nmap section, as well as SYN/FIN and ACK scans.

Now that we are reasonably confident that our iptables firewall will block port scans, we should not become complacent with our success. It is not enough just to block scans; we should also do our level best to detect them, as a port scan could be a precursor to a more advanced attack.