Checking Your Work with Scanners, Part I (of II): nmap

You may have heard horror stories about how easy it is for evil system crackers to probe potential victims' systems for vulnerabilities using software tools readily available on the Internet. The bad news is, these stories are generally true. The good news is, many of these tools are extremely useful (and even designed) for the legitimate purpose of scanning your own system for weaknesses.

This and next month's column will explore some of the ways ordinary system administrators and high-powered security professionals alike can use nmap and nessus, two outstanding open-source packages, to improve system security. But remember, knowledge is power, and it's up to you to use it responsibly (and in ways that won't compel any men in black to confiscate your beloved Red Hat boxen).

Why scan? If you're a system cracker, you scan to determine what services a system is running and to which well-known vulnerabilities those services appear to be subject. If you're a system administrator you scan for essentially the same reasons but in the interest of fixing (or at least understanding) your systems, not breaking into them.

It may sound odd for good guys to use the same tools as the bad guys they're trying to thwart. After all, we don't test our door locks by trying to kick in our own doors. But system security is exponentially more complicated than physical security. It's nowhere near as easy to gauge the relative security of a networked computer system as it is the door to your house. While ideally we should always know which network processes are active on every host we administer, the fact is that in this age of hyper-connectedness it's difficult to keep track of this information.

Therefore, we security-conscious geeks are obliged to take seriously any tool that can provide some sort of sanity check, even an incomplete and imperfect one (as is anything that tries to measure a moving target like system security), despite or even because of that tool's usefulness to the bad guys. However, we also need to remember that neither a security scanner nor any other single tool or technique will magically grant total security. Repeat after me: security is a process, not a product.

Having said all that, there's one more reason we dig security scanners: notoriety. It's fun to pretend to be a s00p3r 3L33T HaX0r. Using tools like nmap and nessus while saying with a straight face that we're “working” is priceless!

There are basically two types of system scans. Port scans look for open TCP and UDP ports, for “listening services”. Security scans go a step further and probe identified services for known weaknesses. In terms of sophistication, doing a port scan is like counting how many doors and windows a house has; running a security scan is more like rattling all the doorknobs and checking the windows for alarm-sensors.

Oh, I almost forgot ping sweeps, arguably a third kind of scan. These are done to identify which IPs in a given range or network are active (i.e., which hosts respond to pinging). While this can be useful depending on the task at hand, in the interests of keeping this month's column to a manageable level, I'll focus on single-system port and security scanning. Everything we discuss here applies whether you're scanning five hosts or 65,500.

The basic premise of port scanning is simple: if you try to connect to a given port, you can determine whether that port is closed/inactive or whether an application (i.e., web server, FTP dæmon, etc.) is accepting connections there. As it happens, it is easy to write a simple port-scanner that uses the local connect( ) system call to TCP connections on various ports; with the right modules, you can even do this with Perl. However, this method also happens to be the most obtrusive and obvious way to scan, and it tends to result in numerous log entries on one's target systems.

Enter nmap, by Fyodor. nmap can do simple connect( ) scans if you like, but its real thing is “stealth scanning”. Stealth scanning involves the use of ersatz TCP packets designed to trigger a response from each target system without actually completing a TCP connection.

nmap supports not one, but four different kinds of stealth scans in addition to TCP Connect scanning, UDP scanning, RPC scanning, ping sweeps and even operating system fingerprinting. It also boasts a number of features more useful to black-hat than white-hat hackers, such as FTP-bounce scanning, ACK scanning and Window firewall scanning, but those are of little interest to Linux Journal's highly ethical readers. In short, nmap is by far the most feature-rich and versatile port-scanner available today.

Here, then, is a summary of the most important types of scans nmap can do:

TCP Connect Scan—uses the OS's native connect( ) system call to attempt a full three-way TCP handshake (SYN, ACK-SYN, ACK) on each probed port. A failed connection (i.e., if the server replies to your SYN packet with an ACK-RST packet) indicates a closed port. It doesn't require root privileges and is one of the faster scanning methods. Not surprisingly, however, most server applications will log connections that are closed immediately after they're open, so this is a fairly “noisy” scan.

TCP SYN Scan—two-thirds of a TCP Connect scan; if the target returns an ACK-SYN packet, nmap immediately sends an RST packet rather than completing the handshake with an ACK packet. “Half-open” connections such as these are far less likely to be logged, so SYN scanning is much less perceptible than TCP Connect scanning. The trade-off is that since nmap, rather than the kernel, builds these packets, you must be root to run nmap in this mode.

TCP FIN Scan—rather than even pretending to initiate a standard TCP connection, nmap sends a single FIN (final) packet. If the target's TCP/IP stack is RFC-793-compliant (MS-anything, HP-UX, IRIX, MVS and Cisco IOS are not) then open ports will drop the packet and closed ports will send an RST.

TCP NULL Scan—similar to a FIN scan but instead a TCP-flagless packet (i.e., a null packet) is sent. Also relies on the RFC-793-compliant behavior described above.

TCP Xmas Tree Scan—similar to a FIN scan but instead sends a packet with its FIN, PSH and URG flags (final, push data and urgent, respectively) set. It also relies on the RFC-793-compliant behavior described above.

UDP Scan—because UDP is a connectionless protocol, i.e., there's no protocol-defined relationship between packets in either direction, UDP has no handshake to play with as in the TCP scans described above. However, most OS' TCP/IP stacks will return an ICMP “Port Unreachable” packet if a UDP packet is sent to a closed UDP port. Thus, a port that doesn't return an ICMP packet can be assumed open. Since neither the probe-packet nor its potential ICMP packet are guaranteed to arrive (remember, UDP is connectionless and so is ICMP) nmap will typically send several UDP packets per UDP probed port to reduce false positives. In my experience the accuracy of nmap's UDP scanning varies among target OSes, but it's better than nothing.

RPC Scan—used in conjunction with other scan types, this feature will cause nmap to attempt to determine which of the ports identified as open are hosting RPC (remote procedure call) services, and what those services and version numbers are.

Ping Scan (Sweep)—see “Types of Scans and Their Uses” above.

Whew! Quite a list of scanning methods—and I've left out ACK scans and Window scans (see the man page on nmap(1) if you're interested). nmap has another very useful feature, OS fingerprinting. Based on characteristics of a target's responses to various arcane packets that nmap sends, nmap can make an educated guess as to which operating system each target host is running.