SQL Comes to Nmap: Power and Convenience

I recently was exchanging e-mail with someone who regularly needs to port scan his own network for vulnerability trending. The port scanning tool of choice for this task is Nmap, but managing the data from Nmap was an entirely different beast. A few weeks later, a patch to Nmap that allows it to log the results directly to MySQL was ready. Although Nmap supports machine-parseable format as well as XML output, the ability to log directly to an SQL database far outruns XML or even machine-parseable output. For one, nmapsql does not involve an extra step in the shell to feed the output to a back end.

nmapsql is a direct patch applied to Fyodor's venerable Nmap v3.48 port scanning tool (at the time of this writing, Nmap v3.50 had just been released; an updated version of nmapsql for v3.50 is available from the Web site). It adds MySQL support, but it goes beyond merely adding the results; it also does target tagging, scanner tagging and simple trending. Once the data has been captured in an SQL database, a whole new set of tasks is possible. nmapsql can be downloaded from sourceforge.net/projects/nmapsql. At the moment, it relies on MySQL's client interface for data manipulation.

As security administrators aren't necessarily database wizards, nmapsql was designed to be simple to use. It's simple enough that most of the information one might want in a network scan can be obtained from a single table. Simplicity is also why IP addresses are stored as plain text instead of with inet_aton() notation. I'm aware of the performance penalties of text manipulation, but the focus is to demonstrate the convenience with a small data set. The target tags, runtime and scanner IDs are there for numeric searches in large data sets where performance is critical.

In this article, we concentrate first on running an SQL-enabled scan to establish a baseline of open ports and live targets on a network. Later, we take a look at the data captured in SQL and find ways of comparing the results.

nmapsql starts out by reading the ~/nmpsql.rc file in the effective user's home directory. So, if you used su to get to root before running nmapsql, ~root/nmapsql.rc is read. At this time, only four items are read from nmapsql.rc, each on a line by itself and in the item=value format common to many other utilities. The items are server=localhost, db=nmaplog, user=nmap and passwd=scanamanga.

The server is the DNS name of the host where MySQL is running, and db is the name of the database on that server. The user and password items are used to connect to the database, and the user listed must have at least SELECT, INSERT and UPDATE rights to the database.

On the command line, nmapsql introduces four new options to those Nmap already provides: --mysql, --runid, --targetid and --scannerid. When the nmapsql binary is executed without any of these options, it behaves exactly as normal Nmap does. None of these options interfere with Nmap's existing output abilities, so it's entirely possible to log to SQL as well as to produce machine-parseable output from the same scan.

The --mysql option, without any of the other nmapsql options on the command line, triggers MySQL logging, with all tags and IDs auto-assigned. All other nmapsql options automatically assume --mysql. Auto-assignment always picks the maximum available value in the respective table and increments by one.

The scanner ID feature, initiated by the --scanner-id xxx option, where xxx is the ID value, is intended for scenarios where more than one scanner is deployed, perhaps in a multisubnet environment. The scanner ID, along with the runtime ID, is stored in the portstat table to allow separation of result sets by the scanning host. It would be simple to separate the results of scanner ten, for instance, using a query like this:

mysql> select * from portstat
    -> where scannerid = 10 and runid = 100;

The --run-id xxx option is used to specify a specific ID for the current nmapsql run. If this option is not specified, a system-generated ID is used. If the runid specified already exists in the database, it is reused. This feature allows results of multiple scans to be grouped conveniently under a single runid.

The runtime ID and its associated information are stored in the runlist table. See the “Tables Used by nmaplog” sidebar for a summary of the tables used. Some of the runtime information is updated at the end of the scan, including the total number of possible targets specified on the command line and the total number found alive. Similarly, the scanner ID and related information go to the scanners table.