Responding to a Security Incident

by Jose Nazario

I spend a good chunk of my time on mailing lists and organizations concerned with monitoring hacker activity. Such lists are the INCIDENTS list from SecurityFocus.com and the SANS GIAC effort, providing a daily update of hacker activities from various parties around the world. Often, the question of the value of reporting an incident is debated. I routinely counsel people to report most incidents they see. What this does for the ISP is help them gather information about a set of independently correlated data about a nefarious customer or a compromised machine on their network. Just don't expect much to be done about it. Most ISPs don't react and aren't very neighborly. Some of us in the business routinely block entire networks from connecting to our networks based on their patterns of allowing unseemly activity to continue.

We'll not go into detecting incidents, but we will define them as port probes, port scans, denial of service (DoS) attempts and unauthorized access attempts. Each of these warrants investigation, some more than others. Combining intrusion detection software with log analysis (which you should be doing anyhow), these events should stand out.

An Example Detect

We'll start with a simple detect and extract information from it, deciding how best to proceed. This first one is a log message of a refused connect request from a service, wrapped using TCP wrappers (tcpd). As of this writing, FTP sweeps are all the rage:

   $ grep refuse /var/log/secure
   Nov 8 15:26:31 linux ftpd[3689]: refused connect from
   7dyn94.ztm.casema.net
 

TCP_wrappers makes an excellent basic intrusion detection system, logging both successes and failures. You should be using at least TCP_wrappers for access control and minimizing what services people can connect to at all.

A brief note about connection attempts: One important thing that gets overlooked by the casual paranoid Internet user is the threshold of what defines a probe or an attack, and what is worth reporting. Usually it's not worth reporting SMTP (port 25/TCP) or HTTP (port 80/TCP) probes, unless you can demonstrate that it was more than a simple connection attempt. Why? Say I see you on a chatline, and I want to send you some mail directly or see if you have an interesting web site. If you are blocking those requests, they would show up as port probes. But they're usually more innocent than they may appear. The same goes for ping requests (ICMP_ECHO), which are used, for example, in gaming and Napster to determine latency. Don't get in a tizzy about simple things; focus on actual intrusion attempts, like an FTP exploit or a network sweep for a service.

Finding Information

One of the first things we do is map an IP address to that hostname. I prefer to work with IP addresses rather than hostnames, as sometimes ISPs will sell their netblocks in smaller quantities, yet retain administrative control of them.

   $ nslookup 7dyn94.ztm.casema.net
   
   Name: 7dyn94.ztm.casema.net
   Address: 212.64.110.94

Now we begin to dig around for domain information using this network address. You can use command line "whois" information to find out about this network address.

Two different major types of databases exist, depending on if you are searching by hostname or network addresses. Network names are handled by registries like Network Solutions, and network numbers are handled by the ARIN (American Registry of Internet Numbers) database. Because we're using a network number, we'll be using the ARIN database, but we'll do both to illustrate how it is done.

A quick note on using whois. You can specify query@whois-engine using fwhois, which is very common on Linux boxes. However, this doesn't work on normal whois clients. You have to use the -h whois-engine option, instead. Hence, the queries fwhois domain.com@whois.networksolutions.com and whois -h whois.networksolutions.com domain.com are equivalent. In these examples, we'll be using the fwhois command-line tool or web-based whois searches, and whois is just a symbolic link to fwhois.

First, the ARIN search:

   $ whois 212.64.110.94@whois.arin.net
[whois.arin.net]
European Regional Internet Registry/RIPE NCC (NET-RIPE-NCC-)
   These addresses have been further assigned to European users.
   Contact information can be found in the RIPE database, via the
   WHOIS and TELNET servers at whois.ripe.net, and at
   http://www.ripe.net/db/whois.html
   Netname: RIPE-NCC-212
   Netblock: 212.0.0.0 - 212.255.255.255
   Maintainer: RIPE
   Coordinator:
      RIPE Network Coordination Centre  (RIPE-NCC-ARIN) nicdb@RIPE.NET
      +31 20 535 4444
Fax- - +31 20 535 4445
   Domain System inverse mapping provided by:
   NS.RIPE.NET                  193.0.0.193
   NS.EU.NET                    192.16.202.11
   AUTH03.NS.UU.NET             198.6.1.83
   NS2.NIC.FR                   192.93.0.4
   SUNIC.SUNET.SE               192.36.125.2
   MUNNARI.OZ.AU                128.250.1.21
   NS.APNIC.NET                 203.37.255.97
   To search on arbitrary strings, see the Database page on
   the RIPE NCC web-site at http://www.ripe.net/db/
   Record last updated on 16-Oct-1998.
   Database last updated on 9-Nov-2000 07:02:34 EDT.
The ARIN Registration Services Host contains ONLY Internet
network information: Networks, ASNs and related POCs.
Use the whois server at rs.internic.net for DOMAIN related
Information and whois.nic.mil for NIPRNET Information.

Aside from all the extraneous information, we can see from this record that ARIN doesn't handle exact information for this network block. We'll then use network name in a query:

   $ whois casema.net@whois.networksolutions.com 
Registrant:
N.V. Casema (CASEMA-DOM)
   P.O. Box 345
   Delft, 2600 AH
   THE NETHERLANDS
   Domain Name: CASEMA.NET
   Administrative Contact:
      Network Operations Centre  (NOC137-ORG)  domain-tech@EURO.NET
      EuroNet Internet BV
      Muiderstraat 1
      Amsterdam
      NL
      +31 20 5355555
      Fax- +31 20 5355400
   Technical Contact, Zone Contact:
      Davids, Marco  (MD2446)  domaintech1@CASEMA.NET
      N.V. Casema - IKC
      Brassersplein 2
      Delft
      ZH
      2612 CT
      NL
      +31(0)15 8881000 (FAX) +31(0)15 8881099
   Billing Contact:
      Finance Departement  (FD5-ORG)  nic-invoices@EURONET.NL
      EuroNet Internet BV
      Postbus 11095
      Amsterdam
      NL
      +31 20 5355555
      Fax- +31 20 5355400
   Record last updated on 13-Jun-2000.
   Record expires on 30-Jan-2001.
   Record created on 28-Jan-1997.
   Database last updated on 7-Nov-2000 19:15:09 EST.
   Domain servers in listed order:
   NS.CASEMA.NL                 195.96.96.97
   NS1.CASEMA.NET               195.96.96.33

We now have the information we need to proceed. We'll be sending our memo to the addresses listed as the billing contact, technical contact and the administrative contact. For good measure, we'll toss in the often used addresses security@ and abuse@, hopefully targeting the right people.

Additional information you may want to gather include a traceroute to find their upstream network providers and the AUP (acceptable use policy) of the ISP. Usually digging around on www.domain.com can turn it up. Rarely do ISPs have hacker-friendly AUP terms, but it's been known to happen.

The Memo

Now we'll put this information into a sample memo to send to the domain contacts to register our complaint. Remember the following things:

* Addresses that you record can be forged quite easily* It's usually not the staff of the ISP or network that are attempting to violate security measures

Further, general rules of thumb that have proven successful for many people over the years are:

* If it's a dedicated line, like a cable modem or a DSL line, it's probably a compromised machine* If it's a dial-up line, it is often a stolen account

With this in mind we can craft our memo.

Things that must go into the memo are:

* Your identification, including name, organization (if any), and role in that organization* The brief purpose of why you are contacting them* Log output to prove your point, time stamped* The address of the host that was hit* Your timezone information, preferably with an offset of GMT (Greenwich Mean Time).* If a compromise occurred

For good measure, I always use a friendly tone.

Another option is to PGP sign your message. This will provide a stronger signature for you, and provide proof that you sent the message to the ISP. This will also help verify that the message was received without being modified in transit. If you do this, include in your memo the location of where you PGP key can be downloaded. In your signature line is a good place.

Don't be stupid, don't make threatening gestures, don't threaten legal action unless you have spoken to a lawyer about things, don't demand action be taken and don't threaten retaliation. Don't retaliate, you'll just become guilty of a violation of your AUP as well. I have seen this numerous times before; please do not fall victim to it as well.

Below is a sample memo, one which mirrors one that I would send regarding the the above detected incident:

To: domain-tech@EURO.NET, domaintech1@CASEMA.NET, nic-invoices@EURONET.NL,
security@casema.net, abuse@casema.net, contact@bigisp.com
From: Jose Nazario (jose@bigisp.com)
Subject: [SECURITY] FTP probe from casema.net domain
Good day,
My name is Jose Nazario and I am a customer of BigCorp ISP. I am writing
to you today to note that a machine I own was probed by a host in the
casema.net network. This was a probe to the FTP daemon, port 21/TCP. You
are listed as a responsible party in the domain records. As there are a
number of problems with FTP servers currently making the rounds with
hackers, this may represent someone attempting to find vulnerable
hosts to compromise. This may represent a legitimate user in violation of
their AUP or a compromised machine on your network.
The host on my network that received the traffic from your domain is
myhost.bigisp.com (10.10.32.4). The log entry for this incident looks like
this:
Nov 8 15:26:31 linux ftpd[3689]: refused connect from
        7dyn94.ztm.casema.net
All times are in US Eastern (GMT-5).
While no reply is expected, the favor of an acknowledgment would be
appreciated.
Thank you for your attention to this matter,
Jose Nazario                       jnazario@bigisp.com
                        http://www.bigisp.com/~jnazario/
Contacting Legal Authorities

This is the first thing that many people think about when documenting a security incident. Most people have this image of a scruffy 15-year-old hacker being led to a prison in handcuffs. The truth is this rarely happens, even with the best of documentation on your end. The reasons for this are many and varied, but can be summarized in large measure by the difficulty in proving who was using what system at what time, and the forensic value of the evidence.

The FBI will not investigate a security incident unless the monetary damage is above $5,000US, someone's life was in danger or interstate commerce was affected. Even then the evidence may have lost forensic value for a criminal prosecution.

If you think the legal authorities should be contacted, you should speak to your site supervisors and any legal advisors to ensure you have a plan before any security incidents take place. Two books you may want to begin with are listed below in the references section.

Contacting CERT Organizations

One questions people have is "Should I contact CERT or a similar organization with this information?" In my experience, it is usually not necessary except under pretty uncommon circumstances.

Most of the incidents you will see are probes of one type or another. Port scans, probes for services like RPC services or DNS servers, maybe even a few web probes for cgi-bin scripts, but just probes. While it's pretty obvious they're sizing you up for an attempt to break in, they didn't get in and they didn't cause any damage.

Times when I have contacted CERT (http://www.cert.org) here in the United States are when legitimate intrusions have taken place, novel exploits have been used which are not documented in the security community and when the system has been used to gain entry into other computers or for DoS attacks. This helps provide a central place for the information on the attack to be stored and evaluated, and, potentially, a third party to show that actions were being taken to remedy the situation. Also, contacting CERT is a good idea if the incident is above a probe, such as a real documented attempt or a successful break-in, and occurred from a host outside of the United States.

CERT-type organizations exist all over the world and are worth reporting to if you file an incident with CERT. File a similar report with the organization in the originating country. A comprehensive list of CERT and similar organizations from around the world can be found at FIRST's contact information page. FIRST is an organization that provides a forum for incident handling.

Note that CERT has no legal authority, but does work with the authorities to investigate security incidents when they are warranted.

Now What?

So, what will happen now? Should you sit back and wait for the ISP you just contacted to report back, "Wow! We got rid of that user!" Think you're going to break up a large international ring of hackers trying to take over the Pentagon?

Think again.

You'll be lucky to get any notification that your memo was received, unless it's an automated reply. You'll probably not get any followup done, either, on their end. So why bother?

Rarely do potential attackers work just on you. Unless you hang out on the Internet and antagonize people for a hobby or a living, chances are they're not targeting you, but they're doing wide sweeps of the Internet to find vulnerable hosts. What you've done is provide the ISP with data that says "They were here, too". This is why you bother. If enough people complain they'll see it's a real problem and probably fix it.

Remember, the ISP staff are busy people. Routers explode, billing situations come up with their customers, new hardware has to be installed, in short, they have daily business to take care of. Their response will be tailored to the severity of the incident. One port probe on one machine is nothing for them to worry about. If you can demonstrate it was a network-scale issue or a recurring problem, then they may start to care.

So, carry on with your network activities, keep a watchful eye out for problems and improve your defenses as needed.

Resources

The SecurityFocus INCIDENTS list is an excellent mailing list about security incidents. Sit back and learn.http://www.securityfocus.com

The SANS GIAC effort is also a good place to learn about detects, what generates them and the like.http://www.sans.org/giac.htm

The GeekTools Whois Proxy is my favorite method of searching for domain information about a hostname, a network name or even a numerical address. It's smart enough to hit the right servers for information. You can also download the source and run your own local instance.http://www.geektools.com/cgi-bin/proxy.cgi

Documentation on using TCP_wrappers and general Linux hardening can be found athttp://www.enteract.com/~lspitz/linux.html

Using grep on your logs will make your life a lot easier. A good piece of documentation on it is athttp://www.sunworld.com/sunworldonline/swol-11-1999/swol-11-unix101.html

Two books in investigating computer crimes for a legal standpoint are:

Criminalistics: An Introduction to Forensic Science by Richard Saferstein. Prentice Hall, 1998.

Investigating Computer Crime by Franklin Clark and Ken Diliberto. CRC Press, 1996.

Load Disqus comments