Responding to a Security Incident
I spend a good chunk of my time on mailing lists and organizations concerned with monitoring hacker activity. Such lists are the INCIDENTS list from SecurityFocus.com and the SANS GIAC effort, providing a daily update of hacker activities from various parties around the world. Often, the question of the value of reporting an incident is debated. I routinely counsel people to report most incidents they see. What this does for the ISP is help them gather information about a set of independently correlated data about a nefarious customer or a compromised machine on their network. Just don't expect much to be done about it. Most ISPs don't react and aren't very neighborly. Some of us in the business routinely block entire networks from connecting to our networks based on their patterns of allowing unseemly activity to continue.
We'll not go into detecting incidents, but we will define them as port probes, port scans, denial of service (DoS) attempts and unauthorized access attempts. Each of these warrants investigation, some more than others. Combining intrusion detection software with log analysis (which you should be doing anyhow), these events should stand out.
We'll start with a simple detect and extract information from it, deciding how best to proceed. This first one is a log message of a refused connect request from a service, wrapped using TCP wrappers (tcpd). As of this writing, FTP sweeps are all the rage:
$ grep refuse /var/log/secure Nov 8 15:26:31 linux ftpd: refused connect from 7dyn94.ztm.casema.net
TCP_wrappers makes an excellent basic intrusion detection system, logging both successes and failures. You should be using at least TCP_wrappers for access control and minimizing what services people can connect to at all.
A brief note about connection attempts: One important thing that gets overlooked by the casual paranoid Internet user is the threshold of what defines a probe or an attack, and what is worth reporting. Usually it's not worth reporting SMTP (port 25/TCP) or HTTP (port 80/TCP) probes, unless you can demonstrate that it was more than a simple connection attempt. Why? Say I see you on a chatline, and I want to send you some mail directly or see if you have an interesting web site. If you are blocking those requests, they would show up as port probes. But they're usually more innocent than they may appear. The same goes for ping requests (ICMP_ECHO), which are used, for example, in gaming and Napster to determine latency. Don't get in a tizzy about simple things; focus on actual intrusion attempts, like an FTP exploit or a network sweep for a service.
One of the first things we do is map an IP address to that hostname. I prefer to work with IP addresses rather than hostnames, as sometimes ISPs will sell their netblocks in smaller quantities, yet retain administrative control of them.
$ nslookup 7dyn94.ztm.casema.net Name: 7dyn94.ztm.casema.net Address: 220.127.116.11
Now we begin to dig around for domain information using this network address. You can use command line "whois" information to find out about this network address.
Two different major types of databases exist, depending on if you are searching by hostname or network addresses. Network names are handled by registries like Network Solutions, and network numbers are handled by the ARIN (American Registry of Internet Numbers) database. Because we're using a network number, we'll be using the ARIN database, but we'll do both to illustrate how it is done.
A quick note on using whois. You can specify query@whois-engine using fwhois, which is very common on Linux boxes. However, this doesn't work on normal whois clients. You have to use the -h whois-engine option, instead. Hence, the queries fwhois email@example.com and whois -h whois.networksolutions.com domain.com are equivalent. In these examples, we'll be using the fwhois command-line tool or web-based whois searches, and whois is just a symbolic link to fwhois.
First, the ARIN search:
$ whois firstname.lastname@example.org [whois.arin.net] European Regional Internet Registry/RIPE NCC (NET-RIPE-NCC-) These addresses have been further assigned to European users. Contact information can be found in the RIPE database, via the WHOIS and TELNET servers at whois.ripe.net, and at http://www.ripe.net/db/whois.html Netname: RIPE-NCC-212 Netblock: 18.104.22.168 - 22.214.171.124 Maintainer: RIPE Coordinator: RIPE Network Coordination Centre (RIPE-NCC-ARIN) nicdb@RIPE.NET +31 20 535 4444 Fax- - +31 20 535 4445 Domain System inverse mapping provided by: NS.RIPE.NET 126.96.36.199 NS.EU.NET 188.8.131.52 AUTH03.NS.UU.NET 184.108.40.206 NS2.NIC.FR 220.127.116.11 SUNIC.SUNET.SE 18.104.22.168 MUNNARI.OZ.AU 22.214.171.124 NS.APNIC.NET 126.96.36.199 To search on arbitrary strings, see the Database page on the RIPE NCC web-site at http://www.ripe.net/db/ Record last updated on 16-Oct-1998. Database last updated on 9-Nov-2000 07:02:34 EDT. The ARIN Registration Services Host contains ONLY Internet network information: Networks, ASNs and related POCs. Use the whois server at rs.internic.net for DOMAIN related Information and whois.nic.mil for NIPRNET Information.
Aside from all the extraneous information, we can see from this record that ARIN doesn't handle exact information for this network block. We'll then use network name in a query:
$ whois email@example.com Registrant: N.V. Casema (CASEMA-DOM) P.O. Box 345 Delft, 2600 AH THE NETHERLANDS Domain Name: CASEMA.NET Administrative Contact: Network Operations Centre (NOC137-ORG) domain-tech@EURO.NET EuroNet Internet BV Muiderstraat 1 Amsterdam NL +31 20 5355555 Fax- +31 20 5355400 Technical Contact, Zone Contact: Davids, Marco (MD2446) domaintech1@CASEMA.NET N.V. Casema - IKC Brassersplein 2 Delft ZH 2612 CT NL +31(0)15 8881000 (FAX) +31(0)15 8881099 Billing Contact: Finance Departement (FD5-ORG) nic-invoices@EURONET.NL EuroNet Internet BV Postbus 11095 Amsterdam NL +31 20 5355555 Fax- +31 20 5355400 Record last updated on 13-Jun-2000. Record expires on 30-Jan-2001. Record created on 28-Jan-1997. Database last updated on 7-Nov-2000 19:15:09 EST. Domain servers in listed order: NS.CASEMA.NL 188.8.131.52 NS1.CASEMA.NET 184.108.40.206
We now have the information we need to proceed. We'll be sending our memo to the addresses listed as the billing contact, technical contact and the administrative contact. For good measure, we'll toss in the often used addresses security@ and abuse@, hopefully targeting the right people.
Additional information you may want to gather include a traceroute to find their upstream network providers and the AUP (acceptable use policy) of the ISP. Usually digging around on www.domain.com can turn it up. Rarely do ISPs have hacker-friendly AUP terms, but it's been known to happen.
Now we'll put this information into a sample memo to send to the domain contacts to register our complaint. Remember the following things:
* Addresses that you record can be forged quite easily* It's usually not the staff of the ISP or network that are attempting to violate security measures
Further, general rules of thumb that have proven successful for many people over the years are:
* If it's a dedicated line, like a cable modem or a DSL line, it's probably a compromised machine* If it's a dial-up line, it is often a stolen account
With this in mind we can craft our memo.
Things that must go into the memo are:
* Your identification, including name, organization (if any), and role in that organization* The brief purpose of why you are contacting them* Log output to prove your point, time stamped* The address of the host that was hit* Your timezone information, preferably with an offset of GMT (Greenwich Mean Time).* If a compromise occurred
For good measure, I always use a friendly tone.
Another option is to PGP sign your message. This will provide a stronger signature for you, and provide proof that you sent the message to the ISP. This will also help verify that the message was received without being modified in transit. If you do this, include in your memo the location of where you PGP key can be downloaded. In your signature line is a good place.
Don't be stupid, don't make threatening gestures, don't threaten legal action unless you have spoken to a lawyer about things, don't demand action be taken and don't threaten retaliation. Don't retaliate, you'll just become guilty of a violation of your AUP as well. I have seen this numerous times before; please do not fall victim to it as well.
Below is a sample memo, one which mirrors one that I would send regarding the the above detected incident:
To: domain-tech@EURO.NET, domaintech1@CASEMA.NET, nic-invoices@EURONET.NL, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org From: Jose Nazario (email@example.com) Subject: [SECURITY] FTP probe from casema.net domain Good day, My name is Jose Nazario and I am a customer of BigCorp ISP. I am writing to you today to note that a machine I own was probed by a host in the casema.net network. This was a probe to the FTP daemon, port 21/TCP. You are listed as a responsible party in the domain records. As there are a number of problems with FTP servers currently making the rounds with hackers, this may represent someone attempting to find vulnerable hosts to compromise. This may represent a legitimate user in violation of their AUP or a compromised machine on your network. The host on my network that received the traffic from your domain is myhost.bigisp.com (10.10.32.4). The log entry for this incident looks like this: Nov 8 15:26:31 linux ftpd: refused connect from 7dyn94.ztm.casema.net All times are in US Eastern (GMT-5). While no reply is expected, the favor of an acknowledgment would be appreciated. Thank you for your attention to this matter, Jose Nazario firstname.lastname@example.org http://www.bigisp.com/~jnazario/
This is the first thing that many people think about when documenting a security incident. Most people have this image of a scruffy 15-year-old hacker being led to a prison in handcuffs. The truth is this rarely happens, even with the best of documentation on your end. The reasons for this are many and varied, but can be summarized in large measure by the difficulty in proving who was using what system at what time, and the forensic value of the evidence.
The FBI will not investigate a security incident unless the monetary damage is above $5,000US, someone's life was in danger or interstate commerce was affected. Even then the evidence may have lost forensic value for a criminal prosecution.
If you think the legal authorities should be contacted, you should speak to your site supervisors and any legal advisors to ensure you have a plan before any security incidents take place. Two books you may want to begin with are listed below in the references section.
One questions people have is "Should I contact CERT or a similar organization with this information?" In my experience, it is usually not necessary except under pretty uncommon circumstances.
Most of the incidents you will see are probes of one type or another. Port scans, probes for services like RPC services or DNS servers, maybe even a few web probes for cgi-bin scripts, but just probes. While it's pretty obvious they're sizing you up for an attempt to break in, they didn't get in and they didn't cause any damage.
Times when I have contacted CERT (http://www.cert.org) here in the United States are when legitimate intrusions have taken place, novel exploits have been used which are not documented in the security community and when the system has been used to gain entry into other computers or for DoS attacks. This helps provide a central place for the information on the attack to be stored and evaluated, and, potentially, a third party to show that actions were being taken to remedy the situation. Also, contacting CERT is a good idea if the incident is above a probe, such as a real documented attempt or a successful break-in, and occurred from a host outside of the United States.
CERT-type organizations exist all over the world and are worth reporting to if you file an incident with CERT. File a similar report with the organization in the originating country. A comprehensive list of CERT and similar organizations from around the world can be found at FIRST's contact information page. FIRST is an organization that provides a forum for incident handling.
Note that CERT has no legal authority, but does work with the authorities to investigate security incidents when they are warranted.
So, what will happen now? Should you sit back and wait for the ISP you just contacted to report back, "Wow! We got rid of that user!" Think you're going to break up a large international ring of hackers trying to take over the Pentagon?
You'll be lucky to get any notification that your memo was received, unless it's an automated reply. You'll probably not get any followup done, either, on their end. So why bother?
Rarely do potential attackers work just on you. Unless you hang out on the Internet and antagonize people for a hobby or a living, chances are they're not targeting you, but they're doing wide sweeps of the Internet to find vulnerable hosts. What you've done is provide the ISP with data that says "They were here, too". This is why you bother. If enough people complain they'll see it's a real problem and probably fix it.
Remember, the ISP staff are busy people. Routers explode, billing situations come up with their customers, new hardware has to be installed, in short, they have daily business to take care of. Their response will be tailored to the severity of the incident. One port probe on one machine is nothing for them to worry about. If you can demonstrate it was a network-scale issue or a recurring problem, then they may start to care.
So, carry on with your network activities, keep a watchful eye out for problems and improve your defenses as needed.
The SecurityFocus INCIDENTS list is an excellent mailing list about security incidents. Sit back and learn.http://www.securityfocus.com
The SANS GIAC effort is also a good place to learn about detects, what generates them and the like.http://www.sans.org/giac.htm
The GeekTools Whois Proxy is my favorite method of searching for domain information about a hostname, a network name or even a numerical address. It's smart enough to hit the right servers for information. You can also download the source and run your own local instance.http://www.geektools.com/cgi-bin/proxy.cgi
Documentation on using TCP_wrappers and general Linux hardening can be found athttp://www.enteract.com/~lspitz/linux.html
Using grep on your logs will make your life a lot easier. A good piece of documentation on it is athttp://www.sunworld.com/sunworldonline/swol-11-1999/swol-11-unix101.html
Two books in investigating computer crimes for a legal standpoint are:
Criminalistics: An Introduction to Forensic Science by Richard Saferstein. Prentice Hall, 1998.
Investigating Computer Crime by Franklin Clark and Ken Diliberto. CRC Press, 1996.