Hack and / - Forensics with Ext4

Learn from my mistakes as I figure out how to gather forensics data on an ext4 filesystem.

One great thing about writing technical articles is that you have a nice collection of documentation you can turn to. I tell people that I probably reference my books and articles more than anyone else, because although I may not always remember specific steps to perform a task, I do always remember whether I wrote about how to do it.

One article I find myself referring to now and then is the "Introduction to Forensics" article I wrote in Linux Journal back in the January 2008 issue (my first feature article in Linux Journal). In that article, I walk through how to use Autopsy, a front end to Sleuthkit, to perform your own forensics investigation on a server that has been hacked. Recently, I had to perform an investigation on a server that fell victim to an SSH brute-force attack (use SSH keys!) and discovered that my personal documentation no longer worked. In this article, I walk through the symptoms of this problem and ultimately how I was able to work around it.

The Victim

As I mentioned, recently I investigated a server (let's call it alvin to protect the innocent) that had been compromised by a brute-force attack. I followed the procedure I document in my forensics article to a T. I pulled the plug from the server the moment I detected it was compromised, immediately created an image of the entire drive, created a second copy of the image on a separate drive that I would work from, and once my evidence was secure, I re-installed the affected server with a clean OS.

The first big difference about this system I ran into compared to past investigations was the sheer size of the data. It's one thing to image a 10–20GB disk and quite another when the drive is hundreds of gigabytes and every action—creating the initial image, image duplication and md5sums—takes hours. This was a slow process to say the least. Once I had my working image, I fired up Autopsy and started my investigation. Initially, everything was fine—Autopsy saw the image and was able to detect its partitions, and when I started to browse the filesystem, I could see the contents of the root directory.

The Problem

The first problem showed up when I tried to navigate through the filesystem with Autopsy—all of the directories below the root directory appeared to be empty. I knew that couldn't be right, but I wasn't sure what the problem was. At first, I thought I might just have a corrupted copy of the image, but since the md5sum seemed to take almost as long as copying the image, I started the image copy again from my gold master just to be sure. When that still didn't work, I tried to use Sleuthkit from the command line and even tried tools from The Coroner's Toolkit, yet I got the same result. I thought perhaps Autopsy couldn't support such a large disk image. Before I did too much more research, I decided to try to mount the image loopback to see if it was a corrupt image. After all, if it were, I'd need to create a fresh image before I went any further.

The challenge here was that I had created an image of the entire disk, not just individual partitions. Normally, when you mount something loopback, you mount an image of a partition, and the syntax is something like:


$ sudo mount -o loop /path/to/image /mnt/image

Then, you can browse /mnt/image like any other mounted filesystem. In this case, since the image was of a full partition, I had to figure out how to skip ahead in the image and mount only the partition. Mount has an option called offset that allows you to specify how far ahead in the file to skip before it mounts it as a filesystem, but the trick is knowing what that offset should be set to. It turns out that if you have parted installed, it's relatively simple to find the offset. First, I run parted against alvin's full disk image and tell it to print out the full partition table in bytes:


$ sudo parted /media/Forensics2/images/alvin-sda
GNU Parted 2.2
Using /media/Forensics2/images/alvin-sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit                                                             
Unit?  [compact]? B                                                       
(parted) print                                                            
Model:  (file)
Disk /media/Forensics2/images/alvin-sda: 251059544064B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start          End            Size           Type     ...
 1      32256B         242242721279B  242242689024B  primary  ...
 2      242242752512B  249999160319B  7756407808B    extended ...
 5      242242753536B  249999160319B  7756406784B    logical  ...

(parted) quit                         

In this case, I'm interested in the first partition, and from this output, I can see that the partition starts at byte 32256. Now I can use that offset to mount the partition loopback:


$ sudo mount -o loop,ro,offset=32256 alvin-sda /mnt 

Note that I mounted the filesystem read-only here. I didn't want to risk writing any new data to this partition! From this point, I found that I could browse the filesystem under /mnt just fine—the problem somehow had to be with Autopsy and Sleuthkit. I did research, but I couldn't really find any evidence that there was an upper limit to the image size that I was close to, but all the same, I decided to try to create an image of just the initial partition. A few hours later, Autopsy still couldn't use the new image, but strangely enough, I could mount the partition loopback on my filesystem just fine. After trying countless other things, I started to realize that it couldn't be the size of my partition that was a problem. It had to be something else, and it was then that I noticed that this filesystem was ext4.

Like Ext3 Plus One More

To be honest, apart from the fact that ext4 touted faster speeds with large numbers of files while claiming backward compatibility with ext3, I hadn't given the filesystem much thought. I had used it on a few new systems (it is the default on modern Ubuntu installs), and it seemed to work fine. Because it was supposed to be backward-compatible, I initially wrote it off as being a cause of my problem. After more research though, I discovered not only did other users complain about lack of support for ext4 in Sleuthkit, but that the backward compatibility isn't as compatible as you might think. Although it's true that you can mount ext2 and ext3 filesystems as ext4, you can mount ext4 filesystems as ext3 only if the filesystem does not use extents (which is a major new feature of ext4, so it's commonly turned on). Because I couldn't treat this ext4 filesystem as ext3, that meant neither could Sleuthkit.

So if I could mount the partition loopback, what is the big deal if Sleuthkit and Autopsy didn't work? Although being able to browse through the filesystem is a useful feature of forensics tools, another incredibly valuable feature is the ability to create a filesystem timeline. A filesystem timeline organizes all the files into a giant text database where they are ordered according to their MAC times (when the file was last Modified or Accessed or Changed its metadata). With a filesystem timeline, if you have a good idea when the attackers might have been on the system, you can start to track their virtual footprints as they execute programs, browse directories and untar their scripts onto the system. Normally, I would have Autopsy generate this file for me. Luckily, it turned out that a tool from The Coroner's Toolkit called mactime was able to generate the timeline either from an image or from a mounted filesystem. Here's the command I used to create a timeline from the filesystem mounted at /mnt:


$ sudo mactime -d /mnt -R -g /mnt/etc/group -p 
 ↪/mnt/etc/passwd 1/2/1970 > timeline.txt

The -d option specifies the directory where the filesystem is mounted, and -R tells mactime to scan recursively through that directory. The -g and -p options tell mactime to get group and user ID information from the group and passwd files in my mounted filesystem, and finally, the date specifies the date range to start from. The date must be after 1/1/1970 (I'll leave why that is as an exercise for the reader), so I chose the next day.

Although I didn't have the same user experience I was used to with Autopsy, once I could browse the filesystem and view the timeline, I could use the same investigation techniques. Ultimately, I was able to piece together the timeline when the attacker compromised the machine via a weak password, changed the password so no one else could break in the same way, and then downloaded and launched SSH brute-force attack scripts to spread onto other servers.

I admit I was really surprised to see that some of my favorite forensics tools no longer worked. Although it is great that Linux filesystems continue to improve, one unintended downside (or possibly an upside if you are an attacker) is that forensics tools must be upgraded continually to work on modern filesystems. Although I was able to mount the partition and create a timeline, I don't know that I would have been able to recover any deleted files from the partition—another valuable use of forensics tools. That said, at least if I (or you) need to analyze an ext4 image again, I now have all of the steps documented.

______________________

Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Ext4 forensic tool

Solal's picture

Hi,

You could have tried with DFF (Digital Forensics Framework) a GPL-v2 tool that support ext4. You can download it at http://www.digital-forensic.org .

Regards.

ext3-4 copy problem

Pravinkumar's picture

Hi I am workinh on system with MIPSEL,debian Lenny 5.0 on it and using alchemy 1250 processor.

I enaled ext3 and ext4 support from the kernel and compiled modules using e2fsprogram and copied that to initramfs of my kernel.

With the resulting image I booted my device and formatted all my partition with ext4 and trying to copy data to and filesystem to respectine hda partition.

The partition mention above has ext2 and vfat on it earlier. On converting them to ext3 or ext4 and copying data to them giving me problem of loss of data. the md5sum of files copied is wrong.I tried forcefully copting but on device reboot again same thing and because of that my device not working.

Can anyone tell me why this is happening.

If you need more info. please feel free to ask.

mounting ext4 image

PhilW's picture

Interesting article, you can also use file to get the offset of paritions within a disk image. Note the the journal will be replayed when you mount ro, writing to the disk image. To prevent this mount with ro,noload. See http://kernel.org/doc/Documentation/filesystems/ext4.txt

Brute-force attack of RSA keys?

Anonymous's picture

Rankin, did you have "PasswordAuthentication no" on the brute-force attacked compromised server? In the third paragraph you said it was brute-force attack on SSH keys, but later in the second last paragraph you said it was due to weak password. Really interested to know whether SSH servers using **only** PubkeyAuthentication can be compromised by brute-force attack. Thanks.

He did not in the third

Anonymous's picture

He did not in the third paragraph say it was a brute-force attack on ssh keys. He wrote it was an SSH brute-force attack and advised the reader to use SSH keys!

ic, use <> using. Thanks.

Anonymous's picture

ic, use <> using. Thanks.

Do you really use sudo to do

Mike Deschane's picture

Do you really use sudo to do forensics on a compromised/sandboxed machine? I would think that this would be a situation where you would be logged in as root.

Good stuff, thanks for sharing!

Recovery

bico's picture

Thanks for the description. I had the same problem with these tools a year ago when a lot of my files were deleted by a badly written (freeware) application. Recovery of deleted files from ext4 is also more difficult. I could do it only using carving tools like foremost, but the whole directory structure was gone for good. Luckily beagle was indexing my files in the background so I managed to reconstruct the directory structure from the index.

look at kpartx

oernii's picture

look at kpartx

libguestfs is better

Anonymous's picture

libguestfs is better, simpler, cleaner and safer than kpartx. Just use:

guestfish -i -a /path/to/image

or mount it on a local directory:

mkdir /tmp/mnt
guestmount -i -a /path/to/image /tmp/mnt

I use kpartx too to mount

Anonymous's picture

I use kpartx too to mount partitioned images:

$ sudo losetup /dev/loop0 /path/to/image
$ sudo kpartx -a /dev/loop0
$ sudo mount /dev/mapper/loop0p1 /mnt/imagepartition

This is the same method I use too!

ka_'s picture

$ sudo losetup /dev/loop0 /path/to/image
$ sudo kpartx -a /dev/loop0
$ sudo mount /dev/mapper/loop0p1 /mnt/imagepartition

I can absolutely recommend this method - I use it all the time with kvm images. I know - I too used to use the mount -o loop with the offset, but that is a pain with multiple partition in particular!!!

To fully unmount afterwards do the following:
$ sudo umount /dev/mapper/loop0p1
$ sudo kpartx -d /dev/loop0
$ sudo losetup -d /dev/loop0

kpartx

gerry's picture

I just do kpartx -a /path/to/image

And then to make sure kpartx -l /path/to/image

and then when I'm done kpartx -d /path/to/image
(don't forget that step)

I didn't know you could even run kpartx on a block device!

Keep what is working

acobar's picture

Another good reason to kepp ext3 on servers I guess. There are a lot more tools to deal with interoperability and problems on ext[23] than with any other linux file system.

You can mount ext2 and ext3

forangej's picture

You can mount ext2 and ext3 as ext4 without changing the format so these tools can work with them.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix