Interrogating a Linux Machine

The other day, a client called upon me to perform a hardware and software inventory on all of the computers on his network. There weren't that many machines to inventory, but we needed to gather quite a bit of information about each one. The client was a Microsoft shop and so I had to deal with about an even mix of Windows XP and Vista with a few Windows 95, 98, and ME machines thrown in for good measure.

So off I went, with an Excel spreadsheet in hand. I visited each machine in person. For each machine, I wanted the network configuration, the workgroup configuration, and the hardware configuration. We also wanted to know what software was installed as well as which drivers were being used. Bonus points for being able to determine what USB devices were being used.

Needless to say, this was tedious work. Some of the data, such as the network configuration, was available in the same place on each platform, the command line. Other information was only available at various control panels, and they varied by platform.

Granted, things might have been easier if I was a bit more proficient in using Windows. But it got me to thinking. How would I do this job if I were working in a Linux shop? How much of this information could I gather? Where would I look? How hard would it be?

It turns out that Linux keeps most of this information in fairly standardized locations. The rest of the information, such as installed software, can be had with just a few platform-specific commands.

I'm fond of telling people that “everything is easier in Linux.” This statement stems from the fact that I don't find myself “hunting” for just the right menu or GUI widget that I need to get a task done. Now sure, I'm demonstrating a bias that I have formed based on years of experience with Linux. But by the end of this article, I hope you see how easy it is to interrogate a machine with just a few commands.

Now let's get started.

First, we need to get the complete network configuration. Once we log in, we see the machine's hostname as part of our shell prompt. We can take a quick look at the DNS configuration with “cat /etc/resolv.conf”. Before going on, we'll simply verify that the DNS configuration looks right. Then we'll get a list of network interfaces, as well as their IP and MAC addresses with “ifconfig -a”.

eth0      Link encap:Ethernet  HWaddr 00:17:31:64:B0:BD
         inet addr:10.0.1.56  Bcast:10.0.1.255  Mask:255.255.255.0
         inet6 addr: fe80::217:31ff:fe64:b0bd/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:127554145 errors:255 dropped:0 overruns:0 frame:255
         TX packets:84151845 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:3916476961 (3735.0 Mb)  TX bytes:4229971584 (4034.0 Mb)
         Interrupt:16 Base address:0x6000

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:16436  Metric:1
         RX packets:68839 errors:0 dropped:0 overruns:0 frame:0
         TX packets:68839 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:2933412 (2.7 Mb)  TX bytes:2933412 (2.7 Mb)

From this, we can gleen the IP address as well as the MAC address of the Ethernet interface. We'll ignore the Loopback interface.

We can take a quick look at the routing table to make sure there's nothing strange going on with the “route -n” command.

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         10.0.1.1        0.0.0.0         UG    0      0        0 eth0

The default gateway, 10.0.1.1, looks right, so we move on.

Well, that's the network configuration, but what kind of machine is this? We can gather a lot of information from the /proc filesystem. For example, we can find out what kind of CPU is installed with the “cat /proc/cpuinfo” command.

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 35
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
stepping        : 2
cpu MHz         : 2200.000
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm
3dnowext 3dnow pni lahf_lm cmp_legacy ts fid vid ttp
bogomips        : 4425.38

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 35
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
stepping        : 2
cpu MHz         : 2200.000
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm
3dnowext 3dnow pni lahf_lm cmp_legacy ts fid vid ttp
bogomips        : 4422.96

As you can see, I'm using an AMD Athlon 64 (4200+) with 2 cores. You can even see that Linux checks for some of the Intel/AMD CPU bugs that have been found over the years, as well some of the special instruction sets that have evolved, such as MMX or 3dnow.

We can determine how much memory is installed by looking at the size of /proc/kcore with the “ls -lah /proc/kcore” command, which will print the file size in human-friendly format.

-r-------- 1 root root 897M Oct 15 12:31 /proc/kcore

We'll just round that 897M up to 1G, since the kernel uses some of the installed memory.

It sure would be nice to find out what hardware is installed without having to open the chassis. This is as easy as using the lspci command.

00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio
Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM
Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron]
Miscellaneous Control
05:06.0 VGA compatible controller: nVidia Corporation NV34 [GeForce FX 5200]
(rev a1)

As you can see, I've got an nVidia GeForce FX 5200 video card and nVidia CK804 AC'97 soundcard on my system's motherboard. You can also see that I've got USB. More on that in a moment. Not every Linux distribution installs lspci by default; you should do this anytime you build a new machine, though.

Sometimes you get called upon to fix a hardware problem where the device just doesn't seem to work right. Soundcards are the typical example and the problem is usually caused by an IRQ conflict or having an IRQ shared with another busy device such as the hard drive or network interface. We can tell what interrupts are being used by what with the “cat /proc/interrupts” command

          CPU0       CPU1
 0:  220186162   56790290   IO-APIC-edge      timer
 1:          0          2   IO-APIC-edge      i8042
 8:          1          1   IO-APIC-edge      rtc
 9:          0          0   IO-APIC-fasteoi   acpi
 12:          0          4   IO-APIC-edge      i8042
 14:    1375087    2345774   IO-APIC-edge      ide0
 15:    9971601         12   IO-APIC-edge      ide1
 16:  173144676  152683932   IO-APIC-fasteoi   ohci_hcd:usb2, eth0
 17:          0          0   IO-APIC-fasteoi   libata, NVIDIA nForce Audio
 18:          0          0   IO-APIC-fasteoi   libata
 19:     242148     231918   IO-APIC-fasteoi   ehci_hcd:usb1
 20:   93937183   75701777   IO-APIC-fasteoi   nvidia
NMI:          0          0
LOC:  276981218  276986794
ERR:          0
MIS:          0

We can quickly assess our hard drive configuration with the “fdisk -l /dev/?d?” command.

Disk /dev/hda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1          13      104391   83  Linux
/dev/hda2              14          76      506047+  82  Linux swap / Solaris
/dev/hda3              77        3812    30009420   83  Linux
/dev/hda4            3813       38913   281948782+  83  Linux

Disk /dev/hdb: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1       30401   244196001   83  Linux

Disk /dev/hdc: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/hdc1               1       30515   245111706   83  Linux

Here you can see that I've got 3 IDE drives. Using the “?” wildcard in the command line makes this go much faster.

By the way, if you wanted to know what type of hard drive /dev/hda was, you could ask with this command, “cat /proc/ide/hda/model”

The results of this command on my machine are “WDC WD3200JB-00KFA0,” which indicates that my /dev/hda is a Western Digital Caviar 320Gb. This is a case where Google is your friend.

Speaking of harddrives, we can determine which filesystems our system supports by looking at the /proc/filesystems file. The results are intuitive, so I'll leave it as an exercise for the reader.

If the kernel was configured to support it, the /proc/config.gz file is a compressed file that contains the configuration of the currently running kernel. This file, once uncompressed, is suitable for use as the .config file found in the root of the Linux Kernel source tree. By capturing this file, you can reproduce, and make changes to the current kernel configuration.

Of course many of the device drivers in the Linux Kernel load as dynamically loaded modules. We can determine which drivers are loaded this way by using the lsmod command.

Module                  Size  Used by
nvidia               7096684  48
vboxdrv                66928  0

Sometimes a driver requires another driver in order to operate. This situation would be indicated in the “Used by” column.

Remember earlier when I said we'd get bonus points for identifying what was plugged into the USB ports? Well we can easily do this with the lsusb command.

Bus 001 Device 001: ID 0000:0000
Bus 002 Device 002: ID 413c:1002 Dell Computer Corp. Keyboard Hub
Bus 002 Device 004: ID 413c:2002 Dell Computer Corp. SK-8125 Keyboard
Bus 002 Device 001: ID 0000:0000

And finally, we need to get a list of the installed software. RPM-based distributions give us this information with the “rpm -qa” command. I don't use Debian, but I'm told you can use the “dpkg -l” command to get a list of installed packages. Under Gentoo, I'd use “emerge --pretend --emptytree world” or install the qpkg tool.

At this point, we know almost everything there is to know about the machine. The Linux Kernel has other secrets though, and I encourage you to poke around in the /proc and /sys filesystems to find them.

Though some of these commands require root privilege, none of them required a GUI, a mouse, or even physical access to the machine. If you had a login and root password, you could do all of this remotely and even automate it with a script. See, I told you it would be easy!