Linux's Tell-Tale Heart, Part 6
Hello, everyone. Nice to see your smiling faces back here on the corner, the SysAdmin's Corner, that is. This week, we're going to wrap up the "Tell-Tale Heart" series with something so cool, you may never look at your Linux system in the same way. You might not know it, but there is a phantom living on your computer: a filesystem with lots of directories and files, much like any other filesystem, except that this one isn't really there at all. In fact, it disappears every time you shut down, and is rebuilt every time you reboot.
While I was writing this article, I realized I was having almost too much fun. Some things in life are so cool that you just want to keep fiddling with them. In the last few articles, I've been comparing your Linux system logs to a kind of digital diary kept by your system. If logs are a peek at your system's private diaries, then the /proc filesystem (or pseudo-filesystem, since it doesn't really exist) is a means of looking into the very soul of your system.
Okay, I've said it twice now. How can the mysterious /proc filesystem not really exist? If I do an ls -l on the root directory, I get a directory of size zero. Try it out yourself.
# ls -l / (note: only showing partial listing) drwxr-xr-x 2 root root 4096 Aug 1 15:07 patch dr-xr-xr-x 90 root root 0 Aug 12 19:57 proc drwxr-x--- 113 root root 12288 Aug 14 14:55 root drwxr-xr-x 3 root root 4096 Jul 12 16:40 sbin drwxrwxrwt 8 root root 4096 Aug 14 15:21 tmp drwxr-xr-x 21 root root 4096 Mar 13 17:07 usr drwxr-xr-x 18 root root 4096 Jun 22 18:59 var
Notice that /proc does indeed have a zero size. Yet when I change to the /proc directory with cd /proc and do an ls there, I see tons of information.
# cd /proc # ls 1 10102 10105 10311 10315 10702 10710 10712 10714 10715 10716 10717 10718 10719 10720 10721 10722 10723 10755 10766 1145 1146 1147 1148 1149 1150 1151 1152 2 3 330 345 346 355 369 4 420 429 443 446 447 449 450 461 475 490 5 508 528 583 6 612 626 675 715 716 717 718 719 720 723 745 752 753 757 771 773 775 776 7922 808 811 8784 8786 8788 8789 8936 9008 9088 911 914 9957 apm bus cmdline cpuinfo devices dma fb filesystems fs ide interrupts ioports kcore kmsg ksyms loadavg locks mdstat meminfo misc modules mounts net partitions pci rtc scsi self slabinfo sound stat swaps sys tty uptime version
For an empty directory, there's quite a bit here. Let's start with some easy stuff.
Oops. Before I move on, let me offer a word of WARNING!!! <begin weasel words> Unless you know exactly what you are doing, do not go changing the contents of your /proc filesystem. There are some pretty neat things you can do with several of the entries and you can tweak your actual running system by changing various things here, but be warned. <end weasel words> Ahem, where was I? Oh, yeah...
Easy stuff. Look at the file called "cpuinfo", for starters. Another zero-size file. Now, cat the file and have a look at the results. This is what mine shows:
# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 5 model : 2 model name : Pentium 75 - 200 stepping : 12 cpu MHz : 150.342254 fdiv_bug : no hlt_bug : no sep_bug : no f00f_bug : yes coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr mce cx8 bogomips : 59.80
Actually it's a Pentium 150, but it is still kind of fascinating to read what my system thinks of itself. Let's see what happens when we look at the file called "interrupts". Guesses, anyone?
# cat /proc/interrupts CPU0 0: 6422178 XT-PIC timer 1: 43093 XT-PIC keyboard 2: 0 XT-PIC cascade 5: 3 XT-PIC soundblaster 8: 1 XT-PIC rtc 10: 34143 XT-PIC eth0 12: 541052 XT-PIC PS/2 Mouse 13: 1 XT-PIC fpu 14: 1622951 XT-PIC ide0 15: 6 XT-PIC ide1 NMI: 0
Similarly, doing a cat on the meminfo file will give you current stats about your memory, both physical and swap. Looking into /proc/scsi will reveal your SCSI assignments, while a look at the sound file will show you your sound card configuration. A cat of your /proc/partitions file will display your disk partition information, and "version" is your running kernel and how it was compiled. Keep looking. There's a lot here, and it's balls of fun.
By now, you may have noticed I did not tell you the whole truth. When you did an ls -l on /proc, you actually found two non-zero-size files. One of them is called "kcore" and it can be a fairly good size for something that takes up no space. If you really want to look into the heart of your Linux system, listen closely; you can almost hear it beating when you look here. This is a "mirror" of sorts for your kernel. Perhaps the lens of a microscope would be a better analogy. Notice how the file takes up about as much space as you have physical memory. That is no coincidence. kcore is your system's memory - its RAM, if you prefer. In fact, it is everything currently in RAM, and as such, it is a dynamic beast. The same can be said for much of what you see in /proc. Looking at "meminfo" from one minute to the next will yield a different set of numbers as the demands of real memory vs swap are handled.
The other "big" file (though much smaller) is something called "self", and if you do an ls -l on /proc, you'll notice that self is actually a pointer to a number. Here's what I get when I do that ls -l right now:
lr>wxrwxrwx 1 root root 64 Aug 16 13:28 self -> 2183
The reason I asked you to ls -l the whole /proc directory rather than just "self" is this: I wanted you to see that the number (in my case, 2183) is also a directory under the /proc directory. Now, do another ls -l. It's gone, isn't it?
What about all those numbers? What are they for? Each one seems to be a directory. What will you find there? Do this: try a ps ax to get a list of the running processes on your system. Even better, do it this way:
ps ax | cut -c1-5
That will get you just the process numbers without the long display. Now, compare that list to the numbers sitting in /proc. Look familiar? Every process currently running on your system has an analog sitting in the /proc filesystem. Let's have a look at what we can discover hidden in there. Just for fun, let's take one of the processes on my system. For this little exploring about, I am using a terminal emulator called Eterm. If you don't have it, get it from http://www.eterm.org. You'll thank me later. Sure, it's eye candy, but a little bit of eye candy isn't all bad, right? (There I go, wandering off again. Sorry.)
The process ID for one of my Eterm sessions is 834 (the result of ps ax | grep Eterm). If I cd to /proc/834 and do an ls -l, this is what I see:
total 0 -r--r--r-- 1 mgagne mgagne 0 Aug 16 13:59 cmdline lrwx------ 1 mgagne mgagne 0 Aug 16 13:59 cwd -> r/share/Eterm/themes/Eterm -r-------- 1 mgagne mgagne 0 Aug 16 13:59 environ lrwx------ 1 mgagne mgagne 0 Aug 16 13:59 exe -> r/bin/Eterm-0.8.10 dr-x------ 2 mgagne mgagne 0 Aug 16 13:59 fd pr--r--r-- 1 mgagne mgagne 0 Aug 16 13:59 maps -rw------- 1 mgagne mgagne 0 Aug 16 13:59 mem lrwx------ 1 mgagne mgagne 0 Aug 16 13:59 root -> / -r--r--r-- 1 mgagne mgagne 0 Aug 16 13:59 stat -r--r--r-- 1 mgagne mgagne 0 Aug 16 13:59 statm -r--r--r-- 1 mgagne mgagne 0 Aug 16 13:59 status
Once again, a number of these can be viewed with cat or more or less. The cmdline file shows the command that executed Eterm, while environ displays the environment variables at work here. Check out maps for a list of all the libraries at work with the current executable. Look inside that fd directory, and see if you can figure out what those numbers represent.
Sure, there are system commands that do a fine job of showing what is happening in /proc. The command top, for instance, is just such a snapshot for your processes. Files such as meminfo and pci can be graphically unwound with the help of KDE's kcontrol program. Just check out the Information tab. If you look long and hard enough, you will likely find a tool to display all this stuff in a nice, graphical format. In some ways, that's part of the reasoning behind the /proc filesystem: to give programmers/administrators/etc. an easier means of accessing core system information and modifying a running kernel. Get the tools later. Look around now. It's a kind of techno adventure, where the prize is a deeper understanding of what happens under your system's "hood". Yes, Bobby, that's the same hood that's not welded shut.
Did I just say modifying a running kernel? Hmmm... that sounds like a good place to stop and get you ready for something brand-new when next we meet here at the SysAdmin's Corner. (Ain't I a stinker?) Until next time, always remember: your Linux system is talking to you. Are you listening?
Practical books for the most technical people on the planet. Newly available books include:
- Agile Product Development by Ted Schmidt
- Improve Business Processes with an Enterprise Job Scheduler by Mike Diehl
- Finding Your Way: Mapping Your Network to Improve Manageability by Bill Childers
- DIY Commerce Site by Reven Lerner
Plus many more.
- diff -u: What's New in Kernel Development
- Server Hardening
- Giving Silos Their Due
- 22 Years of Linux Journal on One DVD - Now Available
- What's New in 3D Printing, Part III: the Software
- Controversy at the Linux Foundation
- Don't Burn Your Android Yet
- Firefox OS
- February 2016 Issue of Linux Journal