Tips for Optimizing Linux Memory Usage

In a previous issue, Jeff discussed ways to reduce disk space usage under Linux. In this sequel article, he shows some useful techniques for making the best use of another valuable resource—memory.
Introduction

Like most Unix-compatible operating systems, the single most important factor in determining the performance you get out of Linux is often the amount of physical memory available. This is often a source of confusion to users accustomed to other systems such as MS-DOS. Since many Linux users are on a tight budget, the option of simply purchasing more memory is not always feasible. This article presents some ways in which you can make better use of the memory you already have.

Background

Linux implements a demand-paged virtual memory system. Processes have a large (4 gigabyte) virtual memory space. As virtual memory is referenced, the appropriate pages are transferred between disk and physical memory.

When there are no more physical memory pages available, the kernel swaps some older pages back to disk. (If they are code pages that have not been changed, then they are just discarded; otherwise they are written to the swap areas.)

Disk drives are mechanical devices; reading and writing to disk is several orders of magnitude slower than accessing physical memory. If the total memory pages required significantly exceed the physical memory available, the kernel starts spending more time swapping pages than executing code. The system begins thrashing, and slows down to a crawl. If this increases to a point where the swap device becomes fully utilized, the system can virtually come to a standstill. This is definitely a situation we want to avoid.

When extra physical memory is not in use, the kernel attempts to put it to work as a disk buffer cache. The disk buffer stores recently accessed disk data in memory; if the same data is needed again it can be quickly retrieved from the cache, improving performance. The buffer grows and shrinks dynamically to use the memory available, although priority is given to using the memory for paging. Thus, all the memory you have is put to good use.

Tools for Measuring Memory Utilization

In order to know what your memory situation is and whether any changes you make are resulting in improvement, you need to have some way of measuring memory usage. What tools do we have at our disposal?

When the system first boots, the ROM BIOS typically performs a memory test. You can use this to identify how much physical memory is installed (and working) in your system, if you don't know already. On my system, it looks something like this:

ROM BIOS (C) 1990
008192 KB OK WAIT......

The next piece of useful information is displayed during the Linux boot process. Output such as the following should be displayed:

Memory: 7100k/8192k available (464k
kernel code, 384k reserved, 244k data) ...
Adding Swap: 19464k swap-space

This shows the amount of RAM available after the kernel has been loaded into memory (in this case 7100K out of the original 8192K). You can also see if the swap space has been properly enabled. If the kernel bootup messages scroll by too quickly to read, on many systems you can recall them at a later time using the “dmesg” command.

Once Linux is running, the “free” command is useful for showing the total memory available (which should match that shown during boot-up), as well as a breakdown showing the amount of memory being used, and the amount free. (If you don't have a “free” command, you can use “cat /proc/meminfo”.) Both physical memory and swap space is shown. Here is a typical output on my system:

Here is a typical output on my system:

total

used

free

shared

buffers

Mem:

7096

5216

1880

2328

Swap:

19464

0

19464

2800

The information is shown in kilobytes (1024 bytes). The “total” memory is the amount available after loading the kernel. Any memory being used for processes or disk buffering is listed as “used.” Memory that is currently unused is listed in the “free” column. Note that the total memory is equal to the sum of the “used” and “free” columns.

The memory indicated as “shared” is an indication of how much memory is common to more than one process. A program such as the shell typically has more than one instance running. The executable code is read-only and can be shared by all processes running the shell.

The “buffers” entry indicates how much of the memory in use is currently being used for disk buffering.

The “free” command also shows very clearly whether the swap space is enabled, and how much swapping is going on.

To better understand how the kernel uses memory, it is instructive to watch the output of the “free” command as the system is used. I'll show some examples taken from my own system; I suggest you try similar experimentation yourself.

On bootup, with one user logged in, my system reports the following:

        total   used    free    shared  buffers
Mem:    7096    2672    4424    1388    1136
Swap:   19464   0       19464

Note that we have considerable free memory (4.4MB) and a relatively small disk buffer (1.1MB). Now watch how the situation changes after running a command that reads data from the disk. (In this case I typed ls -lR /.)

        total   used    free    shared  buffers
Mem:    7096    5104    1992    1396    3460
Swap:   19464   0      19464

We see that the disk buffer has grown by over 2 MB. This brings the “used” memory up correspondingly, and the free memory down. Next, I start up the X Window system and examine the results:

        total   used    free    shared  buffers
Mem:    7096    7016    80      3112    3792
Swap:   19464   8       19456

This has caused the memory used to increase to 7MB, leaving only 80K free. The increase is to support the additional processes running (the X server, window manager, xterm, etc...). Note that the disk buffer didn't shrink, because there is still free memory. Remember: “free” memory means memory that is being wasted.

Now I start up the GNU chess program, having it play against itself. This starts two instances of a rather large program:

        total   used    free    shared   buffers
Mem:    7096    7016    80      1080    860
Swap:   19464   5028    14436

We see now that the disk buffer has shrunk down to less than 1MB and we are 5MB into swap to accommodate the large processes. Because of the swapping, the system has slowed down, and heavy disk drive activity can be heard. There is still a small amount of free memory. (The kernel tries to prevent user processes from taking all of the available memory; it reserves some for the “root” user only.)

The next step is to exit the X Window system and the applications running under it; here is the result.

        total   used    free    shared  buffers
Mem:    7096    2444    4652    412     1480
Swap:   19464   728     18736

We now have lots of free memory, the swap usage is almost gone (some idle programs are still presumably swapped out), and the disk buffer is starting to grow again.

The “top” and “ps” commands are also very useful for showing how memory usage changes dynamically, and how individual processes are using memory. For the scenario described earlier, we can see from the output of “ps” that each of the two chess processes was taking almost 8MB of virtual memory, obviously more than could fit in physical memory, causing the system to thrash.

USER  PID %CPU %MEM SIZE  RSS TTY STAT START   TIME COMMAND ...
tranter   282  4.1 34.4 7859 2448 v01 D  14:08 0:11 gnuchessx 40 5
tranter   285  7.9 30.7 7859 2180 v01 D  14:09 0:21 gnuchessx 40 5
...

Another facility for getting system status information is built into the virtual console driver. This depends on your keyboard mapping, but the default for the US keyboard is to use the Scroll-Lock key. Pressing <Alt><Scroll Lock> shows the current value of the CPU registers. The <Shift><Scroll Lock> combination shows memory information, similar to the “free” command, but more detailed. Finally, <Ctrl><Scroll Lock> will give information on individual processes, much like the “ps” command.

These keys can be particularly handy if your system is slow, or appears to have crashed. Note that if you are running the syslog daemon, this information will probably be logged to a file instead of being displayed on the console. On my Slackware system for example, it is logged to the file /var/adm/syslog.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Buffer+Cache grows with time and is not freed when nedded

Vikash's picture

This is wrt a telecom hardware that is supposed to support some no of calls/UEs. It has a Cavium Octeon processor and an SMP LINUX by WindRiver systems.
Now, when I admit a new call, the used grows (okay), after some 50 UEs are introduced, the system is left with some 40 K of memory and the buffer and cache part has grown so much.
TOTAL USED FREE Buf/Cache
15:28:40 IST 2010 Mem: 1383 702 680 0 0 141
15:45:04 IST 2010 Mem: 1383 965 417 0 0 328
16:06:31 IST 2010 Mem: 1383 1341 41 0 2 693

Isnt the LINUX kernel supposed to free the buffer/cached part when the need for memory needed by new application come up..?
Please someone help me here. Any other info needed, pls let me know.

-Vikash

Need to tune memory

Anonymous's picture

cache memory is getting full very often. server is getting into crashing state. Nothing written in the swap space.

need ideas to tune the memory.

RAM 64G
swap 2 G

...

JShuford's picture

Thank you...

...I'm not just a "troll", but also a subscriber!

nice article. It will be

Manish Madhukar's picture

nice article. It will be great if you also explain the difference between cache and buffers as well...

-bash-3.2$ free -m
total used free shared buffers cached
Mem: 15364 6738 8625 0 211 4011
-/+ buffers/cache: 2514 12849
Swap: 12001 0 12001

Regards
Manish

Its really nice to see the

Madhusudhanan's picture

Its really nice to see the article written in 1994 still helps us, but things have slightly changed a bit since then.

Regarding your query,
The first row, labeled Mem, displays physical memory utilization, including the amount of memory allocated to buffers and caches. A buffer, also called buffer memory, is usually defined as a portion of memory that is set aside as a temporary holding place for data that is being sent to or received from an external device, such as a HDD, keyboard, printer or network.

The second line of data, which begins with -/+ buffers/cache, shows the amount of physical memory currently devoted to system buffer cache. This is particularly meaningful with regard to application programs, as all data accessed from files on the system that are performed through the use of read() and write() system calls1 pass through this cache. This cache can greatly speed up access to data by reducing or eliminating the need to read from or write to the HDD or other disk.

The third row, which begins with Swap, shows the total swap space as well as how much of it is currently in use and how much is still available.

CREDIT: http://www.linfo.org/free.html

- then again...

omelette's picture

... turns out that it was the linux kernel itself that was to blame;

http://www.intellinuxwireless.org/bugzilla/show_bug.cgi?id=1716

let's be careful out there... :)

things are not so clear-cut...

omelette's picture

Nice article, unfortunately like the dozens of others in a similar vein one can google, they all imply that the 'seekers' lack of knowledge of what is discussed above is all that's at issue - imo, this is not (always!) the case, as some (many?) programs leak memory like a sieve! For instance, at this moment my Debian lenny KDE ProDuo 1Gig laptop, up for 6-8hrs but with all apps closed produces the following memory-footprint - again, there's NOTHING running apart form the desktop:

omelette@debian:~$ free
total used free shared buffers cached
Mem: 1034636 956996 77640 0 3944 153828
-/+ buffers/cache: 799224 235412
Swap: 1052216 84856 967360

One reboot later, here is the results:

omelette@debian:~$ free
total used free shared buffers cached
Mem: 1034636 167316 867320 0 10788 88568
-/+ buffers/cache: 67960 966676
Swap: 1052216 0 1052216

Some difference! - clearly, 400-500meg of memory is not being released by apps and linux seems completely oblivious to this! Note, running any decent-sized app in this out-of-memory state sees swap grow proportionally, so linux definitely believes it has run out of memory! And this is not down to Debian or a specific kernel either - I have just transferred over from Ubuntu which performs abysmally as well (far worse it appears, though this may be 'cos Gnome uses more resources than KDE, so it's more noticeable) Ironically, I tested Debian extensively for one day on an 'old' Athlon-based computer, where it performed magnificently, before sticking it on the laptop as well. I'm now starting to wonder might it have something to do with dual-processors... BTW, I'm not out to bash linux here - I have Ubuntu running continuously as a server of sorts on a really old 700mhz Celeron without problems. Anyway, as 'top' is just-about useless in resolving this, I've just installed 'valgrind' in the hope that it will somehow shed some light on the problem (if I can ever figure out how to use it...)

Very useful

Pravin's picture

Nice Article. I was wondering how free memory is so less on my system and large memory is shown in buffers? This article cleared it.

The total of free is always

jithu's picture

The total of free is always less that the amount of physical memory (RAM) actually present on the machine. Why is it ?

you have told The “total” memory is the amount available after loading the kernel. so is the difference the space that is occupied by kernel and its data structures like page table etc ?

Nice article but would be

Anonymous's picture

Nice article but would be good if you explained how to do some of these things - particularly how to stop processes from running at startup like reducing the number of getty's etc.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix