Monitoring Virtual Memory with vmstat
Linux novices often find virtual memory mysterious, but with a grasp of the fundamental concepts, it's easy to understand. With this knowledge, you can monitor your system's memory utilization using vmstat and detect problems that can adversely affect system performance.
Physical memory—the actual RAM installed—is a finite resource on any system. The Linux memory handler manages the allocation of that limited resource by freeing portions of physical memory when possible.
All processes use memory, of course, but each process doesn't need all its allocated memory all the time. Taking advantage of this fact, the kernel frees up physical memory by writing some or all of a process' memory to disk until it's needed again.
The kernel uses paging and swapping to perform this memory management. Paging refers to writing portions, termed pages, of a process' memory to disk. Swapping, strictly speaking, refers to writing the entire process, not just part, to disk. In Linux, true swapping is exceedingly rare, but the terms paging and swapping often are used interchangeably.
When pages are written to disk, the event is called a page-out, and when pages are returned to physical memory, the event is called a page-in. A page fault occurs when the kernel needs a page, finds it doesn't exist in physical memory because it has been paged-out, and re-reads it in from disk.
Page-ins are common, normal and are not a cause for concern. For example, when an application first starts up, its executable image and data are paged-in. This is normal behavior.
Page-outs, however, can be a sign of trouble. When the kernel detects that memory is running low, it attempts to free up memory by paging out. Though this may happen briefly from time to time, if page-outs are plentiful and constant, the kernel can reach a point where it's actually spending more time managing paging activity than running the applications, and system performance suffers. This woeful state is referred to as thrashing.
Using swap space is not inherently bad. Rather, it's intense paging activity that's problematic. For instance, if your most-memory-intensive application is idle, it's fine for portions of it to be set aside when another large job is active. Memory pages belonging to an idle application are better set aside so the kernel can use physical memory for disk buffering.
vmstat, as its name suggests, reports virtual memory statistics. It shows how much virtual memory there is, how much is free and paging activity. Most important, you can observe page-ins and page-outs as they happen. This is extremely useful.
To monitor the virtual memory activity on your system, it's best to use vmstat with a delay. A delay is the number of seconds between updates. If you don't supply a delay, vmstat reports the averages since the last boot and quit. Five seconds is the recommended delay interval.
To run vmstat with a five-second delay, type:
You also can specify a count, which indicates how many updates you want to see before vmstat quits. If you don't specify a count, the count defaults to infinity, but you can stop output with Ctrl-C.
To run vmstat with ten updates, five seconds apart, type:
vmstat 5 10
Here's an example of a system free of paging activity:
procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 29232 116972 4524 244900 0 0 0 0 0 0 0 0 0 0 0 0 29232 116972 4524 244900 0 0 0 0 2560 6 0 1 99 0 0 0 29232 116972 4524 244900 0 0 0 0 2574 10 0 2 98
All fields are explained in the vmstat man page, but the most important columns for this article are free, si and so. The free column shows the amount of free memory, si shows page-ins and so shows page-outs. In this example, the so column is zero consistently, indicating there are no page-outs.
The abbreviations so and si are used instead of the more accurate po and pi for historical reasons.
Here's an example of a system with paging activity:
procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id . . . 1 0 0 13344 1444 1308 19692 0 168 129 42 1505 713 20 11 69 1 0 0 13856 1640 1308 18524 64 516 379 129 4341 646 24 34 42 3 0 0 13856 1084 1308 18316 56 64 14 0 320 1022 84 9 8
Notice the nonzero so values indicating there is not enough physical memory and the kernel is paging out. You can use top and ps to identify the processes that are using the most memory.
You also can use top to show memory and swap statistics. Here is an example of the uppermost portion of a typical top report:
14:23:19 up 348 days, 3:02, 1 user, load average: 0.00, 0.00, 0.00 55 processes: 54 sleeping, 1 running, 0 zombie, 0 stopped CPU states: 0.0% user, 2.4% system, 0.0% nice, 97.6% idle Mem: 481076K total, 367508K used, 113568K free, 4712K buffers Swap: 1004052K total, 29852K used, 974200K free, 244396K cached
For more information about top, see the top man page.
|Designing Electronics with Linux||May 22, 2013|
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
- Designing Electronics with Linux
- New Products
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Validate an E-Mail Address with PHP, the Right Way
- Tech Tip: Really Simple HTTP Server with Python
- Build a Skype Server for Your Home Phone System
- Why Python?
- A Topic for Discussion - Open Source Feature-Richness?
- Reply to comment | Linux Journal
46 min ago
- Not free anymore
4 hours 47 min ago
8 hours 35 min ago
- Reply to comment | Linux Journal
8 hours 43 min ago
- Understanding the Linux Kernel
10 hours 57 min ago
13 hours 27 min ago
- Kernel Problem
23 hours 30 min ago
- BASH script to log IPs on public web server
1 day 3 hours ago
1 day 7 hours ago
- Reply to comment | Linux Journal
1 day 8 hours ago
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?