PostgreSQL Performance Tuning

Tweak your hardware to get the most from this open-source database.
How Big Is Too Big?

You may think, “I will just give all my RAM to the PostgreSQL shared buffer cache.” However, if you do that, there will be no room for the kernel, or for any programs, to run. The proper size for the PostgreSQL shared buffer cache is the largest useful size that does not adversely affect other activity.

To understand adverse activity, you need to understand how UNIX operating systems manage memory. If there is enough memory to hold all programs and data, little memory management is required. However, if everything doesn't fit in RAM, the kernel starts forcing memory pages to a disk area called swap. It moves pages that have not been used recently. This operation is called a swap pageout. Pageouts are not a problem because they happen during periods of inactivity. What is bad is when these pages have to be brought back in from swap, meaning an old page that was moved out to swap has to be moved back into RAM. This is called a swap pagein. This is bad because while the page is moved from swap, the program is suspended until the pagein is complete.

Pagein activity is shown by system analysis tools like vmstat and sar and indicates there is not enough memory available to function efficiently. Do not confuse swap pageins with ordinary pageins, which may include pages read from the filesystem as part of normal system operation. If you can't find swap pageins, many pageouts is a good indicator you are also doing swap pageins.

Effects of Cache Size

You may wonder why cache size is so important. First, imagine the PostgreSQL shared buffer cache is large enough to hold an entire table. Repeated sequential scans of the table will require no disk access because all the data is already in the cache. Now imagine the cache is one block smaller than the table. A sequential scan of the table will load all table blocks into the cache until the last one. When that block is needed, the oldest block is removed, which in this case is the first block of the table. When another sequential scan happens, the first block is no longer in the cache, and to load it in, the oldest block is removed, which in this case is now the second block in the table. This pushing out of the next needed block continues to the end of the table. This is an extreme example, but you can see that a decrease of one block can change the efficiency of the cache from 100% to 0%. It shows that finding the right cache size can dramatically affect performance.

Proper Sizing of Shared Buffer Cache

Ideally, the PostgreSQL shared buffer cache will be large enough to hold most commonly accessed tables and small enough to avoid swap pagein activity. Keep in mind that the postmaster allocates all shared memory when it starts. This area stays the same size even if no one is accessing the database. Some operating systems pageout unreferenced shared memory, while others lock shared memory into RAM. The PostgreSQL 7.2 Administrator's Guide has information about kernel configuration for various operating systems (www.postgresql.org/devel-corner/docs/admin/kernel-resources.html).

Sort Memory Batch Size

Another tuning parameter is the amount of memory used for sort batches. When sorting large tables or results, PostgreSQL will sort them in parts, placing intermediate results in temporary files. These files are then merged and resorted until all rows are sorted. Increasing the batch size creates fewer temporary files and often allows faster sorting. However, if the sort batches are too large, they cause pageins because parts of the sort batch get paged out to swap during sorting. In these cases, it is much faster to use smaller sort batches and more temporary files, so again, swap pageins determine when too much memory has been allocated. Keep in mind that this parameter is used for every backend performing a sort, either for ORDER BY, CREATE INDEX or for a merge join. Several simultaneous sorts will use several times this amount of memory.

Cache Size and Sort Size

Both cache size and sort size affect memory usage, so you cannot maximize one without affecting the other. Keep in mind that cache size is allocated on postmaster startup, while sort size varies depending on the number of sorts being performed. Generally, cache size is more significant than sort size. However, certain queries that use ORDER BY, CREATE INDEX or merge joins may see increases in speed with larger sort batch sizes.

Also, many operating systems limit how much shared memory can be allocated. Increasing this limit requires operating system-specific knowledge to either recompile or reconfigure the kernel. More information can be found in the PostgreSQL 7.1 Administrator's Guide (www.postgresql.org/docs/admin/kernel-resources.html).

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Found this article useful

Anonymous's picture

I found this updated PostgreSQL Tuning article very useful.

This guide was very helpful.

Anonymous's picture

This guide was very helpful. Please ignore the ignorant posts saying otherwise. I'd be scared to have those people managing my data.

A brief overview for database system

Breno Leitao's picture

This article is performance tunning for newbie, nothing to be used as day-a-day book. But one question is important, no one that must tune the pgsql is really newbie about db system. ;-(

Breno Leitao

Re: PostgreSQL Performance Tuning

Anonymous's picture

Hello,
After learning this great article which provides the reader with some very useful and basic understandings of the data path in postgreSQL backend, I must say I have been a little shock by the comments I saw from people who are just waiting others to do what they should also be trying to do. These kinds of comments do not look like an encouragment to Bruce who deserves a lot of credit for the simplicity of his writing, and above all for his willingness toi help others.
Thanks Bruce, your article help explain easily some basic backgrounds required to tune wisely a PostgreSQL database.

Paul

Re: PostgreSQL Performance Tuning

Anonymous's picture

HELLO! CAN SOMEONE TELL ME WHAT THIS ARTICLE IS ABOUT. I EXPECTED TO OPTIMIZE MY DATABASE AFTER READING THIS ARTICLE BUT IT SEEMS THAT I WAS BETTER-OFF WITHOUT READING IT!!!!

Re: PostgreSQL Performance Tuning

Anonymous's picture

I was hoping to learn what the vacuum and vacuum analyze commands do, maybe read about the performance benefits of using them or find out other methods to make my queries run faster. I guess the guy just wanted to advertise his book. Thanks for nothing.

Re: PostgreSQL Performance Tuning

Anonymous's picture

not only is not really about PostgreSQL specifically, but it also has incorrect links.

Re: PostgreSQL Performance Tuning

Anonymous's picture

This document should be called General Overview of Performance Tuning. It provides almost no technical detail or insight into the tuning process. I would like to see an article which delves deeper into the issue and provides some real numbers from real applications on some real hardware. After reading the article, one is no more enabled to do any tuning on a PostgreSQL DB than if one didn't read the article at all. I am disappointed.

Re: PostgreSQL Performance Tuning

Anonymous's picture

This is a test from mso without cookies.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix