Debugging Memory on Linux

Petr explains how programmers can prevent nasty program memory bugs.
Debugging

Memory can cause bugs and usually unwanted memory behavior. One way is by the usage of freed memory, which is the usage of a memory chunk that the program has already freed. Although this will not necessarily cause problems immediately, something will go wrong once a new memory allocation takes over that same area of memory. As a result, the same memory area is used for two different purposes, which causes unexpected values that may lead to a program core dump if the memory area contains pointer values or offsets.

Another problem is trampling over the preamble to a memory chunk. If the program overwrites the preamble to a memory chunk, the memory management system will possibly fail or act unexpectedly when encountering the corrupted memory chunk.

Sometimes trampling occurs over an adjacent memory chunk, and this might corrupt data. The user might only pick up this kind of error later during program execution with odd values and program behavior.

Similarly, if the management information of a freed memory chunk is wrecked by trampling or unwarranted use, it is highly likely that the memory management system will cause an error.

Usage of the unallocated space in the memory arena could also have an effect. It may be possible to use the memory outside of the heap, which is still within the memory arena. This generally will not cause errors until newly allocated memory uses some of this space. This error could be very difficult to detect because the subsequent memory actions could keep within the heap space.

The most obvious and immediate error is when a program attempts to use memory outside of the memory arena and the program memory scope. This results in a SIGSEGV (segmentation violation fault), and the program will automatically dump core.

The most damaging and trickiest-to-debug memory error is when the stack of the program is corrupted. The program stores local variables, parameters and registers from previous frames and, most importantly, the return address in the stack. So if the stack becomes corrupted, the program may become impossible to debug with a conventional debugger, as the stack frames themselves are rendered useless. Debugging stack memory problems is limited to a few open-source (e.g., libsafe) and proprietary memory debuggers because program execution needs to be altered or enhanced to detect stack memory violations.

There are several ways of attempting to catch and find memory misuses. Unfortunately, some have side effects, such as slower program execution speed and more memory usage, and consequently, they may be unusable in memory-intensive programs.

The buggy program examples used with the following memory debuggers can be seen in Listings 2, 3 and 4.

Listing 2. mytest00.c Example Program

Listing 3. mytest01.c Example Program

Listing 4. mytest02.c Example Program

By default there is an environment variable, MALLOC_CHECK_, that can be set to enable rudimentary debugging with the default malloc. MALLOC_CHECK_ can be set to one, in order to provide some error reporting, or set to two to abort the program whenever any malloc error occurs. The output can be cryptic because the debug mode reports problem areas as addresses rather than readable symbols. As a result, it is a good idea to have a debugger on hand to determine where in the program these errors are occurring. The following is an example using default memory debugging:

<home>$ MALLOC_CHECK_=1 ./mytest00
malloc: using debugging hooks
hello Linux users
free(): invalid pointer 0x80496d0
hello again
free(): invalid pointer 0x80496d0
realloc(): invalid pointer 0x80496d0
malloc: top chunk is corrupt
hello there

The output indicates the problem in mytest00.c, line 8 (Listing 2), where the strcpy() function overflows and corrupts the memory chunk pointed to by msg. The subsequent debugging messages are because of this corruption.

There are several excellent open-source memory tools available (see sidebar for a list). Each implementation differs in memory bug coverage, output and interaction.

Open-Source Memory Tools

Electric Fence is one tool that is simple to use. The library performs several memory checks and when encountering an error, stops the program. This usually results in a core dump, which the user then can investigate with a debugger. Electric Fence is most useful when employed within a debugger, such as the GNU debugger (GDB). When Electric Fence stops the program, GDB regains control at the exact location in the program where the error occurred (see Listing 5).

Listing 5. Memory Debugging with Electric Fence within GDB

This example output shows the test built with the Electric Fence library executing under GDB. The very first violation at mytest00.c line 8 results in a SIGSEGV. When examining the stack trace provided by GDB, the user can identify the problem location.

libsafe is used to check a number of possible stack frame boundary violations limited to a few C functions (strcpy, strcat, getwd, gets, scanf, vscanf, fscanf, realpath, sprintf and vsprintf).

The libsafe example output is terse. As soon as a stack error occurs, libsafe displays an error and terminates the program. However, libsafe sends the details of the actual error to various e-mail recipients. Granted, this is a convoluted way of reporting the error, but users primarily use libsafe to detect attempted security breaches that exploit buffer overflow. With a bit of editing, a developer can enhance the libsafe code to report messages that are more informative. Another option is to execute the program in GDB and set a breakpoint on _libsafe_die(), which is hit as soon as a stack violation is detected by libsafe. In the following example libsafe detects stack trampling caused by strcpy() in line 8 of mytest01.c (Listing 3):

<home>$ LD_PRELOAD=/lib/libsafe.so.1.3  ./mytest01
Detected an attempt to write across stack boundary.
Terminating mytest01.
Null message body; hope that's ok
# Email is the sent with the following subject header
libsafe violation for /tmp/mytest01, pid=27265;
overflow caused by strcpy()

debauch limits its output to contain addresses instead of symbols, which makes it necessary to be used with a debugger. debauch has special capabilities that users can activate specifically for GDB use. These capabilities allow better tracking of memory allocation and deallocation calls. debauch is thorough and detects and recovers from many of the memory errors (see Listing 6).

Listing 6. Memory Debugging with debauch

memprof's main feature is the GUI interface, which makes it easy to understand and to see where memory leaks occur. It has fairly powerful capabilities due to the fact that it utilizes functions that GDB uses to control processes via the binary file descriptor (BFD) library. Figure 3 shows that memprof has detected the leak in the function alloc_two() in mytest02.c.

Figure 3. Memory Debugging with memprof

Apart from open-source memory tools, several proprietary tools are available that provide graphical user interfaces and more thorough checks than open-source versions (see sidebar for a list of proprietary memory tools).

Proprietary Memory Tools

Possibly, the last option is to write your own memory handling functions. This might be useful in becoming familiar with memory management or providing performance enhancement due to your particular needs, such a quick allocation and deallocation of large memory areas.

Debugging memory problems is important, for not only program stability, but security as well. There are several memory debuggers available for Linux, each with their own particular set of capabilities and usage criteria. The best approach is to test a program with more than one of these memory debuggers with a debugger such as GDB, as the combined power may detect a wider range of memory problems.

Petr Sorfa (petrs@sco.com) is member of Santa Cruz Operation's Development Systems Group where is the maintainer of the cscope and Sar3D open-source projects. He has a BSc from the University of Cape Town and a BSC Honours from Rhodes University. His interests include open-source projects, computer graphics, development systems and sequential art (comics).

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

missing "static data"?

Anonymous's picture

should you have on the picture also "static data" ?

Re: Debugging Memory on Linux

Anonymous's picture

Dear Mr Petr,

I was reading your article on 'Debugging Memory on Linux' on Linux

Journal.

Recently we have a problem with memory on our Linux machine with 'out

of memory' crashes on one of our application software. I noticed that

the system admin has changed our stacksize limit to unlimited from

8192k. And ever since, all of us had 'out of memory' errors whenever

we at at about 950+M memory usage. We have a 3G memory for our Linux

machines. The system admin stand was that changing the stacksize to

unlimited should not cause this error. My guess is that setting the

stacksize to unlimited reserved 32bit or 2G of memory, leaving 1G

memory for user. That is why it crashes out of memory. On the other

hand, the system admin also claims that it works fine for Solaris when

the stacksize is set to unlimited. My guess is again on the larger

memory available for Solaris, so reserving 32bit or 2G for stack does

not take away so much resources from the available user memory.

I would appreciate your advice on that. I do not see out of memory

problem due to unlimited stacksize being addressed and would like to

know more about it. I hope you can enlighten me on that.

In addition, will most application software uses more user memory than

kernel memory? Can I say that compilers will need more of kernel

memory?

Thanks in advance and best regards,

SS

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState