Stack Backtracing Inside Your Program
At this point, we have in hand a tool that is able to print the list of function calls up to the current execution point. This can be a useful tool in many different contexts. Think of having a complex program and needing to know who's calling a given function with the wrong parameters. With a simple check and a call to our show_stackframe() function, the faulty caller can be spotted easily.
An even more useful application for this technique is putting a stack backtrace inside a signal handler and having the latter catch all the "bad" signals your program can receive (SIGSEGV, SIGBUS, SIGILL, SIGFPE and the like). This way, if your program unfortunately crashes and you were not running it with a debugger, you can get a stack trace and know where the fault happened. This technique also can be used to understand where your program is looping in case it stops responding. All you need to do is set up a SIGUSR1/2 handler and send such a signal when needed. Before presenting an example, we need to open a parenthesis on signal handling.
Backtracing from within a signal handler requires some interesting intricacies that take us on a little detour through signal delivery to processes. Going into deep detail on this matter is outside the scope of this article, but we briefly can summarize it this way:
When the kernel needs to notify a signal of a given process, it prepares some data structures attached to the process' task struct and sets a signal-pending bit.
Later on, when the signalee process is scheduled for execution, its stack frame is altered by the kernel in order to have EIP point to the process' signal handler. This way, when the process runs it behaves as if it had called its own signal handler by itself before being suspended.
The initial steps of user space signal management are taken care of inside libc, which eventually calls the real process' signal handling routines which, in turn, execute our stack backtrace function.
As a consequence of this mechanism, the first two entries in the stack frame chain when you get into the signal handler contain, respectively, a return address inside your signal handler and one inside sigaction() in libc. The stack frame of the last function called before the signal (which, in case of fault signals, also is the one that supposedly caused the problem) is lost. Thus, if function B called function A, which in turn caused a SIGSEGV, a plain backtrace would list these entry points:
your_sig_handler() sigaction() in libc.so func_B() main()
and no trace of the call to function A would be found. For more details, have a look at the manuals for signal() and sigaction().
In order to get a meaningful backtrace, we need a workaround. Luckily, when you have the sources of both the kernel and libc, you can find a workaround for nearly anything. In Listing 2 we exploit an undocumented parameter of type sigcontext that is passed to the signal handler (see the UNDOCUMENTED section in man sigaction) and contains, among other things, the value of EIP when the signal was raised. After the call to backtrace(), we use this value to overwrite the useless entry corresponding to the sigaction() return address in the trace array. When we later call backtrace_symbols(), the address we inserted is resolved the same as any other entry in the array. Finally, when we print the backtrace, we start from the second entry (i=1 in the loop), because the first one always would be inside our signal handler.
Since kernel version 2.2 the undocumented parameter to the signal handler has been declared obsolete in adherence with POSIX.1b. A more correct way to retrieve additional information is to use the SA_SIGINFO option when setting the handler, as shown in Listing 3 and documented in the man page. Unfortunately, the siginfo_t structure provided to the handler does not contain the EIP value we need, so we are forced to resort again to an undocumented feature: the third parameter to the signal handler. No man page is going to tell you that such a parameter points to an ucontext_t structure that contains the values of the CPU registers when the signal was raised. From this structure, we are able to extract the value of EIP and proceed as in the previous case.
Today’s modular x86 servers are compute-centric, designed as a least common denominator to support a wide range of IT workloads. Those generic, virtualized IT workloads have much different resource optimization requirements than hyperscale and cloud applications. They have resulted in a “one size fits all” enterprise IT architecture that is not optimized for a specific set of IT workloads, and especially not emerging hyperscale workloads, such as web applications, big data, and object storage. In this report, you will learn how shifting the focus from traditional compute-centric IT architectures to an innovative disaggregated fabric-based architecture can optimize and scale your data center.
Sponsored by AMD
Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6
Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.
Learn more about catching the bad guy in this free white paper.
Sponsored by DLT Solutions
| Using Salt Stack and Vagrant for Drupal Development | May 20, 2013 |
| Making Linux and Android Get Along (It's Not as Hard as It Sounds) | May 16, 2013 |
| Drupal Is a Framework: Why Everyone Needs to Understand This | May 15, 2013 |
| Home, My Backup Data Center | May 13, 2013 |
| Non-Linux FOSS: Seashore | May 10, 2013 |
| Trying to Tame the Tablet | May 08, 2013 |
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- Validate an E-Mail Address with PHP, the Right Way
- Drupal Is a Framework: Why Everyone Needs to Understand This
- A Topic for Discussion - Open Source Feature-Richness?
- Home, My Backup Data Center
- New Products
- RSS Feeds
- Tech Tip: Really Simple HTTP Server with Python
- Epistle
13 min 9 sec ago - Automatically updating Guest Additions
1 hour 21 min ago - I like your topic on android
2 hours 8 min ago - Reply to comment | Linux Journal
2 hours 29 min ago - This is the easiest tutorial
8 hours 43 min ago - Ahh, the Koolaid.
14 hours 22 min ago - git-annex assistant
20 hours 22 min ago - direct cable connection
20 hours 44 min ago - Agreed on AirDroid. With my
20 hours 54 min ago - I just learned this
20 hours 58 min ago
Enter to Win an Adafruit Prototyping Pi Plate Kit for Raspberry Pi

It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Prototyping Pi Plate Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- Next winner announced on 5-21-13!
Free Webinar: Linux Backup and Recovery
Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.
In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.



Comments
Hi, Excellent article. Is
Hi,
Excellent article. Is there a way to resolve addresses to symbols when the executable is statically linked. It should be possible by reading through elf section i guess. Any pointers pls ?
m68k hacker
on linux it works well, like all other stuff.
better tell me how to do this on m68k, guru!
kernel function like backtrace
Hi,
The function backtrace() is a user space function, does there exist kernel function like that?
I'm modifying some kernel modules(.ko), but it still have bugs :-( .After reading this, I want to
print my stack frame. But I don't know how. Looking forward to your advice.
Thanks.
lineno is off by one
This is helpful.
However, after passing the address to addr2line, it give line no of the next line of the calling statement.
So, in the same example, it gives does not give line no of "func_high()". Instead, it give line no of "return 2*p1;"
Any ideas to get correct line no as well?
Using 3rd arg in sigcontext doesn't work on ARM
Thanks for your article. V. helpful indeed.
I'm trying to get this method to work on ARM running embedded linux (running glibc-2.3.2... I am trying to get the PC/EIP from typecasting the 3rd argument of the signal handler as ucontext_t and finding the appropriate arch-dependent register. However, in ARM none of the registers seem to contain anything that looks like the program counter (not even R15, which should actually be the PC). So, I'm unable to find out where the exact crash happened. Any thoughts/ideas?
Thanks,
M.
backtrace_symbols for ARM
I appriciate the great work that people have put up on this page.I am having a serious problem with getting the call stack information on ARM. I have tried same kind of code, which works fine for linux on x386 with -rdynamic option of gcc but fails on ARM. It would be great if you could help me on this.
Are there any specific options for compiling this on ARM,if YES then what is that option?.
Thanks in advance,
Sharan.
Re: Stack Backtracing Inside Your Program
My problem is to backtrace on a powerPC based Linux platform.
Where can I find address of interrupted program?
powerPC has no eip, nor program counter at all ...
Thank you.
I have tried this and it work
I have tried this and it works great, but now I have no core files generated. Any ideas on how to fix?
Thanks.
Re: Stack Backtracing Inside Your Program
on linux it works well, like all other stuff.
better tell me how to do this on tru64, guru!
Re: Stack Backtracing Inside Your Program
backtrace is neat when it works.
It still lacks many features you get for free in gdb,
or in the Win32 StackWalk facilities:
. It can't seem to properly convert addresses to names
when dealing with static functions.
. It doesn't demangle C++ names, which obliges to
call the obscure and ill-documented __cxa_demangle
. It can't get you your local variables
. It can't get you source references (file+line no)
Re: Stack Backtracing Inside Your Program
main()
{
printf(get the value of i=1,i<=5,i=1)
what will be reasulte
Re: Stack Backtracing Inside Your Program
main()
{
printf(get the value of i=1,i<=5,i=1)
what will be reasulte
interested
same question here. really wondering if you can answer that, guru?
What about locals -vs- arguments?
Is there any easy way to display the rest of the local frame? I'd love to be able to get at the parameters passed. I could just dump the memory added to the stack before the address, but I don't know how far to go.
Re: Stack Backtracing Inside Your Program
You can access the stack in C, but it is still true that some platform-specific knowledge is needed to make sense of it. Just take the address of an argument or automatic variable and work from there:
void func( int arg )
{
void *stackframe = &arg;
...
}
Stack arguments ot a backtrace
I'd appreciate if you could detail the stack argument access a little better. I'm written a wxCrashPrint component for the wxWidgets (wxWindows) framework (here) and would like to show arguments as well.
Re: Stack Backtracing Inside Your Program
Very cool!
I can see where this could help in situations where bugs disappear when debug mode is turned on.
Re: Stack Backtracing Inside Your Program
printf in a signal handler? All y'all are just asking for trouble in the future.