Stack Backtracing Inside Your Program

How to use a backtrace to follow the execution path and find out what went wrong and where.
One Step Farther

At this point, we have in hand a tool that is able to print the list of function calls up to the current execution point. This can be a useful tool in many different contexts. Think of having a complex program and needing to know who's calling a given function with the wrong parameters. With a simple check and a call to our show_stackframe() function, the faulty caller can be spotted easily.

An even more useful application for this technique is putting a stack backtrace inside a signal handler and having the latter catch all the "bad" signals your program can receive (SIGSEGV, SIGBUS, SIGILL, SIGFPE and the like). This way, if your program unfortunately crashes and you were not running it with a debugger, you can get a stack trace and know where the fault happened. This technique also can be used to understand where your program is looping in case it stops responding. All you need to do is set up a SIGUSR1/2 handler and send such a signal when needed. Before presenting an example, we need to open a parenthesis on signal handling.

Signal Handling and Stack Frames

Backtracing from within a signal handler requires some interesting intricacies that take us on a little detour through signal delivery to processes. Going into deep detail on this matter is outside the scope of this article, but we briefly can summarize it this way:

  • When the kernel needs to notify a signal of a given process, it prepares some data structures attached to the process' task struct and sets a signal-pending bit.

  • Later on, when the signalee process is scheduled for execution, its stack frame is altered by the kernel in order to have EIP point to the process' signal handler. This way, when the process runs it behaves as if it had called its own signal handler by itself before being suspended.

  • The initial steps of user space signal management are taken care of inside libc, which eventually calls the real process' signal handling routines which, in turn, execute our stack backtrace function.

As a consequence of this mechanism, the first two entries in the stack frame chain when you get into the signal handler contain, respectively, a return address inside your signal handler and one inside sigaction() in libc. The stack frame of the last function called before the signal (which, in case of fault signals, also is the one that supposedly caused the problem) is lost. Thus, if function B called function A, which in turn caused a SIGSEGV, a plain backtrace would list these entry points:

your_sig_handler()
sigaction() in libc.so
func_B()
main()

and no trace of the call to function A would be found. For more details, have a look at the manuals for signal() and sigaction().

Back to Backtrace

In order to get a meaningful backtrace, we need a workaround. Luckily, when you have the sources of both the kernel and libc, you can find a workaround for nearly anything. In Listing 2 we exploit an undocumented parameter of type sigcontext that is passed to the signal handler (see the UNDOCUMENTED section in man sigaction) and contains, among other things, the value of EIP when the signal was raised. After the call to backtrace(), we use this value to overwrite the useless entry corresponding to the sigaction() return address in the trace array. When we later call backtrace_symbols(), the address we inserted is resolved the same as any other entry in the array. Finally, when we print the backtrace, we start from the second entry (i=1 in the loop), because the first one always would be inside our signal handler.

Listing 2. Using sigcontext

Since kernel version 2.2 the undocumented parameter to the signal handler has been declared obsolete in adherence with POSIX.1b. A more correct way to retrieve additional information is to use the SA_SIGINFO option when setting the handler, as shown in Listing 3 and documented in the man page. Unfortunately, the siginfo_t structure provided to the handler does not contain the EIP value we need, so we are forced to resort again to an undocumented feature: the third parameter to the signal handler. No man page is going to tell you that such a parameter points to an ucontext_t structure that contains the values of the CPU registers when the signal was raised. From this structure, we are able to extract the value of EIP and proceed as in the previous case.

Listing 3. Using the SA_SIGINFO Option

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Hi, Excellent article. Is

Anonymous's picture

Hi,
Excellent article. Is there a way to resolve addresses to symbols when the executable is statically linked. It should be possible by reading through elf section i guess. Any pointers pls ?

m68k hacker

Anonymous's picture

on linux it works well, like all other stuff.
better tell me how to do this on m68k, guru!

kernel function like backtrace

Zhengju Sha's picture

Hi,
The function backtrace() is a user space function, does there exist kernel function like that?

I'm modifying some kernel modules(.ko), but it still have bugs :-( .After reading this, I want to
print my stack frame. But I don't know how. Looking forward to your advice.

Thanks.

lineno is off by one

Vinayak Datar's picture

This is helpful.

However, after passing the address to addr2line, it give line no of the next line of the calling statement.

So, in the same example, it gives does not give line no of "func_high()". Instead, it give line no of "return 2*p1;"

Any ideas to get correct line no as well?

Using 3rd arg in sigcontext doesn't work on ARM

MMysore's picture

Thanks for your article. V. helpful indeed.

I'm trying to get this method to work on ARM running embedded linux (running glibc-2.3.2... I am trying to get the PC/EIP from typecasting the 3rd argument of the signal handler as ucontext_t and finding the appropriate arch-dependent register. However, in ARM none of the registers seem to contain anything that looks like the program counter (not even R15, which should actually be the PC). So, I'm unable to find out where the exact crash happened. Any thoughts/ideas?

Thanks,
M.

backtrace_symbols for ARM

Sharan's picture

I appriciate the great work that people have put up on this page.I am having a serious problem with getting the call stack information on ARM. I have tried same kind of code, which works fine for linux on x386 with -rdynamic option of gcc but fails on ARM. It would be great if you could help me on this.

Are there any specific options for compiling this on ARM,if YES then what is that option?.

Thanks in advance,

Sharan.

Re: Stack Backtracing Inside Your Program

Anonymous's picture

My problem is to backtrace on a powerPC based Linux platform.
Where can I find address of interrupted program?
powerPC has no eip, nor program counter at all ...
Thank you.

I have tried this and it work

Anonymous's picture

I have tried this and it works great, but now I have no core files generated. Any ideas on how to fix?
Thanks.

Re: Stack Backtracing Inside Your Program

Anonymous's picture

on linux it works well, like all other stuff.
better tell me how to do this on tru64, guru!

Re: Stack Backtracing Inside Your Program

Anonymous's picture

backtrace is neat when it works.

It still lacks many features you get for free in gdb,
or in the Win32 StackWalk facilities:

. It can't seem to properly convert addresses to names
when dealing with static functions.

. It doesn't demangle C++ names, which obliges to
call the obscure and ill-documented __cxa_demangle

. It can't get you your local variables

. It can't get you source references (file+line no)

Re: Stack Backtracing Inside Your Program

Anonymous's picture

main()
{
printf(get the value of i=1,i<=5,i=1)

what will be reasulte

Re: Stack Backtracing Inside Your Program

Anonymous's picture

main()
{
printf(get the value of i=1,i<=5,i=1)

what will be reasulte

interested

Anonymous's picture

same question here. really wondering if you can answer that, guru?

What about locals -vs- arguments?

Anonymous's picture

Is there any easy way to display the rest of the local frame? I'd love to be able to get at the parameters passed. I could just dump the memory added to the stack before the address, but I don't know how far to go.

Re: Stack Backtracing Inside Your Program

Anonymous's picture

You can access the stack in C, but it is still true that some platform-specific knowledge is needed to make sense of it. Just take the address of an argument or automatic variable and work from there:
void func( int arg )
{
void *stackframe = &arg;
...
}

Stack arguments ot a backtrace

Otto Wyss's picture

I'd appreciate if you could detail the stack argument access a little better. I'm written a wxCrashPrint component for the wxWidgets (wxWindows) framework (here) and would like to show arguments as well.

Re: Stack Backtracing Inside Your Program

Anonymous's picture

Very cool!

I can see where this could help in situations where bugs disappear when debug mode is turned on.

Re: Stack Backtracing Inside Your Program

Anonymous's picture

printf in a signal handler? All y'all are just asking for trouble in the future.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState