Thread-Specific Data and Signal Handling in Multi-Threaded Applications

Here are the answers to questions about signal handling and taking care of global data when writing multi-threaded programs.

Perhaps the two most common questions I'm asked about multi-threaded programming (after “what is multi-threaded programming?” and “why would you want to do it?”) concern how to handle signals, and how to handle cases where two concurrent threads use a common function that makes use of global data, and yet the two threads need thread-specific data from that function. By definition, global data includes static local variables which are in truth a kind of global variable. In this article I'll explain how these questions can be dealt with in C programs using one of the POSIX (or almost POSIX) multi-threading packages available for Linux. I live in hope of the day when the most common question I'm asked about multi-threaded programming is, “Can we give you lots of money to write this simple multi-threaded application, please?” Hey—I can dream, can't I?

All the examples in this article make use of POSIX compliant functionality. To the best of my knowledge at the time I write this, there are no fully POSIX-compliant multi-threading libraries available for Linux. Which of the available libraries is best is something of a subjective issue. I use Xavier Leroy's LinuxThreads package, and the code fragments and examples were tested using version 0.5 of this library. This package can be obtained from http://pauillac.inria.fr/~xleroy/linuxthreads. Christopher Provenzano has a good user-level library, although the signal handling doesn't yet match the spec, and there were still a number of serious bugs the last time I used it. (These bugs, I believe, are being worked on.) Other library implementations are also available. Information on these and other packages can be found in the comp.programming.threads newsgroup and (to give a less than exhaustive list):

  • http://www.mit.edu:8001/people/proven/pthreads.html

  • http://www.aa.net/~mtp/PCthreads.html

  • ftp://ftp.cs.fsu.edu/pub/PART/PTHREADS

Thread-specific data

As I implied above, I use the term “global data” for any data which persists beyond normal scoping rules, such as static local variables. Given a piece of code like:

void foo(void)
{
        static int i = 1;
        printf( "%d\n", i );
        i = 2;
}

the first call to this function will print the value 1, and all subsequent calls will print the value 2, because the variable i and its value persist from one invocation of the function to the next, rather than disappearing in a puff of smoke as a “normal” local variable would. This, at least as far as POSIX threads are concerned, is global data.

It is commonly said (I've said it myself) that using global data is a bad practice. Whether or not this is true, it is only a rule of thumb. Certainly there are situations where using global data can avoid creating artificial circumstances. The previous article (Linux Journal Issue 34) explained how threads can share global data with careful use of mutual exclusion (mutex) functions to prevent one thread from accessing an item of global data while another thread is changing its value. In this article I will look at a different type of problem, using a real example from a recent project of mine.

Consider the case of a virtual reality system where a client makes several network socket connections to a server. Different types and priorities of data go down different sockets. High priority data, such as information about objects immediately in the field of view of the client, is sent down one socket. Lower priority data such as texture information, background sounds, or information about objects which are out of the current field of view, is sent down another socket to be processed whenever the client has available time. The server could create a collection of new threads every time a new client connects to the server, designating one thread for each of the sockets to be used to talk to each of the clients. Every one of these threads could use the same function to send a lump of data (not a technical term) to the client. The data to be sent details of the client it is to be sent to, the priority and type of data to be sent could all be held in global variables, and yet each thread will make use of different values. So how do we do it?

As a trivial example, suppose the only global data which our lump-sending function needs to use is an integer that indicates the priority of the data. In a non-threaded version, we might have a global integer called priority used as in Listing 1.

In the multi-threaded version we don't have a global integer, instead we have a global key to the integer. It is through the key that the data can be accessed by means of a number of functions:

  1. pthread_key_create() to prepare the key for use

  2. pthread_setspecific() to set a value to thread-specific data

  3. pthread_getspecific() to retrieve the current value

pthread_key_create() is called once, generally before any of the threads which are going to use the key have been created. pthread_getspecific() and pthread_setspecific() never return an error if the key that is used as an argument has not been created. The result of using them on a key which has not been created is undefined. Something will happen, but it could vary from system to system, and it can't be caught simply by using good error handling. This is an excellent source of bugs for the unwary. So our multi-threaded version might look like Listing 2.

There are a few things to note here:

  1. The implementation of POSIX threads can limit the number of keys a process may use. The standard states that this number must be at least 128. The number available in any implementation can be found by looking at the macro PTHREAD_KEYS_MAX. According to this macro, LinuxThreads currently allows 128 keys.

  2. The function pthread_key_delete() can be used to dispose of keys that are no longer needed. Keys, like all “normal” data items, vanish when the process exits, so why bother deleting them? Think of key handling as being similar to file handling. An unsophisticated program need not close any files that it has opened, as they will be automatically closed when the program exits. But since there is a limit to the number of files a program can have open at one time, the best policy is to close files not currently being used so that the limit is not exceeded. This policy also works well for key handling, as you may be limited in the number of thread-specific data keys a process may have.

  3. pthread_getspecific() and pthread_setspecific() access thread-specific data as void* pointers. This ability can be used directly (as in Listing 2), if the data item to be accessed can be cast as type void*, e.g., an int in most, but not necessarily all, implementations. However, if you want your code to be portable or if you need to access larger data objects, then each thread must allocate sufficient memory for the data object, and store the pointer to the object in the thread-specific data rather than storing the data itself.

  4. If you allocate some memory (using the standard function malloc(), for instance) for your thread-specific data, and the thread exits at some point, what happens to the allocated memory? Nothing happens, so it leaks, and this is bad. This is the situation where the extra parameter in the pthread_key_create() function comes into use. This parameter allows you to specify a function to call when a thread exits, and you use that function to free up any memory that has been allocated. To prevent a waste of CPU time, this destructor function is called only in the case where a thread has made use of that particular key. There's little point in tidying up for a thread that has nothing to be tidied. When a thread exits because it called one of the functions exit(), _exit() or abort(), the destructor function is not called. Also, note that pthread_key_delete() does not cause any destructors to be called, that using a key that has been deleted doesn't have a defined behavior, and that pthread_getspecific() and pthread_setspecific() don't return any error indications. Tidy up your keys carefully. One day you'll be glad you did. So a better version of our code is Listing 3.

Some of this code might look a little strange at first sight. Using pthread_getspecific() to store a thread specific value? The idea is to get the memory location this thread is to use, and then the thread specific value is stored there.

Even if global data is anathema to you, you might still have good use for thread-specific data. In particular, you might need to write a multi-threaded version of some existing library code that is also going to be used in a non-threaded program. A good simple example is making a version of the standard C libraries fit for use by multi-threaded programs. That friend of all C programmers, errno, is a global variable that is commonly set by library functions to indicate what went wrong during a function call. If two threads call functions which both set errno to different values, at least one of the threads is going to get the wrong information. This is solved by having thread-specific data areas for errno, rather than one global variable used by all threads.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

SIGALRM - sync or async?

MarkBerg's picture

Is SIGALRM synchronous or async?
i.e. if I use alarm() within a thread, is this thread guaranteed to get the signal, or might some other thread get it?

Listing 4 Contains A Race Condition

Oldfield's picture

Do not rely on this method of deliving signal information to threads.

A race occurs if two or more signals arrive in similar timeframes - there is nothing to prevent "handled_signal" from being overwritten prior to another thread being scheduled and told about the first delivered signal.

Bad advice from the journal!

regarding synchronous signals

sravan's picture

hi,

In the Listing 4 code, the signal handler captures only the asynchronous signals(like SIGINT) but not synchronous signals like (SIGFPE,SIGSEGV etc.).

How can i make sure that the dedicated thread does the job of capturing both types of signals?

Thank you

catching SIGFPE and SIGSEGV

Anonymous's picture

signals SIGSTOP and SIGSEGV cannot be caught or ignored. I am not sure about SIGFPE....according to POSIX ignoring SIGFPE results in undefined behavior....I have tried to catch SIGFPE on my Linux system...but programme terminated with "Floating point exception"

SIGSEGV and SIGFPE can certainly be caught

Darren Kulp's picture

SIGSEGV and SIGFPE can certainly be caught; in fact, I have intentionally generated and caught both in the same program, to sandbox pseudo-simulated code. The SIGSEGV signal is thrown not only when a "truly invalid" memory access is made, but when permissions are incorrect on the desired page; it is possible to change the permissions on the page from within the handler, and return to the original code (this is what my handler did).

From the sigaction man page:

signum specifies the signal and can be any valid signal except SIGKILL and SIGSTOP.

Thread safety

C2H5OH's picture

Regarding Listing 3. I believe not all malloc() implementations are thread-safe (in fact, very few of them actually are); so I'll assume that calling malloc inside a thread without a mutex is a typo.

My real question is. The key destructor in pthread_key_create(), from where is it called? The main thread or the same thread that is terminating? Is not clear anywhere. Perhaps I should check the POSIX specification but it is also difficult to find.

Cheers

Re: Thread safety

Anonymous's picture

That is totally bogus. Any _sane_ environment that supports threads also has thread-safe malloc() and other primitives.

Thanks a million ...

Anonymous's picture

... for posting this article

Cheers !

Re: Thread-Specific Data and Signal Handling in Multi-Threaded A

Anonymous's picture

can anybody give an example code on how to make "errno" as thread specific data.

thanks in advance

rksoni@indts.com

errno should be automatically

Anonymous's picture

errno should be automatically thread specific, if you compile and link with the right flags (if required by your compiler). You shouldn't have to do anything special to make it thread specific.

Getting good info on pthreads, sockets, signals

Anonymous's picture

Unix Systems Programming (Communication, Concurrency and Threads)
by Robbins & Robbins [Pearson Education]

It has got a good coverage of all the related areas on sockets, threads and signals all in one place. Very good book.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix