Kernel Korner - Sleeping in the Kernel

The old sleep_on() function won't work reliably in an age of SMP systems and hyperthreaded processors. Here's how to make a process sleep in a safe, cross-platform way.
Wait Queues

Wait queues are a higher-level mechanism used to put processes to sleep and wake them up. In most instances, you use wait queues. They are needed when more than one process wants to sleep on the occurrence of one or more than one event.

A wait queue for an event is a list of nodes. Each node points to a process waiting for that event. An individual node in this list is called a wait queue entry. Processes that want to sleep while the event occurs add themselves to this list before going to sleep. On the occurrence of the event, one or more processes on the list are woken up. Upon waking up, the processes remove themselves from the list.

A wait queue could be defined and initialised in the following manner:


wait_queue_head_t my_event;
init_waitqueue_head(&my_event);

The same effect could be achieved by using this macro:

DECLARE_WAIT_QUEUE_HEAD(my_event);

Any process that wants to wait on my_event could use either of the following options:

  1. wait_event(&my_event, (event_present == 1) );

  2. wait_event_interruptible(&my_event, (event_present == 1) );

The interruptible version 2 of the options above puts the process to an interruptible sleep, whereas the other (option 1) puts the process into an uninterruptible sleep.

In most instances, a process goes to sleep only after checking some condition for the availability of the resource. To facilitate that, both these functions take an expression as the second argument. The process goes to sleep only if the expression evaluates to false. Care is taken to avoid the lost wake-up problem.

Old kernel versions used the functions sleep_on() and interruptible_sleep_on(), but those two functions can introduce bad race conditions and should not be used.

Let's now take a look at some of the calls for waking up process sleeping on a wait queue:

  1. wake_up(&my_event);: wakes up only one process from the wait queue.

  2. wake_up_all(&my_event);: wakes up all the processes on the wait queue.

  3. wake_up_interruptible(&my_event);: wakes up only one process from the wait queue that is in interruptible sleep.

Wait Queues: Putting It Together

Let us look at a real-life example of how wait queues are used. smbiod is the I/O thread that performs I/O operations for the SMB filesystem. Here is a code snippet for the smbiod thread (linux-2.6.11/fs/smbfs/smbiod.c: 291):


291 static int smbiod(void *unused)
292 {
293     daemonize("smbiod");
294
295     allow_signal(SIGKILL);
296
297     VERBOSE("SMB Kernel thread starting "
                "(%d)...\n", current->pid);
298
299     for (;;) {
300             struct smb_sb_info *server;
301             struct list_head *pos, *n;
302
303             /* FIXME: Use poll? */
304             wait_event_interruptible(smbiod_wait,
305                     test_bit(SMBIOD_DATA_READY,
                                 &smbiod_flags));
...
...             /* Some processing */
312
313             clear_bit(SMBIOD_DATA_READY,
                          &smbiod_flags);
314
...             /* Code to perform the requested I/O */
...
...
337     }
338
339     VERBOSE("SMB Kernel thread exiting (%d)...\n",
                current->pid);
340     module_put_and_exit(0);
341 }
342

As is clear from the code, smbiod is a thread that runs in a continuous loop as it processes I/O requests. When there are no I/O requests to process, the thread goes to sleep on the wait queue smbiod_wait. This is achieved by calling wait_event_interruptible (line 304). This call causes the smbiod to sleep only if the DATA_READY bit is set. As mentioned earlier, wait_event_interruptible takes care to avoid the lost wake-up problem.

Now, when a process wants to get some I/O done, it sets the DATA_READY bit in the smbiod_flags and wakes up the smbiod thread to perform I/O. This can be seen in the following code snippet (linux-2.6.11/fs/smbfs/smbiod.c: 57):


57 void smbiod_wake_up(void)
58 {
59     if (smbiod_state == SMBIOD_DEAD)
60         return;
61     set_bit(SMBIOD_DATA_READY, &smbiod_flags);
62     wake_up_interruptible(&smbiod_wait);
63 }

wake_up_interruptible wakes up one process that was sleeping on the smbiod_wait waitqueue. The function smb_add_request (linux-2.6.11/fs/smbfs/request.c: 279) calls the smbiod_wake_up function when it adds new requests for processing.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

lost wakeup in wait_event_interruptible?

Anonymous's picture

Hello,

I see a 'lost wakeup' even in latest wait_event_interruptible. I know race condition got solved after sleep_on. So, I just want to know how/why this works w/o any problem. I saw a similar question but no answers (code snippet from 2.6.27 below)

thanks,
Shankar.

----

a) Wakeup occurs immediately before the call to prepare_to_wait()

b) Call to prepare_to_wait() sets process state to TASK_INTERRUPTIBLE

c) While the prepare_to_wait exits but just before condition is evaluated again, a h/w interrupt comes in!

d) The process is now still marked as TASK_INTERRUPTIBLE and will therefore not be

re-scheduled and will never execute the call to finish_wait() - so it will sleep

unless/until an interruption comes along...

-----

#define __wait_event_interruptible(wq, condition, ret) \
do { \
DEFINE_WAIT(__wait); \
\
for ( ; ; ) { \
prepare_to_wait(&wq, &__wait, TASK_INTERRUPTIBLE); \
if (condition) \
break; \
if (!signal_pending(current)) { \
schedule(); \
continue; \
} \
ret = -ERESTARTSYS; \
break; \
} \
finish_wait(&wq, &__wait); \
} while (0)

#define wait_event_interruptible(wq, condition) \
({ \
int __ret = 0; \
if (!(condition)) \
__wait_event_interruptible(wq, condition, __ret); \
__ret; \
})

wait_event_interruptible - retuns error (ERESTARTSYS)

Anonymous's picture

Hi,

I am using wait_for_interruptable in
read function and the isr wakes it up using wake_up_interruptable.
Some times wait_for_interruptable returns error ERESTARTSYS.

The application usage scenario is as follows: I have a process which has lots of threads. And one thread has read() function. Sometimes wake_up_interruptable() function return error and the full user-application is killed. But the system contines to run with out any issues.

There are two possibilities this problem can occur:
1) There are many threads in my process, some thread has caused some problem which results in killing the process. So the process sends signal to the thread which is doing read and the read thread which is waiting on wake_up_interruptable() comes out with error. One more observation here is that the "release" function of my driver is called when this happens.
2) Second possibility is that - Some thing wrong is happening in the driver and it is giving the error for wait_for_interruptable() and this inturn kills the userspace process. I am not very sure about this. But can this kind of thing happen?

please provide some inputs on this ..

Waitqueues in bottom halve context

Anonymous's picture

Can we use wait queue in bottom halve context?
For example, we have LOC in the following source
Kernel Version: Linux-2.6.18.5
Path: \net\xfrm\xfrm_policy.c
Ln: 919
Which calls schedule() function.
This function is called when packet is received and looks for
policy in softirq context.
Please correct me if i misunderstood.
Thanks in advance

Re: Waitqueues in bottom halve context

Sathish Kumar's picture

The function xfrm_lookup() is called from both user context
and kernel(softirq) context.
The process is put to sleep when arguemnt:flag of the above
function is -EAGAIN.
User Context:
If this function is called from user context it may have -EAGAIN
Kernel Context:
If this function is called from kernel context (softirq), it will be NULL.
So the softirq context process will not be put to wait queue.

Thanks,
Sathish Kumar

misprint

Anonymous's picture

In the line:
"This call causes the smbiod to sleep only if the DATA_READY bit is set",
"set" should read "clear", and I think this is a misprint here.

SIGKILL arrival

mus_cular_dude's picture

how will wait_event_interruptible will behave if SIGKILL happens to the user space process that is sleeping using this function? My system gives kernel panic when i do 'kill -SIGKILL pid'. is it because of the reason that im sleeping in kernel and i delivered SIGKILL?

In the schedule function

Vinay Raghavan's picture

In the schedule function section, you have mentioned that a way to wake up a process that has just called the schedule function is to call wake_up_process(sleeping_task);

But how does another process have a reference to the task structure of a process that has just called schedule.

In the code you have shown the example to get a reference to the task structure. But that is being done in the process that is calling schedule.

Please elaborate.

Thanks,
Vinay

store it somewhere

Kedar's picture

It is upto you where/how you want to store that reference to the sleeping process's task struct. You may embed it in the appropriate data structures in your code.

Waking up a sleeping process from user space

Anonymous's picture

What is a good way to wake up a process that is asleep in kernel space from a process that is in user space?

Perhaps a system call which invokes 'wake_up_process'?

The 'kill' system call doesn't seem to work...

schedule_timeout_{,un}interruptible()

Kedar Sovani's picture

A patch that includes two new variants of schedule_timeout() has been included in the -mm kernel tree.

Almost all the calls to schedule_timeout(), set the state of the executing task to TASK_INTERRUPTIBLE / TASK_UNINTERRUPTIBLE. This patch embeds such functionality in the following calls,

schedule_timeout_interruptible() and,
schedule_timeout_uninterruptible()

Error in code samples

rjbell4's picture

There is an error in the code samples. wait_event() and wait_event_interuptible() should not be passed the address of my_event, but my_event itself. That is because they are macros, and their implementations will wind up using the address-of operator (&) to take the address of the parameter they are passed.

Re : Error in code samples

Kedar Sovani's picture

Yes, thanks for that correction. In section "Wait Queues", the illustrations should be like this :

  1. wait_event(my_event, (event_present == 1) );
  2. wait_event_interruptible(my_event, (event_present == 1) );

That is because they are

Anonymous's picture

That is because they are macros, and their implementations will wind up using the address-of operator (&) to take the address of the parameter they are passed.mırç mırç Chat chat türkçe mirc türkçe mirc

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix