Slow signal delivery to server process with heavy I/O

Slow signal delivery to server process with heavy I/O

[Dallas Clement]

Hi All,

I've noticed that asynchronous signals such as SIGINT, SIGTERM etc are
delivered to my process long after the signal is sent if the receiving
process is handling lots of I/O.  My process is a multi-threaded web
server.  It's got one thread waiting on 'select' to accept incoming
connections and a thread pool which reads the data with 'recv'.

When I batter the web server with incoming traffic and I try to
shutdown the server by sending a SIGINT or SIGTERM, I have observed
that the web server finishes handling the incoming traffic before the
kernel dispatches the signal to the process.  It appears that the
'select' and 'recv' calls are getting highest priority with regard to
scheduling.

I realize this test may appear unnatural and is perhaps unrealistic,
but I would like to be able to shutdown my server gracefully within a
reasonable amount of time, no matter what kind of load it is handling.
 Don't want to have to wait several minutes for my signals to get
handled under heavy load.  Could someone please explain why signal
delivery is slow under these conditions?

Thanks in advance,

Dallas

Re: Slow signal delivery to server process with heavy I/O

[Dallas Clement]

Yes, I have tried setting this flag.  Doesn't seem to make any
difference.  My main thread doesn't do much.  He registers for the
signal and then waits on a self-pipe and wakes up when a signal
arrives.  All the I/O action is happening in the other threads.  I'm
not expecting the other threads to be interrupted during a system call
like 'select', 'send', 'recv' etc, but even if they were, they detect
EINTR and try again.

On Sat, Jul 10, 2010 at 6:26 PM, Gao Free_Wind <gfree.wind@gmail.com> wrote:
> Do you set the SA_RESTART flag for the signal?
> If set SA_RESTART flag, it will restart the system calls include I/O.
>
> On Sat, Jul 10, 2010 at 4:08 PM, Dallas Clement <dallas.a.clement@gmail.com>
> wrote:
>>
>> Hi All,
>>
>> I've noticed that asynchronous signals such as SIGINT, SIGTERM etc are
>> delivered to my process long after the signal is sent if the receiving
>> process is handling lots of I/O.  My process is a multi-threaded web
>> server.  It's got one thread waiting on 'select' to accept incoming
>> connections and a thread pool which reads the data with 'recv'.
>>
>> When I batter the web server with incoming traffic and I try to
>> shutdown the server by sending a SIGINT or SIGTERM, I have observed
>> that the web server finishes handling the incoming traffic before the
>> kernel dispatches the signal to the process.  It appears that the
>> 'select' and 'recv' calls are getting highest priority with regard to
>> scheduling.
>>
>> I realize this test may appear unnatural and is perhaps unrealistic,
>> but I would like to be able to shutdown my server gracefully within a
>> reasonable amount of time, no matter what kind of load it is handling.
>>  Don't want to have to wait several minutes for my signals to get
>> handled under heavy load.  Could someone please explain why signal
>> delivery is slow under these conditions?
>>
>> Thanks in advance,
>>
>> Dallas
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-c-programming" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Slow signal delivery to server process with heavy I/O

[Glynn Clements]

Dallas Clement wrote:

> I've noticed that asynchronous signals such as SIGINT, SIGTERM etc are
> delivered to my process long after the signal is sent if the receiving
> process is handling lots of I/O.  My process is a multi-threaded web
> server.  It's got one thread waiting on 'select' to accept incoming
> connections and a thread pool which reads the data with 'recv'.
> 
> When I batter the web server with incoming traffic and I try to
> shutdown the server by sending a SIGINT or SIGTERM, I have observed
> that the web server finishes handling the incoming traffic before the
> kernel dispatches the signal to the process.  It appears that the
> 'select' and 'recv' calls are getting highest priority with regard to
> scheduling.
> 
> I realize this test may appear unnatural and is perhaps unrealistic,
> but I would like to be able to shutdown my server gracefully within a
> reasonable amount of time, no matter what kind of load it is handling.
>  Don't want to have to wait several minutes for my signals to get
> handled under heavy load.  Could someone please explain why signal
> delivery is slow under these conditions?

Is it delivery that's slow, or handling? A thread which is executing a
signal handler doesn't get any additional priority. And if there is
intensive disk I/O, paging in the block containing the signal handler
won't get prioritised over other disk I/O.

Also: historically, the kernel hasn't been particularly intelligent
about choosing which thread received the signal (at one time, it
didn't even take into account whether the thread had blocked the
signal). It wouldn't surprise me if it's willing to deliver the signal
to a thread which is in uninterruptible sleep ("D" state).

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: Slow signal delivery to server process with heavy I/O

[Dallas Clement]

It's the delivery that's slow.  If other threads are busy making other
I/O system calls such as 'send', 'recv', 'select' etc, the kernel
seems loathe to interrupt them for the sake of delivering a signal.
Eventually, the signal handling thread gets a turn, but I guess I was
under the false impression that a signal would be like a true
interrupt and preempt any executing user code.

On Sun, Jul 11, 2010 at 8:18 AM, Glynn Clements
<glynn@gclements.plus.com> wrote:
>
> Dallas Clement wrote:
>
>> I've noticed that asynchronous signals such as SIGINT, SIGTERM etc are
>> delivered to my process long after the signal is sent if the receiving
>> process is handling lots of I/O.  My process is a multi-threaded web
>> server.  It's got one thread waiting on 'select' to accept incoming
>> connections and a thread pool which reads the data with 'recv'.
>>
>> When I batter the web server with incoming traffic and I try to
>> shutdown the server by sending a SIGINT or SIGTERM, I have observed
>> that the web server finishes handling the incoming traffic before the
>> kernel dispatches the signal to the process.  It appears that the
>> 'select' and 'recv' calls are getting highest priority with regard to
>> scheduling.
>>
>> I realize this test may appear unnatural and is perhaps unrealistic,
>> but I would like to be able to shutdown my server gracefully within a
>> reasonable amount of time, no matter what kind of load it is handling.
>>  Don't want to have to wait several minutes for my signals to get
>> handled under heavy load.  Could someone please explain why signal
>> delivery is slow under these conditions?
>
> Is it delivery that's slow, or handling? A thread which is executing a
> signal handler doesn't get any additional priority. And if there is
> intensive disk I/O, paging in the block containing the signal handler
> won't get prioritised over other disk I/O.
>
> Also: historically, the kernel hasn't been particularly intelligent
> about choosing which thread received the signal (at one time, it
> didn't even take into account whether the thread had blocked the
> signal). It wouldn't surprise me if it's willing to deliver the signal
> to a thread which is in uninterruptible sleep ("D" state).
>
> --
> Glynn Clements <glynn@gclements.plus.com>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Slow signal delivery to server process with heavy I/O

[Glynn Clements]

Dallas Clement wrote:

> It's the delivery that's slow.

Out of interest, how did you manage to distinguish between delivery
and handling?

> If other threads are busy making other
> I/O system calls such as 'send', 'recv', 'select' etc, the kernel
> seems loathe to interrupt them for the sake of delivering a signal.
> Eventually, the signal handling thread gets a turn, but I guess I was
> under the false impression that a signal would be like a true
> interrupt and preempt any executing user code.

The kernel selects one thread which hasn't blocked the signal. If the
thread is performing a blocking system call (interruptable sleep), it
will be interrupted (changed to runnable, allowing it to receive a
time slice). The next time that the thread is scheduled, the signal
handler will be invoked.

The kernel won't necessarily deliver the signal to the next thread to
be scheduled, and won't necessarily schedule the thread prematurely. 
However, this shouldn't cause a noticeable delay unless the system has
a vast number of runnable threads/processes and a low HZ value (i.e. 
if the time taken to schedule every runnable thread once is long).

If you require urgent delivery of a signal, you need to select
real-time scheduling (sched_setscheduler() with SCHED_FIFO or
SCHED_RR). This allows the thread to "jump the queue", pre-empting
lower-priority threads. In practice, you also need to mlock() any
memory required by the handler, to prevent delays caused by paging.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: Slow signal delivery to server process with heavy I/O

[ern0]

> The kernel selects one thread which hasn't blocked the signal.

So if you want to catch signals ASAP, is it a good strategy to
- leave one thread for signal catching, it may perform sleeps;
- other threads should block all signals?
-- 
ern0
dataflow programmer & evangelist

Re: Slow signal delivery to server process with heavy I/O

[Glynn Clements]

ern0 wrote:

> > The kernel selects one thread which hasn't blocked the signal.
> 
> So if you want to catch signals ASAP, is it a good strategy to
> - leave one thread for signal catching, it may perform sleeps;
> - other threads should block all signals?

You should also set that thread to use real-time scheduling, and
mlock() any memory it requires.

The former ensures that, once the thread becomes runnable, it takes
precedence over other (non-real-time) threads. The latter ensures that
it can actually execute the signal-handling code in a reasonable time
(a thread which is normally idle is prone to having its memory paged
out).

Both of these are subject to resource limits. In particular, the
default for RLIMIT_RTPRIO is usually zero, which prevents selecting
real-time scheduling. The reason is that a real-time process takes
precedence over all non-real-time processes; if a real-time process
gets stuck in a loop, it will prevent non-real-time processes from
getting any CPU. RLIMIT_RTTIME can be used to mitigate this issue.

-- 
Glynn Clements <glynn@gclements.plus.com>