Another signal handling bug?

Another signal handling bug?

[David A. Gatwood]

I've just noticed something that, unless I've really misunderstood the man
page, is another significant bug in signal handling.  Put simply, despite
configuration to the contrary, signals are not being blocked during their
handlers, i.e. I'm getting nested signal handlers that never return.

The code in question basically does this:

sa.sa_handler = alarm_handler;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGALRM, &sa, NULL);


Note that sa.sa_flags should have to be OR'ed with SA_NOMASK in order for
the behaviour seen to occur, unless I'm drastically misreading the
sigaction man page or unless it's incorrect.

The alarm handler's first statement is a print statement, as is the last
statement.  There are no return statements anywhere in the handler.  I'm
getting results that look like

In handler (should be ignoring SIGALRM signals).
Got Lock!
Leaving handler.
In handler (should be ignoring SIGALRM signals).
Got Lock!
In handler (should be ignoring SIGALRM signals).
We're in trouble.
Leaving handler.
In handler (should be ignoring SIGALRM signals).
We're in trouble.
Leaving handler.
In handler (should be ignoring SIGALRM signals).
We're in trouble.
Leaving handler.

And so forth.  IOW, the first few times, it's fine, but then under heavy
load, the signal handler never returns, but yet the kernel is allowing
SIGALRM to be delivered, resulting in a nested handler.

Any ideas?  BTW, this is with kernel 2.2.12.  A similar bug was found and
fixed under MkLinux about a year ago, but I haven't heard of the problem
in the monolithic kernel.  I can't see any other way it could occur,
though.  It certainly shouldn't be happening in any other thread, as
SIGALRM is blocked before threads are created, and only unblocked in the
thread in question.  I've tried both sigprocmask and pthread_setmask for
the initial block, both with the same results.

Any ideas?


Later,
David

---------------------------------------------------------------------
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[David A. Gatwood]

On Wed, 16 Aug 2000, David A. Gatwood wrote:

> I've just noticed something that, unless I've really misunderstood the man
> page, is another significant bug in signal handling.  Put simply, despite
> configuration to the contrary, signals are not being blocked during their
> handlers, i.e. I'm getting nested signal handlers that never return.
>
> The code in question basically does this:
>
> sa.sa_handler = alarm_handler;
> sigemptyset(&sa.sa_mask);
> sa.sa_flags = 0;
> sigaction(SIGALRM, &sa, NULL);

<snip>

> Any ideas?  BTW, this is with kernel 2.2.12.  A similar bug was found and
> fixed under MkLinux about a year ago, but I haven't heard of the problem
> in the monolithic kernel.

Well, the good news is that it is not kernel specific.  Same problem under
MkLinux (plus a handful of race conditions that I'm trying to track down
now... :-).

Any idea how to convince sigaction to block the signal from being
delivered within its signal handler (like the man page says it's always
supposed to do)?


David


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[David A. Gatwood]

On Wed, 16 Aug 2000, David A. Gatwood wrote:

> On Wed, 16 Aug 2000, David A. Gatwood wrote:
>
> > I've just noticed something that, unless I've really misunderstood the man
> > page, is another significant bug in signal handling.  Put simply, despite
> > configuration to the contrary, signals are not being blocked during their
> > handlers, i.e. I'm getting nested signal handlers that never return.
> >
> > The code in question basically does this:
> >
> > sa.sa_handler = alarm_handler;
> > sigemptyset(&sa.sa_mask);
> > sa.sa_flags = 0;
> > sigaction(SIGALRM, &sa, NULL);
>
> <snip>
>
> > Any ideas?  BTW, this is with kernel 2.2.12.  A similar bug was found and
> > fixed under MkLinux about a year ago, but I haven't heard of the problem
> > in the monolithic kernel.
>
> Well, the good news is that it is not kernel specific.  Same problem under
> MkLinux (plus a handful of race conditions that I'm trying to track down
> now... :-).
>
> Any idea how to convince sigaction to block the signal from being
> delivered within its signal handler (like the man page says it's always
> supposed to do)?

Oh, and I tried explicitly adding SIGALRM to the list of signals to be
blocked (with sigaddset).  Same result.


David

---------------------------------------------------------------------
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[David A. Gatwood]

On Wed, 16 Aug 2000, David A. Gatwood wrote:

> > > I've just noticed something that, unless I've really misunderstood the man
> > > page, is another significant bug in signal handling.  Put simply, despite
> > > configuration to the contrary, signals are not being blocked during their
> > > handlers, i.e. I'm getting nested signal handlers that never return.
> > >
> > > The code in question basically does this:
> > >
> > > sa.sa_handler = alarm_handler;
> > > sigemptyset(&sa.sa_mask);
> > > sa.sa_flags = 0;
> > > sigaction(SIGALRM, &sa, NULL);

Well, I know why the one handler didn't return -- there was the nasty
little point of it trying to take a lock twice.  After fixing that, the
signal handler _happens_ to always return before the next signal gets
delivered.  However, obviously that doesn't solve the real issue, since
any heavy system load could cause that "luck" to no longer hold true.
Fortunately, for my code's purposes, that is good enough, as long as the
pthreads locks are correct, but it's still somewhat scary.  :-)


Ideas?
David

---------------------------------------------------------------------
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[Kevin B. Hendricks]

Hi David,

Two things:

1. setting the sa_mask to sigemptyset will not mask out any signals in your
handler (see the sigaction man page).  I think you want sigfillset if you want
to mask all signals while in your handler.  Here is a snippet from the sigaction
man page.

       sa_mask gives a mask of signals which  should  be  blocked
       during  execution of the signal handler.  In addition, the
       signal which triggered the handler will be blocked, unless
       the SA_NODEFER or SA_NOMASK flags are used.

2.  You should techically always be calling async signal safe routines
frominside any signal handlers.  Grabbing a mutex lock is not one of these.  You
can sometimes work around this using semaphore posts (which are technically
allowed) inside signal handlers since they are async signal safe.


Hope this helps.

Kevin

.
"David A. Gatwood" wrote:
>
> On Wed, 16 Aug 2000, David A. Gatwood wrote:
>
> > > > I've just noticed something that, unless I've really misunderstood the man
> > > > page, is another significant bug in signal handling.  Put simply, despite
> > > > configuration to the contrary, signals are not being blocked during their
> > > > handlers, i.e. I'm getting nested signal handlers that never return.
> > > >
> > > > The code in question basically does this:
> > > >
> > > > sa.sa_handler = alarm_handler;
> > > > sigemptyset(&sa.sa_mask);
> > > > sa.sa_flags = 0;
> > > > sigaction(SIGALRM, &sa, NULL);
>
> Well, I know why the one handler didn't return -- there was the nasty
> little point of it trying to take a lock twice.  After fixing that, the
> signal handler _happens_ to always return before the next signal gets
> delivered.  However, obviously that doesn't solve the real issue, since
> any heavy system load could cause that "luck" to no longer hold true.
> Fortunately, for my code's purposes, that is good enough, as long as the
> pthreads locks are correct, but it's still somewhat scary.  :-)
>
> Ideas?
> David
>
> ---------------------------------------------------------------------
> A brief Haiku:
>
> Microsoft is bad.
> It seems secure at first glance.
> Then you read your mail.
>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[David A. Gatwood]

On Thu, 17 Aug 2000, Kevin B. Hendricks wrote:

> Hi David,
>
> Two things:
>
> 1. setting the sa_mask to sigemptyset will not mask out any signals in your
> handler (see the sigaction man page).  I think you want sigfillset if you want
> to mask all signals while in your handler.  Here is a snippet from the sigaction
> man page.
>
>        sa_mask gives a mask of signals which  should  be  blocked
>        during  execution of the signal handler.  In addition, the
>        signal which triggered the handler will be blocked, unless
>        the SA_NODEFER or SA_NOMASK flags are used.

However, I'm not setting SA_NODEFER, and it's the signal that triggered
the handler that's recurring, which _is_ supposed to be blocked.


> 2.  You should techically always be calling async signal safe routines
> frominside any signal handlers.  Grabbing a mutex lock is not one of these.

Wait a sec... how can taking a lock be an atomic operation, but not async
signal safe?  That seems a little odd.


Later,
David

---------------------------------------------------------------------
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[Kevin B. Hendricks]

Hi David,

> > 2.  You should techically always be calling async signal safe routines
> > frominside any signal handlers.  Grabbing a mutex lock is not one of these.
>
> Wait a sec... how can taking a lock be an atomic operation, but not async
> signal safe?  That seems a little odd.

Here is what the pthread_mutex_lock page says:

ASYNC-SIGNAL SAFETY
       The mutex functions are not async-signal safe.  What  this
       means is that they should not be called from a signal han<AD>
       dler.  In  particular,   calling   pthread_mutex_lock   or
       pthread_mutex_unlock  from  a  signal handler may deadlock
       the calling thread.

You have already experienced the deadlock problem first hand!

If you use a spinlock and keep track of the current owner you can implement a
simple mutex with recursion allowed and possibly get away with it but in
general, using mutexes in signal handlers are a big no-no.

Kevin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[David A. Gatwood]

On Thu, 17 Aug 2000, Kevin B. Hendricks wrote:

> Hi David,
>
> > > 2.  You should techically always be calling async signal safe routines
> > > frominside any signal handlers.  Grabbing a mutex lock is not one of these.
> >
> > Wait a sec... how can taking a lock be an atomic operation, but not async
> > signal safe?  That seems a little odd.
>
> Here is what the pthread_mutex_lock page says:
>
> ASYNC-SIGNAL SAFETY
>        The mutex functions are not async-signal safe.  What  this
>        means is that they should not be called from a signal han<AD>
>        dler.  In  particular,   calling   pthread_mutex_lock   or
>        pthread_mutex_unlock  from  a  signal handler may deadlock
>        the calling thread.
>
> You have already experienced the deadlock problem first hand!

Actually, the deadlock was my stupidity.  I took the lock and then
proceeded to call a function that took the lock, which is a no-no with
non-debugging locks.  :-)

Well, I'm rewriting the heck out of this thing, adding Yet Another Thread
(tm) to deal with the signal handling, using nanosleep, which I hope is
interruptable by an alarm, since using sleep is potentially incompatible
with the use of setitimer....  Getting really ugly, but hopefully I can
get all these issues worked out, one way or another.

So anybody know why someone doesn't just write a thread/signal-safe Xlib?
:-)


David

---------------------------------------------------------------------
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[Kevin B. Hendricks]

Hi David,

If you are grabbing mutexes in a signal handler, you are obviously in a
multithreaded environment.  When you set the sa_flags to zero you may prevent
that thread from recursing in the signal handler but the signal could be delived
to another thread which in fact has its own signal mask and things but shares
sigaction strucutres and it could be entering your signal handler.

Just a thought, but something to possibly check out.

Kevin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Another signal handling bug?

[David A. Gatwood]

On Thu, 17 Aug 2000, Kevin B. Hendricks wrote:

> Hi David,
>
> If you are grabbing mutexes in a signal handler, you are obviously in a
> multithreaded environment.  When you set the sa_flags to zero you may prevent
> that thread from recursing in the signal handler but the signal could be delived
> to another thread which in fact has its own signal mask and things but shares
> sigaction strucutres and it could be entering your signal handler.
>
> Just a thought, but something to possibly check out.

Already did.  I thought I said that in the original message, but that's
probably been lost for some time in everyone's inbox.  ;-)

Anyway, the program blocks SIGALRM before it spawns threads, and only
unmasks it in the desired thread.


David

---------------------------------------------------------------------
A brief Haiku:

Microsoft is bad.
It seems secure at first glance.
Then you read your mail.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/