Crash in schedule path after worker thread dies

Crash in schedule path after worker thread dies

[Todd Poynor]

After a worker thread died due to a bug in a work function, a NULL
dereference was seen when schedule calls wq_worker_sleeping calls
kthread_data:

          return to_kthread(task)->data;

mm_release has apparently already set the task's vfork_done = NULL,
causing to_kthread to return a bad address (on 3.0-rc7 on ARM).

I haven't tried a fix because I'm not sure if avoiding this case is
enough to properly recover from death of a worker thread, or if this
has already been discussed and rejected in the past.  I searched
around a little and found some mentions of problems in worker
functions that were probably followed by the kthread_data crash,
but didn't turn up any specific discussion of this crash.  So I
thought I'd start by mentioning this here, and can help fix or test if
needed.


Todd

Re: Crash in schedule path after worker thread dies

[Tejun Heo]

Hello, Todd.

On Tue, Jul 19, 2011 at 05:07:38PM -0700, Todd Poynor wrote:
> After a worker thread died due to a bug in a work function, a NULL
> dereference was seen when schedule calls wq_worker_sleeping calls
> kthread_data:
> 
>           return to_kthread(task)->data;
> 
> mm_release has apparently already set the task's vfork_done = NULL,
> causing to_kthread to return a bad address (on 3.0-rc7 on ARM).
> 
> I haven't tried a fix because I'm not sure if avoiding this case is
> enough to properly recover from death of a worker thread, or if this
> has already been discussed and rejected in the past.  I searched
> around a little and found some mentions of problems in worker
> functions that were probably followed by the kthread_data crash,
> but didn't turn up any specific discussion of this crash.  So I
> thought I'd start by mentioning this here, and can help fix or test if
> needed.

Hmmm... the scheduler code is oopsing after the worker already oopsed,
escalating the failure mode, right?  Avoiding the scheduler oops can
be done simply by clearing PF_WQ_WORKER in do_exit(); however, it
would still leave workqueue code assuming the worker is busy running
and may lead to workqueue stall.

Cleaning that up would require more code but at that point it isn't
really clear what we would be buying with the added complexity.
Hmmm... maybe it would still be worthwhile to at least keep scheduler
working.  Something like this?

diff --git a/kernel/exit.c b/kernel/exit.c
index 6c7fbbe..412ead0 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -915,6 +915,7 @@ NORET_TYPE void do_exit(long code)
 	 * kernel address.
 	 */
 	set_fs(USER_DS);
+	tsk->flags &= ~PF_WQ_WORKER;
 
 	ptrace_event(PTRACE_EVENT_EXIT, code);
 
-- 
tejun

Re: Crash in schedule path after worker thread dies

[Todd Poynor]

Hello Tejun,

On Wed, Jul 20, 2011 at 07:07:17PM +0200, Tejun Heo wrote:
> On Tue, Jul 19, 2011 at 05:07:38PM -0700, Todd Poynor wrote:
> > After a worker thread died due to a bug in a work function, a NULL
> > dereference was seen when schedule calls wq_worker_sleeping calls
> > kthread_data:
> > 
> >           return to_kthread(task)->data;
> > 
> > mm_release has apparently already set the task's vfork_done = NULL,
> > causing to_kthread to return a bad address (on 3.0-rc7 on ARM).
> > [...]
> 
> Hmmm... the scheduler code is oopsing after the worker already oopsed,
> escalating the failure mode, right?  Avoiding the scheduler oops can
> be done simply by clearing PF_WQ_WORKER in do_exit(); however, it
> would still leave workqueue code assuming the worker is busy running
> and may lead to workqueue stall.

Right, the worker oops took out that thread (don't know if
workqueue processing would have gracefully recovered if not for the
scheduler oops), but the scheduler oops was catastrophic.

> Cleaning that up would require more code but at that point it isn't
> really clear what we would be buying with the added complexity.
> Hmmm... maybe it would still be worthwhile to at least keep scheduler
> working.  Something like this?
> 
> diff --git a/kernel/exit.c b/kernel/exit.c
> index 6c7fbbe..412ead0 100644
> --- a/kernel/exit.c
> +++ b/kernel/exit.c
> @@ -915,6 +915,7 @@ NORET_TYPE void do_exit(long code)
>  	 * kernel address.
>  	 */
>  	set_fs(USER_DS);
> +	tsk->flags &= ~PF_WQ_WORKER;
>  
>  	ptrace_event(PTRACE_EVENT_EXIT, code);

Sounds good to me, thanks!

Todd

>  
> -- 
> tejun