Re: [PATCH, RFC] reimplement flush_workqueue()

[Srivatsa Vaddagiri <vatsa@in.ibm.com>]

On Tue, Dec 19, 2006 at 03:43:19AM +0300, Oleg Nesterov wrote:
> > Taking workqueue_mutex() unconditionally in flush_workqueue() means
> > that we'll deadlock if a single-threaded workqueue callback handler calls
> > flush_workqueue().
> 
> Well. But flush_workqueue() drops workqueue_mutex before going to sleep ?

... and acquires it again after woken from sleep. That can be a problem, which 
will lead to the problem described here:

	http://lkml.org/lkml/2006/12/7/374

In brief:

keventd thread					hotplug thread
--------------					--------------

  run_workqueue()
	|
     work_fn()
	 |
	flush_workqueue()
	     |	
	   flush_cpu_workqueue
		|				cpu_down()
	     mutex_unlock(wq_mutex);		     |
	(above opens window for hotplug)	   mutex_lock(wq_mutex);
    		|				   /* bring down cpu */	
	     wait_for_completition();		     notifier(CPU_DEAD, ..)
		| 				       workqueue_cpu_callback
		| 				        cleanup_workqueue_thread
		|					  kthread_stop()
		|
		|
	     mutex_lock(wq_mutex); <- Can deadlock


The kthread_stop() will wait for keventd() thread to exit, but keventd()
is blocked on mutex_lock(wq_mutex) leading to a deadlock.

> 
> 	flush_workqueue(single_threaded_wq);
> 		...
> 		mutex_lock(&workqueue_mutex);
> 		...
> 		mutex_unlock(&workqueue_mutex);
> 		wait_for_completition();
> 							handler runs,
> 							calls flush_workqueue(),
> 							workqueue_mutex is free

-- 
Regards,
vatsa