Re: [PATCH] fix-flush_workqueue-vs-cpu_dead-race-update

[Srivatsa Vaddagiri <vatsa@in.ibm.com>]

On Sun, Jan 07, 2007 at 03:56:03PM +0300, Oleg Nesterov wrote:
> Srivatsa, I'm completely new to cpu-hotplug, so please correct me if I'm
> wrong (in fact I _hope_ I am wrong) but as I see it, the hotplug/workqueue
> interaction is broken by design, it can't be fixed by changing just locking.
> 
> Once again. CPU dies, CPU_DEAD calls kthread_stop() and sleeps until
> cwq->thread exits. To do so, this thread must at least complete the
> currently running work->func().

If run_workqueue() takes a lock_cpu_hotplug() successfully, then we shouldnt 
even reach till this point, as it will block writers (cpu_down/up) until it
completes.


	run_workqueue()
	---------------
	
try_again:
	rc = lock_cpu_hotplug_interruptible();
	
	if (rc && kthread_should_stop())
		return;
	
	if (rc != 0)
		goto try_again;
	
	/* cpu_down/up shouldnt happen now untill we call unlock_cpu_hotplug */
	while (!list_empty(..))
		work->func();
	
	unlock_cpu_hotplug();


If work->func() calls something (say flush_workqueue()) which requires a
lock_cpu_hotplug() again, there are two ways to support it:

Method 1: Add a field, hotplug_lock_held, in task_struct

	If current->hotplug_lock_held > 1, then lock_cpu_hotplug()
	merely increments it and returns success. Its counterpart, 
	unlock_cpu_hotplug() will decrement the count.

	Easiest to implement. However additional field is required in
	each task_struct, which may not be attractive for some.

Method 2 : Bias readers over writers:

	This method will support recursive calls to lock_cpu_hotplug()
	by the same thread, w/o requiring a field in task_struct. To 
	accomplish this, readers are biased over writers i.e 


		reader1_lock(); <- success

					writer1_lock(); <- blocks on reader1


		reader2_lock(); <- success

A fair lock would have blocked reader2_lock() until 
writer1_lock()/writer1_unlock() is complete, but since we are required to 
support recursion w/o maintaining a task_struct field, we let reader2_lock() 
succeed, even though it could be from a different thread.
	
> Andrew, Ingo, this also means that freezer can't solve this particular
> problem either (if i am right).

freezer wont give stable access to cpu_online_map either, as could typically be
required in functions like flush_workqueue.

-- 
Regards,
vatsa