[BUG] cgroups/cpusets: Spurious CPU-hotplug failures

["Paul E. McKenney" <paulmck@kernel.org>]

Hello!

Running rcutorture on v7.0-rc3 results in spurious CPU-hotplug failures,
most frequently on the TREE03 scenario, which suffers about ten such
failures per hundred hours of test time.  Repeat-by is as follows:

tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 80 --duration 100h --configs "100*TREE03" --trust-make

Though a faster repeat-by instead uses kvm-remote.sh and lots of systems.

Bisection converges here:

6df415aa46ec ("cgroup/cpuset: Defer housekeeping_update() calls from CPU hotplug to workqueue")

Reverting this commit gets rid of the spurious CPU-hotplug failures.
Of course, this also gets rid of some ability to do dynamic nohz_full
processing.

Now, the problem might be that the workqueue handler might still be
in flight by the time that rcutorture fired up the next CPU-hotplug
operation, especially given that the TREE03 scenario only waits 200
milliseconds between these operations.  This suggests waiting for this
handler before ending each CPU-hotplug operation.  And the crude patch
below does make the problem go away.

This alleged fix is quite heavy-handed, and also fragile in that if
hk_sd_workfn() uses a different workqueue, this breaks.  It might be
better to call into the cgroups/cpusets code and to use flush_work()
to wait only on hk_sd_workfn() and nothing else.  But it seemed best to
keep things trivial to start with.

Either way, please consider the patch below to be part of this bug report
rather than a proper fix.

Thoughts?

							Thanx, Paul

------------------------------------------------------------------------

diff --git a/kernel/cpu.c b/kernel/cpu.c
index bc4f7a9ba64e6..36a9399be331d 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1514,6 +1514,7 @@ int remove_cpu(unsigned int cpu)
 
 	lock_device_hotplug();
 	ret = device_offline(get_cpu_device(cpu));
+	flush_workqueue(system_unbound_wq);
 	unlock_device_hotplug();
 
 	return ret;
@@ -1730,6 +1731,7 @@ int add_cpu(unsigned int cpu)
 
 	lock_device_hotplug();
 	ret = device_online(get_cpu_device(cpu));
+	flush_workqueue(system_unbound_wq);
 	unlock_device_hotplug();
 
 	return ret;