Re: [PATCH -tip/core/rcu 1/6] Cleanups and fixes for RCU in face of heavy CPU-hotplug stress

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Tue, Aug 18, 2009 at 05:26:43PM +0200, Ingo Molnar wrote:
> 
> FYI, i've started triggering hangs in -tip testing recently, during 
> CPU hotplug tests:
> 
> [   57.632003] eth0: no IPv6 routers present
> [  103.564010] kmemleak: 29 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
> [  200.380003] Hangcheck: hangcheck value past margin!
> [  248.192003] INFO: task S99local:2974 blocked for more than 120 seconds.
> [  248.194532] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [  248.202330] S99local      D 0000000c  6256  2974   2687 0x00000000
> [  248.208929]  9c7ebe90 00000086 6b67ef8b 0000000c 9f25a610 81a69869 00000001 820b6990
> [  248.216123]  820b6990 820b6990 9c6e4c20 9c6e4eb4 82c78990 00000000 6b993559 0000000c
> [  248.220616]  9c7ebe90 8105f22a 9c6e4eb4 9c6e4c20 00000001 9c7ebe98 9c7ebeb4 81a65cb3
> [  248.229990] Call Trace:
> [  248.234049]  [<81a69869>] ? _spin_unlock_irqrestore+0x22/0x37
> [  248.239769]  [<8105f22a>] ? prepare_to_wait+0x48/0x4e
> [  248.244796]  [<81a65cb3>] rcu_barrier_cpu_hotplug+0xaa/0xc9
> [  248.250343]  [<8105f029>] ? autoremove_wake_function+0x0/0x38
> [  248.256063]  [<81062cf2>] notifier_call_chain+0x49/0x71
> [  248.261263]  [<81062da0>] raw_notifier_call_chain+0x11/0x13
> [  248.266809]  [<81a0b475>] _cpu_down+0x272/0x288
> [  248.271316]  [<81a0b4d5>] cpu_down+0x4a/0xa2
> [  248.275563]  [<81a0c48a>] store_online+0x2a/0x5e
> [  248.280156]  [<81a0c460>] ? store_online+0x0/0x5e
> [  248.284836]  [<814ddc35>] sysdev_store+0x20/0x28
> [  248.289429]  [<8112e403>] sysfs_write_file+0xb8/0xe3
> [  248.294369]  [<8112e34b>] ? sysfs_write_file+0x0/0xe3
> [  248.299396]  [<810e4c8f>] vfs_write+0x91/0x120
> [  248.303817]  [<810e4dc1>] sys_write+0x40/0x65
> [  248.308150]  [<81002d73>] sysenter_do_call+0x12/0x28
> 
> config and bootlog attached. I'd suspect one of these patches:
> 
> 684ca5c: rcu: Fix typo in rcu_irq_exit() comment header
> b612ba8: rcu: Make rcupreempt_trace.c look at offline CPUs
> 8064d54: rcu: Make preemptable RCU scan all CPUs when summing RCU counters
> 2e59755: rcu: Simplify RCU CPU-hotplug notification
> 799e64f: cpu hotplug: Introduce cpu_notifier() to handle !HOTPLUG_CPU case
> 2756962: rcu: Split hierarchical RCU initialization into boot-time and CPU-online piece
> 
> Any ideas?

Gah...  I thought I had fixed that one!!!  I was seeing a deadlock
where rcu_barrier_cpu_hotplug() would register the three RCU callbacks,
then wait for them.  But in some situations, it would wait for them in
a state such that grace period could not complete.  I convinced myself
that moving the wait back from CPU_DEAD to CPU_POST_DEAD solved the
problem.

I am going to take a more bullet-proof approach, switching from the
wait_completion() form to wait_event(), which will allow me to wait
for the previous hotplug operation's callbacks at the beginning of the
subsequent hotplug operation.

I reserve the right to insert a short delay in the CPU-hotplug path
outside of any locks, but would imagine that people would prefer that
I avoid that sort of thing, at least until we have bulk CPU-hotplug
operations.

							Thanx, Paul