Re: [PATCH v10 1/3] smpboot: allow excluding cpus from the smpboot threads

[Frederic Weisbecker <fweisbec@gmail.com>]

On Fri, May 01, 2015 at 03:57:51PM -0400, Chris Metcalf wrote:
> On 05/01/2015 04:53 AM, Frederic Weisbecker wrote:
> >>+	/* Unpark any threads that were voluntarily parked. */
> >>+	for_each_cpu_not(cpu, &ht->cpumask) {
> >>+		if (cpu_online(cpu)) {
> >>+			struct task_struct *tsk = *per_cpu_ptr(ht->store, cpu);
> >>+			if (tsk)
> >>+				kthread_unpark(tsk);
> >I'm still not clear why we are doing that. kthread_stop() should be able
> >to handle parked kthreads, otherwise it needs to be fixed.
> 
> Checking without the unpark, it's actually only a problem with nohz_full.
> In a system without nohz_full, the kthreads are able to stop even when
> they are parked; it's only in the nohz_full case that things wedge.

Ok. So this isn't a proper fix but a workaround for a bug that we don't
understand yet. In this case I much prefer that you remove this workaround
(I'm talking about this unpark loop) because it hides the issue. And hiding
the bug is the last thing we want if we plan to fix it properly.

> 
> For example, booting with only cpu 0 as a housekeeping core (and
> therefore all watchdogs 1-35 on my 36-core tilegx are parked), and
> immediately doing "echo 0 > /proc/sys/kernel/watchdog", I see
> (via SysRq ^O-l) the first parked watchdog, on cpu 1, hung with:
> 
>   frame 0: 0xfffffff7000f2928 lock_hrtimer_base+0xb8/0xc0
>   frame 1: 0xfffffff7000f2a28 hrtimer_try_to_cancel+0x40/0x170
>   frame 2: 0xfffffff7000f2a28 hrtimer_try_to_cancel+0x40/0x170
>   frame 3: 0xfffffff7000f2b98 hrtimer_cancel+0x40/0x68
>   frame 4: 0xfffffff70014cce0 watchdog_disable+0x50/0x70
>   frame 5: 0xfffffff70008c2d0 smpboot_thread_fn+0x350/0x438
>   frame 6: 0xfffffff700084b28 kthread+0x160/0x178

Have you tried to do that before your patchset?

> The other cores are all idle.
> 
> I have no idea why lock_hrtimer_base() is hanging; perhaps the
> hrtimer_cpu_base lock is taken by some other task that is now
> scheduled out.

No, it's a spinlock, tasks can't sleep while holding it. But it looks like
a deadlock.

> 
> The config does not have NO_HZ_FULL_ALL or NO_HZ_FULL_SYSIDLE
> set, and does have RCU_FAST_NO_HZ and RCU_NOCB_CPU_ALL.
> 
> I don't really know how to start debugging this, but I do know that
> unparking the threads first avoids the issue :-)

Do you have CONFIG_PROVE_LOCKING=y ?

I can't check that myself until the middle of next week.

> 
> -- 
> Chris Metcalf, EZChip Semiconductor
> http://www.ezchip.com
>