Re: [tip:timers/nohz] nohz: Move full nohz kick to its own IPI

[Frederic Weisbecker <fweisbec@gmail.com>]

On Wed, May 07, 2014 at 05:37:36PM +0200, Peter Zijlstra wrote:
> On Wed, May 07, 2014 at 05:29:24PM +0200, Frederic Weisbecker wrote:
> > On Wed, May 07, 2014 at 05:17:35PM +0200, Peter Zijlstra wrote:
> > > On Mon, May 05, 2014 at 05:34:08PM +0200, Frederic Weisbecker wrote:
> > > > On Mon, May 05, 2014 at 05:12:28PM +0200, Peter Zijlstra wrote:
> > > > > > Note the current ordering:
> > > > > > 
> > > > > >     cmpxchg(&qsd->pending, 0, 1)       get ipi
> > > > > >     csd_lock(qsd->csd)                 xchg(&qsd->pending, 1)
> > > > > >     send ipi                           csd_unlock(qsd->csd)
> > > > > > 
> > > > > > 
> > > > > > So there shouldn't be racing updaters. Also ipi sender shouldn't
> > > > > > race with ipi receiver, the update shouldn't always eventually see
> > > > > > the unlock happening.
> > > > > 
> > > > > Yeah, I've not spotted how this particular train wreck happens either.
> > > > > 
> > > > > The problem is reproduction, it took me 9 hours to confirm I could
> > > > > reproduce the problem on my machine. So how long to I run it with this
> > > > > patch reverted to show its gone..
> > > > 
> > > > Maybe it could be favoured cpu hotplug. Anyway converting to irq_work should
> > > > fix it.
> > > 
> > > Ingo needs a commit msg for the revert of this patch; do you think you
> > > have time to look into _why_ this patch is broken and write such a
> > > thing?
> > 
> > I can try but I need to reproduce it. Do you have any clue on how to do so?
> > Also which HEAD were you guys using?
> 
> Ha!, so I was running a tip/master with that commit in -- a few days
> ago, v3.15-rc4-1644-g5c658b0cdf22 might've been it.
> 
> Then I ran it on my dual socket AMD interlagos, with:
> 
> while :; make O=allyesconfig-build/ clean; make O=allyesconfig-build/
> -j96 -s; done
> 
> for 9 hours, and then got empty RCU stall warns and a bricked machine.
> 
> I might still have the .config, but I don't think there was anything
> particularly odd about the config other than having NOHZ_FULL enabled.
> 
> The only way I found this patch was by staring at some RCU stall warns
> Ingo managed to get, sometimes they actually got backtraces in them
> apparently.
> 
> According to Ingo the bigger the machine the faster it reproduces, but
> reproduction times, even for these 32 cpu machines, are in the many
> hours range.

Ok then, I'll try something.

But note that those commits aren't upstream yet and they are in a seperate
branch tip:timers/nohz with no other non-upstream commits.

And I work alone on this branch.

So we can as well zap these commits and replace them with the irq_work_on()
conversion (still preparing that).