Re: Bug in on_each_cpu?

Ernie Petrides
2007-03-02T04:47:01Z

On Thursday, 1-Mar-2007 at 7:22 PST, Andrew Morton wrote:

> On Thu, 01 Mar 2007 03:47:39 -0800 Zachary Amsden <zach@vmware.com> wrote:
>
> > Rusty Russell wrote:
> > > On Thu, 2007-03-01 at 03:34 -0800, Zachary Amsden wrote:
> > >
> > >> What would be really, really nice would be to statically check all
> > >> callsites that issue irq disables actually keep irqs disabled.
> > >> Presumably, there was a reason they disabled irqs, and re-enabling them
> > >> underneath their noses, even if it is to avoid a race, breaks the logic
> > >> behind that reason.
> > >>
> > >
> > > For the moment, how about a BUG_ON() in on_each_cpu()?
> > >
> >
> > Sounds quite decent. But why does on_each_cpu need to disable
> > interrupts at all? It just calls func(), then re-enables interrupts.
> > So whatever was going to happen during func() that might not be
> > interrupt safe could just be done in the callee, avoiding the rather
> > expensive mess of disabling and re-enabling interrupts for those cases
> > where it doesn't matter.
>
> The handler for smp_call_function() is called with local interrupts
> disabled (from the IPI handler).
>
> So to provide a consistent call environment for that handler, on_each_cpu()
> will also disable local interrupts when making the direct call on this CPU.

And further, this "consistent call environment" is *required* for correct
operation of certain callers, e.g. invalidate_bh_lrus(), whose callback
function is invalidate_bh_lru(). If invalidate_bh_lru() is called without
IRQs blocked, it might be interrupted by an IPI that causes nested execution
of that same function on behalf of another cpu's call to on_each_cpu(), and
this can lead to duplicate brelse() calls on a buf head (and ultimately to
ext3 journaling crashes due to invalid concurrent use of that buf head).

Cheers. -ernie