IRQ_ONESHOT expectations vs mask/unmask

IRQ_ONESHOT expectations vs mask/unmask

[Benjamin Herrenschmidt]

Hi Thomas !

I noticed a problem with some powerpc systems with threaded IRQs. I
hadn't looked at threaded irq in the past so there was a bit of
discovery involved. That lead to a few questions:

 - Threaded IRQs rely on IRQF_ONESHOT. Now, is that a flag/feature that
is generally exposed to drivers even in the non-threaded case, and if
yes, what core callback are drivers expected to use to "unmask" the
oneshot interrupt ?

 - I don't see any provisions for dealing with interrupts lost while
masked. So on some PICs on powerpc, while we do use "fast EOI", we also
have a chance of edge interrupts (MSIs) being lost while masked. This
wasn't a problem until now because we used lazy disabling. However, it
looks like IRQF_ONESHOT (and thus threaded irqs) rely on masked
interrupts being latched in HW, otherwise an interrupt might be lost
between the threaded handler completing and the interrupt being
unmasked, or am I missing something ?

 - I noticed that other flow handlers (edge, edge_eoi, ...) don't have
any provision for IRQF_ONESHOT. Isn't that a problem ? Or will the core
silently swallow subsequent interrupts until the thread has completed
anyway ? (I might be missing something here).

Generally, how do you suggest I fix the threaded irq problem for XICS ?

(For the newer P9 XIVE interrupt controller I can probably fix things
by using a different masking mechanism in HW but for the old XICS,
especially with the hypervisor under the hood, I'm less confident).

I'd rather not write my own flow handler, that's a recipe for missing
fixes going into the generic ones and horrible bitrot...

Cheers,
Ben.

Re: IRQ_ONESHOT expectations vs mask/unmask

[Thomas Gleixner]

Ben,

On Mon, 31 Jul 2017, Benjamin Herrenschmidt wrote:
> I noticed a problem with some powerpc systems with threaded IRQs. I
> hadn't looked at threaded irq in the past so there was a bit of
> discovery involved. That lead to a few questions:
> 
>  - Threaded IRQs rely on IRQF_ONESHOT. Now, is that a flag/feature that
> is generally exposed to drivers even in the non-threaded case, and if
> yes, what core callback are drivers expected to use to "unmask" the
> oneshot interrupt ?

The oneshot flag should not have any effects when the interrupt is
non-threaded. Emphasis on should. I'm not sure whether that's true, will
have a look.

>  - I don't see any provisions for dealing with interrupts lost while
> masked. So on some PICs on powerpc, while we do use "fast EOI", we also
> have a chance of edge interrupts (MSIs) being lost while masked. This
> wasn't a problem until now because we used lazy disabling. However, it
> looks like IRQF_ONESHOT (and thus threaded irqs) rely on masked
> interrupts being latched in HW, otherwise an interrupt might be lost
> between the threaded handler completing and the interrupt being
> unmasked, or am I missing something ?
>
>  - I noticed that other flow handlers (edge, edge_eoi, ...) don't have
> any provision for IRQF_ONESHOT. Isn't that a problem ? Or will the core
> silently swallow subsequent interrupts until the thread has completed
> anyway ? (I might be missing something here).

The only case where IRQF_ONESHOT should have an effect is with level type
interrupts. That's required, because otherwise the hardware interrupt would
fire for ever. Level type interrupts should not need any hardware latching,
we assume that they fire once they are unmasked.

For edge type interrupts we do not mask the interrupt in order not to lose
an interrupt. If the interrupt fires while the thread handler is running,
we mark the thread and once it the handler returns it gets called again.

> Generally, how do you suggest I fix the threaded irq problem for XICS ?

You asked a lot of questions, but you failed to explain the problem for
XICS.

Thanks,

	tglx

Re: IRQ_ONESHOT expectations vs mask/unmask

[Benjamin Herrenschmidt]

On Mon, 2017-07-31 at 09:09 +0200, Thomas Gleixner wrote:
> Ben,
> 
> On Mon, 31 Jul 2017, Benjamin Herrenschmidt wrote:
> > I noticed a problem with some powerpc systems with threaded IRQs. I
> > hadn't looked at threaded irq in the past so there was a bit of
> > discovery involved. That lead to a few questions:
> > 
> >  - Threaded IRQs rely on IRQF_ONESHOT. Now, is that a flag/feature that
> > is generally exposed to drivers even in the non-threaded case, and if
> > yes, what core callback are drivers expected to use to "unmask" the
> > oneshot interrupt ?
> 
> The oneshot flag should not have any effects when the interrupt is
> non-threaded. Emphasis on should. I'm not sure whether that's true, will
> have a look.
> 
> >  - I don't see any provisions for dealing with interrupts lost while
> > masked. So on some PICs on powerpc, while we do use "fast EOI", we also
> > have a chance of edge interrupts (MSIs) being lost while masked. This
> > wasn't a problem until now because we used lazy disabling. However, it
> > looks like IRQF_ONESHOT (and thus threaded irqs) rely on masked
> > interrupts being latched in HW, otherwise an interrupt might be lost
> > between the threaded handler completing and the interrupt being
> > unmasked, or am I missing something ?
> > 
> >  - I noticed that other flow handlers (edge, edge_eoi, ...) don't have
> > any provision for IRQF_ONESHOT. Isn't that a problem ? Or will the core
> > silently swallow subsequent interrupts until the thread has completed
> > anyway ? (I might be missing something here).
> 
> The only case where IRQF_ONESHOT should have an effect is with level type
> interrupts. That's required, because otherwise the hardware interrupt would
> fire for ever. Level type interrupts should not need any hardware latching,
> we assume that they fire once they are unmasked.
> 
> For edge type interrupts we do not mask the interrupt in order not to lose
> an interrupt. If the interrupt fires while the thread handler is running,
> we mark the thread and once it the handler returns it gets called again.
> 
> > Generally, how do you suggest I fix the threaded irq problem for XICS ?
> 
> You asked a lot of questions, but you failed to explain the problem for
> XICS.

I did but maybe it wasn't clear :-)

"So on some PICs on powerpc, while we do use "fast EOI", we also> >
have a chance of edge interrupts (MSIs) being lost while masked."

Basically, a masked interrupt might get dropped rather than "latched",
so if we use the existing fasteoi handler with IRQF_ONESHOT, we'll
lose them if they occur at the wrong time.

I could use something like the edge_eoi flow handler instead I suppose
but that will *never* lazy disable which is somewhat unfortunate, very
fast paced MSIs benefit from being HW masked if we already recorded
that they occurred.

So what I would need is something along the line of ONESHOT as done in
fasteoi but only for level interrupts.

Cheers,
Ben.

> Thanks,
> 
> 	tglx

Re: IRQ_ONESHOT expectations vs mask/unmask

[Thomas Gleixner]

On Mon, 31 Jul 2017, Benjamin Herrenschmidt wrote:
> On Mon, 2017-07-31 at 09:09 +0200, Thomas Gleixner wrote:
> > You asked a lot of questions, but you failed to explain the problem for
> > XICS.
> 
> I did but maybe it wasn't clear :-)
> 
> "So on some PICs on powerpc, while we do use "fast EOI", we also> >
> have a chance of edge interrupts (MSIs) being lost while masked."
> 
> Basically, a masked interrupt might get dropped rather than "latched",
> so if we use the existing fasteoi handler with IRQF_ONESHOT, we'll
> lose them if they occur at the wrong time.
> 
> I could use something like the edge_eoi flow handler instead I suppose
> but that will *never* lazy disable which is somewhat unfortunate, very
> fast paced MSIs benefit from being HW masked if we already recorded
> that they occurred.
> 
> So what I would need is something along the line of ONESHOT as done in
> fasteoi but only for level interrupts.

If that MSI interrupt is using a separate interrupt chip, then the solution
is simple. You just have to add IRQCHIP_ONESHOT_SAFE to chip.flags

Thanks,

	tglx