Re: About a shortcoming of the verbs API

Bart Van Assche
2010-07-27T19:28:54Z

On Tue, Jul 27, 2010 at 8:20 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>
> On Tue, Jul 27, 2010 at 08:03:25PM +0200, Bart Van Assche wrote:
>
> > As far as I know it is not possible for a HCA to tell whether or not a
> > CPU has finished executing the interrupt it triggered. So it is not
> > possible for the HCA to implement the above requirement by delaying
> > the generation of a new interrupt -- implementing the above
>
> Linux does not allow interrupts to re-enter.. Read through
> kernel/irq/chip.c handle_edge_irq to get a sense of how that is done
> for MSI. Looked to me like all the CQ call backs flowed from the
> interrupt handler in mlx4?

Thanks for the feedback -- I had tried to look up that information but
hadn't found it yet.

I have two more questions:
- Some time ago I observed that the kernel reported soft lockups
because of spin_lock() calls inside a completion handler. These
spinlocks were not locked in any other context than the completion
handler itself. And the lockups disappeared after having replaced the
spin_lock() calls by spin_lock_irqsave(). Can it be concluded from
this observation that completion handlers are not always invoked from
interrupt context ?
- The function handle_edge_irq() in kernel/irq/chip.c invokes the
actual interrupt handler while the spinlock desc->lock is not locked.
Does that mean that a completion interrupt can get lost due to the
following (unlikely) event order ?
* completion interrupt is triggered on CPU 1.
* handle_edge_irq() sets the IRQ_INPROGRESS flag and invokes the
completion handler on CPU 1.
* The completion handler reenables the completion interrupt via
ib_req_notify_cq().
* Before the IRQ_INPROGRESS flag is cleared by handle_edge_irq(), a
new completion interrupt is triggered on CPU 2.
* handle_edge_irq() is invoked on CPU 2 and exits immediately because
IRQ_INPROGRESS is still set.
* handle_edge_irq() clears IRQ_INPROGRESS on CPU 1.

Bart.