Re: About a shortcoming of the verbs API

Bart Van Assche
2010-07-28T18:01:24Z

On Wed, Jul 28, 2010 at 7:51 PM, Ralph Campbell
<ralph.campbell-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org> wrote:
> On Wed, 2010-07-28 at 10:42 -0700, Roland Dreier wrote:
>> > - Some time ago I observed that the kernel reported soft lockups
>>  > because of spin_lock() calls inside a completion handler. These
>>  > spinlocks were not locked in any other context than the completion
>>  > handler itself. And the lockups disappeared after having replaced the
>>  > spin_lock() calls by spin_lock_irqsave(). Can it be concluded from
>>  > this observation that completion handlers are not always invoked from
>>  > interrupt context ?
>>
>> Did you get a soft lockup report or a lockdep report?  Anyway, the very
>> next paragraph of the documentation I quoted says:
>>
>>   The context in which completion event and asynchronous event
>>   callbacks run is not defined.  Depending on the low-level driver, it
>>   may be process context, softirq context, or interrupt context.
>>   Upper level protocol consumers may not sleep in a callback.
>>
>> So yes, it is possible that a completion callback gets called in
>> non-interrupt context.
>>
>> However as far as I know, at least mthca and mlx4 only call completion
>> callbacks from the interrupt handler.  But without the actual code in
>> question it's hard to know what the real problem was.
>
> Actually, I tried to implement the completion callback
> in a workqueue thread but ipoib_cm_handle_tx_wc() calls
> netif_tx_lock() which isn't safe unless it is called
> from an IRQ handler or netif_tx_lock_bh() is called first.

Has anyone already tried to enable threaded IRQ mode for IB completion
interrupts ? Threaded IRQs are one of the cornerstones of the
real-time Linux patch for reducing interrupt latency.

Bart.