Re: [PATCH v2 12/12] IB/srp: Add multichannel support

[Sagi Grimberg <sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>]

On 11/5/2014 6:57 AM, Elliott, Robert (Server Storage) wrote:
>
>
>> -----Original Message-----
>> From: Sagi Grimberg [mailto:sagig-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org]
>> Sent: Tuesday, November 04, 2014 6:15 AM
>> To: Bart Van Assche; Elliott, Robert (Server Storage); Christoph Hellwig
>> Cc: Jens Axboe; Sagi Grimberg; Sebastian Parschauer; Ming Lei; linux-
>> scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-rdma
>> Subject: Re: [PATCH v2 12/12] IB/srp: Add multichannel support
>>
> ...
>> I think that Rob and I are not talking about the same issue. In
>> case only a single core is servicing interrupts it is indeed expected
>> that it will spend 100% in hard-irq, that's acceptable since it is
>> pounded with completions all the time.
>>
>> However, I'm referring to a condition where SRP will spend infinite
>> time servicing a single interrupt (while loop on ib_poll_cq that never
>> drains) which will lead to a hard lockup.
>>
>> This *can* happen, and I do believe that with an optimized IO path
>> it is even more likely to.
>
> If the IB completions/interrupts are only for IOs submitted on this
> CPU, then the CQ will eventually drain, because this CPU is not
> submitting anything new while stuck in the loop.

They're not (or not necessarily). I'm talking about the case where the
IO completions are submitted from another CPU. This creates a cycle
where the submitter is generating completions on CPU X and the completer
is evacuating room for more submissions on CPU Y. This process can
never end while the completer is in hard-irq context.

>
> This can become bursty, though - submit a lot of IOs, then be busy
> completing all of them and not submitting more, resulting in the
> queue depth bouncing from 0 to high to 0 to high.  I've seen
> that with both hpsa and mpt3sas drivers.  The fio options
> iodepth_batch, iodepth_batch_complete, and iodepth_low
> can amplify and reduce that effect (using libaio).
>

blk-iopoll (or some other form of budgeting completions) should take
care of that.

Sagi.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html