Re: [PATCH] IB/srp: Fix initiator lockup

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Jan 13, 2010 at 07:57:26PM +0100, Bart Van Assche wrote:
> On Wed, Jan 13, 2010 at 7:16 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> >
> > On Wed, Jan 13, 2010 at 08:23:27AM +0100, Bart Van Assche wrote:
> > > On Wed, Jan 13, 2010 at 12:24 AM, Jason Gunthorpe
> > > <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > > > [ ... ]
> > > >
> > > > Also, I couldn't tell for sure from a cursory examination of the
> > > > patch, but the initiator must be designed to not stall processing the
> > > > RQ dependent on the SQ or the credit level, when using a credit scheme
> > > > like this. Or you will deadlock.
> > > >
> > > > For instance it isn't clear to me how the control flow around
> > > > srp_process_cred_req is ment to work - it tries to send a reply, but
> > > > if it can't due to srp_get_txp_iu failing it just gives up forever?
> > >
> > > For a standards-conforming SRP target, srp_get_txp_iu() will never
> > > fail. A quote from section 5.5.2 of the SRP r16a document:
> >
> > What guarantees that that the send completion for the reply is
> > processed before a receive completion for the next request?
> 
> Did you notice that I increased the number of receive slots reserved
> for target-initiated requests from one to two ?

Yes, but I don't see how more receive slots helps exhausting send
slots.

The IBA doesn't guarentee ordering between completions originating
from different work queues. Getting a receive completion does not
guarantee that a related send completion has been received. Even if
the work queues are mapped to the same CQ.

Since srp_get_txp_iu depends on txp_head/tail which is incremented by
send completions, but srp_get_txp_iu is called on the receive
completion path, it is possible to call srp_get_txp_iu multiple times
before the send completions are generated - even though the target is
only generating new request sends in response to reply receives.

The two main causes are ack coalescing algorithms and lost ack
packets.

That said, with SRP_TXP_SIZE set to 2, I think it would be really hard
to break this in real life, with current hardware.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html