Re: [PATCH for-next 0/2] RDMA/erdma: Introduce custom implementation of drain_sq and drain_rq

[Cheng Xu <chengyou@linux.alibaba.com>]

On 8/24/22 10:08 PM, Tom Talpey wrote:
> On 8/24/2022 5:42 AM, Cheng Xu wrote:
>> Hi,
>>
>> This series introduces erdma's implementation of drain_sq and drain_rq.
>> Our hardware will stop processing any new WRs if QP state is error.
> 
> Doesn't this violate the IB specification? Failing newly posted WRs
> before older WRs have flushed to the CQ means that ordering is not
> preserved.

I agree with Bernard's point.

I'm not very familiar with with IB specification. But for RNIC/iWarp [1],
post WR in Error state has two optional actions: "Post WQE, and then Flush it"
or "Return an Immediate Error" (Showed in Figure 10). So, I think failing
newly posted WRs is reasonable.

> Many upper layers depend on this.

For kernel verbs, erdma currently supports NoF. we tested the NoF cases,
and found that it's never happened that newly WRs were posted after QP
changed to error, and the drain_qp should be the terminal of IO process.

Also, in userspace, I find that many providers prevents new WRs if QP state is
not right.

So, it seems ok in practice.

Thanks,
Cheng Xu


[1] http://www.rdmaconsortium.org/home/draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf