Re: CX314A WCE error: WR_FLUSH_ERR

Doug Ledford
2019-08-21T18:47:07Z

[-- Attachment #1: Type: text/plain, Size: 4679 bytes --]

On Wed, 2019-08-21 at 23:38 +0800, Liu, Changcheng wrote:
> On 09:36 Wed 21 Aug, Tom Talpey wrote:
> > On 8/21/2019 8:09 AM, Liu, Changcheng wrote:
> > > Hi all,
> > > In one system, it always frequently hit "IBV_WC_WR_FLUSH_ERR"
> > > in the WCE(work completion element) polled from completion queue
> > > bound with RQ(Receive Queue).
> > > Does anyone has some idea to debug "IBV_WC_WR_FLUSH_ERR"
> > > problem?
> > >
> > > With CX314A/40Gb NIC, I hit this error when using RC transport
> > > type with only Send Operation(IBV_WR_SEND) WR(work request) on
> > > SQ(Send Queue).
> > > Every WR only has one SGE(scatter/gather element) and all the
> > > SGE on RQ has the same size. The SGE size in SQ WR is not greater
> > > than the SGE size in RQ WR.
> > >
> > > There’s one explanation about IBV_WC_WR_FLUSH_ERR on page 114
> > > in the "RDMA Aware Networks Programming User Manual"
> > > http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf
> > > But I still didn't understand it well. How to trigger this
> > > error with a short demo program?
> > > "
> > > IBV_WC_WR_FLUSH_ERR
> > > This event is generated when an invalid remote error is
> > > thrown when the responder detects an
> > > invalid request. It may be that the operation is not
> > > supported by the request queue or there is
> > > insufficient buffer space to receive the request.
> > > "
> >
> > The most common reason for a flushed work request is loss of
> > the connection to the remote peer. This can be caused by any
> > number of conditions.
> Good diretion. I'll debug it in this way first.
> > The second-most common is a programming error in the upper
> > layer protocol. A shortage of posted receives on either peer,
> > a protection error on some buffer, etc.
> Do you mean the protection key such as l_key/r_key isn't set well?
> What's kind of protection error could trigger IBV_WC_WR_FLUSH_ERR?

FLUSH_ERR is the error used whenever a queue pair goes into an error
state and there are still WQEs posted to the queue pair. All
outstanding WQEs are returned with the state IBV_WC_WR_FLUSH_ERR. This
is how you make sure you don't loose WQEs when the QP hits an error
state. So, literally *anything* that can cause a QP to go into an ERROR
state will result in all WQEs currently posted to the QP being sent back
with this FLUSH_ERR. FLUSH_ERR literally just means that the card is
flushing out the QP's work queue because now that the QP is in an error
state it can't process the WQEs and, presumably, the application needs
to know which ones completed and which ones didn't so it knows what to
requeue once the QP is no longer in an error state.

As Tom has already pointed out, all of these things will throw the queue
pair into an error state and cause all posted WQEs to be flushed with
the FLUSH_ERR condition:

1) Loss of queue pair connection
2) Any memory permission violation (attempt to write to read only
memory, attempt to RDMA read/write to an invalid rkey, etc)
3) Receipt of any post_send message without a waiting post_recv buffer
to accept the message
4) Receipt of a post_send message that is too large to fit in the first
available post_recv buffer

A common cause of this sort of thing is when you don't do proper flow
control on the queue pair and the sending side floods the receiving side
and runs it out of posted recv WQEs. Although, in your case, you did
say this was happening on the receive queue, so that implies this is
happening on the receiving side, so if that is what's happenining here,
the process would have to be something like:

sender starts sending data (maybe without any flow control)
receiver starts receiving data and refilling buffers
...
receiver runs totally dry of buffers and gets an incoming recv
causing qp to go into error state

receiver then posts refill buffers to the RQ after the QP
went into error state but before acknowledging the error state
and shutting down the recv processing thread

all recv buffers posted as WQEs are flushed back to the process
with FLUSH_ERR because they were posted to a QP in ERROR state

> > If you're looking to actually trigger this error for testing,
> > well, try one of the above. If you're trying to figure out
> > why it's happening, that can take some digging, but not in
> > the RDMA stack, typically.
> Many thanks.
>
> --Changcheng
> > Tom.
> >

--
Doug Ledford <dledford@redhat.com>
GPG KeyID: B826A3330E572FDD
Fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]