RE: [PATCH v3 05/11] xprtrdma: Do not wait if ib_post_send() fails

["Steve Wise" <swise@opengridcomputing.com>]

> > On Mar 10, 2016, at 10:04 AM, Steve Wise <swise@opengridcomputing.com>
> wrote:
> >
> >>>> Moving the QP into error state right after with rdma_disconnect
> >>>> you are not sure that none of the subset of the invalidations
> >>>> that _were_ posted completed and you get the corresponding MRs
> >>>> in a bogus state...
> >>>
> >>> Moving the QP to error state and then draining the CQs means
> >>> that all LOCAL_INV WRs that managed to get posted will get
> >>> completed or flushed. That's already handled today.
> >>>
> >>> It's the WRs that didn't get posted that I'm worried about
> >>> in this patch.
> >>>
> >>> Are there RDMA consumers in the kernel that use that third
> >>> argument to recover when LOCAL_INV WRs cannot be posted?
> >>
> >> None :)
> >>
> >>>>> I suppose I could reset these MRs instead (that is,
> >>>>> pass them to ib_dereg_mr).
> >>>>
> >>>> Or, just wait for a completion for those that were posted
> >>>> and then all the MRs are in a consistent state.
> >>>
> >>> When a LOCAL_INV completes with IB_WC_SUCCESS, the associated
> >>> MR is in a known state (ie, invalid).
> >>>
> >>> The WRs that flush mean the associated MRs are not in a known
> >>> state. Sometimes the MR state is different than the hardware
> >>> state, for example. Trying to do anything with one of these
> >>> inconsistent MRs results in IB_WC_BIND_MW_ERR until the thing
> >>> is deregistered.
> >>
> >> Correct.
> >>
> >
> > It is legal to invalidate an MR that is not in the valid state.  So you
don't
> > have to deregister it, you can assume it is valid and post another LINV WR.
> 
> I've tried that. Once the MR is inconsistent, even LOCAL_INV
> does not work.
>

Maybe IB Verbs don't mandate that invalidating an invalid MR must be allowed?
(looking at the verbs spec now).