Work completions generated after a queue pair has made the transition to an error state

Work completions generated after a queue pair has made the transition to an error state

[Bart Van Assche]

Hello,

Has anyone already tried to process the work completions generated by
a HCA after the state of a queue pair has been changed to IB_QPS_ERR ?
With the hardware/firmware/driver combination I have tested I have
observed the following:
* Multiple completions with the same wr_id and nonzero (error) status
were received by the application, while all work requests queued with
the flag IB_SEND_SIGNALED had a unique wr_id.
* Completions with non-zero (error) status and a wr_id / opcode
combination were received that were never queued by the application.
Note: some work requests were queued with and some without the flag
IB_SEND_SIGNALED. I'm not sure however whether that has anything to do
with the observed behavior.

This behavior is easy to reproduce. If I interpret the InfiniBand
Architecture Specification correctly, this behavior is non-compliant.

Has anyone been looking into this before ?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Ralph Campbell]

I haven't seen it. It isn't supposed to happen.

What hardware and software are you using and how do you
reproduce it?


On Tue, 2010-10-12 at 11:38 -0700, Bart Van Assche wrote:
> Hello,
> 
> Has anyone already tried to process the work completions generated by
> a HCA after the state of a queue pair has been changed to IB_QPS_ERR ?
> With the hardware/firmware/driver combination I have tested I have
> observed the following:
> * Multiple completions with the same wr_id and nonzero (error) status
> were received by the application, while all work requests queued with
> the flag IB_SEND_SIGNALED had a unique wr_id.
> * Completions with non-zero (error) status and a wr_id / opcode
> combination were received that were never queued by the application.
> Note: some work requests were queued with and some without the flag
> IB_SEND_SIGNALED. I'm not sure however whether that has anything to do
> with the observed behavior.
> 
> This behavior is easy to reproduce. If I interpret the InfiniBand
> Architecture Specification correctly, this behavior is non-compliant.
> 
> Has anyone been looking into this before ?
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Bart Van Assche]

On Tue, Oct 12, 2010 at 8:50 PM, Ralph Campbell
<ralph.campbell-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org> wrote:
> On Tue, 2010-10-12 at 11:38 -0700, Bart Van Assche wrote:
>> Hello,
>>
>> Has anyone already tried to process the work completions generated by
>> a HCA after the state of a queue pair has been changed to IB_QPS_ERR ?
>> With the hardware/firmware/driver combination I have tested I have
>> observed the following:
>> * Multiple completions with the same wr_id and nonzero (error) status
>> were received by the application, while all work requests queued with
>> the flag IB_SEND_SIGNALED had a unique wr_id.
>> * Completions with non-zero (error) status and a wr_id / opcode
>> combination were received that were never queued by the application.
>> Note: some work requests were queued with and some without the flag
>> IB_SEND_SIGNALED. I'm not sure however whether that has anything to do
>> with the observed behavior.
>>
>> This behavior is easy to reproduce. If I interpret the InfiniBand
>> Architecture Specification correctly, this behavior is non-compliant.
>>
>> Has anyone been looking into this before ?
>
> I haven't seen it. It isn't supposed to happen.
>
> What hardware and software are you using and how do you
> reproduce it?

Hello Ralph and Or,

The way I reproduce that behavior is by modifying the state of a queue
pair into IB_QPS_ERR while RDMA is ongoing. The application, which is
multithreaded, performs RDMA by calling ib_post_recv() and
ib_post_send() (opcodes IB_WR_SEND, IB_WR_RDMA_READ and
IB_WR_RDMA_WRITE). This has been observed with the mlx4 driver, a
ConnectX HCA and firmware version 2.7.0.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Eli Cohen]

On Tue, Oct 12, 2010 at 08:58:59PM +0200, Bart Van Assche wrote:
> On Tue, Oct 12, 2010 at 8:50 PM, Ralph Campbell
> <ralph.campbell-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org> wrote:
> > On Tue, 2010-10-12 at 11:38 -0700, Bart Van Assche wrote:
> >> Hello,
> >>
> >> Has anyone already tried to process the work completions generated by
> >> a HCA after the state of a queue pair has been changed to IB_QPS_ERR ?
> >> With the hardware/firmware/driver combination I have tested I have
> >> observed the following:
> >> * Multiple completions with the same wr_id and nonzero (error) status
> >> were received by the application, while all work requests queued with
> >> the flag IB_SEND_SIGNALED had a unique wr_id.
I assume your QP is configured for selective signalling, right? This
means that for succcessful processing of the work request there will 
not be  any completion. But for unsuccessful WR, the hardware should
generate a completion. For these casese it is worth having a 
meaningfull wrid.
> >> * Completions with non-zero (error) status and a wr_id / opcode
> >> combination were received that were never queued by the application.
In case of error the opcode of the completed operation is not provided.
I am not sure why.

> >> Note: some work requests were queued with and some without the flag
> >> IB_SEND_SIGNALED. I'm not sure however whether that has anything to do
> >> with the observed behavior.
If you have WRs for which you did not set IB_SEND_SIGNALED, they are
not considered completed before a comletion entry is pushed to the CQ
that correspnds to that send queue. I am not sure if it means that all
the WR in the send queue should be completed with error.
> >>
> >> This behavior is easy to reproduce. If I interpret the InfiniBand
> >> Architecture Specification correctly, this behavior is non-compliant.
> >>
> >> Has anyone been looking into this before ?
> >
> > I haven't seen it. It isn't supposed to happen.
> >
> > What hardware and software are you using and how do you
> > reproduce it?
> 
> Hello Ralph and Or,
> 
> The way I reproduce that behavior is by modifying the state of a queue
> pair into IB_QPS_ERR while RDMA is ongoing. The application, which is
> multithreaded, performs RDMA by calling ib_post_recv() and
> ib_post_send() (opcodes IB_WR_SEND, IB_WR_RDMA_READ and
> IB_WR_RDMA_WRITE). This has been observed with the mlx4 driver, a
> ConnectX HCA and firmware version 2.7.0.
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Eli Cohen]

libmlx4 does not report the opcode of the operation if it completed
with error. We will fix this and post a patch for review.

On Tue, Oct 12, 2010 at 10:22 PM, Eli Cohen <eli-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> wrote:
> On Tue, Oct 12, 2010 at 08:58:59PM +0200, Bart Van Assche wrote:
>> On Tue, Oct 12, 2010 at 8:50 PM, Ralph Campbell
>> <ralph.campbell-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org> wrote:
>> > On Tue, 2010-10-12 at 11:38 -0700, Bart Van Assche wrote:
>> >> Hello,
>> >>
>> >> Has anyone already tried to process the work completions generated by
>> >> a HCA after the state of a queue pair has been changed to IB_QPS_ERR ?
>> >> With the hardware/firmware/driver combination I have tested I have
>> >> observed the following:
>> >> * Multiple completions with the same wr_id and nonzero (error) status
>> >> were received by the application, while all work requests queued with
>> >> the flag IB_SEND_SIGNALED had a unique wr_id.
> I assume your QP is configured for selective signalling, right? This
> means that for succcessful processing of the work request there will
> not be  any completion. But for unsuccessful WR, the hardware should
> generate a completion. For these casese it is worth having a
> meaningfull wrid.
>> >> * Completions with non-zero (error) status and a wr_id / opcode
>> >> combination were received that were never queued by the application.
> In case of error the opcode of the completed operation is not provided.
> I am not sure why.
>
>> >> Note: some work requests were queued with and some without the flag
>> >> IB_SEND_SIGNALED. I'm not sure however whether that has anything to do
>> >> with the observed behavior.
> If you have WRs for which you did not set IB_SEND_SIGNALED, they are
> not considered completed before a comletion entry is pushed to the CQ
> that correspnds to that send queue. I am not sure if it means that all
> the WR in the send queue should be completed with error.
>> >>
>> >> This behavior is easy to reproduce. If I interpret the InfiniBand
>> >> Architecture Specification correctly, this behavior is non-compliant.
>> >>
>> >> Has anyone been looking into this before ?
>> >
>> > I haven't seen it. It isn't supposed to happen.
>> >
>> > What hardware and software are you using and how do you
>> > reproduce it?
>>
>> Hello Ralph and Or,
>>
>> The way I reproduce that behavior is by modifying the state of a queue
>> pair into IB_QPS_ERR while RDMA is ongoing. The application, which is
>> multithreaded, performs RDMA by calling ib_post_recv() and
>> ib_post_send() (opcodes IB_WR_SEND, IB_WR_RDMA_READ and
>> IB_WR_RDMA_WRITE). This has been observed with the mlx4 driver, a
>> ConnectX HCA and firmware version 2.7.0.
>>
>> Bart.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Or Gerlitz]

Eli Cohen wrote:
> Completions with non-zero (error) status and a wr_id / opcode 
> combination were received that were never queued by the application.
> In case of error the opcode of the completed operation is not provided. I am not sure why.
Eli, there's nothing in the IB spec that mandates the WC.opcode of a non 
successful work request to be valid, the only WC fields that must be 
valid are the work-request ID (cookie) and the status code, I believe 
that hardware vendors would also make sure to have the vendor id valid...

Bart, reading your initial posting, I was under the impression that the 
wr_id is something your app didn't post, so in that respect I take back 
my response, so, of-course, when you program to IB you can't assume 
anything on WC.opcode of an error-ed WR.

Or.


>
>>>> Note: some work requests were queued with and some without the flag
>>>> IB_SEND_SIGNALED. I'm not sure however whether that has anything to do
>>>> with the observed behavior.
> If you have WRs for which you did not set IB_SEND_SIGNALED, they are
> not considered completed before a comletion entry is pushed to the CQ
> that correspnds to that send queue. I am not sure if it means that all
> the WR in the send queue should be completed with error.
>>>> This behavior is easy to reproduce. If I interpret the InfiniBand
>>>> Architecture Specification correctly, this behavior is non-compliant.
>>>>
>>>> Has anyone been looking into this before ?
>>> I haven't seen it. It isn't supposed to happen.
>>>
>>> What hardware and software are you using and how do you
>>> reproduce it?
>> Hello Ralph and Or,
>>
>> The way I reproduce that behavior is by modifying the state of a queue
>> pair into IB_QPS_ERR while RDMA is ongoing. The application, which is
>> multithreaded, performs RDMA by calling ib_post_recv() and
>> ib_post_send() (opcodes IB_WR_SEND, IB_WR_RDMA_READ and
>> IB_WR_RDMA_WRITE). This has been observed with the mlx4 driver, a
>> ConnectX HCA and firmware version 2.7.0.
>>
>> Bart.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Eli Cohen]

On Wed, Oct 13, 2010 at 03:51:10PM +0200, Or Gerlitz wrote:
> Eli Cohen wrote:
> >Completions with non-zero (error) status and a wr_id / opcode
> >combination were received that were never queued by the
> >application.
> >In case of error the opcode of the completed operation is not provided. I am not sure why.
> Eli, there's nothing in the IB spec that mandates the WC.opcode of a
> non successful work request to be valid, the only WC fields that
> must be valid are the work-request ID (cookie) and the status code,
> I believe that hardware vendors would also make sure to have the
> vendor id valid...

Maybe I am misinterpreting the spec. Looking at volume 1.2.1 of the
spec, 11.4.2.1, it says:

"If the status of the operation that generates the Work Completion is
anything other than success, the contents of the Work Completion are
undefined except as noted below. The contents of a Work Completion
are:"

Then it lists the reported fileds with wrid first and the opcodes
following.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Roland Dreier]

 > "If the status of the operation that generates the Work Completion is
 > anything other than success, the contents of the Work Completion are
 > undefined except as noted below. The contents of a Work Completion
 > are:"

Yes, that is exactly right.  And the "noted below" fields are the status
and the WR ID.  Everything else is undefined if the status is not success.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Eli Cohen]

On Wed, Oct 13, 2010 at 09:05:47AM -0700, Roland Dreier wrote:
>  > "If the status of the operation that generates the Work Completion is
>  > anything other than success, the contents of the Work Completion are
>  > undefined except as noted below. The contents of a Work Completion
>  > are:"
> 
> Yes, that is exactly right.  And the "noted below" fields are the status
> and the WR ID.  Everything else is undefined if the status is not success.
> 
I see... those that have "This is always valid".
Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Roland Dreier]

I'm not clear on the problem observed here.  A few notes:

 - If a QP transitions to error state, then *all* work requests, whether
   or not they were signaled, generate a completion with status "flush".

 - If a work request completes with an error status, then the opcode
   field is not defined.

With that understood, can you explain what part of the IB spec is not
being implemented correctly?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Bart Van Assche]

On Wed, Oct 13, 2010 at 6:18 PM, Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
>
> I'm not clear on the problem observed here.  A few notes:
>
>  - If a QP transitions to error state, then *all* work requests, whether
>   or not they were signaled, generate a completion with status "flush".
>
>  - If a work request completes with an error status, then the opcode
>   field is not defined.
>
> With that understood, can you explain what part of the IB spec is not
> being implemented correctly?

I was posting my question to the this mailing list because the spec is
somewhat cryptic with regard to which fields are defined and which
ones not if the error status is set. Thanks for taking the time to
clarify the specs.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Work completions generated after a queue pair has made the transition to an error state

[Or Gerlitz]

Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org> wrote:
> Has anyone been looking into this before ?

nope, never ever, what hca is that?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html