Re: modify srq pyverbs test (new?) failing for the rxe driver

Re: modify srq pyverbs test (new?) failing for the rxe driver

[Bob Pearson]

On 10/23/23 17:47, Bob Pearson wrote:
> Edward,
> 
> There is a test which may be new since this failure didn't happen
> before. Modify srq tries to set max_wr to 4094 but the rxe driver
> rounds this up to 2^n-1 or 4095. My understanding of the IBA
> spec is that queue sizes can be set larger than requested.
> Also this same test tries to change max_sge using the same
> MAX_WR mask bit. There is no mention (as far as I can recall)
> in the IBA spec of being able to change the max_sge setting.
> 
> Is this really the correct behavior?
> 
> Bob Pearson

Just for reference

11.2.3.3 MODIFY SHARED RECEIVE QUEUE
Description:
   Modifies the attributes of an SRQ for the specified HCA.
   If any of the modify attributes are invalid, none of the attributes
   shall be modified.
   Input Modifiers:
     •HCA handle.
     •SRQ handle.
     •The SRQ attributes to modify and their new values. The SRQ at-
     tributes that can be modified after the SRQ has been created are:
       •The maximum number of outstanding Work Requests the
       Consumer expects to submit to the Shared Receive Queue, if
       resizing of the SRQ is supported by the HCA.

[It does *not* include max_sge.]

       •SRQ Limit. If the SRQ Limit is greater than zero, then it shall
       be armed upon returning from this verb.
   Output Modifiers:
     •The actual number of outstanding Work Requests supported on
     the Shared Receive Queue. If an error is not returned, this is
     guaranteed to be greater than or equal to the number requested.
     (This may require the Consumer to increase the size of the CQ.)

[Unfortunately the rdma verbs API does not support returning the
new value of max_wr. You have to call ib_query_srq to get the new
value which does *not* have to equal the requested size.]

     •Verb Results:
     •Operation completed successfully.
     •Insufficient resources to complete request.
     •Invalid HCA handle.
     •Invalid SRQ handle.
     •SRQ is in the Error State.
     •HCA does not support resizing SRQ.
     •Maximum number of Work Requests requested exceeds HCA
     capability.
     •SRQ Limit exceeds maximum number of Work Requests al-
     lowed on the SRQ.
     •More outstanding entries on WQ than size specified.
     •HCA does not support SRQ.

[The test_modify_srq just seems incorrect.]

Re: modify srq pyverbs test (new?) failing for the rxe driver

[Edward Srouji]

You're right.
I've patched the test and pushed it Upstream (see PR link below [1]).
Please notify me if you're still having an issue after applying the patch.

On 10/24/2023 8:47 PM, Bob Pearson wrote:
> External email: Use caution opening links or attachments
>
>
> On 10/23/23 17:47, Bob Pearson wrote:
>> Edward,
>>
>> There is a test which may be new since this failure didn't happen
>> before. Modify srq tries to set max_wr to 4094 but the rxe driver
>> rounds this up to 2^n-1 or 4095. My understanding of the IBA
>> spec is that queue sizes can be set larger than requested.
>> Also this same test tries to change max_sge using the same
>> MAX_WR mask bit. There is no mention (as far as I can recall)
>> in the IBA spec of being able to change the max_sge setting.
>>
>> Is this really the correct behavior?
>>
>> Bob Pearson
>
> Just for reference
>
> 11.2.3.3 MODIFY SHARED RECEIVE QUEUE
> Description:
>   Modifies the attributes of an SRQ for the specified HCA.
>   If any of the modify attributes are invalid, none of the attributes
>   shall be modified.
>   Input Modifiers:
>     •HCA handle.
>     •SRQ handle.
>     •The SRQ attributes to modify and their new values. The SRQ at-
>     tributes that can be modified after the SRQ has been created are:
>       •The maximum number of outstanding Work Requests the
>       Consumer expects to submit to the Shared Receive Queue, if
>       resizing of the SRQ is supported by the HCA.
>
> [It does *not* include max_sge.]
>
>       •SRQ Limit. If the SRQ Limit is greater than zero, then it shall
>       be armed upon returning from this verb.
>   Output Modifiers:
>     •The actual number of outstanding Work Requests supported on
>     the Shared Receive Queue. If an error is not returned, this is
>     guaranteed to be greater than or equal to the number requested.
>     (This may require the Consumer to increase the size of the CQ.)
>
> [Unfortunately the rdma verbs API does not support returning the
> new value of max_wr. You have to call ib_query_srq to get the new
> value which does *not* have to equal the requested size.]
>
>     •Verb Results:
>     •Operation completed successfully.
>     •Insufficient resources to complete request.
>     •Invalid HCA handle.
>     •Invalid SRQ handle.
>     •SRQ is in the Error State.
>     •HCA does not support resizing SRQ.
>     •Maximum number of Work Requests requested exceeds HCA
>     capability.
>     •SRQ Limit exceeds maximum number of Work Requests al-
>     lowed on the SRQ.
>     •More outstanding entries on WQ than size specified.
>     •HCA does not support SRQ.
>
> [The test_modify_srq just seems incorrect.]

[1] https://github.com/linux-rdma/rdma-core/pull/1402

Thanks.

Re: modify srq pyverbs test (new?) failing for the rxe driver

[Bob Pearson]

On 10/25/23 06:29, Edward Srouji wrote:
> You're right.
> I've patched the test and pushed it Upstream (see PR link below [1]).
> Please notify me if you're still having an issue after applying the patch.

Will do. Thanks!

Bob
> 
> On 10/24/2023 8:47 PM, Bob Pearson wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> On 10/23/23 17:47, Bob Pearson wrote:
>>> Edward,
>>>
>>> There is a test which may be new since this failure didn't happen
>>> before. Modify srq tries to set max_wr to 4094 but the rxe driver
>>> rounds this up to 2^n-1 or 4095. My understanding of the IBA
>>> spec is that queue sizes can be set larger than requested.
>>> Also this same test tries to change max_sge using the same
>>> MAX_WR mask bit. There is no mention (as far as I can recall)
>>> in the IBA spec of being able to change the max_sge setting.
>>>
>>> Is this really the correct behavior?
>>>
>>> Bob Pearson

Re: modify srq pyverbs test (new?) failing for the rxe driver

[Bob Pearson]

On 10/25/23 06:29, Edward Srouji wrote:
> You're right.
> I've patched the test and pushed it Upstream (see PR link below [1]).
> Please notify me if you're still having an issue after applying the patch.
> 

Works fine now. Thanks.

Bob