[SPDK] [NVMf] Invalid SQSIZE 128

[SPDK] [NVMf] Invalid SQSIZE 128

[Alex Bowden]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

It seems SQSIZE in linux-nvmf is by default set to 128 and seems to be 127 in the NVMf code, but I think the RDMA driver does a +1 to it? This causes SPDK to not accept the cmd.

https://github.com/spdk/spdk/commit/a5be12c376f42108085bd9cfd06fc27818fdb6c9

I suggest either changing this to ">" instead of ">=" or changing the nvmf.conf.in<http://nvmf.conf.in> MaxQueueDepth size to 129 by default.

Here's the error msg:
***ERROR*** Invalid SQSIZE 128 (min 1, max 127)

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 864 bytes --]

Re: [SPDK] [NVMf] Invalid SQSIZE 128

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

On Wed, 2016-09-28 at 22:11 +0000, Alex Bowden wrote:
> It seems SQSIZE in linux-nvmf is by default set to 128 and seems to be 127 in
> the NVMf code, but I think the RDMA driver does a +1 to it? This causes SPDK
> to not accept the cmd.
> 

You are using an old version of the kernel initiator. We reported this bug to
the kernel
developers and it is now fixed in the latest version (4.8).  SPDK was doing this
correctly,
but the kernel was not treating the value as 0's based.

> 
> https://github.com/spdk/spdk/commit/a5be12c376f42108085bd9cfd06fc27818fdb6c9
> 
> 
> 
> I suggest either changing this to ">" instead of ">=" or changing the
> nvmf.conf.in MaxQueueDepth size to 129 by default.
> 
> 
> Here's the error msg:
> ***ERROR*** Invalid SQSIZE 128 (min 1, max 127)
> 
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] [NVMf] Invalid SQSIZE 128

[Daniel Verkamp]

[-- Attachment #1: Type: text/plain, Size: 1376 bytes --]

On 09/28/2016 03:32 PM, Walker, Benjamin wrote:
> On Wed, 2016-09-28 at 22:11 +0000, Alex Bowden wrote:
>> It seems SQSIZE in linux-nvmf is by default set to 128 and seems to be 127 in
>> the NVMf code, but I think the RDMA driver does a +1 to it? This causes SPDK
>> to not accept the cmd.
>>
>
> You are using an old version of the kernel initiator. We reported this bug to
> the kernel
> developers and it is now fixed in the latest version (4.8).  SPDK was doing this
> correctly,
> but the kernel was not treating the value as 0's based.
>
>>
>> https://github.com/spdk/spdk/commit/a5be12c376f42108085bd9cfd06fc27818fdb6c9
>>
>>
>>
>> I suggest either changing this to ">" instead of ">=" or changing the
>> nvmf.conf.in MaxQueueDepth size to 129 by default.
>>
>>
>> Here's the error msg:
>> ***ERROR*** Invalid SQSIZE 128 (min 1, max 127)

Specifically, the fix is in these Linux kernel commits:

commit c5af8654c422cfdd8480be3a244748e18cace6c5
     nvme-rdma: fix sqsize/hsqsize per spec

commit f994d9dc28bc27353acde2caaf718222d92a3e24
     fabrics: define admin sqsize min default, per spec

The buggy version of the SQSIZE code was never in a released kernel 
version, only Linux 4.8-rc4. The final Linux 4.8 release (which is the 
first kernel release with NVMe over Fabrics support) will contain the fix.

Thanks,
-- Daniel