[SPDK] error while submitting readv cmd

[SPDK] error while submitting readv cmd

[txcy uio]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

Hello,

While submitting a readv command (spdk_nvme_ns_cmd_readv) I see the
following error:

READ sqid:3 cid:127 nsid:1 lba:100 len:64
INVALID_FIELD (00/02) sqid:3 cid:127 cdw0:0 sqhd:0000 p:0 m:0 dnr:1

Based on this error what can I infer about the command that went wrong?

--Tyc

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 422 bytes --]

Re: [SPDK] error while submitting readv cmd

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

On Mon, 2016-08-01 at 16:50 -0700, txcy uio wrote:
> Hello,
> 
> While submitting a readv command (spdk_nvme_ns_cmd_readv) I see the following error:
> 
> READ sqid:3 cid:127 nsid:1 lba:100 len:64
> INVALID_FIELD (00/02) sqid:3 cid:127 cdw0:0 sqhd:0000 p:0 m:0 dnr:1
> 
> 
> Based on this error what can I infer about the command that went wrong?

The SPDK NVMe driver is zero-copy, meaning we program the hardware to transfer data directly from
the buffers you specify. That's why all of the buffers need to be allocated from pinned memory (in
practice, that usually means they need to be allocated using rte_malloc from DPDK). NVMe devices
also have physical alignment restrictions. For instance, version 1.0 of the specification requires
us to program the hardware using a "physical resource page list" (PRP list), which consists of a
list of 4k physical memory pages. The first page does not have to be 4k aligned, but all others do,
and all but the first and last page must be 4k in length. Version 1.2 of the NVMe specification
optionally allows fully flexible scatter gather lists, but almost no commercially available devices
support scatter gather lists yet.

spdk_nvme_ns_cmd_readv is our interface for using the NVMe 1.2 scatter gather list capabilities. If your device doesn't support that, it's going to fail with an error like you are seeing. I now see that the purpose of spdk_nvme_ns_cmd_readv isn't clearly specified in the documentation (http://www.spdk.io/spdk/doc/nvme_8h.html#acb47ce7de6b6e963ec9fb8de261466ae) and if you aren't intimately familiar with the NVMe specification I can definitely see how you wouldn't realize that scatter gather lists require special hardware support. The documentation needs to be updated.

> 
> --Tyc
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] error while submitting readv cmd

[Bhadauria, Varun]

[-- Attachment #1: Type: text/plain, Size: 2754 bytes --]

Hello Benjamin,

Is there a way of knowing if the H/W supports SGL by looking at the cmd supports flags of the namespace, similar to SPDK_NVME_NS_FLUSH_SUPPORTED and others?

Also if the H/W doesn’t support a specific command like FLUSH, TRIM, READV, WRITEV , the corresponding spdk_nvme_ns_cmd_* () should return ENOTSUP as an error which can be handled appropriately in the caller?
For failures due to non-alignment or memory not being pinned appropriate errors should be returned as well.

-Varun







On 8/2/16, 8:56 AM, "SPDK on behalf of Walker, Benjamin" <spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:

>On Mon, 2016-08-01 at 16:50 -0700, txcy uio wrote:
>> Hello,
>> 
>> While submitting a readv command (spdk_nvme_ns_cmd_readv) I see the following error:
>> 
>> READ sqid:3 cid:127 nsid:1 lba:100 len:64
>> INVALID_FIELD (00/02) sqid:3 cid:127 cdw0:0 sqhd:0000 p:0 m:0 dnr:1
>> 
>> 
>> Based on this error what can I infer about the command that went wrong?
>
>The SPDK NVMe driver is zero-copy, meaning we program the hardware to transfer data directly from
>the buffers you specify. That's why all of the buffers need to be allocated from pinned memory (in
>practice, that usually means they need to be allocated using rte_malloc from DPDK). NVMe devices
>also have physical alignment restrictions. For instance, version 1.0 of the specification requires
>us to program the hardware using a "physical resource page list" (PRP list), which consists of a
>list of 4k physical memory pages. The first page does not have to be 4k aligned, but all others do,
>and all but the first and last page must be 4k in length. Version 1.2 of the NVMe specification
>optionally allows fully flexible scatter gather lists, but almost no commercially available devices
>support scatter gather lists yet.
>
>spdk_nvme_ns_cmd_readv is our interface for using the NVMe 1.2 scatter gather list capabilities. If your device doesn't support that, it's going to fail with an error like you are seeing. I now see that the purpose of spdk_nvme_ns_cmd_readv isn't clearly specified in the documentation (http://www.spdk.io/spdk/doc/nvme_8h.html#acb47ce7de6b6e963ec9fb8de261466ae) and if you aren't intimately familiar with the NVMe specification I can definitely see how you wouldn't realize that scatter gather lists require special hardware support. The documentation needs to be updated.
>
>> 
>> --Tyc
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] error while submitting readv cmd

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4100 bytes --]

On Tue, 2016-08-02 at 17:07 +0000, Bhadauria, Varun wrote:
> Hello Benjamin,
> 
> Is there a way of knowing if the H/W supports SGL by looking at the cmd
> supports flags of the namespace, similar to SPDK_NVME_NS_FLUSH_SUPPORTED and
> others?

You can determine if the hardware supports SGLs by calling
spdk_nvme_ctrlr_get_data and then looking at the sgls.supported field of the
resulting data structure, as per the NVMe specification.

> 
> Also if the H/W doesn’t support a specific command like FLUSH, TRIM, READV,
> WRITEV , the corresponding spdk_nvme_ns_cmd_* () should return ENOTSUP as an
> error which can be handled appropriately in the caller?
> For failures due to non-alignment or memory not being pinned appropriate
> errors should be returned as well.

Agreed - the errors should be clearer. We'd accept patches for any improvements
to error reporting. I have some longer term plans to make memory allocation and
alignment much more explicit than it is today which should help clear up
confusion. Right now I know some of the requirements are magic unless you are
fluent with the NVMe specification.


One other thing - I was mistaken before when I said that spdk_nvme_ns_cmd_readv
can only be used with devices that support SGL. That interface can be used with
devices that don't support full SGLs, but you have to only specify scatter
gather lists that can be translated into PRP lists or it will fail with the
error below.
> 
> -Varun
> 
> 
> 
> 
> 
> 
> 
> On 8/2/16, 8:56 AM, "SPDK on behalf of Walker, Benjamin" <spdk-bounces(a)lists.0
> 1.org on behalf of benjamin.walker(a)intel.com> wrote:
> 
> > 
> > On Mon, 2016-08-01 at 16:50 -0700, txcy uio wrote:
> > > 
> > > Hello,
> > > 
> > > While submitting a readv command (spdk_nvme_ns_cmd_readv) I see the
> > > following error:
> > > 
> > > READ sqid:3 cid:127 nsid:1 lba:100 len:64
> > > INVALID_FIELD (00/02) sqid:3 cid:127 cdw0:0 sqhd:0000 p:0 m:0 dnr:1
> > > 
> > > 
> > > Based on this error what can I infer about the command that went wrong?
> > 
> > The SPDK NVMe driver is zero-copy, meaning we program the hardware to
> > transfer data directly from
> > the buffers you specify. That's why all of the buffers need to be allocated
> > from pinned memory (in
> > practice, that usually means they need to be allocated using rte_malloc from
> > DPDK). NVMe devices
> > also have physical alignment restrictions. For instance, version 1.0 of the
> > specification requires
> > us to program the hardware using a "physical resource page list" (PRP list),
> > which consists of a
> > list of 4k physical memory pages. The first page does not have to be 4k
> > aligned, but all others do,
> > and all but the first and last page must be 4k in length. Version 1.2 of the
> > NVMe specification
> > optionally allows fully flexible scatter gather lists, but almost no
> > commercially available devices
> > support scatter gather lists yet.
> > 
> > spdk_nvme_ns_cmd_readv is our interface for using the NVMe 1.2 scatter
> > gather list capabilities. If your device doesn't support that, it's going to
> > fail with an error like you are seeing. I now see that the purpose of
> > spdk_nvme_ns_cmd_readv isn't clearly specified in the documentation (http://
> > www.spdk.io/spdk/doc/nvme_8h.html#acb47ce7de6b6e963ec9fb8de261466ae) and if
> > you aren't intimately familiar with the NVMe specification I can definitely
> > see how you wouldn't realize that scatter gather lists require special
> > hardware support. The documentation needs to be updated.
> > 
> > > 
> > > 
> > > --Tyc
> > > 
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] error while submitting readv cmd

[Bhadauria, Varun]

[-- Attachment #1: Type: text/plain, Size: 4524 bytes --]

Thank you Benjamin.

-Varun








On 8/2/16, 10:21 AM, "SPDK on behalf of Walker, Benjamin" <spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:

>On Tue, 2016-08-02 at 17:07 +0000, Bhadauria, Varun wrote:
>> Hello Benjamin,
>> 
>> Is there a way of knowing if the H/W supports SGL by looking at the cmd
>> supports flags of the namespace, similar to SPDK_NVME_NS_FLUSH_SUPPORTED and
>> others?
>
>You can determine if the hardware supports SGLs by calling
>spdk_nvme_ctrlr_get_data and then looking at the sgls.supported field of the
>resulting data structure, as per the NVMe specification.
>
>> 
>> Also if the H/W doesn’t support a specific command like FLUSH, TRIM, READV,
>> WRITEV , the corresponding spdk_nvme_ns_cmd_* () should return ENOTSUP as an
>> error which can be handled appropriately in the caller?
>> For failures due to non-alignment or memory not being pinned appropriate
>> errors should be returned as well.
>
>Agreed - the errors should be clearer. We'd accept patches for any improvements
>to error reporting. I have some longer term plans to make memory allocation and
>alignment much more explicit than it is today which should help clear up
>confusion. Right now I know some of the requirements are magic unless you are
>fluent with the NVMe specification.
>
>
>One other thing - I was mistaken before when I said that spdk_nvme_ns_cmd_readv
>can only be used with devices that support SGL. That interface can be used with
>devices that don't support full SGLs, but you have to only specify scatter
>gather lists that can be translated into PRP lists or it will fail with the
>error below.
>> 
>> -Varun
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On 8/2/16, 8:56 AM, "SPDK on behalf of Walker, Benjamin" <spdk-bounces(a)lists.0
>> 1.org on behalf of benjamin.walker(a)intel.com> wrote:
>> 
>> > 
>> > On Mon, 2016-08-01 at 16:50 -0700, txcy uio wrote:
>> > > 
>> > > Hello,
>> > > 
>> > > While submitting a readv command (spdk_nvme_ns_cmd_readv) I see the
>> > > following error:
>> > > 
>> > > READ sqid:3 cid:127 nsid:1 lba:100 len:64
>> > > INVALID_FIELD (00/02) sqid:3 cid:127 cdw0:0 sqhd:0000 p:0 m:0 dnr:1
>> > > 
>> > > 
>> > > Based on this error what can I infer about the command that went wrong?
>> > 
>> > The SPDK NVMe driver is zero-copy, meaning we program the hardware to
>> > transfer data directly from
>> > the buffers you specify. That's why all of the buffers need to be allocated
>> > from pinned memory (in
>> > practice, that usually means they need to be allocated using rte_malloc from
>> > DPDK). NVMe devices
>> > also have physical alignment restrictions. For instance, version 1.0 of the
>> > specification requires
>> > us to program the hardware using a "physical resource page list" (PRP list),
>> > which consists of a
>> > list of 4k physical memory pages. The first page does not have to be 4k
>> > aligned, but all others do,
>> > and all but the first and last page must be 4k in length. Version 1.2 of the
>> > NVMe specification
>> > optionally allows fully flexible scatter gather lists, but almost no
>> > commercially available devices
>> > support scatter gather lists yet.
>> > 
>> > spdk_nvme_ns_cmd_readv is our interface for using the NVMe 1.2 scatter
>> > gather list capabilities. If your device doesn't support that, it's going to
>> > fail with an error like you are seeing. I now see that the purpose of
>> > spdk_nvme_ns_cmd_readv isn't clearly specified in the documentation (http://
>> > www.spdk.io/spdk/doc/nvme_8h.html#acb47ce7de6b6e963ec9fb8de261466ae) and if
>> > you aren't intimately familiar with the NVMe specification I can definitely
>> > see how you wouldn't realize that scatter gather lists require special
>> > hardware support. The documentation needs to be updated.
>> > 
>> > > 
>> > > 
>> > > --Tyc
>> > > 
>> > > _______________________________________________
>> > > SPDK mailing list
>> > > SPDK(a)lists.01.org
>> > > https://lists.01.org/mailman/listinfo/spdk
>> > _______________________________________________
>> > SPDK mailing list
>> > SPDK(a)lists.01.org
>> > https://lists.01.org/mailman/listinfo/spdk
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>_______________________________________________
>SPDK mailing list
>SPDK(a)lists.01.org
>https://lists.01.org/mailman/listinfo/spdk