From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5017484369224422053==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] error while submitting readv cmd Date: Tue, 02 Aug 2016 17:21:50 +0000 Message-ID: <1470158509.3005.5.camel@intel.com> In-Reply-To: 99174110-F8A2-40F1-8024-065AD43CF77E@playstation.sony.com List-ID: To: spdk@lists.01.org --===============5017484369224422053== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Tue, 2016-08-02 at 17:07 +0000, Bhadauria, Varun wrote: > Hello Benjamin, > = > Is there a way of knowing if the H/W supports SGL by looking at the cmd > supports flags of the namespace, similar to SPDK_NVME_NS_FLUSH_SUPPORTED = and > others? You can determine if the hardware supports SGLs by calling spdk_nvme_ctrlr_get_data and then looking at the sgls.supported field of the resulting data structure, as per the NVMe specification. > = > Also if the H/W doesn=E2=80=99t support a specific command like FLUSH, TR= IM, READV, > WRITEV , the corresponding spdk_nvme_ns_cmd_* () should return ENOTSUP as= an > error which can be handled appropriately in the caller? > For failures due to non-alignment or memory not being pinned appropriate > errors should be returned as well. Agreed - the errors should be clearer. We'd accept patches for any improvem= ents to error reporting. I have some longer term plans to make memory allocation= and alignment much more explicit than it is today which should help clear up confusion. Right now I know some of the requirements are magic unless you a= re fluent with the NVMe specification. One other thing - I was mistaken before when I said that spdk_nvme_ns_cmd_r= eadv can only be used with devices that support SGL. That interface can be used = with devices that don't support full SGLs, but you have to only specify scatter gather lists that can be translated into PRP lists or it will fail with the error below. > = > -Varun > = > = > = > = > = > = > = > On 8/2/16, 8:56 AM, "SPDK on behalf of Walker, Benjamin" 1.org on behalf of benjamin.walker(a)intel.com> wrote: > = > > = > > On Mon, 2016-08-01 at 16:50 -0700, txcy uio wrote: > > > = > > > Hello, > > > = > > > While submitting a readv command (spdk_nvme_ns_cmd_readv) I see the > > > following error: > > > = > > > READ sqid:3 cid:127 nsid:1 lba:100 len:64 > > > INVALID_FIELD (00/02) sqid:3 cid:127 cdw0:0 sqhd:0000 p:0 m:0 dnr:1 > > > = > > > = > > > Based on this error what can I infer about the command that went wron= g? > > = > > The SPDK NVMe driver is zero-copy, meaning we program the hardware to > > transfer data directly from > > the buffers you specify. That's why all of the buffers need to be alloc= ated > > from pinned memory (in > > practice, that usually means they need to be allocated using rte_malloc= from > > DPDK). NVMe devices > > also have physical alignment restrictions. For instance, version 1.0 of= the > > specification requires > > us to program the hardware using a "physical resource page list" (PRP l= ist), > > which consists of a > > list of 4k physical memory pages. The first page does not have to be 4k > > aligned, but all others do, > > and all but the first and last page must be 4k in length. Version 1.2 o= f the > > NVMe specification > > optionally allows fully flexible scatter gather lists, but almost no > > commercially available devices > > support scatter gather lists yet. > > = > > spdk_nvme_ns_cmd_readv is our interface for using the NVMe 1.2 scatter > > gather list capabilities. If your device doesn't support that, it's goi= ng to > > fail with an error like you are seeing. I now see that the purpose of > > spdk_nvme_ns_cmd_readv isn't clearly specified in the documentation (ht= tp:// > > www.spdk.io/spdk/doc/nvme_8h.html#acb47ce7de6b6e963ec9fb8de261466ae) an= d if > > you aren't intimately familiar with the NVMe specification I can defini= tely > > see how you wouldn't realize that scatter gather lists require special > > hardware support. The documentation needs to be updated. > > = > > > = > > > = > > > --Tyc > > > = > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============5017484369224422053==--