From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============3341130733523312886==" MIME-Version: 1.0 From: Sreeni (Sreenivasa) Busam (Stellus) Subject: Re: [SPDK] Request for more details on SPDK driver API. Date: Mon, 13 Nov 2017 21:07:10 +0000 Message-ID: <35b5c01fd2a64a7ca229ce8085816e60@stellus.com> In-Reply-To: 95734624-848C-4621-872F-9874E2BA3A96@intel.com List-ID: To: spdk@lists.01.org --===============3341130733523312886== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Great! Thanks Jim. -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R Sent: Monday, November 13, 2017 11:55 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Request for more details on SPDK driver API. > On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) wrote: > = > Hi guys, > = > A simple follow-up question to Paul=E2=80=99s answer. > Let us assume that I/O to NVMe device has been issued and submit has comp= leted and is successful. Is it a safe assumption that the callback function= provided in the spdk_nvme_ns_cmd_read() gets called when the read or write= I/O has completed successfully without errors? If the I/O has failed for a= ny reason, the callback is not called correct? The callback function is called when the I/O has completed - both success a= nd error. The callback function takes a const struct spdk_nvme_cpl * param= eter, which is used to determine if the I/O completed successfully or not. = include/spdk/nvme_spec.h includes a helper macro spdk_nvme_cpl_is_error(),= and if it is an error, the individual fields of spdk_nvme_cpl can be decod= ed to determine what type of error. -Jim > I am using the case when the request is issued without the bdev layer. > = > Thanks, > Sreeni > = > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni = > (Sreenivasa) Busam (Stellus) > Sent: Sunday, November 12, 2017 10:37 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Request for more details on SPDK driver API. > = > Hi Paul, > = > Thank you very much for taking the time to write a clear and lengthy repl= y addressing my questions. I have got a fair understanding about the bdev l= ayer and the use of it. > I looked at the hello_world and fio example code already and tested the c= ode, and they were working well with a local NVMe SSD device connected. I n= eed to check out the blob hello_world example though. > I will take a look at the presentations and follow-up if I have any other= questions. I have to join the #spdk channel, and will contact you if I nee= d any help there. > I have a few questions regarding how the failures of read and write reque= sts are handled in the lower layer and controller level, if they are handle= d and other questions. > = > Sreeni > = > = > = > = > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul = > E > Sent: Saturday, November 11, 2017 6:41 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Request for more details on SPDK driver API. > = > Hi Sreeni, > = > Welcome! Before I give you the start of an answer (others will chime = > in as well I=E2=80=99m sure) I should mention that we=E2=80=99re also on = IRC at = > freenode in the #spdk channel. You can generally find an expert there = > that can help via real-time or slightly delayed discussions J > = > Also, if you haven=E2=80=99t checked out the presentations from our summi= t earlier this year they are athttp://www.spdk.io/news/2017/05/03/summit_pr= esentations/ and there=E2=80=99s some good ones in there that might help yo= u. > = > In most of those decks you=E2=80=99ll find an architecture diagram that m= ight help pull all of this together for you. The key point for understandin= g your questions is to realize that SPDK is a collection of layered modules= , some optional, that all work together to form a storage stack in user spa= ce. So if you think of the top of the stack being front end protocol modul= es like iSCSI and NVMeoF and then the next big layer down being a generic b= lock layer followed by a lower device driver layer it should start to make = more sense. The generic block layer is a lightweight block abstraction at = the top and then device specific modules at the bottom to interface with de= vice drivers. So, anywhere you see an API with =E2=80=9Cbdev=E2=80=9D in it= , like the 2 you mention, you=E2=80=99re in the generic block layer. If th= e application you=E2=80=99ve constructed is based on NVMe then the next lay= er down will be the NVMe driver where you=E2=80=99ll find the API that you = listed in (1) below. As I mentioned though, a lot of the layers are option= al including the bdev layer. If you wanted to write an application that di= dn=E2=80=99t use bdevs (you knew it was only ever going to be NVMe and didn= =E2=80=99t want the minimal overhead of bdev) you could use those NVMe driv= er APIs directly in your application. > = > Hope that makes sense, again I=E2=80=99m sure others will add more clarif= ying bits of information as well. Here are some examples: > = > https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is = > an application that doesn=E2=80=99t use bdev, talks directly to NVMe = > https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is = > a simple blobstore example (I didn=E2=80=99t mention blobstore above but = its = > another optional layer) that does use a bdev. In this example it uses = > a ram disk (malloc) back end but could also easily use an NVMe back = > end directly and there=E2=80=99s an open patch that shows how that works = at = > https://review.gerrithub.io/#/c/375460/ - both of these might not be = > the best example for your question because blobstore is thrown in the = > mix but you can see the bdev portion in there as part of blobstore in = > a subdir called bdev under /lib/blob > = > Anyway, hope that helps more than it confuses J Feel free to keep = > asking questions until it makes sense=E2=80=A6 > = > Thx > Paul > = > PS: there=E2=80=99s also a bunch of existing bdev back end modules = > herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help = > clarify the concepts. Note that you can also stack bdevs in order to = > intercept IO coming and going so you can do value added things easily, = > we call those =E2=80=9Cvirtual bdevs=E2=80=9D and you=E2=80=99ll see some= examples in the dir = > I just mentioned (ie split, lvol) > = > = > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni = > (Sreenivasa) Busam (Stellus) > Sent: Friday, November 10, 2017 6:25 PM > To: spdk(a)lists.01.org > Subject: [SPDK] Request for more details on SPDK driver API. > = > I am new to SPDK driver and trying to understand the various API=E2=80=99= s available to do I/O on NVMe local device and using NVMeoF protocol on a t= arget device. > I see different interfaces for the read and write I/O operations. > It seems there are two types of interfaces available. > 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related funct= ion. This is used for PCIe devices. It seems to apply to devices attached l= ocally. > 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API= =E2=80=99s in lib/bdev. > I have been looking at the code and trying to understand how this API and= related APIs for the module has been used, the API=E2=80=99s in 2. are use= d for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob files= ystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/= nvmf/*. > Is this a correct assumption? Is the lib/bdev layer developed to address = only these cases? > Can the lib/bdev API be used on a local NVMe device? If so, is there a te= st program to understand how to create a bdev device and do the I/O on NVMe= SSD block device? > Please let me know if there is an example to understand how to use the li= b/nvmf interfaces. I want to test a device on remote target using NVMeoF pr= otocol. > = > Thanks very much for any help in the lib/bdev API. > = > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk --===============3341130733523312886==--