From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5497055447904932142==" MIME-Version: 1.0 From: Sreeni (Sreenivasa) Busam (Stellus) Subject: Re: [SPDK] Request for more details on SPDK driver API. Date: Mon, 13 Nov 2017 23:14:35 +0000 Message-ID: <5013ff1a19614344bfe9cd3f9eac8534@stellus.com> In-Reply-To: 1510610060.2168.5.camel@intel.com List-ID: To: spdk@lists.01.org --===============5497055447904932142== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Thanks Ben. I see the spdk_nvme_qpair_process_completions() being called in= hello_world program. = Thanks for clarifying it. -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjam= in Sent: Monday, November 13, 2017 1:54 PM To: spdk(a)lists.01.org Subject: Re: [SPDK] Request for more details on SPDK driver API. On Mon, 2017-11-13 at 21:07 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrot= e: > Great! Thanks Jim. Just to double clarify things - the completion callback isn't automatically= called. It's called in response to your application polling for completion= s (spdk_nvme_qpair_process_completions). Nothing will happen if you don't p= oll at the nvme layer. The bdev layer works a bit differently - it sets up pollers that will poll = on your behalf. It's the only way to make the polling efficient once you ge= t into layered bdevs. So if you submit an I/O through the bdev layer, your = callback will just be called automatically when the I/O completes. > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, = > James R > Sent: Monday, November 13, 2017 11:55 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Request for more details on SPDK driver API. > = > = > > On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) = > > wrote: > > = > > Hi guys, > > = > > A simple follow-up question to Paul=E2=80=99s answer. > > Let us assume that I/O to NVMe device has been issued and submit has = > > completed and is successful. Is it a safe assumption that the = > > callback function provided in the spdk_nvme_ns_cmd_read() gets = > > called when the read or write I/O has completed successfully without = > > errors? If the I/O has failed for any reason, the callback is not calle= d correct? > = > The callback function is called when the I/O has completed - both = > success and error. The callback function takes a const struct = > spdk_nvme_cpl * parameter, which is used to determine if the I/O = > completed successfully or not. include/spdk/nvme_spec.h includes a = > helper macro spdk_nvme_cpl_is_error(), and if it is an error, the = > individual fields of spdk_nvme_cpl can be decoded to determine what type = of error. > = > -Jim > = > = > = > > I am using the case when the request is issued without the bdev layer. > > = > > Thanks, > > Sreeni > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni > > (Sreenivasa) Busam (Stellus) > > Sent: Sunday, November 12, 2017 10:37 PM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Request for more details on SPDK driver API. > > = > > Hi Paul, > > = > > Thank you very much for taking the time to write a clear and lengthy = > > reply addressing my questions. I have got a fair understanding about = > > the bdev layer and the use of it. > > I looked at the hello_world and fio example code already and tested = > > the code, and they were working well with a local NVMe SSD device = > > connected. I need to check out the blob hello_world example though. > > I will take a look at the presentations and follow-up if I have any = > > other questions. I have to join the #spdk channel, and will contact = > > you if I need any help there. > > I have a few questions regarding how the failures of read and write = > > requests are handled in the lower layer and controller level, if = > > they are handled and other questions. > > = > > Sreeni > > = > > = > > = > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, = > > Paul E > > Sent: Saturday, November 11, 2017 6:41 AM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Request for more details on SPDK driver API. > > = > > Hi Sreeni, > > = > > Welcome! Before I give you the start of an answer (others will = > > chime in as well I=E2=80=99m sure) I should mention that we=E2=80=99re = also on IRC = > > at freenode in the #spdk channel. You can generally find an expert = > > there that can help via real-time or slightly delayed discussions J > > = > > Also, if you haven=E2=80=99t checked out the presentations from our sum= mit = > > earlier this year they are = > > athttp://www.spdk.io/news/2017/05/03/summit_presentations > > / and there=E2=80=99s some good ones in there that might help you. > > = > > In most of those decks you=E2=80=99ll find an architecture diagram that = > > might help pull all of this together for you. The key point for = > > understanding your questions is to realize that SPDK is a collection = > > of layered modules, some optional, that all work together to form a = > > storage stack in user space. So if you think of the top of the = > > stack being front end protocol modules like iSCSI and NVMeoF and = > > then the next big layer down being a generic block layer followed by = > > a lower device driver layer it should start to make more sense. The = > > generic block layer is a lightweight block abstraction at the top = > > and then device specific modules at the bottom to interface with = > > device drivers. So, anywhere you see an API with =E2=80=9Cbdev=E2=80=9D= in it, like = > > the 2 you mention, you=E2=80=99re in the generic block layer. If the = > > application you=E2=80=99ve constructed is based on NVMe then the next l= ayer = > > down will be the NVMe driver where you=E2=80=99ll find the API that you = > > listed in (1) below. As I mentioned though, a lot of the layers are = > > optional including the bdev layer. If you wanted to write an = > > application that didn=E2=80=99t use bdevs (you knew it was only ever go= ing to be NVMe and didn=E2=80=99t want the minimal overhead of bdev) you co= uld use those NVMe driver APIs directly in your application. > > = > > Hope that makes sense, again I=E2=80=99m sure others will add more = > > clarifying bits of information as well. Here are some examples: > > = > > https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world = > > is an application that doesn=E2=80=99t use bdev, talks directly to NVMe = > > https://github.com/spdk/spdk/tree/master/examples/blob/hello_world = > > is a simple blobstore example (I didn=E2=80=99t mention blobstore above= but = > > its another optional layer) that does use a bdev. In this example it = > > uses a ram disk (malloc) back end but could also easily use an NVMe = > > back end directly and there=E2=80=99s an open patch that shows how that = > > works at https://review.gerrithub.io/#/c/375460/ - both of these = > > might not be the best example for your question because blobstore is = > > thrown in the mix but you can see the bdev portion in there as part = > > of blobstore in a subdir called bdev under /lib/blob > > = > > Anyway, hope that helps more than it confuses J Feel free to keep = > > asking questions until it makes sense=E2=80=A6 > > = > > Thx > > Paul > > = > > PS: there=E2=80=99s also a bunch of existing bdev back end modules = > > herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help = > > clarify the concepts. Note that you can also stack bdevs in order to = > > intercept IO coming and going so you can do value added things = > > easily, we call those =E2=80=9Cvirtual bdevs=E2=80=9D and you=E2=80=99l= l see some examples = > > in the dir I just mentioned (ie split, lvol) > > = > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni > > (Sreenivasa) Busam (Stellus) > > Sent: Friday, November 10, 2017 6:25 PM > > To: spdk(a)lists.01.org > > Subject: [SPDK] Request for more details on SPDK driver API. > > = > > I am new to SPDK driver and trying to understand the various API=E2=80= =99s = > > available to do I/O on NVMe local device and using NVMeoF protocol on a= target device. > > I see different interfaces for the read and write I/O operations. > > It seems there are two types of interfaces available. > > 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related = > > function. This is used for PCIe devices. It seems to apply to = > > devices attached locally. > > 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related = > > API=E2=80=99s in lib/bdev. > > I have been looking at the code and trying to understand how this = > > API and related APIs for the module has been used, the API=E2=80=99s in= 2. = > > are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there = > > is a blob filesystem and for I/O operation on NVMe devices using NVMeoF= protocol spdk/lib/nvmf/*. > > Is this a correct assumption? Is the lib/bdev layer developed to = > > address only these cases? > > Can the lib/bdev API be used on a local NVMe device? If so, is there = > > a test program to understand how to create a bdev device and do the = > > I/O on NVMe SSD block device? > > Please let me know if there is an example to understand how to use = > > the lib/nvmf interfaces. I want to test a device on remote target = > > using NVMeoF protocol. > > = > > Thanks very much for any help in the lib/bdev API. > > = > > = > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============5497055447904932142==--