On Mon, 2017-11-13 at 21:07 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrote: > Great! Thanks Jim. Just to double clarify things - the completion callback isn't automatically called. It's called in response to your application polling for completions (spdk_nvme_qpair_process_completions). Nothing will happen if you don't poll at the nvme layer. The bdev layer works a bit differently - it sets up pollers that will poll on your behalf. It's the only way to make the polling efficient once you get into layered bdevs. So if you submit an I/O through the bdev layer, your callback will just be called automatically when the I/O completes. > > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R > Sent: Monday, November 13, 2017 11:55 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Request for more details on SPDK driver API. > > > > On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) > tellus.com> wrote: > > > > Hi guys, > > > > A simple follow-up question to Paul’s answer. > > Let us assume that I/O to NVMe device has been issued and submit has > > completed and is successful. Is it a safe assumption that the callback > > function provided in the spdk_nvme_ns_cmd_read() gets called when the read > > or write I/O has completed successfully without errors? If the I/O has > > failed for any reason, the callback is not called correct? > > The callback function is called when the I/O has completed - both success and > error. The callback function takes a const struct spdk_nvme_cpl * parameter, > which is used to determine if the I/O completed successfully or > not. include/spdk/nvme_spec.h includes a helper macro > spdk_nvme_cpl_is_error(), and if it is an error, the individual fields of > spdk_nvme_cpl can be decoded to determine what type of error. > > -Jim > > > > > I am using the case when the request is issued without the bdev layer. > > > > Thanks, > > Sreeni > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni > > (Sreenivasa) Busam (Stellus) > > Sent: Sunday, November 12, 2017 10:37 PM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Request for more details on SPDK driver API. > > > > Hi Paul, > > > > Thank you very much for taking the time to write a clear and lengthy reply > > addressing my questions. I have got a fair understanding about the bdev > > layer and the use of it. > > I looked at the hello_world and fio example code already and tested the > > code, and they were working well with a local NVMe SSD device connected. I > > need to check out the blob hello_world example though. > > I will take a look at the presentations and follow-up if I have any other > > questions. I have to join the #spdk channel, and will contact you if I need > > any help there. > > I have a few questions regarding how the failures of read and write requests > > are handled in the lower layer and controller level, if they are handled and > > other questions. > > > > Sreeni > > > > > > > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul > > E > > Sent: Saturday, November 11, 2017 6:41 AM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Request for more details on SPDK driver API. > > > > Hi Sreeni, > > > > Welcome! Before I give you the start of an answer (others will chime > > in as well I’m sure) I should mention that we’re also on IRC at > > freenode in the #spdk channel. You can generally find an expert there > > that can help via real-time or slightly delayed discussions J > > > > Also, if you haven’t checked out the presentations from our summit earlier > > this year they are athttp://www.spdk.io/news/2017/05/03/summit_presentations > > / and there’s some good ones in there that might help you. > > > > In most of those decks you’ll find an architecture diagram that might help > > pull all of this together for you. The key point for understanding your > > questions is to realize that SPDK is a collection of layered modules, some > > optional, that all work together to form a storage stack in user space. So > > if you think of the top of the stack being front end protocol modules like > > iSCSI and NVMeoF and then the next big layer down being a generic block > > layer followed by a lower device driver layer it should start to make more > > sense. The generic block layer is a lightweight block abstraction at the > > top and then device specific modules at the bottom to interface with device > > drivers. So, anywhere you see an API with “bdev” in it, like the 2 you > > mention, you’re in the generic block layer. If the application you’ve > > constructed is based on NVMe then the next layer down will be the NVMe > > driver where you’ll find the API that you listed in (1) below. As I > > mentioned though, a lot of the layers are optional including the bdev > > layer. If you wanted to write an application that didn’t use bdevs (you > > knew it was only ever going to be NVMe and didn’t want the minimal overhead > > of bdev) you could use those NVMe driver APIs directly in your application. > > > > Hope that makes sense, again I’m sure others will add more clarifying bits > > of information as well. Here are some examples: > > > > https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is > > an application that doesn’t use bdev, talks directly to NVMe > > https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is > > a simple blobstore example (I didn’t mention blobstore above but its > > another optional layer) that does use a bdev. In this example it uses > > a ram disk (malloc) back end but could also easily use an NVMe back > > end directly and there’s an open patch that shows how that works at > > https://review.gerrithub.io/#/c/375460/ - both of these might not be > > the best example for your question because blobstore is thrown in the > > mix but you can see the bdev portion in there as part of blobstore in > > a subdir called bdev under /lib/blob > > > > Anyway, hope that helps more than it confuses J Feel free to keep > > asking questions until it makes sense… > > > > Thx > > Paul > > > > PS: there’s also a bunch of existing bdev back end modules > > herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help > > clarify the concepts. Note that you can also stack bdevs in order to > > intercept IO coming and going so you can do value added things easily, > > we call those “virtual bdevs” and you’ll see some examples in the dir > > I just mentioned (ie split, lvol) > > > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni > > (Sreenivasa) Busam (Stellus) > > Sent: Friday, November 10, 2017 6:25 PM > > To: spdk(a)lists.01.org > > Subject: [SPDK] Request for more details on SPDK driver API. > > > > I am new to SPDK driver and trying to understand the various API’s available > > to do I/O on NVMe local device and using NVMeoF protocol on a target device. > > I see different interfaces for the read and write I/O operations. > > It seems there are two types of interfaces available. > > 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related > > function. This is used for PCIe devices. It seems to apply to devices > > attached locally. > > 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API’s in > > lib/bdev. > > I have been looking at the code and trying to understand how this API and > > related APIs for the module has been used, the API’s in 2. are used for > > blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem > > and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*. > > Is this a correct assumption? Is the lib/bdev layer developed to address > > only these cases? > > Can the lib/bdev API be used on a local NVMe device? If so, is there a test > > program to understand how to create a bdev device and do the I/O on NVMe SSD > > block device? > > Please let me know if there is an example to understand how to use the > > lib/nvmf interfaces. I want to test a device on remote target using NVMeoF > > protocol. > > > > Thanks very much for any help in the lib/bdev API. > > > > > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk