Re: [SPDK] Request for more details on SPDK driver API.

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 9042 bytes --]

On Mon, 2017-11-13 at 21:07 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrote:
> Great! Thanks Jim.

Just to double clarify things - the completion callback isn't automatically
called. It's called in response to your application polling for completions
(spdk_nvme_qpair_process_completions). Nothing will happen if you don't poll at
the nvme layer.

The bdev layer works a bit differently - it sets up pollers that will poll on
your behalf. It's the only way to make the polling efficient once you get into
layered bdevs. So if you submit an I/O through the bdev layer, your callback
will just be called automatically when the I/O completes.

> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
> Sent: Monday, November 13, 2017 11:55 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
> 
> 
> > On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) <s.busam(a)s
> > tellus.com> wrote:
> > 
> > Hi guys,
> >  
> > A simple follow-up question to Paul’s answer.
> > Let us assume that I/O to NVMe device has been issued and submit has
> > completed and is successful. Is it a safe assumption that the callback
> > function provided in the spdk_nvme_ns_cmd_read() gets called when the read
> > or write I/O has completed successfully without errors? If the I/O has
> > failed for any reason, the callback is not called correct?
> 
> The callback function is called when the I/O has completed - both success and
> error.  The callback function takes a const struct spdk_nvme_cpl * parameter,
> which is used to determine if the I/O completed successfully or
> not.  include/spdk/nvme_spec.h includes a helper macro
> spdk_nvme_cpl_is_error(), and if it is an error, the individual fields of
> spdk_nvme_cpl can be decoded to determine what type of error.
> 
> -Jim
> 
> 
> 
> > I am using the case when the request is issued without the bdev layer.
> >  
> > Thanks,
> > Sreeni
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> > (Sreenivasa) Busam (Stellus)
> > Sent: Sunday, November 12, 2017 10:37 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Request for more details on SPDK driver API.
> >  
> > Hi Paul,
> >  
> > Thank you very much for taking the time to write a clear and lengthy reply
> > addressing my questions. I have got a fair understanding about the bdev
> > layer and the use of it.
> > I looked at the hello_world and fio example code already and tested the
> > code, and they were working well with a local NVMe SSD device connected. I
> > need to check out the blob hello_world example though.
> > I will take a look at the presentations and follow-up if I have any other
> > questions. I have to join the #spdk channel, and will contact you if I need
> > any help there.
> > I have a few questions regarding how the failures of read and write requests
> > are handled in the lower layer and controller level, if they are handled and
> > other questions.
> >  
> > Sreeni
> >  
> >  
> >  
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul 
> > E
> > Sent: Saturday, November 11, 2017 6:41 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Request for more details on SPDK driver API.
> >  
> > Hi Sreeni,
> >  
> > Welcome!  Before I give you the start of an answer (others will chime 
> > in as well I’m sure) I should mention that we’re also on IRC at 
> > freenode in the #spdk channel.  You can generally find an expert there 
> > that can help via real-time or slightly delayed discussions J
> >  
> > Also, if you haven’t checked out the presentations from our summit earlier
> > this year they are athttp://www.spdk.io/news/2017/05/03/summit_presentations
> > / and there’s some good ones in there that might help you.
> >  
> > In most of those decks you’ll find an architecture diagram that might help
> > pull all of this together for you. The key point for understanding your
> > questions is to realize that SPDK is a collection of layered modules, some
> > optional, that all work together to form a storage stack in user space.  So
> > if you think of the top of the stack being front end protocol modules like
> > iSCSI and NVMeoF and then the next big layer down being a generic block
> > layer followed by a lower device driver layer it should start to make more
> > sense.  The generic block layer is a lightweight block abstraction at the
> > top and then device specific modules at the bottom to interface with device
> > drivers. So, anywhere you see an API with “bdev” in it, like the 2 you
> > mention, you’re in the generic block layer.  If the application you’ve
> > constructed is based on NVMe then the next layer down will be the NVMe
> > driver where you’ll find the API that you listed in (1) below.  As I
> > mentioned though, a lot of the layers are optional including the bdev
> > layer.  If you wanted to write an application that didn’t use bdevs (you
> > knew it was only ever going to be NVMe and didn’t want the minimal overhead
> > of bdev) you could use those NVMe driver APIs directly in your application.
> >  
> > Hope that makes sense, again I’m sure others will add more clarifying bits
> > of information as well.  Here are some examples:
> >  
> > https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is 
> > an application that doesn’t use bdev, talks directly to NVMe 
> > https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is 
> > a simple blobstore example (I didn’t mention blobstore above but its 
> > another optional layer) that does use a bdev. In this example it uses 
> > a ram disk (malloc) back end but could also easily use an NVMe back 
> > end directly and there’s an open patch that shows how that works at 
> > https://review.gerrithub.io/#/c/375460/ - both of these might not be 
> > the best example for your question because blobstore is thrown in the 
> > mix but you can see the bdev portion in there as part of blobstore in 
> > a subdir called bdev under /lib/blob
> >  
> > Anyway, hope that helps more than it confuses J Feel free to keep 
> > asking questions until it makes sense…
> >  
> > Thx
> > Paul
> >  
> > PS: there’s also a bunch of existing bdev back end modules 
> > herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help 
> > clarify the concepts. Note that you can also stack bdevs in order to 
> > intercept IO coming and going so you can do value added things easily, 
> > we call those “virtual bdevs” and you’ll see some examples in the dir 
> > I just mentioned (ie split, lvol)
> >  
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> > (Sreenivasa) Busam (Stellus)
> > Sent: Friday, November 10, 2017 6:25 PM
> > To: spdk(a)lists.01.org
> > Subject: [SPDK] Request for more details on SPDK driver API.
> >  
> > I am new to SPDK driver and trying to understand the various API’s available
> > to do I/O on NVMe local device and using NVMeoF protocol on a target device.
> > I see different interfaces for the read and write I/O operations.
> > It seems there are two types of interfaces available.
> > 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related
> > function. This is used for PCIe devices. It seems to apply to devices
> > attached locally.
> > 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API’s in
> > lib/bdev.
> > I have been looking at the code and trying to understand how this API and
> > related APIs for the module has been used, the API’s in 2. are used for
> > blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem
> > and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
> > Is this a correct assumption? Is the lib/bdev layer developed to address
> > only these cases?
> > Can the lib/bdev API be used on a local NVMe device? If so, is there a test
> > program to understand how to create a bdev device and do the I/O on NVMe SSD
> > block device?
> > Please let me know if there is an example to understand how to use the
> > lib/nvmf interfaces. I want to test a device on remote target using NVMeoF
> > protocol.
> >  
> > Thanks very much for any help in the lib/bdev API.
> >  
> >  
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3274 bytes --]