Re: [SPDK] Request for more details on SPDK driver API.

[Sreeni (Sreenivasa) Busam (Stellus) <s.busam at stellus.com>]

[-- Attachment #1: Type: text/plain, Size: 7795 bytes --]

Great! Thanks Jim.

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Monday, November 13, 2017 11:55 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Request for more details on SPDK driver API.


> On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) <s.busam(a)stellus.com> wrote:
> 
> Hi guys,
>  
> A simple follow-up question to Paul’s answer.
> Let us assume that I/O to NVMe device has been issued and submit has completed and is successful. Is it a safe assumption that the callback function provided in the spdk_nvme_ns_cmd_read() gets called when the read or write I/O has completed successfully without errors? If the I/O has failed for any reason, the callback is not called correct?

The callback function is called when the I/O has completed - both success and error.  The callback function takes a const struct spdk_nvme_cpl * parameter, which is used to determine if the I/O completed successfully or not.  include/spdk/nvme_spec.h includes a helper macro spdk_nvme_cpl_is_error(), and if it is an error, the individual fields of spdk_nvme_cpl can be decoded to determine what type of error.

-Jim



> I am using the case when the request is issued without the bdev layer.
>  
> Thanks,
> Sreeni
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> (Sreenivasa) Busam (Stellus)
> Sent: Sunday, November 12, 2017 10:37 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
>  
> Hi Paul,
>  
> Thank you very much for taking the time to write a clear and lengthy reply addressing my questions. I have got a fair understanding about the bdev layer and the use of it.
> I looked at the hello_world and fio example code already and tested the code, and they were working well with a local NVMe SSD device connected. I need to check out the blob hello_world example though.
> I will take a look at the presentations and follow-up if I have any other questions. I have to join the #spdk channel, and will contact you if I need any help there.
> I have a few questions regarding how the failures of read and write requests are handled in the lower layer and controller level, if they are handled and other questions.
>  
> Sreeni
>  
>  
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul 
> E
> Sent: Saturday, November 11, 2017 6:41 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
>  
> Hi Sreeni,
>  
> Welcome!  Before I give you the start of an answer (others will chime 
> in as well I’m sure) I should mention that we’re also on IRC at 
> freenode in the #spdk channel.  You can generally find an expert there 
> that can help via real-time or slightly delayed discussions J
>  
> Also, if you haven’t checked out the presentations from our summit earlier this year they are athttp://www.spdk.io/news/2017/05/03/summit_presentations/ and there’s some good ones in there that might help you.
>  
> In most of those decks you’ll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with “bdev” in it, like the 2 you mention, you’re in the generic block layer.  If the application you’ve constructed is based on NVMe then the next layer down will be the NVMe driver where you’ll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn’t use bdevs (you knew it was only ever going to be NVMe and didn’t want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.
>  
> Hope that makes sense, again I’m sure others will add more clarifying bits of information as well.  Here are some examples:
>  
> https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is 
> an application that doesn’t use bdev, talks directly to NVMe 
> https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is 
> a simple blobstore example (I didn’t mention blobstore above but its 
> another optional layer) that does use a bdev. In this example it uses 
> a ram disk (malloc) back end but could also easily use an NVMe back 
> end directly and there’s an open patch that shows how that works at 
> https://review.gerrithub.io/#/c/375460/ - both of these might not be 
> the best example for your question because blobstore is thrown in the 
> mix but you can see the bdev portion in there as part of blobstore in 
> a subdir called bdev under /lib/blob
>  
> Anyway, hope that helps more than it confuses J Feel free to keep 
> asking questions until it makes sense…
>  
> Thx
> Paul
>  
> PS: there’s also a bunch of existing bdev back end modules 
> herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help 
> clarify the concepts. Note that you can also stack bdevs in order to 
> intercept IO coming and going so you can do value added things easily, 
> we call those “virtual bdevs” and you’ll see some examples in the dir 
> I just mentioned (ie split, lvol)
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> (Sreenivasa) Busam (Stellus)
> Sent: Friday, November 10, 2017 6:25 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] Request for more details on SPDK driver API.
>  
> I am new to SPDK driver and trying to understand the various API’s available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
> I see different interfaces for the read and write I/O operations.
> It seems there are two types of interfaces available.
> 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
> 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API’s in lib/bdev.
> I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API’s in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
> Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
> Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
> Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.
>  
> Thanks very much for any help in the lib/bdev API.
>  
>  
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk