[SPDK] Request for more details on SPDK driver API.

[SPDK] Request for more details on SPDK driver API.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

I am new to SPDK driver and trying to understand the various API's available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
I see different interfaces for the read and write I/O operations.
It seems there are two types of interfaces available.
1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API's in lib/bdev.
I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API's in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.

Thanks very much for any help in the lib/bdev API.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 5106 bytes --]

Re: [SPDK] Request for more details on SPDK driver API.

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 4627 bytes --]

Hi Sreeni,

Welcome!  Before I give you the start of an answer (others will chime in as well I'm sure) I should mention that we're also on IRC at freenode in the #spdk channel.  You can generally find an expert there that can help via real-time or slightly delayed discussions :)

Also, if you haven't checked out the presentations from our summit earlier this year they are at http://www.spdk.io/news/2017/05/03/summit_presentations/ and there's some good ones in there that might help you.

In most of those decks you'll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with "bdev" in it, like the 2 you mention, you're in the generic block layer.  If the application you've constructed is based on NVMe then the next layer down will be the NVMe driver where you'll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn't use bdevs (you knew it was only ever going to be NVMe and didn't want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.

Hope that makes sense, again I'm sure others will add more clarifying bits of information as well.  Here are some examples:

https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is an application that doesn't use bdev, talks directly to NVMe
https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is a simple blobstore example (I didn't mention blobstore above but its another optional layer) that does use a bdev. In this example it uses a ram disk (malloc) back end but could also easily use an NVMe back end directly and there's an open patch that shows how that works at https://review.gerrithub.io/#/c/375460/ - both of these might not be the best example for your question because blobstore is thrown in the mix but you can see the bdev portion in there as part of blobstore in a subdir called bdev under /lib/blob

Anyway, hope that helps more than it confuses :) Feel free to keep asking questions until it makes sense...

Thx
Paul

PS: there's also a bunch of existing bdev back end modules here https://github.com/spdk/spdk/tree/master/lib/bdev that may help clarify the concepts. Note that you can also stack bdevs in order to intercept IO coming and going so you can do value added things easily, we call those "virtual bdevs" and you'll see some examples in the dir I just mentioned (ie split, lvol)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 10, 2017 6:25 PM
To: spdk(a)lists.01.org
Subject: [SPDK] Request for more details on SPDK driver API.

I am new to SPDK driver and trying to understand the various API's available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
I see different interfaces for the read and write I/O operations.
It seems there are two types of interfaces available.
1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API's in lib/bdev.
I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API's in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.

Thanks very much for any help in the lib/bdev API.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9590 bytes --]

Re: [SPDK] Request for more details on SPDK driver API.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 5649 bytes --]

Hi Paul,

Thank you very much for taking the time to write a clear and lengthy reply addressing my questions. I have got a fair understanding about the bdev layer and the use of it.
I looked at the hello_world and fio example code already and tested the code, and they were working well with a local NVMe SSD device connected. I need to check out the blob hello_world example though.
I will take a look at the presentations and follow-up if I have any other questions. I have to join the #spdk channel, and will contact you if I need any help there.
I have a few questions regarding how the failures of read and write requests are handled in the lower layer and controller level, if they are handled and other questions.

Sreeni




From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Saturday, November 11, 2017 6:41 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi Sreeni,

Welcome!  Before I give you the start of an answer (others will chime in as well I'm sure) I should mention that we're also on IRC at freenode in the #spdk channel.  You can generally find an expert there that can help via real-time or slightly delayed discussions :)

Also, if you haven't checked out the presentations from our summit earlier this year they are at http://www.spdk.io/news/2017/05/03/summit_presentations/ and there's some good ones in there that might help you.

In most of those decks you'll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with "bdev" in it, like the 2 you mention, you're in the generic block layer.  If the application you've constructed is based on NVMe then the next layer down will be the NVMe driver where you'll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn't use bdevs (you knew it was only ever going to be NVMe and didn't want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.

Hope that makes sense, again I'm sure others will add more clarifying bits of information as well.  Here are some examples:

https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is an application that doesn't use bdev, talks directly to NVMe
https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is a simple blobstore example (I didn't mention blobstore above but its another optional layer) that does use a bdev. In this example it uses a ram disk (malloc) back end but could also easily use an NVMe back end directly and there's an open patch that shows how that works at https://review.gerrithub.io/#/c/375460/ - both of these might not be the best example for your question because blobstore is thrown in the mix but you can see the bdev portion in there as part of blobstore in a subdir called bdev under /lib/blob

Anyway, hope that helps more than it confuses :) Feel free to keep asking questions until it makes sense...

Thx
Paul

PS: there's also a bunch of existing bdev back end modules here https://github.com/spdk/spdk/tree/master/lib/bdev that may help clarify the concepts. Note that you can also stack bdevs in order to intercept IO coming and going so you can do value added things easily, we call those "virtual bdevs" and you'll see some examples in the dir I just mentioned (ie split, lvol)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 10, 2017 6:25 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Request for more details on SPDK driver API.

I am new to SPDK driver and trying to understand the various API's available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
I see different interfaces for the read and write I/O operations.
It seems there are two types of interfaces available.
1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API's in lib/bdev.
I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API's in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.

Thanks very much for any help in the lib/bdev API.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 12021 bytes --]

Re: [SPDK] Request for more details on SPDK driver API.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 6446 bytes --]

Hi guys,

A simple follow-up question to Paul's answer.
Let us assume that I/O to NVMe device has been issued and submit has completed and is successful. Is it a safe assumption that the callback function provided in the spdk_nvme_ns_cmd_read() gets called when the read or write I/O has completed successfully without errors? If the I/O has failed for any reason, the callback is not called correct?
I am using the case when the request is issued without the bdev layer.

Thanks,
Sreeni

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Sunday, November 12, 2017 10:37 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi Paul,

Thank you very much for taking the time to write a clear and lengthy reply addressing my questions. I have got a fair understanding about the bdev layer and the use of it.
I looked at the hello_world and fio example code already and tested the code, and they were working well with a local NVMe SSD device connected. I need to check out the blob hello_world example though.
I will take a look at the presentations and follow-up if I have any other questions. I have to join the #spdk channel, and will contact you if I need any help there.
I have a few questions regarding how the failures of read and write requests are handled in the lower layer and controller level, if they are handled and other questions.

Sreeni




From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Saturday, November 11, 2017 6:41 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi Sreeni,

Welcome!  Before I give you the start of an answer (others will chime in as well I'm sure) I should mention that we're also on IRC at freenode in the #spdk channel.  You can generally find an expert there that can help via real-time or slightly delayed discussions :)

Also, if you haven't checked out the presentations from our summit earlier this year they are at http://www.spdk.io/news/2017/05/03/summit_presentations/ and there's some good ones in there that might help you.

In most of those decks you'll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with "bdev" in it, like the 2 you mention, you're in the generic block layer.  If the application you've constructed is based on NVMe then the next layer down will be the NVMe driver where you'll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn't use bdevs (you knew it was only ever going to be NVMe and didn't want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.

Hope that makes sense, again I'm sure others will add more clarifying bits of information as well.  Here are some examples:

https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is an application that doesn't use bdev, talks directly to NVMe
https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is a simple blobstore example (I didn't mention blobstore above but its another optional layer) that does use a bdev. In this example it uses a ram disk (malloc) back end but could also easily use an NVMe back end directly and there's an open patch that shows how that works at https://review.gerrithub.io/#/c/375460/ - both of these might not be the best example for your question because blobstore is thrown in the mix but you can see the bdev portion in there as part of blobstore in a subdir called bdev under /lib/blob

Anyway, hope that helps more than it confuses :) Feel free to keep asking questions until it makes sense...

Thx
Paul

PS: there's also a bunch of existing bdev back end modules here https://github.com/spdk/spdk/tree/master/lib/bdev that may help clarify the concepts. Note that you can also stack bdevs in order to intercept IO coming and going so you can do value added things easily, we call those "virtual bdevs" and you'll see some examples in the dir I just mentioned (ie split, lvol)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 10, 2017 6:25 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Request for more details on SPDK driver API.

I am new to SPDK driver and trying to understand the various API's available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
I see different interfaces for the read and write I/O operations.
It seems there are two types of interfaces available.
1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API's in lib/bdev.
I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API's in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.

Thanks very much for any help in the lib/bdev API.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13853 bytes --]

Re: [SPDK] Request for more details on SPDK driver API.

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 7276 bytes --]


> On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) <s.busam(a)stellus.com> wrote:
> 
> Hi guys,
>  
> A simple follow-up question to Paul’s answer.
> Let us assume that I/O to NVMe device has been issued and submit has completed and is successful. Is it a safe assumption that the callback function provided in the spdk_nvme_ns_cmd_read() gets called when the read or write I/O has completed successfully without errors? If the I/O has failed for any reason, the callback is not called correct?

The callback function is called when the I/O has completed - both success and error.  The callback function takes a const struct spdk_nvme_cpl * parameter, which is used to determine if the I/O completed successfully or not.  include/spdk/nvme_spec.h includes a helper macro spdk_nvme_cpl_is_error(), and if it is an error, the individual fields of spdk_nvme_cpl can be decoded to determine what type of error.

-Jim



> I am using the case when the request is issued without the bdev layer.
>  
> Thanks,
> Sreeni
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
> Sent: Sunday, November 12, 2017 10:37 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
>  
> Hi Paul,
>  
> Thank you very much for taking the time to write a clear and lengthy reply addressing my questions. I have got a fair understanding about the bdev layer and the use of it.
> I looked at the hello_world and fio example code already and tested the code, and they were working well with a local NVMe SSD device connected. I need to check out the blob hello_world example though.
> I will take a look at the presentations and follow-up if I have any other questions. I have to join the #spdk channel, and will contact you if I need any help there.
> I have a few questions regarding how the failures of read and write requests are handled in the lower layer and controller level, if they are handled and other questions.
>  
> Sreeni
>  
>  
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
> Sent: Saturday, November 11, 2017 6:41 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
>  
> Hi Sreeni,
>  
> Welcome!  Before I give you the start of an answer (others will chime in as well I’m sure) I should mention that we’re also on IRC at freenode in the #spdk channel.  You can generally find an expert there that can help via real-time or slightly delayed discussions J
>  
> Also, if you haven’t checked out the presentations from our summit earlier this year they are athttp://www.spdk.io/news/2017/05/03/summit_presentations/ and there’s some good ones in there that might help you.
>  
> In most of those decks you’ll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with “bdev” in it, like the 2 you mention, you’re in the generic block layer.  If the application you’ve constructed is based on NVMe then the next layer down will be the NVMe driver where you’ll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn’t use bdevs (you knew it was only ever going to be NVMe and didn’t want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.
>  
> Hope that makes sense, again I’m sure others will add more clarifying bits of information as well.  Here are some examples:
>  
> https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is an application that doesn’t use bdev, talks directly to NVMe
> https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is a simple blobstore example (I didn’t mention blobstore above but its another optional layer) that does use a bdev. In this example it uses a ram disk (malloc) back end but could also easily use an NVMe back end directly and there’s an open patch that shows how that works at https://review.gerrithub.io/#/c/375460/ - both of these might not be the best example for your question because blobstore is thrown in the mix but you can see the bdev portion in there as part of blobstore in a subdir called bdev under /lib/blob
>  
> Anyway, hope that helps more than it confuses J Feel free to keep asking questions until it makes sense…
>  
> Thx
> Paul
>  
> PS: there’s also a bunch of existing bdev back end modules herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help clarify the concepts. Note that you can also stack bdevs in order to intercept IO coming and going so you can do value added things easily, we call those “virtual bdevs” and you’ll see some examples in the dir I just mentioned (ie split, lvol)
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
> Sent: Friday, November 10, 2017 6:25 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] Request for more details on SPDK driver API.
>  
> I am new to SPDK driver and trying to understand the various API’s available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
> I see different interfaces for the read and write I/O operations.
> It seems there are two types of interfaces available.
> 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
> 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API’s in lib/bdev.
> I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API’s in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
> Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
> Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
> Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.
>  
> Thanks very much for any help in the lib/bdev API.
>  
>  
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Request for more details on SPDK driver API.

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 7672 bytes --]

So if you checkout the nvme hello_world read completion for example:

static void
read_complete(void *arg, const struct spdk_nvme_cpl *completion)
{
}

You'll see it doesn't check for errors (ugh) but if you look in the completion struct you'll see a status field that you can check....

struct spdk_nvme_cpl {
              /* dword 0 */
              uint32_t                           cdw0;   /* command-specific */

              /* dword 1 */
              uint32_t                           rsvd1;

              /* dword 2 */
              uint16_t                           sqhd;    /* submission queue head pointer */
              uint16_t                           sqid;      /* submission queue identifier */

              /* dword 3 */
              uint16_t                           cid;        /* command identifier */
              struct spdk_nvme_status           status;
};




From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, November 13, 2017 12:48 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi guys,

A simple follow-up question to Paul's answer.
Let us assume that I/O to NVMe device has been issued and submit has completed and is successful. Is it a safe assumption that the callback function provided in the spdk_nvme_ns_cmd_read() gets called when the read or write I/O has completed successfully without errors? If the I/O has failed for any reason, the callback is not called correct?
I am using the case when the request is issued without the bdev layer.

Thanks,
Sreeni

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Sunday, November 12, 2017 10:37 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi Paul,

Thank you very much for taking the time to write a clear and lengthy reply addressing my questions. I have got a fair understanding about the bdev layer and the use of it.
I looked at the hello_world and fio example code already and tested the code, and they were working well with a local NVMe SSD device connected. I need to check out the blob hello_world example though.
I will take a look at the presentations and follow-up if I have any other questions. I have to join the #spdk channel, and will contact you if I need any help there.
I have a few questions regarding how the failures of read and write requests are handled in the lower layer and controller level, if they are handled and other questions.

Sreeni




From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Saturday, November 11, 2017 6:41 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi Sreeni,

Welcome!  Before I give you the start of an answer (others will chime in as well I'm sure) I should mention that we're also on IRC at freenode in the #spdk channel.  You can generally find an expert there that can help via real-time or slightly delayed discussions :)

Also, if you haven't checked out the presentations from our summit earlier this year they are at http://www.spdk.io/news/2017/05/03/summit_presentations/ and there's some good ones in there that might help you.

In most of those decks you'll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with "bdev" in it, like the 2 you mention, you're in the generic block layer.  If the application you've constructed is based on NVMe then the next layer down will be the NVMe driver where you'll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn't use bdevs (you knew it was only ever going to be NVMe and didn't want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.

Hope that makes sense, again I'm sure others will add more clarifying bits of information as well.  Here are some examples:

https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is an application that doesn't use bdev, talks directly to NVMe
https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is a simple blobstore example (I didn't mention blobstore above but its another optional layer) that does use a bdev. In this example it uses a ram disk (malloc) back end but could also easily use an NVMe back end directly and there's an open patch that shows how that works at https://review.gerrithub.io/#/c/375460/ - both of these might not be the best example for your question because blobstore is thrown in the mix but you can see the bdev portion in there as part of blobstore in a subdir called bdev under /lib/blob

Anyway, hope that helps more than it confuses :) Feel free to keep asking questions until it makes sense...

Thx
Paul

PS: there's also a bunch of existing bdev back end modules here https://github.com/spdk/spdk/tree/master/lib/bdev that may help clarify the concepts. Note that you can also stack bdevs in order to intercept IO coming and going so you can do value added things easily, we call those "virtual bdevs" and you'll see some examples in the dir I just mentioned (ie split, lvol)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 10, 2017 6:25 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Request for more details on SPDK driver API.

I am new to SPDK driver and trying to understand the various API's available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
I see different interfaces for the read and write I/O operations.
It seems there are two types of interfaces available.
1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API's in lib/bdev.
I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API's in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.

Thanks very much for any help in the lib/bdev API.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 18784 bytes --]

Re: [SPDK] Request for more details on SPDK driver API.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 7795 bytes --]

Great! Thanks Jim.

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
Sent: Monday, November 13, 2017 11:55 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Request for more details on SPDK driver API.


> On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) <s.busam(a)stellus.com> wrote:
> 
> Hi guys,
>  
> A simple follow-up question to Paul’s answer.
> Let us assume that I/O to NVMe device has been issued and submit has completed and is successful. Is it a safe assumption that the callback function provided in the spdk_nvme_ns_cmd_read() gets called when the read or write I/O has completed successfully without errors? If the I/O has failed for any reason, the callback is not called correct?

The callback function is called when the I/O has completed - both success and error.  The callback function takes a const struct spdk_nvme_cpl * parameter, which is used to determine if the I/O completed successfully or not.  include/spdk/nvme_spec.h includes a helper macro spdk_nvme_cpl_is_error(), and if it is an error, the individual fields of spdk_nvme_cpl can be decoded to determine what type of error.

-Jim



> I am using the case when the request is issued without the bdev layer.
>  
> Thanks,
> Sreeni
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> (Sreenivasa) Busam (Stellus)
> Sent: Sunday, November 12, 2017 10:37 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
>  
> Hi Paul,
>  
> Thank you very much for taking the time to write a clear and lengthy reply addressing my questions. I have got a fair understanding about the bdev layer and the use of it.
> I looked at the hello_world and fio example code already and tested the code, and they were working well with a local NVMe SSD device connected. I need to check out the blob hello_world example though.
> I will take a look at the presentations and follow-up if I have any other questions. I have to join the #spdk channel, and will contact you if I need any help there.
> I have a few questions regarding how the failures of read and write requests are handled in the lower layer and controller level, if they are handled and other questions.
>  
> Sreeni
>  
>  
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul 
> E
> Sent: Saturday, November 11, 2017 6:41 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
>  
> Hi Sreeni,
>  
> Welcome!  Before I give you the start of an answer (others will chime 
> in as well I’m sure) I should mention that we’re also on IRC at 
> freenode in the #spdk channel.  You can generally find an expert there 
> that can help via real-time or slightly delayed discussions J
>  
> Also, if you haven’t checked out the presentations from our summit earlier this year they are athttp://www.spdk.io/news/2017/05/03/summit_presentations/ and there’s some good ones in there that might help you.
>  
> In most of those decks you’ll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with “bdev” in it, like the 2 you mention, you’re in the generic block layer.  If the application you’ve constructed is based on NVMe then the next layer down will be the NVMe driver where you’ll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn’t use bdevs (you knew it was only ever going to be NVMe and didn’t want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.
>  
> Hope that makes sense, again I’m sure others will add more clarifying bits of information as well.  Here are some examples:
>  
> https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is 
> an application that doesn’t use bdev, talks directly to NVMe 
> https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is 
> a simple blobstore example (I didn’t mention blobstore above but its 
> another optional layer) that does use a bdev. In this example it uses 
> a ram disk (malloc) back end but could also easily use an NVMe back 
> end directly and there’s an open patch that shows how that works at 
> https://review.gerrithub.io/#/c/375460/ - both of these might not be 
> the best example for your question because blobstore is thrown in the 
> mix but you can see the bdev portion in there as part of blobstore in 
> a subdir called bdev under /lib/blob
>  
> Anyway, hope that helps more than it confuses J Feel free to keep 
> asking questions until it makes sense…
>  
> Thx
> Paul
>  
> PS: there’s also a bunch of existing bdev back end modules 
> herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help 
> clarify the concepts. Note that you can also stack bdevs in order to 
> intercept IO coming and going so you can do value added things easily, 
> we call those “virtual bdevs” and you’ll see some examples in the dir 
> I just mentioned (ie split, lvol)
>  
>  
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> (Sreenivasa) Busam (Stellus)
> Sent: Friday, November 10, 2017 6:25 PM
> To: spdk(a)lists.01.org
> Subject: [SPDK] Request for more details on SPDK driver API.
>  
> I am new to SPDK driver and trying to understand the various API’s available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
> I see different interfaces for the read and write I/O operations.
> It seems there are two types of interfaces available.
> 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
> 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API’s in lib/bdev.
> I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API’s in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
> Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
> Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
> Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.
>  
> Thanks very much for any help in the lib/bdev API.
>  
>  
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Request for more details on SPDK driver API.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 8109 bytes --]

That is good. I thought the callback was taking only one parameter. Your comments below is helpful to understand the status of the I/O request.
Thanks Paul.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Monday, November 13, 2017 11:56 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

So if you checkout the nvme hello_world read completion for example:

static void
read_complete(void *arg, const struct spdk_nvme_cpl *completion)
{
}

You'll see it doesn't check for errors (ugh) but if you look in the completion struct you'll see a status field that you can check....

struct spdk_nvme_cpl {
              /* dword 0 */
              uint32_t                           cdw0;   /* command-specific */

              /* dword 1 */
              uint32_t                           rsvd1;

              /* dword 2 */
              uint16_t                           sqhd;    /* submission queue head pointer */
              uint16_t                           sqid;      /* submission queue identifier */

              /* dword 3 */
              uint16_t                           cid;        /* command identifier */
              struct spdk_nvme_status           status;
};




From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, November 13, 2017 12:48 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi guys,

A simple follow-up question to Paul's answer.
Let us assume that I/O to NVMe device has been issued and submit has completed and is successful. Is it a safe assumption that the callback function provided in the spdk_nvme_ns_cmd_read() gets called when the read or write I/O has completed successfully without errors? If the I/O has failed for any reason, the callback is not called correct?
I am using the case when the request is issued without the bdev layer.

Thanks,
Sreeni

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Sunday, November 12, 2017 10:37 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi Paul,

Thank you very much for taking the time to write a clear and lengthy reply addressing my questions. I have got a fair understanding about the bdev layer and the use of it.
I looked at the hello_world and fio example code already and tested the code, and they were working well with a local NVMe SSD device connected. I need to check out the blob hello_world example though.
I will take a look at the presentations and follow-up if I have any other questions. I have to join the #spdk channel, and will contact you if I need any help there.
I have a few questions regarding how the failures of read and write requests are handled in the lower layer and controller level, if they are handled and other questions.

Sreeni




From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul E
Sent: Saturday, November 11, 2017 6:41 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] Request for more details on SPDK driver API.

Hi Sreeni,

Welcome!  Before I give you the start of an answer (others will chime in as well I'm sure) I should mention that we're also on IRC at freenode in the #spdk channel.  You can generally find an expert there that can help via real-time or slightly delayed discussions :)

Also, if you haven't checked out the presentations from our summit earlier this year they are at http://www.spdk.io/news/2017/05/03/summit_presentations/ and there's some good ones in there that might help you.

In most of those decks you'll find an architecture diagram that might help pull all of this together for you. The key point for understanding your questions is to realize that SPDK is a collection of layered modules, some optional, that all work together to form a storage stack in user space.  So if you think of the top of the stack being front end protocol modules like iSCSI and NVMeoF and then the next big layer down being a generic block layer followed by a lower device driver layer it should start to make more sense.  The generic block layer is a lightweight block abstraction at the top and then device specific modules at the bottom to interface with device drivers. So, anywhere you see an API with "bdev" in it, like the 2 you mention, you're in the generic block layer.  If the application you've constructed is based on NVMe then the next layer down will be the NVMe driver where you'll find the API that you listed in (1) below.  As I mentioned though, a lot of the layers are optional including the bdev layer.  If you wanted to write an application that didn't use bdevs (you knew it was only ever going to be NVMe and didn't want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.

Hope that makes sense, again I'm sure others will add more clarifying bits of information as well.  Here are some examples:

https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is an application that doesn't use bdev, talks directly to NVMe
https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is a simple blobstore example (I didn't mention blobstore above but its another optional layer) that does use a bdev. In this example it uses a ram disk (malloc) back end but could also easily use an NVMe back end directly and there's an open patch that shows how that works at https://review.gerrithub.io/#/c/375460/ - both of these might not be the best example for your question because blobstore is thrown in the mix but you can see the bdev portion in there as part of blobstore in a subdir called bdev under /lib/blob

Anyway, hope that helps more than it confuses :) Feel free to keep asking questions until it makes sense...

Thx
Paul

PS: there's also a bunch of existing bdev back end modules here https://github.com/spdk/spdk/tree/master/lib/bdev that may help clarify the concepts. Note that you can also stack bdevs in order to intercept IO coming and going so you can do value added things easily, we call those "virtual bdevs" and you'll see some examples in the dir I just mentioned (ie split, lvol)


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Friday, November 10, 2017 6:25 PM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] Request for more details on SPDK driver API.

I am new to SPDK driver and trying to understand the various API's available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
I see different interfaces for the read and write I/O operations.
It seems there are two types of interfaces available.
1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related function. This is used for PCIe devices. It seems to apply to devices attached locally.
2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API's in lib/bdev.
I have been looking at the code and trying to understand how this API and related APIs for the module has been used, the API's in 2. are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
Is this a correct assumption? Is the lib/bdev layer developed to address only these cases?
Can the lib/bdev API be used on a local NVMe device? If so, is there a test program to understand how to create a bdev device and do the I/O on NVMe SSD block device?
Please let me know if there is an example to understand how to use the lib/nvmf interfaces. I want to test a device on remote target using NVMeoF protocol.

Thanks very much for any help in the lib/bdev API.



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 19969 bytes --]

Re: [SPDK] Request for more details on SPDK driver API.

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 9042 bytes --]

On Mon, 2017-11-13 at 21:07 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrote:
> Great! Thanks Jim.

Just to double clarify things - the completion callback isn't automatically
called. It's called in response to your application polling for completions
(spdk_nvme_qpair_process_completions). Nothing will happen if you don't poll at
the nvme layer.

The bdev layer works a bit differently - it sets up pollers that will poll on
your behalf. It's the only way to make the polling efficient once you get into
layered bdevs. So if you submit an I/O through the bdev layer, your callback
will just be called automatically when the I/O completes.

> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, James R
> Sent: Monday, November 13, 2017 11:55 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
> 
> 
> > On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) <s.busam(a)s
> > tellus.com> wrote:
> > 
> > Hi guys,
> >  
> > A simple follow-up question to Paul’s answer.
> > Let us assume that I/O to NVMe device has been issued and submit has
> > completed and is successful. Is it a safe assumption that the callback
> > function provided in the spdk_nvme_ns_cmd_read() gets called when the read
> > or write I/O has completed successfully without errors? If the I/O has
> > failed for any reason, the callback is not called correct?
> 
> The callback function is called when the I/O has completed - both success and
> error.  The callback function takes a const struct spdk_nvme_cpl * parameter,
> which is used to determine if the I/O completed successfully or
> not.  include/spdk/nvme_spec.h includes a helper macro
> spdk_nvme_cpl_is_error(), and if it is an error, the individual fields of
> spdk_nvme_cpl can be decoded to determine what type of error.
> 
> -Jim
> 
> 
> 
> > I am using the case when the request is issued without the bdev layer.
> >  
> > Thanks,
> > Sreeni
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> > (Sreenivasa) Busam (Stellus)
> > Sent: Sunday, November 12, 2017 10:37 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Request for more details on SPDK driver API.
> >  
> > Hi Paul,
> >  
> > Thank you very much for taking the time to write a clear and lengthy reply
> > addressing my questions. I have got a fair understanding about the bdev
> > layer and the use of it.
> > I looked at the hello_world and fio example code already and tested the
> > code, and they were working well with a local NVMe SSD device connected. I
> > need to check out the blob hello_world example though.
> > I will take a look at the presentations and follow-up if I have any other
> > questions. I have to join the #spdk channel, and will contact you if I need
> > any help there.
> > I have a few questions regarding how the failures of read and write requests
> > are handled in the lower layer and controller level, if they are handled and
> > other questions.
> >  
> > Sreeni
> >  
> >  
> >  
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul 
> > E
> > Sent: Saturday, November 11, 2017 6:41 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Request for more details on SPDK driver API.
> >  
> > Hi Sreeni,
> >  
> > Welcome!  Before I give you the start of an answer (others will chime 
> > in as well I’m sure) I should mention that we’re also on IRC at 
> > freenode in the #spdk channel.  You can generally find an expert there 
> > that can help via real-time or slightly delayed discussions J
> >  
> > Also, if you haven’t checked out the presentations from our summit earlier
> > this year they are athttp://www.spdk.io/news/2017/05/03/summit_presentations
> > / and there’s some good ones in there that might help you.
> >  
> > In most of those decks you’ll find an architecture diagram that might help
> > pull all of this together for you. The key point for understanding your
> > questions is to realize that SPDK is a collection of layered modules, some
> > optional, that all work together to form a storage stack in user space.  So
> > if you think of the top of the stack being front end protocol modules like
> > iSCSI and NVMeoF and then the next big layer down being a generic block
> > layer followed by a lower device driver layer it should start to make more
> > sense.  The generic block layer is a lightweight block abstraction at the
> > top and then device specific modules at the bottom to interface with device
> > drivers. So, anywhere you see an API with “bdev” in it, like the 2 you
> > mention, you’re in the generic block layer.  If the application you’ve
> > constructed is based on NVMe then the next layer down will be the NVMe
> > driver where you’ll find the API that you listed in (1) below.  As I
> > mentioned though, a lot of the layers are optional including the bdev
> > layer.  If you wanted to write an application that didn’t use bdevs (you
> > knew it was only ever going to be NVMe and didn’t want the minimal overhead
> > of bdev) you could use those NVMe driver APIs directly in your application.
> >  
> > Hope that makes sense, again I’m sure others will add more clarifying bits
> > of information as well.  Here are some examples:
> >  
> > https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is 
> > an application that doesn’t use bdev, talks directly to NVMe 
> > https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is 
> > a simple blobstore example (I didn’t mention blobstore above but its 
> > another optional layer) that does use a bdev. In this example it uses 
> > a ram disk (malloc) back end but could also easily use an NVMe back 
> > end directly and there’s an open patch that shows how that works at 
> > https://review.gerrithub.io/#/c/375460/ - both of these might not be 
> > the best example for your question because blobstore is thrown in the 
> > mix but you can see the bdev portion in there as part of blobstore in 
> > a subdir called bdev under /lib/blob
> >  
> > Anyway, hope that helps more than it confuses J Feel free to keep 
> > asking questions until it makes sense…
> >  
> > Thx
> > Paul
> >  
> > PS: there’s also a bunch of existing bdev back end modules 
> > herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help 
> > clarify the concepts. Note that you can also stack bdevs in order to 
> > intercept IO coming and going so you can do value added things easily, 
> > we call those “virtual bdevs” and you’ll see some examples in the dir 
> > I just mentioned (ie split, lvol)
> >  
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni 
> > (Sreenivasa) Busam (Stellus)
> > Sent: Friday, November 10, 2017 6:25 PM
> > To: spdk(a)lists.01.org
> > Subject: [SPDK] Request for more details on SPDK driver API.
> >  
> > I am new to SPDK driver and trying to understand the various API’s available
> > to do I/O on NVMe local device and using NVMeoF protocol on a target device.
> > I see different interfaces for the read and write I/O operations.
> > It seems there are two types of interfaces available.
> > 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related
> > function. This is used for PCIe devices. It seems to apply to devices
> > attached locally.
> > 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API’s in
> > lib/bdev.
> > I have been looking at the code and trying to understand how this API and
> > related APIs for the module has been used, the API’s in 2. are used for
> > blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesystem
> > and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
> > Is this a correct assumption? Is the lib/bdev layer developed to address
> > only these cases?
> > Can the lib/bdev API be used on a local NVMe device? If so, is there a test
> > program to understand how to create a bdev device and do the I/O on NVMe SSD
> > block device?
> > Please let me know if there is an example to understand how to use the
> > lib/nvmf interfaces. I want to test a device on remote target using NVMeoF
> > protocol.
> >  
> > Thanks very much for any help in the lib/bdev API.
> >  
> >  
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3274 bytes --]

Re: [SPDK] Request for more details on SPDK driver API.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 9461 bytes --]

Thanks Ben. I see the spdk_nvme_qpair_process_completions() being called in hello_world program. 
Thanks for clarifying it.

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Monday, November 13, 2017 1:54 PM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] Request for more details on SPDK driver API.

On Mon, 2017-11-13 at 21:07 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrote:
> Great! Thanks Jim.

Just to double clarify things - the completion callback isn't automatically called. It's called in response to your application polling for completions (spdk_nvme_qpair_process_completions). Nothing will happen if you don't poll at the nvme layer.

The bdev layer works a bit differently - it sets up pollers that will poll on your behalf. It's the only way to make the polling efficient once you get into layered bdevs. So if you submit an I/O through the bdev layer, your callback will just be called automatically when the I/O completes.

> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, 
> James R
> Sent: Monday, November 13, 2017 11:55 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] Request for more details on SPDK driver API.
> 
> 
> > On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) 
> > <s.busam(a)s tellus.com> wrote:
> > 
> > Hi guys,
> >  
> > A simple follow-up question to Paul’s answer.
> > Let us assume that I/O to NVMe device has been issued and submit has 
> > completed and is successful. Is it a safe assumption that the 
> > callback function provided in the spdk_nvme_ns_cmd_read() gets 
> > called when the read or write I/O has completed successfully without 
> > errors? If the I/O has failed for any reason, the callback is not called correct?
> 
> The callback function is called when the I/O has completed - both 
> success and error.  The callback function takes a const struct 
> spdk_nvme_cpl * parameter, which is used to determine if the I/O 
> completed successfully or not.  include/spdk/nvme_spec.h includes a 
> helper macro spdk_nvme_cpl_is_error(), and if it is an error, the 
> individual fields of spdk_nvme_cpl can be decoded to determine what type of error.
> 
> -Jim
> 
> 
> 
> > I am using the case when the request is issued without the bdev layer.
> >  
> > Thanks,
> > Sreeni
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni
> > (Sreenivasa) Busam (Stellus)
> > Sent: Sunday, November 12, 2017 10:37 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Request for more details on SPDK driver API.
> >  
> > Hi Paul,
> >  
> > Thank you very much for taking the time to write a clear and lengthy 
> > reply addressing my questions. I have got a fair understanding about 
> > the bdev layer and the use of it.
> > I looked at the hello_world and fio example code already and tested 
> > the code, and they were working well with a local NVMe SSD device 
> > connected. I need to check out the blob hello_world example though.
> > I will take a look at the presentations and follow-up if I have any 
> > other questions. I have to join the #spdk channel, and will contact 
> > you if I need any help there.
> > I have a few questions regarding how the failures of read and write 
> > requests are handled in the lower layer and controller level, if 
> > they are handled and other questions.
> >  
> > Sreeni
> >  
> >  
> >  
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, 
> > Paul E
> > Sent: Saturday, November 11, 2017 6:41 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] Request for more details on SPDK driver API.
> >  
> > Hi Sreeni,
> >  
> > Welcome!  Before I give you the start of an answer (others will 
> > chime in as well I’m sure) I should mention that we’re also on IRC 
> > at freenode in the #spdk channel.  You can generally find an expert 
> > there that can help via real-time or slightly delayed discussions J
> >  
> > Also, if you haven’t checked out the presentations from our summit 
> > earlier this year they are 
> > athttp://www.spdk.io/news/2017/05/03/summit_presentations
> > / and there’s some good ones in there that might help you.
> >  
> > In most of those decks you’ll find an architecture diagram that 
> > might help pull all of this together for you. The key point for 
> > understanding your questions is to realize that SPDK is a collection 
> > of layered modules, some optional, that all work together to form a 
> > storage stack in user space.  So if you think of the top of the 
> > stack being front end protocol modules like iSCSI and NVMeoF and 
> > then the next big layer down being a generic block layer followed by 
> > a lower device driver layer it should start to make more sense.  The 
> > generic block layer is a lightweight block abstraction at the top 
> > and then device specific modules at the bottom to interface with 
> > device drivers. So, anywhere you see an API with “bdev” in it, like 
> > the 2 you mention, you’re in the generic block layer.  If the 
> > application you’ve constructed is based on NVMe then the next layer 
> > down will be the NVMe driver where you’ll find the API that you 
> > listed in (1) below.  As I mentioned though, a lot of the layers are 
> > optional including the bdev layer.  If you wanted to write an 
> > application that didn’t use bdevs (you knew it was only ever going to be NVMe and didn’t want the minimal overhead of bdev) you could use those NVMe driver APIs directly in your application.
> >  
> > Hope that makes sense, again I’m sure others will add more 
> > clarifying bits of information as well.  Here are some examples:
> >  
> > https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world 
> > is an application that doesn’t use bdev, talks directly to NVMe 
> > https://github.com/spdk/spdk/tree/master/examples/blob/hello_world 
> > is a simple blobstore example (I didn’t mention blobstore above but 
> > its another optional layer) that does use a bdev. In this example it 
> > uses a ram disk (malloc) back end but could also easily use an NVMe 
> > back end directly and there’s an open patch that shows how that 
> > works at https://review.gerrithub.io/#/c/375460/ - both of these 
> > might not be the best example for your question because blobstore is 
> > thrown in the mix but you can see the bdev portion in there as part 
> > of blobstore in a subdir called bdev under /lib/blob
> >  
> > Anyway, hope that helps more than it confuses J Feel free to keep 
> > asking questions until it makes sense…
> >  
> > Thx
> > Paul
> >  
> > PS: there’s also a bunch of existing bdev back end modules 
> > herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help 
> > clarify the concepts. Note that you can also stack bdevs in order to 
> > intercept IO coming and going so you can do value added things 
> > easily, we call those “virtual bdevs” and you’ll see some examples 
> > in the dir I just mentioned (ie split, lvol)
> >  
> >  
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni
> > (Sreenivasa) Busam (Stellus)
> > Sent: Friday, November 10, 2017 6:25 PM
> > To: spdk(a)lists.01.org
> > Subject: [SPDK] Request for more details on SPDK driver API.
> >  
> > I am new to SPDK driver and trying to understand the various API’s 
> > available to do I/O on NVMe local device and using NVMeoF protocol on a target device.
> > I see different interfaces for the read and write I/O operations.
> > It seems there are two types of interfaces available.
> > 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related 
> > function. This is used for PCIe devices. It seems to apply to 
> > devices attached locally.
> > 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related 
> > API’s in lib/bdev.
> > I have been looking at the code and trying to understand how this 
> > API and related APIs for the module has been used, the API’s in 2. 
> > are used for blobfs I/O operation spdk/lib/blob/bdev/*, when there 
> > is a blob filesystem and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nvmf/*.
> > Is this a correct assumption? Is the lib/bdev layer developed to 
> > address only these cases?
> > Can the lib/bdev API be used on a local NVMe device? If so, is there 
> > a test program to understand how to create a bdev device and do the 
> > I/O on NVMe SSD block device?
> > Please let me know if there is an example to understand how to use 
> > the lib/nvmf interfaces. I want to test a device on remote target 
> > using NVMeoF protocol.
> >  
> > Thanks very much for any help in the lib/bdev API.
> >  
> >  
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk