From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2741228592128554949==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] Request for more details on SPDK driver API. Date: Mon, 13 Nov 2017 21:54:22 +0000 Message-ID: <1510610060.2168.5.camel@intel.com> In-Reply-To: 35b5c01fd2a64a7ca229ce8085816e60@stellus.com List-ID: To: spdk@lists.01.org --===============2741228592128554949== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, 2017-11-13 at 21:07 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrot= e: > Great! Thanks Jim. Just to double clarify things - the completion callback isn't automatically called. It's called in response to your application polling for completions (spdk_nvme_qpair_process_completions). Nothing will happen if you don't pol= l at the nvme layer. The bdev layer works a bit differently - it sets up pollers that will poll = on your behalf. It's the only way to make the polling efficient once you get i= nto layered bdevs. So if you submit an I/O through the bdev layer, your callback will just be called automatically when the I/O completes. > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Harris, Jame= s R > Sent: Monday, November 13, 2017 11:55 AM > To: Storage Performance Development Kit > Subject: Re: [SPDK] Request for more details on SPDK driver API. > = > = > > On Nov 13, 2017, at 12:47 PM, Sreeni (Sreenivasa) Busam (Stellus) > tellus.com> wrote: > > = > > Hi guys, > > = > > A simple follow-up question to Paul=E2=80=99s answer. > > Let us assume that I/O to NVMe device has been issued and submit has > > completed and is successful. Is it a safe assumption that the callback > > function provided in the spdk_nvme_ns_cmd_read() gets called when the r= ead > > or write I/O has completed successfully without errors? If the I/O has > > failed for any reason, the callback is not called correct? > = > The callback function is called when the I/O has completed - both success= and > error. The callback function takes a const struct spdk_nvme_cpl * parame= ter, > which is used to determine if the I/O completed successfully or > not. include/spdk/nvme_spec.h includes a helper macro > spdk_nvme_cpl_is_error(), and if it is an error, the individual fields of > spdk_nvme_cpl can be decoded to determine what type of error. > = > -Jim > = > = > = > > I am using the case when the request is issued without the bdev layer. > > = > > Thanks, > > Sreeni > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni = > > (Sreenivasa) Busam (Stellus) > > Sent: Sunday, November 12, 2017 10:37 PM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Request for more details on SPDK driver API. > > = > > Hi Paul, > > = > > Thank you very much for taking the time to write a clear and lengthy re= ply > > addressing my questions. I have got a fair understanding about the bdev > > layer and the use of it. > > I looked at the hello_world and fio example code already and tested the > > code, and they were working well with a local NVMe SSD device connected= . I > > need to check out the blob hello_world example though. > > I will take a look at the presentations and follow-up if I have any oth= er > > questions. I have to join the #spdk channel, and will contact you if I = need > > any help there. > > I have a few questions regarding how the failures of read and write req= uests > > are handled in the lower layer and controller level, if they are handle= d and > > other questions. > > = > > Sreeni > > = > > = > > = > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Luse, Paul = > > E > > Sent: Saturday, November 11, 2017 6:41 AM > > To: Storage Performance Development Kit > > Subject: Re: [SPDK] Request for more details on SPDK driver API. > > = > > Hi Sreeni, > > = > > Welcome! Before I give you the start of an answer (others will chime = > > in as well I=E2=80=99m sure) I should mention that we=E2=80=99re also o= n IRC at = > > freenode in the #spdk channel. You can generally find an expert there = > > that can help via real-time or slightly delayed discussions J > > = > > Also, if you haven=E2=80=99t checked out the presentations from our sum= mit earlier > > this year they are athttp://www.spdk.io/news/2017/05/03/summit_presenta= tions > > / and there=E2=80=99s some good ones in there that might help you. > > = > > In most of those decks you=E2=80=99ll find an architecture diagram that= might help > > pull all of this together for you. The key point for understanding your > > questions is to realize that SPDK is a collection of layered modules, s= ome > > optional, that all work together to form a storage stack in user space.= So > > if you think of the top of the stack being front end protocol modules l= ike > > iSCSI and NVMeoF and then the next big layer down being a generic block > > layer followed by a lower device driver layer it should start to make m= ore > > sense. The generic block layer is a lightweight block abstraction at t= he > > top and then device specific modules at the bottom to interface with de= vice > > drivers. So, anywhere you see an API with =E2=80=9Cbdev=E2=80=9D in it,= like the 2 you > > mention, you=E2=80=99re in the generic block layer. If the application= you=E2=80=99ve > > constructed is based on NVMe then the next layer down will be the NVMe > > driver where you=E2=80=99ll find the API that you listed in (1) below. = As I > > mentioned though, a lot of the layers are optional including the bdev > > layer. If you wanted to write an application that didn=E2=80=99t use b= devs (you > > knew it was only ever going to be NVMe and didn=E2=80=99t want the mini= mal overhead > > of bdev) you could use those NVMe driver APIs directly in your applicat= ion. > > = > > Hope that makes sense, again I=E2=80=99m sure others will add more clar= ifying bits > > of information as well. Here are some examples: > > = > > https://github.com/spdk/spdk/tree/master/examples/nvme/hello_world is = > > an application that doesn=E2=80=99t use bdev, talks directly to NVMe = > > https://github.com/spdk/spdk/tree/master/examples/blob/hello_world is = > > a simple blobstore example (I didn=E2=80=99t mention blobstore above bu= t its = > > another optional layer) that does use a bdev. In this example it uses = > > a ram disk (malloc) back end but could also easily use an NVMe back = > > end directly and there=E2=80=99s an open patch that shows how that work= s at = > > https://review.gerrithub.io/#/c/375460/ - both of these might not be = > > the best example for your question because blobstore is thrown in the = > > mix but you can see the bdev portion in there as part of blobstore in = > > a subdir called bdev under /lib/blob > > = > > Anyway, hope that helps more than it confuses J Feel free to keep = > > asking questions until it makes sense=E2=80=A6 > > = > > Thx > > Paul > > = > > PS: there=E2=80=99s also a bunch of existing bdev back end modules = > > herehttps://github.com/spdk/spdk/tree/master/lib/bdev that may help = > > clarify the concepts. Note that you can also stack bdevs in order to = > > intercept IO coming and going so you can do value added things easily, = > > we call those =E2=80=9Cvirtual bdevs=E2=80=9D and you=E2=80=99ll see so= me examples in the dir = > > I just mentioned (ie split, lvol) > > = > > = > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni = > > (Sreenivasa) Busam (Stellus) > > Sent: Friday, November 10, 2017 6:25 PM > > To: spdk(a)lists.01.org > > Subject: [SPDK] Request for more details on SPDK driver API. > > = > > I am new to SPDK driver and trying to understand the various API=E2=80= =99s available > > to do I/O on NVMe local device and using NVMeoF protocol on a target de= vice. > > I see different interfaces for the read and write I/O operations. > > It seems there are two types of interfaces available. > > 1. spdk_nvme_ns_cmd_read() and spdk_nvme_ns_cmd_write() and related > > function. This is used for PCIe devices. It seems to apply to devices > > attached locally. > > 2. spdk_bdev_read_blocks() and spdk_bdev_write_blocks() and related API= =E2=80=99s in > > lib/bdev. > > I have been looking at the code and trying to understand how this API a= nd > > related APIs for the module has been used, the API=E2=80=99s in 2. are = used for > > blobfs I/O operation spdk/lib/blob/bdev/*, when there is a blob filesys= tem > > and for I/O operation on NVMe devices using NVMeoF protocol spdk/lib/nv= mf/*. > > Is this a correct assumption? Is the lib/bdev layer developed to address > > only these cases? > > Can the lib/bdev API be used on a local NVMe device? If so, is there a = test > > program to understand how to create a bdev device and do the I/O on NVM= e SSD > > block device? > > Please let me know if there is an example to understand how to use the > > lib/nvmf interfaces. I want to test a device on remote target using NVM= eoF > > protocol. > > = > > Thanks very much for any help in the lib/bdev API. > > = > > = > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============2741228592128554949== Content-Type: application/x-pkcs7-signature MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKdTCCBOsw ggPToAMCAQICEFLpAsoR6ESdlGU4L6MaMLswDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAgTmV0 d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0xMzAzMTkwMDAwMDBa Fw0yMDA1MzAxMDQ4MzhaMHkxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEUMBIGA1UEBxMLU2Fu dGEgQ2xhcmExGjAYBgNVBAoTEUludGVsIENvcnBvcmF0aW9uMSswKQYDVQQDEyJJbnRlbCBFeHRl cm5hbCBCYXNpYyBJc3N1aW5nIENBIDRBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA 4LDMgJ3YSVX6A9sE+jjH3b+F3Xa86z3LLKu/6WvjIdvUbxnoz2qnvl9UKQI3sE1zURQxrfgvtP0b Pgt1uDwAfLc6H5eqnyi+7FrPsTGCR4gwDmq1WkTQgNDNXUgb71e9/6sfq+WfCDpi8ScaglyLCRp7 ph/V60cbitBvnZFelKCDBh332S6KG3bAdnNGB/vk86bwDlY6omDs6/RsfNwzQVwo/M3oPrux6y6z yIoRulfkVENbM0/9RrzQOlyK4W5Vk4EEsfW2jlCV4W83QKqRccAKIUxw2q/HoHVPbbETrrLmE6RR Z/+eWlkGWl+mtx42HOgOmX0BRdTRo9vH7yeBowIDAQABo4IBdzCCAXMwHwYDVR0jBBgwFoAUrb2Y ejS0Jvf6xCZU7wO94CTLVBowHQYDVR0OBBYEFB5pKrTcKP5HGE4hCz+8rBEv8Jj1MA4GA1UdDwEB /wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMDYGA1UdJQQvMC0GCCsGAQUFBwMEBgorBgEEAYI3 CgMEBgorBgEEAYI3CgMMBgkrBgEEAYI3FQUwFwYDVR0gBBAwDjAMBgoqhkiG+E0BBQFpMEkGA1Ud HwRCMEAwPqA8oDqGOGh0dHA6Ly9jcmwudHJ1c3QtcHJvdmlkZXIuY29tL0FkZFRydXN0RXh0ZXJu YWxDQVJvb3QuY3JsMDoGCCsGAQUFBwEBBC4wLDAqBggrBgEFBQcwAYYeaHR0cDovL29jc3AudHJ1 c3QtcHJvdmlkZXIuY29tMDUGA1UdHgQuMCygKjALgQlpbnRlbC5jb20wG6AZBgorBgEEAYI3FAID oAsMCWludGVsLmNvbTANBgkqhkiG9w0BAQUFAAOCAQEAKcLNo/2So1Jnoi8G7W5Q6FSPq1fmyKW3 sSDf1amvyHkjEgd25n7MKRHGEmRxxoziPKpcmbfXYU+J0g560nCo5gPF78Wd7ZmzcmCcm1UFFfIx fw6QA19bRpTC8bMMaSSEl8y39Pgwa+HENmoPZsM63DdZ6ziDnPqcSbcfYs8qd/m5d22rpXq5IGVU tX6LX7R/hSSw/3sfATnBLgiJtilVyY7OGGmYKCAS2I04itvSS1WtecXTt9OZDyNbl7LtObBrgMLh ZkpJW+pOR9f3h5VG2S5uKkA7Th9NC9EoScdwQCAIw+UWKbSQ0Isj2UFL7fHKvmqWKVTL98sRzvI3 seNC4DCCBYIwggRqoAMCAQICEzMAAIu5Kz5Fe8d0qN0AAAAAi7kwDQYJKoZIhvcNAQEFBQAweTEL MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRQwEgYDVQQHEwtTYW50YSBDbGFyYTEaMBgGA1UEChMR SW50ZWwgQ29ycG9yYXRpb24xKzApBgNVBAMTIkludGVsIEV4dGVybmFsIEJhc2ljIElzc3Vpbmcg Q0EgNEEwHhcNMTcwMTA5MjEyMzU4WhcNMTgwMTA0MjEyMzU4WjBFMRkwFwYDVQQDExBXYWxrZXIs IEJlbmphbWluMSgwJgYJKoZIhvcNAQkBFhliZW5qYW1pbi53YWxrZXJAaW50ZWwuY29tMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxFugJYk4Vd/Yvdmr8BdnGDdCkN1bc1KNCAQBhzC/ BWXw5nxpXWMYFBkTxahM78PtuwdtPDFqoHsMNEaX0miWeYjB6zKbKl7y0LEsSxlu9wjllEdWTYOP 9/m3UC0oITDn7L01adbsD5Sin6W1FMmjcBVrD51oy2orpwfvan3TNVRRQxt8dQz38hivXnona5tt toi+V8ved7o251HApvEwW7QtDfdML+RmBKBSf0MzGjZHPzoBfRrsBUZ0yRHJxlkYNeY99EAUUHwT npsySQSf0cxLmvA6/a4qPOUSitHit+cJQ58/EOt6PLrPGAbdu5sz9O+Iv+FUJakwUtg0sAY4RQID AQABo4ICNTCCAjEwHQYDVR0OBBYEFAU2hsr+3sx/M5e5WafmYD18VvX1MB8GA1UdIwQYMBaAFB5p KrTcKP5HGE4hCz+8rBEv8Jj1MGUGA1UdHwReMFwwWqBYoFaGVGh0dHA6Ly93d3cuaW50ZWwuY29t L3JlcG9zaXRvcnkvQ1JML0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElzc3VpbmclMjBDQSUy MDRBLmNybDCBnwYIKwYBBQUHAQEEgZIwgY8waQYIKwYBBQUHMAKGXWh0dHA6Ly93d3cuaW50ZWwu Y29tL3JlcG9zaXRvcnkvY2VydGlmaWNhdGVzL0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElz c3VpbmclMjBDQSUyMDRBLmNydDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AuaW50ZWwuY29tLzAL BgNVHQ8EBAMCB4AwPAYJKwYBBAGCNxUHBC8wLQYlKwYBBAGCNxUIhsOMdYSZ5VGD/YEohY6fU4KR wAlngd69OZXwQwIBZAIBCTAfBgNVHSUEGDAWBggrBgEFBQcDBAYKKwYBBAGCNwoDDDApBgkrBgEE AYI3FQoEHDAaMAoGCCsGAQUFBwMEMAwGCisGAQQBgjcKAwwwTwYDVR0RBEgwRqApBgorBgEEAYI3 FAIDoBsMGWJlbmphbWluLndhbGtlckBpbnRlbC5jb22BGWJlbmphbWluLndhbGtlckBpbnRlbC5j b20wDQYJKoZIhvcNAQEFBQADggEBAMQUzXgrfwDLl92M7wNqp24Xe1poeurJ8YVAy5a2UukwC/uX uXE8Duoz2jMJL90QETn17H7EQQu1J7kc059H6GyDU42MkzPA3mqZQimrTgOaalPXxWXoVl/UUoLB PJZXGF3Ef1p8b1UVdSnZZ8wTD/QTUw7UhgljKZ1td/raLV1h96x6lKCVkZ0UKU8be5M3FHQ/GZJ9 CgUjvN0m2mYOUHDkNzsUTJb4bsV7vZDa3zixm4Gxu2F/uq328AEJ6JJmXA+jjFOzQ0FI8sa7XOSR 1UPvZSrwyA00M/zFZaDTln+sFPFNseYYGYFU7P711D8Wj1Hv1V/C2G4rSRBJG5f1WF8xggIXMIIC EwIBATCBkDB5MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFDASBgNVBAcTC1NhbnRhIENsYXJh MRowGAYDVQQKExFJbnRlbCBDb3Jwb3JhdGlvbjErMCkGA1UEAxMiSW50ZWwgRXh0ZXJuYWwgQmFz aWMgSXNzdWluZyBDQSA0QQITMwAAi7krPkV7x3So3QAAAACLuTAJBgUrDgMCGgUAoF0wGAYJKoZI hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcxMTEzMjE1NDIwWjAjBgkqhkiG 9w0BCQQxFgQUh9QFNz0Vhr6Pnt59WvH2m5+uwJYwDQYJKoZIhvcNAQEBBQAEggEALIv5sMNAcFR0 pxqJ16S0ZLwwGdAIVMOokk96bA1pPZv7RE/PvXyhYFDYuXYiEm0Kpqvq8i0JVQuIMFB3SJN2F3F4 kYHEfPJncVeaIavT6eONr61ZNs2LG40Wn4pMo4kkqRjjgAdx+xNz81sCNwRQ0/sJOZfADT0iRoQ5 WJvYdNuSovDXMUij5kARu6CWkUYGLa17mn8dcEa+CcMLB/qJbF4XprH73la0y2M10K2gocNEyS1u 6CriQ1AaImjhgzkP1+2NUSH/htg3q9O1/p4v0zHKhDNskipltmfOIUETcOrvHttqdaicHrtTJ8nG /DZXiRj3Fvv56SFiApaX6kge5QAAAAAAAA== --===============2741228592128554949==--