From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============7779328955787175723==" MIME-Version: 1.0 From: Walker, Benjamin Subject: [SPDK] NVMe-oF Target Library Date: Wed, 26 Apr 2017 21:06:10 +0000 Message-ID: <1493240768.6290.1.camel@intel.com> List-ID: To: spdk@lists.01.org --===============7779328955787175723== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi all, I was hoping to start a bit of a design discussion about the future of the NVMe-oF target library (lib/nvmf). The NVMe-oF target was originally create= d as part of a skunkworks project and was very much an application. It wasn't divided into a library and an app as it is today. Right before we released = it, I decided to attempt to break it up into a library and an application, but I never really finished that task. I'd like to resume that work now, but let = the entire community weigh in on what the library looks like. First, libraries in SPDK (most things that live in lib/) shouldn't enforce a threading model. They should, as much as possible, be entirely passive C libraries with as few dependencies as we can manage. Applications in SPDK (things that live in app/), on the other hand, necessarily must choose a particular threading model. We universally use our application/event framew= ork (lib/event) for apps, which spawns one thread per core, etc. We'll continue this model for NVMe-oF where app/nvmf_tgt will be a full application with a threading model dictated by the application/event framework, while lib/nvmf will be a passive C library that will depend only on other passive C librar= ies. I don't think this distinction is at all reality today, but let's work to m= ake it so. The other major issue with the NVMe-oF target implementation is that it has quite a few baked in assumptions about what the backing storage device looks like. In particular, it was written assuming that it was talking directly t= o an NVMe device (Direct mode), and the ability to route I/O to the bdev layer (Virtual mode) was added much later and isn't entirely fleshed out yet. One= of these assumptions is that real NVMe devices don't benefit from multiple que= ues - you can get the full performance from an NVMe device using just one queue pair. That isn't necessarily true for bdevs, which may be arbitrarily complex virtualized devices. Given that assumption, the NVMe-oF target today only creates a single queue pair to the backing storage device and on= ly uses a single thread to route I/O to it. We're definitely going to need to break that assumption. The first discussion that I want to have is around what the high level conc= epts should be. We clearly need to expose things like "subsystem", "queue pair/connection", "namespace", and "port". We should probably have an object that represents the entire target too, maybe "nvmf_tgt". However, in order = to separate the threading model from the library I think we'll need at least t= wo more concepts. First, some thread has to be in charge of polling for new connections. We typically refer to this as the "acceptor" thread today. Maybe the best way = to handle this is to add an "accept" function that takes the nvmf_tgt object a= s an argument. This function can only be called one a single thread at a time an= d is repeatedly called to discover new connections. I think the user will end up passing a callback in to this function that will be called for each new connection discovered. Second, once a new connection is discovered, we need to hand it off to some collection that a dedicated thread can poll. This collection of connections would be tied specifically to that dedicated thread, but it wouldn't necessarily be tied to a subsystem or a particular storage device. I don't really know what to call this thing - right now I'm kind of thing "io_handl= er". So the general flow for an application would be to construct a target, add subsystems, namespaces, and ports as needed, and then poll the target for incoming connections. For each new connection, the application would assign= it to an io_handler (using whatever algorithm it wanted) and then poll the io_handlers to actually handle I/O on the connections. Does this seem like a reasonable design at a very high level? Feedback is very much welcome and encouraged. If I don't hear back with a bunch of "you're wrong!" or "that's stupid!" ty= pe replies over the next few days, the next step will be to write up a new hea= der file for the library that we can discuss in more detail. Thanks, Ben --===============7779328955787175723== Content-Type: application/x-pkcs7-signature MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKdTCCBOsw ggPToAMCAQICEFLpAsoR6ESdlGU4L6MaMLswDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAgTmV0 d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0xMzAzMTkwMDAwMDBa Fw0yMDA1MzAxMDQ4MzhaMHkxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEUMBIGA1UEBxMLU2Fu dGEgQ2xhcmExGjAYBgNVBAoTEUludGVsIENvcnBvcmF0aW9uMSswKQYDVQQDEyJJbnRlbCBFeHRl cm5hbCBCYXNpYyBJc3N1aW5nIENBIDRBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA 4LDMgJ3YSVX6A9sE+jjH3b+F3Xa86z3LLKu/6WvjIdvUbxnoz2qnvl9UKQI3sE1zURQxrfgvtP0b Pgt1uDwAfLc6H5eqnyi+7FrPsTGCR4gwDmq1WkTQgNDNXUgb71e9/6sfq+WfCDpi8ScaglyLCRp7 ph/V60cbitBvnZFelKCDBh332S6KG3bAdnNGB/vk86bwDlY6omDs6/RsfNwzQVwo/M3oPrux6y6z yIoRulfkVENbM0/9RrzQOlyK4W5Vk4EEsfW2jlCV4W83QKqRccAKIUxw2q/HoHVPbbETrrLmE6RR Z/+eWlkGWl+mtx42HOgOmX0BRdTRo9vH7yeBowIDAQABo4IBdzCCAXMwHwYDVR0jBBgwFoAUrb2Y ejS0Jvf6xCZU7wO94CTLVBowHQYDVR0OBBYEFB5pKrTcKP5HGE4hCz+8rBEv8Jj1MA4GA1UdDwEB /wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMDYGA1UdJQQvMC0GCCsGAQUFBwMEBgorBgEEAYI3 CgMEBgorBgEEAYI3CgMMBgkrBgEEAYI3FQUwFwYDVR0gBBAwDjAMBgoqhkiG+E0BBQFpMEkGA1Ud HwRCMEAwPqA8oDqGOGh0dHA6Ly9jcmwudHJ1c3QtcHJvdmlkZXIuY29tL0FkZFRydXN0RXh0ZXJu YWxDQVJvb3QuY3JsMDoGCCsGAQUFBwEBBC4wLDAqBggrBgEFBQcwAYYeaHR0cDovL29jc3AudHJ1 c3QtcHJvdmlkZXIuY29tMDUGA1UdHgQuMCygKjALgQlpbnRlbC5jb20wG6AZBgorBgEEAYI3FAID oAsMCWludGVsLmNvbTANBgkqhkiG9w0BAQUFAAOCAQEAKcLNo/2So1Jnoi8G7W5Q6FSPq1fmyKW3 sSDf1amvyHkjEgd25n7MKRHGEmRxxoziPKpcmbfXYU+J0g560nCo5gPF78Wd7ZmzcmCcm1UFFfIx fw6QA19bRpTC8bMMaSSEl8y39Pgwa+HENmoPZsM63DdZ6ziDnPqcSbcfYs8qd/m5d22rpXq5IGVU tX6LX7R/hSSw/3sfATnBLgiJtilVyY7OGGmYKCAS2I04itvSS1WtecXTt9OZDyNbl7LtObBrgMLh ZkpJW+pOR9f3h5VG2S5uKkA7Th9NC9EoScdwQCAIw+UWKbSQ0Isj2UFL7fHKvmqWKVTL98sRzvI3 seNC4DCCBYIwggRqoAMCAQICEzMAAIu5Kz5Fe8d0qN0AAAAAi7kwDQYJKoZIhvcNAQEFBQAweTEL MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRQwEgYDVQQHEwtTYW50YSBDbGFyYTEaMBgGA1UEChMR SW50ZWwgQ29ycG9yYXRpb24xKzApBgNVBAMTIkludGVsIEV4dGVybmFsIEJhc2ljIElzc3Vpbmcg Q0EgNEEwHhcNMTcwMTA5MjEyMzU4WhcNMTgwMTA0MjEyMzU4WjBFMRkwFwYDVQQDExBXYWxrZXIs IEJlbmphbWluMSgwJgYJKoZIhvcNAQkBFhliZW5qYW1pbi53YWxrZXJAaW50ZWwuY29tMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxFugJYk4Vd/Yvdmr8BdnGDdCkN1bc1KNCAQBhzC/ BWXw5nxpXWMYFBkTxahM78PtuwdtPDFqoHsMNEaX0miWeYjB6zKbKl7y0LEsSxlu9wjllEdWTYOP 9/m3UC0oITDn7L01adbsD5Sin6W1FMmjcBVrD51oy2orpwfvan3TNVRRQxt8dQz38hivXnona5tt toi+V8ved7o251HApvEwW7QtDfdML+RmBKBSf0MzGjZHPzoBfRrsBUZ0yRHJxlkYNeY99EAUUHwT npsySQSf0cxLmvA6/a4qPOUSitHit+cJQ58/EOt6PLrPGAbdu5sz9O+Iv+FUJakwUtg0sAY4RQID AQABo4ICNTCCAjEwHQYDVR0OBBYEFAU2hsr+3sx/M5e5WafmYD18VvX1MB8GA1UdIwQYMBaAFB5p KrTcKP5HGE4hCz+8rBEv8Jj1MGUGA1UdHwReMFwwWqBYoFaGVGh0dHA6Ly93d3cuaW50ZWwuY29t L3JlcG9zaXRvcnkvQ1JML0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElzc3VpbmclMjBDQSUy MDRBLmNybDCBnwYIKwYBBQUHAQEEgZIwgY8waQYIKwYBBQUHMAKGXWh0dHA6Ly93d3cuaW50ZWwu Y29tL3JlcG9zaXRvcnkvY2VydGlmaWNhdGVzL0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElz c3VpbmclMjBDQSUyMDRBLmNydDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AuaW50ZWwuY29tLzAL BgNVHQ8EBAMCB4AwPAYJKwYBBAGCNxUHBC8wLQYlKwYBBAGCNxUIhsOMdYSZ5VGD/YEohY6fU4KR wAlngd69OZXwQwIBZAIBCTAfBgNVHSUEGDAWBggrBgEFBQcDBAYKKwYBBAGCNwoDDDApBgkrBgEE AYI3FQoEHDAaMAoGCCsGAQUFBwMEMAwGCisGAQQBgjcKAwwwTwYDVR0RBEgwRqApBgorBgEEAYI3 FAIDoBsMGWJlbmphbWluLndhbGtlckBpbnRlbC5jb22BGWJlbmphbWluLndhbGtlckBpbnRlbC5j b20wDQYJKoZIhvcNAQEFBQADggEBAMQUzXgrfwDLl92M7wNqp24Xe1poeurJ8YVAy5a2UukwC/uX uXE8Duoz2jMJL90QETn17H7EQQu1J7kc059H6GyDU42MkzPA3mqZQimrTgOaalPXxWXoVl/UUoLB PJZXGF3Ef1p8b1UVdSnZZ8wTD/QTUw7UhgljKZ1td/raLV1h96x6lKCVkZ0UKU8be5M3FHQ/GZJ9 CgUjvN0m2mYOUHDkNzsUTJb4bsV7vZDa3zixm4Gxu2F/uq328AEJ6JJmXA+jjFOzQ0FI8sa7XOSR 1UPvZSrwyA00M/zFZaDTln+sFPFNseYYGYFU7P711D8Wj1Hv1V/C2G4rSRBJG5f1WF8xggIXMIIC EwIBATCBkDB5MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFDASBgNVBAcTC1NhbnRhIENsYXJh MRowGAYDVQQKExFJbnRlbCBDb3Jwb3JhdGlvbjErMCkGA1UEAxMiSW50ZWwgRXh0ZXJuYWwgQmFz aWMgSXNzdWluZyBDQSA0QQITMwAAi7krPkV7x3So3QAAAACLuTAJBgUrDgMCGgUAoF0wGAYJKoZI hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcwNDI2MjEwNjA4WjAjBgkqhkiG 9w0BCQQxFgQU7HauMWHzlW0iVtFAJL50s92mxqEwDQYJKoZIhvcNAQEBBQAEggEAxEsGKqq12x87 DHuQApgz/AGSOYoqZwzqFNAaluJvCaa8XFuWdsJs4+YsfPplJipUnT4rP/l6/ngndhA+vCutrLDO bMoWVJmFHlA6bDGPtaiYa53jjFxGZXlT3YOHbCL0aUTux2JnB98I2eJXWT5jCPfb3g5FbZ1f5Syo 7jWovriAmd+O9lDXFBS2Z9SMmxNJvwD8u7zscQ0xdRPM3Ct7/ZhqQcrHqcFjRuAoqJty274jDq+D TMoQ+cl6BDptFdVU2zGAkhNOJbX2V7YAb1J7U6Xqt9qNWeaiiifZJKsLAjYpWGKzakr6ANKcskNU BdwufTrCdw7Q3amE7rEzoJg1LwAAAAAAAA== --===============7779328955787175723==--