From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8121447425713101029==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] SPDK Dependencies Date: Wed, 26 Apr 2017 22:09:24 +0000 Message-ID: <1493244563.6290.3.camel@intel.com> In-Reply-To: 37B08312E007AE46A00101F43DA919DD43FA83EB@FMSMSX105.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============8121447425713101029== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable At the summit we did end up discussing this quite a bit. For DPDK it's fair= ly clear that we're going to continue to abstract away all of those dependenci= es inside of the env library. There are pull requests out to finish this up no= w and we'll get those merged relatively quickly. We're also working with the DPDK= team to improve DPDK so that there are hopefully fewer reasons to forego using i= t. As far as POSIX, I have a proposal. Assuming POSIX is available makes it significantly easier to work on SPDK itself, so I'm very hesitant to move a= ll POSIX calls inside of the env library and force people to use wrapper funct= ions. I'm concerned that it raises the barrier to contribute too much (can I call malloc? what about printf?). Further, the vast majority of SPDK users actua= lly do have POSIX available to them. That said, there are probably parts that w= e can abstract out and some other refactoring we can do here. First, I think pthreads can be refactored into the env library. Locking and spawning threads is rare enough in SPDK that it isn't an undue burden to use wrappers, and it also requires some care regarding CPU affinity. So the bel= ow discussion excludes that. Second, I think we can get away with moving all of our POSIX headers to a s= ingle SPDK header file at spdk/stdinc.h. We can then include that file everywhere either directly or by using a -include option in the Makefile. This will ev= en play nicely with precompiling spdk/stdinc.h (thanks Daniel for the suggesti= on), although SPDK's build times are not a concern right now. Third, for the small subset of SPDK users that really don't have POSIX available, they can carry a patch on top of SPDK that simply replaces spdk/stdinc.h with their own implementation. Most environments have at least some part of the POSIX standard, and 90% of what we're using is just the C standard library, so in reality this header shouldn't be too difficult to replace. Thoughts? Opinions? Thanks, Ben On Mon, 2017-04-17 at 18:55 +0000, Walker, Benjamin wrote: > Hi all, > = > There has been a lot of activity lately dealing with abstracting SPDK's > dependencies to allow porting SPDK to additional platforms and frameworks= . I > think we'll end up talking about this extensively at the summit this week= . The > ability to use SPDK in a variety of environments is very important to us,= so I > look forward to the discussions. I wanted to write a basic summary of the > current state of SPDK's dependencies for everyone to have, and so that th= ose > who are unable to attend the summit can provide feedback. I'll try to upd= ate > this thread with the results of the discussions at the summit for everyon= e's > benefit. > = > We've already eliminated all SPDK dependencies except for two: > = > 1) POSIX > 2) DPDK > = > SPDK already has a mechanism for abstracting away the "environment" it is > running in. We do this by moving all calls to dependencies to the "env" > library, whose interface is defined by include/spdk/env.h. The build syst= em > allows users to point to an alternate implementation of the env library if > required. SPDK's default implementation of env is based on DPDK and runs = in > Linux and FreeBSD user space. > = > DPDK is a very large toolkit, of course, but until just recently SPDK only > used DPDK's 'eal' module (environment abstraction library). The original = goal > of the SPDK env library was just to abstract all uses of DPDK's eal, but = not > all uses of POSIX APIs. We made some progress in this area, but there are= a > number of remaining calls as of this writing. Fortunately, the wider comm= unity > has really come through. There are at least two pull requests out from two > independent users that mostly finish the job. We'll work through those pu= ll > requests and try to get them merged shortly. The only part of the code th= at > will continue to explicitly depend on DPDK will be the newly released vho= st- > scsi target because that relies on DPDK vhost infrastructure. This > infrastructure is outside of DPDK's eal and is quite extensive. We feel t= his > is acceptable because building vhost in SPDK is optional - you can just t= urn > it off to eliminate the dependency. > = > There are a number of legitimate reasons to want to remove the DPDK depen= dency > from SPDK. For instance, DPDK dictates a particular memory management mod= el, > requires a specific threading model, and performs PCI device management t= hat > may conflict with other libraries. However, DPDK provides tremendous valu= e to > SPDK in a number of areas, especially by implementing memory allocation > suitable for DMA in user space. Memory management is a major challenge wi= th > all sorts of hidden traps and challenges, so we see significant value in = using > a production-tested library here. Re-implement the env without DPDK at yo= ur > own risk! > = > Abstracting away our POSIX dependency is a similar but larger challenge. = While > we foresaw the need for people to replace DPDK, we didn't see any use cas= e for > dropping POSIX. We were definitely wrong though - people are using SPDK i= n all > sorts of environments that we didn't predict, from embedded firmware to > operating system kernels and beyond. So we'll need to address this as time > goes on. I think this will result in further expansion of the 'env' libra= ry. > Fortunately, we don't require too much from POSIX - mostly pthreads, the C > standard library, and maybe a couple of other miscellaneous headers. > = > There are other less obvious dependencies in SPDK. For instance, it also > depends on a relatively recent version of gcc or clang. This is mostly a > practical concern and not so much by intention. Our continuous integration > system (which I'll talk about at the summit - exciting developments here) > currently consists of 4 machines, 3 of which are running Fedora Linux 25 = while > the other is running FreeBSD 11.0. These are very much up to date systems= , so > we tend not to catch bugs on older systems today. We also use cutting edg= e C > features and tools, so we'll probably over time need to select a particul= ar > minimum version of the C standard that we're going to develop to. > = > I'm also concerned with the scope of the env library and I'd like to work= as > hard as possible to limit the size of the API. This is for two reasons - = first > and foremost to minimize the number of calls to these wrapper functions so > that it doesn't impact performance. Performance really is everything and = we > can't sacrifice that. This isn't shaping up to be much of a problem so far > because very few calls to our dependencies happen within the I/O path. > Secondarily, I'd like to make it as straightforward as possible for users= to > re-implement the env. The best way to make it easy to do that is to keep = the > API small. > = > I look forward to the discussions at the summit this week. We'd love to h= ear > about all of the different environments, platforms, and frameworks where = SPDK > is being used and how we can make it easier to integrate SPDK wherever it= is > valuable. > = > Thanks, > Ben > = > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============8121447425713101029== Content-Type: application/x-pkcs7-signature MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKdTCCBOsw ggPToAMCAQICEFLpAsoR6ESdlGU4L6MaMLswDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAgTmV0 d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0xMzAzMTkwMDAwMDBa Fw0yMDA1MzAxMDQ4MzhaMHkxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEUMBIGA1UEBxMLU2Fu dGEgQ2xhcmExGjAYBgNVBAoTEUludGVsIENvcnBvcmF0aW9uMSswKQYDVQQDEyJJbnRlbCBFeHRl cm5hbCBCYXNpYyBJc3N1aW5nIENBIDRBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA 4LDMgJ3YSVX6A9sE+jjH3b+F3Xa86z3LLKu/6WvjIdvUbxnoz2qnvl9UKQI3sE1zURQxrfgvtP0b Pgt1uDwAfLc6H5eqnyi+7FrPsTGCR4gwDmq1WkTQgNDNXUgb71e9/6sfq+WfCDpi8ScaglyLCRp7 ph/V60cbitBvnZFelKCDBh332S6KG3bAdnNGB/vk86bwDlY6omDs6/RsfNwzQVwo/M3oPrux6y6z yIoRulfkVENbM0/9RrzQOlyK4W5Vk4EEsfW2jlCV4W83QKqRccAKIUxw2q/HoHVPbbETrrLmE6RR Z/+eWlkGWl+mtx42HOgOmX0BRdTRo9vH7yeBowIDAQABo4IBdzCCAXMwHwYDVR0jBBgwFoAUrb2Y ejS0Jvf6xCZU7wO94CTLVBowHQYDVR0OBBYEFB5pKrTcKP5HGE4hCz+8rBEv8Jj1MA4GA1UdDwEB /wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMDYGA1UdJQQvMC0GCCsGAQUFBwMEBgorBgEEAYI3 CgMEBgorBgEEAYI3CgMMBgkrBgEEAYI3FQUwFwYDVR0gBBAwDjAMBgoqhkiG+E0BBQFpMEkGA1Ud HwRCMEAwPqA8oDqGOGh0dHA6Ly9jcmwudHJ1c3QtcHJvdmlkZXIuY29tL0FkZFRydXN0RXh0ZXJu YWxDQVJvb3QuY3JsMDoGCCsGAQUFBwEBBC4wLDAqBggrBgEFBQcwAYYeaHR0cDovL29jc3AudHJ1 c3QtcHJvdmlkZXIuY29tMDUGA1UdHgQuMCygKjALgQlpbnRlbC5jb20wG6AZBgorBgEEAYI3FAID oAsMCWludGVsLmNvbTANBgkqhkiG9w0BAQUFAAOCAQEAKcLNo/2So1Jnoi8G7W5Q6FSPq1fmyKW3 sSDf1amvyHkjEgd25n7MKRHGEmRxxoziPKpcmbfXYU+J0g560nCo5gPF78Wd7ZmzcmCcm1UFFfIx fw6QA19bRpTC8bMMaSSEl8y39Pgwa+HENmoPZsM63DdZ6ziDnPqcSbcfYs8qd/m5d22rpXq5IGVU tX6LX7R/hSSw/3sfATnBLgiJtilVyY7OGGmYKCAS2I04itvSS1WtecXTt9OZDyNbl7LtObBrgMLh ZkpJW+pOR9f3h5VG2S5uKkA7Th9NC9EoScdwQCAIw+UWKbSQ0Isj2UFL7fHKvmqWKVTL98sRzvI3 seNC4DCCBYIwggRqoAMCAQICEzMAAIu5Kz5Fe8d0qN0AAAAAi7kwDQYJKoZIhvcNAQEFBQAweTEL MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRQwEgYDVQQHEwtTYW50YSBDbGFyYTEaMBgGA1UEChMR SW50ZWwgQ29ycG9yYXRpb24xKzApBgNVBAMTIkludGVsIEV4dGVybmFsIEJhc2ljIElzc3Vpbmcg Q0EgNEEwHhcNMTcwMTA5MjEyMzU4WhcNMTgwMTA0MjEyMzU4WjBFMRkwFwYDVQQDExBXYWxrZXIs IEJlbmphbWluMSgwJgYJKoZIhvcNAQkBFhliZW5qYW1pbi53YWxrZXJAaW50ZWwuY29tMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxFugJYk4Vd/Yvdmr8BdnGDdCkN1bc1KNCAQBhzC/ BWXw5nxpXWMYFBkTxahM78PtuwdtPDFqoHsMNEaX0miWeYjB6zKbKl7y0LEsSxlu9wjllEdWTYOP 9/m3UC0oITDn7L01adbsD5Sin6W1FMmjcBVrD51oy2orpwfvan3TNVRRQxt8dQz38hivXnona5tt toi+V8ved7o251HApvEwW7QtDfdML+RmBKBSf0MzGjZHPzoBfRrsBUZ0yRHJxlkYNeY99EAUUHwT npsySQSf0cxLmvA6/a4qPOUSitHit+cJQ58/EOt6PLrPGAbdu5sz9O+Iv+FUJakwUtg0sAY4RQID AQABo4ICNTCCAjEwHQYDVR0OBBYEFAU2hsr+3sx/M5e5WafmYD18VvX1MB8GA1UdIwQYMBaAFB5p KrTcKP5HGE4hCz+8rBEv8Jj1MGUGA1UdHwReMFwwWqBYoFaGVGh0dHA6Ly93d3cuaW50ZWwuY29t L3JlcG9zaXRvcnkvQ1JML0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElzc3VpbmclMjBDQSUy MDRBLmNybDCBnwYIKwYBBQUHAQEEgZIwgY8waQYIKwYBBQUHMAKGXWh0dHA6Ly93d3cuaW50ZWwu Y29tL3JlcG9zaXRvcnkvY2VydGlmaWNhdGVzL0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElz c3VpbmclMjBDQSUyMDRBLmNydDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AuaW50ZWwuY29tLzAL BgNVHQ8EBAMCB4AwPAYJKwYBBAGCNxUHBC8wLQYlKwYBBAGCNxUIhsOMdYSZ5VGD/YEohY6fU4KR wAlngd69OZXwQwIBZAIBCTAfBgNVHSUEGDAWBggrBgEFBQcDBAYKKwYBBAGCNwoDDDApBgkrBgEE AYI3FQoEHDAaMAoGCCsGAQUFBwMEMAwGCisGAQQBgjcKAwwwTwYDVR0RBEgwRqApBgorBgEEAYI3 FAIDoBsMGWJlbmphbWluLndhbGtlckBpbnRlbC5jb22BGWJlbmphbWluLndhbGtlckBpbnRlbC5j b20wDQYJKoZIhvcNAQEFBQADggEBAMQUzXgrfwDLl92M7wNqp24Xe1poeurJ8YVAy5a2UukwC/uX uXE8Duoz2jMJL90QETn17H7EQQu1J7kc059H6GyDU42MkzPA3mqZQimrTgOaalPXxWXoVl/UUoLB PJZXGF3Ef1p8b1UVdSnZZ8wTD/QTUw7UhgljKZ1td/raLV1h96x6lKCVkZ0UKU8be5M3FHQ/GZJ9 CgUjvN0m2mYOUHDkNzsUTJb4bsV7vZDa3zixm4Gxu2F/uq328AEJ6JJmXA+jjFOzQ0FI8sa7XOSR 1UPvZSrwyA00M/zFZaDTln+sFPFNseYYGYFU7P711D8Wj1Hv1V/C2G4rSRBJG5f1WF8xggIXMIIC EwIBATCBkDB5MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFDASBgNVBAcTC1NhbnRhIENsYXJh MRowGAYDVQQKExFJbnRlbCBDb3Jwb3JhdGlvbjErMCkGA1UEAxMiSW50ZWwgRXh0ZXJuYWwgQmFz aWMgSXNzdWluZyBDQSA0QQITMwAAi7krPkV7x3So3QAAAACLuTAJBgUrDgMCGgUAoF0wGAYJKoZI hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcwNDI2MjIwOTIzWjAjBgkqhkiG 9w0BCQQxFgQUMSL/fBPZAjrnYM+1KrLYwEFVNTkwDQYJKoZIhvcNAQEBBQAEggEAIiLff7gh6zs5 yYHFQ4h/wCUr2Ar8Pg++qBSp65iSBrvofjyyJ6CC9mo8oVwi1UhB6xA2n/40lEGYdvWV+SeDXOt9 RsZGg7ERqGdZ8KIIOXyVoyWGmIRLdeSjCgZ3feL35Wp8JiBMMZJ1G21bvpYOIVD7mvHnmk2XBNfW Mu9Oisl4sZqTEW2wNC8/DmmsviE/JQ6OGvzCE3qRY6hYvJoaobK0W4158qSIse+bKMdhdcrrqFqp CoDnhEdrLl4wKLzfO3Ho+8v6zTchLWFADCxl/+Iied6KHhAb/cQIk+YFShS72EvnV4jgMS/zvC0T xjX+xv1MtNUfN31oYixUGSi5ngAAAAAAAA== --===============8121447425713101029==--