From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============5459425571622874684==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] queue pair vs cpu cores Date: Wed, 29 Mar 2017 20:46:04 +0000 Message-ID: <1490820363.62307.22.camel@intel.com> In-Reply-To: BAF7572087063A4BAD2F325FC7533F42049143061EE0@FMSAMAIL.fmsa.local List-ID: To: spdk@lists.01.org --===============5459425571622874684== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, 2017-03-27 at 17:06 -0700, Isaac Otsiabah wrote: > =C2=A0 > Ben, the implementation of your second paragraph may be a little > easier, correct? Is there an example somewhere that shows how to use > SPDK to do blocking I/O. It does not have to be robust but, can serve > as an example of how to structure the code (using threads or > processes) to do blocking I/O=C2=A0 for legacy applications. The second approach is easier if you are trying to integrate with an existing application that is implemented with a pool of threads performing blocking I/O. We happen to have a great example of how to do this that we're about to release. See lib/blobfs on the master branch. This is implementing blocking filesystem-like calls by message passing to a single core performing operations asynchronously. We use this to implement an alternate backend to RocksDB that bypasses the kernel filesystem. RocksDB just happens to be implemented as a pool of threads performing blocking I/O. > =C2=A0 > Isaac > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, > Benjamin > Sent: Monday, March 27, 2017 3:55 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] queue pair vs cpu cores > =C2=A0 > Hi Raj, > =C2=A0 > It=E2=80=99s true - the SPDK event framework spins in reactor loops on ea= ch > core, using them all up even if you aren=E2=80=99t busy, and it=E2=80=99s= a model of > how we think applications should generally be structured for best > performance. However, the vast majority of the components in SPDK > don=E2=80=99t require the event framework. These components don=E2=80=99t= actually do > anything unless you call one of their functions from your > application. In particular, your application is in charge of making > the call to poll for completions, so one quick solution is to poll > less often. The =E2=80=98less often=E2=80=99 can even be intelligent =E2= =80=93 for instance > if no I/O is outstanding don=E2=80=99t bother polling and maybe sleep for= a > short period instead. The SPDK event framework is actually capable of > sleeping to save CPU cycles (off by default), so you can use that as > a concrete example. > =C2=A0 > There are other designs that work well too. For instance, many legacy > applications are written with a pool of threads (or processes) doing > blocking I/O operations. You can dedicate one thread to actually > doing I/O with SPDK asynchronously, and then have the pool of threads > pass a message to the main I/O thread and block on a semaphore until > it is completed. Then, the pool of threads all goes to sleep just as > if they were doing a blocking I/O operation, and only the one > dedicated I/O thread spins and polls. You can, of course, combine > this technique with the first one. > =C2=A0 > Another design is to simply limit your application to fewer CPU > cores. With SPDK making the I/O path so efficient, does that enable > you to get away with fewer cores? Or you can funnel more work to your > application such that it is rarely idle, which is often possible in > the cloud. > =C2=A0 > These are at least a few ideas. I=E2=80=99m sure others will come up with > additional clever ways of managing this. > =C2=A0 > -Ben > =C2=A0 > =C2=A0 > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of > Rajinikanth Pandurangan > Sent: Wednesday, March 22, 2017 2:21 PM > To: spdk(a)lists.01.org > Subject: [SPDK] queue pair vs cpu cores > Hi > As per understanding, using SPDK we can achieve full performance of a > SSD with 1 core and there is 1 queue pair per thread and 1 thread per > core.=C2=A0=C2=A0 Typically applications spans multiple threads.=C2=A0 As= locking is > discouraged in SPDK, we have to map each thread to a queue pair.=C2=A0 By > doing so, we might end up keeping all the cores busy just for IO > polling.=C2=A0 How do you guys compare this scenario with Kernel mode in- > terms of overall performance and utilization?=C2=A0 How do we control cpu > utilization with no locks while satisfying application that spans > multiple threads? > =C2=A0 > Thanks, > Raj > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============5459425571622874684== Content-Type: application/x-pkcs7-signature MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKdTCCBOsw ggPToAMCAQICEFLpAsoR6ESdlGU4L6MaMLswDQYJKoZIhvcNAQEFBQAwbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5hbCBUVFAgTmV0 d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9vdDAeFw0xMzAzMTkwMDAwMDBa Fw0yMDA1MzAxMDQ4MzhaMHkxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEUMBIGA1UEBxMLU2Fu dGEgQ2xhcmExGjAYBgNVBAoTEUludGVsIENvcnBvcmF0aW9uMSswKQYDVQQDEyJJbnRlbCBFeHRl cm5hbCBCYXNpYyBJc3N1aW5nIENBIDRBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA 4LDMgJ3YSVX6A9sE+jjH3b+F3Xa86z3LLKu/6WvjIdvUbxnoz2qnvl9UKQI3sE1zURQxrfgvtP0b Pgt1uDwAfLc6H5eqnyi+7FrPsTGCR4gwDmq1WkTQgNDNXUgb71e9/6sfq+WfCDpi8ScaglyLCRp7 ph/V60cbitBvnZFelKCDBh332S6KG3bAdnNGB/vk86bwDlY6omDs6/RsfNwzQVwo/M3oPrux6y6z yIoRulfkVENbM0/9RrzQOlyK4W5Vk4EEsfW2jlCV4W83QKqRccAKIUxw2q/HoHVPbbETrrLmE6RR Z/+eWlkGWl+mtx42HOgOmX0BRdTRo9vH7yeBowIDAQABo4IBdzCCAXMwHwYDVR0jBBgwFoAUrb2Y ejS0Jvf6xCZU7wO94CTLVBowHQYDVR0OBBYEFB5pKrTcKP5HGE4hCz+8rBEv8Jj1MA4GA1UdDwEB /wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMDYGA1UdJQQvMC0GCCsGAQUFBwMEBgorBgEEAYI3 CgMEBgorBgEEAYI3CgMMBgkrBgEEAYI3FQUwFwYDVR0gBBAwDjAMBgoqhkiG+E0BBQFpMEkGA1Ud HwRCMEAwPqA8oDqGOGh0dHA6Ly9jcmwudHJ1c3QtcHJvdmlkZXIuY29tL0FkZFRydXN0RXh0ZXJu YWxDQVJvb3QuY3JsMDoGCCsGAQUFBwEBBC4wLDAqBggrBgEFBQcwAYYeaHR0cDovL29jc3AudHJ1 c3QtcHJvdmlkZXIuY29tMDUGA1UdHgQuMCygKjALgQlpbnRlbC5jb20wG6AZBgorBgEEAYI3FAID oAsMCWludGVsLmNvbTANBgkqhkiG9w0BAQUFAAOCAQEAKcLNo/2So1Jnoi8G7W5Q6FSPq1fmyKW3 sSDf1amvyHkjEgd25n7MKRHGEmRxxoziPKpcmbfXYU+J0g560nCo5gPF78Wd7ZmzcmCcm1UFFfIx fw6QA19bRpTC8bMMaSSEl8y39Pgwa+HENmoPZsM63DdZ6ziDnPqcSbcfYs8qd/m5d22rpXq5IGVU tX6LX7R/hSSw/3sfATnBLgiJtilVyY7OGGmYKCAS2I04itvSS1WtecXTt9OZDyNbl7LtObBrgMLh ZkpJW+pOR9f3h5VG2S5uKkA7Th9NC9EoScdwQCAIw+UWKbSQ0Isj2UFL7fHKvmqWKVTL98sRzvI3 seNC4DCCBYIwggRqoAMCAQICEzMAAIu5Kz5Fe8d0qN0AAAAAi7kwDQYJKoZIhvcNAQEFBQAweTEL MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRQwEgYDVQQHEwtTYW50YSBDbGFyYTEaMBgGA1UEChMR SW50ZWwgQ29ycG9yYXRpb24xKzApBgNVBAMTIkludGVsIEV4dGVybmFsIEJhc2ljIElzc3Vpbmcg Q0EgNEEwHhcNMTcwMTA5MjEyMzU4WhcNMTgwMTA0MjEyMzU4WjBFMRkwFwYDVQQDExBXYWxrZXIs IEJlbmphbWluMSgwJgYJKoZIhvcNAQkBFhliZW5qYW1pbi53YWxrZXJAaW50ZWwuY29tMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxFugJYk4Vd/Yvdmr8BdnGDdCkN1bc1KNCAQBhzC/ BWXw5nxpXWMYFBkTxahM78PtuwdtPDFqoHsMNEaX0miWeYjB6zKbKl7y0LEsSxlu9wjllEdWTYOP 9/m3UC0oITDn7L01adbsD5Sin6W1FMmjcBVrD51oy2orpwfvan3TNVRRQxt8dQz38hivXnona5tt toi+V8ved7o251HApvEwW7QtDfdML+RmBKBSf0MzGjZHPzoBfRrsBUZ0yRHJxlkYNeY99EAUUHwT npsySQSf0cxLmvA6/a4qPOUSitHit+cJQ58/EOt6PLrPGAbdu5sz9O+Iv+FUJakwUtg0sAY4RQID AQABo4ICNTCCAjEwHQYDVR0OBBYEFAU2hsr+3sx/M5e5WafmYD18VvX1MB8GA1UdIwQYMBaAFB5p KrTcKP5HGE4hCz+8rBEv8Jj1MGUGA1UdHwReMFwwWqBYoFaGVGh0dHA6Ly93d3cuaW50ZWwuY29t L3JlcG9zaXRvcnkvQ1JML0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElzc3VpbmclMjBDQSUy MDRBLmNybDCBnwYIKwYBBQUHAQEEgZIwgY8waQYIKwYBBQUHMAKGXWh0dHA6Ly93d3cuaW50ZWwu Y29tL3JlcG9zaXRvcnkvY2VydGlmaWNhdGVzL0ludGVsJTIwRXh0ZXJuYWwlMjBCYXNpYyUyMElz c3VpbmclMjBDQSUyMDRBLmNydDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AuaW50ZWwuY29tLzAL BgNVHQ8EBAMCB4AwPAYJKwYBBAGCNxUHBC8wLQYlKwYBBAGCNxUIhsOMdYSZ5VGD/YEohY6fU4KR wAlngd69OZXwQwIBZAIBCTAfBgNVHSUEGDAWBggrBgEFBQcDBAYKKwYBBAGCNwoDDDApBgkrBgEE AYI3FQoEHDAaMAoGCCsGAQUFBwMEMAwGCisGAQQBgjcKAwwwTwYDVR0RBEgwRqApBgorBgEEAYI3 FAIDoBsMGWJlbmphbWluLndhbGtlckBpbnRlbC5jb22BGWJlbmphbWluLndhbGtlckBpbnRlbC5j b20wDQYJKoZIhvcNAQEFBQADggEBAMQUzXgrfwDLl92M7wNqp24Xe1poeurJ8YVAy5a2UukwC/uX uXE8Duoz2jMJL90QETn17H7EQQu1J7kc059H6GyDU42MkzPA3mqZQimrTgOaalPXxWXoVl/UUoLB PJZXGF3Ef1p8b1UVdSnZZ8wTD/QTUw7UhgljKZ1td/raLV1h96x6lKCVkZ0UKU8be5M3FHQ/GZJ9 CgUjvN0m2mYOUHDkNzsUTJb4bsV7vZDa3zixm4Gxu2F/uq328AEJ6JJmXA+jjFOzQ0FI8sa7XOSR 1UPvZSrwyA00M/zFZaDTln+sFPFNseYYGYFU7P711D8Wj1Hv1V/C2G4rSRBJG5f1WF8xggIXMIIC EwIBATCBkDB5MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFDASBgNVBAcTC1NhbnRhIENsYXJh MRowGAYDVQQKExFJbnRlbCBDb3Jwb3JhdGlvbjErMCkGA1UEAxMiSW50ZWwgRXh0ZXJuYWwgQmFz aWMgSXNzdWluZyBDQSA0QQITMwAAi7krPkV7x3So3QAAAACLuTAJBgUrDgMCGgUAoF0wGAYJKoZI hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTcwMzI5MjA0NjAzWjAjBgkqhkiG 9w0BCQQxFgQU9kCYJBgsQEyd+zufIbX3+AdTdbowDQYJKoZIhvcNAQEBBQAEggEAOFeh5T1GChrw DRPwVMz0fAZ+JWgRDrvxJcTRNhlDYfd2ONRD8yiJ/P2G4P1TWptzbmuVRNjh5eHuGIf6E7z/mAU+ bzbep548wLvAl1z7LvQM2QDYusD7xB/Ri0ygFho2AG7V6YbrcRb9j/2Tem9QdnmyqcZ3REI5/Jvl 9/FB0moKFGXYWJRMdFYZcaKgONmLJ9swbyOYJl3WEGviVx5rNUworlX08Bi1QOdbZuPTgmTJIxw7 VFxVv7wcCCv/BlFP5PXyH+H9fg6ta7m+noP3eAwIkQEowLenwdftegLLzvobCnKnAI0ki6tU0yey qFoYmGwnKk0eihB23OkoN+RP3QAAAAAAAA== --===============5459425571622874684==--