From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============1837571703595495538==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Number of NVMe devices per core Date: Thu, 15 Feb 2018 14:50:38 +0000 Message-ID: <89940C85-FD9D-4890-9812-47F7A0FC27A2@intel.com> In-Reply-To: CAHK_hpK8r85mqFQ3-b97Otb5fYHOFsVURoRMKr6ZNTBB=y41jQ@mail.gmail.com List-ID: To: spdk@lists.01.org --===============1837571703595495538== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Ernest, The answer depends on many different factors =E2=80=93 the IOPs limit per d= evice, CPU core frequency, Turbo, etc. As a frame of reference, the team h= ere at Intel has measured over 3M IO/s on a single Intel Xeon core[1]. -Jim [1] https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2016/= 20160809_FA12_P3_Prepalli.pdf - slide 63 From: SPDK on behalf of Ernest Zed Reply-To: Storage Performance Development Kit Date: Thursday, February 15, 2018 at 6:46 AM To: Storage Performance Development Kit Subject: [SPDK] Number of NVMe devices per core Hi All, I ran 'perf' with different settings to see how to get most of NVMe disks. = I see that performance degrades once cpu mask puts more than 2 devises per = core. Is it OK? what is the theoretical (or empirical) limit of devices one= core can handle? Sincerely, Ernest --===============1837571703595495538== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iVGl0bGUiIGNvbnRlbnQ9IiI+DQo8bWV0YSBuYW1lPSJLZXl3b3JkcyIgY29udGVu dD0iIj4NCjxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUg KGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxlPjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8N CkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0 IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkRlbmdYaWFuOw0K CXBhbm9zZS0xOjIgMSA2IDAgMyAxIDEgMSAxIDE7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWls eTpDYWxpYnJpOw0KCXBhbm9zZS0xOjIgMTUgNSAyIDIgMiA0IDMgMiA0O30NCi8qIFN0eWxlIERl ZmluaXRpb25zICovDQpwLk1zb05vcm1hbCwgbGkuTXNvTm9ybWFsLCBkaXYuTXNvTm9ybWFsDQoJ e21hcmdpbjowaW47DQoJbWFyZ2luLWJvdHRvbTouMDAwMXB0Ow0KCWZvbnQtc2l6ZToxMi4wcHQ7 DQoJZm9udC1mYW1pbHk6IlRpbWVzIE5ldyBSb21hbiI7fQ0KYTpsaW5rLCBzcGFuLk1zb0h5cGVy bGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6IzA1NjNDMTsNCgl0ZXh0LWRl Y29yYXRpb246dW5kZXJsaW5lO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dl ZA0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6Izk1NEY3MjsNCgl0ZXh0LWRlY29y YXRpb246dW5kZXJsaW5lO30NCnNwYW4uRW1haWxTdHlsZTE3DQoJe21zby1zdHlsZS10eXBlOnBl cnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5OkNhbGlicmk7DQoJY29sb3I6d2luZG93dGV4dDt9 DQpzcGFuLm1zb0lucw0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTsNCgltc28tc3R5bGUt bmFtZToiIjsNCgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lOw0KCWNvbG9yOnRlYWw7fQ0KLk1z b0NocERlZmF1bHQNCgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1zaXplOjEw LjBwdDt9DQpAcGFnZSBXb3JkU2VjdGlvbjENCgl7c2l6ZTo4LjVpbiAxMS4waW47DQoJbWFyZ2lu OjEuMGluIDEuMGluIDEuMGluIDEuMGluO30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3Jk U2VjdGlvbjE7fQ0KLS0+PC9zdHlsZT4NCjwvaGVhZD4NCjxib2R5IGJnY29sb3I9IndoaXRlIiBs YW5nPSJFTi1VUyIgbGluaz0iIzA1NjNDMSIgdmxpbms9IiM5NTRGNzIiPg0KPGRpdiBjbGFzcz0i V29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPkhpIEVybmVzdCw8bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw dDtmb250LWZhbWlseTpDYWxpYnJpIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWls eTpDYWxpYnJpIj5UaGUgYW5zd2VyIGRlcGVuZHMgb24gbWFueSBkaWZmZXJlbnQgZmFjdG9ycyDi gJMgdGhlIElPUHMgbGltaXQgcGVyIGRldmljZSwgQ1BVIGNvcmUgZnJlcXVlbmN5LCBUdXJibywg ZXRjLiZuYnNwOyBBcyBhIGZyYW1lIG9mIHJlZmVyZW5jZSwgdGhlIHRlYW0gaGVyZSBhdCBJbnRl bCBoYXMgbWVhc3VyZWQgb3ZlciAzTSBJTy9zIG9uIGENCiBzaW5nbGUgSW50ZWwgWGVvbiBjb3Jl WzFdLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0 eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPjxvOnA+Jm5ic3A7PC9v OnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmkiPi1KaW08bzpwPjwvbzpwPjwvc3Bhbj48L3A+ DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250 LWZhbWlseTpDYWxpYnJpIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxp YnJpIj5bMV0gPGEgaHJlZj0iaHR0cHM6Ly93d3cuZmxhc2htZW1vcnlzdW1taXQuY29tL0VuZ2xp c2gvQ29sbGF0ZXJhbHMvUHJvY2VlZGluZ3MvMjAxNi8yMDE2MDgwOV9GQTEyX1AzX1ByZXBhbGxp LnBkZiI+DQpodHRwczovL3d3dy5mbGFzaG1lbW9yeXN1bW1pdC5jb20vRW5nbGlzaC9Db2xsYXRl cmFscy9Qcm9jZWVkaW5ncy8yMDE2LzIwMTYwODA5X0ZBMTJfUDNfUHJlcGFsbGkucGRmPC9hPiAt IHNsaWRlIDYzPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaSI+PG86cD4mbmJz cDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaSI+PG86cD4mbmJzcDs8L286cD48L3Nw YW4+PC9wPg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAjQjVDNERG IDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIg c3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTpDYWxp YnJpO2NvbG9yOmJsYWNrIj5Gcm9tOg0KPC9zcGFuPjwvYj48c3BhbiBzdHlsZT0iZm9udC1mYW1p bHk6Q2FsaWJyaTtjb2xvcjpibGFjayI+U1BESyAmbHQ7c3Bkay1ib3VuY2VzQGxpc3RzLjAxLm9y ZyZndDsgb24gYmVoYWxmIG9mIEVybmVzdCBaZWQgJmx0O2tyZXV6ZXJrcmllZ0BnbWFpbC5jb20m Z3Q7PGJyPg0KPGI+UmVwbHktVG86IDwvYj5TdG9yYWdlIFBlcmZvcm1hbmNlIERldmVsb3BtZW50 IEtpdCAmbHQ7c3Bka0BsaXN0cy4wMS5vcmcmZ3Q7PGJyPg0KPGI+RGF0ZTogPC9iPlRodXJzZGF5 LCBGZWJydWFyeSAxNSwgMjAxOCBhdCA2OjQ2IEFNPGJyPg0KPGI+VG86IDwvYj5TdG9yYWdlIFBl cmZvcm1hbmNlIERldmVsb3BtZW50IEtpdCAmbHQ7c3Bka0BsaXN0cy4wMS5vcmcmZ3Q7PGJyPg0K PGI+U3ViamVjdDogPC9iPltTUERLXSBOdW1iZXIgb2YgTlZNZSBkZXZpY2VzIHBlciBjb3JlPG86 cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIg c3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0Oi41aW4iPkhpIEFs bCwgPG86cD48L286cD48L3A+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1h cmdpbi1sZWZ0Oi41aW4iPkkgcmFuICdwZXJmJyB3aXRoIGRpZmZlcmVudCBzZXR0aW5ncyB0byBz ZWUgaG93IHRvIGdldCBtb3N0IG9mIE5WTWUgZGlza3MuIEkgc2VlIHRoYXQgcGVyZm9ybWFuY2Ug ZGVncmFkZXMgb25jZSBjcHUgbWFzayBwdXRzIG1vcmUgdGhhbiAyIGRldmlzZXMgcGVyIGNvcmUu IElzIGl0IE9LPyB3aGF0IGlzIHRoZSB0aGVvcmV0aWNhbCAob3IgZW1waXJpY2FsKSBsaW1pdA0K IG9mIGRldmljZXMgb25lIGNvcmUgY2FuIGhhbmRsZT8mbmJzcDs8bzpwPjwvbzpwPjwvcD4NCjwv ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVmdDouNWlu Ij5TaW5jZXJlbHksPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6LjVpbiI+RXJuZXN0PG86cD48L286cD48L3A+DQo8 L2Rpdj4NCjwvZGl2Pg0KPC9kaXY+DQo8L2JvZHk+DQo8L2h0bWw+DQo= --===============1837571703595495538==--