From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0327116142022880523==" MIME-Version: 1.0 From: Ernest Zaslavsky Subject: Re: [SPDK] Memory buffer allocation in SPDK Date: Wed, 07 Feb 2018 07:51:02 +0200 Message-ID: <5a7a93c8.d4f4500a.21cf3.30c4@mx.google.com> In-Reply-To: 8F16ED7A-281B-4658-AB6E-016F6C4D5A30@intel.com List-ID: To: spdk@lists.01.org --===============0327116142022880523== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi James, Thanks for the info, sounds like we have a way to solve our problem. I have= some additional questions. The spdk_mem_register() function, it just adds = the region to the SPDK allocation map, right? It does not touch or modifies= the memory in any way, right? Is it safe to say that we may register with = SPDK all our pre-allocated memory when the application starts and just keep= it that way? Sincerely, E. From: Harris, James R Sent: Wednesday, February 7, 2018 2:11 AM To: Storage Performance Development Kit Subject: Re: [SPDK] Memory buffer allocation in SPDK From: SPDK on behalf of Ernest Zaslavsky Reply-To: Storage Performance Development Kit Date: Tuesday, February 6, 2018 at 6:32 AM To: "spdk(a)lists.01.org" Subject: [SPDK] Memory buffer allocation in SPDK Hi, I=E2=80=99ve found this discussion https://lists.01.org/pipermail/spdk/2016= -December/000251.html on memory allocation. I have similar situation =E2=80= =93 our application uses some allocator to create IO buffers, but of course= , SPDK would not accept such a buffer. So I have two questions: 1) I cant find in code the spdk_vtophys_register() mentioned in the above l= ink, is it removed? If not, where is it? How to use it? Hi Ernest, spdk_mem_register() is the equivalent function now.=C2=A0 This will registe= r the specified buffer not only for virtual-to-physical address translation= , but also for RDMA operations using the SPDK NVMe-oF driver. 2) In case our allocator uses hugepages and it is aligned on 4Kib boundary,= is there a way to make the SPDK believe that this buffer could be used for= NVMe IO? spdk_mem_register() is the function to use.=C2=A0 Note that currently for v= tophys translation, the specified buffer must be 2MiB aligned and an even 2= MiB multiple. Regards, -Jim --===============0327116142022880523== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj48aGVhZD48bWV0YSBodHRwLWVxdWl2PUNv bnRlbnQtVHlwZSBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPjxtZXRhIG5hbWU9 R2VuZXJhdG9yIGNvbnRlbnQ9Ik1pY3Jvc29mdCBXb3JkIDE1IChmaWx0ZXJlZCBtZWRpdW0pIj48 c3R5bGU+PCEtLQovKiBGb250IERlZmluaXRpb25zICovCkBmb250LWZhY2UKCXtmb250LWZhbWls eToiQ2FtYnJpYSBNYXRoIjsKCXBhbm9zZS0xOjIgNCA1IDMgNSA0IDYgMyAyIDQ7fQpAZm9udC1m YWNlCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsKCXBhbm9zZS0xOjIgMTUgNSAyIDIgMiA0IDMgMiA0 O30KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8KcC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2 Lk1zb05vcm1hbAoJe21hcmdpbjowaW47CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7Cglmb250LXNp emU6MTEuMHB0OwoJZm9udC1mYW1pbHk6IkNhbGlicmkiLHNhbnMtc2VyaWY7fQphOmxpbmssIHNw YW4uTXNvSHlwZXJsaW5rCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5OwoJY29sb3I6Ymx1ZTsKCXRl eHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQphOnZpc2l0ZWQsIHNwYW4uTXNvSHlwZXJsaW5rRm9s bG93ZWQKCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7Cgljb2xvcjojOTU0RjcyOwoJdGV4dC1kZWNv cmF0aW9uOnVuZGVybGluZTt9Ci5Nc29DaHBEZWZhdWx0Cgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0 LW9ubHk7fQpAcGFnZSBXb3JkU2VjdGlvbjEKCXtzaXplOjguNWluIDExLjBpbjsKCW1hcmdpbjox LjBpbiAxLjBpbiAxLjBpbiAxLjBpbjt9CmRpdi5Xb3JkU2VjdGlvbjEKCXtwYWdlOldvcmRTZWN0 aW9uMTt9Ci8qIExpc3QgRGVmaW5pdGlvbnMgKi8KQGxpc3QgbDAKCXttc28tbGlzdC1pZDo2OTU1 NDIwNjg7Cgltc28tbGlzdC10eXBlOmh5YnJpZDsKCW1zby1saXN0LXRlbXBsYXRlLWlkczotMjM0 Njg4NDE4IC0xIDY3Njk4NzEzIDY3Njk4NzE1IDY3Njk4NzAzIDY3Njk4NzEzIDY3Njk4NzE1IDY3 Njk4NzAzIDY3Njk4NzEzIDY3Njk4NzE1O30KQGxpc3QgbDA6bGV2ZWwxCgl7bXNvLWxldmVsLXRl eHQ6IiUxXCkiOwoJbXNvLWxldmVsLXRhYi1zdG9wOm5vbmU7Cgltc28tbGV2ZWwtbnVtYmVyLXBv c2l0aW9uOmxlZnQ7Cgl0ZXh0LWluZGVudDotLjI1aW47Cglmb250LWZhbWlseToiQ2FsaWJyaSIs c2Fucy1zZXJpZjsKCW1zby1iaWRpLWZvbnQtZmFtaWx5OkFyaWFsO30KQGxpc3QgbDA6bGV2ZWwy Cgl7bXNvLWxldmVsLW51bWJlci1mb3JtYXQ6YWxwaGEtbG93ZXI7Cgltc28tbGV2ZWwtdGFiLXN0 b3A6bm9uZTsKCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsKCXRleHQtaW5kZW50Oi0u MjVpbjt9CkBsaXN0IGwwOmxldmVsMwoJe21zby1sZXZlbC1udW1iZXItZm9ybWF0OnJvbWFuLWxv d2VyOwoJbXNvLWxldmVsLXRhYi1zdG9wOm5vbmU7Cgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9u OnJpZ2h0OwoJdGV4dC1pbmRlbnQ6LTkuMHB0O30KQGxpc3QgbDA6bGV2ZWw0Cgl7bXNvLWxldmVs LXRhYi1zdG9wOm5vbmU7Cgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7Cgl0ZXh0LWlu ZGVudDotLjI1aW47fQpAbGlzdCBsMDpsZXZlbDUKCXttc28tbGV2ZWwtbnVtYmVyLWZvcm1hdDph bHBoYS1sb3dlcjsKCW1zby1sZXZlbC10YWItc3RvcDpub25lOwoJbXNvLWxldmVsLW51bWJlci1w b3NpdGlvbjpsZWZ0OwoJdGV4dC1pbmRlbnQ6LS4yNWluO30KQGxpc3QgbDA6bGV2ZWw2Cgl7bXNv LWxldmVsLW51bWJlci1mb3JtYXQ6cm9tYW4tbG93ZXI7Cgltc28tbGV2ZWwtdGFiLXN0b3A6bm9u ZTsKCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246cmlnaHQ7Cgl0ZXh0LWluZGVudDotOS4wcHQ7 fQpAbGlzdCBsMDpsZXZlbDcKCXttc28tbGV2ZWwtdGFiLXN0b3A6bm9uZTsKCW1zby1sZXZlbC1u dW1iZXItcG9zaXRpb246bGVmdDsKCXRleHQtaW5kZW50Oi0uMjVpbjt9CkBsaXN0IGwwOmxldmVs OAoJe21zby1sZXZlbC1udW1iZXItZm9ybWF0OmFscGhhLWxvd2VyOwoJbXNvLWxldmVsLXRhYi1z dG9wOm5vbmU7Cgltc28tbGV2ZWwtbnVtYmVyLXBvc2l0aW9uOmxlZnQ7Cgl0ZXh0LWluZGVudDot LjI1aW47fQpAbGlzdCBsMDpsZXZlbDkKCXttc28tbGV2ZWwtbnVtYmVyLWZvcm1hdDpyb21hbi1s b3dlcjsKCW1zby1sZXZlbC10YWItc3RvcDpub25lOwoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlv bjpyaWdodDsKCXRleHQtaW5kZW50Oi05LjBwdDt9Cm9sCgl7bWFyZ2luLWJvdHRvbTowaW47fQp1 bAoJe21hcmdpbi1ib3R0b206MGluO30KLS0+PC9zdHlsZT48L2hlYWQ+PGJvZHkgbGFuZz1FTi1V UyBsaW5rPWJsdWUgdmxpbms9IiM5NTRGNzIiPjxkaXYgY2xhc3M9V29yZFNlY3Rpb24xPjxwIGNs YXNzPU1zb05vcm1hbD5IaSBKYW1lcyw8L3A+PHAgY2xhc3M9TXNvTm9ybWFsPlRoYW5rcyBmb3Ig dGhlIGluZm8sIHNvdW5kcyBsaWtlIHdlIGhhdmUgYSB3YXkgdG8gc29sdmUgb3VyIHByb2JsZW0u IEkgaGF2ZSBzb21lIGFkZGl0aW9uYWwgcXVlc3Rpb25zLiBUaGUgc3Bka19tZW1fcmVnaXN0ZXIo KSBmdW5jdGlvbiwgaXQganVzdCBhZGRzIHRoZSByZWdpb24gdG8gdGhlIFNQREsgYWxsb2NhdGlv biBtYXAsIHJpZ2h0PyBJdCBkb2VzIG5vdCB0b3VjaCBvciBtb2RpZmllcyB0aGUgbWVtb3J5IGlu IGFueSB3YXksIHJpZ2h0PyBJcyBpdCBzYWZlIHRvIHNheSB0aGF0IHdlIG1heSByZWdpc3RlciB3 aXRoIFNQREsgYWxsIG91ciBwcmUtYWxsb2NhdGVkIG1lbW9yeSB3aGVuIHRoZSBhcHBsaWNhdGlv biBzdGFydHMgYW5kIGp1c3Qga2VlcCBpdCB0aGF0IHdheT88L3A+PHAgY2xhc3M9TXNvTm9ybWFs PlNpbmNlcmVseSw8bzpwPjwvbzpwPjwvcD48cCBjbGFzcz1Nc29Ob3JtYWw+RS48bzpwPjwvbzpw PjwvcD48cCBjbGFzcz1Nc29Ob3JtYWw+PG86cD4mbmJzcDs8L286cD48L3A+PHAgY2xhc3M9TXNv Tm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPjxkaXYgc3R5bGU9J21zby1lbGVtZW50OnBhcmEt Ym9yZGVyLWRpdjtib3JkZXI6bm9uZTtib3JkZXItdG9wOnNvbGlkICNFMUUxRTEgMS4wcHQ7cGFk ZGluZzozLjBwdCAwaW4gMGluIDBpbic+PHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdib3JkZXI6 bm9uZTtwYWRkaW5nOjBpbic+PGI+RnJvbTogPC9iPjxhIGhyZWY9Im1haWx0bzpqYW1lcy5yLmhh cnJpc0BpbnRlbC5jb20iPkhhcnJpcywgSmFtZXMgUjwvYT48YnI+PGI+U2VudDogPC9iPldlZG5l c2RheSwgRmVicnVhcnkgNywgMjAxOCAyOjExIEFNPGJyPjxiPlRvOiA8L2I+PGEgaHJlZj0ibWFp bHRvOnNwZGtAbGlzdHMuMDEub3JnIj5TdG9yYWdlIFBlcmZvcm1hbmNlIERldmVsb3BtZW50IEtp dDwvYT48YnI+PGI+U3ViamVjdDogPC9iPlJlOiBbU1BES10gTWVtb3J5IGJ1ZmZlciBhbGxvY2F0 aW9uIGluIFNQREs8L3A+PC9kaXY+PHAgY2xhc3M9TXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+ PC9wPjxwIGNsYXNzPU1zb05vcm1hbD48bzpwPiZuYnNwOzwvbzpwPjwvcD48cCBjbGFzcz1Nc29O b3JtYWw+PG86cD4mbmJzcDs8L286cD48L3A+PGRpdiBzdHlsZT0nYm9yZGVyOm5vbmU7Ym9yZGVy LXRvcDpzb2xpZCAjQjVDNERGIDEuMHB0O3BhZGRpbmc6My4wcHQgMGluIDBpbiAwaW4nPjxwIGNs YXNzPU1zb05vcm1hbCBzdHlsZT0nbWFyZ2luLWxlZnQ6LjVpbic+PGI+PHNwYW4gc3R5bGU9J2Nv bG9yOmJsYWNrJz5Gcm9tOiA8L3NwYW4+PC9iPjxzcGFuIHN0eWxlPSdjb2xvcjpibGFjayc+U1BE SyAmbHQ7c3Bkay1ib3VuY2VzQGxpc3RzLjAxLm9yZyZndDsgb24gYmVoYWxmIG9mIEVybmVzdCBa YXNsYXZza3kgJmx0O2tyZXV6ZXJrcmllZ0BnbWFpbC5jb20mZ3Q7PGJyPjxiPlJlcGx5LVRvOiA8 L2I+U3RvcmFnZSBQZXJmb3JtYW5jZSBEZXZlbG9wbWVudCBLaXQgJmx0O3NwZGtAbGlzdHMuMDEu b3JnJmd0Ozxicj48Yj5EYXRlOiA8L2I+VHVlc2RheSwgRmVicnVhcnkgNiwgMjAxOCBhdCA2OjMy IEFNPGJyPjxiPlRvOiA8L2I+JnF1b3Q7c3Bka0BsaXN0cy4wMS5vcmcmcXVvdDsgJmx0O3NwZGtA bGlzdHMuMDEub3JnJmd0Ozxicj48Yj5TdWJqZWN0OiA8L2I+W1NQREtdIE1lbW9yeSBidWZmZXIg YWxsb2NhdGlvbiBpbiBTUERLPC9zcGFuPjxzcGFuIHN0eWxlPSdmb250LXNpemU6MTIuMHB0O2Nv bG9yOmJsYWNrJz48bzpwPjwvbzpwPjwvc3Bhbj48L3A+PC9kaXY+PGRpdj48cCBjbGFzcz1Nc29O b3JtYWwgc3R5bGU9J21hcmdpbi1sZWZ0Oi41aW4nPjxzcGFuIHN0eWxlPSdmb250LWZhbWlseToi VGltZXMgTmV3IFJvbWFuIixzZXJpZic+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPjwvZGl2 PjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0nbWFyZ2luLWxlZnQ6LjVpbic+SGksPG86cD48L286 cD48L3A+PHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtYXJnaW4tbGVmdDouNWluJz5J4oCZdmUg Zm91bmQgdGhpcyBkaXNjdXNzaW9uIDxhIGhyZWY9Imh0dHBzOi8vbGlzdHMuMDEub3JnL3BpcGVy bWFpbC9zcGRrLzIwMTYtRGVjZW1iZXIvMDAwMjUxLmh0bWwiPmh0dHBzOi8vbGlzdHMuMDEub3Jn L3BpcGVybWFpbC9zcGRrLzIwMTYtRGVjZW1iZXIvMDAwMjUxLmh0bWw8L2E+IG9uIG1lbW9yeSBh bGxvY2F0aW9uLiBJIGhhdmUgc2ltaWxhciBzaXR1YXRpb24g4oCTIG91ciBhcHBsaWNhdGlvbiB1 c2VzIHNvbWUgYWxsb2NhdG9yIHRvIGNyZWF0ZSBJTyBidWZmZXJzLCBidXQgb2YgY291cnNlLCBT UERLIHdvdWxkIG5vdCBhY2NlcHQgc3VjaCBhIGJ1ZmZlci4gU28gSSBoYXZlIHR3byBxdWVzdGlv bnM6PG86cD48L286cD48L3A+PHAgY2xhc3M9TXNvTm9ybWFsIHN0eWxlPSdtYXJnaW4tbGVmdDox LjBpbjt0ZXh0LWluZGVudDotLjI1aW47bXNvLWxpc3Q6bDAgbGV2ZWwxIGxmbzEnPjwhW2lmICFz dXBwb3J0TGlzdHNdPjxzcGFuIHN0eWxlPSdtc28tbGlzdDpJZ25vcmUnPjEpPHNwYW4gc3R5bGU9 J2ZvbnQ6Ny4wcHQgIlRpbWVzIE5ldyBSb21hbiInPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyA8L3NwYW4+PC9zcGFuPjwhW2VuZGlmXT48c3BhbiBkaXI9TFRSPjwvc3Bhbj5JIGNhbnQg ZmluZCBpbiBjb2RlIHRoZSBzcGRrX3Z0b3BoeXNfcmVnaXN0ZXIoKSBtZW50aW9uZWQgaW4gdGhl IGFib3ZlIGxpbmssIGlzIGl0IHJlbW92ZWQ/IElmIG5vdCwgd2hlcmUgaXMgaXQ/IEhvdyB0byB1 c2UgaXQ/PG86cD48L286cD48L3A+PHAgY2xhc3M9TXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+ PC9wPjxwIGNsYXNzPU1zb05vcm1hbD5IaSBFcm5lc3QsPG86cD48L286cD48L3A+PHAgY2xhc3M9 TXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPjxwIGNsYXNzPU1zb05vcm1hbD5zcGRrX21l bV9yZWdpc3RlcigpIGlzIHRoZSBlcXVpdmFsZW50IGZ1bmN0aW9uIG5vdy4mbmJzcDsgVGhpcyB3 aWxsIHJlZ2lzdGVyIHRoZSBzcGVjaWZpZWQgYnVmZmVyIG5vdCBvbmx5IGZvciB2aXJ0dWFsLXRv LXBoeXNpY2FsIGFkZHJlc3MgdHJhbnNsYXRpb24sIGJ1dCBhbHNvIGZvciBSRE1BIG9wZXJhdGlv bnMgdXNpbmcgdGhlIFNQREsgTlZNZS1vRiBkcml2ZXIuPG86cD48L286cD48L3A+PHAgY2xhc3M9 TXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPjxwIGNsYXNzPU1zb05vcm1hbCBzdHlsZT0n bWFyZ2luLWxlZnQ6MS4waW47dGV4dC1pbmRlbnQ6LS4yNWluO21zby1saXN0OmwwIGxldmVsMSBs Zm8xJz48IVtpZiAhc3VwcG9ydExpc3RzXT48c3BhbiBzdHlsZT0nbXNvLWxpc3Q6SWdub3JlJz4y KTxzcGFuIHN0eWxlPSdmb250OjcuMHB0ICJUaW1lcyBOZXcgUm9tYW4iJz4mbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsgPC9zcGFuPjwvc3Bhbj48IVtlbmRpZl0+PHNwYW4gZGlyPUxUUj48 L3NwYW4+SW4gY2FzZSBvdXIgYWxsb2NhdG9yIHVzZXMgaHVnZXBhZ2VzIGFuZCBpdCBpcyBhbGln bmVkIG9uIDRLaWIgYm91bmRhcnksIGlzIHRoZXJlIGEgd2F5IHRvIG1ha2UgdGhlIFNQREsgYmVs aWV2ZSB0aGF0IHRoaXMgYnVmZmVyIGNvdWxkIGJlIHVzZWQgZm9yIE5WTWUgSU8/PG86cD48L286 cD48L3A+PHAgY2xhc3M9TXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPjxwIGNsYXNzPU1z b05vcm1hbD5zcGRrX21lbV9yZWdpc3RlcigpIGlzIHRoZSBmdW5jdGlvbiB0byB1c2UuJm5ic3A7 IE5vdGUgdGhhdCBjdXJyZW50bHkgZm9yIHZ0b3BoeXMgdHJhbnNsYXRpb24sIHRoZSBzcGVjaWZp ZWQgYnVmZmVyIG11c3QgYmUgMk1pQiBhbGlnbmVkIGFuZCBhbiBldmVuIDJNaUIgbXVsdGlwbGUu PG86cD48L286cD48L3A+PHAgY2xhc3M9TXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPjxw IGNsYXNzPU1zb05vcm1hbD5SZWdhcmRzLDxvOnA+PC9vOnA+PC9wPjxwIGNsYXNzPU1zb05vcm1h bD48bzpwPiZuYnNwOzwvbzpwPjwvcD48cCBjbGFzcz1Nc29Ob3JtYWw+LUppbTxvOnA+PC9vOnA+ PC9wPjxwIGNsYXNzPU1zb05vcm1hbD48bzpwPiZuYnNwOzwvbzpwPjwvcD48cCBjbGFzcz1Nc29O b3JtYWw+PG86cD4mbmJzcDs8L286cD48L3A+PC9kaXY+PC9ib2R5PjwvaHRtbD4= --===============0327116142022880523==--