From mboxrd@z Thu Jan 1 00:00:00 1970 From: Yongseok Koh Subject: Re: [PATCH 0/4] Allow using external memory without malloc Date: Wed, 12 Dec 2018 12:55:43 +0000 Message-ID: <20181212125528.GA5538@minint-98vp2qg> References: <7F52E2C1-4552-48E7-9AFA-BEAF4D8E45D7@mellanox.com> <57d7ad23-bebd-f788-5ac9-540804ecf5da@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Cc: Shahaf Shuler , "dev@dpdk.org" , Thomas Monjalon , "shreyansh.jain@nxp.com" To: "Burakov, Anatoly" Return-path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10061.outbound.protection.outlook.com [40.107.1.61]) by dpdk.org (Postfix) with ESMTP id A16D42C16 for ; Wed, 12 Dec 2018 13:55:52 +0100 (CET) In-Reply-To: <57d7ad23-bebd-f788-5ac9-540804ecf5da@intel.com> Content-Language: en-US Content-ID: <44B154FFEC94584193FAF3EC508DFCF2@eurprd05.prod.outlook.com> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, Dec 03, 2018 at 10:23:03AM +0000, Burakov, Anatoly wrote: > On 02-Dec-18 11:28 PM, Yongseok Koh wrote: > >=20 > > > On Dec 1, 2018, at 9:48 PM, Shahaf Shuler wrot= e: > > >=20 > > > Hi Anatoly, > > >=20 > > > Thursday, November 29, 2018 3:49 PM, Anatoly Burakov: > > > > Subject: [PATCH 0/4] Allow using external memory without malloc > > > >=20 > > > > Currently, the only way to use externally allocated memory is throu= gh > > > > rte_malloc API's. While this is fine for a lot of use cases, it may= not be suitable > > > > for certain other use cases like manual memory management, etc. > > > >=20 > > > > This patchset adds another API to register memory segments with DPD= K (so > > > > that API's like ``rte_mem_virt2memseg`` could be relied on by PMD's= and > > > > such), but not create a malloc heap out of them. > > > >=20 > > > > Aside from the obvious (not adding memory to a heap), the other maj= or > > > > difference between this API and the ``rte_malloc_heap_*`` external = memory > > > > functions is the fact that no DMA mapping is performed automaticall= y. > > > >=20 > > > > This really draws a line in the sand, and there are now two ways of= doing > > > > things - do everything automatically (using the ``rte_malloc_heap_*= `` API's), > > > > or do everything manually (``rte_extmem_*`` and future DMA mapping = API > > > > [1] that would replace ``rte_vfio_dma_map``). This way, the consist= ency of > > > > API is kept, and flexibility is also allowed. > > > >=20 > > >=20 > > > As you know I like the idea. > > > One question though, do you see a use case for application to have ex= ternally allocated memory which needs to be registered to the DPDK subsyste= m however not being used for DMA? > > > My only guess would be so helper libraries which requires the memory = allocation from user (however it doesn't seems like a good API). > > >=20 > > > If no use case, maybe it is better to merge between the two (rte_extm= em_* and rte_dma_map) to have a single call for app to register and DMA map= the memory. The rte_mem_virt2memseg is not something application needs to = understand, it is used internally by PMDs or other libs. > >=20 > > Just FYI. > > My implementation for mlx4/5 doesn't need to have a separate registrati= on for > > DMA by rte_dma_map() as long as it is included in the memseg list. Regi= stration > > is done only if Lkey lookup misses and only mem free event is taken to = release > > it. From my end, the reason why we wanted to have a generic DMA registr= ation was > > because some people doesn't want to use the new API to make the ext mem= included > > in the memseg list but want to simply call the API for DMA registration= . > >=20 > > In a nutshell, mlx4/5 needs users use either rte_extmem_register() or > > rte_dma_map(). However, it is no problem to call both. >=20 > It would be good to create a segment when using rte_dma_map(). > Unfortunately, that's not realistic :) >=20 > Registering memory within DPDK does not necessarily have to be performed = by > the primary process - whichever process that wants to create the table, c= an > do so, and later processes have to attach to the memory area. There's als= o > no way to know if memory segment can be attached to - this is a question > only application can answer. >=20 > In other words, there's no way to combine rte_dma_map() and > rte_extmem_register() into one call and keep multiprocess support. Sorry for late reply. I was away for a while. I understood your point that rte_dma_map() can't create a segment but isn't= the opposite possible? I still have a question about rte_extmem_register/unregister/attach/detach(). Why don't these APIs genera= te memory events? Do you define the memory events are limited to memories for malloc? What if some app wants to know the events even if it is extmem? Wha= t makes difference between two types of extmem (one for malloc heap and the o= ther for just memseg) in generating the events? I've reviewed your patches and all look good :-) But it is still unclear to= me. Thanks, Yongseok > > > > [1] > > > > https://emea01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%= 2Fma > > > > ils.dpdk.org%2Farchives%2Fdev%2F2018- > > > > November%2F118175.html&data=3D02%7C01%7Cshahafs%40mellanox.co > > > > m%7C007a9234feaf42c82f6508d656015eb1%7Ca652971c7d2e4d9ba6a4d1492 > > > > 56f461b%7C0%7C0%7C636790961244424277&sdata=3DYqwcPEEhJM3I7Toe > > > > Ne%2BGcbeo%2FmPbYEnNFckoA7ES2Hg%3D&reserved=3D0 > > > >=20 > > > > Note: at the time of this writing, there's no release notes > > > > template, so no release notes updates in this patchset. > > > > They will be added in later revisions. > > > >=20 > > > > Anatoly Burakov (4): > > > > malloc: separate creating memseg list and malloc heap > > > > malloc: separate destroying memseg list and heap data > > > > mem: allow registering external memory areas > > > > mem: allow usage of non-heap external memory in multiprocess > > > >=20 > > > > .../prog_guide/env_abstraction_layer.rst | 63 +++++++-- > > > > lib/librte_eal/common/eal_common_memory.c | 116 > > > > +++++++++++++++++ > > > > lib/librte_eal/common/include/rte_memory.h | 122 > > > > ++++++++++++++++++ > > > > lib/librte_eal/common/malloc_heap.c | 104 +++++++++++---- > > > > lib/librte_eal/common/malloc_heap.h | 15 ++- > > > > lib/librte_eal/common/rte_malloc.c | 115 +++++++--------= -- > > > > lib/librte_eal/rte_eal_version.map | 4 + > > > > 7 files changed, 434 insertions(+), 105 deletions(-) > > > >=20 > > > > -- > > > > 2.17.1 > >=20 > >=20 >=20 >=20 > --=20 > Thanks, > Anatoly