From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Burakov, Anatoly" Subject: Re: [PATCH v2] mem: add --match-allocations Date: Mon, 17 Dec 2018 09:45:09 +0000 Message-ID: <44e46a5a-7edc-5146-9fa3-d3c3a62abc4e@intel.com> References: <20181214171303.21817-1-james.r.harris@intel.com> <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: dariusz.stojaczyk@intel.com, benjamin.walker@intel.com, seth.howell@intel.com To: Jim Harris , dev@dpdk.org Return-path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 2CA7B374 for ; Mon, 17 Dec 2018 10:45:12 +0100 (CET) In-Reply-To: <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> Content-Language: en-US List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 17-Dec-18 9:40 AM, Burakov, Anatoly wrote: > On 14-Dec-18 5:13 PM, Jim Harris wrote: >> SPDK uses the rte_mem_event_callback_register API to >> create RDMA memory regions (MRs) for newly allocated regions >> of memory. This is used in both the SPDK NVMe-oF target >> and the NVMe-oF host driver. >> >> DPDK creates internal malloc_elem structures for these >> allocated regions. As users malloc and free memory, DPDK >> will sometimes merge malloc_elems that originated from >> different allocations that were notified through the >> registered mem_event callback routine. This results >> in subsequent allocations that can span across multiple >> RDMA MRs. This requires SPDK to check each DPDK buffer to >> see if it crosses an MR boundary, and if so, would have to >> add considerable logic and complexity to describe that >> buffer before it can be accessed by the RNIC. It is somewhat >> analagous to rte_malloc returning a buffer that is not >> IOVA-contiguous. >> >> As a malloc_elem gets split and some of these elements >> get freed, it can also result in DPDK sending an >> RTE_MEM_EVENT_FREE notification for a subset of the >> original RTE_MEM_EVENT_ALLOC notification. This is also >> problematic for RDMA memory regions, since unregistering >> the memory region is all-or-nothing. It is not possible >> to unregister part of a memory region. >> >> To support these types of applications, this patch adds >> a new --match-allocations EAL init flag. When this >> flag is specified, malloc elements from different >> hugepage allocations will never be merged. Memory will >> also only be freed back to the system (with the requisite >> memory event callback) exactly as it was originally >> allocated. >> >> Since part of this patch is extending the size of struct >> malloc_elem, we also fix up the malloc autotests so they >> do not assume its size exactly fits in one cacheline. >> >> Signed-off-by: Jim Harris >> --- > > Reviewed-by: Anatoly Burakov > To those reviewing, some relevant discussion: https://mails.dpdk.org/archives/dev/2018-December/120225.html -- Thanks, Anatoly