From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Burakov, Anatoly" Subject: Re: [PATCH v2] mem: add --match-allocations Date: Mon, 17 Dec 2018 09:40:00 +0000 Message-ID: <677d302e-6583-babe-edc7-3c462aa09a62@intel.com> References: <20181214171303.21817-1-james.r.harris@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: dariusz.stojaczyk@intel.com, benjamin.walker@intel.com, seth.howell@intel.com To: Jim Harris , dev@dpdk.org Return-path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 1E6B61B728 for ; Mon, 17 Dec 2018 10:40:02 +0100 (CET) In-Reply-To: <20181214171303.21817-1-james.r.harris@intel.com> Content-Language: en-US List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 14-Dec-18 5:13 PM, Jim Harris wrote: > SPDK uses the rte_mem_event_callback_register API to > create RDMA memory regions (MRs) for newly allocated regions > of memory. This is used in both the SPDK NVMe-oF target > and the NVMe-oF host driver. > > DPDK creates internal malloc_elem structures for these > allocated regions. As users malloc and free memory, DPDK > will sometimes merge malloc_elems that originated from > different allocations that were notified through the > registered mem_event callback routine. This results > in subsequent allocations that can span across multiple > RDMA MRs. This requires SPDK to check each DPDK buffer to > see if it crosses an MR boundary, and if so, would have to > add considerable logic and complexity to describe that > buffer before it can be accessed by the RNIC. It is somewhat > analagous to rte_malloc returning a buffer that is not > IOVA-contiguous. > > As a malloc_elem gets split and some of these elements > get freed, it can also result in DPDK sending an > RTE_MEM_EVENT_FREE notification for a subset of the > original RTE_MEM_EVENT_ALLOC notification. This is also > problematic for RDMA memory regions, since unregistering > the memory region is all-or-nothing. It is not possible > to unregister part of a memory region. > > To support these types of applications, this patch adds > a new --match-allocations EAL init flag. When this > flag is specified, malloc elements from different > hugepage allocations will never be merged. Memory will > also only be freed back to the system (with the requisite > memory event callback) exactly as it was originally > allocated. > > Since part of this patch is extending the size of struct > malloc_elem, we also fix up the malloc autotests so they > do not assume its size exactly fits in one cacheline. > > Signed-off-by: Jim Harris > --- Reviewed-by: Anatoly Burakov -- Thanks, Anatoly