From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Burakov, Anatoly" Subject: Re: [PATCH v2] mem: accelerate dpdk program startup by reuse page from page cache Date: Fri, 9 Nov 2018 12:20:59 +0000 Message-ID: <6a7fcafc-5ac7-417a-fea6-2fd33b8b6c90@intel.com> References: <20181109075830.27265-1-jianmingfan@126.com> <20181109092338.30097-1-jianmingfan@126.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: Jianming Fan To: jianmingfan , dev@dpdk.org Return-path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 6B6C74C9F for ; Fri, 9 Nov 2018 13:21:02 +0100 (CET) In-Reply-To: <20181109092338.30097-1-jianmingfan@126.com> Content-Language: en-US List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 09-Nov-18 9:23 AM, jianmingfan wrote: > --- fix coding style of the previous patch > > During procless startup, dpdk invokes clear_hugedir() to unlink all > hugepage files under /dev/hugepages. Then in map_all_hugepages(), > it invokes mmap to allocate and zero all the huge pages as configured > in /sys/kernel/mm/hugepages/xxx/nr_hugepages. > > This cause startup process extreamly slow with large size of huge page > configured. > > In our use case, we usually configure as large as 200GB hugepages in our > router. It takes more than 50s each time dpdk process startup to clear > the pages. > > To address this issue, user can turn on --reuse-map switch. With it, > dpdk will check the validity of the exiting page cache under > /dev/hugespages. If valid, the cache will be reused not deleted, > so that the os doesn't need to zero the pages again. > > However, as there are a lot of users ,e.g. rte_kni_alloc, rely on the > os zeor page behavior. To keep things work, I add memset during > malloc_heap_alloc(). This makes sense due to the following reason. > 1) user often configure hugepage size too large to be used by the program. > In our router, 200GB is configured, but less than 2GB is actually used. > 2) dpdk users don't call heap allocation in performance-critical path. > They alloc memory during process bootup. > > Signed-off-by: Jianming Fan > --- I believe this issue is better solved by actually fixing all of the memory that DPDK leaves behind. We already have rte_eal_cleanup() call which will deallocate any EAL-allocated memory that have been reserved, and an exited application should free any memory it was using so that memory subsystem could free it back to the system, thereby not needing any cleaning of hugepages at startup. If your application does not e.g. free its mempools on exit, it should :) Chances are, the problem will go away. The only circumstance where this may not work is if you preallocated your memory using -m/--socket-mem flag. -- Thanks, Anatoly