From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADD01EB64D7 for ; Mon, 19 Jun 2023 02:34:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4178F8D0002; Sun, 18 Jun 2023 22:34:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 39FE08D0001; Sun, 18 Jun 2023 22:34:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 240D18D0002; Sun, 18 Jun 2023 22:34:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1166A8D0001 for ; Sun, 18 Jun 2023 22:34:29 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D55621603BE for ; Mon, 19 Jun 2023 02:34:28 +0000 (UTC) X-FDA: 80917928616.29.F888524 Received: from out-54.mta1.migadu.com (out-54.mta1.migadu.com [95.215.58.54]) by imf09.hostedemail.com (Postfix) with ESMTP id 1FD0C14000C for ; Mon, 19 Jun 2023 02:34:26 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Mn0IXsSF; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf09.hostedemail.com: domain of yajun.deng@linux.dev designates 95.215.58.54 as permitted sender) smtp.mailfrom=yajun.deng@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687142067; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=oVQbxjpuybR0tDH9ddBEqcix10WINBbUnDacUemPWWA=; b=NwB5f0BhzdFTu6pJh54Vg44e4pwUlTOAvK1wIfROOkOeQbbeO8DJHQOkRVHps1gVpu03SC i+TZyn2jkzThUqIbvTm2kqU7P0UAmje0e80piMD7VNA3zXFqi3srmGbahDvfYAOemHSfOn J3387bzGl7Dssms09yNG/nogbtDxj8M= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Mn0IXsSF; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf09.hostedemail.com: domain of yajun.deng@linux.dev designates 95.215.58.54 as permitted sender) smtp.mailfrom=yajun.deng@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687142067; a=rsa-sha256; cv=none; b=TRy2pELLpID+KAMemcnWG2OrgoNsIpi8pRHl89Pl9K/0XibJcMcnD7aDgDH7qTH/T7maAt qA+6J+i8d1m3LOS7CLrHvu9K1zkXJxB76GVq86TpQj1pEaVXoXj7gnCckaanyIxOIgJ1jM RNLRAFETVkllE5X48eHf5rLf2x9rP88= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1687142065; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=oVQbxjpuybR0tDH9ddBEqcix10WINBbUnDacUemPWWA=; b=Mn0IXsSF0w1YIqlm5F+yQWMVV4RZqBO5LgK1xOC66NLlKvuf4FZdDHF9JB7cnIVUAC1Vk5 I51mMpf4cNHS3ZgZ/PvIOeTojrmPt7TuNwHY/RByoqyXtLxyLemfKimVYOB5L5jSR4qWSE FDk2p6b9errHejR2r9GD1BazcIA9lEY= From: Yajun Deng To: akpm@linux-foundation.org, rppt@kernel.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yajun Deng , kernel test robot Subject: [PATCH v4] mm: pass nid to reserve_bootmem_region() Date: Mon, 19 Jun 2023 10:34:06 +0800 Message-Id: <20230619023406.424298-1-yajun.deng@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1FD0C14000C X-Stat-Signature: k4ehe4rd3az86ttjfxus6caitj5j9ydf X-Rspam-User: X-HE-Tag: 1687142066-993448 X-HE-Meta: U2FsdGVkX18Zsb0aafBPHYWEZTy0iX1DPW/+IPiJGB4TUOSt2VXMMCvGKwN6EUF9PmnvBr2Vy6ovOtCQC2CNrz/eTbHuqmBg7rMq+coyyuZjy2sbNyXv2MlPZyFoBIoLG25fXwjLtn9AQ33QCPIKe/wKagi8Agb+Ixpa3z7t0Ksfo135Z47Kl45UHHxtWGWrOS+C+Yjv7fKavMp01gzrPZ6g4oDmLyCGk+fob284CqKTIJz/lFZaJZsaCCt+zHMGnwv2rAhcAtLdbWGGapXofTgYDQIgR8y4xW0zwGMjJhKxyEQ+g7J496zStc2Gzt73ZBDp66CMgmkuSOS8HwTrDmRPuD0gSo6VjqkiGu/Uafi0lt8S1MCchyyLscKK4+7DTcz3MTYUR5vx40ZogNVPpSULthu0sL9zKsIQWjI+77085nYBIf/N+mAWKxUPAtveW2S9J5qxtiXnhHA0ZjTZ57a0DZo2n35eyF3lG3bvgL4j0OquMxK+/SFsFB+6r7djmAUnWQh6U0JrTPPdaL4ecAF8g97IuAZFp7d31tR20XYj+BIecQs1/5GF2M6P0bxGGr4sshOvdtKBFUVIuhzI8fUy/56PU/lTKwLKztwvZl1tqI9UTsYdqP/9waRdU2wmWJpYEwt4mwxqV6LeTuXfR6icC+OZqTgGOHNK9+IJJzavBWZIe7kmV2N6XD7Me6s5t5y2Ni2Pmh/kRdZujrWGUvQJLpPBBNk9dKjO2XmTUHzWZKSy27fZpzt6i0tFQ4lqTMBYFa2DxTxkP8SFL9efG8oiUxZ9yOJCXBk5n4BLhdu0RWIMK5+5/lKDhWf0SEK2yEMnITlzNfdQGJdAWtAhDaxWserglo6W69rzMzCuW4G7UABDQBxOWJMhfUqQlxKJ7iS5wR8l2NExgW2PXfUJpGhxUB6uVifCcD3K+npbRy4uXnmqd2ojL8wUv7RaSjSHT5p9CFoEH7PWmX+OfnV B/MI6ifj 8R9njOsajV50wa6G5txnWqcCNM+CnQvhR6tpiZSb4RvdwDGv7U5HsQlHnZQ4mfYyTWRzGnnveDTlRW012ZLZFEQsirjUBd/wBpbMWrjxjUgH+YVoIEMOlOdce5U9QpUk8+SmdMoPKv0mRkiEOQEYA41Lcsb9AaBE1bBluKESTeJ1HWOtDExIiC237YzAUchyMwAhG4nIY0gplIXw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: early_pfn_to_nid() is called frequently in init_reserved_page(), it returns the node id of the PFN. These PFN are probably from the same memory region, they have the same node id. It's not necessary to call early_pfn_to_nid() for each PFN. Pass nid to reserve_bootmem_region() and drop the call to early_pfn_to_nid() in init_reserved_page(). Also, set nid on all reserved pages before doing this, as some reserved memory regions may not be set nid. The most beneficial function is memmap_init_reserved_pages() if CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled. The following data was tested on an x86 machine with 190GB of RAM. before: memmap_init_reserved_pages() 67ms after: memmap_init_reserved_pages() 20ms Signed-off-by: Yajun Deng Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202306160145.juJMr3Bi-lkp@intel.com --- V3 -> V4: make the test for early_page_initialised() inside if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT)) V2 -> V3: set nid on all reserved pages before pass nid. V1 -> V2: fix build error when CONFIG_NUMA is not enabled. --- include/linux/mm.h | 3 ++- mm/memblock.c | 31 +++++++++++++++++++++---------- mm/mm_init.c | 30 +++++++++++++++++------------- 3 files changed, 40 insertions(+), 24 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fdd966b11f79..a7a0e692d44d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2960,7 +2960,8 @@ extern unsigned long free_reserved_area(void *start, void *end, extern void adjust_managed_page_count(struct page *page, long count); -extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end); +extern void reserve_bootmem_region(phys_addr_t start, + phys_addr_t end, int nid); /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) diff --git a/mm/memblock.c b/mm/memblock.c index ff0da1858778..f9e61e565a53 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -2091,19 +2091,30 @@ static void __init memmap_init_reserved_pages(void) { struct memblock_region *region; phys_addr_t start, end; - u64 i; + int nid; + + /* + * set nid on all reserved pages and also treat struct + * pages for the NOMAP regions as PageReserved + */ + for_each_mem_region(region) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; + + if (memblock_is_nomap(region)) + reserve_bootmem_region(start, end, nid); + + memblock_set_node(start, end, &memblock.reserved, nid); + } /* initialize struct pages for the reserved regions */ - for_each_reserved_mem_range(i, &start, &end) - reserve_bootmem_region(start, end); + for_each_reserved_mem_region(region) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; - /* and also treat struct pages for the NOMAP regions as PageReserved */ - for_each_mem_region(region) { - if (memblock_is_nomap(region)) { - start = region->base; - end = start + region->size; - reserve_bootmem_region(start, end); - } + reserve_bootmem_region(start, end, nid); } } diff --git a/mm/mm_init.c b/mm/mm_init.c index d393631599a7..a1963c3322af 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -646,10 +646,8 @@ static inline void pgdat_set_deferred_range(pg_data_t *pgdat) } /* Returns true if the struct page for the pfn is initialised */ -static inline bool __meminit early_page_initialised(unsigned long pfn) +static inline bool __meminit early_page_initialised(unsigned long pfn, int nid) { - int nid = early_pfn_to_nid(pfn); - if (node_online(nid) && pfn >= NODE_DATA(nid)->first_deferred_pfn) return false; @@ -695,15 +693,14 @@ defer_init(int nid, unsigned long pfn, unsigned long end_pfn) return false; } -static void __meminit init_reserved_page(unsigned long pfn) +static void __meminit init_reserved_page(unsigned long pfn, int nid) { pg_data_t *pgdat; - int nid, zid; + int zid; - if (early_page_initialised(pfn)) + if (early_page_initialised(pfn, nid)) return; - nid = early_pfn_to_nid(pfn); pgdat = NODE_DATA(nid); for (zid = 0; zid < MAX_NR_ZONES; zid++) { @@ -717,7 +714,7 @@ static void __meminit init_reserved_page(unsigned long pfn) #else static inline void pgdat_set_deferred_range(pg_data_t *pgdat) {} -static inline bool early_page_initialised(unsigned long pfn) +static inline bool early_page_initialised(unsigned long pfn, int nid) { return true; } @@ -727,7 +724,7 @@ static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) return false; } -static inline void init_reserved_page(unsigned long pfn) +static inline void init_reserved_page(unsigned long pfn, int nid) { } #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ @@ -738,7 +735,8 @@ static inline void init_reserved_page(unsigned long pfn) * marks the pages PageReserved. The remaining valid pages are later * sent to the buddy page allocator. */ -void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) +void __meminit reserve_bootmem_region(phys_addr_t start, + phys_addr_t end, int nid) { unsigned long start_pfn = PFN_DOWN(start); unsigned long end_pfn = PFN_UP(end); @@ -747,7 +745,7 @@ void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) if (pfn_valid(start_pfn)) { struct page *page = pfn_to_page(start_pfn); - init_reserved_page(start_pfn); + init_reserved_page(start_pfn, nid); /* Avoid false-positive PageTail() */ INIT_LIST_HEAD(&page->lru); @@ -2579,8 +2577,14 @@ void __init set_dma_reserve(unsigned long new_dma_reserve) void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (!early_page_initialised(pfn)) - return; + + if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT)) { + int nid = early_pfn_to_nid(pfn); + + if (!early_page_initialised(pfn, nid)) + return; + } + if (!kmsan_memblock_free_pages(page, order)) { /* KMSAN will take care of these pages. */ return; -- 2.25.1