From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8C8BDC3DA5D for ; Mon, 22 Jul 2024 08:07:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=IO1ypQhPJJGVoWKSnsnPOVclcRMGA5MlPVSimGsxnd0=; b=Z7Wc1EBbAILi0oKauUX2YeVDtC 3EaGjr3sK+TyMB49aP1en4rqVFz2R791+sywmNIHZQ3hFvqN3biCssyLPTz0sKpXdbHJgyT3X8gSS CR3wZpJQQbimJSL7b7j5CEqEEYrU1MYz141NFDpxnMNnFbu0McMScQYx8wD9WTlJGOnkD/U+Jsx9W y2CDVPoMGKhtjCoJnrRkVzJVLNwCq2WBB9IzSysE8CaSpqkVcaxgTn24Nnz1mnZ0GUdsyjbwyCcgv GIjakpXKD35K9U2zE7Kv1GGAIqOUVmXnX3zCi7AlfG1d+B4ZyQWMJbjlXMsToFq5xRUqmmHp6QsX6 QbrZ3hFA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sVo52-00000008oRu-02ik; Mon, 22 Jul 2024 08:07:36 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sVo4e-00000008oK7-0mhq; Mon, 22 Jul 2024 08:07:14 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 11541CE0B18; Mon, 22 Jul 2024 08:07:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C7D33C32782; Mon, 22 Jul 2024 08:06:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721635625; bh=H9F6V5OYzjr4gKPKPFe3Wqpvyc1OTcMklndZSekeRHA=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Q5Dp/726iOiXcRsY4FjpIdOLw46JsxdnylRfDnLOAnzjGf18EfC22uDduEG/56577 mF9VR0M7MePJHH6qducKTMLl4+3wJJgy2MXlYIyme/Z0oGQmEBaV2fnZN8cTUTZZvp e4AxGAr/yPA5Th2/BAQI9KKOEa1B0mXrWv1iZfa496Ragj+yYsX+mzlc5vma9rVTY9 zTTiTTmtETJN6g5cdK6hrsVEV55nEGUvysy6ROPM9ZclNIfLuk9SbEP9IB4BHpP+D6 6+f3Pr37n33T6UhUEbbN4W+dS5FVFyy9KfDtqZZwxaNb+ifX6zJtDsjsuk0MBnmqT3 FAlsWVe5OvzTA== Date: Mon, 22 Jul 2024 11:03:57 +0300 From: Mike Rapoport To: Jonathan Cameron Cc: linux-kernel@vger.kernel.org, Alexander Gordeev , Andreas Larsson , Andrew Morton , Arnd Bergmann , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dan Williams , Dave Hansen , David Hildenbrand , "David S. Miller" , Greg Kroah-Hartman , Heiko Carstens , Huacai Chen , Ingo Molnar , Jiaxun Yang , John Paul Adrian Glaubitz , Michael Ellerman , Palmer Dabbelt , "Rafael J. Wysocki" , Rob Herring , Thomas Bogendoerfer , Thomas Gleixner , Vasily Gorbik , Will Deacon , linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-acpi@vger.kernel.org, linux-cxl@vger.kernel.org, nvdimm@lists.linux.dev, devicetree@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Subject: Re: [PATCH 12/17] mm: introduce numa_memblks Message-ID: References: <20240716111346.3676969-1-rppt@kernel.org> <20240716111346.3676969-13-rppt@kernel.org> <20240719191647.000072f6@Huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240719191647.000072f6@Huawei.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240722_010712_781285_5FBFDD30 X-CRM114-Status: GOOD ( 42.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Jul 19, 2024 at 07:16:47PM +0100, Jonathan Cameron wrote: > On Tue, 16 Jul 2024 14:13:41 +0300 > Mike Rapoport wrote: > > > From: "Mike Rapoport (Microsoft)" > > > > Move code dealing with numa_memblks from arch/x86 to mm/ and add Kconfig > > options to let x86 select it in its Kconfig. > > > > This code will be later reused by arch_numa. > > > > No functional changes. > > > > Signed-off-by: Mike Rapoport (Microsoft) > Hi Mike, > > My only real concern in here is there are a few places where > the lifted code makes changes to memblocks that are x86 only today. > I need to do some more digging to work out if those are safe > in all cases. > > Jonathan > > > > > +/** > > + * numa_cleanup_meminfo - Cleanup a numa_meminfo > > + * @mi: numa_meminfo to clean up > > + * > > + * Sanitize @mi by merging and removing unnecessary memblks. Also check for > > + * conflicts and clear unused memblks. > > + * > > + * RETURNS: > > + * 0 on success, -errno on failure. > > + */ > > +int __init numa_cleanup_meminfo(struct numa_meminfo *mi) > > +{ > > + const u64 low = 0; > > Given always zero, why not just use that value inline? Actually it seems to me that it should be memblock_start_of_DRAM(). The blocks outside system memory are moved to numa_reserved_meminfo, so AFAIU on arm64/riscv such blocks can be below the RAM. > > + const u64 high = PFN_PHYS(max_pfn); > > + int i, j, k; > > + > > + /* first, trim all entries */ > > + for (i = 0; i < mi->nr_blks; i++) { > > + struct numa_memblk *bi = &mi->blk[i]; > > + > > + /* move / save reserved memory ranges */ > > + if (!memblock_overlaps_region(&memblock.memory, > > + bi->start, bi->end - bi->start)) { > > + numa_move_tail_memblk(&numa_reserved_meminfo, i--, mi); > > + continue; > > + } > > + > > + /* make sure all non-reserved blocks are inside the limits */ > > + bi->start = max(bi->start, low); > > + > > + /* preserve info for non-RAM areas above 'max_pfn': */ > > + if (bi->end > high) { > > + numa_add_memblk_to(bi->nid, high, bi->end, > > + &numa_reserved_meminfo); > > + bi->end = high; > > + } > > + > > + /* and there's no empty block */ > > + if (bi->start >= bi->end) > > + numa_remove_memblk_from(i--, mi); > > + } > > + > > + /* merge neighboring / overlapping entries */ > > + for (i = 0; i < mi->nr_blks; i++) { > > + struct numa_memblk *bi = &mi->blk[i]; > > + > > + for (j = i + 1; j < mi->nr_blks; j++) { > > + struct numa_memblk *bj = &mi->blk[j]; > > + u64 start, end; > > + > > + /* > > + * See whether there are overlapping blocks. Whine > > + * about but allow overlaps of the same nid. They > > + * will be merged below. > > + */ > > + if (bi->end > bj->start && bi->start < bj->end) { > > + if (bi->nid != bj->nid) { > > + pr_err("node %d [mem %#010Lx-%#010Lx] overlaps with node %d [mem %#010Lx-%#010Lx]\n", > > + bi->nid, bi->start, bi->end - 1, > > + bj->nid, bj->start, bj->end - 1); > > + return -EINVAL; > > + } > > + pr_warn("Warning: node %d [mem %#010Lx-%#010Lx] overlaps with itself [mem %#010Lx-%#010Lx]\n", > > + bi->nid, bi->start, bi->end - 1, > > + bj->start, bj->end - 1); > > + } > > + > > + /* > > + * Join together blocks on the same node, holes > > + * between which don't overlap with memory on other > > + * nodes. > > + */ > > + if (bi->nid != bj->nid) > > + continue; > > + start = min(bi->start, bj->start); > > + end = max(bi->end, bj->end); > > + for (k = 0; k < mi->nr_blks; k++) { > > + struct numa_memblk *bk = &mi->blk[k]; > > + > > + if (bi->nid == bk->nid) > > + continue; > > + if (start < bk->end && end > bk->start) > > + break; > > + } > > + if (k < mi->nr_blks) > > + continue; > > + pr_info("NUMA: Node %d [mem %#010Lx-%#010Lx] + [mem %#010Lx-%#010Lx] -> [mem %#010Lx-%#010Lx]\n", > > + bi->nid, bi->start, bi->end - 1, bj->start, > > + bj->end - 1, start, end - 1); > > + bi->start = start; > > + bi->end = end; > > + numa_remove_memblk_from(j--, mi); > > + } > > + } > > + > > + /* clear unused ones */ > > + for (i = mi->nr_blks; i < ARRAY_SIZE(mi->blk); i++) { > > + mi->blk[i].start = mi->blk[i].end = 0; > > + mi->blk[i].nid = NUMA_NO_NODE; > > + } > > + > > + return 0; > > +} > > ... > > > > +/* > > + * Mark all currently memblock-reserved physical memory (which covers the > > + * kernel's own memory ranges) as hot-unswappable. > > + */ > > +static void __init numa_clear_kernel_node_hotplug(void) > > This will be a change for non x86 architectures. 'should' be fine > but I'm not 100% sure. This function sets nid to memblock.reserved which does not change anything except the dump in debugfs and then uses the node info in memblock.reserve to clear MEMBLOCK_HOTPLUG from the regions in memblock.memory that contain the reserved memory because they cannot be hot(un)plugged anyway. > > +{ > > + nodemask_t reserved_nodemask = NODE_MASK_NONE; > > + struct memblock_region *mb_region; > > + int i; > > + > > + /* > > + * We have to do some preprocessing of memblock regions, to > > + * make them suitable for reservation. > > + * > > + * At this time, all memory regions reserved by memblock are > > + * used by the kernel, but those regions are not split up > > + * along node boundaries yet, and don't necessarily have their > > + * node ID set yet either. > > + * > > + * So iterate over all memory known to the x86 architecture, > > Comment needs an update at least given not x86 specific any more. Sure, will fix. > > + * and use those ranges to set the nid in memblock.reserved. > > + * This will split up the memblock regions along node > > + * boundaries and will set the node IDs as well. > > + */ > > + for (i = 0; i < numa_meminfo.nr_blks; i++) { > > + struct numa_memblk *mb = numa_meminfo.blk + i; > > + int ret; > > + > > + ret = memblock_set_node(mb->start, mb->end - mb->start, > > + &memblock.reserved, mb->nid); > > + WARN_ON_ONCE(ret); > > + } > > + > > + /* > > + * Now go over all reserved memblock regions, to construct a > > + * node mask of all kernel reserved memory areas. > > + * > > + * [ Note, when booting with mem=nn[kMG] or in a kdump kernel, > > + * numa_meminfo might not include all memblock.reserved > > + * memory ranges, because quirks such as trim_snb_memory() > > + * reserve specific pages for Sandy Bridge graphics. ] > > + */ > > + for_each_reserved_mem_region(mb_region) { > > + int nid = memblock_get_region_node(mb_region); > > + > > + if (nid != MAX_NUMNODES) > > + node_set(nid, reserved_nodemask); > > + } > > + > > + /* > > + * Finally, clear the MEMBLOCK_HOTPLUG flag for all memory > > + * belonging to the reserved node mask. > > + * > > + * Note that this will include memory regions that reside > > + * on nodes that contain kernel memory - entire nodes > > + * become hot-unpluggable: > > + */ > > + for (i = 0; i < numa_meminfo.nr_blks; i++) { > > + struct numa_memblk *mb = numa_meminfo.blk + i; > > + > > + if (!node_isset(mb->nid, reserved_nodemask)) > > + continue; > > + > > + memblock_clear_hotplug(mb->start, mb->end - mb->start); > > + } > > +} > -- Sincerely yours, Mike.