From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FDFBC05027 for ; Tue, 14 Feb 2023 09:43:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 405C16B0073; Tue, 14 Feb 2023 04:43:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B5166B0074; Tue, 14 Feb 2023 04:43:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27C9B6B0075; Tue, 14 Feb 2023 04:43:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 15C1C6B0073 for ; Tue, 14 Feb 2023 04:43:21 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C9CED140DB3 for ; Tue, 14 Feb 2023 09:43:20 +0000 (UTC) X-FDA: 80465409360.26.2A7D31D Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf07.hostedemail.com (Postfix) with ESMTP id F24464000B for ; Tue, 14 Feb 2023 09:43:17 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=lWLSMcar; spf=pass (imf07.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676367798; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zqV6CFANw7uhmSo1W1cLSOHFWLHB/I1+U2S+G+Ah/E0=; b=6f1V8QTS66UrvjMD6WPZa9nShNUH5r/2oz9fV0iFdh6rLCJ9s09Bndhty4BFMjOAwIyv+g yPaRvttgXGpxtl2PC8i9DdcnSeLY69BpRuPV9HIuZK8uhMnSWze8nUMX41tw4Wum6s38AV kqHKzLq32SyjGHyCJyJPFFs/h/REy1s= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=lWLSMcar; spf=pass (imf07.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676367798; a=rsa-sha256; cv=none; b=o4tHCo0HKNsUcuObLFFEpqe4QjNwvwV8OJvbqP2nM2ST1P+f3lGwIya8fS8HtNUhjPdNkO dmGDUB52YFdMgAxEgWsi3HnuaMlNfzXMTIsPNdXjMQl474xgFh1Wl3vcQL4YQiAMU3pg/Q Vvad+IPMjivjodAqrG5IxkL/9eTiZX8= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 70C53B81BF4; Tue, 14 Feb 2023 09:43:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5763EC433D2; Tue, 14 Feb 2023 09:43:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676367795; bh=JgiePJOXuSHNcYNssB2GnhcYmi8KQG3kSVOb03KPjqg=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=lWLSMcarZUjF2q75pRtslttXe5d9W5s3zjYdOqjCaqj+eqmveF8cULy4HEV49RCCG NIjeJG3dK9R3fA2phdX8XYFjLoR0Z0AeE36wLTLOifLxBDgDQE5zQU23GuLHi4WLUg ShXAinWiRKmrxlbBY2ppiw2bqayPQly7Njf3L9HjAQkI277kpiuFrEvuRYOyj4vF2M G040hImP5Q+f0TSzFYzLIw/vWWCSgjxlqtRQOqXzTg1uuWbYmznA/r59UXP7HCy9z3 6ua4GpJyZFJ5XCqBtTrPdfVPFsxU5b/ETNi0fO0ej9BW3qYnYP1ILoo7teg8zGYxLE mdn0T4MdlSmLQ== Date: Tue, 14 Feb 2023 11:43:08 +0200 From: Mike Rapoport To: David Hildenbrand Cc: Vlastimil Babka , Qi Zheng , akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Teng Hu , Matthew Wilcox , Mel Gorman , Oscar Salvador , Muchun Song Subject: Re: [PATCH] mm: page_alloc: don't allocate page from memoryless nodes Message-ID: References: <20230212110305.93670-1-zhengqi.arch@bytedance.com> <2484666e-e78e-549d-e075-b2c39d460d71@suse.cz> <85af4ada-96c8-1f99-90fa-9b6d63d0016e@bytedance.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: F24464000B X-Stat-Signature: ppi67yo8t4w9mj5xfsozpzqfjfpks63x X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1676367797-983054 X-HE-Meta: U2FsdGVkX194MJuK7+ZnK7zf0rnqMYpiAO0rXK19Grg3lAodhe+yx+cHiNqWQwjTtyBAvbtiGQQ7OvZxUE29O3+34UZ13xl7XxVFxd/5lEjOdEib7QmXt/ZYzrmUeCUKC0+70zklGHw8VrT/4EiLoyo9l/nOxA+r7Eqq9Q4XA3oy/FbHQPxd1ojMgDWsCQg6tL5RkPxc9F+phf0hK3RV+IvcyCr2VqraGZx4RFDYO5A3Le3RY9bxqcXEG6IOZbvglW5U4llh0e3TRvzdT+eWYHHyh5jCCl19KhlPv9NkGAUusnssXqEsuB6sXx8jU4eaFz2nGDC94maAIn0HdjpXG3HEzBkUuUHLWrmqUkjUmgVFCehgY4KM7ITjcIF2uC7FWQS/G1fgbgYhgs8ENUBMApZiK8r8uspSgdfIQBYgcewmow0/Kyvt5glvWi4UcbNLe3yIxN3wHRHzSmnPuvVXw9L/heX0Oh+pjuXGugDID2XFpjBMyZFoSxy+0J4Ty4WBLon/VzcFKXN2Mpayz5QT+6YbtSO/oTCwG9GQ8Yseq57uaxEP9EAQT/U6v+tI8SA/kAKxdswntK4w+pVeKCrCu78gJK2nNaB9Lv4h3A6Fk/w4fcRXD8Y07WCl9lRhoR7Z7/AwagC4jbL2hZXyZoLff/6/sGs79aQ/Td96+RLCOlqE0h2y0C5mvjKDOcQhXMtKBKrsxe1Nu6Fl4NXY90PYZsl3H7J6yS/DjUka1FqV7yxIrT0bEWtW3AjFHwQ1ZN5m8zCzLQupm8XJStOLX4bbPbj8gIqGuk9fUPUo+CarWSInMzZ2PwyQT/wdmwaqGh6meuHsI8BWBMwWPRrU4LQGC4zh1CGISAfS9iDH+hsDrYhZIScU9+BzVNR8xzjcW/AMTjF6Noz27Gkb64KbWOx/GLyGGIWJC/kiIbEtUIxwUAOZ+XcyvVPF5IyGTkLJTtrJOYrOUguhJM9Yub/zasm yBSok90P pEzeIzpObvfdixA3boLtXcmN8WzSf8bUhH0Y8zn0kajo7cX35HCAQLtUrch6hKgQBqho581ln5qvKqB5GGUoCDqhsJSb0Fq+HQv8Xcjbg5fQfVqcCXye3niXySrX2zUJxKyBUSmF0d5G1SFsiBKGIpV5rgxiMxxI1HcHybwtUpoAXadXoHlhu+PLmRLnEh8Gx84L3hlluD0GvSmtAnEr1Ivet8/S7LKGNRuBdQ+u9PE1NW+EwtlfAHiyCYRPyIEigD37JIRJtyMydHMhrgOQUPfvuqVGtwVtd5aEdvZk2s+WPlBDQUrWth47x9Q0oj/1VLpkqJJph9gAqLjTTqU8R7KcVNSbOgSKYY+CTpNAsGUtP7/+vuZB1+m1sIyG3/fSALHHe5yTvztkrMwpcmBMgUjDZQCRxRe+iP2CKtU0HC/2EQhaG0igzGnIQY/hm0eSxv/9r X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 14, 2023 at 10:17:03AM +0100, David Hildenbrand wrote: > On 14.02.23 09:42, Vlastimil Babka wrote: > > On 2/13/23 12:00, Qi Zheng wrote: > > > > > > > > > On 2023/2/13 16:47, Vlastimil Babka wrote: > > > > On 2/12/23 12:03, Qi Zheng wrote: > > > > > In x86, numa_register_memblks() is only interested in > > > > > those nodes which have enough memory, so it skips over > > > > > all nodes with memory below NODE_MIN_SIZE (treated as > > > > > a memoryless node). Later on, we will initialize these > > > > > memoryless nodes (allocate pgdat in free_area_init() > > > > > and build zonelist etc), and will online these nodes > > > > > in init_cpu_to_node() and init_gi_nodes(). > > > > > > > > > > After boot, these memoryless nodes are in N_ONLINE > > > > > state but not in N_MEMORY state. But we can still allocate > > > > > pages from these memoryless nodes. > > > > > > > > > > In SLUB, we only process nodes in the N_MEMORY state, > > > > > such as allocating their struct kmem_cache_node. So if > > > > > we allocate a page from the memoryless node above to > > > > > SLUB, the struct kmem_cache_node of the node corresponding > > > > > to this page is NULL, which will cause panic. > > > > > > > > > > For example, if we use qemu to start a two numa node kernel, > > > > > one of the nodes has 2M memory (less than NODE_MIN_SIZE), > > > > > and the other node has 2G, then we will encounter the > > > > > following panic: > > > > > > > > > > [ 0.149844] BUG: kernel NULL pointer dereference, address: 0000000000000000 > > > > > [ 0.150783] #PF: supervisor write access in kernel mode > > > > > [ 0.151488] #PF: error_code(0x0002) - not-present page > > > > > <...> > > > > > [ 0.156056] RIP: 0010:_raw_spin_lock_irqsave+0x22/0x40 > > > > > <...> > > > > > [ 0.169781] Call Trace: > > > > > [ 0.170159] > > > > > [ 0.170448] deactivate_slab+0x187/0x3c0 > > > > > [ 0.171031] ? bootstrap+0x1b/0x10e > > > > > [ 0.171559] ? preempt_count_sub+0x9/0xa0 > > > > > [ 0.172145] ? kmem_cache_alloc+0x12c/0x440 > > > > > [ 0.172735] ? bootstrap+0x1b/0x10e > > > > > [ 0.173236] bootstrap+0x6b/0x10e > > > > > [ 0.173720] kmem_cache_init+0x10a/0x188 > > > > > [ 0.174240] start_kernel+0x415/0x6ac > > > > > [ 0.174738] secondary_startup_64_no_verify+0xe0/0xeb > > > > > [ 0.175417] > > > > > [ 0.175713] Modules linked in: > > > > > [ 0.176117] CR2: 0000000000000000 > > > > > > > > > > In addition, we can also encountered this panic in the actual > > > > > production environment. We set up a 2c2g container with two > > > > > numa nodes, and then reserved 128M for kdump, and then we > > > > > can encountered the above panic in the kdump kernel. > > > > > > > > > > To fix it, we can filter memoryless nodes when allocating > > > > > pages. > > > > > > > > > > Signed-off-by: Qi Zheng > > > > > Reported-by: Teng Hu > > > > > > > > Well AFAIK the key mechanism to only allocate from "good" nodes is the > > > > zonelist, we shouldn't need to start putting extra checks like this. So it > > > > seems to me that the code building the zonelists should take the > > > > NODE_MIN_SIZE constraint in mind. > > > > > > Indeed. How about the following patch: > > > > +Cc also David, forgot earlier. > > > > Looks good to me, at least. > > > > > @@ -6382,8 +6378,11 @@ int find_next_best_node(int node, nodemask_t > > > *used_node_mask) > > > int min_val = INT_MAX; > > > int best_node = NUMA_NO_NODE; > > > > > > - /* Use the local node if we haven't already */ > > > - if (!node_isset(node, *used_node_mask)) { > > > + /* > > > + * Use the local node if we haven't already. But for memoryless > > > local > > > + * node, we should skip it and fallback to other nodes. > > > + */ > > > + if (!node_isset(node, *used_node_mask) && node_state(node, > > > N_MEMORY)) { > > > node_set(node, *used_node_mask); > > > return node; > > > } > > > > > > For memoryless node, we skip it and fallback to other nodes when > > > build its zonelists. > > > > > > Say we have node0 and node1, and node0 is memoryless, then: > > > > > > [ 0.102400] Fallback order for Node 0: 1 > > > [ 0.102931] Fallback order for Node 1: 1 > > > > > > In this way, we will not allocate pages from memoryless node0. > > > > > In offline_pages(), we'll first build_all_zonelists() to then > node_states_clear_node()->node_clear_state(node, N_MEMORY); > > So at least on the offlining path, we wouldn't detect it properly yet I > assume, and build a zonelist that contains a now-memory-less node? Another question is what happens if a new memory is plugged into a node that had < NODE_MIN_SIZE of memory and after hotplug it stops being "memoryless". > -- > Thanks, > > David / dhildenb > -- Sincerely yours, Mike.