From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01A2BC433F5 for ; Tue, 9 Nov 2021 02:08:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7B0A661152 for ; Tue, 9 Nov 2021 02:08:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7B0A661152 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C7A226B0071; Mon, 8 Nov 2021 21:08:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C291F6B0072; Mon, 8 Nov 2021 21:08:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B18176B0073; Mon, 8 Nov 2021 21:08:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A37106B0071 for ; Mon, 8 Nov 2021 21:08:55 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5E6DD82499A8 for ; Tue, 9 Nov 2021 02:08:55 +0000 (UTC) X-FDA: 78787758630.22.7C4F8E2 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf21.hostedemail.com (Postfix) with ESMTP id 98C84D036A5C for ; Tue, 9 Nov 2021 02:08:49 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id r5so18108830pls.1 for ; Mon, 08 Nov 2021 18:08:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=nuIPdnvIF8rGskbyb/raN9C5Xy0Y5NftDXzCqbiXGYE=; b=AH9QWPEsMWY5Y2X1LmK1rIKY8YOFSC4hn1T1xUOe3HTCJem0PwGq8aROL5NbN3eRHx eoKcbYWdCC219ty2wZ3ZQDfwEvfdWTzSPocW5ulhayeRrdcvU5YRvXg70b2lC791TkHR kVJ+d1/Ez5E4tL+BPajaqMSKqncNnsbRduIKSITNs3ZW/iMJoL5w0EqoP559HZARIIAQ NDPvsYSlD7ZL8qr8dAgCwqae0AUa3n7nownxF3lbkCKg2KjAMLWjbDopN/kfVtYPU6Jq 6mIXYPoK37s1JjqAh5C0y7F8U2rdVRiH2p0GMgdaRTvoZ0Xb7VG40Qw7khaMgvNt5DJQ UpKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=nuIPdnvIF8rGskbyb/raN9C5Xy0Y5NftDXzCqbiXGYE=; b=T1JY+A5Ogz9GVQNpwgu2R30UjV1/cwMbka3i3qVowGhEOjbAH1GedQZyXCM8WXkbXN 6dz+Zd3AWOiNms1F6VT8MzzUeQCl9gaj5brUR0JNkaghbhae3yIpNQvi2xGZyoP/yH32 f+oKudh8fP4hcFGeqJUSEUOvi28tekji5v1gqm466nRHkjXMHtlsT8PEVIS6KZca4Vzj TdhqOKyeGgzwO4j3eDnifGBBoqA9g+DnCkW0Qb+E+DZsiNzZ1DXTbO53Le3xi5244cBa BYazxyAC7F2/eRfqeJlI/+by2SScMKtwWweEqmc3ZNcEhZyfqBHUyUREyF7MruJo3XRW q/bw== X-Gm-Message-State: AOAM5337HyN0CLIHIKK+ExOTDZ53+Y6rNxxtyOIoX8rjVNk44To5DE6o c355RuzRn5Xo7K2zY/4jeHU= X-Google-Smtp-Source: ABdhPJzFVMYq7FKSbBlhbIK/GaWghuRe2076O7z19n4If5/0Ax5qCk8bLqKRsiWDPtHSi91iC2kWtw== X-Received: by 2002:a17:902:e5ce:b0:142:780:78db with SMTP id u14-20020a170902e5ce00b00142078078dbmr3807610plf.12.1636423733999; Mon, 08 Nov 2021 18:08:53 -0800 (PST) Received: from [192.168.86.235] (c-73-241-150-58.hsd1.ca.comcast.net. [73.241.150.58]) by smtp.gmail.com with ESMTPSA id cv1sm552622pjb.48.2021.11.08.18.08.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 08 Nov 2021 18:08:53 -0800 (PST) Subject: Re: [PATCH v3] mm: fix panic in __alloc_pages To: Alexey Makhalov , linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , Michal Hocko , Oscar Salvador , Dennis Zhou , Tejun Heo , Christoph Lameter , linux-kernel@vger.kernel.org, stable@vger.kernel.org References: <908909e0-4815-b580-7ff5-d824d36a141c@redhat.com> <20211108202325.20304-1-amakhalov@vmware.com> From: Eric Dumazet Message-ID: <2e191db3-286f-90c6-bf96-3f89891e9926@gmail.com> Date: Mon, 8 Nov 2021 18:08:52 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <20211108202325.20304-1-amakhalov@vmware.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=AH9QWPEs; spf=pass (imf21.hostedemail.com: domain of eric.dumazet@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=eric.dumazet@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 98C84D036A5C X-Stat-Signature: 4h9txqqnx4cxym8ydnj37fsk1dqmqt97 X-HE-Tag: 1636423729-580033 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 11/8/21 12:23 PM, Alexey Makhalov wrote: > There is a kernel panic caused by pcpu_alloc_pages() passing > offlined and uninitialized node to alloc_pages_node() leading > to panic by NULL dereferencing uninitialized NODE_DATA(nid). > > CPU2 has been hot-added > BUG: unable to handle page fault for address: 0000000000001608 > #PF: supervisor read access in kernel mode > #PF: error_code(0x0000) - not-present page > PGD 0 P4D 0 > Oops: 0000 [#1] SMP PTI > CPU: 0 PID: 1 Comm: systemd Tainted: G E 5.15.0-rc7+ #11 > Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW > > RIP: 0010:__alloc_pages+0x127/0x290 > Code: 4c 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 44 89 e0 48 8b 55 b8 c1 e8 0c 83 e0 01 88 45 d0 4c 89 c8 48 85 d2 0f 85 1a 01 00 00 <45> 3b 41 08 0f 82 10 01 00 00 48 89 45 c0 48 8b 00 44 89 e2 81 e2 > RSP: 0018:ffffc900006f3bc8 EFLAGS: 00010246 > RAX: 0000000000001600 RBX: 0000000000000000 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000cc2 > RBP: ffffc900006f3c18 R08: 0000000000000001 R09: 0000000000001600 > R10: ffffc900006f3a40 R11: ffff88813c9fffe8 R12: 0000000000000cc2 > R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000cc2 > FS: 00007f27ead70500(0000) GS:ffff88807ce00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000000000001608 CR3: 000000000582c003 CR4: 00000000001706b0 > Call Trace: > pcpu_alloc_pages.constprop.0+0xe4/0x1c0 > pcpu_populate_chunk+0x33/0xb0 > pcpu_alloc+0x4d3/0x6f0 > __alloc_percpu_gfp+0xd/0x10 > alloc_mem_cgroup_per_node_info+0x54/0xb0 > mem_cgroup_alloc+0xed/0x2f0 > mem_cgroup_css_alloc+0x33/0x2f0 > css_create+0x3a/0x1f0 > cgroup_apply_control_enable+0x12b/0x150 > cgroup_mkdir+0xdd/0x110 > kernfs_iop_mkdir+0x4f/0x80 > vfs_mkdir+0x178/0x230 > do_mkdirat+0xfd/0x120 > __x64_sys_mkdir+0x47/0x70 > ? syscall_exit_to_user_mode+0x21/0x50 > do_syscall_64+0x43/0x90 > entry_SYSCALL_64_after_hwframe+0x44/0xae > > Panic can be easily reproduced by disabling udev rule for > automatic onlining hot added CPU followed by CPU with > memoryless node (NUMA node with CPU only) hot add. > > Hot adding CPU and memoryless node does not bring the node > to online state. Memoryless node will be onlined only during > the onlining its CPU. > > Node can be in one of the following states: > 1. not present.(nid == NUMA_NO_NODE) > 2. present, but offline (nid > NUMA_NO_NODE, node_online(nid) == 0, > NODE_DATA(nid) == NULL) > 3. present and online (nid > NUMA_NO_NODE, node_online(nid) > 0, > NODE_DATA(nid) != NULL) > > Percpu code is doing allocations for all possible CPUs. The > issue happens when it serves hot added but not yet onlined > CPU when its node is in 2nd state. This node is not ready > to use, fallback to numa_mem_id(). > > Signed-off-by: Alexey Makhalov > Reviewed-by: David Hildenbrand > Cc: Andrew Morton > Cc: David Hildenbrand > Cc: Michal Hocko > Cc: Oscar Salvador > Cc: Dennis Zhou > Cc: Tejun Heo > Cc: Christoph Lameter > Cc: linux-mm@kvack.org > Cc: linux-kernel@vger.kernel.org > Cc: stable@vger.kernel.org > --- > mm/percpu-vm.c | 8 ++++++-- > 1 file changed, 6 insertions(+), 2 deletions(-) > > diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c > index 2054c9213..f58d73c92 100644 > --- a/mm/percpu-vm.c > +++ b/mm/percpu-vm.c > @@ -84,15 +84,19 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk, > gfp_t gfp) > { > unsigned int cpu, tcpu; > - int i; > + int i, nid; > > gfp |= __GFP_HIGHMEM; > > for_each_possible_cpu(cpu) { > + nid = cpu_to_node(cpu); > + if (nid == NUMA_NO_NODE || !node_online(nid)) > + nid = numa_mem_id(); Maybe we should fail this fallback if (gfp & __GFP_THISNODE) ? Or maybe there is no support for this constraint in per-cpu allocator anyway. I am a bit worried that we do not really know if pages are allocated on the right node or not. Some workloads could really be hurt if all per-cpu pages were put on a single NUMA node. > + > for (i = page_start; i < page_end; i++) { > struct page **pagep = &pages[pcpu_page_idx(cpu, i)]; > > - *pagep = alloc_pages_node(cpu_to_node(cpu), gfp, 0); > + *pagep = alloc_pages_node(nid, gfp, 0); > if (!*pagep) > goto err; > } >