From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E01C433FE for ; Tue, 9 Nov 2021 07:03:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4890A61159 for ; Tue, 9 Nov 2021 07:03:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4890A61159 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 9B0F36B0071; Tue, 9 Nov 2021 02:03:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 95EFD6B00DD; Tue, 9 Nov 2021 02:03:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 827776B00DF; Tue, 9 Nov 2021 02:03:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id 6F87E6B0071 for ; Tue, 9 Nov 2021 02:03:06 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2ABE11838A437 for ; Tue, 9 Nov 2021 07:03:06 +0000 (UTC) X-FDA: 78788499972.03.44034CB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id DAFA960020BB for ; Tue, 9 Nov 2021 07:02:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1636441384; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nPrvGm053cP3UgyADu8h6v/paiJaHwAsCpY2M8NmnWc=; b=YDI/26QaPYqn9PPb4s8xZKfdE0KQg9LqnKbHkKyWwaQcW9CDeAy1W6qA/PIKdqXvoEyMdW sGiMn4/hFwGaf0pdT0719PWqae5b2zMyoC9btR7h37cRLe924U7rw7Rpj/C9FMheo2IykS 6P5dAZ5VWNBr0lWUo92osmHYlKEMVFU= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-384-QWA_yMUyP1qUDqlnSpLscw-1; Tue, 09 Nov 2021 02:03:03 -0500 X-MC-Unique: QWA_yMUyP1qUDqlnSpLscw-1 Received: by mail-wr1-f69.google.com with SMTP id c4-20020adfed84000000b00185ca4eba36so4572701wro.21 for ; Mon, 08 Nov 2021 23:03:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent :content-language:to:cc:references:from:organization:subject :in-reply-to:content-transfer-encoding; bh=nPrvGm053cP3UgyADu8h6v/paiJaHwAsCpY2M8NmnWc=; b=vplYsdLCJIb8wZc12N3EkQT9gM1H093NYyOyQie9vV9iKtQs8OUYpYE4gdDj0eGFty X+htoiFSHjkBJ7Xw6jYQOVUULl9xeh/XfFXynTPrK2RglHxmei53gofII7Cb9p6q0nyZ Q/DFNLhKCiBQffioQq16ptTZjIprRUQKOILabG7YvRfRF/LAwhZMEY1mjidM/k5EcmL1 TQLCpdjrywviCeFVrEHFmnrRuB6Yvl8Af1lTW8XTd4hMzd4S65tzzWz5cX3NYHgXDa8o Cc70hqGQNfoVSCBGBxAAWoEKmEDsBk4B4cYK6MBBE90iSE76TdUMNIesHfG1MHlYSf1G jnAg== X-Gm-Message-State: AOAM532PcukrHiYb9rW0qZ6/FoECl4OBtTaeRh++TZDxX1A929eDzjhu lIVy8DUkdq6vXsCxvCD3SIIuT2kK/j7NUdczcGyXLMcmZ42l9KyikJGzzKnqXwTJ2MS2LfVlGGu +qsuyH3EBufQ= X-Received: by 2002:a05:600c:19c8:: with SMTP id u8mr4568925wmq.155.1636441382012; Mon, 08 Nov 2021 23:03:02 -0800 (PST) X-Google-Smtp-Source: ABdhPJyxcqyin3pn5o2FEe5VqM8C7p+RoQONeS3JPmpGgKWji9b1WR2P59esCbKX+Pqdm+YLH93Fyg== X-Received: by 2002:a05:600c:19c8:: with SMTP id u8mr4568899wmq.155.1636441381808; Mon, 08 Nov 2021 23:03:01 -0800 (PST) Received: from [192.168.3.132] (p4ff23c2b.dip0.t-ipconnect.de. [79.242.60.43]) by smtp.gmail.com with ESMTPSA id l5sm1558193wms.16.2021.11.08.23.03.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 08 Nov 2021 23:03:01 -0800 (PST) Message-ID: Date: Tue, 9 Nov 2021 08:03:00 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.2.0 To: Eric Dumazet , Alexey Makhalov , linux-mm@kvack.org Cc: Andrew Morton , Michal Hocko , Oscar Salvador , Dennis Zhou , Tejun Heo , Christoph Lameter , linux-kernel@vger.kernel.org, stable@vger.kernel.org References: <908909e0-4815-b580-7ff5-d824d36a141c@redhat.com> <20211108202325.20304-1-amakhalov@vmware.com> <2e191db3-286f-90c6-bf96-3f89891e9926@gmail.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v3] mm: fix panic in __alloc_pages In-Reply-To: <2e191db3-286f-90c6-bf96-3f89891e9926@gmail.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: DAFA960020BB X-Stat-Signature: fddghgiu6xxz4ykxn4wcrzja7g1tyoiq Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="YDI/26Qa"; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf10.hostedemail.com: domain of david@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=david@redhat.com X-HE-Tag: 1636441371-442090 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 09.11.21 03:08, Eric Dumazet wrote: > > > On 11/8/21 12:23 PM, Alexey Makhalov wrote: >> There is a kernel panic caused by pcpu_alloc_pages() passing >> offlined and uninitialized node to alloc_pages_node() leading >> to panic by NULL dereferencing uninitialized NODE_DATA(nid). >> >> CPU2 has been hot-added >> BUG: unable to handle page fault for address: 0000000000001608 >> #PF: supervisor read access in kernel mode >> #PF: error_code(0x0000) - not-present page >> PGD 0 P4D 0 >> Oops: 0000 [#1] SMP PTI >> CPU: 0 PID: 1 Comm: systemd Tainted: G E 5.15.0-rc7+ #11 >> Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW >> >> RIP: 0010:__alloc_pages+0x127/0x290 >> Code: 4c 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 44 89 e0 48 8b 55 b8 c1 e8 0c 83 e0 01 88 45 d0 4c 89 c8 48 85 d2 0f 85 1a 01 00 00 <45> 3b 41 08 0f 82 10 01 00 00 48 89 45 c0 48 8b 00 44 89 e2 81 e2 >> RSP: 0018:ffffc900006f3bc8 EFLAGS: 00010246 >> RAX: 0000000000001600 RBX: 0000000000000000 RCX: 0000000000000000 >> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000cc2 >> RBP: ffffc900006f3c18 R08: 0000000000000001 R09: 0000000000001600 >> R10: ffffc900006f3a40 R11: ffff88813c9fffe8 R12: 0000000000000cc2 >> R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000cc2 >> FS: 00007f27ead70500(0000) GS:ffff88807ce00000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 0000000000001608 CR3: 000000000582c003 CR4: 00000000001706b0 >> Call Trace: >> pcpu_alloc_pages.constprop.0+0xe4/0x1c0 >> pcpu_populate_chunk+0x33/0xb0 >> pcpu_alloc+0x4d3/0x6f0 >> __alloc_percpu_gfp+0xd/0x10 >> alloc_mem_cgroup_per_node_info+0x54/0xb0 >> mem_cgroup_alloc+0xed/0x2f0 >> mem_cgroup_css_alloc+0x33/0x2f0 >> css_create+0x3a/0x1f0 >> cgroup_apply_control_enable+0x12b/0x150 >> cgroup_mkdir+0xdd/0x110 >> kernfs_iop_mkdir+0x4f/0x80 >> vfs_mkdir+0x178/0x230 >> do_mkdirat+0xfd/0x120 >> __x64_sys_mkdir+0x47/0x70 >> ? syscall_exit_to_user_mode+0x21/0x50 >> do_syscall_64+0x43/0x90 >> entry_SYSCALL_64_after_hwframe+0x44/0xae >> >> Panic can be easily reproduced by disabling udev rule for >> automatic onlining hot added CPU followed by CPU with >> memoryless node (NUMA node with CPU only) hot add. >> >> Hot adding CPU and memoryless node does not bring the node >> to online state. Memoryless node will be onlined only during >> the onlining its CPU. >> >> Node can be in one of the following states: >> 1. not present.(nid == NUMA_NO_NODE) >> 2. present, but offline (nid > NUMA_NO_NODE, node_online(nid) == 0, >> NODE_DATA(nid) == NULL) >> 3. present and online (nid > NUMA_NO_NODE, node_online(nid) > 0, >> NODE_DATA(nid) != NULL) >> >> Percpu code is doing allocations for all possible CPUs. The >> issue happens when it serves hot added but not yet onlined >> CPU when its node is in 2nd state. This node is not ready >> to use, fallback to numa_mem_id(). >> >> Signed-off-by: Alexey Makhalov >> Reviewed-by: David Hildenbrand >> Cc: Andrew Morton >> Cc: David Hildenbrand >> Cc: Michal Hocko >> Cc: Oscar Salvador >> Cc: Dennis Zhou >> Cc: Tejun Heo >> Cc: Christoph Lameter >> Cc: linux-mm@kvack.org >> Cc: linux-kernel@vger.kernel.org >> Cc: stable@vger.kernel.org >> --- >> mm/percpu-vm.c | 8 ++++++-- >> 1 file changed, 6 insertions(+), 2 deletions(-) >> >> diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c >> index 2054c9213..f58d73c92 100644 >> --- a/mm/percpu-vm.c >> +++ b/mm/percpu-vm.c >> @@ -84,15 +84,19 @@ static int pcpu_alloc_pages(struct pcpu_chunk *chunk, >> gfp_t gfp) >> { >> unsigned int cpu, tcpu; >> - int i; >> + int i, nid; >> >> gfp |= __GFP_HIGHMEM; >> >> for_each_possible_cpu(cpu) { >> + nid = cpu_to_node(cpu); >> + if (nid == NUMA_NO_NODE || !node_online(nid)) >> + nid = numa_mem_id(); > > Maybe we should fail this fallback if (gfp & __GFP_THISNODE) ? ... and what to do then? Fail the allocation? We could do that, but ... > > Or maybe there is no support for this constraint in per-cpu allocator anyway. > ... looking at mm/percpu.c, I don't think there are any users (IOW not supported?). > I am a bit worried that we do not really know if pages are > allocated on the right node or not. Even without __GFP_THISNODE it's sub-optimal. But if there is no memory on that node, there is barely anything we can do than falling back. -- Thanks, David / dhildenb