From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AF25C4332F for ; Thu, 3 Nov 2022 23:58:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EED4F6B0072; Thu, 3 Nov 2022 19:58:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E76426B0073; Thu, 3 Nov 2022 19:58:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D17008E0001; Thu, 3 Nov 2022 19:58:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BAF2D6B0072 for ; Thu, 3 Nov 2022 19:58:08 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 861891C60DF for ; Thu, 3 Nov 2022 23:58:08 +0000 (UTC) X-FDA: 80093797056.03.5AD54DB Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf10.hostedemail.com (Postfix) with ESMTP id 21D48C0005 for ; Thu, 3 Nov 2022 23:58:06 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id u8-20020a17090a5e4800b002106dcdd4a0so6765788pji.1 for ; Thu, 03 Nov 2022 16:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=caIkGO4tLNLGZYOqGews9YjV3/Xeu1Urs/paPcqE4u0=; b=gncVQ2ibapGM657f3pM0seh2su0K+edIr1NO9NkQqx2VAFxVA1DmKBPFZMT+ecPDi2 2+VVOafLED2V9jm5zMUB0W8cw27F1PA6EDxp30VIRfwdZ0w0AkVtOxhXC1DIDKpWaxES HTIy3rWmGfzUjZ5mkTGcqfub4gRludxFsu3VH7ETlLJkok7mPqVYdSrQKWLkAnU0waUJ xFjGaZXPvpiPsSVInSK8oMWRTmA7gR7JPGjg5ZPtCsLxAP8/tzZ3RPt1iuGGKsCjgtCE 84Vc7X+1ypORPzAdkRpRfglXki7giAtFQT77MPYk7ApTH3CIU9/bQ0MWQAzC3oWiKxCw nvEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=caIkGO4tLNLGZYOqGews9YjV3/Xeu1Urs/paPcqE4u0=; b=JFSXy2l3zk6o2jJkkq7aWpLZ/jJxiOMx5lxau8ifJoTL66382HMqThL8jkJp8K6s4Z yomrhSfLgmt0TJ5BzD3QB0A4a3BdwMPmaNFT38nYu2I6Ljne7vcfZVSKEMCCggqotCSX u+Q9QW4zrc/xVLSTQdctcSWo67XWDPp+dXpFZiDsfskbmBAPpdkXJxmEoom4/mTt5Jo6 xPGESwdoF+86s6jmoffjSMukJfK2wqy9vmtQJaFVuD+uy4Tp2C2dZFVVztlNahqc8Jki la+Dg/jyPLfr0wLe7kuTchqu8SAehB224TVcg9CF7o6SIRhBjwfnc6onXowC3lLnjhQf QuRA== X-Gm-Message-State: ACrzQf0LAKHRqKLaoC1aaCgsfxvughru0OTSKT1xLMtfvHlnvNrYVXP/ vAogZkqtFSkfDmFXYhSfgFCuPQ== X-Google-Smtp-Source: AMsMyM7JaJxQI+vGozir4TiXoBa57nXGubugzECel+ALy0oUWxqBMdPV3oG+Dqhe5kDwUOzOOnPGsQ== X-Received: by 2002:a17:90a:198a:b0:213:8a8:dbb with SMTP id 10-20020a17090a198a00b0021308a80dbbmr33753229pji.73.1667519885755; Thu, 03 Nov 2022 16:58:05 -0700 (PDT) Received: from google.com (33.5.83.34.bc.googleusercontent.com. [34.83.5.33]) by smtp.gmail.com with ESMTPSA id x8-20020aa78f08000000b005625d5ae760sm1399049pfr.11.2022.11.03.16.58.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Nov 2022 16:58:05 -0700 (PDT) Date: Thu, 3 Nov 2022 16:58:01 -0700 From: Zach O'Keefe To: Yang Shi Cc: mhocko@suse.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [v2 PATCH 1/2] mm: khugepaged: allow page allocation fallback to eligible nodes Message-ID: References: <20221103213641.7296-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221103213641.7296-1-shy828301@gmail.com> ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667519887; a=rsa-sha256; cv=none; b=0dornGMwkKGq9lB/MAVSnEH+fwrf306gYJUfaqQo77FOV+9NTvFL13o6eesu6r/8ty8gYu 3xXAH0DPH7WFFJoWGnzAFegE6lwSAuQhai3Ez1uv0qfclwm8MMGTd5KZfOhL74Sg4R51lj nKMIff5A/w//R/EfZSid39v27QrrIbA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gncVQ2ib; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of zokeefe@google.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zokeefe@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667519887; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=caIkGO4tLNLGZYOqGews9YjV3/Xeu1Urs/paPcqE4u0=; b=l7FI/QBukWzqa5eNwOI5fIKNwHuXxmR0EQhlVgy4/3Z0NxjH4YJExfuqJDcwKlBzgCMbsp /Vk34NtiI6zIFZeoGsepPKtIi9wxMhRcgdp/QWX2W6n+bQASYX4BAfM/nFVePk5pKjnR3w xfqtWdm40+ys7KcZkgHflay1JqOV1kE= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 21D48C0005 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gncVQ2ib; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of zokeefe@google.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=zokeefe@google.com X-Stat-Signature: inf53qj7cdi9u64k4c74foroqokjzd4h X-HE-Tag: 1667519886-134543 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Nov 03 14:36, Yang Shi wrote: > Syzbot reported the below splat: > > WARNING: CPU: 1 PID: 3646 at include/linux/gfp.h:221 __alloc_pages_node include/linux/gfp.h:221 [inline] > WARNING: CPU: 1 PID: 3646 at include/linux/gfp.h:221 hpage_collapse_alloc_page mm/khugepaged.c:807 [inline] > WARNING: CPU: 1 PID: 3646 at include/linux/gfp.h:221 alloc_charge_hpage+0x802/0xaa0 mm/khugepaged.c:963 > Modules linked in: > CPU: 1 PID: 3646 Comm: syz-executor210 Not tainted 6.1.0-rc1-syzkaller-00454-ga70385240892 #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/11/2022 > RIP: 0010:__alloc_pages_node include/linux/gfp.h:221 [inline] > RIP: 0010:hpage_collapse_alloc_page mm/khugepaged.c:807 [inline] > RIP: 0010:alloc_charge_hpage+0x802/0xaa0 mm/khugepaged.c:963 > Code: e5 01 4c 89 ee e8 6e f9 ae ff 4d 85 ed 0f 84 28 fc ff ff e8 70 fc ae ff 48 8d 6b ff 4c 8d 63 07 e9 16 fc ff ff e8 5e fc ae ff <0f> 0b e9 96 fa ff ff 41 bc 1a 00 00 00 e9 86 fd ff ff e8 47 fc ae > RSP: 0018:ffffc90003fdf7d8 EFLAGS: 00010293 > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 > RDX: ffff888077f457c0 RSI: ffffffff81cd8f42 RDI: 0000000000000001 > RBP: ffff888079388c0c R08: 0000000000000001 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 > R13: dffffc0000000000 R14: 0000000000000000 R15: 0000000000000000 > FS: 00007f6b48ccf700(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007f6b48a819f0 CR3: 00000000171e7000 CR4: 00000000003506e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > Call Trace: > > collapse_file+0x1ca/0x5780 mm/khugepaged.c:1715 > hpage_collapse_scan_file+0xd6c/0x17a0 mm/khugepaged.c:2156 > madvise_collapse+0x53a/0xb40 mm/khugepaged.c:2611 > madvise_vma_behavior+0xd0a/0x1cc0 mm/madvise.c:1066 > madvise_walk_vmas+0x1c7/0x2b0 mm/madvise.c:1240 > do_madvise.part.0+0x24a/0x340 mm/madvise.c:1419 > do_madvise mm/madvise.c:1432 [inline] > __do_sys_madvise mm/madvise.c:1432 [inline] > __se_sys_madvise mm/madvise.c:1430 [inline] > __x64_sys_madvise+0x113/0x150 mm/madvise.c:1430 > do_syscall_x64 arch/x86/entry/common.c:50 [inline] > do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 > entry_SYSCALL_64_after_hwframe+0x63/0xcd > RIP: 0033:0x7f6b48a4eef9 > Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 b1 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 > RSP: 002b:00007f6b48ccf318 EFLAGS: 00000246 ORIG_RAX: 000000000000001c > RAX: ffffffffffffffda RBX: 00007f6b48af0048 RCX: 00007f6b48a4eef9 > RDX: 0000000000000019 RSI: 0000000000600003 RDI: 0000000020000000 > RBP: 00007f6b48af0040 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6b48aa53a4 > R13: 00007f6b48bffcbf R14: 00007f6b48ccf400 R15: 0000000000022000 > > > The khugepaged code would pick up the node with the most hit as the preferred > node, and also tries to do some balance if several nodes have the same > hit record. Basically it does conceptually: > * If the target_node <= last_target_node, then iterate from > last_target_node + 1 to MAX_NUMNODES (1024 on default config) > * If the max_value == node_load[nid], then target_node = nid > > But there is a corner case, paritucularly for MADV_COLLAPSE, that the > non-existing node may be returned as preferred node. > > Assuming the system has 2 nodes, the target_node is 0 and the > last_target_node is 1, if MADV_COLLAPSE path is hit, the max_value may > be 0, then it may return 2 for target_node, but it is actually not > existing (offline), so the warn is triggered. > > The node balance was introduced by commit 9f1b868a13ac ("mm: thp: > khugepaged: add policy for finding target node") to satisfy > "numactl --interleave=all". But interleaving is a mere hint rather than > something that has hard requirements. > > So use nodemask to record the nodes which have the same hit record, the > hugepage allocation could fallback to those nodes. And remove > __GFP_THISNODE since it does disallow fallback. And if nodemask is > empty (no node is set), it means there is one single node has the most > hist record, the nodemask approach actually behaves like __GFP_THISNODE. > > Reported-by: syzbot+0044b22d177870ee974f@syzkaller.appspotmail.com > Suggested-by: Zach O'Keefe > Suggested-by: Michal Hocko > Signed-off-by: Yang Shi Reviewed-by: Zach O'Keefe > --- > mm/khugepaged.c | 32 ++++++++++++++------------------ > 1 file changed, 14 insertions(+), 18 deletions(-) > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > index ea0d186bc9d4..572ce7dbf4b0 100644 > --- a/mm/khugepaged.c > +++ b/mm/khugepaged.c > @@ -97,8 +97,8 @@ struct collapse_control { > /* Num pages scanned per node */ > u32 node_load[MAX_NUMNODES]; > > - /* Last target selected in hpage_collapse_find_target_node() */ > - int last_target_node; > + /* nodemask for allocation fallback */ > + nodemask_t alloc_nmask; > }; > > /** > @@ -734,7 +734,6 @@ static void khugepaged_alloc_sleep(void) > > struct collapse_control khugepaged_collapse_control = { > .is_khugepaged = true, > - .last_target_node = NUMA_NO_NODE, > }; > > static bool hpage_collapse_scan_abort(int nid, struct collapse_control *cc) > @@ -783,16 +782,11 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) > target_node = nid; > } > > - /* do some balance if several nodes have the same hit record */ > - if (target_node <= cc->last_target_node) > - for (nid = cc->last_target_node + 1; nid < MAX_NUMNODES; > - nid++) > - if (max_value == cc->node_load[nid]) { > - target_node = nid; > - break; > - } > + for_each_online_node(nid) { > + if (max_value == cc->node_load[nid]) > + node_set(nid, cc->alloc_nmask); > + } > > - cc->last_target_node = target_node; > return target_node; > } > #else > @@ -802,9 +796,10 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) > } > #endif > > -static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node) > +static bool hpage_collapse_alloc_page(struct page **hpage, gfp_t gfp, int node, > + nodemask_t *nmask) > { > - *hpage = __alloc_pages_node(node, gfp, HPAGE_PMD_ORDER); > + *hpage = __alloc_pages(gfp, HPAGE_PMD_ORDER, node, nmask); > if (unlikely(!*hpage)) { > count_vm_event(THP_COLLAPSE_ALLOC_FAILED); > return false; > @@ -955,12 +950,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, > static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, > struct collapse_control *cc) > { > - /* Only allocate from the target node */ > gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : > - GFP_TRANSHUGE) | __GFP_THISNODE; > + GFP_TRANSHUGE); > int node = hpage_collapse_find_target_node(cc); > > - if (!hpage_collapse_alloc_page(hpage, gfp, node)) > + if (!hpage_collapse_alloc_page(hpage, gfp, node, &cc->alloc_nmask)) > return SCAN_ALLOC_HUGE_PAGE_FAIL; > if (unlikely(mem_cgroup_charge(page_folio(*hpage), mm, gfp))) > return SCAN_CGROUP_CHARGE_FAIL; > @@ -1144,6 +1138,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, > goto out; > > memset(cc->node_load, 0, sizeof(cc->node_load)); > + nodes_clear(cc->alloc_nmask); > pte = pte_offset_map_lock(mm, pmd, address, &ptl); > for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; > _pte++, _address += PAGE_SIZE) { > @@ -2078,6 +2073,7 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, > present = 0; > swap = 0; > memset(cc->node_load, 0, sizeof(cc->node_load)); > + nodes_clear(cc->alloc_nmask); > rcu_read_lock(); > xas_for_each(&xas, page, start + HPAGE_PMD_NR - 1) { > if (xas_retry(&xas, page)) > @@ -2581,7 +2577,6 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, > if (!cc) > return -ENOMEM; > cc->is_khugepaged = false; > - cc->last_target_node = NUMA_NO_NODE; > > mmgrab(mm); > lru_add_drain_all(); > @@ -2607,6 +2602,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, > } > mmap_assert_locked(mm); > memset(cc->node_load, 0, sizeof(cc->node_load)); > + nodes_clear(cc->alloc_nmask); > if (IS_ENABLED(CONFIG_SHMEM) && vma->vm_file) { > struct file *file = get_file(vma->vm_file); > pgoff_t pgoff = linear_page_index(vma, addr); > -- > 2.26.3 > Thanks for the patch, Yang! Looks good. khugepaged selftest is good too.