From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BDDDC5320E for ; Tue, 20 Aug 2024 09:02:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A2676B007B; Tue, 20 Aug 2024 05:02:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 652146B0082; Tue, 20 Aug 2024 05:02:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 519D16B0083; Tue, 20 Aug 2024 05:02:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 34DA86B007B for ; Tue, 20 Aug 2024 05:02:42 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CA4161218B7 for ; Tue, 20 Aug 2024 09:02:41 +0000 (UTC) X-FDA: 82472033322.29.94A7C2B Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) by imf27.hostedemail.com (Postfix) with ESMTP id C944440018 for ; Tue, 20 Aug 2024 09:02:39 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="VWV/5kun"; spf=pass (imf27.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724144544; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HQvEj+i2KB/Df0ml1vf1DAW64uAm8Q1UNpNL60RFnfs=; b=x6J8R/RwadyYmnbujy/NKxzECvmdk+z4sjoWShNpNckaaMuWGVRD2TzSPHukv+R8Dy8zGB ZFRuCIP4e38BaSWEdcCTXVFGWdjC5vZzSP+jQpc/0OrJjO3CXF6XlW2ZxCI54lXvFLvrTT UcMk+4O85VJiB7BhAl0OKZBBQ1A16NM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="VWV/5kun"; spf=pass (imf27.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724144544; a=rsa-sha256; cv=none; b=xwoygFpYOmpdr7shnXjPsEoS8HP3eaGFacGY2r8XgmqTr/LNkgTgDe2y1t8qfhgphwCLXy 1Rc3AwTQYBPydRvaUbtySMkhSicjaSj+DjZQ/5uH/n6URFk50Svh+Gqlv1aA7/3PTjvM9w QjhYivX8j580z5m3YR/p1wNCwmIfeIg= Received: by mail-lj1-f169.google.com with SMTP id 38308e7fff4ca-2f3f07ac2dcso2859221fa.2 for ; Tue, 20 Aug 2024 02:02:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724144558; x=1724749358; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=HQvEj+i2KB/Df0ml1vf1DAW64uAm8Q1UNpNL60RFnfs=; b=VWV/5kunT5/RChi0UH4jTvqZ3jryKjHUHbAS08n/l8kjUINYSt7hn+fVGGSPiTRyCK qUCjrWrOA28603x+pucUS00xs/B1vUYTaOrApbAyB4A6QeqSZN6dfu0nGhsUlCkAz3I/ RTYNb8Btu+NQBDJzbzHq56f/PcCklK9jWeG5ElRmiTlUtpcBXcmSe74wooC3ioWo1HqG 8O7SECAJ0A6cW0I7OmITfclmMRpO7l2juZ5obxGV+TP+W+7m6dttKe0LOGMn6o+IX2Or Z8Sua1THZmuQNa6AJ7cIbc/DJihLHcvZotvMoI2bCucDirvGKQuZjxNEqeYg0cCVpicw wGMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724144558; x=1724749358; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HQvEj+i2KB/Df0ml1vf1DAW64uAm8Q1UNpNL60RFnfs=; b=c/i3AohZ6QtfVfVsI+R2499XZlQXPt5ykgSZecc3lD9xtLU7Sqc8PzH2GHbZoRYU/O SWQD1rhYomRq9JHSTO1nbHFAJSR4yPrQpDAoqcD8ptYQMmEGjUC1rh0U/XA7Exw5rS1g tJ11js5ZyW7MYVbkR7H8Q6ktEOl09owkzL2Uf1IwLZEtelUwgz4gFFWiMoTNqguKL9aK ppZsCO4HV4V9vBe9cDYex7qUC7haMNnkXyidsLg+yH9tSh0fWSVo1vDKJ5iSEsGF9RRJ elP1CzyrWzBnwz5UjwL9jlZ0eANFHXLgab9wTRYZ+MVzNLTWYsXQt2qTKaDp+6SY519C JFVw== X-Forwarded-Encrypted: i=1; AJvYcCVZU8Z//xfjKLLL+5giSrfIj4pS/m0QH2vstYJRgt/rfKqOn1gDOUgAsW6e5/2QvIZZJtmnhfsLHErXfZheofiW5z8= X-Gm-Message-State: AOJu0YwwKdixtGHjhdhrhm0SRG5FDlBDy/Icl5fpwsLc/YZ5qC5mIlvf VbsHfXljiGESVC4h0Dg/oGOB1iUzZHXYYAjbHjPPDirHXNQpEjrIe7eEMI6SzmS8fif4R0tmUHe pyUgyn4xgRo7b2ZI/ucUzTpXHYN8= X-Google-Smtp-Source: AGHT+IEsiXwFlzQo0g+R6eE/DS/Ct1wvhPZjsaLOCdc2PyEJ7YhxP5z2ilu5qAyZI20if7MzIy4b3o6XTdVQxmD1Re4= X-Received: by 2002:a05:651c:504:b0:2f1:56a6:6057 with SMTP id 38308e7fff4ca-2f3be57849emr85755641fa.7.1724144557428; Tue, 20 Aug 2024 02:02:37 -0700 (PDT) MIME-Version: 1.0 References: <00000000000060cf79061fd24ca8@google.com> In-Reply-To: From: Kairui Song Date: Tue, 20 Aug 2024 17:02:20 +0800 Message-ID: Subject: Re: [syzbot] [mm?] WARNING in zswap_swapoff To: Yosry Ahmed , Barry Song <21cnbao@gmail.com> Cc: syzbot , akpm@linux-foundation.org, chengming.zhou@linux.dev, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, syzkaller-bugs@googlegroups.com, Chris Li , Ying , Ryan Roberts Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: dpwtkg8ow7nnrkxy53i575iseqzbdgjm X-Rspamd-Queue-Id: C944440018 X-Rspamd-Server: rspam11 X-HE-Tag: 1724144559-807443 X-HE-Meta: U2FsdGVkX19kKai3666RGz7xfQ3KQUPwSV5Tf8eBYcqmi4eohP/9P/e0QxDRWfrtTdFo7w3GCRXBvFG+tOgxOKh8Reok0JnqRXo6E2qnwnoTQRuzZNfviaSMFFQUUGyruIEkEru8kxgDrcp9VUpt7Yqb/grg5pbrrbZsxwCk8sY5g5LKJOCWH+T9KMr0cdqBecQRWZhzI2G1PBs8ZOzM7eenmMHwOeDiJJuyAQZgjjZLwQe1LE6eNPSnPrvUzh5yHDLg4hfRC0UOcH1ZwAc6iaaA5p2izFkUJZN9Fipacbrpyw8bbyNLuK2rxqMK5But6NKuPYPDw3Vzizfug8V1PnJWc1nz6KhcGFX2PFYu4j6AeJIshb15w1g7iVr6YXSAZbfpVmPyyuoFCFrLz0Crv5mf8W503OWsO88MG7BVKTT1j46Fi3B3kZJMu9vim/LuE5d/2ARKBUxw1EZzo63ByKt8xs6SLgfSN+xbJzP/lgiwk2rrF+v6Sdhku0pfbDj4fd7ohLFBjCvd4G1xcp32KsAnflGEUPl7HFnHnFb9XopgKLuUzfakqAQEb9JVq77UrPL810FF5MX0gHZKDXuAtAjSbPWp20uDhTzYQwHwcrunoNa5q28MoaNc7cCN+tTssJh1ev+Eh7EKggBtYSvv3yOHTx228Vv9VqT4Po6KXPL0RkfqNzVF6yozGuAu6TEzAKeDSrcyXs5Enzf7m4uq9hWBdu3PVY7J/flxoq6M38fsvNGZp5oiWb5vOoyElsaGU5nnBV44WV2UzAakcyM9YiYcHm9DZdHQciReeL97zaZeuE/Iun54nQFtlr6UNmLIFKbPk2fYtN6HpFr4dHhzrL2cWk4XA1dm+oWZ0QLlTQnMQtjb/LWPOJMnzU/79qgsYWEKOnh4ONznp8InYLaakWrTTUlaghB50A5iUOoBtzUHcbKEnhAU1mYbNT8H6bG8C1t940mZLrB1hxceJMJ NXP00DSa WgPYAzc/KNY9II3xU4FiCaptwwh+Oa0Rw4ffuODby2e4Hs4h3ribdxeddPfbBtTftVMK+lKyUeOlyFx9vIFl4SgRk49clK418DrMi9zNuEQYR9NlZAQfoIUvY6+T+hkcwS6PJkV0HvtEiqoaUzKSDbgnGZ/j61/2cdDEYDBLuWfHnpJtetfqrnNR6ViIbiodOaEsVGK4p1Jtz+YYTQIvAOxpHhZTlcHi0G019goVPIiGfvUYBLacLPGFWUoPWQBv/ThL7/A8yPvqhtZO1F0C1R/nR+tAolqrObetAa4tBuv28Cl3TpbEZMiJ3UnVOj5S2bWnajXXNSN7kHeW+SgOjt5pbX26xMbImmaxLLyjCSPFVAlIoUGz4DiTlde9RUs8qn1Q6pvk4zbx9TBM295xQ6vaZmHYPxqJp9drSJQWvQURzEsOmvPvquTrpFd2mvTtq2992h6H0hKXMGxZMKAPOmFSycFLsmwkI8u3J0O69wbZdVIMfbQdryhuulr+s8XQu6OIfmCPh9Ut/yu+Z0ULfN1GeXlubGuwM024rmvPqLD3d849HreeyS2Oq7PwTIDgH5XbeY7uBXO5mP8FxJYwKBf41aCav4NGzKrm64fC+PRAd67/kJvI6Y2DMofJOwIpwjMnFikfCJay4R5G3duccUe2nscujfqWr8fSWhC8YKFZWxjM36GWDu8KQiRX9arr05uz3n+GkDlx4BkrNba4cDNM46lSTzgXUkR1ByJRkOp2yKfEJ5/lFS16yaQsg3lsaunkwlFmO1TC0yPTSO4U75fDWHcfNM5uPlq1ZZXbPIvPzEQXSX7L2yH6rJ1iY0p6hZ9EKHPKSIp5oSEKLED0D+e8RwrigrOL4LRxk+l6DyEl5Shxrg3kCHTWgKdMMUsK80w0CSS/QuJzq7yYjsoiYGzLvgslwG/2hmYH2W/uI4jsPbNwbp7lzEOPYMQ1/Wu3BLk1Vlra9xNRTHc63jcSUgviP1YPV UDpfghte NrkyCPKiWt0/qYdQc0jZ3TRnKSbKpDR8a2KWLDMQaPc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Aug 20, 2024 at 4:47=E2=80=AFPM Kairui Song wrot= e: > > On Tue, Aug 20, 2024 at 4:13=E2=80=AFAM Yosry Ahmed wrote: > > On Fri, Aug 16, 2024 at 12:52=E2=80=AFPM syzbot > > wrote: > > > > > > Hello, > > > > > > syzbot found the following issue on: > > > > > > HEAD commit: 367b5c3d53e5 Add linux-next specific files for 202408= 16 > > I can't find this commit, seems this commit is not in linux-next any more= ? > > > > git tree: linux-next > > > console output: https://syzkaller.appspot.com/x/log.txt?x=3D124891059= 80000 > > > kernel config: https://syzkaller.appspot.com/x/.config?x=3D61ba6f3b2= 2ee5467 > > > dashboard link: https://syzkaller.appspot.com/bug?extid=3Dce6029250d7= fd4d0476d > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for= Debian) 2.40 > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > Downloadable assets: > > > disk image: https://storage.googleapis.com/syzbot-assets/0b1b4e3cad3c= /disk-367b5c3d.raw.xz > > > vmlinux: https://storage.googleapis.com/syzbot-assets/5bb090f7813c/vm= linux-367b5c3d.xz > > > kernel image: https://storage.googleapis.com/syzbot-assets/6674cb0709= b1/bzImage-367b5c3d.xz > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the = commit: > > > Reported-by: syzbot+ce6029250d7fd4d0476d@syzkaller.appspotmail.com > > > > > > ------------[ cut here ]------------ > > > WARNING: CPU: 0 PID: 11298 at mm/zswap.c:1700 zswap_swapoff+0x11b/0x2= b0 mm/zswap.c:1700 > > > Modules linked in: > > > CPU: 0 UID: 0 PID: 11298 Comm: swapoff Not tainted 6.11.0-rc3-next-20= 240816-syzkaller #0 > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BI= OS Google 06/27/2024 > > > RIP: 0010:zswap_swapoff+0x11b/0x2b0 mm/zswap.c:1700 > > > Code: 74 05 e8 78 73 07 00 4b 83 7c 35 00 00 75 15 e8 1b bd 9e ff 48 = ff c5 49 83 c6 50 83 7c 24 0c 17 76 9b eb 24 e8 06 bd 9e ff 90 <0f> 0b 90 e= b e5 48 8b 0c 24 80 e1 07 80 c1 03 38 c1 7c 90 48 8b 3c > > > RSP: 0018:ffffc9000302fa38 EFLAGS: 00010293 > > > RAX: ffffffff81f4d66a RBX: dffffc0000000000 RCX: ffff88802c19bc00 > > > RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff888015986248 > > > RBP: 0000000000000000 R08: ffffffff81f4d620 R09: 1ffffffff1d476ac > > > R10: dffffc0000000000 R11: fffffbfff1d476ad R12: dffffc0000000000 > > > R13: ffff888015986200 R14: 0000000000000048 R15: 0000000000000002 > > > FS: 00007f9e628a5380(0000) GS:ffff8880b9000000(0000) knlGS:000000000= 0000000 > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > CR2: 0000001b30f15ff8 CR3: 000000006c5f0000 CR4: 00000000003506f0 > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > Call Trace: > > > > > > __do_sys_swapoff mm/swapfile.c:2837 [inline] > > > __se_sys_swapoff+0x4653/0x4cf0 mm/swapfile.c:2706 > > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > RIP: 0033:0x7f9e629feb37 > > > Code: 73 01 c3 48 8b 0d f1 52 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 = 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 a8 00 00 00 0f 05 <48> 3d 01 f= 0 ff ff 73 01 c3 48 8b 0d c1 52 0d 00 f7 d8 64 89 01 48 > > > RSP: 002b:00007fff17734f68 EFLAGS: 00000246 ORIG_RAX: 00000000000000a= 8 > > > RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9e629feb37 > > > RDX: 00007f9e62a9e7e8 RSI: 00007f9e62b9beed RDI: 0000563090942a20 > > > RBP: 0000563090942a20 R08: 0000000000000000 R09: 77872e07ed164f94 > > > R10: 000000000000001f R11: 0000000000000246 R12: 00007fff17735188 > > > R13: 00005630909422a0 R14: 0000563073724169 R15: 00007f9e62bdda80 > > > > > > > I am hoping syzbot would find a reproducer and bisect this for us. > > Meanwhile, from a high-level it looks to me like we are missing a > > zswap_invalidate() call in some paths. > > > > If I have to guess, I would say it's related to the latest mTHP swap > > changes, but I am not following closely. Perhaps one of the following > > things happened: > > > > (1) We are not calling zswap_invalidate() in some invalidation paths. > > It used to not be called for the cluster freeing path, so maybe we end > > up with some order-0 swap entries in a cluster? or maybe there is an > > entirely new invalidation path that does not go through > > free_swap_slot() for order-0 entries? > > > > (2) Some higher order swap entries (i.e. a cluster) end up in zswap > > somehow. zswap_store() has a warning to cover that though. Maybe > > somehow some swap entries are allocated as a cluster, but then pages > > are swapped out one-by-one as order-0 (which can go to zswap), but > > then we still free the swap entries as a cluster? > > Hi Yosry, thanks for the report. > > There are many mTHP related optimizations recently, for this problem I > can reproduce this locally. Can confirm the problem is gone for me > after reverting: > > "mm: attempt to batch free swap entries for zap_pte_range()" > > Hi Barry, > > If a set of continuous slots are having the same value, they are > considered a mTHP and freed, bypassing the slot cache, and causing > zswap leak. > This didn't happen in put_swap_folio because that function is > expecting an actual mTHP folio behind the slots but > free_swap_and_cache_nr is simply walking the slots. > > For the testing, I actually have to disable mTHP, because linux-next > will panic with mTHP due to lack of following fixes: > https://lore.kernel.org/linux-mm/a4b1b34f-0d8c-490d-ab00-eaedbf3fe780@gma= il.com/ > https://lore.kernel.org/linux-mm/403b7f3c-6e5b-4030-ab1c-3198f36e3f73@gma= il.com/ > > > > > I am not closely following the latest changes so I am not sure. CCing > > folks who have done work in that area recently. > > > > I am starting to think maybe it would be more reliable to just call > > zswap_invalidate() for all freed swap entries anyway. Would that be > > too expensive? We used to do that before the zswap_invalidate() call > > was moved by commit 0827a1fb143f ("mm/zswap: invalidate zswap entry > > when swap entry free"), and that was before we started using the > > xarray (so it was arguably worse than it would be now). > > > > That might be a good idea, I suggest moving zswap_invalidate to > swap_range_free and call it for every freed slot. > > Below patch can be squash into or put before "mm: attempt to batch > free swap entries for zap_pte_range()". Hmm, on second thought, the commit message in the attachment commit might be not suitable, current zswap_invalidate is also designed to only work for order 0 ZSWAP, so things are not clean even after this. And for performance, it will cause unnecessary heavier contention for the mTHP page on ZSWAP Xarray. It does fix the leak though, please ignore this fix, let's try find a better fix.