From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F87EC61D92 for ; Tue, 21 Nov 2023 00:59:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D08CE6B0390; Mon, 20 Nov 2023 19:59:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC3E06B0391; Mon, 20 Nov 2023 19:59:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA87E6B0392; Mon, 20 Nov 2023 19:59:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AB16C6B0390 for ; Mon, 20 Nov 2023 19:59:18 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6F69F80A4B for ; Tue, 21 Nov 2023 00:59:18 +0000 (UTC) X-FDA: 81480152796.05.8C50F26 Received: from out-176.mta0.migadu.com (out-176.mta0.migadu.com [91.218.175.176]) by imf18.hostedemail.com (Postfix) with ESMTP id 1528C1C0008 for ; Tue, 21 Nov 2023 00:59:14 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Zu2Mevij; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf18.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.176 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700528355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TMiJoyjEZW4sOevoestRnpHPcZV3tDd1QOoL9UFCKUI=; b=DzEI7Tofy+8SFCQsZEmMdzJSwO2OuVwfVfKnLwiwawry3f74+wAM1DBN/8Q9zTaUK04ZL1 ZhEBZuhbndTfzHb28C9m+YkuBlle4vMJpuGBPgyqoIaf8KxkzibqeBi03OfjKuaSOw2sPx qRx/e87g11ccjD8OuKDGXzis1h/NSjs= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Zu2Mevij; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf18.hostedemail.com: domain of chengming.zhou@linux.dev designates 91.218.175.176 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700528355; a=rsa-sha256; cv=none; b=4c3fW08NpuoW+GFWshiY0jafViUc0jsMpnS250juLCatV0wjNd13zrBkwjIh5fmSLqBMnQ 2CHIQhThs/XUrWNR4zauc4AuGU0q0xcpABEc9H2rLkTMv4hLaqUGjGP1NcMCMOgdIsHVJO OmB0AsUkYOBwKEsEyagVe2vagJnvjHM= Message-ID: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1700528352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TMiJoyjEZW4sOevoestRnpHPcZV3tDd1QOoL9UFCKUI=; b=Zu2MevijGntArris/N302EBD2ByzB1hgor9dSXLc+/pJ14O5QzRU4V+hMDnKNdZaBn55VS 0BtPbmrjIdyUup1fJjb85fH4MNz2VsKvBw2dxrDAaRLEzlpujyx+9wpqtaA4sJv+khruA5 G20Kkg2H72Xzh6CuF3HDWaIPnv6kHQY= Date: Tue, 21 Nov 2023 08:58:40 +0800 MIME-Version: 1.0 Subject: Re: [PATCH v5 6/9] slub: Delay freezing of partial slabs To: Mark Brown Cc: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou References: <20231102032330.1036151-1-chengming.zhou@linux.dev> <20231102032330.1036151-7-chengming.zhou@linux.dev> <4f3bc1bd-ea87-465d-b58a-0ed57b15187b@sirena.org.uk> Content-Language: en-US X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou In-Reply-To: <4f3bc1bd-ea87-465d-b58a-0ed57b15187b@sirena.org.uk> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 1528C1C0008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3woro1ksxy8t7qgtyft1p1yxsodatsbm X-HE-Tag: 1700528354-742223 X-HE-Meta: U2FsdGVkX19YwX0/sUvk3Mfx4vlf0aKDJ0ACzLGT4xs+EFxnPsu06ixhlCG16Iq5Zdr96UWv6e7pAzkq6IjqLCDgFkqzNZ/yvOc+9TMa+wBa6UP0FFKofcj40p4U+8h6/a3rL1UYSTUunezBGIJTLJAu8y3DnGEKkljjh8Y5Rt4d0R5C0Pfcq6TLSWi4Jz8l2MIKGsgo8czbGLILVvyycb51boIN7WV84Au6bVaW0Zeux/hwTSh4GP/m6WwFQD23epjg78cqh9Od1bdRIzQ/IXNR7hFNGafDOEB6PLaOTb7x8ZcGQLuCJFga6FsSnAb285GYIMecC7NmszEdP2pkqlPv+JMc7+w2ql4IjbG0Bd0WVRySm2U+7BH+rsgE1bxfAzKvc41jtBjbgnuinDNSjWqNU/Uz7K72zQkpeBaGFSC0MmReIG6acuA2S60Z9SSkmw8NL3mdNF59U//Pp1FokEhi+1wfO8X16PArYG/bTZZI5ZFO2QffnP+m/w9/ZiGWSBsrOEihVrkT/e6nW55p/kBQdurDGZ3zGHNMSYcke7VLhjlGl+yJdm15f2EhlzNb2O6lPUAF8dDWlGwXoyPFvIZcf3ye3zJ9BxkO7QktNbFBPD5KnswObsmMvXGPhCQs1pMrmCRzGi48OAT+he74K85Y5H5YCvCscZQh356MCZqK0eGRBlvHqAe9Nt4PHQDfwACcQyXnfWOlyHpaUQlo+pbbhHdkeE8r4yfIj43kBvVXuMX28wCChygZJLC3LbSM3VYBOslEBqjXn+j5ho9CgSLbXJquc+E8OhZu8ztqJuOYSrpqqd0S+Rr1SJBbgcqRvqQYe/SW/x/rkxV+s6ttSUZzSm9/A7FxTm4tfbRNNnpkTOp/Qxkzqsg4f0jYKSBYEQBm9TZqryvme3F/K3QqRvrqozO2kQf+g6Tz/Z3cqboZmQN3v/SQDvq9YgGsD6kt40JYOfNny5vpik91f28 bn6nF3+T M1D/S+5olOy1ubRlEToqt9Xxqy1ecn08d8R7xa00Eyia8/ythJzADzxHs10pp6XTHmiajzi9sTS9zrKf2zPNqcy0C+DxdqMjeIeSGn14nusnFcQgxa2LFPxpgtZ/Fri9NOCtwsxmRaMdDjk9LJYEcwq5tF+bGbbjwjpsswlEFs0HcOEP/AyBCbxsPQaKjdh43GIKKyhSvGUT10agvMRyOH9DeJ2ogNx4h/BnwZjgV2vEJ40np4MBkaAfEs4JwbxwyWvv6wHdK+YxeRFBF83kM8YJrUayBxlbKDvmmKtJX9T8hl92ONy4hjx743cTvG7PpKjN1/32J7qwiQRn7woQi6RRdbY2sFcRO4CI4ckcFOCXdJqjZnlo5hfSNMaBWkEFZd7oQrBx4kSYI5XU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2023/11/21 02:49, Mark Brown wrote: > On Thu, Nov 02, 2023 at 03:23:27AM +0000, chengming.zhou@linux.dev wrote: >> From: Chengming Zhou >> >> Now we will freeze slabs when moving them out of node partial list to >> cpu partial list, this method needs two cmpxchg_double operations: >> >> 1. freeze slab (acquire_slab()) under the node list_lock >> 2. get_freelist() when pick used in ___slab_alloc() > > Recently -next has been failing to boot on a Raspberry Pi 3 with an arm > multi_v7_defconfig and a NFS rootfs, a bisect appears to point to this > patch (in -next as c8d312e039030edab25836a326bcaeb2a3d4db14) as having > introduced the issue. I've included the full bisect log below. > > When we see problems we see RCU stalls while logging in, for example: > > debian-testing-armhf login: root (automatic login) > Linux debian-testing-armhf 6.7.0-rc1-00006-gc8d312e03903 #1 SMP @1699864348 armv7l > The programs included with the Debian GNU/Linux system are free software; > the exact distribution terms for each program are described in the > individual files in /usr/share/doc/*/copyright. > Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent > permitted by applicable law. > [ 46.453323] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: > [ 46.459361] rcu: 3-...0: (1 GPs behind) idle=def4/1/0x40000000 softirq=1304/1304 fqs=951 > [ 46.467669] rcu: (detected by 0, t=2103 jiffies, g=1161, q=499 ncpus=4) > [ 46.474472] Sending NMI from CPU 0 to CPUs 3: IIUC, here should print the backtrace of CPU 3, right? It looks like CPU 3 is the cause, but we couldn't see what it's doing from the log. Thanks! > [ 56.478894] rcu: rcu_sched kthread timer wakeup didn't happen for 1002 jiffies! g1161 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 > [ 56.490195] rcu: Possible timer handling issue on cpu=0 timer-softirq=1650 > [ 56.497259] rcu: rcu_sched kthread starved for 1005 jiffies! g1161 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0 > [ 56.507589] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. > [ 56.516681] rcu: RCU grace-period kthread stack dump: > [ 56.521803] task:rcu_sched state:I stack:0 pid:13 tgid:13 ppid:2 flags:0x00000000 > [ 56.531267] __schedule from schedule+0x20/0xe8 > [ 56.535883] schedule from schedule_timeout+0xa0/0x158 > [ 56.541111] schedule_timeout from rcu_gp_fqs_loop+0x104/0x594 > [ 56.547048] rcu_gp_fqs_loop from rcu_gp_kthread+0x14c/0x1c0 > [ 56.552801] rcu_gp_kthread from kthread+0xe0/0xfc > [ 56.557674] kthread from ret_from_fork+0x14/0x28 > [ 56.562457] Exception stack(0xf084dfb0 to 0xf084dff8) > [ 56.567584] dfa0: 00000000 00000000 00000000 00000000 > [ 56.575886] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > [ 56.584191] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 > [ 56.590907] rcu: Stack dump where RCU GP kthread last ran: > [ 56.596474] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.7.0-rc1-00006-gc8d312e03903 #1 > [ 56.604515] Hardware name: BCM2835 > [ 56.607965] PC is at default_idle_call+0x1c/0xb0 > [ 56.612654] LR is at ct_kernel_enter.constprop.0+0x48/0x11c > [ 56.618311] pc : [] lr : [] psr: 60010013 > [ 56.624672] sp : c1b01f70 ip : c1d5af7c fp : 00000000 > [ 56.629974] r10: c19cda60 r9 : 00000000 r8 : 00000000 > [ 56.635277] r7 : c1b04d50 r6 : c1b04d18 r5 : c1d5b684 r4 : c1b09740 > [ 56.641902] r3 : 00000000 r2 : 00000000 r1 : 00000001 r0 : 002a3114 > [ 56.648528] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none > [ 56.655774] Control: 10c5383d Table: 0237406a DAC: 00000051 > [ 56.661605] default_idle_call from do_idle+0x208/0x290 > [ 56.666920] do_idle from cpu_startup_entry+0x28/0x2c > [ 56.672059] cpu_startup_entry from rest_init+0xac/0xb0 > [ 56.677371] rest_init from arch_post_acpi_subsys_init+0x0/0x8 > Login ti > > A full log for that run can be seen at: > > https://validation.linaro.org/scheduler/job/4017095 > > Boots to initramfs with the same kernel image seem fine. Other systems, > including other 32 bit arm ones, don't seem to be having similar issues > with this userspace. I've not investigated beyond running the bisect, > the log for which is below: > > git bisect start > # good: [64e6d94bfb47ed0732ad06aedf8ec6af5dd2ab84] Merge branch 'for-linux-next-fixes' of git://anongit.freedesktop.org/drm/drm-misc > git bisect good 64e6d94bfb47ed0732ad06aedf8ec6af5dd2ab84 > # bad: [5a82d69d48c82e89aef44483d2a129f869f3506a] Add linux-next specific files for 20231120 > git bisect bad 5a82d69d48c82e89aef44483d2a129f869f3506a > # good: [ce252a92a867da8a6622463bff637e5f7b904a46] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next.git > git bisect good ce252a92a867da8a6622463bff637e5f7b904a46 > # good: [c22e026efad504e3b056d4436920d173a09c580e] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator.git > git bisect good c22e026efad504e3b056d4436920d173a09c580e > # good: [b7fc58ffb105470cb339163cc2b04e3f59387a45] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/fpga/linux-fpga.git > git bisect good b7fc58ffb105470cb339163cc2b04e3f59387a45 > # good: [26f89f0614f03e4016578a992fc2e86b048a5cb4] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl.git > git bisect good 26f89f0614f03e4016578a992fc2e86b048a5cb4 > # good: [602bf18307981f3bfd9ebf19921791a4256d3fd1] Merge branch 'for-6.7' into for-next > git bisect good 602bf18307981f3bfd9ebf19921791a4256d3fd1 > # good: [9f16a68069822b1df6bfb8a9ef7258a1e32b25e7] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching > git bisect good 9f16a68069822b1df6bfb8a9ef7258a1e32b25e7 > # good: [3ff57db6f6569ebc2cc333437e7e949749e59424] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/krisman/unicode.git > git bisect good 3ff57db6f6569ebc2cc333437e7e949749e59424 > # bad: [dd374e220ba492f95344a638b1efe5b2744fdd73] slub: Update frozen slabs documentations in the source > git bisect bad dd374e220ba492f95344a638b1efe5b2744fdd73 > # good: [a3058965bb35490454953aa2c87ea51004839f2f] slub: Prepare __slab_free() for unfrozen partial slab out of node partial list > git bisect good a3058965bb35490454953aa2c87ea51004839f2f > # bad: [c8d312e039030edab25836a326bcaeb2a3d4db14] slub: Delay freezing of partial slabs > git bisect bad c8d312e039030edab25836a326bcaeb2a3d4db14 > # good: [00b15a19ee543f0117cb217fcbab8b7b3fd50677] slub: Introduce freeze_slab() > git bisect good 00b15a19ee543f0117cb217fcbab8b7b3fd50677 > # first bad commit: [c8d312e039030edab25836a326bcaeb2a3d4db14] slub: Delay freezing of partial slabs