* [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] @ 2023-08-23 4:06 Ming Lei 2023-08-23 8:47 ` Christian Brauner 0 siblings, 1 reply; 14+ messages in thread From: Ming Lei @ 2023-08-23 4:06 UTC (permalink / raw) To: linux-fsdevel, Christian Brauner, Alexander Viro, linux-kernel Cc: ming.lei, linux-scsi, Changhui Zhong Looks the issue is more related with vfs, so forward to vfs list. ----- Forwarded message from Changhui Zhong <czhong@redhat.com> ----- Date: Wed, 23 Aug 2023 11:17:55 +0800 From: Changhui Zhong <czhong@redhat.com> To: linux-scsi@vger.kernel.org Cc: Ming Lei <ming.lei@redhat.com> Subject: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 Hello, triggered below warning issue with branch " Tree: mainline.kernel.org-clang Repository: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git @ master Commit Hash: 89bf6209cad66214d3774dac86b6bbf2aec6a30d Commit Name: v6.5-rc7-18-g89bf6209cad6 Kernel information: Commit message: Merge tag 'devicetree-fixes-for-6.5-2' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux " for more detail，please check https://datawarehouse.cki-project.org/kcidb/tests/9232643 #modprobe scsi_debug virtual_gb=128 #echo none > /sys/block/sdb/queue/scheduler #fio --bs=4k --ioengine=libaio --iodepth=1 --numjobs=4 --rw=randrw --name=sdb-libaio-randrw-4k --filename=/dev/sdb --direct=1 --size=60G --runtime=60 [ 3056.092761] Device: sdb Engine: libaio Sched: none Pattern: randrw Direct: 1 Depth: 1 Block size: 4K Size: 60G [ 3117.055168] ------------[ cut here ]------------ [ 3117.059778] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 [ 3117.067601] Modules linked in: scsi_debug nvme nvme_core nvme_common null_blk pktcdvd ipmi_watchdog ipmi_poweroff rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif arm_spe_pmu igb ipmi_devintf ipmi_msghandler arm_cmn arm_dmc620_pmu cppc_cpufreq arm_dsu_pmu acpi_tad loop fuse zram xfs crct10dif_ce polyval_ce polyval_generic ghash_ce sbsa_gwdt ast onboard_usb_hub i2c_algo_bit xgene_hwmon [last unloaded: scsi_debug] [ 3117.103572] CPU: 121 PID: 93233 Comm: bash Not tainted 6.5.0-rc7 #1 [ 3117.109827] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022 [ 3117.119119] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3117.126068] pc : __dentry_kill+0x214/0x278 [ 3117.130152] lr : __dentry_kill+0x194/0x278 [ 3117.134236] sp : ffff800084993870 [ 3117.137537] x29: ffff800084993870 x28: ffff0800830e2200 x27: 0000000000080400 [ 3117.144661] x26: 00000000fff7fbff x25: 0000000000000001 x24: ffff07ff88473198 [ 3117.151783] x23: ffff07ff884731c0 x22: ffff07ff9d033c80 x21: ffff07ff884731d0 [ 3117.158906] x20: ffff07ff88473198 x19: ffff07ff88473140 x18: ffffbc0d0739ceb4 [ 3117.166028] x17: 0000000000000000 x16: ffffbc0d073528c0 x15: ffff07ff89c761f8 [ 3117.173151] x14: 0000000000000002 x13: 0000000000000000 x12: 0000000000000001 [ 3117.180273] x11: ffff080f33fe0850 x10: ffffffffffffffff x9 : 0000000100000000 [ 3117.187395] x8 : ffffbc0d08aa1e98 x7 : 0000000000000000 x6 : 000000000000003f [ 3117.194518] x5 : ffff800084993a30 x4 : ffff800084993908 x3 : ffff07ff9cb7e210 [ 3117.201640] x2 : ffff07ff884731d0 x1 : 0000000000000000 x0 : ffff07ff884731f0 [ 3117.208763] Call trace: [ 3117.211197] __dentry_kill+0x214/0x278 [ 3117.214934] shrink_dentry_list+0x134/0x2b0 [ 3117.219105] prune_dcache_sb+0x64/0xa0 [ 3117.222842] super_cache_scan+0x144/0x198 [ 3117.226841] do_shrink_slab+0x1dc/0x420 [ 3117.230666] shrink_slab+0x114/0x388 [ 3117.234229] drop_slab+0xb0/0x118 [ 3117.237532] drop_caches_sysctl_handler+0xac/0x170 [ 3117.242313] proc_sys_call_handler+0x184/0x2d0 [ 3117.246745] proc_sys_write+0x20/0x38 [ 3117.250396] vfs_write+0x24c/0x368 [ 3117.253786] ksys_write+0x84/0xf8 [ 3117.257089] __arm64_sys_write+0x28/0x40 [ 3117.261000] invoke_syscall+0x78/0x110 [ 3117.264739] el0_svc_common+0xc0/0xf8 [ 3117.268390] do_el0_svc+0x3c/0xb8 [ 3117.271693] el0_svc+0x34/0x110 [ 3117.274825] el0t_64_sync_handler+0x84/0x100 [ 3117.279083] el0t_64_sync+0x194/0x198 [ 3117.282733] ---[ end trace 0000000000000000 ]--- [ 3119.372909] Device: sdb Engine: libaio Sched: none Pattern: randrw Direct: 1 Depth: 1 Block size: 16K Size: 60G [ 3144.068984] watchdog: BUG: soft lockup - CPU#97 stuck for 26s! [fio:93777] [ 3144.069984] watchdog: BUG: soft lockup - CPU#99 stuck for 26s! [systemd-udevd:1680] [ 3144.075849] Modules linked in: scsi_debug nvme [ 3144.083493] Modules linked in: [ 3144.087924] nvme_core nvme_common null_blk pktcdvd [ 3144.090967] scsi_debug [ 3144.090968] ipmi_watchdog ipmi_poweroff rfkill [ 3144.098267] nvme [ 3144.098267] sunrpc vfat fat [ 3144.102785] nvme_core [ 3144.104698] acpi_ipmi [ 3144.107566] nvme_common [ 3144.109912] ipmi_ssif [ 3144.112259] null_blk [ 3144.114779] arm_spe_pmu [ 3144.117126] pktcdvd [ 3144.119385] igb ipmi_devintf [ 3144.121905] ipmi_watchdog [ 3144.124078] ipmi_msghandler [ 3144.127033] ipmi_poweroff [ 3144.129728] arm_cmn [ 3144.132596] rfkill [ 3144.135289] arm_dmc620_pmu [ 3144.137462] sunrpc [ 3144.139548] cppc_cpufreq [ 3144.142329] vfat [ 3144.144415] arm_dsu_pmu [ 3144.147022] fat [ 3144.148934] acpi_tad loop [ 3144.151455] acpi_ipmi [ 3144.153280] fuse [ 3144.155974] ipmi_ssif [ 3144.158320] zram [ 3144.160233] arm_spe_pmu [ 3144.162579] xfs [ 3144.164493] igb [ 3144.167012] crct10dif_ce [ 3144.168838] ipmi_devintf [ 3144.170664] polyval_ce [ 3144.173271] ipmi_msghandler [ 3144.175877] polyval_generic [ 3144.178312] arm_cmn [ 3144.181178] ghash_ce [ 3144.184047] arm_dmc620_pmu [ 3144.186219] sbsa_gwdt [ 3144.188479] cppc_cpufreq [ 3144.191259] ast [ 3144.193606] arm_dsu_pmu [ 3144.196213] onboard_usb_hub [ 3144.198039] acpi_tad [ 3144.200559] i2c_algo_bit [ 3144.203427] loop [ 3144.205686] xgene_hwmon [ 3144.208294] fuse [ 3144.210206] [last unloaded: scsi_debug] [ 3144.212726] zram [ 3144.214638] [ 3144.214640] CPU: 97 PID: 93777 Comm: fio Tainted: G W 6.5.0-rc7 #1 [ 3144.218548] xfs [ 3144.220459] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022 [ 3144.221939] crct10dif_ce [ 3144.229493] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3144.231320] polyval_ce [ 3144.240610] pc : d_alloc_parallel+0x204/0x520 [ 3144.243218] polyval_generic [ 3144.250164] lr : d_alloc_parallel+0x128/0x520 [ 3144.252598] ghash_ce [ 3144.256940] sp : ffff800083ecb860 [ 3144.259809] sbsa_gwdt [ 3144.264151] x29: ffff800083ecb8b0 [ 3144.266411] ast [ 3144.269712] x28: 00000000e28e44d4 [ 3144.272059] onboard_usb_hub [ 3144.275360] x27: ffff0800c7e6c400 [ 3144.277186] i2c_algo_bit [ 3144.280574] [ 3144.280575] x26: ffff07ff9dfb48d0 [ 3144.283442] xgene_hwmon [ 3144.286830] x25: 0000000000011b30 [ 3144.289437] [last unloaded: scsi_debug] [ 3144.290915] x24: ffff07ffe89be298 [ 3144.294217] [ 3144.296737] [ 3144.296738] x23: ffff07ffe89be240 [ 3144.300126] CPU: 99 PID: 1680 Comm: systemd-udevd Tainted: G W 6.5.0-rc7 #1 [ 3144.304034] x22: ffff800083ecba20 x21: ffffbc0d084e9000 [ 3144.307424] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022 [ 3144.308902] [ 3144.308903] x20: ffffbc0d08aa1e98 [ 3144.310381] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 3144.313682] x19: ffff07ff9eeac918 x18: 00000000fffffffb [ 3144.322019] pc : d_alloc_parallel+0x17c/0x520 [ 3144.327316] [ 3144.327317] x17: ffff07ff93f1c021 [ 3144.336608] lr : d_alloc_parallel+0x128/0x520 [ 3144.338086] x16: 636f6c622f000000 x15: fefefefeff727372 [ 3144.341388] sp : ffff8000844ab9b0 [ 3144.348335] [ 3144.348336] x14: fefefefeff7b7b7b [ 3144.353634] x29: ffff8000844aba00 [ 3144.357976] x13: 0000000000000000 x12: 00000000e28e44d4 [ 3144.359456] x28: 00000000e777e234 [ 3144.362757] [ 3144.362757] x11: ffff080f32b00000 [ 3144.367100] x27: ffff07ff96888000 [ 3144.372398] x10: 0000000000000000 x9 : ffff07ff93f1c037 [ 3144.375700] [ 3144.377178] [ 3144.377179] x8 : 0000000000000000 [ 3144.380480] x26: ffff07ff9de47890 [ 3144.383781] x7 : 7374736575716572 [ 3144.389080] x25: 0000000000011b30 [ 3144.392467] x6 : 65757165725f726e [ 3144.393947] x24: ffff07ff88faf618 [ 3144.397248] [ 3144.397249] x5 : ffff07ff9dfb4883 [ 3144.400637] [ 3144.405934] x4 : ffff07ff93f1c042 [ 3144.407414] x23: ffff07ff88faf5c0 [ 3144.408892] x3 : ffff07ff9dfb4bd0 [ 3144.412193] x22: ffff8000844abb30 [ 3144.415494] [ 3144.415495] x2 : ffff800083ecb87c [ 3144.418883] x21: ffffbc0d084e9000 [ 3144.422271] x1 : ffff800083ecba20 x0 : 0000000000000000 [ 3144.425660] [ 3144.429048] [ 3144.429049] Call trace: [ 3144.430527] x20: ffffbc0d08aa1e98 [ 3144.433828] d_alloc_parallel+0x204/0x520 [ 3144.435307] x19: 00000000000000e8 [ 3144.438695] __lookup_slow+0x6c/0x158 [ 3144.441997] x18: 00000000fffffffb [ 3144.445385] lookup_slow+0x4c/0x78 [ 3144.448774] [ 3144.450252] walk_component+0x10c/0x128 [ 3144.453553] x17: ffff07ff9400a021 [ 3144.456941] path_lookupat+0x60/0x140 [ 3144.462240] x16: 697665642f000000 [ 3144.463718] filename_lookup+0xd8/0x1d0 [ 3144.465197] x15: 722e6a626e6b612e [ 3144.467629] vfs_statx+0x90/0x220 [ 3144.470932] [ 3144.474927] __arm64_sys_newfstatat+0xa0/0x100 [ 3144.478316] x14: 7aff6b6b7f6b6bff [ 3144.481964] invoke_syscall+0x78/0x110 [ 3144.485353] x13: 0000000000000000 [ 3144.488741] el0_svc_common+0xc0/0xf8 [ 3144.490219] x12: 00000000e777e234 [ 3144.494041] do_el0_svc+0x3c/0xb8 [ 3144.497343] [ 3144.500991] el0_svc+0x34/0x110 [ 3144.504380] x11: ffff080f32b00000 [ 3144.508201] el0t_64_sync_handler+0x84/0x100 [ 3144.511590] x10: 0000000000000000 [ 3144.514891] el0t_64_sync+0x194/0x198 [ 3144.516370] x9 : ffff07ff9400a06b [ 3144.564153] x8 : ffff07ff884731f1 x7 : 25732500716d016b x6 : 0000000032757063 [ 3144.571280] x5 : ffff07ff9de4783d x4 : ffff07ff9400a070 x3 : ffff07ff9de46c90 [ 3144.578407] x2 : ffff8000844ab9cc x1 : ffff8000844abb30 x0 : 0000000000000000 [ 3144.585534] Call trace: [ 3144.587969] d_alloc_parallel+0x17c/0x520 [ 3144.591971] path_openat+0x238/0xc70 [ 3144.595539] do_filp_open+0xc4/0x178 [ 3144.599107] do_sys_openat2+0x90/0x100 [ 3144.602847] __arm64_sys_openat+0x7c/0xb0 [ 3144.606848] invoke_syscall+0x78/0x110 [ 3144.610591] el0_svc_common+0x94/0xf8 [ 3144.614247] do_el0_svc+0x3c/0xb8 [ 3144.617555] el0_svc+0x34/0x110 [ 3144.620690] el0t_64_sync_handler+0x84/0x100 [ 3144.624954] el0t_64_sync+0x194/0x198 [ 3168.068867] watchdog: BUG: soft lockup - CPU#97 stuck for 48s! [fio:93777] [ 3168.069867] watchdog: BUG: soft lockup - CPU#99 stuck for 48s! [systemd-udevd:1680] Thanks， ----- End forwarded message ----- -- Ming ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-08-23 4:06 [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] Ming Lei @ 2023-08-23 8:47 ` Christian Brauner 2023-08-28 10:43 ` Ming Lei 0 siblings, 1 reply; 14+ messages in thread From: Christian Brauner @ 2023-08-23 8:47 UTC (permalink / raw) To: Ming Lei Cc: linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong On Wed, Aug 23, 2023 at 12:06:14PM +0800, Ming Lei wrote: > > Looks the issue is more related with vfs, so forward to vfs list. > > ----- Forwarded message from Changhui Zhong <czhong@redhat.com> ----- > > Date: Wed, 23 Aug 2023 11:17:55 +0800 > From: Changhui Zhong <czhong@redhat.com> > To: linux-scsi@vger.kernel.org > Cc: Ming Lei <ming.lei@redhat.com> > Subject: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 > > Hello, > > triggered below warning issue with branch > " > Tree: mainline.kernel.org-clang > Repository: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > @ master > Commit Hash: 89bf6209cad66214d3774dac86b6bbf2aec6a30d > Commit Name: v6.5-rc7-18-g89bf6209cad6 > Kernel information: > Commit message: Merge tag 'devicetree-fixes-for-6.5-2' of > git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux > " > for more detail，please check > https://datawarehouse.cki-project.org/kcidb/tests/9232643 > > #modprobe scsi_debug virtual_gb=128 > #echo none > /sys/block/sdb/queue/scheduler > #fio --bs=4k --ioengine=libaio --iodepth=1 --numjobs=4 --rw=randrw > --name=sdb-libaio-randrw-4k --filename=/dev/sdb --direct=1 --size=60G > --runtime=60 Looking at this issue it seems unlikely that this is a vfs bug. We should see this all over the place and specifically not just on arm64. The sequence here seems to be: echo 4 > /proc/sys/vm/drop_caches rmmod scsi_debug > /dev/null 3>&1 [ 3117.059778] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 [ 3117.067601] Modules linked in: scsi_debug nvme nvme_core nvme_common null_blk pktcdvd ipmi_watchdog ipmi_poweroff rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif arm_spe_pmu igb ipmi_devintf ipmi_msghandler arm_cmn arm_dmc620_pmu cppc_cpufreq arm_dsu_pmu acpi_tad loop fuse zram xfs crct10dif_ce polyval_ce polyval_generic ghash_ce sbsa_gwdt ast onboard_usb_hub i2c_algo_bit xgene_hwmon [last unloaded: scsi_debug] So my money is on some device that gets removed still having an increased refcount and pinning the dentry. Immediate suspects would be: 7882541ca06d ("of/platform: increase refcount of fwnode") but that part is complete speculation on my part. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-08-23 8:47 ` Christian Brauner @ 2023-08-28 10:43 ` Ming Lei 2023-09-13 8:59 ` Yi Zhang 0 siblings, 1 reply; 14+ messages in thread From: Ming Lei @ 2023-08-28 10:43 UTC (permalink / raw) To: Christian Brauner Cc: linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, Yi Zhang On Wed, Aug 23, 2023 at 4:47 PM Christian Brauner <brauner@kernel.org> wrote: > > On Wed, Aug 23, 2023 at 12:06:14PM +0800, Ming Lei wrote: > > > > Looks the issue is more related with vfs, so forward to vfs list. > > > > ----- Forwarded message from Changhui Zhong <czhong@redhat.com> ----- > > > > Date: Wed, 23 Aug 2023 11:17:55 +0800 > > From: Changhui Zhong <czhong@redhat.com> > > To: linux-scsi@vger.kernel.org > > Cc: Ming Lei <ming.lei@redhat.com> > > Subject: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 > > > > Hello, > > > > triggered below warning issue with branch > > " > > Tree: mainline.kernel.org-clang > > Repository: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > @ master > > Commit Hash: 89bf6209cad66214d3774dac86b6bbf2aec6a30d > > Commit Name: v6.5-rc7-18-g89bf6209cad6 > > Kernel information: > > Commit message: Merge tag 'devicetree-fixes-for-6.5-2' of > > git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux > > " > > for more detail，please check > > https://datawarehouse.cki-project.org/kcidb/tests/9232643 > > > > #modprobe scsi_debug virtual_gb=128 > > #echo none > /sys/block/sdb/queue/scheduler > > #fio --bs=4k --ioengine=libaio --iodepth=1 --numjobs=4 --rw=randrw > > --name=sdb-libaio-randrw-4k --filename=/dev/sdb --direct=1 --size=60G > > --runtime=60 > > Looking at this issue it seems unlikely that this is a vfs bug. > We should see this all over the place and specifically not just on arm64. > > The sequence here seems to be: > > echo 4 > /proc/sys/vm/drop_caches > rmmod scsi_debug > /dev/null 3>&1 > > [ 3117.059778] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 > [ 3117.067601] Modules linked in: scsi_debug nvme nvme_core nvme_common null_blk pktcdvd ipmi_watchdog ipmi_poweroff rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif arm_spe_pmu igb ipmi_devintf ipmi_msghandler arm_cmn arm_dmc620_pmu cppc_cpufreq arm_dsu_pmu acpi_tad loop fuse zram xfs crct10dif_ce polyval_ce polyval_generic ghash_ce sbsa_gwdt ast onboard_usb_hub i2c_algo_bit xgene_hwmon [last unloaded: scsi_debug] > > So my money is on some device that gets removed still having an > increased refcount and pinning the dentry. Immediate suspects would be: > > 7882541ca06d ("of/platform: increase refcount of fwnode") > > but that part is complete speculation on my part. BTW, just saw another panic on 6.5-rc7, still scsi_debug test on arm64: [ 959.371726] sr 50:0:0:0: Attached scsi generic sg1 type 5 [ 959.603145] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 959.603155] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 959.603950] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 959.604052] scsi 49:0:0:0: Power-on or device reset occurred [ 959.609336] sr 49:0:0:0: [sr1] scsi-1 drive [ 959.611360] scsi 48:0:0:0: Power-on or device reset occurred [ 959.614540] Unable to handle kernel paging request at virtual address 65888c2e6fe694d5 [ 959.614544] Mem abort info: [ 959.614545] ESR = 0x0000000096000004 [ 959.614547] EC = 0x25: DABT (current EL), IL = 32 bits [ 959.614550] SET = 0, FnV = 0 [ 959.614552] EA = 0, S1PTW = 0 [ 959.614553] FSC = 0x04: level 0 translation fault [ 959.614555] Data abort info: [ 959.614556] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 [ 959.614559] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 [ 959.614561] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 [ 959.614563] [65888c2e6fe694d5] address between user and kernel address ranges [ 959.614566] Internal error: Oops: 0000000096000004 [#1] SMP [ 959.614570] Modules linked in: pktcdvd scsi_debug ipmi_watchdog ipmi_poweroff rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif arm_spe_pmu igb ipmi_devintf arm_cmn ipmi_msghandler arm_dmc620_pmu arm_dsu_pmu cppc_cpufreq acpi_tad loop fuse zram xfs nvme crct10dif_ce polyval_ce nvme_core polyval_generic ghash_ce sbsa_gwdt nvme_common ast onboard_usb_hub i2c_algo_bit xgene_hwmon [last unloaded: scsi_debug] [ 959.614620] CPU: 108 PID: 19529 Comm: check Not tainted 6.5.0-rc7 #1 [ 959.614625] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022 [ 959.614627] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 959.614632] pc : d_alloc_parallel+0x140/0x440 [ 959.614641] lr : d_alloc_parallel+0xcc/0x440 [ 959.614646] sp : ffff80008a7d3290 [ 959.614647] x29: ffff80008a7d3290 x28: ffff07ff8230c530 x27: 65888c2e6fe69565 [ 959.614654] x26: ffffcb72eac9e1d0 x25: ffff80008a7d33d8 x24: ffff07ffa0fd3800 [ 959.614659] x23: 00000000000003c0 x22: 000000007a701548 x21: ffffcb72eac9ffd0 [ 959.614664] x20: ffff07ffa0fd35c0 x19: ffffcb72ea6e9600 x18: ffffffffffffffff [ 959.614670] x17: 00000000440fd8e0 x16: 00000000b6431329 x15: ffff80008a7d3360 [ 959.614675] x14: ffff80008a7d3508 x13: ffffcb72e94da6d0 x12: ffff80008a7d334c [ 959.614680] x11: 0000000c7a701548 x10: ffff3c8cb54590b8 x9 : ffffcb72e80832f4 [ 959.614685] x8 : ffff07ffa0fd35c0 x7 : 7473696c5f71725f x6 : 0000000000200008 [ 959.614690] x5 : ffffcb72ea6f4000 x4 : 00000000003d380a x3 : 0000000000000004 [ 959.614696] x2 : ffff80008a7d331c x1 : ffff07ffe1760000 x0 : 0000000000005000 [ 959.614701] Call trace: [ 959.614703] d_alloc_parallel+0x140/0x440 [ 959.614708] __lookup_slow+0x64/0x158 [ 959.614714] lookup_one_len+0xac/0xc8 [ 959.614719] start_creating.part.0+0x88/0x198 [ 959.614725] __debugfs_create_file+0x70/0x230 [ 959.614730] debugfs_create_file+0x34/0x48 [ 959.614734] blk_mq_debugfs_register_hctx+0x154/0x1d0 [ 959.614740] blk_mq_debugfs_register+0xfc/0x1e0 [ 959.614745] blk_register_queue+0xc0/0x1f0 [ 959.614750] device_add_disk+0x1dc/0x3e0 [ 959.614754] sr_probe+0x2c0/0x368 [ 959.614760] really_probe+0x190/0x3d8 [ 959.614766] __driver_probe_device+0x84/0x180 [ 959.614771] driver_probe_device+0x44/0x120 [ 959.614776] __device_attach_driver+0xc4/0x168 [ 959.614781] bus_for_each_drv+0x8c/0xf0 [ 959.614785] __device_attach+0xa4/0x1c0 [ 959.614790] device_initial_probe+0x1c/0x30 [ 959.614795] bus_probe_device+0xb4/0xc0 [ 959.614799] device_add+0x508/0x6f8 [ 959.614803] scsi_sysfs_add_sdev+0x8c/0x250 [ 959.614809] scsi_add_lun+0x424/0x558 [ 959.614813] scsi_probe_and_add_lun+0x11c/0x430 [ 959.614817] __scsi_scan_target+0xb8/0x258 [ 959.614821] scsi_scan_channel+0xa0/0xb8 [ 959.614825] scsi_scan_host_selected+0x170/0x188 [ 959.614830] store_scan+0x194/0x1a8 [ 959.614835] dev_attr_store+0x20/0x40 [ 959.614840] sysfs_kf_write+0x4c/0x68 [ 959.614845] kernfs_fop_write_iter+0x13c/0x1d8 [ 959.614849] vfs_write+0x1c0/0x310 [ 959.614855] ksys_write+0x78/0x118 [ 959.614859] __arm64_sys_write+0x24/0x38 [ 959.614864] invoke_syscall+0x78/0x100 [ 959.614868] el0_svc_common.constprop.0+0x4c/0xf8 [ 959.614871] do_el0_svc+0x34/0x50 [ 959.614874] el0_svc+0x34/0x108 [ 959.614879] el0t_64_sync_handler+0x120/0x130 [ 959.614884] el0t_64_sync+0x194/0x198 [ 959.614889] Code: 54000088 14000067 f940037b b4000cbb (b8570360) [ 959.614892] ---[ end trace 0000000000000000 ]--- [ 959.614895] Kernel panic - not syncing: Oops: Fatal exception [ 959.614897] SMP: stopping secondary CPUs [ 959.619492] Kernel Offset: 0x4b7267c40000 from 0xffff800080000000 [ 959.619494] PHYS_OFFSET: 0x80000000 [ 959.619496] CPU features: 0x00000010,b80140a1,8841720b [ 959.619498] Memory Limit: none [ 960.040819] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]--- ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-08-28 10:43 ` Ming Lei @ 2023-09-13 8:59 ` Yi Zhang 2023-09-16 6:55 ` Baokun Li 2023-09-17 0:35 ` Bagas Sanjaya 0 siblings, 2 replies; 14+ messages in thread From: Yi Zhang @ 2023-09-13 8:59 UTC (permalink / raw) To: Ming Lei Cc: Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, mark.rutland The issue still can be reproduced on the latest linux tree[2]. To reproduce I need to run about 1000 times blktests block/001, and bisect shows it was introduced with commit[1], as it was not 100% reproduced, not sure if it's the culprit? [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery [2] [ 2304.536339] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2304.540805] sr 50:0:0:0: Attached scsi CD-ROM sr3 [ 2304.544574] scsi 48:0:0:0: Power-on or device reset occurred [ 2304.600645] sr 48:0:0:0: [sr1] scsi-1 drive [ 2304.616364] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2304.624639] scsi 51:0:0:0: Power-on or device reset occurred [ 2304.626634] sr 48:0:0:0: Attached scsi CD-ROM sr1 [ 2304.680537] sr 51:0:0:0: [sr2] scsi-1 drive [ 2304.706394] sr 51:0:0:0: Attached scsi CD-ROM sr2 [ 2304.746329] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2304.754569] scsi 49:0:0:0: Power-on or device reset occurred [ 2304.756302] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2304.768483] scsi 50:0:0:0: Power-on or device reset occurred [ 2304.806321] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2304.810587] sr 49:0:0:0: [sr0] scsi-1 drive [ 2304.814561] scsi 48:0:0:0: Power-on or device reset occurred [ 2304.824475] sr 50:0:0:0: [sr3] scsi-1 drive [ 2304.836384] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2304.840364] sr 49:0:0:0: Attached scsi CD-ROM sr0 [ 2304.844619] scsi 51:0:0:0: Power-on or device reset occurred [ 2304.850444] sr 50:0:0:0: Attached scsi CD-ROM sr3 [ 2304.874563] sr 48:0:0:0: [sr1] scsi-1 drive [ 2304.900660] sr 51:0:0:0: [sr2] scsi-1 drive [ 2304.901506] sr 48:0:0:0: Attached scsi CD-ROM sr1 [ 2304.926306] sr 51:0:0:0: Attached scsi CD-ROM sr2 [ 2305.056432] scsi 50:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2305.056572] scsi 49:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2305.064635] scsi 50:0:0:0: Power-on or device reset occurred [ 2305.072821] scsi 49:0:0:0: Power-on or device reset occurred [ 2305.086286] scsi 51:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2305.086357] scsi 48:0:0:0: CD-ROM Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 2305.094521] scsi 51:0:0:0: Power-on or device reset occurred [ 2305.102693] scsi 48:0:0:0: Power-on or device reset occurred [ 2305.128785] sr 50:0:0:0: [sr0] scsi-1 drive [ 2305.134445] sr 49:0:0:0: [sr1] scsi-1 drive [ 2305.154728] sr 50:0:0:0: Attached scsi CD-ROM sr0 [ 2305.158607] sr 51:0:0:0: [sr2] scsi-1 drive [ 2305.160392] sr 49:0:0:0: Attached scsi CD-ROM sr1 [ 2305.164254] sr 48:0:0:0: [sr3] scsi-1 drive [ 2305.184185] sr 51:0:0:0: Attached scsi CD-ROM sr2 [ 2305.190086] sr 48:0:0:0: Attached scsi CD-ROM sr3 [ 2305.555658] Unable to handle kernel execute from non-executable memory at virtual address ffffc61b656052e8 [ 2305.565301] Mem abort info: [ 2305.568086] ESR = 0x000000008600000e [ 2305.571822] EC = 0x21: IABT (current EL), IL = 32 bits [ 2305.577123] SET = 0, FnV = 0 [ 2305.580164] EA = 0, S1PTW = 0 [ 2305.583292] FSC = 0x0e: level 2 permission fault [ 2305.588074] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000080731fa0000 [ 2305.594761] [ffffc61b656052e8] pgd=1000080ffffff003, p4d=1000080ffffff003, pud=1000080fffffe003, pmd=0068080732e00f01 [ 2305.605362] Internal error: Oops: 000000008600000e [#1] SMP [ 2305.610922] Modules linked in: scsi_debug sr_mod pktcdvd cdrom rfkill sunrpc vfat fat acpi_ipmi arm_spe_pmu ipmi_ssif ipmi_devintf ipmi_msghandler arm_cmn arm_dmc620_pmu arm_dsu_pmu cppc_cpufreq loop fuse zram xfs crct10dif_ce ghash_ce nvme sha2_ce nvme_core sha256_arm64 igb sha1_ce ast sbsa_gwdt nvme_common i2c_designware_platform i2c_algo_bit i2c_designware_core xgene_hwmon dm_mod [last unloaded: scsi_debug] [ 2305.647236] CPU: 85 PID: 1 Comm: systemd Kdump: loaded Not tainted 6.6.0-rc1+ #13 [ 2305.654706] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS F31n (SCP: 2.10.20220810) 09/30/2022 [ 2305.663997] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 2305.670946] pc : in_lookup_hashtable+0x1138/0x2000 [ 2305.675728] lr : rcu_do_batch+0x194/0x488 [ 2305.679727] sp : ffff8000802abe60 [ 2305.683029] x29: ffff8000802abe60 x28: ffffc61b6524c7c0 x27: ffffc61b63452f40 [ 2305.690152] x26: ffff080f37ab6438 x25: 000000000000000a x24: 0000000000000000 [ 2305.697274] x23: 0000000000000002 x22: ffff8000802abec0 x21: ffff080f37ab63c0 [ 2305.704396] x20: ffff07ff8136a580 x19: 0000000000000003 x18: 0000000000000000 [ 2305.711519] x17: ffff41f3d3161000 x16: ffff8000802a8000 x15: 0000000000000000 [ 2305.718641] x14: 0000000000000000 x13: ffff07ffa131802d x12: ffff80008041bb94 [ 2305.725764] x11: 0000000000000040 x10: ffff07ff802622e8 x9 : ffffc61b63452e30 [ 2305.732887] x8 : 000002189dce1780 x7 : ffff07ff8d5c1000 x6 : ffff41f3d3161000 [ 2305.740009] x5 : ffff07ff8136a580 x4 : ffff080f37aba960 x3 : 000000001550a055 [ 2305.747131] x2 : 0000000000000000 x1 : ffffc61b656052e8 x0 : ffff080184c565f0 [ 2305.754254] Call trace: [ 2305.756687] in_lookup_hashtable+0x1138/0x2000 [ 2305.761119] rcu_core+0x268/0x350 [ 2305.764422] rcu_core_si+0x18/0x30 [ 2305.767812] __do_softirq+0x120/0x3d4 [ 2305.771462] ____do_softirq+0x18/0x30 [ 2305.775112] call_on_irq_stack+0x24/0x30 [ 2305.779022] do_softirq_own_stack+0x24/0x38 [ 2305.783192] __irq_exit_rcu+0xfc/0x130 [ 2305.786929] irq_exit_rcu+0x18/0x30 [ 2305.790404] el1_interrupt+0x4c/0xe8 [ 2305.793969] el1h_64_irq_handler+0x18/0x28 [ 2305.798052] el1h_64_irq+0x78/0x80 [ 2305.801441] d_same_name+0x50/0xd0 [ 2305.804832] __lookup_slow+0x64/0x158 [ 2305.808482] walk_component+0xe0/0x1a0 [ 2305.812219] path_lookupat+0x7c/0x1b8 [ 2305.815869] filename_lookup+0xb4/0x1b8 [ 2305.819692] vfs_statx+0x94/0x1a8 [ 2305.822995] vfs_fstatat+0xd4/0x110 [ 2305.826471] __do_sys_newfstatat+0x58/0xa8 [ 2305.830556] __arm64_sys_newfstatat+0x28/0x40 [ 2305.834901] invoke_syscall.constprop.0+0x80/0xd8 [ 2305.839592] do_el0_svc+0x48/0xd0 [ 2305.842894] el0_svc+0x4c/0x1c0 [ 2305.846023] el0t_64_sync_handler+0x120/0x130 [ 2305.850367] el0t_64_sync+0x1a4/0x1a8 [ 2305.854017] Code: 00000000 00000000 00000000 00000000 (84c565f1) [ 2305.860098] SMP: stopping secondary CPUs [ 2305.865048] Starting crashdump kernel... [ 2305.868958] Bye! On Mon, Aug 28, 2023 at 6:43 PM Ming Lei <ming.lei@redhat.com> wrote: > > On Wed, Aug 23, 2023 at 4:47 PM Christian Brauner <brauner@kernel.org> wrote: > > > > On Wed, Aug 23, 2023 at 12:06:14PM +0800, Ming Lei wrote: > > > > > > Looks the issue is more related with vfs, so forward to vfs list. > > > > > > ----- Forwarded message from Changhui Zhong <czhong@redhat.com> ----- > > > > > > Date: Wed, 23 Aug 2023 11:17:55 +0800 > > > From: Changhui Zhong <czhong@redhat.com> > > > To: linux-scsi@vger.kernel.org > > > Cc: Ming Lei <ming.lei@redhat.com> > > > Subject: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 > > > > > > Hello, > > > > > > triggered below warning issue with branch > > > " > > > Tree: mainline.kernel.org-clang > > > Repository: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > > @ master > > > Commit Hash: 89bf6209cad66214d3774dac86b6bbf2aec6a30d > > > Commit Name: v6.5-rc7-18-g89bf6209cad6 > > > Kernel information: > > > Commit message: Merge tag 'devicetree-fixes-for-6.5-2' of > > > git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux > > > " > > > for more detail，please check > > > https://datawarehouse.cki-project.org/kcidb/tests/9232643 > > > > > > #modprobe scsi_debug virtual_gb=128 > > > #echo none > /sys/block/sdb/queue/scheduler > > > #fio --bs=4k --ioengine=libaio --iodepth=1 --numjobs=4 --rw=randrw > > > --name=sdb-libaio-randrw-4k --filename=/dev/sdb --direct=1 --size=60G > > > --runtime=60 > > > > Looking at this issue it seems unlikely that this is a vfs bug. > > We should see this all over the place and specifically not just on arm64. > > > > The sequence here seems to be: > > > > echo 4 > /proc/sys/vm/drop_caches > > rmmod scsi_debug > /dev/null 3>&1 > > > > [ 3117.059778] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278 > > [ 3117.067601] Modules linked in: scsi_debug nvme nvme_core nvme_common null_blk pktcdvd ipmi_watchdog ipmi_poweroff rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif arm_spe_pmu igb ipmi_devintf ipmi_msghandler arm_cmn arm_dmc620_pmu cppc_cpufreq arm_dsu_pmu acpi_tad loop fuse zram xfs crct10dif_ce polyval_ce polyval_generic ghash_ce sbsa_gwdt ast onboard_usb_hub i2c_algo_bit xgene_hwmon [last unloaded: scsi_debug] > > > > So my money is on some device that gets removed still having an > > increased refcount and pinning the dentry. Immediate suspects would be: > > > > 7882541ca06d ("of/platform: increase refcount of fwnode") > > > > but that part is complete speculation on my part. > > BTW, just saw another panic on 6.5-rc7, still scsi_debug test on arm64: > > [ 959.371726] sr 50:0:0:0: Attached scsi generic sg1 type 5 > [ 959.603145] scsi 48:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 959.603155] scsi 50:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 959.603950] scsi 49:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 959.604052] scsi 49:0:0:0: Power-on or device reset occurred > [ 959.609336] sr 49:0:0:0: [sr1] scsi-1 drive > [ 959.611360] scsi 48:0:0:0: Power-on or device reset occurred > [ 959.614540] Unable to handle kernel paging request at virtual > address 65888c2e6fe694d5 > [ 959.614544] Mem abort info: > [ 959.614545] ESR = 0x0000000096000004 > [ 959.614547] EC = 0x25: DABT (current EL), IL = 32 bits > [ 959.614550] SET = 0, FnV = 0 > [ 959.614552] EA = 0, S1PTW = 0 > [ 959.614553] FSC = 0x04: level 0 translation fault > [ 959.614555] Data abort info: > [ 959.614556] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000 > [ 959.614559] CM = 0, WnR = 0, TnD = 0, TagAccess = 0 > [ 959.614561] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0 > [ 959.614563] [65888c2e6fe694d5] address between user and kernel address ranges > [ 959.614566] Internal error: Oops: 0000000096000004 [#1] SMP > [ 959.614570] Modules linked in: pktcdvd scsi_debug ipmi_watchdog > ipmi_poweroff rfkill sunrpc vfat fat acpi_ipmi ipmi_ssif arm_spe_pmu > igb ipmi_devintf arm_cmn ipmi_msghandler arm_dmc620_pmu arm_dsu_pmu > cppc_cpufreq acpi_tad loop fuse zram xfs nvme crct10dif_ce polyval_ce > nvme_core polyval_generic ghash_ce sbsa_gwdt nvme_common ast > onboard_usb_hub i2c_algo_bit xgene_hwmon [last unloaded: scsi_debug] > [ 959.614620] CPU: 108 PID: 19529 Comm: check Not tainted 6.5.0-rc7 #1 > [ 959.614625] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS > F31n (SCP: 2.10.20220810) 09/30/2022 > [ 959.614627] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > [ 959.614632] pc : d_alloc_parallel+0x140/0x440 > [ 959.614641] lr : d_alloc_parallel+0xcc/0x440 > [ 959.614646] sp : ffff80008a7d3290 > [ 959.614647] x29: ffff80008a7d3290 x28: ffff07ff8230c530 x27: 65888c2e6fe69565 > [ 959.614654] x26: ffffcb72eac9e1d0 x25: ffff80008a7d33d8 x24: ffff07ffa0fd3800 > [ 959.614659] x23: 00000000000003c0 x22: 000000007a701548 x21: ffffcb72eac9ffd0 > [ 959.614664] x20: ffff07ffa0fd35c0 x19: ffffcb72ea6e9600 x18: ffffffffffffffff > [ 959.614670] x17: 00000000440fd8e0 x16: 00000000b6431329 x15: ffff80008a7d3360 > [ 959.614675] x14: ffff80008a7d3508 x13: ffffcb72e94da6d0 x12: ffff80008a7d334c > [ 959.614680] x11: 0000000c7a701548 x10: ffff3c8cb54590b8 x9 : ffffcb72e80832f4 > [ 959.614685] x8 : ffff07ffa0fd35c0 x7 : 7473696c5f71725f x6 : 0000000000200008 > [ 959.614690] x5 : ffffcb72ea6f4000 x4 : 00000000003d380a x3 : 0000000000000004 > [ 959.614696] x2 : ffff80008a7d331c x1 : ffff07ffe1760000 x0 : 0000000000005000 > [ 959.614701] Call trace: > [ 959.614703] d_alloc_parallel+0x140/0x440 > [ 959.614708] __lookup_slow+0x64/0x158 > [ 959.614714] lookup_one_len+0xac/0xc8 > [ 959.614719] start_creating.part.0+0x88/0x198 > [ 959.614725] __debugfs_create_file+0x70/0x230 > [ 959.614730] debugfs_create_file+0x34/0x48 > [ 959.614734] blk_mq_debugfs_register_hctx+0x154/0x1d0 > [ 959.614740] blk_mq_debugfs_register+0xfc/0x1e0 > [ 959.614745] blk_register_queue+0xc0/0x1f0 > [ 959.614750] device_add_disk+0x1dc/0x3e0 > [ 959.614754] sr_probe+0x2c0/0x368 > [ 959.614760] really_probe+0x190/0x3d8 > [ 959.614766] __driver_probe_device+0x84/0x180 > [ 959.614771] driver_probe_device+0x44/0x120 > [ 959.614776] __device_attach_driver+0xc4/0x168 > [ 959.614781] bus_for_each_drv+0x8c/0xf0 > [ 959.614785] __device_attach+0xa4/0x1c0 > [ 959.614790] device_initial_probe+0x1c/0x30 > [ 959.614795] bus_probe_device+0xb4/0xc0 > [ 959.614799] device_add+0x508/0x6f8 > [ 959.614803] scsi_sysfs_add_sdev+0x8c/0x250 > [ 959.614809] scsi_add_lun+0x424/0x558 > [ 959.614813] scsi_probe_and_add_lun+0x11c/0x430 > [ 959.614817] __scsi_scan_target+0xb8/0x258 > [ 959.614821] scsi_scan_channel+0xa0/0xb8 > [ 959.614825] scsi_scan_host_selected+0x170/0x188 > [ 959.614830] store_scan+0x194/0x1a8 > [ 959.614835] dev_attr_store+0x20/0x40 > [ 959.614840] sysfs_kf_write+0x4c/0x68 > [ 959.614845] kernfs_fop_write_iter+0x13c/0x1d8 > [ 959.614849] vfs_write+0x1c0/0x310 > [ 959.614855] ksys_write+0x78/0x118 > [ 959.614859] __arm64_sys_write+0x24/0x38 > [ 959.614864] invoke_syscall+0x78/0x100 > [ 959.614868] el0_svc_common.constprop.0+0x4c/0xf8 > [ 959.614871] do_el0_svc+0x34/0x50 > [ 959.614874] el0_svc+0x34/0x108 > [ 959.614879] el0t_64_sync_handler+0x120/0x130 > [ 959.614884] el0t_64_sync+0x194/0x198 > [ 959.614889] Code: 54000088 14000067 f940037b b4000cbb (b8570360) > [ 959.614892] ---[ end trace 0000000000000000 ]--- > [ 959.614895] Kernel panic - not syncing: Oops: Fatal exception > [ 959.614897] SMP: stopping secondary CPUs > [ 959.619492] Kernel Offset: 0x4b7267c40000 from 0xffff800080000000 > [ 959.619494] PHYS_OFFSET: 0x80000000 > [ 959.619496] CPU features: 0x00000010,b80140a1,8841720b > [ 959.619498] Memory Limit: none > [ 960.040819] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]--- > -- Best Regards, Yi Zhang ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-13 8:59 ` Yi Zhang @ 2023-09-16 6:55 ` Baokun Li 2023-09-17 9:10 ` Peter Zijlstra 2023-09-19 15:10 ` Mark Rutland 2023-09-17 0:35 ` Bagas Sanjaya 1 sibling, 2 replies; 14+ messages in thread From: Baokun Li @ 2023-09-16 6:55 UTC (permalink / raw) To: Yi Zhang, Ming Lei, mark.rutland Cc: Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), Baokun Li, peterz, Kees Cook, chengzhihao On 2023/9/13 16:59, Yi Zhang wrote: > The issue still can be reproduced on the latest linux tree[2]. > To reproduce I need to run about 1000 times blktests block/001, and > bisect shows it was introduced with commit[1], as it was not 100% > reproduced, not sure if it's the culprit? > > > [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery Hello, everyone！ We have confirmed that the merge-in of this patch caused hlist_bl_lock (aka, bit_spin_lock) to fail, which in turn triggered the issue above. The process in which VFS issue arise is as follows: 1. bl_head >>> first==dentry2 >>> dentry1 dentry2->next = dentry1 dentry2->pprev = head dentry1->next = NULL dentry1->pprev = dentry2 2. Concurrent deletion of dentry, hlist_bl_lock lock protection failure ``` __hlist_bl_del(dentry2) __hlist_bl_del(dentry1) dentry2->next = NULL; dentry1->next = NULL; dentry1->pprev = NULL; head->first = dentry1 dentry1->pprev = head dentry2->next = NULL; dentry2->pprev = NULL; ``` 3. WARN_ON/BUG_ON is triggered because dentry1 is still on the hlist after being deleted. dentry1->next = NULL dentry1->pprev = head Verify that hlist_bl_lock is not working with the following mod： mymod.c ``` #include <linux/kallsyms.h> #include <linux/module.h> #include <linux/moduleloader.h> #include <linux/kernel.h> #include <linux/jiffies.h> #include <linux/sched.h> #include <linux/smp.h> #include <linux/cpu.h> #include <linux/delay.h> #include <linux/percpu.h> #include <linux/threads.h> #include <linux/kthread.h> #include <linux/kernel_stat.h> #include <linux/version.h> #include <linux/slab.h> #include <linux/smpboot.h> #include <linux/pagemap.h> #include <linux/notifier.h> #include <linux/syscalls.h> #include <linux/namei.h> #include <asm/atomic.h> #include <asm/bitops.h> static unsigned long long a = 0, b = 0; static struct hlist_bl_head bl_head; struct task_struct *Thread1; struct task_struct *Thread2; struct task_struct *Thread3; struct task_struct *Thread4; struct task_struct *Thread5; struct task_struct *Thread6; int increase_ab(void *arg); int increase_ab(void *arg) { while (1) { hlist_bl_lock(&bl_head); if (a != b) { pr_err(">>> a = %llu, b = %llu \n", a, b); BUG(); return -1; } if (a > (ULLONG_MAX - 4096)) { a = 0; b = 0; } a++; b++; hlist_bl_unlock(&bl_head); schedule(); } return 0; } static int mymod_init(void) { INIT_HLIST_BL_HEAD(&bl_head); Thread1 = kthread_create(increase_ab, NULL, "bl_lock_thread1"); wake_up_process(Thread1); Thread2 = kthread_create(increase_ab, NULL, "bl_lock_thread2"); wake_up_process(Thread2); Thread3 = kthread_create(increase_ab, NULL, "bl_lock_thread3"); wake_up_process(Thread3); Thread4 = kthread_create(increase_ab, NULL, "bl_lock_thread4"); wake_up_process(Thread4); Thread5 = kthread_create(increase_ab, NULL, "bl_lock_thread5"); wake_up_process(Thread5); Thread6 = kthread_create(increase_ab, NULL, "bl_lock_thread6"); wake_up_process(Thread6); return 0; } static void mymod_exit(void) { if (Thread1) kthread_stop(Thread1); if (Thread2) kthread_stop(Thread2); if (Thread3) kthread_stop(Thread3); if (Thread4) kthread_stop(Thread4); if (Thread5) kthread_stop(Thread5); if (Thread6) kthread_stop(Thread6); } module_init(mymod_init); module_exit(mymod_exit); MODULE_LICENSE("Dual BSD/GPL"); ``` After 9257959a6e5b ("locking/atomic: scripts: restructure fallback ifdeffery") is merged in, we can see the problem when inserting the ko: ``` [root@localhost ~]# insmod mymod.ko [ 37.994787][ T621] >>> a = 725, b = 724 [ 37.995313][ T621] ------------[ cut here ]------------ [ 37.995951][ T621] kernel BUG at fs/mymod/mymod.c:42! [r[ oo 3t7@.l996o4c61al]h[o s T6t21] ~ ]#Int ernal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 37.997420][ T621] Modules linked in: mymod(E) [ 37.997891][ T621] CPU: 9 PID: 621 Comm: bl_lock_thread2 Tainted: G E 6.4.0-rc2-00034-g9257959a6e5b-dirty #117 [ 37.999038][ T621] Hardware name: linux,dummy-virt (DT) [ 37.999571][ T621] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 38.000344][ T621] pc : increase_ab+0xcc/0xe70 [mymod] [ 38.000882][ T621] lr : increase_ab+0xcc/0xe70 [mymod] [ 38.001416][ T621] sp : ffff800008b4be40 [ 38.001822][ T621] x29: ffff800008b4be40 x28: 0000000000000000 x27: 0000000000000000 [ 38.002605][ T621] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 [ 38.003385][ T621] x23: ffffd9930c698190 x22: ffff800008a0ba38 x21: 0000000000000001 [ 38.004174][ T621] x20: ffffffffffffefff x19: ffffd9930c69a580 x18: 0000000000000000 [ 38.004955][ T621] x17: 0000000000000000 x16: ffffd9933011bd38 x15: ffffffffffffffff [ 38.005754][ T621] x14: 0000000000000000 x13: 205d313236542020 x12: ffffd99332175b80 [ 38.006538][ T621] x11: 0000000000000003 x10: 0000000000000001 x9 : ffffd9933022a9d8 [ 38.007325][ T621] x8 : 00000000000bffe8 x7 : c0000000ffff7fff x6 : ffffd993320b5b40 [ 38.008124][ T621] x5 : ffff0001f7d1c708 x4 : 0000000000000000 x3 : 0000000000000000 [ 38.008912][ T621] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000015 [ 38.009709][ T621] Call trace: [ 38.010035][ T621] increase_ab+0xcc/0xe70 [mymod] [ 38.010539][ T621] kthread+0xdc/0xf0 [ 38.010927][ T621] ret_from_fork+0x10/0x20 [ 38.011370][ T621] Code: 17ffffe0 90000020 91044000 9400000d (d4210000) [ 38.012067][ T621] ---[ end trace 0000000000000000 ]--- [ 38.012603][ T621] Kernel panic - not syncing: Oops - BUG: Fatal exception [ 38.013311][ T621] SMP: stopping secondary CPUs [ 38.013818][ T621] Kernel Offset: 0x599328000000 from 0xffff800008000000 [ 38.014508][ T621] PHYS_OFFSET: 0x40000000 [ 38.014933][ T621] CPU features: 0x000000,0220080c,44016203 [ 38.015510][ T621] Memory Limit: none [ 38.015950][ T621] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception ]--- ``` ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-16 6:55 ` Baokun Li @ 2023-09-17 9:10 ` Peter Zijlstra 2023-09-17 9:26 ` Peter Zijlstra 2023-09-18 1:10 ` Baokun Li 2023-09-19 15:10 ` Mark Rutland 1 sibling, 2 replies; 14+ messages in thread From: Peter Zijlstra @ 2023-09-17 9:10 UTC (permalink / raw) To: Baokun Li Cc: Yi Zhang, Ming Lei, mark.rutland, Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), Kees Cook, chengzhihao On Sat, Sep 16, 2023 at 02:55:47PM +0800, Baokun Li wrote: > On 2023/9/13 16:59, Yi Zhang wrote: > > The issue still can be reproduced on the latest linux tree[2]. > > To reproduce I need to run about 1000 times blktests block/001, and > > bisect shows it was introduced with commit[1], as it was not 100% > > reproduced, not sure if it's the culprit? > > > > > > [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery > Hello, everyone！ > > We have confirmed that the merge-in of this patch caused hlist_bl_lock > (aka, bit_spin_lock) to fail, which in turn triggered the issue above. > [root@localhost ~]# insmod mymod.ko > [ 37.994787][ T621] >>> a = 725, b = 724 > [ 37.995313][ T621] ------------[ cut here ]------------ > [ 37.995951][ T621] kernel BUG at fs/mymod/mymod.c:42! > [r[ oo 3t7@.l996o4c61al]h[o s T6t21] ~ ]#Int ernal error: Oops - BUG: > 00000000f2000800 [#1] SMP > [ 37.997420][ T621] Modules linked in: mymod(E) > [ 37.997891][ T621] CPU: 9 PID: 621 Comm: bl_lock_thread2 Tainted: > G E 6.4.0-rc2-00034-g9257959a6e5b-dirty #117 > [ 37.999038][ T621] Hardware name: linux,dummy-virt (DT) > [ 37.999571][ T621] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS > BTYPE=--) > [ 38.000344][ T621] pc : increase_ab+0xcc/0xe70 [mymod] > [ 38.000882][ T621] lr : increase_ab+0xcc/0xe70 [mymod] > [ 38.001416][ T621] sp : ffff800008b4be40 > [ 38.001822][ T621] x29: ffff800008b4be40 x28: 0000000000000000 x27: > 0000000000000000 > [ 38.002605][ T621] x26: 0000000000000000 x25: 0000000000000000 x24: > 0000000000000000 > [ 38.003385][ T621] x23: ffffd9930c698190 x22: ffff800008a0ba38 x21: > 0000000000000001 > [ 38.004174][ T621] x20: ffffffffffffefff x19: ffffd9930c69a580 x18: > 0000000000000000 > [ 38.004955][ T621] x17: 0000000000000000 x16: ffffd9933011bd38 x15: > ffffffffffffffff > [ 38.005754][ T621] x14: 0000000000000000 x13: 205d313236542020 x12: > ffffd99332175b80 > [ 38.006538][ T621] x11: 0000000000000003 x10: 0000000000000001 x9 : > ffffd9933022a9d8 > [ 38.007325][ T621] x8 : 00000000000bffe8 x7 : c0000000ffff7fff x6 : > ffffd993320b5b40 > [ 38.008124][ T621] x5 : ffff0001f7d1c708 x4 : 0000000000000000 x3 : > 0000000000000000 > [ 38.008912][ T621] x2 : 0000000000000000 x1 : 0000000000000000 x0 : > 0000000000000015 > [ 38.009709][ T621] Call trace: > [ 38.010035][ T621] increase_ab+0xcc/0xe70 [mymod] > [ 38.010539][ T621] kthread+0xdc/0xf0 > [ 38.010927][ T621] ret_from_fork+0x10/0x20 > [ 38.011370][ T621] Code: 17ffffe0 90000020 91044000 9400000d (d4210000) > [ 38.012067][ T621] ---[ end trace 0000000000000000 ]--- Is this arm64 or something? You seem to have forgotten to mention what platform you're using. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-17 9:10 ` Peter Zijlstra @ 2023-09-17 9:26 ` Peter Zijlstra 2023-09-18 1:52 ` Baokun Li 2023-09-18 1:10 ` Baokun Li 1 sibling, 1 reply; 14+ messages in thread From: Peter Zijlstra @ 2023-09-17 9:26 UTC (permalink / raw) To: Baokun Li Cc: Yi Zhang, Ming Lei, mark.rutland, Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), Kees Cook, chengzhihao On Sun, Sep 17, 2023 at 11:10:32AM +0200, Peter Zijlstra wrote: > On Sat, Sep 16, 2023 at 02:55:47PM +0800, Baokun Li wrote: > > On 2023/9/13 16:59, Yi Zhang wrote: > > > The issue still can be reproduced on the latest linux tree[2]. > > > To reproduce I need to run about 1000 times blktests block/001, and > > > bisect shows it was introduced with commit[1], as it was not 100% > > > reproduced, not sure if it's the culprit? > > > > > > > > > [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery > > Hello, everyone！ > > > > We have confirmed that the merge-in of this patch caused hlist_bl_lock > > (aka, bit_spin_lock) to fail, which in turn triggered the issue above. > > > [root@localhost ~]# insmod mymod.ko > > [ 37.994787][ T621] >>> a = 725, b = 724 > > [ 37.995313][ T621] ------------[ cut here ]------------ > > [ 37.995951][ T621] kernel BUG at fs/mymod/mymod.c:42! > > [r[ oo 3t7@.l996o4c61al]h[o s T6t21] ~ ]#Int ernal error: Oops - BUG: > > 00000000f2000800 [#1] SMP > > [ 37.997420][ T621] Modules linked in: mymod(E) > > [ 37.997891][ T621] CPU: 9 PID: 621 Comm: bl_lock_thread2 Tainted: > > G E 6.4.0-rc2-00034-g9257959a6e5b-dirty #117 > > [ 37.999038][ T621] Hardware name: linux,dummy-virt (DT) > > [ 37.999571][ T621] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS > > BTYPE=--) > > [ 38.000344][ T621] pc : increase_ab+0xcc/0xe70 [mymod] > > [ 38.000882][ T621] lr : increase_ab+0xcc/0xe70 [mymod] > > [ 38.001416][ T621] sp : ffff800008b4be40 > > [ 38.001822][ T621] x29: ffff800008b4be40 x28: 0000000000000000 x27: > > 0000000000000000 > > [ 38.002605][ T621] x26: 0000000000000000 x25: 0000000000000000 x24: > > 0000000000000000 > > [ 38.003385][ T621] x23: ffffd9930c698190 x22: ffff800008a0ba38 x21: > > 0000000000000001 > > [ 38.004174][ T621] x20: ffffffffffffefff x19: ffffd9930c69a580 x18: > > 0000000000000000 > > [ 38.004955][ T621] x17: 0000000000000000 x16: ffffd9933011bd38 x15: > > ffffffffffffffff > > [ 38.005754][ T621] x14: 0000000000000000 x13: 205d313236542020 x12: > > ffffd99332175b80 > > [ 38.006538][ T621] x11: 0000000000000003 x10: 0000000000000001 x9 : > > ffffd9933022a9d8 > > [ 38.007325][ T621] x8 : 00000000000bffe8 x7 : c0000000ffff7fff x6 : > > ffffd993320b5b40 > > [ 38.008124][ T621] x5 : ffff0001f7d1c708 x4 : 0000000000000000 x3 : > > 0000000000000000 > > [ 38.008912][ T621] x2 : 0000000000000000 x1 : 0000000000000000 x0 : > > 0000000000000015 > > [ 38.009709][ T621] Call trace: > > [ 38.010035][ T621] increase_ab+0xcc/0xe70 [mymod] > > [ 38.010539][ T621] kthread+0xdc/0xf0 > > [ 38.010927][ T621] ret_from_fork+0x10/0x20 > > [ 38.011370][ T621] Code: 17ffffe0 90000020 91044000 9400000d (d4210000) > > [ 38.012067][ T621] ---[ end trace 0000000000000000 ]--- > > Is this arm64 or something? You seem to have forgotten to mention what > platform you're using. Is that an LSE or LLSC arm64 ? Anyway, it seems that ARM64 shouldn't be using the fallback as it does everything itself. Mark, can you have a look please? At first glance the atomic64_fetch_or_acquire() that's being used by generic bitops/lock.h seems in order.. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-17 9:26 ` Peter Zijlstra @ 2023-09-18 1:52 ` Baokun Li 2023-09-18 18:42 ` Darrick J. Wong 0 siblings, 1 reply; 14+ messages in thread From: Baokun Li @ 2023-09-18 1:52 UTC (permalink / raw) To: Peter Zijlstra Cc: Yi Zhang, Ming Lei, mark.rutland, Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), Kees Cook, chengzhihao, Baokun Li On 2023/9/17 17:26, Peter Zijlstra wrote: > On Sun, Sep 17, 2023 at 11:10:32AM +0200, Peter Zijlstra wrote: >> On Sat, Sep 16, 2023 at 02:55:47PM +0800, Baokun Li wrote: >>> On 2023/9/13 16:59, Yi Zhang wrote: >>>> The issue still can be reproduced on the latest linux tree[2]. >>>> To reproduce I need to run about 1000 times blktests block/001, and >>>> bisect shows it was introduced with commit[1], as it was not 100% >>>> reproduced, not sure if it's the culprit? >>>> >>>> >>>> [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery >>> Hello, everyone！ >>> >>> We have confirmed that the merge-in of this patch caused hlist_bl_lock >>> (aka, bit_spin_lock) to fail, which in turn triggered the issue above. >>> [root@localhost ~]# insmod mymod.ko >>> [ 37.994787][ T621] >>> a = 725, b = 724 >>> [ 37.995313][ T621] ------------[ cut here ]------------ >>> [ 37.995951][ T621] kernel BUG at fs/mymod/mymod.c:42! >>> [r[ oo 3t7@.l996o4c61al]h[o s T6t21] ~ ]#Int ernal error: Oops - BUG: >>> 00000000f2000800 [#1] SMP >>> [ 37.997420][ T621] Modules linked in: mymod(E) >>> [ 37.997891][ T621] CPU: 9 PID: 621 Comm: bl_lock_thread2 Tainted: >>> G E 6.4.0-rc2-00034-g9257959a6e5b-dirty #117 >>> [ 37.999038][ T621] Hardware name: linux,dummy-virt (DT) >>> [ 37.999571][ T621] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS >>> BTYPE=--) >>> [ 38.000344][ T621] pc : increase_ab+0xcc/0xe70 [mymod] >>> [ 38.000882][ T621] lr : increase_ab+0xcc/0xe70 [mymod] >>> [ 38.001416][ T621] sp : ffff800008b4be40 >>> [ 38.001822][ T621] x29: ffff800008b4be40 x28: 0000000000000000 x27: >>> 0000000000000000 >>> [ 38.002605][ T621] x26: 0000000000000000 x25: 0000000000000000 x24: >>> 0000000000000000 >>> [ 38.003385][ T621] x23: ffffd9930c698190 x22: ffff800008a0ba38 x21: >>> 0000000000000001 >>> [ 38.004174][ T621] x20: ffffffffffffefff x19: ffffd9930c69a580 x18: >>> 0000000000000000 >>> [ 38.004955][ T621] x17: 0000000000000000 x16: ffffd9933011bd38 x15: >>> ffffffffffffffff >>> [ 38.005754][ T621] x14: 0000000000000000 x13: 205d313236542020 x12: >>> ffffd99332175b80 >>> [ 38.006538][ T621] x11: 0000000000000003 x10: 0000000000000001 x9 : >>> ffffd9933022a9d8 >>> [ 38.007325][ T621] x8 : 00000000000bffe8 x7 : c0000000ffff7fff x6 : >>> ffffd993320b5b40 >>> [ 38.008124][ T621] x5 : ffff0001f7d1c708 x4 : 0000000000000000 x3 : >>> 0000000000000000 >>> [ 38.008912][ T621] x2 : 0000000000000000 x1 : 0000000000000000 x0 : >>> 0000000000000015 >>> [ 38.009709][ T621] Call trace: >>> [ 38.010035][ T621] increase_ab+0xcc/0xe70 [mymod] >>> [ 38.010539][ T621] kthread+0xdc/0xf0 >>> [ 38.010927][ T621] ret_from_fork+0x10/0x20 >>> [ 38.011370][ T621] Code: 17ffffe0 90000020 91044000 9400000d (d4210000) >>> [ 38.012067][ T621] ---[ end trace 0000000000000000 ]--- >> Is this arm64 or something? You seem to have forgotten to mention what >> platform you're using. > Is that an LSE or LLSC arm64 ? I'm not sure how to distinguish if it's LSE or LLSC, here's some info on the cpu: $ cat /sys/devices/system/cpu/cpu0/regs/identification/midr_el1 0x00000000481fd010 $ lscpu Architecture: aarch64 Byte Order: Little Endian CPU(s): 96 On-line CPU(s) list: 0-95 Thread(s) per core: 1 Core(s) per socket: 48 Socket(s): 2 NUMA node(s): 4 Vendor ID: HiSilicon BIOS Vendor ID: HiSilicon Model: 0 Model name: Kunpeng-920 BIOS Model name: Kunpeng 920-4826 Stepping: 0x1 BogoMIPS: 200.00 L1d cache: 64K L1i cache: 64K L2 cache: 512K L3 cache: 49152K NUMA node0 CPU(s): 0-23 NUMA node1 CPU(s): 24-47 NUMA node2 CPU(s): 48-71 NUMA node3 CPU(s): 72-95 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm > Anyway, it seems that ARM64 shouldn't be using the fallback as it does > everything itself. > > Mark, can you have a look please? At first glance the > atomic64_fetch_or_acquire() that's being used by generic bitops/lock.h > seems in order.. > We also suspect some implicit mechanism change in raw_atomic64_fetch_or_acquire. You can reproduce the problem with the above mod that can reproduce the problem to make it easier to locate. I can help reproduce it and grab some information if you can't reproduce it on your end. -- With Best Regards, Baokun Li . ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-18 1:52 ` Baokun Li @ 2023-09-18 18:42 ` Darrick J. Wong 0 siblings, 0 replies; 14+ messages in thread From: Darrick J. Wong @ 2023-09-18 18:42 UTC (permalink / raw) To: Baokun Li Cc: Peter Zijlstra, Yi Zhang, Ming Lei, mark.rutland, Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), Kees Cook, chengzhihao On Mon, Sep 18, 2023 at 09:52:28AM +0800, Baokun Li wrote: > On 2023/9/17 17:26, Peter Zijlstra wrote: > > On Sun, Sep 17, 2023 at 11:10:32AM +0200, Peter Zijlstra wrote: > > > On Sat, Sep 16, 2023 at 02:55:47PM +0800, Baokun Li wrote: > > > > On 2023/9/13 16:59, Yi Zhang wrote: > > > > > The issue still can be reproduced on the latest linux tree[2]. > > > > > To reproduce I need to run about 1000 times blktests block/001, and > > > > > bisect shows it was introduced with commit[1], as it was not 100% > > > > > reproduced, not sure if it's the culprit? > > > > > > > > > > > > > > > [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery > > > > Hello, everyone！ > > > > > > > > We have confirmed that the merge-in of this patch caused hlist_bl_lock > > > > (aka, bit_spin_lock) to fail, which in turn triggered the issue above. > > > > [root@localhost ~]# insmod mymod.ko > > > > [ 37.994787][ T621] >>> a = 725, b = 724 > > > > [ 37.995313][ T621] ------------[ cut here ]------------ > > > > [ 37.995951][ T621] kernel BUG at fs/mymod/mymod.c:42! > > > > [r[ oo 3t7@.l996o4c61al]h[o s T6t21] ~ ]#Int ernal error: Oops - BUG: > > > > 00000000f2000800 [#1] SMP > > > > [ 37.997420][ T621] Modules linked in: mymod(E) > > > > [ 37.997891][ T621] CPU: 9 PID: 621 Comm: bl_lock_thread2 Tainted: > > > > G E 6.4.0-rc2-00034-g9257959a6e5b-dirty #117 > > > > [ 37.999038][ T621] Hardware name: linux,dummy-virt (DT) > > > > [ 37.999571][ T621] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS > > > > BTYPE=--) > > > > [ 38.000344][ T621] pc : increase_ab+0xcc/0xe70 [mymod] > > > > [ 38.000882][ T621] lr : increase_ab+0xcc/0xe70 [mymod] > > > > [ 38.001416][ T621] sp : ffff800008b4be40 > > > > [ 38.001822][ T621] x29: ffff800008b4be40 x28: 0000000000000000 x27: > > > > 0000000000000000 > > > > [ 38.002605][ T621] x26: 0000000000000000 x25: 0000000000000000 x24: > > > > 0000000000000000 > > > > [ 38.003385][ T621] x23: ffffd9930c698190 x22: ffff800008a0ba38 x21: > > > > 0000000000000001 > > > > [ 38.004174][ T621] x20: ffffffffffffefff x19: ffffd9930c69a580 x18: > > > > 0000000000000000 > > > > [ 38.004955][ T621] x17: 0000000000000000 x16: ffffd9933011bd38 x15: > > > > ffffffffffffffff > > > > [ 38.005754][ T621] x14: 0000000000000000 x13: 205d313236542020 x12: > > > > ffffd99332175b80 > > > > [ 38.006538][ T621] x11: 0000000000000003 x10: 0000000000000001 x9 : > > > > ffffd9933022a9d8 > > > > [ 38.007325][ T621] x8 : 00000000000bffe8 x7 : c0000000ffff7fff x6 : > > > > ffffd993320b5b40 > > > > [ 38.008124][ T621] x5 : ffff0001f7d1c708 x4 : 0000000000000000 x3 : > > > > 0000000000000000 > > > > [ 38.008912][ T621] x2 : 0000000000000000 x1 : 0000000000000000 x0 : > > > > 0000000000000015 > > > > [ 38.009709][ T621] Call trace: > > > > [ 38.010035][ T621] increase_ab+0xcc/0xe70 [mymod] > > > > [ 38.010539][ T621] kthread+0xdc/0xf0 > > > > [ 38.010927][ T621] ret_from_fork+0x10/0x20 > > > > [ 38.011370][ T621] Code: 17ffffe0 90000020 91044000 9400000d (d4210000) > > > > [ 38.012067][ T621] ---[ end trace 0000000000000000 ]--- > > > Is this arm64 or something? You seem to have forgotten to mention what > > > platform you're using. > > Is that an LSE or LLSC arm64 ? > > I'm not sure how to distinguish if it's LSE or LLSC, here's some info on the > cpu: > > $ cat /sys/devices/system/cpu/cpu0/regs/identification/midr_el1 > 0x00000000481fd010 > > $ lscpu > Architecture: aarch64 > Byte Order: Little Endian > CPU(s): 96 > On-line CPU(s) list: 0-95 > Thread(s) per core: 1 > Core(s) per socket: 48 > Socket(s): 2 > NUMA node(s): 4 > Vendor ID: HiSilicon > BIOS Vendor ID: HiSilicon > Model: 0 > Model name: Kunpeng-920 > BIOS Model name: Kunpeng 920-4826 > Stepping: 0x1 > BogoMIPS: 200.00 > L1d cache: 64K > L1i cache: 64K > L2 cache: 512K > L3 cache: 49152K > NUMA node0 CPU(s): 0-23 > NUMA node1 CPU(s): 24-47 > NUMA node2 CPU(s): 48-71 > NUMA node3 CPU(s): 72-95 > Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp > asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm > > > Anyway, it seems that ARM64 shouldn't be using the fallback as it does > > everything itself. > > > > Mark, can you have a look please? At first glance the > > atomic64_fetch_or_acquire() that's being used by generic bitops/lock.h > > seems in order.. > > > We also suspect some implicit mechanism change in > raw_atomic64_fetch_or_acquire. You can reproduce the problem with the > above mod that can reproduce the problem to make it easier to locate. > I can help reproduce it and grab some information if you can't reproduce > it on your end. FWIW this looks a lot like the crash I reported last week: https://lore.kernel.org/linux-fsdevel/ZQep0OR0uMmR%2Fwg3@dread.disaster.area/T/#t Also arm64, but virtualized. I /think/ the host is some Ampere box, though I have no idea what kind since it's just some Oracle Cloud A1 instance. The internet claims "Ampere Altra" processors[1]. # lscpu Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: ARM Model name: Neoverse-N1 Model: 1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: r3p1 BogoMIPS: 50.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0,1 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Vulnerable Spectre v1: Mitigation; __user pointer sanitization Spectre v2: Mitigation; CSV2, but not BHB Srbds: Not affected Tsx async abort: Not affected [1] https://www.oracle.com/cloud/compute/arm/ --D > -- > With Best Regards, > Baokun Li > . ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-17 9:10 ` Peter Zijlstra 2023-09-17 9:26 ` Peter Zijlstra @ 2023-09-18 1:10 ` Baokun Li 2023-09-18 10:20 ` Yi Zhang 1 sibling, 1 reply; 14+ messages in thread From: Baokun Li @ 2023-09-18 1:10 UTC (permalink / raw) To: Peter Zijlstra Cc: Yi Zhang, Ming Lei, mark.rutland, Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), Kees Cook, chengzhihao, Baokun Li On 2023/9/17 17:10, Peter Zijlstra wrote: > On Sat, Sep 16, 2023 at 02:55:47PM +0800, Baokun Li wrote: >> On 2023/9/13 16:59, Yi Zhang wrote: >>> The issue still can be reproduced on the latest linux tree[2]. >>> To reproduce I need to run about 1000 times blktests block/001, and >>> bisect shows it was introduced with commit[1], as it was not 100% >>> reproduced, not sure if it's the culprit? >>> >>> >>> [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery >> Hello, everyone！ >> >> We have confirmed that the merge-in of this patch caused hlist_bl_lock >> (aka, bit_spin_lock) to fail, which in turn triggered the issue above. >> [root@localhost ~]# insmod mymod.ko >> [ 37.994787][ T621] >>> a = 725, b = 724 >> [ 37.995313][ T621] ------------[ cut here ]------------ >> [ 37.995951][ T621] kernel BUG at fs/mymod/mymod.c:42! >> [r[ oo 3t7@.l996o4c61al]h[o s T6t21] ~ ]#Int ernal error: Oops - BUG: >> 00000000f2000800 [#1] SMP >> [ 37.997420][ T621] Modules linked in: mymod(E) >> [ 37.997891][ T621] CPU: 9 PID: 621 Comm: bl_lock_thread2 Tainted: >> G E 6.4.0-rc2-00034-g9257959a6e5b-dirty #117 >> [ 37.999038][ T621] Hardware name: linux,dummy-virt (DT) >> [ 37.999571][ T621] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS >> BTYPE=--) >> [ 38.000344][ T621] pc : increase_ab+0xcc/0xe70 [mymod] >> [ 38.000882][ T621] lr : increase_ab+0xcc/0xe70 [mymod] >> [ 38.001416][ T621] sp : ffff800008b4be40 >> [ 38.001822][ T621] x29: ffff800008b4be40 x28: 0000000000000000 x27: >> 0000000000000000 >> [ 38.002605][ T621] x26: 0000000000000000 x25: 0000000000000000 x24: >> 0000000000000000 >> [ 38.003385][ T621] x23: ffffd9930c698190 x22: ffff800008a0ba38 x21: >> 0000000000000001 >> [ 38.004174][ T621] x20: ffffffffffffefff x19: ffffd9930c69a580 x18: >> 0000000000000000 >> [ 38.004955][ T621] x17: 0000000000000000 x16: ffffd9933011bd38 x15: >> ffffffffffffffff >> [ 38.005754][ T621] x14: 0000000000000000 x13: 205d313236542020 x12: >> ffffd99332175b80 >> [ 38.006538][ T621] x11: 0000000000000003 x10: 0000000000000001 x9 : >> ffffd9933022a9d8 >> [ 38.007325][ T621] x8 : 00000000000bffe8 x7 : c0000000ffff7fff x6 : >> ffffd993320b5b40 >> [ 38.008124][ T621] x5 : ffff0001f7d1c708 x4 : 0000000000000000 x3 : >> 0000000000000000 >> [ 38.008912][ T621] x2 : 0000000000000000 x1 : 0000000000000000 x0 : >> 0000000000000015 >> [ 38.009709][ T621] Call trace: >> [ 38.010035][ T621] increase_ab+0xcc/0xe70 [mymod] >> [ 38.010539][ T621] kthread+0xdc/0xf0 >> [ 38.010927][ T621] ret_from_fork+0x10/0x20 >> [ 38.011370][ T621] Code: 17ffffe0 90000020 91044000 9400000d (d4210000) >> [ 38.012067][ T621] ---[ end trace 0000000000000000 ]--- > Is this arm64 or something? You seem to have forgotten to mention what > platform you're using. > Sorry for the late reply. We tested both x86 and arm64, and the problem is only encountered under arm64. -- With Best Regards, Baokun Li . ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-18 1:10 ` Baokun Li @ 2023-09-18 10:20 ` Yi Zhang 0 siblings, 0 replies; 14+ messages in thread From: Yi Zhang @ 2023-09-18 10:20 UTC (permalink / raw) To: Baokun Li Cc: Peter Zijlstra, Ming Lei, mark.rutland, Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), Kees Cook, chengzhihao On Mon, Sep 18, 2023 at 9:10 AM Baokun Li <libaokun1@huawei.com> wrote: > > On 2023/9/17 17:10, Peter Zijlstra wrote: > > On Sat, Sep 16, 2023 at 02:55:47PM +0800, Baokun Li wrote: > >> On 2023/9/13 16:59, Yi Zhang wrote: > >>> The issue still can be reproduced on the latest linux tree[2]. > >>> To reproduce I need to run about 1000 times blktests block/001, and > >>> bisect shows it was introduced with commit[1], as it was not 100% > >>> reproduced, not sure if it's the culprit? > >>> > >>> > >>> [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery > >> Hello, everyone！ > >> > >> We have confirmed that the merge-in of this patch caused hlist_bl_lock > >> (aka, bit_spin_lock) to fail, which in turn triggered the issue above. > >> [root@localhost ~]# insmod mymod.ko > >> [ 37.994787][ T621] >>> a = 725, b = 724 > >> [ 37.995313][ T621] ------------[ cut here ]------------ > >> [ 37.995951][ T621] kernel BUG at fs/mymod/mymod.c:42! > >> [r[ oo 3t7@.l996o4c61al]h[o s T6t21] ~ ]#Int ernal error: Oops - BUG: > >> 00000000f2000800 [#1] SMP > >> [ 37.997420][ T621] Modules linked in: mymod(E) > >> [ 37.997891][ T621] CPU: 9 PID: 621 Comm: bl_lock_thread2 Tainted: > >> G E 6.4.0-rc2-00034-g9257959a6e5b-dirty #117 > >> [ 37.999038][ T621] Hardware name: linux,dummy-virt (DT) > >> [ 37.999571][ T621] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS > >> BTYPE=--) > >> [ 38.000344][ T621] pc : increase_ab+0xcc/0xe70 [mymod] > >> [ 38.000882][ T621] lr : increase_ab+0xcc/0xe70 [mymod] > >> [ 38.001416][ T621] sp : ffff800008b4be40 > >> [ 38.001822][ T621] x29: ffff800008b4be40 x28: 0000000000000000 x27: > >> 0000000000000000 > >> [ 38.002605][ T621] x26: 0000000000000000 x25: 0000000000000000 x24: > >> 0000000000000000 > >> [ 38.003385][ T621] x23: ffffd9930c698190 x22: ffff800008a0ba38 x21: > >> 0000000000000001 > >> [ 38.004174][ T621] x20: ffffffffffffefff x19: ffffd9930c69a580 x18: > >> 0000000000000000 > >> [ 38.004955][ T621] x17: 0000000000000000 x16: ffffd9933011bd38 x15: > >> ffffffffffffffff > >> [ 38.005754][ T621] x14: 0000000000000000 x13: 205d313236542020 x12: > >> ffffd99332175b80 > >> [ 38.006538][ T621] x11: 0000000000000003 x10: 0000000000000001 x9 : > >> ffffd9933022a9d8 > >> [ 38.007325][ T621] x8 : 00000000000bffe8 x7 : c0000000ffff7fff x6 : > >> ffffd993320b5b40 > >> [ 38.008124][ T621] x5 : ffff0001f7d1c708 x4 : 0000000000000000 x3 : > >> 0000000000000000 > >> [ 38.008912][ T621] x2 : 0000000000000000 x1 : 0000000000000000 x0 : > >> 0000000000000015 > >> [ 38.009709][ T621] Call trace: > >> [ 38.010035][ T621] increase_ab+0xcc/0xe70 [mymod] > >> [ 38.010539][ T621] kthread+0xdc/0xf0 > >> [ 38.010927][ T621] ret_from_fork+0x10/0x20 > >> [ 38.011370][ T621] Code: 17ffffe0 90000020 91044000 9400000d (d4210000) > >> [ 38.012067][ T621] ---[ end trace 0000000000000000 ]--- > > Is this arm64 or something? You seem to have forgotten to mention what > > platform you're using. > > > Sorry for the late reply. > We tested both x86 and arm64, and the problem is only encountered under > arm64. Yeah, my reproduced environment is also aarch64. > > -- > With Best Regards, > Baokun Li > . > -- Best Regards, Yi Zhang ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-16 6:55 ` Baokun Li 2023-09-17 9:10 ` Peter Zijlstra @ 2023-09-19 15:10 ` Mark Rutland 1 sibling, 0 replies; 14+ messages in thread From: Mark Rutland @ 2023-09-19 15:10 UTC (permalink / raw) To: Baokun Li Cc: Yi Zhang, Ming Lei, Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, yangerkun, zhangyi (F), peterz, Kees Cook, chengzhihao, Will Deacon On Sat, Sep 16, 2023 at 02:55:47PM +0800, Baokun Li wrote: > On 2023/9/13 16:59, Yi Zhang wrote: > > The issue still can be reproduced on the latest linux tree[2]. > > To reproduce I need to run about 1000 times blktests block/001, and > > bisect shows it was introduced with commit[1], as it was not 100% > > reproduced, not sure if it's the culprit? > > > > > > [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery > Hello, everyone！ > > We have confirmed that the merge-in of this patch caused hlist_bl_lock > (aka, bit_spin_lock) to fail, which in turn triggered the issue above. Thanks for this! I believe I know what the issue is. I took a look at the generated assembly for hlist_bl_lock() and hlist_bl_unlock(), and for the latter I see a plain store rather than a store-release as was intended. I believe that in 9257959a6e5b, I messed up the fallback logic for atomic*_set_release(): | static __always_inline void | raw_atomic64_set_release(atomic64_t *v, s64 i) | { | #if defined(arch_atomic64_set_release) | arch_atomic64_set_release(v, i); | #elif defined(arch_atomic64_set) | arch_atomic64_set(v, i); | #else | if (__native_word(atomic64_t)) { | smp_store_release(&(v)->counter, i); | } else { | __atomic_release_fence(); | raw_atomic64_set(v, i); | } | #endif | } On arm64 we want to use smp_store_release(), and don't provide arch_atomic64_set_release(). Unfortunately we *do* provide arch_atomic64_set(), and the ifdeffery above will choose that in preference. Prior to that commit, the ifdeffery would do what we want: | #ifndef arch_atomic64_set_release | static __always_inline void | arch_atomic64_set_release(atomic64_t *v, s64 i) | { | if (__native_word(atomic64_t)) { | smp_store_release(&(v)->counter, i); | } else { | __atomic_release_fence(); | arch_atomic64_set(v, i); | } | } | #define arch_atomic64_set_release arch_atomic64_set_release | #endif That explains the lock going wrong -- we lose the RELEASE semantic on hlist_bl_unlock(), and so loads and stores within the critical section aren't guaranteed to be visible to the next hlist_bl_lock(). On x86 this happens to work becauase of TSO. I'm working on fixing that now; I'll try to have a patch shortly. Thanks, Mark. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-13 8:59 ` Yi Zhang 2023-09-16 6:55 ` Baokun Li @ 2023-09-17 0:35 ` Bagas Sanjaya 2023-09-29 13:24 ` Linux regression tracking #update (Thorsten Leemhuis) 1 sibling, 1 reply; 14+ messages in thread From: Bagas Sanjaya @ 2023-09-17 0:35 UTC (permalink / raw) To: Yi Zhang, Ming Lei Cc: Christian Brauner, linux-fsdevel, Alexander Viro, linux-kernel, linux-scsi, Changhui Zhong, Mark Rutland, Peter Zijlstra, Kees Cook, Linux Regressions [-- Attachment #1: Type: text/plain, Size: 7266 bytes --] On Wed, Sep 13, 2023 at 04:59:31PM +0800, Yi Zhang wrote: > The issue still can be reproduced on the latest linux tree[2]. > To reproduce I need to run about 1000 times blktests block/001, and > bisect shows it was introduced with commit[1], as it was not 100% > reproduced, not sure if it's the culprit? > > > [1] 9257959a6e5b locking/atomic: scripts: restructure fallback ifdeffery > [2] > [ 2304.536339] scsi 48:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2304.540805] sr 50:0:0:0: Attached scsi CD-ROM sr3 > [ 2304.544574] scsi 48:0:0:0: Power-on or device reset occurred > [ 2304.600645] sr 48:0:0:0: [sr1] scsi-1 drive > [ 2304.616364] scsi 51:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2304.624639] scsi 51:0:0:0: Power-on or device reset occurred > [ 2304.626634] sr 48:0:0:0: Attached scsi CD-ROM sr1 > [ 2304.680537] sr 51:0:0:0: [sr2] scsi-1 drive > [ 2304.706394] sr 51:0:0:0: Attached scsi CD-ROM sr2 > [ 2304.746329] scsi 49:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2304.754569] scsi 49:0:0:0: Power-on or device reset occurred > [ 2304.756302] scsi 50:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2304.768483] scsi 50:0:0:0: Power-on or device reset occurred > [ 2304.806321] scsi 48:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2304.810587] sr 49:0:0:0: [sr0] scsi-1 drive > [ 2304.814561] scsi 48:0:0:0: Power-on or device reset occurred > [ 2304.824475] sr 50:0:0:0: [sr3] scsi-1 drive > [ 2304.836384] scsi 51:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2304.840364] sr 49:0:0:0: Attached scsi CD-ROM sr0 > [ 2304.844619] scsi 51:0:0:0: Power-on or device reset occurred > [ 2304.850444] sr 50:0:0:0: Attached scsi CD-ROM sr3 > [ 2304.874563] sr 48:0:0:0: [sr1] scsi-1 drive > [ 2304.900660] sr 51:0:0:0: [sr2] scsi-1 drive > [ 2304.901506] sr 48:0:0:0: Attached scsi CD-ROM sr1 > [ 2304.926306] sr 51:0:0:0: Attached scsi CD-ROM sr2 > [ 2305.056432] scsi 50:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2305.056572] scsi 49:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2305.064635] scsi 50:0:0:0: Power-on or device reset occurred > [ 2305.072821] scsi 49:0:0:0: Power-on or device reset occurred > [ 2305.086286] scsi 51:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2305.086357] scsi 48:0:0:0: CD-ROM Linux scsi_debug > 0191 PQ: 0 ANSI: 7 > [ 2305.094521] scsi 51:0:0:0: Power-on or device reset occurred > [ 2305.102693] scsi 48:0:0:0: Power-on or device reset occurred > [ 2305.128785] sr 50:0:0:0: [sr0] scsi-1 drive > [ 2305.134445] sr 49:0:0:0: [sr1] scsi-1 drive > [ 2305.154728] sr 50:0:0:0: Attached scsi CD-ROM sr0 > [ 2305.158607] sr 51:0:0:0: [sr2] scsi-1 drive > [ 2305.160392] sr 49:0:0:0: Attached scsi CD-ROM sr1 > [ 2305.164254] sr 48:0:0:0: [sr3] scsi-1 drive > [ 2305.184185] sr 51:0:0:0: Attached scsi CD-ROM sr2 > [ 2305.190086] sr 48:0:0:0: Attached scsi CD-ROM sr3 > [ 2305.555658] Unable to handle kernel execute from non-executable > memory at virtual address ffffc61b656052e8 > [ 2305.565301] Mem abort info: > [ 2305.568086] ESR = 0x000000008600000e > [ 2305.571822] EC = 0x21: IABT (current EL), IL = 32 bits > [ 2305.577123] SET = 0, FnV = 0 > [ 2305.580164] EA = 0, S1PTW = 0 > [ 2305.583292] FSC = 0x0e: level 2 permission fault > [ 2305.588074] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000080731fa0000 > [ 2305.594761] [ffffc61b656052e8] pgd=1000080ffffff003, > p4d=1000080ffffff003, pud=1000080fffffe003, pmd=0068080732e00f01 > [ 2305.605362] Internal error: Oops: 000000008600000e [#1] SMP > [ 2305.610922] Modules linked in: scsi_debug sr_mod pktcdvd cdrom > rfkill sunrpc vfat fat acpi_ipmi arm_spe_pmu ipmi_ssif ipmi_devintf > ipmi_msghandler arm_cmn arm_dmc620_pmu arm_dsu_pmu cppc_cpufreq loop > fuse zram xfs crct10dif_ce ghash_ce nvme sha2_ce nvme_core > sha256_arm64 igb sha1_ce ast sbsa_gwdt nvme_common > i2c_designware_platform i2c_algo_bit i2c_designware_core xgene_hwmon > dm_mod [last unloaded: scsi_debug] > [ 2305.647236] CPU: 85 PID: 1 Comm: systemd Kdump: loaded Not tainted > 6.6.0-rc1+ #13 > [ 2305.654706] Hardware name: GIGABYTE R152-P31-00/MP32-AR1-00, BIOS > F31n (SCP: 2.10.20220810) 09/30/2022 > [ 2305.663997] pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > [ 2305.670946] pc : in_lookup_hashtable+0x1138/0x2000 > [ 2305.675728] lr : rcu_do_batch+0x194/0x488 > [ 2305.679727] sp : ffff8000802abe60 > [ 2305.683029] x29: ffff8000802abe60 x28: ffffc61b6524c7c0 x27: ffffc61b63452f40 > [ 2305.690152] x26: ffff080f37ab6438 x25: 000000000000000a x24: 0000000000000000 > [ 2305.697274] x23: 0000000000000002 x22: ffff8000802abec0 x21: ffff080f37ab63c0 > [ 2305.704396] x20: ffff07ff8136a580 x19: 0000000000000003 x18: 0000000000000000 > [ 2305.711519] x17: ffff41f3d3161000 x16: ffff8000802a8000 x15: 0000000000000000 > [ 2305.718641] x14: 0000000000000000 x13: ffff07ffa131802d x12: ffff80008041bb94 > [ 2305.725764] x11: 0000000000000040 x10: ffff07ff802622e8 x9 : ffffc61b63452e30 > [ 2305.732887] x8 : 000002189dce1780 x7 : ffff07ff8d5c1000 x6 : ffff41f3d3161000 > [ 2305.740009] x5 : ffff07ff8136a580 x4 : ffff080f37aba960 x3 : 000000001550a055 > [ 2305.747131] x2 : 0000000000000000 x1 : ffffc61b656052e8 x0 : ffff080184c565f0 > [ 2305.754254] Call trace: > [ 2305.756687] in_lookup_hashtable+0x1138/0x2000 > [ 2305.761119] rcu_core+0x268/0x350 > [ 2305.764422] rcu_core_si+0x18/0x30 > [ 2305.767812] __do_softirq+0x120/0x3d4 > [ 2305.771462] ____do_softirq+0x18/0x30 > [ 2305.775112] call_on_irq_stack+0x24/0x30 > [ 2305.779022] do_softirq_own_stack+0x24/0x38 > [ 2305.783192] __irq_exit_rcu+0xfc/0x130 > [ 2305.786929] irq_exit_rcu+0x18/0x30 > [ 2305.790404] el1_interrupt+0x4c/0xe8 > [ 2305.793969] el1h_64_irq_handler+0x18/0x28 > [ 2305.798052] el1h_64_irq+0x78/0x80 > [ 2305.801441] d_same_name+0x50/0xd0 > [ 2305.804832] __lookup_slow+0x64/0x158 > [ 2305.808482] walk_component+0xe0/0x1a0 > [ 2305.812219] path_lookupat+0x7c/0x1b8 > [ 2305.815869] filename_lookup+0xb4/0x1b8 > [ 2305.819692] vfs_statx+0x94/0x1a8 > [ 2305.822995] vfs_fstatat+0xd4/0x110 > [ 2305.826471] __do_sys_newfstatat+0x58/0xa8 > [ 2305.830556] __arm64_sys_newfstatat+0x28/0x40 > [ 2305.834901] invoke_syscall.constprop.0+0x80/0xd8 > [ 2305.839592] do_el0_svc+0x48/0xd0 > [ 2305.842894] el0_svc+0x4c/0x1c0 > [ 2305.846023] el0t_64_sync_handler+0x120/0x130 > [ 2305.850367] el0t_64_sync+0x1a4/0x1a8 > [ 2305.854017] Code: 00000000 00000000 00000000 00000000 (84c565f1) > [ 2305.860098] SMP: stopping secondary CPUs > [ 2305.865048] Starting crashdump kernel... > [ 2305.868958] Bye! > > Please don't top-post; reply inline with appropriate context instead. Anyway, thanks for bisecting this regression. I'm adding it to regzbot: #regzbot ^introduced: 9257959a6e5b4f #regzbot title: restructuring atomic locking conditionals causes vfs dentry lock protection failure -- An old man doll... just what I always wanted! - Clara [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] 2023-09-17 0:35 ` Bagas Sanjaya @ 2023-09-29 13:24 ` Linux regression tracking #update (Thorsten Leemhuis) 0 siblings, 0 replies; 14+ messages in thread From: Linux regression tracking #update (Thorsten Leemhuis) @ 2023-09-29 13:24 UTC (permalink / raw) To: Linux Regressions; +Cc: linux-fsdevel, linux-kernel, linux-scsi On 17.09.23 02:35, Bagas Sanjaya wrote: > On Wed, Sep 13, 2023 at 04:59:31PM +0800, Yi Zhang wrote: >> The issue still can be reproduced on the latest linux tree[2]. >> To reproduce I need to run about 1000 times blktests block/001, and >> bisect shows it was introduced with commit[1], as it was not 100% >> reproduced, not sure if it's the culprit? > > Anyway, thanks for bisecting this regression. I'm adding it to regzbot: > > #regzbot ^introduced: 9257959a6e5b4f > #regzbot title: restructuring atomic locking conditionals causes vfs dentry lock protection failure #regzbot fix: 6d2779ecaeb56f #regzbot ignore-activity Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you. ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-09-29 13:24 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-08-23 4:06 [czhong@redhat.com: [bug report] WARNING: CPU: 121 PID: 93233 at fs/dcache.c:365 __dentry_kill+0x214/0x278] Ming Lei 2023-08-23 8:47 ` Christian Brauner 2023-08-28 10:43 ` Ming Lei 2023-09-13 8:59 ` Yi Zhang 2023-09-16 6:55 ` Baokun Li 2023-09-17 9:10 ` Peter Zijlstra 2023-09-17 9:26 ` Peter Zijlstra 2023-09-18 1:52 ` Baokun Li 2023-09-18 18:42 ` Darrick J. Wong 2023-09-18 1:10 ` Baokun Li 2023-09-18 10:20 ` Yi Zhang 2023-09-19 15:10 ` Mark Rutland 2023-09-17 0:35 ` Bagas Sanjaya 2023-09-29 13:24 ` Linux regression tracking #update (Thorsten Leemhuis)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).