[next-20250212] FStests generic/451 on EXT4 FS resulting in kernel OOPs

[next-20250212] FStests generic/451 on EXT4 FS resulting in kernel OOPs

[Venkat Rao Bagalkote]

Greetings!!!


I am observing kernel OOPs, while running FStests generic/451 on EXT4 
with linux-next kernel(next-20250212) on IBM Power Servers.


Issue is seen on: next-20250212

Issue not seen on: next-20250210


Traces:

[ 2977.768678] run fstests generic/451 at 2025-02-12 16:42:14
[ 3001.242006] list_add double add: new=c00000000e2398c8, 
prev=c00000000e2398c8, next=c0000000c1f2f0b8.
[ 3001.242047] ------------[ cut here ]------------
[ 3001.242052] kernel BUG at lib/list_debug.c:35!
[ 3001.242059] Oops: Exception in kernel mode, sig: 5 [#1]
[ 3001.242065] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=8192 NUMA pSeries
[ 3001.242072] Modules linked in: overlay dm_zero dm_thin_pool 
dm_persistent_data dm_bio_prison dm_snapshot dm_bufio dm_flakey ext4 
mbcache jbd2 loop dm_mod nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib 
nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct 
nft_chain_nat nf_nat nf_conntrack bonding nf_defrag_ipv6 nf_defrag_ipv4 
tls ip_set rfkill nf_tables nfnetlink pseries_rng aes_gcm_p10_crypto 
crypto_simd vmx_crypto xfs sr_mod sd_mod cdrom sg ibmvscsi ibmveth 
scsi_transport_srp fuse [last unloaded: scsi_debug]
[ 3001.242165] CPU: 26 UID: 0 PID: 894003 Comm: kworker/u228:17 Not 
tainted 6.14.0-rc2-next-20250212 #1
[ 3001.242174] Hardware name: IBM,8375-42A POWER9 (architected) 0x4e0202 
0xf000005 of:IBM,FW950.80 (VL950_131) hv:phyp pSeries
[ 3001.242183] Workqueue: loop0 loop_workfn [loop]
[ 3001.242194] NIP:  c00000000090d528 LR: c00000000090d524 CTR: 
00000000007088ec
[ 3001.242201] REGS: c0000000e5adfa70 TRAP: 0700   Not tainted 
(6.14.0-rc2-next-20250212)
[ 3001.242208] MSR:  800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  
CR: 2800024f  XER: 00000009
[ 3001.242229] CFAR: c000000000224238 IRQMASK: 1
[ 3001.242229] GPR00: c00000000090d524 c0000000e5adfd10 c000000001657500 
0000000000000058
[ 3001.242229] GPR04: c000001179a87c88 c000001179a96000 c0000000e5adfb58 
0000001177810000
[ 3001.242229] GPR08: 0000000000000027 0000000000000000 0000000000000000 
0000000000000001
[ 3001.242229] GPR12: c000000002a49cd0 c00000000f75ff00 c0000000001abf00 
c0000000fda72fc0
[ 3001.242229] GPR16: 0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[ 3001.242229] GPR20: c000000007acbc00 0000000004208060 c00000000e239880 
fffffffffffffef7
[ 3001.242229] GPR24: 0000000000000402 c0000000924b8fc0 c0000000c1f2f07c 
c0000000c1f2f000
[ 3001.242229] GPR28: c0000000c1f2f0b8 c00000000e2398c8 c00000000e2398c8 
c0000000c1f2f0b8
[ 3001.242309] NIP [c00000000090d528] __list_add_valid_or_report+0x158/0x15c
[ 3001.242318] LR [c00000000090d524] __list_add_valid_or_report+0x154/0x15c
[ 3001.242324] Call Trace:
[ 3001.242327] [c0000000e5adfd10] [c00000000090d524] 
__list_add_valid_or_report+0x154/0x15c (unreliable)
[ 3001.242336] [c0000000e5adfd90] [c0080000073c359c] 
loop_process_work+0x594/0x630 [loop]
[ 3001.242344] [c0000000e5adfe40] [c00000000019dc5c] 
process_one_work+0x1fc/0x4bc
[ 3001.242354] [c0000000e5adfef0] [c00000000019ebec] 
worker_thread+0x344/0x508
[ 3001.242362] [c0000000e5adff90] [c0000000001ac024] kthread+0x12c/0x14c
[ 3001.242370] [c0000000e5adffe0] [c00000000000df98] 
start_kernel_thread+0x14/0x18
[ 3001.242377] Code: 4b916cf9 60000000 0fe00000 7c0802a6 7c641b78 
3c62ffe5 7fe6fb78 7fc5f378 386343a0 f8010090 4b916cd1 60000000 
<0fe00000> 3c4c00d5 38429fd4 fbe1fff8
[ 3001.242407] ---[ end trace 0000000000000000 ]---
[ 3001.284444] pstore: backend (nvram) writing error (-1)
[ 3001.284452]
[ 3002.284455] Kernel panic - not syncing: Fatal exception
[ 3002.431023] Rebooting in 10 seconds..


Regards,

Venkat.

Re: [next-20250212] FStests generic/451 on EXT4 FS resulting in kernel OOPs

[Sourabh Jain]

Hello Venkat,

Thanks for reporting the issue.

I noticed that system rebooted after kernel panic.

Just wondering did you configured kdump/fadump?

Thanks,
Sourabh Jain


On 13/02/25 11:21, Venkat Rao Bagalkote wrote:
> Greetings!!!
>
>
> I am observing kernel OOPs, while running FStests generic/451 on EXT4 
> with linux-next kernel(next-20250212) on IBM Power Servers.
>
>
> Issue is seen on: next-20250212
>
> Issue not seen on: next-20250210
>
>
> Traces:
>
> [ 2977.768678] run fstests generic/451 at 2025-02-12 16:42:14
> [ 3001.242006] list_add double add: new=c00000000e2398c8, 
> prev=c00000000e2398c8, next=c0000000c1f2f0b8.
> [ 3001.242047] ------------[ cut here ]------------
> [ 3001.242052] kernel BUG at lib/list_debug.c:35!
> [ 3001.242059] Oops: Exception in kernel mode, sig: 5 [#1]
> [ 3001.242065] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=8192 NUMA pSeries
> [ 3001.242072] Modules linked in: overlay dm_zero dm_thin_pool 
> dm_persistent_data dm_bio_prison dm_snapshot dm_bufio dm_flakey ext4 
> mbcache jbd2 loop dm_mod nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 
> nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject 
> nft_ct nft_chain_nat nf_nat nf_conntrack bonding nf_defrag_ipv6 
> nf_defrag_ipv4 tls ip_set rfkill nf_tables nfnetlink pseries_rng 
> aes_gcm_p10_crypto crypto_simd vmx_crypto xfs sr_mod sd_mod cdrom sg 
> ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: scsi_debug]
> [ 3001.242165] CPU: 26 UID: 0 PID: 894003 Comm: kworker/u228:17 Not 
> tainted 6.14.0-rc2-next-20250212 #1
> [ 3001.242174] Hardware name: IBM,8375-42A POWER9 (architected) 
> 0x4e0202 0xf000005 of:IBM,FW950.80 (VL950_131) hv:phyp pSeries
> [ 3001.242183] Workqueue: loop0 loop_workfn [loop]
> [ 3001.242194] NIP:  c00000000090d528 LR: c00000000090d524 CTR: 
> 00000000007088ec
> [ 3001.242201] REGS: c0000000e5adfa70 TRAP: 0700   Not tainted 
> (6.14.0-rc2-next-20250212)
> [ 3001.242208] MSR:  800000000282b033 
> <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 2800024f  XER: 00000009
> [ 3001.242229] CFAR: c000000000224238 IRQMASK: 1
> [ 3001.242229] GPR00: c00000000090d524 c0000000e5adfd10 
> c000000001657500 0000000000000058
> [ 3001.242229] GPR04: c000001179a87c88 c000001179a96000 
> c0000000e5adfb58 0000001177810000
> [ 3001.242229] GPR08: 0000000000000027 0000000000000000 
> 0000000000000000 0000000000000001
> [ 3001.242229] GPR12: c000000002a49cd0 c00000000f75ff00 
> c0000000001abf00 c0000000fda72fc0
> [ 3001.242229] GPR16: 0000000000000000 0000000000000000 
> 0000000000000000 0000000000000000
> [ 3001.242229] GPR20: c000000007acbc00 0000000004208060 
> c00000000e239880 fffffffffffffef7
> [ 3001.242229] GPR24: 0000000000000402 c0000000924b8fc0 
> c0000000c1f2f07c c0000000c1f2f000
> [ 3001.242229] GPR28: c0000000c1f2f0b8 c00000000e2398c8 
> c00000000e2398c8 c0000000c1f2f0b8
> [ 3001.242309] NIP [c00000000090d528] 
> __list_add_valid_or_report+0x158/0x15c
> [ 3001.242318] LR [c00000000090d524] 
> __list_add_valid_or_report+0x154/0x15c
> [ 3001.242324] Call Trace:
> [ 3001.242327] [c0000000e5adfd10] [c00000000090d524] 
> __list_add_valid_or_report+0x154/0x15c (unreliable)
> [ 3001.242336] [c0000000e5adfd90] [c0080000073c359c] 
> loop_process_work+0x594/0x630 [loop]
> [ 3001.242344] [c0000000e5adfe40] [c00000000019dc5c] 
> process_one_work+0x1fc/0x4bc
> [ 3001.242354] [c0000000e5adfef0] [c00000000019ebec] 
> worker_thread+0x344/0x508
> [ 3001.242362] [c0000000e5adff90] [c0000000001ac024] kthread+0x12c/0x14c
> [ 3001.242370] [c0000000e5adffe0] [c00000000000df98] 
> start_kernel_thread+0x14/0x18
> [ 3001.242377] Code: 4b916cf9 60000000 0fe00000 7c0802a6 7c641b78 
> 3c62ffe5 7fe6fb78 7fc5f378 386343a0 f8010090 4b916cd1 60000000 
> <0fe00000> 3c4c00d5 38429fd4 fbe1fff8
> [ 3001.242407] ---[ end trace 0000000000000000 ]---
> [ 3001.284444] pstore: backend (nvram) writing error (-1)
> [ 3001.284452]
> [ 3002.284455] Kernel panic - not syncing: Fatal exception
> [ 3002.431023] Rebooting in 10 seconds..
>
>
> Regards,
>
> Venkat.
>
>

Re: [next-20250212] FStests generic/451 on EXT4 FS resulting in kernel OOPs

[Theodore Ts'o]

On Thu, Feb 13, 2025 at 11:21:22AM +0530, Venkat Rao Bagalkote wrote:
> Greetings!!!
> 
> I am observing kernel OOPs, while running FStests generic/451 on EXT4 with
> linux-next kernel(next-20250212) on IBM Power Servers.

I'm running daily spinnner tests on the fs-next branch on the
linux-next tree, via:

   gce-xfstests ltm -c ext4/all,xfs/all,btrfs/all,f2fs/all -g auto --repo \
       https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next \
       --watch fs-next

The fs-next branch is a subset of linux-next which only has file
system related branches.  This avoids instability caused by non-fs
related changes.  I'm not seeing any kernel oops on today's fs-next
running on an x86 cloud server, using a standardized config.

Looking at the kernel stack trace of your report, it appears that a
linked list used by the workqueue handler (in process_one_work) had
gotten corrupted.  This could be caused by anything (which is one of
the reasons why I test using fs-next instead of linux-next; we didn't
want to spend time debugging problems that aren't under our control).

Is this something which you can easily reproduce?  If so, can you try
seeing if it reproduces on the fs-next branch, and could you try
bisecting the to find the guilty commit?  If this was something we
could reproduce in my test infrastructure, the bisection could be
trivially accomplished via:

   gce-xfstests ltm -c ext4/4k generic/451 --repo linux-next.git \
      --bisect-bad linux-next --bisect-good v6.14-rc1

... and then wait for an e-mailed report to land in my inbox.   :-)

Can you do something similar using your test infrastructure?

Unfortuantely, given the stack trace, I doubt kernel developers would
be able to do much more with your report.

Thanks,

					- Ted

Re: [next-20250212] FStests generic/451 on EXT4 FS resulting in kernel OOPs

[Venkat]

I did attempt git bisect, and the first bad commit is d49e636ed7014be354d1ee279b8f4957e8def389.

Git bisect log:

 #git bisect log
#git bisect start
# status: waiting for both good and bad commits
# good: [a64dcfb451e254085a7daee5fe51bf22959d52d3] Linux 6.14-rc2
git bisect good a64dcfb451e254085a7daee5fe51bf22959d52d3
# status: waiting for bad commit, 1 good commit known
# bad: [c674aa7c289e51659e40dda0f954886ef7f80042] Add linux-next specific files for 20250212
git bisect bad c674aa7c289e51659e40dda0f954886ef7f80042
# good: [0a354ebcff3e3564281f2ee70b7094f5d16b762e] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next.git
git bisect good 0a354ebcff3e3564281f2ee70b7094f5d16b762e
# good: [21abbfe377b86bac23e3d167f002a492391555c7] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git
git bisect good 21abbfe377b86bac23e3d167f002a492391555c7
# good: [fcd8e79b9c555ea05cca4ab4593e3fa02722158f] Merge branch 'usb-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git
git bisect good fcd8e79b9c555ea05cca4ab4593e3fa02722158f
# bad: [8b5e9ed3e14f98a0dfa9a2e15654c0f4d94ac64c] Merge branch 'pwm/for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/ukleinek/linux.git
git bisect bad 8b5e9ed3e14f98a0dfa9a2e15654c0f4d94ac64c
# bad: [4e91a64bcf478f6f7cf7aab043d80ca671367348] Merge branch 'staging-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git
git bisect bad 4e91a64bcf478f6f7cf7aab043d80ca671367348
# good: [7b465a0d58c19a45ddf81c90bc8ba04693de038b] iio: light: adux1020: Drop unneeded assignment for cache_type
git bisect good 7b465a0d58c19a45ddf81c90bc8ba04693de038b
# good: [66b800119cc5ced59aa45d523e8ef9af54211abd] Merge branch 'togreg' of git://git.kernel.org/pub/scm/linux/kernel/git/jic23/iio.git
git bisect good 66b800119cc5ced59aa45d523e8ef9af54211abd
# bad: [1fad5171135d31d3d9258715c7d3fefa378797e0] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux-w1.git
git bisect bad 1fad5171135d31d3d9258715c7d3fefa378797e0
# good: [0d8db251dd15d2e284f5a6a53bc2b869f3eca711] phy: qcom: qmp-pcie: Add X1P42100 Gen4x4 PHY
git bisect good 0d8db251dd15d2e284f5a6a53bc2b869f3eca711
# bad: [d49e636ed7014be354d1ee279b8f4957e8def389] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/phy/linux-phy.git
git bisect bad d49e636ed7014be354d1ee279b8f4957e8def389
# good: [88c0053baed659acd85b87dd52cbd75f3d8806be] phy: Use (of|device)_property_present() for non-boolean properties
git bisect good 88c0053baed659acd85b87dd52cbd75f3d8806be
# first bad commit: [d49e636ed7014be354d1ee279b8f4957e8def389] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/phy/linux-phy.git
#

Regards,
Venkat.