* transaction assertion failure in next-20251103
@ 2025-11-04 15:53 Matthew Wilcox
2025-11-04 16:01 ` Darrick J. Wong
2025-11-04 22:25 ` Dave Chinner
0 siblings, 2 replies; 4+ messages in thread
From: Matthew Wilcox @ 2025-11-04 15:53 UTC (permalink / raw)
To: linux-xfs
Two runs of xfstests, two assertion failures. One while running
generic/083, one while running generic/561.
Here's the g/561 failure:
generic/561 run fstests generic/561 at 2025-11-03 22:20:18
XFS (vdb): Mounting V5 Filesystem e0d0d737-4733-4583-8d2e-deaedb725697
XFS (vdb): Ending clean mount
XFS (vdc): Mounting V5 Filesystem 522795a0-828a-4476-9928-c71a0ff20619
XFS (vdc): Ending clean mount
XFS (vdc): Unmounting Filesystem 522795a0-828a-4476-9928-c71a0ff20619
XFS (vdc): Mounting V5 Filesystem 2a587011-9c3c-41df-b507-5d3b27f8616c
XFS (vdc): Ending clean mount
iomap_finish_ioend_buffered: 80 callbacks suppressed
vdc: writeback error on inode 166, offset 1589248, sector 5544
vdc: writeback error on inode 25165978, offset 3338240, sector 22021584
vdc: writeback error on inode 174, offset 1327104, sector 14832056
vdc: writeback error on inode 8388743, offset 1122304, sector 7342704
vdc: writeback error on inode 8388743, offset 4091904, sector 7344688
vdc: writeback error on inode 8388743, offset 7483392, sector 7362280
vdc: writeback error on inode 16908430, offset 5103616, sector 22031648
vdc: writeback error on inode 16908430, offset 5865472, sector 22033136
vdc: writeback error on inode 16908438, offset 778240, sector 22031064
vdc: writeback error on inode 16908438, offset 3309568, sector 22031216
XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem.
XFS (vdc): Please unmount the filesystem and rectify the problem(s)
XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120
------------[ cut here ]------------
kernel BUG at fs/xfs/xfs_message.c:102!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 3 UID: 0 PID: 1631375 Comm: kworker/3:4 Tainted: G W 6.18.0-rc4-next-20251103-ktest-00016-g8c6f8121e488 #113 NONE
Tainted: [W]=WARN
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: xfs-conv/vdc xfs_end_io
RIP: 0010:assfail+0x3c/0x46
Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202
RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff
RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a
R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878
R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948
FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0
PKRU: 55555554
Call Trace:
<TASK>
xfs_trans_dup+0x258/0x270
xfs_trans_roll+0x48/0x120
xfs_defer_trans_roll+0x5f/0x1a0
xfs_defer_finish_noroll+0x3d5/0x5d0
xfs_trans_commit+0x4e/0x70
xfs_iomap_write_unwritten+0xe5/0x350
xfs_end_ioend+0x219/0x2c0
xfs_end_io+0xae/0xd0
process_one_work+0x1ed/0x530
? move_linked_works+0x77/0xb0
worker_thread+0x1cf/0x3d0
? __pfx_worker_thread+0x10/0x10
kthread+0x100/0x220
? _raw_spin_unlock_irq+0x2b/0x40
? __pfx_kthread+0x10/0x10
ret_from_fork+0x1f6/0x250
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in: [last unloaded: crc_t10dif]
---[ end trace 0000000000000000 ]---
RIP: 0010:assfail+0x3c/0x46
Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202
RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff
RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a
R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878
R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948
FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0
PKRU: 55555554
Kernel panic - not syncing: Fatal exception
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception ]---
The 081 failure looks similar:
vdc: writeback error on inode 131223, offset 1552384, sector 194360
XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4
/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem.
XFS (vdc): Please unmount the filesystem and rectify the problem(s)
XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_tra
ns.c, line: 120
------------[ cut here ]------------
kernel BUG at fs/xfs/xfs_message.c:102!
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110
3-ktest-00016-g8c6f8121e488 #113 NONE
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2
04/01/2014
Workqueue: xfs-conv/vdc xfs_end_io
RIP: 0010:assfail+0x3c/0x46
Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202
RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff
RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a
R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00
R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0
FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0
PKRU: 55555554
Call Trace:
<TASK>
xfs_trans_dup+0x258/0x270
xfs_trans_roll+0x48/0x120
xfs_defer_trans_roll+0x5f/0x1a0
xfs_defer_finish_noroll+0x3d5/0x5d0
xfs_trans_commit+0x4e/0x70
xfs_iomap_write_unwritten+0xe5/0x350
xfs_end_ioend+0x219/0x2c0
xfs_end_io+0xae/0xd0
process_one_work+0x1ed/0x530
? move_linked_works+0x77/0xb0
worker_thread+0x1cf/0x3d0
? __pfx_worker_thread+0x10/0x10
kthread+0x100/0x220
? _raw_spin_unlock_irq+0x2b/0x40
? __pfx_kthread+0x10/0x10
ret_from_fork+0x1f6/0x250
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:assfail+0x3c/0x46
Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202
RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff
RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a
R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00
R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0
FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0
PKRU: 55555554
Kernel panic - not syncing: Fatal exception
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception ]---
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: transaction assertion failure in next-20251103
2025-11-04 15:53 transaction assertion failure in next-20251103 Matthew Wilcox
@ 2025-11-04 16:01 ` Darrick J. Wong
2025-11-04 16:19 ` Matthew Wilcox
2025-11-04 22:25 ` Dave Chinner
1 sibling, 1 reply; 4+ messages in thread
From: Darrick J. Wong @ 2025-11-04 16:01 UTC (permalink / raw)
To: Matthew Wilcox; +Cc: linux-xfs
On Tue, Nov 04, 2025 at 03:53:49PM +0000, Matthew Wilcox wrote:
> Two runs of xfstests, two assertion failures. One while running
> generic/083, one while running generic/561.
Can you also post the output of xfs_info /dev/vd[bc] (whichever one is
TEST_DEV)? 561 is the duperemove + write stress test, and 083 is
another stresser.
--D
> Here's the g/561 failure:
>
> generic/561 run fstests generic/561 at 2025-11-03 22:20:18
> XFS (vdb): Mounting V5 Filesystem e0d0d737-4733-4583-8d2e-deaedb725697
> XFS (vdb): Ending clean mount
> XFS (vdc): Mounting V5 Filesystem 522795a0-828a-4476-9928-c71a0ff20619
> XFS (vdc): Ending clean mount
> XFS (vdc): Unmounting Filesystem 522795a0-828a-4476-9928-c71a0ff20619
> XFS (vdc): Mounting V5 Filesystem 2a587011-9c3c-41df-b507-5d3b27f8616c
> XFS (vdc): Ending clean mount
> iomap_finish_ioend_buffered: 80 callbacks suppressed
> vdc: writeback error on inode 166, offset 1589248, sector 5544
> vdc: writeback error on inode 25165978, offset 3338240, sector 22021584
> vdc: writeback error on inode 174, offset 1327104, sector 14832056
> vdc: writeback error on inode 8388743, offset 1122304, sector 7342704
> vdc: writeback error on inode 8388743, offset 4091904, sector 7344688
> vdc: writeback error on inode 8388743, offset 7483392, sector 7362280
> vdc: writeback error on inode 16908430, offset 5103616, sector 22031648
> vdc: writeback error on inode 16908430, offset 5865472, sector 22033136
> vdc: writeback error on inode 16908438, offset 778240, sector 22031064
> vdc: writeback error on inode 16908438, offset 3309568, sector 22031216
> XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem.
> XFS (vdc): Please unmount the filesystem and rectify the problem(s)
> XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120
> ------------[ cut here ]------------
> kernel BUG at fs/xfs/xfs_message.c:102!
> Oops: invalid opcode: 0000 [#1] SMP NOPTI
> CPU: 3 UID: 0 PID: 1631375 Comm: kworker/3:4 Tainted: G W 6.18.0-rc4-next-20251103-ktest-00016-g8c6f8121e488 #113 NONE
> Tainted: [W]=WARN
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> Workqueue: xfs-conv/vdc xfs_end_io
> RIP: 0010:assfail+0x3c/0x46
> Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202
> RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff
> RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a
> R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878
> R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948
> FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0
> PKRU: 55555554
> Call Trace:
> <TASK>
> xfs_trans_dup+0x258/0x270
> xfs_trans_roll+0x48/0x120
> xfs_defer_trans_roll+0x5f/0x1a0
> xfs_defer_finish_noroll+0x3d5/0x5d0
> xfs_trans_commit+0x4e/0x70
> xfs_iomap_write_unwritten+0xe5/0x350
> xfs_end_ioend+0x219/0x2c0
> xfs_end_io+0xae/0xd0
> process_one_work+0x1ed/0x530
> ? move_linked_works+0x77/0xb0
> worker_thread+0x1cf/0x3d0
> ? __pfx_worker_thread+0x10/0x10
> kthread+0x100/0x220
> ? _raw_spin_unlock_irq+0x2b/0x40
> ? __pfx_kthread+0x10/0x10
> ret_from_fork+0x1f6/0x250
> ? __pfx_kthread+0x10/0x10
> ret_from_fork_asm+0x1a/0x30
> </TASK>
> Modules linked in: [last unloaded: crc_t10dif]
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:assfail+0x3c/0x46
> Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202
> RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff
> RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a
> R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878
> R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948
> FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0
> PKRU: 55555554
> Kernel panic - not syncing: Fatal exception
> Kernel Offset: disabled
> ---[ end Kernel panic - not syncing: Fatal exception ]---
>
> The 081 failure looks similar:
>
> vdc: writeback error on inode 131223, offset 1552384, sector 194360
> XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4
> /0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem.
> XFS (vdc): Please unmount the filesystem and rectify the problem(s)
> XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_tra
> ns.c, line: 120
> ------------[ cut here ]------------
> kernel BUG at fs/xfs/xfs_message.c:102!
> Oops: invalid opcode: 0000 [#1] SMP NOPTI
> CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110
> 3-ktest-00016-g8c6f8121e488 #113 NONE
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2
> 04/01/2014
> Workqueue: xfs-conv/vdc xfs_end_io
> RIP: 0010:assfail+0x3c/0x46
> Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202
> RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff
> RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a
> R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00
> R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0
> FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0
> PKRU: 55555554
> Call Trace:
> <TASK>
> xfs_trans_dup+0x258/0x270
> xfs_trans_roll+0x48/0x120
> xfs_defer_trans_roll+0x5f/0x1a0
> xfs_defer_finish_noroll+0x3d5/0x5d0
> xfs_trans_commit+0x4e/0x70
> xfs_iomap_write_unwritten+0xe5/0x350
> xfs_end_ioend+0x219/0x2c0
> xfs_end_io+0xae/0xd0
> process_one_work+0x1ed/0x530
> ? move_linked_works+0x77/0xb0
> worker_thread+0x1cf/0x3d0
> ? __pfx_worker_thread+0x10/0x10
> kthread+0x100/0x220
> ? _raw_spin_unlock_irq+0x2b/0x40
> ? __pfx_kthread+0x10/0x10
> ret_from_fork+0x1f6/0x250
> ? __pfx_kthread+0x10/0x10
> ret_from_fork_asm+0x1a/0x30
> </TASK>
> Modules linked in:
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:assfail+0x3c/0x46
> Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202
> RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff
> RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a
> R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00
> R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0
> FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0
> PKRU: 55555554
> Kernel panic - not syncing: Fatal exception
> Kernel Offset: disabled
> ---[ end Kernel panic - not syncing: Fatal exception ]---
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: transaction assertion failure in next-20251103
2025-11-04 16:01 ` Darrick J. Wong
@ 2025-11-04 16:19 ` Matthew Wilcox
0 siblings, 0 replies; 4+ messages in thread
From: Matthew Wilcox @ 2025-11-04 16:19 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: linux-xfs
On Tue, Nov 04, 2025 at 08:01:02AM -0800, Darrick J. Wong wrote:
> On Tue, Nov 04, 2025 at 03:53:49PM +0000, Matthew Wilcox wrote:
> > Two runs of xfstests, two assertion failures. One while running
> > generic/083, one while running generic/561.
>
> Can you also post the output of xfs_info /dev/vd[bc] (whichever one is
> TEST_DEV)? 561 is the duperemove + write stress test, and 083 is
> another stresser.
Maybe? It's all wrapped up inside a CI system I don't fully understand,
and since it's paniced, I can't log into it. Restarting the VM will
presuambly blow away the storage. So here's what I can do:
$ /usr/sbin/xfs_info ktest-out/vm/dev-1
meta-data=ktest-out/vm/dev-1 isize=512 agcount=4, agsize=917504 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=1
= reflink=1 bigtime=1 inobtcount=1 nrext64=1
= exchange=0 metadir=0
data = bsize=4096 blocks=3670016, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
= rgcount=0 rgsize=0 extents
$ /usr/sbin/xfs_info ktest-out/vm/dev-2
meta-data=ktest-out/vm/dev-2 isize=512 agcount=4, agsize=917504 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=1
= reflink=1 bigtime=1 inobtcount=1 nrext64=1
= exchange=0 metadir=0
data = bsize=4096 blocks=3670016, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
= rgcount=0 rgsize=0 extents
$ /usr/sbin/xfs_info ktest-out/vm/dev-3
xfs_info: ktest-out/vm/dev-3 is not a valid XFS filesystem (unexpected SB magic number 0x72687377)
Use -F to force a read attempt.
So my guess is that dev-1 is vdb, dev-2 is vdc and dev-3 is vdd.
> --D
>
> > Here's the g/561 failure:
> >
> > generic/561 run fstests generic/561 at 2025-11-03 22:20:18
> > XFS (vdb): Mounting V5 Filesystem e0d0d737-4733-4583-8d2e-deaedb725697
> > XFS (vdb): Ending clean mount
> > XFS (vdc): Mounting V5 Filesystem 522795a0-828a-4476-9928-c71a0ff20619
> > XFS (vdc): Ending clean mount
> > XFS (vdc): Unmounting Filesystem 522795a0-828a-4476-9928-c71a0ff20619
> > XFS (vdc): Mounting V5 Filesystem 2a587011-9c3c-41df-b507-5d3b27f8616c
> > XFS (vdc): Ending clean mount
> > iomap_finish_ioend_buffered: 80 callbacks suppressed
> > vdc: writeback error on inode 166, offset 1589248, sector 5544
> > vdc: writeback error on inode 25165978, offset 3338240, sector 22021584
> > vdc: writeback error on inode 174, offset 1327104, sector 14832056
> > vdc: writeback error on inode 8388743, offset 1122304, sector 7342704
> > vdc: writeback error on inode 8388743, offset 4091904, sector 7344688
> > vdc: writeback error on inode 8388743, offset 7483392, sector 7362280
> > vdc: writeback error on inode 16908430, offset 5103616, sector 22031648
> > vdc: writeback error on inode 16908430, offset 5865472, sector 22033136
> > vdc: writeback error on inode 16908438, offset 778240, sector 22031064
> > vdc: writeback error on inode 16908438, offset 3309568, sector 22031216
> > XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem.
> > XFS (vdc): Please unmount the filesystem and rectify the problem(s)
> > XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120
> > ------------[ cut here ]------------
> > kernel BUG at fs/xfs/xfs_message.c:102!
> > Oops: invalid opcode: 0000 [#1] SMP NOPTI
> > CPU: 3 UID: 0 PID: 1631375 Comm: kworker/3:4 Tainted: G W 6.18.0-rc4-next-20251103-ktest-00016-g8c6f8121e488 #113 NONE
> > Tainted: [W]=WARN
> > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> > Workqueue: xfs-conv/vdc xfs_end_io
> > RIP: 0010:assfail+0x3c/0x46
> > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> > RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202
> > RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff
> > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> > RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a
> > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878
> > R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948
> > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0
> > PKRU: 55555554
> > Call Trace:
> > <TASK>
> > xfs_trans_dup+0x258/0x270
> > xfs_trans_roll+0x48/0x120
> > xfs_defer_trans_roll+0x5f/0x1a0
> > xfs_defer_finish_noroll+0x3d5/0x5d0
> > xfs_trans_commit+0x4e/0x70
> > xfs_iomap_write_unwritten+0xe5/0x350
> > xfs_end_ioend+0x219/0x2c0
> > xfs_end_io+0xae/0xd0
> > process_one_work+0x1ed/0x530
> > ? move_linked_works+0x77/0xb0
> > worker_thread+0x1cf/0x3d0
> > ? __pfx_worker_thread+0x10/0x10
> > kthread+0x100/0x220
> > ? _raw_spin_unlock_irq+0x2b/0x40
> > ? __pfx_kthread+0x10/0x10
> > ret_from_fork+0x1f6/0x250
> > ? __pfx_kthread+0x10/0x10
> > ret_from_fork_asm+0x1a/0x30
> > </TASK>
> > Modules linked in: [last unloaded: crc_t10dif]
> > ---[ end trace 0000000000000000 ]---
> > RIP: 0010:assfail+0x3c/0x46
> > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> > RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202
> > RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff
> > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> > RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a
> > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878
> > R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948
> > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0
> > PKRU: 55555554
> > Kernel panic - not syncing: Fatal exception
> > Kernel Offset: disabled
> > ---[ end Kernel panic - not syncing: Fatal exception ]---
> >
> > The 081 failure looks similar:
> >
> > vdc: writeback error on inode 131223, offset 1552384, sector 194360
> > XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4
> > /0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem.
> > XFS (vdc): Please unmount the filesystem and rectify the problem(s)
> > XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_tra
> > ns.c, line: 120
> > ------------[ cut here ]------------
> > kernel BUG at fs/xfs/xfs_message.c:102!
> > Oops: invalid opcode: 0000 [#1] SMP NOPTI
> > CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110
> > 3-ktest-00016-g8c6f8121e488 #113 NONE
> > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2
> > 04/01/2014
> > Workqueue: xfs-conv/vdc xfs_end_io
> > RIP: 0010:assfail+0x3c/0x46
> > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> > RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202
> > RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff
> > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> > RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a
> > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00
> > R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0
> > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0
> > PKRU: 55555554
> > Call Trace:
> > <TASK>
> > xfs_trans_dup+0x258/0x270
> > xfs_trans_roll+0x48/0x120
> > xfs_defer_trans_roll+0x5f/0x1a0
> > xfs_defer_finish_noroll+0x3d5/0x5d0
> > xfs_trans_commit+0x4e/0x70
> > xfs_iomap_write_unwritten+0xe5/0x350
> > xfs_end_ioend+0x219/0x2c0
> > xfs_end_io+0xae/0xd0
> > process_one_work+0x1ed/0x530
> > ? move_linked_works+0x77/0xb0
> > worker_thread+0x1cf/0x3d0
> > ? __pfx_worker_thread+0x10/0x10
> > kthread+0x100/0x220
> > ? _raw_spin_unlock_irq+0x2b/0x40
> > ? __pfx_kthread+0x10/0x10
> > ret_from_fork+0x1f6/0x250
> > ? __pfx_kthread+0x10/0x10
> > ret_from_fork_asm+0x1a/0x30
> > </TASK>
> > Modules linked in:
> > ---[ end trace 0000000000000000 ]---
> > RIP: 0010:assfail+0x3c/0x46
> > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> > RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202
> > RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff
> > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955
> > RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a
> > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00
> > R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0
> > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0
> > PKRU: 55555554
> > Kernel panic - not syncing: Fatal exception
> > Kernel Offset: disabled
> > ---[ end Kernel panic - not syncing: Fatal exception ]---
> >
> >
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: transaction assertion failure in next-20251103
2025-11-04 15:53 transaction assertion failure in next-20251103 Matthew Wilcox
2025-11-04 16:01 ` Darrick J. Wong
@ 2025-11-04 22:25 ` Dave Chinner
1 sibling, 0 replies; 4+ messages in thread
From: Dave Chinner @ 2025-11-04 22:25 UTC (permalink / raw)
To: Matthew Wilcox; +Cc: linux-xfs
On Tue, Nov 04, 2025 at 03:53:49PM +0000, Matthew Wilcox wrote:
> Two runs of xfstests, two assertion failures. One while running
> generic/083, one while running generic/561.
>
> Here's the g/561 failure:
>
> generic/561 run fstests generic/561 at 2025-11-03 22:20:18
.....
> XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem.
> XFS (vdc): Please unmount the filesystem and rectify the problem(s)
> XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120
....
> xfs_trans_dup+0x258/0x270
> xfs_trans_roll+0x48/0x120
> xfs_defer_trans_roll+0x5f/0x1a0
> xfs_defer_finish_noroll+0x3d5/0x5d0
> xfs_trans_commit+0x4e/0x70
> xfs_iomap_write_unwritten+0xe5/0x350
So we have a block reservation for a double split of the BMBT
(which technically cannot happen for a double adjacent record
insert) yet we apparently exhausted the entire block allocation.
Thing is, modifications to tp->t_blk_res_used can only be done
through xfs_trans_mod_sb() during the transaction, which
does:
case XFS_TRANS_SB_FDBLOCKS:
/*
* Track the number of blocks allocated in the transaction.
* Make sure it does not exceed the number reserved. If so,
* shutdown as this can lead to accounting inconsistency.
*/
if (delta < 0) {
tp->t_blk_res_used += (uint)-delta;
if (tp->t_blk_res_used > tp->t_blk_res)
xfs_force_shutdown(mp, SHUTDOWN_CORRUPT_INCORE);
Which will shut down the filesystem the moment a block allocation
overrun occurs. This should happen long before we get to the
transaction commit code...
IOWs, the internal runtime transaction accounting check whenever
tp->t_blk_res_used is updated passed just fine, but it was then
detected as broken a short time later at transaction commit time.
This implies something external modified either tp->t_blk_res or
tp->t_blk_res_used between the last time it was modified by the
filesystem and when it bounds checked and when the transaction was
committed and rolled.
This, to me, smells of external memory corruption, not an XFS bug...
> CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110
... and you are testing on a -next kernel, so it's entirely possible
that there is newly introduced memory corruption bug somewhere
outside of XFS.
Can you reproduce this on a vanilla v6.18-rc4 kernel?
-Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-11-04 22:25 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-11-04 15:53 transaction assertion failure in next-20251103 Matthew Wilcox
2025-11-04 16:01 ` Darrick J. Wong
2025-11-04 16:19 ` Matthew Wilcox
2025-11-04 22:25 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox