* transaction assertion failure in next-20251103 @ 2025-11-04 15:53 Matthew Wilcox 2025-11-04 16:01 ` Darrick J. Wong 2025-11-04 22:25 ` Dave Chinner 0 siblings, 2 replies; 4+ messages in thread From: Matthew Wilcox @ 2025-11-04 15:53 UTC (permalink / raw) To: linux-xfs Two runs of xfstests, two assertion failures. One while running generic/083, one while running generic/561. Here's the g/561 failure: generic/561 run fstests generic/561 at 2025-11-03 22:20:18 XFS (vdb): Mounting V5 Filesystem e0d0d737-4733-4583-8d2e-deaedb725697 XFS (vdb): Ending clean mount XFS (vdc): Mounting V5 Filesystem 522795a0-828a-4476-9928-c71a0ff20619 XFS (vdc): Ending clean mount XFS (vdc): Unmounting Filesystem 522795a0-828a-4476-9928-c71a0ff20619 XFS (vdc): Mounting V5 Filesystem 2a587011-9c3c-41df-b507-5d3b27f8616c XFS (vdc): Ending clean mount iomap_finish_ioend_buffered: 80 callbacks suppressed vdc: writeback error on inode 166, offset 1589248, sector 5544 vdc: writeback error on inode 25165978, offset 3338240, sector 22021584 vdc: writeback error on inode 174, offset 1327104, sector 14832056 vdc: writeback error on inode 8388743, offset 1122304, sector 7342704 vdc: writeback error on inode 8388743, offset 4091904, sector 7344688 vdc: writeback error on inode 8388743, offset 7483392, sector 7362280 vdc: writeback error on inode 16908430, offset 5103616, sector 22031648 vdc: writeback error on inode 16908430, offset 5865472, sector 22033136 vdc: writeback error on inode 16908438, offset 778240, sector 22031064 vdc: writeback error on inode 16908438, offset 3309568, sector 22031216 XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem. XFS (vdc): Please unmount the filesystem and rectify the problem(s) XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120 ------------[ cut here ]------------ kernel BUG at fs/xfs/xfs_message.c:102! Oops: invalid opcode: 0000 [#1] SMP NOPTI CPU: 3 UID: 0 PID: 1631375 Comm: kworker/3:4 Tainted: G W 6.18.0-rc4-next-20251103-ktest-00016-g8c6f8121e488 #113 NONE Tainted: [W]=WARN Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 Workqueue: xfs-conv/vdc xfs_end_io RIP: 0010:assfail+0x3c/0x46 Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202 RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878 R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948 FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0 PKRU: 55555554 Call Trace: <TASK> xfs_trans_dup+0x258/0x270 xfs_trans_roll+0x48/0x120 xfs_defer_trans_roll+0x5f/0x1a0 xfs_defer_finish_noroll+0x3d5/0x5d0 xfs_trans_commit+0x4e/0x70 xfs_iomap_write_unwritten+0xe5/0x350 xfs_end_ioend+0x219/0x2c0 xfs_end_io+0xae/0xd0 process_one_work+0x1ed/0x530 ? move_linked_works+0x77/0xb0 worker_thread+0x1cf/0x3d0 ? __pfx_worker_thread+0x10/0x10 kthread+0x100/0x220 ? _raw_spin_unlock_irq+0x2b/0x40 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x1f6/0x250 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: [last unloaded: crc_t10dif] ---[ end trace 0000000000000000 ]--- RIP: 0010:assfail+0x3c/0x46 Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202 RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878 R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948 FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0 PKRU: 55555554 Kernel panic - not syncing: Fatal exception Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception ]--- The 081 failure looks similar: vdc: writeback error on inode 131223, offset 1552384, sector 194360 XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4 /0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem. XFS (vdc): Please unmount the filesystem and rectify the problem(s) XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_tra ns.c, line: 120 ------------[ cut here ]------------ kernel BUG at fs/xfs/xfs_message.c:102! Oops: invalid opcode: 0000 [#1] SMP NOPTI CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110 3-ktest-00016-g8c6f8121e488 #113 NONE Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 Workqueue: xfs-conv/vdc xfs_end_io RIP: 0010:assfail+0x3c/0x46 Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202 RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00 R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0 FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0 PKRU: 55555554 Call Trace: <TASK> xfs_trans_dup+0x258/0x270 xfs_trans_roll+0x48/0x120 xfs_defer_trans_roll+0x5f/0x1a0 xfs_defer_finish_noroll+0x3d5/0x5d0 xfs_trans_commit+0x4e/0x70 xfs_iomap_write_unwritten+0xe5/0x350 xfs_end_ioend+0x219/0x2c0 xfs_end_io+0xae/0xd0 process_one_work+0x1ed/0x530 ? move_linked_works+0x77/0xb0 worker_thread+0x1cf/0x3d0 ? __pfx_worker_thread+0x10/0x10 kthread+0x100/0x220 ? _raw_spin_unlock_irq+0x2b/0x40 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x1f6/0x250 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Modules linked in: ---[ end trace 0000000000000000 ]--- RIP: 0010:assfail+0x3c/0x46 Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202 RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00 R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0 FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0 PKRU: 55555554 Kernel panic - not syncing: Fatal exception Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception ]--- ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: transaction assertion failure in next-20251103 2025-11-04 15:53 transaction assertion failure in next-20251103 Matthew Wilcox @ 2025-11-04 16:01 ` Darrick J. Wong 2025-11-04 16:19 ` Matthew Wilcox 2025-11-04 22:25 ` Dave Chinner 1 sibling, 1 reply; 4+ messages in thread From: Darrick J. Wong @ 2025-11-04 16:01 UTC (permalink / raw) To: Matthew Wilcox; +Cc: linux-xfs On Tue, Nov 04, 2025 at 03:53:49PM +0000, Matthew Wilcox wrote: > Two runs of xfstests, two assertion failures. One while running > generic/083, one while running generic/561. Can you also post the output of xfs_info /dev/vd[bc] (whichever one is TEST_DEV)? 561 is the duperemove + write stress test, and 083 is another stresser. --D > Here's the g/561 failure: > > generic/561 run fstests generic/561 at 2025-11-03 22:20:18 > XFS (vdb): Mounting V5 Filesystem e0d0d737-4733-4583-8d2e-deaedb725697 > XFS (vdb): Ending clean mount > XFS (vdc): Mounting V5 Filesystem 522795a0-828a-4476-9928-c71a0ff20619 > XFS (vdc): Ending clean mount > XFS (vdc): Unmounting Filesystem 522795a0-828a-4476-9928-c71a0ff20619 > XFS (vdc): Mounting V5 Filesystem 2a587011-9c3c-41df-b507-5d3b27f8616c > XFS (vdc): Ending clean mount > iomap_finish_ioend_buffered: 80 callbacks suppressed > vdc: writeback error on inode 166, offset 1589248, sector 5544 > vdc: writeback error on inode 25165978, offset 3338240, sector 22021584 > vdc: writeback error on inode 174, offset 1327104, sector 14832056 > vdc: writeback error on inode 8388743, offset 1122304, sector 7342704 > vdc: writeback error on inode 8388743, offset 4091904, sector 7344688 > vdc: writeback error on inode 8388743, offset 7483392, sector 7362280 > vdc: writeback error on inode 16908430, offset 5103616, sector 22031648 > vdc: writeback error on inode 16908430, offset 5865472, sector 22033136 > vdc: writeback error on inode 16908438, offset 778240, sector 22031064 > vdc: writeback error on inode 16908438, offset 3309568, sector 22031216 > XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem. > XFS (vdc): Please unmount the filesystem and rectify the problem(s) > XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120 > ------------[ cut here ]------------ > kernel BUG at fs/xfs/xfs_message.c:102! > Oops: invalid opcode: 0000 [#1] SMP NOPTI > CPU: 3 UID: 0 PID: 1631375 Comm: kworker/3:4 Tainted: G W 6.18.0-rc4-next-20251103-ktest-00016-g8c6f8121e488 #113 NONE > Tainted: [W]=WARN > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 > Workqueue: xfs-conv/vdc xfs_end_io > RIP: 0010:assfail+0x3c/0x46 > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202 > RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878 > R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948 > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0 > PKRU: 55555554 > Call Trace: > <TASK> > xfs_trans_dup+0x258/0x270 > xfs_trans_roll+0x48/0x120 > xfs_defer_trans_roll+0x5f/0x1a0 > xfs_defer_finish_noroll+0x3d5/0x5d0 > xfs_trans_commit+0x4e/0x70 > xfs_iomap_write_unwritten+0xe5/0x350 > xfs_end_ioend+0x219/0x2c0 > xfs_end_io+0xae/0xd0 > process_one_work+0x1ed/0x530 > ? move_linked_works+0x77/0xb0 > worker_thread+0x1cf/0x3d0 > ? __pfx_worker_thread+0x10/0x10 > kthread+0x100/0x220 > ? _raw_spin_unlock_irq+0x2b/0x40 > ? __pfx_kthread+0x10/0x10 > ret_from_fork+0x1f6/0x250 > ? __pfx_kthread+0x10/0x10 > ret_from_fork_asm+0x1a/0x30 > </TASK> > Modules linked in: [last unloaded: crc_t10dif] > ---[ end trace 0000000000000000 ]--- > RIP: 0010:assfail+0x3c/0x46 > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202 > RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878 > R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948 > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0 > PKRU: 55555554 > Kernel panic - not syncing: Fatal exception > Kernel Offset: disabled > ---[ end Kernel panic - not syncing: Fatal exception ]--- > > The 081 failure looks similar: > > vdc: writeback error on inode 131223, offset 1552384, sector 194360 > XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4 > /0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem. > XFS (vdc): Please unmount the filesystem and rectify the problem(s) > XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_tra > ns.c, line: 120 > ------------[ cut here ]------------ > kernel BUG at fs/xfs/xfs_message.c:102! > Oops: invalid opcode: 0000 [#1] SMP NOPTI > CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110 > 3-ktest-00016-g8c6f8121e488 #113 NONE > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 > 04/01/2014 > Workqueue: xfs-conv/vdc xfs_end_io > RIP: 0010:assfail+0x3c/0x46 > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202 > RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00 > R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0 > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0 > PKRU: 55555554 > Call Trace: > <TASK> > xfs_trans_dup+0x258/0x270 > xfs_trans_roll+0x48/0x120 > xfs_defer_trans_roll+0x5f/0x1a0 > xfs_defer_finish_noroll+0x3d5/0x5d0 > xfs_trans_commit+0x4e/0x70 > xfs_iomap_write_unwritten+0xe5/0x350 > xfs_end_ioend+0x219/0x2c0 > xfs_end_io+0xae/0xd0 > process_one_work+0x1ed/0x530 > ? move_linked_works+0x77/0xb0 > worker_thread+0x1cf/0x3d0 > ? __pfx_worker_thread+0x10/0x10 > kthread+0x100/0x220 > ? _raw_spin_unlock_irq+0x2b/0x40 > ? __pfx_kthread+0x10/0x10 > ret_from_fork+0x1f6/0x250 > ? __pfx_kthread+0x10/0x10 > ret_from_fork_asm+0x1a/0x30 > </TASK> > Modules linked in: > ---[ end trace 0000000000000000 ]--- > RIP: 0010:assfail+0x3c/0x46 > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202 > RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00 > R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0 > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0 > PKRU: 55555554 > Kernel panic - not syncing: Fatal exception > Kernel Offset: disabled > ---[ end Kernel panic - not syncing: Fatal exception ]--- > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: transaction assertion failure in next-20251103 2025-11-04 16:01 ` Darrick J. Wong @ 2025-11-04 16:19 ` Matthew Wilcox 0 siblings, 0 replies; 4+ messages in thread From: Matthew Wilcox @ 2025-11-04 16:19 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs On Tue, Nov 04, 2025 at 08:01:02AM -0800, Darrick J. Wong wrote: > On Tue, Nov 04, 2025 at 03:53:49PM +0000, Matthew Wilcox wrote: > > Two runs of xfstests, two assertion failures. One while running > > generic/083, one while running generic/561. > > Can you also post the output of xfs_info /dev/vd[bc] (whichever one is > TEST_DEV)? 561 is the duperemove + write stress test, and 083 is > another stresser. Maybe? It's all wrapped up inside a CI system I don't fully understand, and since it's paniced, I can't log into it. Restarting the VM will presuambly blow away the storage. So here's what I can do: $ /usr/sbin/xfs_info ktest-out/vm/dev-1 meta-data=ktest-out/vm/dev-1 isize=512 agcount=4, agsize=917504 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=1 = exchange=0 metadir=0 data = bsize=4096 blocks=3670016, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0 log =internal log bsize=4096 blocks=16384, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 = rgcount=0 rgsize=0 extents $ /usr/sbin/xfs_info ktest-out/vm/dev-2 meta-data=ktest-out/vm/dev-2 isize=512 agcount=4, agsize=917504 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=1 = exchange=0 metadir=0 data = bsize=4096 blocks=3670016, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0 log =internal log bsize=4096 blocks=16384, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 = rgcount=0 rgsize=0 extents $ /usr/sbin/xfs_info ktest-out/vm/dev-3 xfs_info: ktest-out/vm/dev-3 is not a valid XFS filesystem (unexpected SB magic number 0x72687377) Use -F to force a read attempt. So my guess is that dev-1 is vdb, dev-2 is vdc and dev-3 is vdd. > --D > > > Here's the g/561 failure: > > > > generic/561 run fstests generic/561 at 2025-11-03 22:20:18 > > XFS (vdb): Mounting V5 Filesystem e0d0d737-4733-4583-8d2e-deaedb725697 > > XFS (vdb): Ending clean mount > > XFS (vdc): Mounting V5 Filesystem 522795a0-828a-4476-9928-c71a0ff20619 > > XFS (vdc): Ending clean mount > > XFS (vdc): Unmounting Filesystem 522795a0-828a-4476-9928-c71a0ff20619 > > XFS (vdc): Mounting V5 Filesystem 2a587011-9c3c-41df-b507-5d3b27f8616c > > XFS (vdc): Ending clean mount > > iomap_finish_ioend_buffered: 80 callbacks suppressed > > vdc: writeback error on inode 166, offset 1589248, sector 5544 > > vdc: writeback error on inode 25165978, offset 3338240, sector 22021584 > > vdc: writeback error on inode 174, offset 1327104, sector 14832056 > > vdc: writeback error on inode 8388743, offset 1122304, sector 7342704 > > vdc: writeback error on inode 8388743, offset 4091904, sector 7344688 > > vdc: writeback error on inode 8388743, offset 7483392, sector 7362280 > > vdc: writeback error on inode 16908430, offset 5103616, sector 22031648 > > vdc: writeback error on inode 16908430, offset 5865472, sector 22033136 > > vdc: writeback error on inode 16908438, offset 778240, sector 22031064 > > vdc: writeback error on inode 16908438, offset 3309568, sector 22031216 > > XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem. > > XFS (vdc): Please unmount the filesystem and rectify the problem(s) > > XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120 > > ------------[ cut here ]------------ > > kernel BUG at fs/xfs/xfs_message.c:102! > > Oops: invalid opcode: 0000 [#1] SMP NOPTI > > CPU: 3 UID: 0 PID: 1631375 Comm: kworker/3:4 Tainted: G W 6.18.0-rc4-next-20251103-ktest-00016-g8c6f8121e488 #113 NONE > > Tainted: [W]=WARN > > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 > > Workqueue: xfs-conv/vdc xfs_end_io > > RIP: 0010:assfail+0x3c/0x46 > > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > > RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202 > > RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff > > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > > RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a > > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878 > > R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948 > > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0 > > PKRU: 55555554 > > Call Trace: > > <TASK> > > xfs_trans_dup+0x258/0x270 > > xfs_trans_roll+0x48/0x120 > > xfs_defer_trans_roll+0x5f/0x1a0 > > xfs_defer_finish_noroll+0x3d5/0x5d0 > > xfs_trans_commit+0x4e/0x70 > > xfs_iomap_write_unwritten+0xe5/0x350 > > xfs_end_ioend+0x219/0x2c0 > > xfs_end_io+0xae/0xd0 > > process_one_work+0x1ed/0x530 > > ? move_linked_works+0x77/0xb0 > > worker_thread+0x1cf/0x3d0 > > ? __pfx_worker_thread+0x10/0x10 > > kthread+0x100/0x220 > > ? _raw_spin_unlock_irq+0x2b/0x40 > > ? __pfx_kthread+0x10/0x10 > > ret_from_fork+0x1f6/0x250 > > ? __pfx_kthread+0x10/0x10 > > ret_from_fork_asm+0x1a/0x30 > > </TASK> > > Modules linked in: [last unloaded: crc_t10dif] > > ---[ end trace 0000000000000000 ]--- > > RIP: 0010:assfail+0x3c/0x46 > > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > > RSP: 0018:ffff8881582d3ba8 EFLAGS: 00010202 > > RAX: 00000000ffffff01 RBX: ffff8880174a9790 RCX: 000000007fffffff > > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > > RBP: ffff8881582d3ba8 R08: 0000000000000000 R09: 000000000000000a > > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff8880174a9878 > > R13: ffff888110460000 R14: 0000000000000000 R15: ffff8880174a9948 > > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00007f7098803000 CR3: 0000000155342000 CR4: 0000000000750eb0 > > PKRU: 55555554 > > Kernel panic - not syncing: Fatal exception > > Kernel Offset: disabled > > ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > > The 081 failure looks similar: > > > > vdc: writeback error on inode 131223, offset 1552384, sector 194360 > > XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4 > > /0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem. > > XFS (vdc): Please unmount the filesystem and rectify the problem(s) > > XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_tra > > ns.c, line: 120 > > ------------[ cut here ]------------ > > kernel BUG at fs/xfs/xfs_message.c:102! > > Oops: invalid opcode: 0000 [#1] SMP NOPTI > > CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110 > > 3-ktest-00016-g8c6f8121e488 #113 NONE > > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 > > 04/01/2014 > > Workqueue: xfs-conv/vdc xfs_end_io > > RIP: 0010:assfail+0x3c/0x46 > > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > > RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202 > > RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff > > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > > RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a > > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00 > > R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0 > > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0 > > PKRU: 55555554 > > Call Trace: > > <TASK> > > xfs_trans_dup+0x258/0x270 > > xfs_trans_roll+0x48/0x120 > > xfs_defer_trans_roll+0x5f/0x1a0 > > xfs_defer_finish_noroll+0x3d5/0x5d0 > > xfs_trans_commit+0x4e/0x70 > > xfs_iomap_write_unwritten+0xe5/0x350 > > xfs_end_ioend+0x219/0x2c0 > > xfs_end_io+0xae/0xd0 > > process_one_work+0x1ed/0x530 > > ? move_linked_works+0x77/0xb0 > > worker_thread+0x1cf/0x3d0 > > ? __pfx_worker_thread+0x10/0x10 > > kthread+0x100/0x220 > > ? _raw_spin_unlock_irq+0x2b/0x40 > > ? __pfx_kthread+0x10/0x10 > > ret_from_fork+0x1f6/0x250 > > ? __pfx_kthread+0x10/0x10 > > ret_from_fork_asm+0x1a/0x30 > > </TASK> > > Modules linked in: > > ---[ end trace 0000000000000000 ]--- > > RIP: 0010:assfail+0x3c/0x46 > > Code: c2 f8 d1 41 82 48 89 f1 48 89 fe 48 c7 c7 55 69 46 82 48 89 e5 e8 e4 fd ff ff 8a 05 e6 79 55 01 3c 01 76 02 0f 0b a8 01 74 02 <0f> 0b 0f 0b 5d c3 cc cc cc cc 48 8d 45 10 4c 8d 6c 24 10 48 89 e2 > > RSP: 0018:ffff888115b3fba8 EFLAGS: 00010202 > > RAX: 00000000ffffff01 RBX: ffff88815d857878 RCX: 000000007fffffff > > RDX: 0000000000000021 RSI: 0000000000000000 RDI: ffffffff82466955 > > RBP: ffff888115b3fba8 R08: 0000000000000000 R09: 000000000000000a > > R10: 000000000000000a R11: 0fffffffffffffff R12: ffff88815d857d00 > > R13: ffff888114c28000 R14: 0000000000000000 R15: ffff88815d857dd0 > > FS: 0000000000000000(0000) GS:ffff8881f6b8d000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00007f585a04a000 CR3: 000000010ebd3000 CR4: 0000000000750eb0 > > PKRU: 55555554 > > Kernel panic - not syncing: Fatal exception > > Kernel Offset: disabled > > ---[ end Kernel panic - not syncing: Fatal exception ]--- > > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: transaction assertion failure in next-20251103 2025-11-04 15:53 transaction assertion failure in next-20251103 Matthew Wilcox 2025-11-04 16:01 ` Darrick J. Wong @ 2025-11-04 22:25 ` Dave Chinner 1 sibling, 0 replies; 4+ messages in thread From: Dave Chinner @ 2025-11-04 22:25 UTC (permalink / raw) To: Matthew Wilcox; +Cc: linux-xfs On Tue, Nov 04, 2025 at 03:53:49PM +0000, Matthew Wilcox wrote: > Two runs of xfstests, two assertion failures. One while running > generic/083, one while running generic/561. > > Here's the g/561 failure: > > generic/561 run fstests generic/561 at 2025-11-03 22:20:18 ..... > XFS (vdc): Corruption of in-memory data (0x8) detected at xfs_trans_mod_sb+0x2a4/0x310 (fs/xfs/xfs_trans.c:353). Shutting down filesystem. > XFS (vdc): Please unmount the filesystem and rectify the problem(s) > XFS: Assertion failed: tp->t_blk_res >= tp->t_blk_res_used, file: fs/xfs/xfs_trans.c, line: 120 .... > xfs_trans_dup+0x258/0x270 > xfs_trans_roll+0x48/0x120 > xfs_defer_trans_roll+0x5f/0x1a0 > xfs_defer_finish_noroll+0x3d5/0x5d0 > xfs_trans_commit+0x4e/0x70 > xfs_iomap_write_unwritten+0xe5/0x350 So we have a block reservation for a double split of the BMBT (which technically cannot happen for a double adjacent record insert) yet we apparently exhausted the entire block allocation. Thing is, modifications to tp->t_blk_res_used can only be done through xfs_trans_mod_sb() during the transaction, which does: case XFS_TRANS_SB_FDBLOCKS: /* * Track the number of blocks allocated in the transaction. * Make sure it does not exceed the number reserved. If so, * shutdown as this can lead to accounting inconsistency. */ if (delta < 0) { tp->t_blk_res_used += (uint)-delta; if (tp->t_blk_res_used > tp->t_blk_res) xfs_force_shutdown(mp, SHUTDOWN_CORRUPT_INCORE); Which will shut down the filesystem the moment a block allocation overrun occurs. This should happen long before we get to the transaction commit code... IOWs, the internal runtime transaction accounting check whenever tp->t_blk_res_used is updated passed just fine, but it was then detected as broken a short time later at transaction commit time. This implies something external modified either tp->t_blk_res or tp->t_blk_res_used between the last time it was modified by the filesystem and when it bounds checked and when the transaction was committed and rolled. This, to me, smells of external memory corruption, not an XFS bug... > CPU: 3 UID: 0 PID: 338999 Comm: kworker/3:12 Not tainted 6.18.0-rc4-next-2025110 ... and you are testing on a -next kernel, so it's entirely possible that there is newly introduced memory corruption bug somewhere outside of XFS. Can you reproduce this on a vanilla v6.18-rc4 kernel? -Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-11-04 22:25 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-11-04 15:53 transaction assertion failure in next-20251103 Matthew Wilcox 2025-11-04 16:01 ` Darrick J. Wong 2025-11-04 16:19 ` Matthew Wilcox 2025-11-04 22:25 ` Dave Chinner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox