Re: generic/648 metadata corruption

["Darrick J. Wong" <djwong@kernel.org>]

On Sat, Nov 08, 2025 at 09:11:27AM +1100, Dave Chinner wrote:
> On Fri, Nov 07, 2025 at 10:42:21AM +0100, Carlos Maiolino wrote:
> > Hello, has anybody has found any issues with generic/648 recently?
> > 
> > I've hit it on my test batch this evening, running a 2k block size a
> > metadata corruption error o generic/648.
> > 
> > I'll rerun the tests now and it later today, sharing it for a broader
> > audience.
> > 
> > This is running xfs's branch xfs-6.18-fixes.
> > 
> > I don't remember have seen this on my previous runs, but
> > I'll check the logs just in case.
> > 
> > The fsstress process ended up getting stuck at:
> > 
> > $ sudo cat /proc/2969171/stack 
> > [<0>] folio_wait_bit_common+0x138/0x340
> > [<0>] folio_wait_bit+0x1c/0x30
> > [<0>] folio_wait_writeback+0x2f/0x90
> > [<0>] __filemap_fdatawait_range+0x8d/0xf0
> > [<0>] filemap_fdatawait_keep_errors+0x22/0x50
> > [<0>] sync_inodes_sb+0x22c/0x2d0
> > [<0>] sync_filesystem+0x70/0xb0
> > [<0>] __x64_sys_syncfs+0x4e/0xd0
> > [<0>] x64_sys_call+0x778/0x1da0
> > [<0>] do_syscall_64+0x7f/0x7b0
> > [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> 
> Yeah, no surprise, the kernel is oopsing in IO completion with the
> folio still in writeback state - nothing will ever change the state
> on that folio now, so sync operations will block forever on it.
> 
> > The kernel log from the last mount.
> > 
> > [ 7467.362544] XFS (loop0): EXPERIMENTAL metadata directory tree feature enabled.  Use at your own risk!
> > [ 7467.363481] XFS (loop0): Mounting V5 Filesystem 2b40a1e4-f2f6-4a87-8f86-bbfc8a748329
> > [ 7467.880205] XFS (loop0): Starting recovery (logdev: internal)
> > [ 7468.006067] XFS (loop0): Ending recovery (logdev: internal)
> > [ 7470.131605] buffer_io_error: 8 callbacks suppressed
> > [ 7470.131613] Buffer I/O error on dev dm-1, logical block 243952, async page read
> > [ 7470.148095] I/O error, dev loop0, sector 10071568 op 0x0:(READ) flags 0x81700 phys_seg 1 prio class 2
> > [ 7470.148145] dm-0: writeback error on inode 71, offset 239466496, sector 668620
> ....
> > [ 7470.195987] XFS (loop0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x1fe/0x4c0 [xfs] (fs/xfs/xfs_trans_buf.c:311).  Shutting down filesystem.
> > [ 7470.200555] XFS (loop0): Please unmount the filesystem and rectify the problem(s)
> > [ 7470.201821] XFS (loop0): Metadata corruption detected at xfs_dinode_verify.part.0+0x434/0xcb0 [xfs], inode 0x40d422 xfs_inode_item_precommit_check
> 
> So what check did this fail? Convert
> xfs_dinode_verify.part.0+0x434xfs_dinode_verify.part.0+0x434 to a
> line number and that will tell us what the actual corruption
> detected was.
> 
> > [ 7470.206186] XFS (loop0): Unmount and run xfs_repair
> > [ 7470.207577] XFS (loop0): First 128 bytes of corrupted metadata buffer:
> > [ 7470.209043] 00000000: 49 4e 81 b6 03 02 00 00 00 00 03 8b 00 00 02 1c  IN..............
> > [ 7470.210242] 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 0d  ................
> > [ 7470.211633] 00000020: 36 42 da 8b dd 84 5e ec 36 42 da 8b ea d0 f9 2c  6B....^.6B.....,
> > [ 7470.212668] 00000030: 36 42 da 8b ea d0 f9 2c 00 00 00 00 00 25 b8 00  6B.....,.....%..
> > [ 7470.213878] 00000040: 00 00 00 00 00 00 03 32 00 00 00 00 00 00 00 00  .......2........
> > [ 7470.215056] 00000050: 00 00 18 01 00 00 00 00 00 00 00 02 6e b2 b8 ce  ............n...
> > [ 7470.216375] 00000060: 00 00 00 00 9f bb e2 1f 00 00 00 00 00 00 00 2f  .............../
> > [ 7470.217157] 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 1a  ................
> > [ 7470.218462] XFS: Assertion failed: fa == NULL, file: fs/xfs/xfs_inode_item.c, line: 62
> > [ 7470.219749] ------------[ cut here ]------------
> > [ 7470.220602] kernel BUG at fs/xfs/xfs_message.c:102!
> > [ 7470.221232] Oops: invalid opcode: 0000 [#1] SMP NOPTI
> > [ 7470.221907] CPU: 9 UID: 0 PID: 2967999 Comm: kworker/9:2 Not tainted 6.18.0-rc2.xfsRC5+ #23 PREEMPT(voluntary) 
> > [ 7470.223443] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-2.fc40 04/01/2014
> > [ 7470.224773] Workqueue: xfs-conv/loop0 xfs_end_io [xfs]
> > [ 7470.225855] RIP: 0010:assfail+0x35/0x3f [xfs]
> > [ 7470.226665] Code: 89 d0 41 89 c9 48 c7 c2 98 04 a0 c0 48 89 f1 48 89 fe 48 c7 c7 48 d6 9e c0 48 89 e5 e8 a4 fd ff ff 80 3d b5 62 26 00 00 74 02 <0f> 0b 0f 0b 5d e9 91 1d ba f8 48 8d 45 10 4c 8d 6c 24 10 48 89 e2
> > [ 7470.228907] RSP: 0018:ffffb2e087bcfc60 EFLAGS: 00010202
> > [ 7470.229492] RAX: 0000000000000000 RBX: ffff9e399129e400 RCX: 000000007fffffff
> > [ 7470.230298] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffc09ed648
> > [ 7470.231094] RBP: ffffb2e087bcfc60 R08: 0000000000000000 R09: 000000000000000a
> > [ 7470.231871] R10: 000000000000000a R11: 0fffffffffffffff R12: ffff9e3988311800
> > [ 7470.232670] R13: ffff9e399a358000 R14: ffff9e3c05054318 R15: ffff9e3999d0d790
> > [ 7470.233457] FS:  0000000000000000(0000) GS:ffff9e3d3e65f000(0000) knlGS:0000000000000000
> > [ 7470.234362] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 7470.235002] CR2: 00007f2f373b20d8 CR3: 0000000114904005 CR4: 0000000000772ef0
> > [ 7470.235805] PKRU: 55555554
> > [ 7470.236113] Call Trace:
> > [ 7470.236417]  <TASK>
> > [ 7470.236688]  xfs_inode_item_precommit+0x1b8/0x370 [xfs]
> > [ 7470.237601]  __xfs_trans_commit+0xba/0x410 [xfs]
> > [ 7470.238453]  xfs_trans_commit+0x3b/0x70 [xfs]
> > [ 7470.239245]  xfs_setfilesize+0xff/0x160 [xfs]
> 
> Hmmmm. I wonder. The issue was detected from IO completion
> processing.....
> 
> We've just written the in-memory inode to the buffer, calculated the
> CRC, and then we verify what we've written. Something in the dinode
> is coming out invalid, so either there is a code bug writing an
> invalid value somewhere, or the in-memory VFS/XFS inode metadata has
> been corrupted prior to this IO completion transaction commit being
> run.
> 
> Willy has been seeing unexpected transaction overruns on similar IO
> error based tests in IO completion processing that smell of
> memory corruption. These have been on 6.18-rc4-next and
> 6.18-rc4-fs-next kernels, IIUC.
> 
> Now we have a debug check of an inode in IO completion detecting
> in-memory corruption during a test that has triggered IO error
> processing on a plain 6.18-rc2 kernel
> 
> Coincidence? Maybe, but I'm is starting to think a new memory
> corruption bug has been introduced in the 6.18 merge cycle
> somewhere in the IO error processing paths....

Oh, there definitely is *something* wrong.  I turned on kasan to try to
take another stab at fixing the maple_node leak.  KASAN reported a UAF
in blkdev_put (which iirc has been reported already) in two of the VMs.
The rest of the VMs reported inode precommit errors in the UUID check
but no KASAN reports.

--D

> -Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
>