weird splats in xfs/561 on 6.10-rc1?

weird splats in xfs/561 on 6.10-rc1?

[Darrick J. Wong]

Hi Christoph,

I noticed the following assert triggering on xfs/561 after about 8
minutes of operation on a 64k-page arm64 vm with a rtgroups filesystem:

FSTYP         -- xfs (debug)
PLATFORM      -- Linux/aarch64 oci-mtra46 6.10.0-rc1-xfsa #rc1 SMP PREEMPT Wed May 29 11:26:10 PDT 2024
MKFS_OPTIONS  -- -f -rrtdev=/dev/sdb4 -i verity=1,exchange=1, -d rtinherit=1, -n parent=1, -r rtgroups=1, /dev/sda3
MOUNT_OPTIONS -- -o usrquota,grpquota,prjquota, -ortdev=/dev/sdb4 /dev/sda3 /opt

XFS (sda3): ino 414aa33 data fork has delalloc extent at [0x16a:0x6]
XFS: Assertion failed: 0, file: fs/xfs/xfs_icache.c, line: 1853
------------[ cut here ]------------
WARNING: CPU: 0 PID: 44 at fs/xfs/xfs_message.c:104 assfail+0x48/0x68 [xfs]
Modules linked in: xfs rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_REDIRECT nf_nat nf_
CPU: 0 PID: 44 Comm: kswapd0 Tainted: G        W          6.10.0-rc1-xfsa #rc1 1ccdfe2f
Hardware name: QEMU KVM Virtual Machine, BIOS 1.6.6 08/22/2023
pstate: 60401005 (nZCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)
pc : assfail+0x48/0x68 [xfs]
lr : assfail+0x3c/0x68 [xfs]
sp : fffffe0081fef7b0
x29: fffffe0081fef7b0 x28: fffffe0081fefab8 x27: 000000000000f35c
x26: 0000000000000005 x25: 00000000000003fb x24: 0000000000000000
x23: fffffc013f518000 x22: fffffe007a40ebe4 x21: fffffe0080f30008
x20: fffffe0081289458 x19: fffffc00e5f25f00 x18: 0000000000000006
x17: 3a61363178305b20 x16: 746120746e657478 x15: fffffe0081fef1a0
x14: 0000000000000000 x13: 33353831203a656e x12: fffffe0081fef6e0
x11: fffffe007a644790 x10: fffffe00fa64478f x9 : 0fffffffffffffff
x8 : 000000000000000a x7 : 00000000ffffffc0 x6 : 0000000000000021
x5 : fffffe007a644791 x4 : 00000000ffffffca x3 : 0000000000000000
x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
Call trace:
 assfail+0x48/0x68 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 xfs_inodegc_set_reclaimable+0x164/0x180 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 xfs_inode_mark_reclaimable+0xd0/0x460 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 xfs_fs_destroy_inode+0xf4/0x1b8 [xfs 33c3f304945dadccc25d02a0520df74527538fda]
 destroy_inode+0x48/0x88
 evict+0x158/0x1a8
 dispose_list+0x6c/0xa8
 prune_icache_sb+0x64/0xa0
 super_cache_scan+0x148/0x1a8
 do_shrink_slab+0x14c/0x470
 shrink_slab+0x304/0x528
 shrink_one+0xa8/0x240
 shrink_node+0xb18/0xe50
 balance_pgdat+0x398/0x8b0
 kswapd+0x244/0x518
 kthread+0x110/0x128
 ret_from_fork+0x10/0x20
---[ end trace 0000000000000000 ]---
Direct I/O collision with buffered writes! File: d32/d36/d5a/f54 Comm: fsstress
Direct I/O collision with buffered writes! File: /rt/p3/f25 Comm: fsstress

From what I can tell, this wasn't happening with my 6.9 djwong-dev
branch, so I suspect it's something that came in from when I rebased
against 6.10-rc1.  It seems to happen all over the place (and not just
with realtime files) if I leave Zhang Yi's iomap patches applied.  If I
revert them, the screaming seems to go down to just this one test.

The file itself is ~1482KB, or enough for the file to have 0x169 actual
blocks of written data to it, so the delalloc reservation is beyond the
eof block.  Any thoughts?

--D

Re: weird splats in xfs/561 on 6.10-rc1?

[Christoph Hellwig]

On Thu, May 30, 2024 at 03:59:12PM -0700, Darrick J. Wong wrote:
> >From what I can tell, this wasn't happening with my 6.9 djwong-dev
> branch, so I suspect it's something that came in from when I rebased
> against 6.10-rc1.  It seems to happen all over the place (and not just
> with realtime files) if I leave Zhang Yi's iomap patches applied.  If I
> revert them, the screaming seems to go down to just this one test.

When testing Linus' tree I haven't seen this yet, but I also haven't
done a lot of testing yet.  I mostly triggered odd MM warnings that
Johannes fixes, but I haven't seen something like this yet. 

> The file itself is ~1482KB, or enough for the file to have 0x169 actual
> blocks of written data to it, so the delalloc reservation is beyond the
> eof block.  Any thoughts?

I'll see if I can reproduce it with your tree.  I have a pretty full
plate given that yesterday was a public holiday here which doesn't
really help my catch up rate..

Re: weird splats in xfs/561 on 6.10-rc1?

[Darrick J. Wong]

On Thu, May 30, 2024 at 10:39:29PM -0700, Christoph Hellwig wrote:
> On Thu, May 30, 2024 at 03:59:12PM -0700, Darrick J. Wong wrote:
> > >From what I can tell, this wasn't happening with my 6.9 djwong-dev
> > branch, so I suspect it's something that came in from when I rebased
> > against 6.10-rc1.  It seems to happen all over the place (and not just
> > with realtime files) if I leave Zhang Yi's iomap patches applied.  If I
> > revert them, the screaming seems to go down to just this one test.
> 
> When testing Linus' tree I haven't seen this yet, but I also haven't
> done a lot of testing yet.  I mostly triggered odd MM warnings that
> Johannes fixes, but I haven't seen something like this yet. 
> 
> > The file itself is ~1482KB, or enough for the file to have 0x169 actual
> > blocks of written data to it, so the delalloc reservation is beyond the
> > eof block.  Any thoughts?
> 
> I'll see if I can reproduce it with your tree.  I have a pretty full
> plate given that yesterday was a public holiday here which doesn't
> really help my catch up rate..

Yeah.  You might want to revert all of Zhang Yi's patches first, though
maybe his new series actually fixes all the problems.

(Hm, no, it's still missing that cow-over-hole thing Dave was musing
about.)

--D

Re: weird splats in xfs/561 on 6.10-rc1?

[Christoph Hellwig]

On Thu, May 30, 2024 at 11:18:22PM -0700, Darrick J. Wong wrote:
> Yeah.  You might want to revert all of Zhang Yi's patches first, though
> maybe his new series actually fixes all the problems.
> 
> (Hm, no, it's still missing that cow-over-hole thing Dave was musing
> about.)

I've kicked off a run with realtime and rthinherit on your current
realtime reflink tree and haven't seen any warning yet.  Do you know
if it's limited to > 4K page sizes?  My testing ability on that is
unfortunately limited to a used macbook running linux in a VM as that
Asahi installer sucks ass, so rather limited.

Re: weird splats in xfs/561 on 6.10-rc1?

[Darrick J. Wong]

On Thu, May 30, 2024 at 11:22:13PM -0700, Christoph Hellwig wrote:
> On Thu, May 30, 2024 at 11:18:22PM -0700, Darrick J. Wong wrote:
> > Yeah.  You might want to revert all of Zhang Yi's patches first, though
> > maybe his new series actually fixes all the problems.
> > 
> > (Hm, no, it's still missing that cow-over-hole thing Dave was musing
> > about.)
> 
> I've kicked off a run with realtime and rthinherit on your current
> realtime reflink tree and haven't seen any warning yet.  Do you know
> if it's limited to > 4K page sizes?  My testing ability on that is
> unfortunately limited to a used macbook running linux in a VM as that
> Asahi installer sucks ass, so rather limited.

I don't know specifically if it's limited to > 4K pagesize, other than
to say that I haven't seen it show up on x86 yet.

--D