Re: [bug report] fstests generic/774 hang again

["Darrick J. Wong" <djwong@kernel.org>]

On Thu, Sep 17, 2026 at 07:45:04AM +1000, Dave Chinner wrote:
> On Tue, Sep 15, 2026 at 06:34:50PM +0900, Shin'ichiro Kawasaki wrote:
> > I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB
> > TCMU fileio devices. Actually I once reported this hang symptom in last October,
> > and Darrick and John kindly took some actions [1]. Since then, the hang
> > disappeared and I have not observed the hang almost one year. However, my test
> > system started reporting the hang again last week.
> > 
> > [1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t
> > 
> > FYI, here I attache the kernel message [2]. I observed the hang last week with
> > the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove
> > an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel
> > at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in
> > xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated
> > in stable manner by repeating the test case 20 times or so.
> > 
> > I also tried other test cases in atmoicwrites group, and observed no failure.
> > 
> >  generic/765        [not run] write atomic not supported by this block device
> >  generic/767         11s
> >  generic/768         13s
> >  generic/769         13s
> >  generic/770         32s
> >  generic/773        [not run] write atomic not supported by this block device
> >  generic/774        (skipped)
> >  generic/775         293s
> >  generic/776        [notrun] write atomic not supported by this block device
> >  generic/778         48s
> >  xfs/838            [not run] External volumes not in use, skipped this test
> >  xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
> >  xfs/840            [not run] write atomic not supported by this block device
> > 
> > Action for fix will be appreciated. If I can do anything on my test node, please
> > let me know.
> .....
> 
> A bunch of threads waiting on the ILOCK here:
> 
>  down_write_nested+0x1c0/0x1f0
>  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
>  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
>  iomap_dio_complete+0x140/0xb20
>  iomap_dio_complete_work+0x58/0x90
>  process_one_work+0x947/0x1760
> 
> And the holder:
> 
>  schedule+0xe5/0x2e0
>  xlog_grant_head_wait+0x175/0xac0 [xfs]
>  xlog_grant_head_check+0x312/0x3f0 [xfs]
>  xfs_log_regrant+0x380/0x7d0 [xfs]
>  xfs_trans_roll+0x2d9/0x420 [xfs]
>  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
>  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
>  xfs_trans_commit+0xfc/0x180 [xfs]
>  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
>  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
>  iomap_dio_complete+0x140/0xb20
>  iomap_dio_complete_work+0x58/0x90
>  process_one_work+0x947/0x1760
> 
> is waiting on log space whilst holding the ILOCK.
> 
> This looks to me like the test runs out of log space because of all
> the IO completions holding transaction reservations waiting on the
> ILOCK, whilst the ILOCK holder can't get enough log space to regrant
> on transaction roll to continue the transaction.

Agreed.

Why are we calling xlog_grant_head_check from within
xfs_reflink_end_atomic_cow?  I think the reason for doing that is
because we've exhausted t_cnt in the ticket (i.e. we've already rolled
more than tr_logcount times).

Oh.  tr_atomic_ioend.tr_logcount is 5 on a rmap+reflink filesystem,
which it inherits from tr_itruncate.  However, tr_itruncate is only
intended to remove two extents from a file, so it only needs 5 rolls.
Coincidentally we calculate 5 rolls for each atomic extent remapping
operation.

For atomic write ioends, what if we increased tr_logcount to 5x the
number of remappings that would have to occur to finish the write?
That would preallocate all the permanent reservation we'd need before we
take the ILOCK, which avoids the situation of needing to obtain more log
space while holding ILOCK.

The downside is that you'd have to limit the software awu_max even
further, perhaps to 40% of the log size divided by
(tr_logres*tr_logcount).  However, we'd still be able to handle
concurrent atomic writes to different parts of the file, at least until
fragmentation got bad.

<shrug>

--D

> 
> Oh:
> 
> STATIC void
> xfs_calc_default_atomic_ioend_reservation(
>         struct xfs_mount        *mp,
>         struct xfs_trans_resv   *resp)
> {
>         /* Pick a default that will scale reasonably for the log size. */
>         resp->tr_atomic_ioend = resp->tr_itruncate;
> }
> 
> Which means each reservation is probably holding hundreds of KB to
> MB of journal reservation. If the journal is 64MB in size, then a
> few dozen o these might be all that is needed to consume all the
> reservation space.
> 
> Hmmm - there are at least 30 tasks stuck in
> xfs_reflink_end_atomic_cow(), another 30 stuck in
> xfs_vn_update_time() holding reservations, and significant number
> xfs_atomic_write_cow_iomap_begin()->xfs_trans_alloc_inode() holding
> tr_write reservations.
> 
> IOWs, smells of the journal being run out of reservation space, and
> no new space being able to be freed because all the dirty inodes
> in the journal that need to be written back to free up space are
> locked waiting for journal space to come free....
> 
> How big is the journal in the filesystem being tested? If you
> increase the size of the journal, does it go away? Can you get a
> dump of the transaction reservation sizes for the filesystem in
> question (the xfs_db logres command can do this, IIRC) and then run
> the math on the reservations held from the dump of all the blocked
> tasks holding locks to see if this matches the size of the journal
> in the fs?
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> dgc@kernel.org
>