[RFC] generic/301: flaky failure on btrfs after metadata overcommit change

[RFC] generic/301: flaky failure on btrfs after metadata overcommit change

[Leo Martins]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2414 bytes --]

Hi,

generic/301 has become flaky on btrfs after commit 0dc118b3c327 ("btrfs:
be less aggressive with metadata overcommit when we can do full
flushing") which landed in btrfs/for-next. Out of 30 runs, 8 fail with:

    +file2 badly fragmented

I bisected this to the above commit, which reduces the metadata
overcommit limit from 1/8th to 1/64th of available space for
full-flushing contexts. This is a legitimate fix for -ENOSPC transaction
aborts on small filesystems, but as a side effect it causes more
frequent transaction commits during writeback. The reduced batching
means the extent allocator has less opportunity to coalesce adjacent CoW
extents, resulting in higher extent counts that sometimes cross the
test's threshold.

The fragmentation check in question is:

    test $new_extents -lt $((internal_blks * 2 / 3)) || echo "file2 badly fragmented"

The 2/3 threshold was introduced in 9184ca155d7c ("xfs: test
fragmentation characteristics of copy-on-write") as part of a series
testing XFS's CoW extent size hint (cowextsize) mechanism. For btrfs,
this threshold is arbitrary — btrfs doesn't have XFS's cowextsize hint,
and its CoW extent allocation depends on factors like transaction commit
frequency and metadata reservation behavior, which is exactly what the
overcommit commit changed.

I see two possible fixes and would appreciate input on which is
preferred:

Option A: _notrun for btrfs
----------------------------

Skip the entire test since the fragmentation threshold is not applicable
to btrfs:

    test $FSTYP = "btrfs" && \
        _notrun "CoW fragmentation threshold not applicable to btrfs"

Option B: Skip only the extent count assertion for btrfs
---------------------------------------------------------

Keep the CoW + data integrity portion of the test (the md5sum checks
after random CoW writes and remount are still useful) and only skip the
fragmentation assertion:

    if [ "$FSTYP" != "btrfs" ]; then
        test $new_extents -lt $((internal_blks * 2 / 3)) || \
            echo "file2 badly fragmented"
    fi

I lean towards option B since the CoW write + remount + md5sum
verification is still a reasonable smoke test, but option A is cleaner
if the consensus is that this test isn't adding value for btrfs.

Thoughts?

Thanks,
Leo Martins

Re: [RFC] generic/301: flaky failure on btrfs after metadata overcommit change

[Darrick J. Wong]

On Mon, Mar 23, 2026 at 01:15:29PM -0700, Leo Martins wrote:
> Hi,
> 
> generic/301 has become flaky on btrfs after commit 0dc118b3c327 ("btrfs:
> be less aggressive with metadata overcommit when we can do full
> flushing") which landed in btrfs/for-next. Out of 30 runs, 8 fail with:
> 
>     +file2 badly fragmented
> 
> I bisected this to the above commit, which reduces the metadata
> overcommit limit from 1/8th to 1/64th of available space for
> full-flushing contexts. This is a legitimate fix for -ENOSPC transaction
> aborts on small filesystems, but as a side effect it causes more
> frequent transaction commits during writeback. The reduced batching
> means the extent allocator has less opportunity to coalesce adjacent CoW
> extents, resulting in higher extent counts that sometimes cross the
> test's threshold.
> 
> The fragmentation check in question is:
> 
>     test $new_extents -lt $((internal_blks * 2 / 3)) || echo "file2 badly fragmented"
> 
> The 2/3 threshold was introduced in 9184ca155d7c ("xfs: test
> fragmentation characteristics of copy-on-write") as part of a series
> testing XFS's CoW extent size hint (cowextsize) mechanism. For btrfs,
> this threshold is arbitrary — btrfs doesn't have XFS's cowextsize hint,
> and its CoW extent allocation depends on factors like transaction commit
> frequency and metadata reservation behavior, which is exactly what the
> overcommit commit changed.
> 
> I see two possible fixes and would appreciate input on which is
> preferred:
> 
> Option A: _notrun for btrfs
> ----------------------------
> 
> Skip the entire test since the fragmentation threshold is not applicable
> to btrfs:
> 
>     test $FSTYP = "btrfs" && \
>         _notrun "CoW fragmentation threshold not applicable to btrfs"
> 
> Option B: Skip only the extent count assertion for btrfs
> ---------------------------------------------------------
> 
> Keep the CoW + data integrity portion of the test (the md5sum checks
> after random CoW writes and remount are still useful) and only skip the
> fragmentation assertion:
> 
>     if [ "$FSTYP" != "btrfs" ]; then
>         test $new_extents -lt $((internal_blks * 2 / 3)) || \
>             echo "file2 badly fragmented"
>     fi
> 
> I lean towards option B since the CoW write + remount + md5sum
> verification is still a reasonable smoke test, but option A is cleaner
> if the consensus is that this test isn't adding value for btrfs.
> 
> Thoughts?

B, since it's checking data integrity across a mount cycle.

--D

> Thanks,
> Leo Martins
> 

Re: [RFC] generic/301: flaky failure on btrfs after metadata overcommit change

[Qu Wenruo]

在 2026/3/24 06:45, Leo Martins 写道:
> Hi,
> 
> generic/301 has become flaky on btrfs after commit 0dc118b3c327 ("btrfs:
> be less aggressive with metadata overcommit when we can do full
> flushing") which landed in btrfs/for-next. Out of 30 runs, 8 fail with:
> 
>      +file2 badly fragmented
> 
> I bisected this to the above commit, which reduces the metadata
> overcommit limit from 1/8th to 1/64th of available space for
> full-flushing contexts. This is a legitimate fix for -ENOSPC transaction
> aborts on small filesystems, but as a side effect it causes more
> frequent transaction commits during writeback. The reduced batching
> means the extent allocator has less opportunity to coalesce adjacent CoW
> extents, resulting in higher extent counts that sometimes cross the
> test's threshold.
> 
> The fragmentation check in question is:
> 
>      test $new_extents -lt $((internal_blks * 2 / 3)) || echo "file2 badly fragmented"
> 
> The 2/3 threshold was introduced in 9184ca155d7c ("xfs: test
> fragmentation characteristics of copy-on-write") as part of a series
> testing XFS's CoW extent size hint (cowextsize) mechanism. For btrfs,
> this threshold is arbitrary — btrfs doesn't have XFS's cowextsize hint,
> and its CoW extent allocation depends on factors like transaction commit
> frequency and metadata reservation behavior, which is exactly what the
> overcommit commit changed.
> 
> I see two possible fixes and would appreciate input on which is
> preferred:
> 
> Option A: _notrun for btrfs
> ----------------------------
> 
> Skip the entire test since the fragmentation threshold is not applicable
> to btrfs:
> 
>      test $FSTYP = "btrfs" && \
>          _notrun "CoW fragmentation threshold not applicable to btrfs"
> 
> Option B: Skip only the extent count assertion for btrfs
> ---------------------------------------------------------
> 
> Keep the CoW + data integrity portion of the test (the md5sum checks
> after random CoW writes and remount are still useful) and only skip the
> fragmentation assertion:
> 
>      if [ "$FSTYP" != "btrfs" ]; then
>          test $new_extents -lt $((internal_blks * 2 / 3)) || \
>              echo "file2 badly fragmented"
>      fi
> 
> I lean towards option B since the CoW write + remount + md5sum
> verification is still a reasonable smoke test, but option A is cleaner
> if the consensus is that this test isn't adding value for btrfs.

I also agree on the option B.

The fragmentation behavior is really specific to each fs, thus leaving 
the number of extents test to xfs, and keeping the contents check looks 
good to me.

Thanks,
Qu

> 
> Thoughts?
> 
> Thanks,
> Leo Martins
>