linux-xfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCHSET] xfs: improve online repair reap calculations
@ 2025-08-28 14:28 Darrick J. Wong
  2025-08-28 14:28 ` [PATCH 1/9] xfs: prepare reaping code for dynamic limits Darrick J. Wong
                   ` (8 more replies)
  0 siblings, 9 replies; 24+ messages in thread
From: Darrick J. Wong @ 2025-08-28 14:28 UTC (permalink / raw)
  To: cem, djwong; +Cc: stable, hch, linux-xfs

Hi all,

A few months ago, the multi-fsblock untorn writes patchset added a bunch
of log intent item helper functions to estimate the number of intent
items that could be added to a particular transaction.  Those helpers
enabled us to compute a safe upper bound on the number of blocks that
could be written in an untorn fashion with filesystem-provided out of
place writes.

Currently, the online fsck code employs static limits on the number of
intent items that it's willing to accrue to a single transaction when
it's trying to reap what it thinks are the old blocks from a corrupt
structure.  There have been no problems reported with this approach
after years of testing, but static limits are scary and gross because
overestimating the intent item limit could result in transaction
overflows and dead filesystems; and underestimating causes unnecessary
overhead.

This series uses the new log intent item size helpers to estimate the
limits dynamically based on worst-case per-block repair work vs. the
size of the scrub transaction.  After several months of testing this,
there don't seem to be any problems here either.

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

This has been running on the djcloud for months with no problems.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=fix-scrub-reap-calculations
---
Commits in this patchset:
 * xfs: prepare reaping code for dynamic limits
 * xfs: convert the ifork reap code to use xreap_state
 * xfs: use deferred intent items for reaping crosslinked blocks
 * xfs: compute per-AG extent reap limits dynamically
 * xfs: compute data device CoW staging extent reap limits dynamically
 * xfs: compute realtime device CoW staging extent reap limits dynamically
 * xfs: compute file mapping reap limits dynamically
 * xfs: remove static reap limits
 * xfs: use deferred reaping for data device cow extents
---
 fs/xfs/scrub/repair.h |    8 -
 fs/xfs/scrub/trace.h  |   45 ++++
 fs/xfs/scrub/newbt.c  |    7 +
 fs/xfs/scrub/reap.c   |  622 +++++++++++++++++++++++++++++++++++++++----------
 fs/xfs/scrub/trace.c  |    1 
 5 files changed, 552 insertions(+), 131 deletions(-)


^ permalink raw reply	[flat|nested] 24+ messages in thread
* [PATCHSET 6.18 v2 1/2] xfs: improve online repair reap calculations
@ 2025-09-05 15:55 Darrick J. Wong
  2025-09-05 15:57 ` [PATCH 7/9] xfs: compute file mapping reap limits dynamically Darrick J. Wong
  0 siblings, 1 reply; 24+ messages in thread
From: Darrick J. Wong @ 2025-09-05 15:55 UTC (permalink / raw)
  To: djwong, cem; +Cc: stable, hch, linux-xfs, hch

Hi all,

A few months ago, the multi-fsblock untorn writes patchset added a bunch
of log intent item helper functions to estimate the number of intent
items that could be added to a particular transaction.  Those helpers
enabled us to compute a safe upper bound on the number of blocks that
could be written in an untorn fashion with filesystem-provided out of
place writes.

Currently, the online fsck code employs static limits on the number of
intent items that it's willing to accrue to a single transaction when
it's trying to reap what it thinks are the old blocks from a corrupt
structure.  There have been no problems reported with this approach
after years of testing, but static limits are scary and gross because
overestimating the intent item limit could result in transaction
overflows and dead filesystems; and underestimating causes unnecessary
overhead.

This series uses the new log intent item size helpers to estimate the
limits dynamically based on worst-case per-block repair work vs. the
size of the scrub transaction.  After several months of testing this,
there don't seem to be any problems here either.

v2: rearrange patches, add review tags

If you're going to start using this code, I strongly recommend pulling
from my git trees, which are linked below.

This has been running on the djcloud for months with no problems.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=fix-scrub-reap-calculations
---
Commits in this patchset:
 * xfs: use deferred intent items for reaping crosslinked blocks
 * xfs: prepare reaping code for dynamic limits
 * xfs: convert the ifork reap code to use xreap_state
 * xfs: compute per-AG extent reap limits dynamically
 * xfs: compute data device CoW staging extent reap limits dynamically
 * xfs: compute realtime device CoW staging extent reap limits dynamically
 * xfs: compute file mapping reap limits dynamically
 * xfs: remove static reap limits from repair.h
 * xfs: use deferred reaping for data device cow extents
---
 fs/xfs/scrub/repair.h |    8 -
 fs/xfs/scrub/trace.h  |   45 ++++
 fs/xfs/scrub/newbt.c  |    9 +
 fs/xfs/scrub/reap.c   |  622 +++++++++++++++++++++++++++++++++++++++----------
 fs/xfs/scrub/trace.c  |    1 
 5 files changed, 554 insertions(+), 131 deletions(-)


^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2025-09-05 15:57 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-28 14:28 [PATCHSET] xfs: improve online repair reap calculations Darrick J. Wong
2025-08-28 14:28 ` [PATCH 1/9] xfs: prepare reaping code for dynamic limits Darrick J. Wong
2025-09-02  6:23   ` Christoph Hellwig
2025-08-28 14:28 ` [PATCH 2/9] xfs: convert the ifork reap code to use xreap_state Darrick J. Wong
2025-09-02  6:24   ` Christoph Hellwig
2025-08-28 14:28 ` [PATCH 3/9] xfs: use deferred intent items for reaping crosslinked blocks Darrick J. Wong
2025-09-02  6:25   ` Christoph Hellwig
2025-09-02 22:22     ` Darrick J. Wong
2025-08-28 14:28 ` [PATCH 4/9] xfs: compute per-AG extent reap limits dynamically Darrick J. Wong
2025-09-02  6:26   ` Christoph Hellwig
2025-08-28 14:29 ` [PATCH 5/9] xfs: compute data device CoW staging " Darrick J. Wong
2025-09-03  6:02   ` Christoph Hellwig
2025-08-28 14:29 ` [PATCH 6/9] xfs: compute realtime " Darrick J. Wong
2025-09-02  6:27   ` Christoph Hellwig
2025-08-28 14:29 ` [PATCH 7/9] xfs: compute file mapping " Darrick J. Wong
2025-09-02  6:27   ` Christoph Hellwig
2025-08-28 14:29 ` [PATCH 8/9] xfs: remove static reap limits Darrick J. Wong
2025-09-02  6:28   ` Christoph Hellwig
2025-09-02 22:32     ` Darrick J. Wong
2025-09-03  6:03       ` Christoph Hellwig
2025-09-03  6:27         ` Darrick J. Wong
2025-08-28 14:30 ` [PATCH 9/9] xfs: use deferred reaping for data device cow extents Darrick J. Wong
2025-09-02  6:28   ` Christoph Hellwig
  -- strict thread matches above, loose matches on Subject: below --
2025-09-05 15:55 [PATCHSET 6.18 v2 1/2] xfs: improve online repair reap calculations Darrick J. Wong
2025-09-05 15:57 ` [PATCH 7/9] xfs: compute file mapping reap limits dynamically Darrick J. Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).