[GIT PULLBOMB] xfsprogs: catch us up to 6.8, at least

[GIT PULLBOMB] xfsprogs: catch us up to 6.8, at least

[Darrick J. Wong]

Hi Carlos,

Please pull these branches containing a series of bugfixes, cleanups,
and conversions that didn't make it for 6.7; the 6.8 libxfs sync; and
then more fixes and cleanups for 6.8.  With these changes, xfs_repair
gains the ability to deal with extremely fragmented files; the ability
to rebuild a bmbt if the rmapbt data look intact; and slightly faster
btree index rebuilding.

Everyone else's patches notwithstanding, this gets xfsprogs to the point
where it will be ready for the 6.9 sync and more enhancements.

--D

[GIT PULL 01/11] xfsprogs: packaging fixes for 6.7

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 09ba6420a1ee2ca4bfc763e498b4ee6be415b131:

xfsprogs: Release v6.7.0 (2024-04-17 09:55:22 +0200)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/packaging-fixes-6.7_2024-04-17

for you to fetch changes up to d27e715c3081306e1b210e64d21775457c9f087a:

libxfs: fix incorrect porting to 6.7 (2024-04-17 14:06:22 -0700)

----------------------------------------------------------------
xfsprogs: packaging fixes for 6.7 [01/20]

This series fixes some bugs that I and others have found in the
userspace tools.  At this point 6.7 is released, so these target 6.8.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (2):
debian: fix package configuration after removing platform_defs.h.in
libxfs: fix incorrect porting to 6.7

db/check.c            | 1 -
debian/rules          | 6 ++++--
include/libxfs.h      | 4 ++++
libxfs/Makefile       | 1 +
libxfs/xfs_rtbitmap.c | 2 +-
libxfs/xfs_rtbitmap.h | 3 ---
repair/rt.c           | 1 -
7 files changed, 10 insertions(+), 8 deletions(-)

Re: [GIT PULL 01/11] xfsprogs: packaging fixes for 6.7

[Christoph Hellwig]

On Wed, Apr 17, 2024 at 03:07:35PM -0700, Darrick J. Wong wrote:
> Hi Carlos,
> 
> Please pull this branch with changes for xfsprogs for 6.6-rc1.

I think the scripts need a little tweak, anything from now on should
be for 6.8-rc or later.

Re: [GIT PULL 01/11] xfsprogs: packaging fixes for 6.7

[Carlos Maiolino]

On Wed, Apr 17, 2024 at 03:07:35PM -0700, Darrick J. Wong wrote:
> Hi Carlos,
> 
> Please pull this branch with changes for xfsprogs for 6.6-rc1.
> 
> As usual, I did a test-merge with the main upstream branch as of a few
> minutes ago, and didn't see any conflicts.  Please let me know if you
> encounter any problems.
> 
> The following changes since commit 09ba6420a1ee2ca4bfc763e498b4ee6be415b131:
> 
> xfsprogs: Release v6.7.0 (2024-04-17 09:55:22 +0200)
> 
> are available in the Git repository at:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/packaging-fixes-6.7_2024-04-17

I'm actually pulling this before 6.8 libxfs-sync, as it fixes the xfs_rtbitmap.h include mistake I
did previous sync, so I can synchronize libxfs with this already fixed.

Carlos

> 
> for you to fetch changes up to d27e715c3081306e1b210e64d21775457c9f087a:
> 
> libxfs: fix incorrect porting to 6.7 (2024-04-17 14:06:22 -0700)
> 
> ----------------------------------------------------------------
> xfsprogs: packaging fixes for 6.7 [01/20]
> 
> This series fixes some bugs that I and others have found in the
> userspace tools.  At this point 6.7 is released, so these target 6.8.
> 
> This has been running on the djcloud for months with no problems.  Enjoy!
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> 
> ----------------------------------------------------------------
> Darrick J. Wong (2):
> debian: fix package configuration after removing platform_defs.h.in
> libxfs: fix incorrect porting to 6.7
> 
> db/check.c            | 1 -
> debian/rules          | 6 ++++--
> include/libxfs.h      | 4 ++++
> libxfs/Makefile       | 1 +
> libxfs/xfs_rtbitmap.c | 2 +-
> libxfs/xfs_rtbitmap.h | 3 ---
> repair/rt.c           | 1 -
> 7 files changed, 10 insertions(+), 8 deletions(-)
> 
> 

[GIT PULL 02/11] xfsprogs: minor fixes for 6.7

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit d27e715c3081306e1b210e64d21775457c9f087a:

libxfs: fix incorrect porting to 6.7 (2024-04-17 14:06:22 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/random-fixes-6.7_2024-04-17

for you to fetch changes up to 94f4f0a7321d52edaa998367cccbe4dd16f1053a:

mkfs: fix log sunit rounding when external logs are in use (2024-04-17 14:06:22 -0700)

----------------------------------------------------------------
xfsprogs: minor fixes for 6.7 [02/20]

This series fixes some bugs that I and others have found in the
userspace tools.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (1):
mkfs: fix log sunit rounding when external logs are in use

mkfs/xfs_mkfs.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)

[GIT PULL 03/11] xfsprogs: convert utilities to use new rt helpers

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 94f4f0a7321d52edaa998367cccbe4dd16f1053a:

mkfs: fix log sunit rounding when external logs are in use (2024-04-17 14:06:22 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/fix-realtime-units-6.7_2024-04-17

for you to fetch changes up to 9c69d1c725391f9a65fa8d6d2be337466918e248:

xfs_{db,repair}: use m_blockwsize instead of sb_blocksize for rt blocks (2024-04-17 14:06:23 -0700)

----------------------------------------------------------------
xfsprogs: convert utilities to use new rt helpers [03/20]

The patches in this series clean up a lot of realtime space usage code
the userspace utilities.  This involves correcting incorrect type usage,
renaming variables to reflect their actual usage; and converting open
code logic to use the new helpers that were just added to libxfs.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (11):
xfs_repair: fix confusing rt space units in the duplicate detection code
libxfs: create a helper to compute leftovers of realtime extents
libxfs: use helpers to convert rt block numbers to rt extent numbers
xfs_repair: convert utility to use new rt extent helpers and types
mkfs: convert utility to use new rt extent helpers and types
xfs_{db,repair}: convert open-coded xfs_rtword_t pointer accesses to helper
xfs_repair: convert helpers for rtbitmap block/wordcount computations
xfs_{db,repair}: use accessor functions for bitmap words
xfs_{db,repair}: use helpers for rtsummary block/wordcount computations
xfs_{db,repair}: use accessor functions for summary info words
xfs_{db,repair}: use m_blockwsize instead of sb_blocksize for rt blocks

db/check.c               | 89 +++++++++++++++++++++++++++++++++++++-----------
libxfs/init.c            |  8 ++---
libxfs/libxfs_api_defs.h |  8 +++++
libxfs/logitem.c         |  3 +-
libxfs/trans.c           |  3 +-
mkfs/proto.c             | 41 +++++++++++++++-------
repair/agheader.h        |  2 +-
repair/dinode.c          | 21 +++++++-----
repair/globals.c         |  4 +--
repair/globals.h         |  4 +--
repair/incore.c          | 16 ++++-----
repair/incore.h          | 15 ++++----
repair/incore_ext.c      | 74 +++++++++++++++++++++-------------------
repair/phase4.c          | 16 ++++-----
repair/phase6.c          | 28 +++++++++++----
repair/rt.c              | 63 +++++++++++++++++++++++-----------
repair/rt.h              |  6 ++--
repair/scan.c            |  2 +-
18 files changed, 261 insertions(+), 142 deletions(-)

[GIT PULL 04/11] libxfs: sync with 6.8

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 9c69d1c725391f9a65fa8d6d2be337466918e248:

xfs_{db,repair}: use m_blockwsize instead of sb_blocksize for rt blocks (2024-04-17 14:06:23 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/libxfs-sync-6.8_2024-04-17

for you to fetch changes up to a6e7d369929423b10df05d11dddd1dd9550e4f20:

xfs: remove conditional building of rt geometry validator functions (2024-04-17 14:06:27 -0700)

----------------------------------------------------------------
libxfs: sync with 6.8 [04/20]

Synchronize libxfs with the kernel.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Andrey Albershteyn (1):
xfs: reset XFS_ATTR_INCOMPLETE filter on node removal

Christoph Hellwig (24):
xfs: move xfs_ondisk.h to libxfs/
xfs: consolidate the xfs_attr_defer_* helpers
xfs: store an ops pointer in struct xfs_defer_pending
xfs: pass the defer ops instead of type to xfs_defer_start_recovery
xfs: pass the defer ops directly to xfs_defer_add
xfs: remove the xfs_alloc_arg argument to xfs_bmap_btalloc_accounting
xfs: also use xfs_bmap_btalloc_accounting for RT allocations
xfs: return -ENOSPC from xfs_rtallocate_*
xfs: indicate if xfs_bmap_adjacent changed ap->blkno
xfs: move xfs_rtget_summary to xfs_rtbitmap.c
xfs: split xfs_rtmodify_summary_int
xfs: remove rt-wrappers from xfs_format.h
xfs: remove XFS_RTMIN/XFS_RTMAX
xfs: make if_data a void pointer
xfs: return if_data from xfs_idata_realloc
xfs: move the xfs_attr_sf_lookup tracepoint
xfs: simplify xfs_attr_sf_findname
xfs: remove xfs_attr_shortform_lookup
xfs: use xfs_attr_sf_findname in xfs_attr_shortform_getvalue
xfs: remove struct xfs_attr_shortform
xfs: remove xfs_attr_sf_hdr_t
xfs: turn the XFS_DA_OP_REPLACE checks in xfs_attr_shortform_addname into asserts
xfs: fix a use after free in xfs_defer_finish_recovery
xfs: use the op name in trace_xlog_intent_recovery_failed

Darrick J. Wong (37):
xfs: use xfs_defer_pending objects to recover intent items
xfs: recreate work items when recovering intent items
xfs: use xfs_defer_finish_one to finish recovered work items
xfs: move ->iop_recover to xfs_defer_op_type
xfs: hoist intent done flag setting to ->finish_item callsite
xfs: hoist ->create_intent boilerplate to its callsite
xfs: use xfs_defer_create_done for the relogging operation
xfs: clean out XFS_LI_DIRTY setting boilerplate from ->iop_relog
xfs: hoist xfs_trans_add_item calls to defer ops functions
xfs: move ->iop_relog to struct xfs_defer_op_type
xfs: make rextslog computation consistent with mkfs
xfs: fix 32-bit truncation in xfs_compute_rextslog
xfs: don't allow overly small or large realtime volumes
xfs: elide ->create_done calls for unlogged deferred work
xfs: don't append work items to logged xfs_defer_pending objects
xfs: allow pausing of pending deferred work items
xfs: remove __xfs_free_extent_later
xfs: automatic freeing of freshly allocated unwritten space
xfs: remove unused fields from struct xbtree_ifakeroot
xfs: force small EFIs for reaping btree extents
xfs: force all buffers to be written during btree bulk load
xfs: set XBF_DONE on newly formatted btree block that are ready for writing
xfs: read leaf blocks when computing keys for bulkloading into node blocks
xfs: move btree bulkload record initialization to ->get_record implementations
xfs: constrain dirty buffers while formatting a staged btree
xfs: repair free space btrees
xfs: repair inode btrees
xfs: repair refcount btrees
xfs: dont cast to char * for XFS_DFORK_*PTR macros
xfs: set inode sick state flags when we zap either ondisk fork
xfs: zap broken inode forks
xfs: repair inode fork block mapping data structures
xfs: create a ranged query function for refcount btrees
xfs: create a new inode fork block unmap helper
xfs: improve dquot iteration for scrub
xfs: fix backwards logic in xfs_bmap_alloc_account
xfs: remove conditional building of rt geometry validator functions

Jiachen Zhang (1):
xfs: ensure logflagsp is initialized in xfs_bmap_del_extent_real

Long Li (2):
xfs: add lock protection when remove perag from radix tree
xfs: fix perag leak when growfs fails

Zhang Tianci (2):
xfs: update dir3 leaf block metadata after swap
xfs: extract xfs_da_buf_copy() helper function

db/attrshort.c              |  35 ++--
db/check.c                  |  12 +-
db/inode.c                  |   6 +-
db/metadump.c               |  16 +-
db/namei.c                  |   4 +-
include/list.h              |  14 ++
include/xfs_trace.h         |   5 +
include/xfs_trans.h         |   1 -
libxfs/defer_item.c         |  16 +-
libxfs/init.c               |   6 +
libxfs/libxfs_api_defs.h    |   4 +
libxfs/libxfs_io.h          |  11 ++
libxfs/libxfs_priv.h        |   7 +-
libxfs/util.c               |   2 +-
libxfs/xfs_ag.c             |  38 +++-
libxfs/xfs_ag.h             |  12 ++
libxfs/xfs_ag_resv.c        |   2 +
libxfs/xfs_alloc.c          | 116 ++++++++++--
libxfs/xfs_alloc.h          |  24 +--
libxfs/xfs_alloc_btree.c    |  13 +-
libxfs/xfs_attr.c           | 131 ++++---------
libxfs/xfs_attr_leaf.c      | 244 ++++++++----------------
libxfs/xfs_attr_leaf.h      |   8 +-
libxfs/xfs_attr_sf.h        |  24 ++-
libxfs/xfs_bmap.c           | 201 ++++++++++++--------
libxfs/xfs_bmap.h           |   9 +-
libxfs/xfs_bmap_btree.c     | 124 ++++++++++--
libxfs/xfs_bmap_btree.h     |   5 +
libxfs/xfs_btree.c          |  28 ++-
libxfs/xfs_btree.h          |   5 +
libxfs/xfs_btree_staging.c  |  89 ++++++---
libxfs/xfs_btree_staging.h  |  33 ++--
libxfs/xfs_da_btree.c       |  69 +++----
libxfs/xfs_da_btree.h       |   2 +
libxfs/xfs_da_format.h      |  31 +--
libxfs/xfs_defer.c          | 452 ++++++++++++++++++++++++++++++++++++--------
libxfs/xfs_defer.h          |  59 ++++--
libxfs/xfs_dir2.c           |   2 +-
libxfs/xfs_dir2_block.c     |   6 +-
libxfs/xfs_dir2_priv.h      |   3 +-
libxfs/xfs_dir2_sf.c        |  91 ++++-----
libxfs/xfs_format.h         |  19 +-
libxfs/xfs_health.h         |  10 +
libxfs/xfs_ialloc.c         |  36 ++--
libxfs/xfs_ialloc.h         |   3 +-
libxfs/xfs_ialloc_btree.c   |   2 +-
libxfs/xfs_iext_tree.c      |  59 +++---
libxfs/xfs_inode_fork.c     |  78 ++++----
libxfs/xfs_inode_fork.h     |  13 +-
libxfs/xfs_ondisk.h         | 199 +++++++++++++++++++
libxfs/xfs_refcount.c       |  57 +++++-
libxfs/xfs_refcount.h       |  12 +-
libxfs/xfs_refcount_btree.c |  15 +-
libxfs/xfs_rmap.c           |   2 +-
libxfs/xfs_rtbitmap.c       | 134 ++++++-------
libxfs/xfs_rtbitmap.h       |   4 +-
libxfs/xfs_sb.c             |  20 +-
libxfs/xfs_sb.h             |   2 +
libxfs/xfs_shared.h         |   2 +-
libxfs/xfs_symlink_remote.c |  12 +-
libxfs/xfs_types.h          |  20 +-
mkfs/proto.c                |   4 +-
mkfs/xfs_mkfs.c             |   8 +-
repair/agbtree.c            |  57 ++++--
repair/attr_repair.c        |  48 +++--
repair/dinode.c             |  23 +--
repair/phase6.c             |   9 +-
repair/rt.c                 |   6 +-
repair/sb.c                 |   6 +-
69 files changed, 1861 insertions(+), 959 deletions(-)
create mode 100644 libxfs/xfs_ondisk.h

[GIT PULL 05/11] xfs_repair: faster btree bulkloading

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit a6e7d369929423b10df05d11dddd1dd9550e4f20:

xfs: remove conditional building of rt geometry validator functions (2024-04-17 14:06:27 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-bulkload-faster-6.8_2024-04-17

for you to fetch changes up to 2f2e6b36a22c510964fa920b15726526aa102e2c:

xfs_repair: bulk load records into new btree blocks (2024-04-17 14:06:27 -0700)

----------------------------------------------------------------
xfs_repair: faster btree bulkloading [05/20]

Two improvements for xfs_repair: first, we adjust the btree bulk
loader's slack computation to match the kernel.  Second, we make the
bulk loader write as many records as w can per ->get_records call.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (2):
xfs_repair: adjust btree bulkloading slack computations to match online repair
xfs_repair: bulk load records into new btree blocks

repair/agbtree.c  | 161 ++++++++++++++++++++++++++++++------------------------
repair/bulkload.c |   9 +--
2 files changed, 95 insertions(+), 75 deletions(-)

[GIT PULL 06/11] xfsprogs: bug fixes for 6.8

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 2f2e6b36a22c510964fa920b15726526aa102e2c:

xfs_repair: bulk load records into new btree blocks (2024-04-17 14:06:27 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/xfsprogs-fixes-6.8_2024-04-17

for you to fetch changes up to 16a75c71b4c04373c0c726b8e6ddfd030a81c93e:

xfs_io: add linux madvise advice codes (2024-04-17 14:06:27 -0700)

----------------------------------------------------------------
xfsprogs: bug fixes for 6.8 [06/20]

Bug fixes for xfsprogs for 6.8.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (5):
xfs_repair: double-check with shortform attr verifiers
xfs_db: improve number extraction in getbitval
xfs_scrub: fix threadcount estimates for phase 6
xfs_scrub: don't fail while reporting media scan errors
xfs_io: add linux madvise advice codes

db/bit.c             | 37 +++++++++++--------------
io/madvise.c         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++-
repair/attr_repair.c | 17 ++++++++++++
scrub/phase6.c       | 36 ++++++++++++++++++------
4 files changed, 137 insertions(+), 30 deletions(-)

[GIT PULL 07/11] xfsprogs: fix log sector size detection

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 16a75c71b4c04373c0c726b8e6ddfd030a81c93e:

xfs_io: add linux madvise advice codes (2024-04-17 14:06:27 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/mkfs-fix-log-sector-size-6.8_2024-04-17

for you to fetch changes up to 53bf0604e104bc053778495faef94b3570582aac:

mkfs: use a sensible log sector size default (2024-04-17 14:06:27 -0700)

----------------------------------------------------------------
xfsprogs: fix log sector size detection [V3 07/20]

From Christoph Hellwig,

this series cleans up the libxfs toplogy code and then fixes detection
of the log sector size in mkfs.xfs, so that it doesn't create smaller
than possible log sectors by default on > 512 byte sector size devices.

Note that this doesn't cleanup the types of the topology members, as
that creeps all the way into platform_findsize.  Which has a lot more
cruft that should be dealth with and is worth it's own series.

Changes since v2:
- rebased to the lastest for-next branch

Changes since v1:
- fix a spelling mistake
- add a few more cleanups

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (5):
libxfs: remove the unused fs_topology_t typedef
libxfs: refactor the fs_topology structure
libxfs: remove the S_ISREG check from blkid_get_topology
libxfs: also query log device topology in get_topology
mkfs: use a sensible log sector size default

libxfs/topology.c | 109 +++++++++++++++++++++++++++---------------------------
libxfs/topology.h |  19 ++++++----
mkfs/xfs_mkfs.c   |  71 +++++++++++++++++------------------
repair/sb.c       |   2 +-
4 files changed, 100 insertions(+), 101 deletions(-)

[GIT PULL 08/11] mkfs: scale shards on ssds

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 53bf0604e104bc053778495faef94b3570582aac:

mkfs: use a sensible log sector size default (2024-04-17 14:06:27 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/mkfs-scale-geo-on-ssds-6.8_2024-04-17

for you to fetch changes up to c02a18733fc0a0e1b607f75e90962b3adc27c8fa:

mkfs: allow sizing internal logs for concurrency (2024-04-17 14:06:27 -0700)

----------------------------------------------------------------
mkfs: scale shards on ssds [08/20]

For a long time, the maintainers have had a gut feeling that we could
optimize performance of XFS filesystems on non-mechanical storage by
scaling the number of allocation groups to be a multiple of the CPU
count.

With modern ~2022 hardware, it is common for systems to have more than
four CPU cores and non-striped SSDs ranging in size from 256GB to 4TB.
The default mkfs geometry still defaults to 4 AGs regardless of core
count, which was settled on in the age of spinning rust.

This patchset adds a different computation for AG count and log size
that is based entirely on a desired level of concurrency.  If we detect
storage that is non-rotational (or the sysadmin provides a CLI option),
then we will try to match the AG count to the CPU count to minimize AGF
contention and make the log large enough to minimize grant head
contention.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (2):
mkfs: allow sizing allocation groups for concurrency
mkfs: allow sizing internal logs for concurrency

man/man8/mkfs.xfs.8.in |  46 +++++++++
mkfs/xfs_mkfs.c        | 251 +++++++++++++++++++++++++++++++++++++++++++++++--
2 files changed, 291 insertions(+), 6 deletions(-)

[GIT PULL 09/11] xfs_scrub: scan metadata files in parallel

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit c02a18733fc0a0e1b607f75e90962b3adc27c8fa:

mkfs: allow sizing internal logs for concurrency (2024-04-17 14:06:27 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-metafile-parallel-6.8_2024-04-17

for you to fetch changes up to bd35f31ce91bcf5ed9370e94a4a5da89638f37f4:

xfs_scrub: scan whole-fs metadata files in parallel (2024-04-17 14:06:27 -0700)

----------------------------------------------------------------
xfs_scrub: scan metadata files in parallel [v30.3 09/20]

At this point, we need to clean up the libfrog and xfs_scrub code a
little bit.  First, correct some of the weird naming and organizing
choices I made in libfrog for scrub types and fs summary counter scans.
Second, break out metadata file scans as a separate group, and teach
xfs_scrub that it can ask the kernel to scan them in parallel.  On
filesystems with quota or realtime volumes, this can speed up that part
significantly.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
libfrog: rename XFROG_SCRUB_TYPE_* to XFROG_SCRUB_GROUP_*
libfrog: promote XFROG_SCRUB_DESCR_SUMMARY to a scrub type
xfs_scrub: scan whole-fs metadata files in parallel

io/scrub.c      |  13 +++---
libfrog/scrub.c |  51 +++++++++++----------
libfrog/scrub.h |  24 ++++------
scrub/phase2.c  | 135 ++++++++++++++++++++++++++++++++++++++++++--------------
scrub/phase4.c  |   2 +-
scrub/phase7.c  |   4 +-
scrub/scrub.c   |  75 +++++++++++++++++--------------
scrub/scrub.h   |   6 ++-
8 files changed, 194 insertions(+), 116 deletions(-)

[GIT PULL 10/11] xfs_repair: rebuild inode fork mappings

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit bd35f31ce91bcf5ed9370e94a4a5da89638f37f4:

xfs_scrub: scan whole-fs metadata files in parallel (2024-04-17 14:06:27 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-rebuild-forks-6.8_2024-04-17

for you to fetch changes up to b3bcb8f0a8b5763defc09bc6d9a04da275ad780a:

xfs_repair: rebuild block mappings from rmapbt data (2024-04-17 14:06:28 -0700)

----------------------------------------------------------------
xfs_repair: rebuild inode fork mappings [v30.3 10/20]

Add the ability to regenerate inode fork mappings if the rmapbt
otherwise looks ok.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
xfs_repair: push inode buf and dinode pointers all the way to inode fork processing
xfs_repair: sync bulkload data structures with kernel newbt code
xfs_repair: rebuild block mappings from rmapbt data

include/xfs_trans.h      |   2 +
libfrog/util.h           |   5 +
libxfs/libxfs_api_defs.h |  16 +-
libxfs/trans.c           |  48 +++
repair/Makefile          |   2 +
repair/agbtree.c         |  24 +-
repair/bmap_repair.c     | 748 +++++++++++++++++++++++++++++++++++++++++++++++
repair/bmap_repair.h     |  13 +
repair/bulkload.c        | 260 ++++++++++++++--
repair/bulkload.h        |  34 ++-
repair/dino_chunks.c     |   5 +-
repair/dinode.c          | 142 ++++++---
repair/dinode.h          |   7 +-
repair/phase5.c          |   2 +-
repair/rmap.c            |   2 +-
repair/rmap.h            |   1 +
16 files changed, 1231 insertions(+), 80 deletions(-)
create mode 100644 repair/bmap_repair.c
create mode 100644 repair/bmap_repair.h

[GIT PULL 11/11] xfs_repair: support more than 4 billion records

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit b3bcb8f0a8b5763defc09bc6d9a04da275ad780a:

xfs_repair: rebuild block mappings from rmapbt data (2024-04-17 14:06:28 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-support-4bn-records-6.8_2024-04-17

for you to fetch changes up to 90ee2c3a94511da87929989a06199fd537c94db4:

xfs_repair: support more than INT_MAX block maps (2024-04-17 14:06:28 -0700)

----------------------------------------------------------------
xfs_repair: support more than 4 billion records [11/20]

I started looking through all the places where XFS has to deal with the
rc_refcount attribute of refcount records, and noticed that offline
repair doesn't handle the situation where there are more than 2^32
reverse mappings in an AG, or that there are more than 2^32 owners of a
particular piece of AG space.  I've estimated that it would take several
months to produce a filesystem with this many records, but we really
ought to do better at handling them than crashing or (worse) not
crashing and writing out corrupt btrees due to integer truncation.

Once I started using the bmap_inflate debugger command to create extreme
reflink scenarios, I noticed that the memory usage of xfs_repair was
astronomical.  This I observed to be due to the fact that it allocates a
single huge block mapping array for all files on the system, even though
it only uses that array for data and attr forks that map metadata blocks
(e.g. directories, xattrs, symlinks) and does not use it for regular
data files.

So I got rid of the 2^31-1 limits on the block map array and turned off
the block mapping for regular data files.  This doesn't answer the
question of what to do if there are a lot of extents, but it kicks the
can down the road until someone creates a maximally sized xattr tree,
which so far nobody's ever stuck to long enough to complain about.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (8):
xfs_db: add a bmbt inflation command
xfs_repair: slab and bag structs need to track more than 2^32 items
xfs_repair: support more than 2^32 rmapbt records per AG
xfs_repair: support more than 2^32 owners per physical block
xfs_repair: clean up lock resources
xfs_repair: constrain attr fork extent count
xfs_repair: don't create block maps for data files
xfs_repair: support more than INT_MAX block maps

db/Makefile       |  65 ++++++-
db/bmap_inflate.c | 551 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
db/command.c      |   1 +
db/command.h      |   1 +
man/man8/xfs_db.8 |  23 +++
repair/bmap.c     |  23 +--
repair/bmap.h     |   7 +-
repair/dinode.c   |  18 +-
repair/dir2.c     |   2 +-
repair/incore.c   |   9 +
repair/rmap.c     |  25 ++-
repair/rmap.h     |   4 +-
repair/slab.c     |  36 ++--
repair/slab.h     |  36 ++--
14 files changed, 725 insertions(+), 76 deletions(-)
create mode 100644 db/bmap_inflate.c