[GIT PULLBOMB v30.5] xfsprogs: catch us up to 6.9 (final)

[GIT PULLBOMB v30.5] xfsprogs: catch us up to 6.9 (final)

[Darrick J. Wong]

Hi Carlos,

Please accept these pull requests for all the xfsprogs 6.9 patches that
have gone through review.  The only patch remaining from my stack is the
one about null pointers being passed to duration() in xfs_repair:

https://lore.kernel.org/linux-xfs/20240603161446.GA52987@frogsfrogsfrogs/T/#u

(Just thought I'd mention it since there's been a lot of noise today.)

--D

[GIT PULL 01/10] libxfs: prepare to sync with 6.9

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit df4bd2d27189a98711fd35965c18bee25a25a9ea:

xfsprogs: Release v6.8.0 (2024-05-17 13:20:34 +0200)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/libxfs-6.9-sync-prep_2024-06-03

for you to fetch changes up to 493d04e97479b36f34a895171ed852cd2e3298b2:

libfrog: create a new scrub group for things requiring full inode scans (2024-06-03 11:37:35 -0700)

----------------------------------------------------------------
libxfs: prepare to sync with 6.9 [v30.5 01/35]

Apply some cleanups to libxfs before we synchronize it with the kernel.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
libxfs: actually set m_fsname
libxfs: clean up xfs_da_unmount usage
libfrog: create a new scrub group for things requiring full inode scans

io/scrub.c        |  1 +
libfrog/scrub.h   |  1 +
libxfs/init.c     | 24 +++++++++++++++++-------
scrub/phase5.c    | 22 ++++++++++++++++++++--
scrub/scrub.c     | 33 +++++++++++++++++++++++++++++++++
scrub/scrub.h     |  1 +
scrub/xfs_scrub.h |  1 +
7 files changed, 74 insertions(+), 9 deletions(-)

[GIT PULL 02/10] libxfs: sync with 6.9

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 493d04e97479b36f34a895171ed852cd2e3298b2:

libfrog: create a new scrub group for things requiring full inode scans (2024-06-03 11:37:35 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/libxfs-sync-6.9_2024-06-03

for you to fetch changes up to a5974a2d10bdbfbfff9a870b832f2ca2b711ec14:

xfs: allow sunit mount option to repair bad primary sb stripe values (2024-06-03 11:37:41 -0700)

----------------------------------------------------------------
libxfs: sync with 6.9 [v30.5 02/35]

Synchronize libxfs with the kernel.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (44):
xfs: remove bc_ino.flags
xfs: consolidate the xfs_alloc_lookup_* helpers
xfs: turn the allocbt cursor active field into a btree flag
xfs: move the btree stats offset into struct btree_ops
xfs: split out a btree type from the btree ops geometry flags
xfs: split the per-btree union in struct xfs_btree_cur
xfs: move comment about two 2 keys per pointer in the rmap btree
xfs: add a xfs_btree_init_ptr_from_cur
xfs: don't override bc_ops for staging btrees
xfs: fold xfs_allocbt_init_common into xfs_allocbt_init_cursor
xfs: remove xfs_allocbt_stage_cursor
xfs: fold xfs_inobt_init_common into xfs_inobt_init_cursor
xfs: remove xfs_inobt_stage_cursor
xfs: fold xfs_refcountbt_init_common into xfs_refcountbt_init_cursor
xfs: remove xfs_refcountbt_stage_cursor
xfs: fold xfs_rmapbt_init_common into xfs_rmapbt_init_cursor
xfs: remove xfs_rmapbt_stage_cursor
xfs: make full use of xfs_btree_stage_ifakeroot in xfs_bmbt_stage_cursor
xfs: fold xfs_bmbt_init_common into xfs_bmbt_init_cursor
xfs: remove xfs_bmbt_stage_cursor
xfs: split the agf_roots and agf_levels arrays
xfs: add a name field to struct xfs_btree_ops
xfs: add a sick_mask to struct xfs_btree_ops
xfs: split xfs_allocbt_init_cursor
xfs: remove xfs_inobt_cur
xfs: remove the btnum argument to xfs_inobt_count_blocks
xfs: split xfs_inobt_insert_sprec
xfs: split xfs_inobt_init_cursor
xfs: pass a 'bool is_finobt' to xfs_inobt_insert
xfs: remove xfs_btnum_t
xfs: simplify xfs_btree_check_sblock_siblings
xfs: simplify xfs_btree_check_lblock_siblings
xfs: open code xfs_btree_check_lptr in xfs_bmap_btree_to_extents
xfs: consolidate btree ptr checking
xfs: misc cleanups for __xfs_btree_check_sblock
xfs: remove the crc variable in __xfs_btree_check_lblock
xfs: tighten up validation of root block in inode forks
xfs: consolidate btree block verification
xfs: rename btree helpers that depends on the block number representation
xfs: factor out a __xfs_btree_check_lblock_hdr helper
xfs: remove xfs_btree_reada_bufl
xfs: remove xfs_btree_reada_bufs
xfs: move and rename xfs_btree_read_bufl
xfs: add a xfs_btree_ptrs_equal helper

Darrick J. Wong (56):
xfs: create a static name for the dot entry too
xfs: create a predicate to determine if two xfs_names are the same
xfs: create a macro for decoding ftypes in tracepoints
xfs: report the health of quota counts
xfs: implement live quotacheck inode scan
xfs: report health of inode link counts
xfs: teach scrub to check file nlinks
xfs: separate the marking of sick and checked metadata
xfs: report fs corruption errors to the health tracking system
xfs: report ag header corruption errors to the health tracking system
xfs: report block map corruption errors to the health tracking system
xfs: report btree block corruption errors to the health system
xfs: report dir/attr block corruption errors to the health system
xfs: report inode corruption errors to the health system
xfs: report realtime metadata corruption errors to the health system
xfs: report XFS_IS_CORRUPT errors to the health system
xfs: add secondary and indirect classes to the health tracking system
xfs: remember sick inodes that get inactivated
xfs: update health status if we get a clean bill of health
xfs: consolidate btree block freeing tracepoints
xfs: consolidate btree block allocation tracepoints
xfs: set the btree cursor bc_ops in xfs_btree_alloc_cursor
xfs: drop XFS_BTREE_CRC_BLOCKS
xfs: encode the btree geometry flags in the btree ops structure
xfs: extern some btree ops structures
xfs: initialize btree blocks using btree_ops structure
xfs: rename btree block/buffer init functions
xfs: btree convert xfs_btree_init_block to xfs_btree_init_buf calls
xfs: remove the unnecessary daddr paramter to _init_block
xfs: set btree block buffer ops in _init_buf
xfs: move lru refs to the btree ops structure
xfs: factor out a xfs_btree_owner helper
xfs: factor out a btree block owner check
xfs: store the btree pointer length in struct xfs_btree_ops
xfs: create predicate to determine if cursor is at inode root level
xfs: make staging file forks explicit
libxfs: teach buftargs to maintain their own buffer hashtable
libxfs: add xfile support
libxfs: partition memfd files to avoid using too many fds
xfs: teach buftargs to maintain their own buffer hashtable
libxfs: support in-memory buffer cache targets
xfs: support in-memory btrees
xfs: launder in-memory btree buffers before transaction commit
xfs: create a helper to decide if a file mapping targets the rt volume
xfs: repair the rmapbt
xfs: create a shadow rmap btree during rmap repair
xfs: hook live rmap operations during a repair operation
xfs: clean up bmap log intent item tracepoint callsites
xfs: move xfs_bmap_defer_add to xfs_bmap_item.c
xfs: fix xfs_bunmapi to allow unmapping of partial rt extents
xfs: add a realtime flag to the bmap update log redo items
xfs: support deferred bmap updates on the attr fork
xfs: xfs_bmap_finish_one should map unwritten extents properly
xfs: move xfs_symlink_remote.c declarations to xfs_symlink_remote.h
xfs: move remote symlink target read function to libxfs
xfs: move symlink target write function to libxfs

Dave Chinner (10):
xfs: convert kmem_zalloc() to kzalloc()
xfs: convert kmem_alloc() to kmalloc()
xfs: convert remaining kmem_free() to kfree()
xfs: use __GFP_NOLOCKDEP instead of GFP_NOFS
xfs: use GFP_KERNEL in pure transaction contexts
xfs: clean up remaining GFP_NOFS users
xfs: use xfs_defer_alloc a bit more
xfs: xfs_btree_bload_prep_block() should use __GFP_NOFAIL
xfs: shrink failure needs to hold AGI buffer
xfs: allow sunit mount option to repair bad primary sb stripe values

Matthew Wilcox (Oracle) (1):
xfs: Replace xfs_isilocked with xfs_assert_ilocked

copy/xfs_copy.c             |    4 +-
db/agf.c                    |   28 +-
db/bmap_inflate.c           |    8 +-
db/check.c                  |   14 +-
db/freesp.c                 |    8 +-
db/metadump.c               |   12 +-
include/kmem.h              |    5 +-
include/libxfs.h            |    4 +-
include/xfs_mount.h         |    5 +
include/xfs_trace.h         |   17 +-
include/xfs_trans.h         |    1 +
libxfs/Makefile             |    9 +-
libxfs/buf_mem.c            |  313 ++++++++++++
libxfs/buf_mem.h            |   30 ++
libxfs/defer_item.c         |   15 +-
libxfs/defer_item.h         |   13 +
libxfs/init.c               |   52 +-
libxfs/libxfs_api_defs.h    |   10 +
libxfs/libxfs_io.h          |   42 +-
libxfs/libxfs_priv.h        |   19 +-
libxfs/logitem.c            |    2 +-
libxfs/rdwr.c               |   86 ++--
libxfs/trans.c              |   40 ++
libxfs/util.c               |   10 +
libxfs/xfile.c              |  393 ++++++++++++++++
libxfs/xfile.h              |   34 ++
libxfs/xfs_ag.c             |   79 ++--
libxfs/xfs_ag.h             |   18 +-
libxfs/xfs_alloc.c          |  258 ++++++----
libxfs/xfs_alloc_btree.c    |  191 ++++----
libxfs/xfs_alloc_btree.h    |   10 +-
libxfs/xfs_attr.c           |    5 +-
libxfs/xfs_attr_leaf.c      |   22 +-
libxfs/xfs_attr_remote.c    |   37 +-
libxfs/xfs_bmap.c           |  365 +++++++++-----
libxfs/xfs_bmap.h           |   19 +-
libxfs/xfs_bmap_btree.c     |  152 +++---
libxfs/xfs_bmap_btree.h     |    5 +-
libxfs/xfs_btree.c          | 1097 ++++++++++++++++++++++++++-----------------
libxfs/xfs_btree.h          |  274 +++++------
libxfs/xfs_btree_mem.c      |  346 ++++++++++++++
libxfs/xfs_btree_mem.h      |   75 +++
libxfs/xfs_btree_staging.c  |  133 +-----
libxfs/xfs_btree_staging.h  |   10 +-
libxfs/xfs_da_btree.c       |   59 ++-
libxfs/xfs_da_format.h      |   11 +
libxfs/xfs_defer.c          |   25 +-
libxfs/xfs_dir2.c           |   59 +--
libxfs/xfs_dir2.h           |   13 +
libxfs/xfs_dir2_block.c     |    8 +-
libxfs/xfs_dir2_data.c      |    3 +
libxfs/xfs_dir2_leaf.c      |    3 +
libxfs/xfs_dir2_node.c      |    7 +
libxfs/xfs_dir2_sf.c        |   16 +-
libxfs/xfs_format.h         |   21 +-
libxfs/xfs_fs.h             |    8 +-
libxfs/xfs_health.h         |   95 +++-
libxfs/xfs_ialloc.c         |  232 ++++++---
libxfs/xfs_ialloc_btree.c   |  173 ++++---
libxfs/xfs_ialloc_btree.h   |   11 +-
libxfs/xfs_iext_tree.c      |   26 +-
libxfs/xfs_inode_buf.c      |   12 +-
libxfs/xfs_inode_fork.c     |   49 +-
libxfs/xfs_inode_fork.h     |    1 +
libxfs/xfs_log_format.h     |    4 +-
libxfs/xfs_refcount.c       |   69 ++-
libxfs/xfs_refcount_btree.c |   78 ++-
libxfs/xfs_refcount_btree.h |    2 -
libxfs/xfs_rmap.c           |  284 ++++++++---
libxfs/xfs_rmap.h           |   31 +-
libxfs/xfs_rmap_btree.c     |  240 ++++++++--
libxfs/xfs_rmap_btree.h     |    8 +-
libxfs/xfs_rtbitmap.c       |   11 +-
libxfs/xfs_sb.c             |   42 +-
libxfs/xfs_sb.h             |    5 +-
libxfs/xfs_shared.h         |   67 ++-
libxfs/xfs_symlink_remote.c |  155 +++++-
libxfs/xfs_symlink_remote.h |   26 +
libxfs/xfs_trans_inode.c    |    6 +-
libxfs/xfs_types.h          |   26 +-
logprint/log_misc.c         |    8 +-
logprint/log_print_all.c    |    8 +-
mkfs/xfs_mkfs.c             |    8 +-
repair/agbtree.c            |   28 +-
repair/bmap_repair.c        |    4 +-
repair/bulkload.c           |    2 +-
repair/phase5.c             |   28 +-
repair/phase6.c             |    4 -
repair/prefetch.c           |   12 +-
repair/prefetch.h           |    1 +
repair/progress.c           |   14 +-
repair/progress.h           |    2 +-
repair/scan.c               |   18 +-
repair/xfs_repair.c         |   47 +-
94 files changed, 4425 insertions(+), 1915 deletions(-)
create mode 100644 libxfs/buf_mem.c
create mode 100644 libxfs/buf_mem.h
create mode 100644 libxfs/defer_item.h
create mode 100644 libxfs/xfile.c
create mode 100644 libxfs/xfile.h
create mode 100644 libxfs/xfs_btree_mem.c
create mode 100644 libxfs/xfs_btree_mem.h
create mode 100644 libxfs/xfs_symlink_remote.h

[GIT PULL 03/10] xfsprogs: bmap log intent cleanups

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit a5974a2d10bdbfbfff9a870b832f2ca2b711ec14:

xfs: allow sunit mount option to repair bad primary sb stripe values (2024-06-03 11:37:41 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/bmap-intent-cleanups-6.9_2024-06-03

for you to fetch changes up to 5ac16998a64422509f3123304891aae905e1ff04:

libxfs: add a xattr_entry helper (2024-06-03 11:37:41 -0700)

----------------------------------------------------------------
xfsprogs: bmap log intent cleanups [v30.5 03/35]

The next major target of online repair are metadata that are persisted
in blocks mapped by a file fork.  In other words, we want to repair
directories, extended attributes, symbolic links, and the realtime free
space information.  For file-based metadata, we assume that the space
metadata is correct, which enables repair to construct new versions of
the metadata in a temporary file.  We then need to swap the file fork
mappings of the two files atomically.  With this patchset, we begin
constructing such a facility based on the existing bmap log items and a
new extent swap log item.

This series cleans up a few parts of the file block mapping log intent
code before we start adding support for realtime bmap intents.  Most of
it involves cleaning up tracepoints so that more of the data extraction
logic ends up in the tracepoint code and not the tracepoint call site,
which should reduce overhead further when tracepoints are disabled.
There is also a change to pass bmap intents all the way back to the bmap
code instead of unboxing the intent values and re-boxing them after the
_finish_one function completes.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (4):
libxfs: remove kmem_alloc, kmem_zalloc, and kmem_free
libxfs: add a bi_entry helper
libxfs: reuse xfs_bmap_update_cancel_item
libxfs: add a xattr_entry helper

db/bmap_inflate.c         |  2 +-
include/kmem.h            | 10 +-------
libxfs/defer_item.c       | 58 +++++++++++++++++++++++++----------------------
libxfs/init.c             |  2 +-
libxfs/kmem.c             | 32 ++++++++------------------
libxlog/xfs_log_recover.c | 19 ++++++++--------
repair/bmap_repair.c      |  4 ++--
7 files changed, 55 insertions(+), 72 deletions(-)

[GIT PULL 04/10] xfsprogs: widen BUI formats to support realtime

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 5ac16998a64422509f3123304891aae905e1ff04:

libxfs: add a xattr_entry helper (2024-06-03 11:37:41 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/realtime-bmap-intents-6.9_2024-06-03

for you to fetch changes up to a552c62f50207db2346732f8171de992f61ef894:

libxfs: add a realtime flag to the bmap update log redo items (2024-06-03 11:37:41 -0700)

----------------------------------------------------------------
xfsprogs: widen BUI formats to support realtime [v30.5 04/35]

Atomic extent swapping (and later, reverse mapping and reflink) on the
realtime device needs to be able to defer file mapping and extent
freeing work in much the same manner as is required on the data volume.
Make the BUI log items operate on rt extents in preparation for atomic
swapping and realtime rmap.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (1):
libxfs: add a realtime flag to the bmap update log redo items

libxfs/defer_item.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

[GIT PULL 05/10] xfs_spaceman: updates for 6.9

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit a552c62f50207db2346732f8171de992f61ef894:

libxfs: add a realtime flag to the bmap update log redo items (2024-06-03 11:37:41 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/spaceman-updates-6.9_2024-06-03

for you to fetch changes up to 60cf6755a65f544665bca5b887fa4926a3253658:

xfs_spaceman: report health of inode link counts (2024-06-03 11:37:42 -0700)

----------------------------------------------------------------
xfs_spaceman: updates for 6.9 [v30.5 05/35]

Update xfs_spaceman to handle the new health reporting code that was
merged in 6.9.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (2):
xfs_spaceman: report the health of quota counts
xfs_spaceman: report health of inode link counts

man/man2/ioctl_xfs_fsgeometry.2 | 3 +++
spaceman/health.c               | 8 ++++++++
2 files changed, 11 insertions(+)

[GIT PULL 06/10] xfs_scrub: updates for 6.9

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 60cf6755a65f544665bca5b887fa4926a3253658:

xfs_spaceman: report health of inode link counts (2024-06-03 11:37:42 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-fixes-6.9_2024-06-03

for you to fetch changes up to 2025aa7a29e60c89047711922c1340a8b3c9d99e:

xfs_scrub: upload clean bills of health (2024-06-03 11:37:42 -0700)

----------------------------------------------------------------
xfs_scrub: updates for 6.9 [v30.5 06/35]

Now that the kernel has the code for userspace to upload a clean bill of
health (which clears out all the secondary markers of ill health that
hint at forgotten sicknesses), let's make xfs_scrub do that if the
filesystem is actually clean.

Second, restructure the xfs_scrub program so that it scrubs file link
counts and quotacheck in parallel.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (5):
xfs_scrub: implement live quotacheck inode scan
xfs_scrub: check file link counts
xfs_scrub: update health status if we get a clean bill of health
xfs_scrub: use multiple threads to run in-kernel metadata scrubs that scan inodes
xfs_scrub: upload clean bills of health

libfrog/scrub.c                     |  15 ++++
man/man2/ioctl_xfs_scrub_metadata.2 |  10 +++
scrub/phase1.c                      |  38 +++++++++
scrub/phase4.c                      |  17 ++++
scrub/phase5.c                      | 150 ++++++++++++++++++++++++++++++++----
scrub/repair.c                      |  18 +++++
scrub/repair.h                      |   1 +
scrub/scrub.c                       |  43 +++++++----
scrub/scrub.h                       |   3 +
9 files changed, 265 insertions(+), 30 deletions(-)

[GIT PULL 07/10] xfs_repair: minor fixes

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 2025aa7a29e60c89047711922c1340a8b3c9d99e:

xfs_scrub: upload clean bills of health (2024-06-03 11:37:42 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-fixes-6.9_2024-06-03

for you to fetch changes up to 842676ed999f0baae79c3de3ad6e2d3b90733f49:

xfs_repair: check num before bplist[num] (2024-06-03 11:37:42 -0700)

----------------------------------------------------------------
xfs_repair: minor fixes [v30.5 07/35]

Fix some random minor problems in xfs_repair.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (2):
xfs_repair: log when buffers fail CRC checks even if we just recompute it
xfs_repair: check num before bplist[num]

repair/attr_repair.c | 15 ++++++++++++---
repair/da_util.c     | 12 ++++++++----
repair/dir2.c        | 28 +++++++++++++++++++++-------
repair/prefetch.c    |  2 +-
4 files changed, 42 insertions(+), 15 deletions(-)

[GIT PULL 08/10] xfs_repair: use in-memory rmap btrees

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 842676ed999f0baae79c3de3ad6e2d3b90733f49:

xfs_repair: check num before bplist[num] (2024-06-03 11:37:42 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-use-in-memory-btrees-6.9_2024-06-03

for you to fetch changes up to 47307ecef44599b2caf0546c7e518b544e14d9c8:

xfs_repair: remove the old rmap collection slabs (2024-06-03 11:37:42 -0700)

----------------------------------------------------------------
xfs_repair: use in-memory rmap btrees [v30.5 08/35]

Now that we've ported support for in-memory btrees to userspace, port
xfs_repair to use them instead of the clunky slab interface that we
currently use.  This has the effect of moving memory consumption for
tracking reverse mappings into a memfd file, which means that we could
(theoretically) reduce the memory requirements by pointing it at an
on-disk file or something.  It also enables us to remove the sorting
step and to avoid having to coalesce adjacent contiguous bmap records
into a single rmap record.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (1):
libxfs: provide a kernel-compatible kasprintf

Darrick J. Wong (5):
xfs_repair: convert regular rmap repair to use in-memory btrees
xfs_repair: verify on-disk rmap btrees with in-memory btree data
xfs_repair: compute refcount data from in-memory rmap btrees
xfs_repair: reduce rmap bag memory usage when creating refcounts
xfs_repair: remove the old rmap collection slabs

include/kmem.h           |   3 +
include/libxfs.h         |   3 +
libxfs/buf_mem.h         |   5 +
libxfs/kmem.c            |  13 +
libxfs/libxfs_api_defs.h |  13 +
repair/agbtree.c         |  18 +-
repair/agbtree.h         |   1 +
repair/dinode.c          |   9 +-
repair/phase4.c          |  25 +-
repair/phase5.c          |   2 +-
repair/rmap.c            | 762 ++++++++++++++++++++++++++++++-----------------
repair/rmap.h            |  25 +-
repair/scan.c            |   7 +-
repair/slab.c            |  49 +--
repair/slab.h            |   2 +-
repair/xfs_repair.c      |   6 +
16 files changed, 602 insertions(+), 341 deletions(-)

[GIT PULL 09/10] xfs_repair: reduce refcount repair memory usage

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 47307ecef44599b2caf0546c7e518b544e14d9c8:

xfs_repair: remove the old rmap collection slabs (2024-06-03 11:37:42 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/repair-refcount-scalability-6.9_2024-06-03

for you to fetch changes up to 9a51e91a017bf285ea5fd5c5a6a7534e2ca56587:

xfs_repair: remove the old bag implementation (2024-06-03 11:37:42 -0700)

----------------------------------------------------------------
xfs_repair: reduce refcount repair memory usage [v30.5 09/35]

The refcountbt repair code has serious memory usage problems when the
block sharing factor of the filesystem is very high.  This can happen if
a deduplication tool has been run against the filesystem, or if the fs
stores reflinked VM images that have been aging for a long time.

Recall that the original reference counting algorithm walks the reverse
mapping records of the filesystem to generate reference counts.  For any
given block in the AG, the rmap bag structure contains the all rmap
records that cover that block; the refcount is the size of that bag.

For online repair, the bag doesn't need the owner, offset, or state flag
information, so it discards those.  This halves the record size, but the
bag structure still stores one excerpted record for each reverse
mapping.  If the sharing count is high, this will use a LOT of memory
storing redundant records.  In the extreme case, 100k mappings to the
same piece of space will consume 100k*16 bytes = 1.6M of memory.

For offline repair, the bag stores the owner values so that we know
which inodes need to be marked as being reflink inodes.  If a
deduplication tool has been run and there are many blocks within a file
pointing to the same physical space, this will stll use a lot of memory
to store redundant records.

The solution to this problem is to deduplicate the bag records when
possible by adding a reference count to the bag record, and changing the
bag add function to detect an existing record to bump the refcount.  In
the above example, the 100k mappings will now use 24 bytes of memory.
These lookups can be done efficiently with a btree, so we create a new
refcount bag btree type (inside of online repair).  This is why we
refactored the btree code in the previous patchset.

The btree conversion also dramatically reduces the runtime of the
refcount generation algorithm, because the code to delete all bag
records that end at a given agblock now only has to delete one record
instead of (using the example above) 100k records.  As an added benefit,
record deletion now gives back the unused xfile space, which it did not
do previously.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (4):
xfs_repair: define an in-memory btree for storing refcount bag info
xfs_repair: create refcount bag
xfs_repair: port to the new refcount bag structure
xfs_repair: remove the old bag implementation

libxfs/libxfs_api_defs.h |   9 ++
repair/Makefile          |   4 +
repair/rcbag.c           | 371 ++++++++++++++++++++++++++++++++++++++++++++
repair/rcbag.h           |  32 ++++
repair/rcbag_btree.c     | 390 +++++++++++++++++++++++++++++++++++++++++++++++
repair/rcbag_btree.h     |  77 ++++++++++
repair/rmap.c            | 157 ++++++-------------
repair/slab.c            | 130 ----------------
repair/slab.h            |  19 ---
repair/xfs_repair.c      |   6 +
10 files changed, 934 insertions(+), 261 deletions(-)
create mode 100644 repair/rcbag.c
create mode 100644 repair/rcbag.h
create mode 100644 repair/rcbag_btree.c
create mode 100644 repair/rcbag_btree.h

[GIT PULL 10/10] mkfs: cleanups for 6.9

[Darrick J. Wong]

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.6-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 9a51e91a017bf285ea5fd5c5a6a7534e2ca56587:

xfs_repair: remove the old bag implementation (2024-06-03 11:37:42 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/mkfs-cleanups-6.9_2024-06-03

for you to fetch changes up to d1f942f19859103ccbca6ba7de9ed65024be2ae9:

mkfs: use libxfs to create symlinks (2024-06-03 11:37:43 -0700)

----------------------------------------------------------------
mkfs: cleanups for 6.9 [v30.5 11/35]

Clean up mkfs' open-coded symlink handling code.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
xfs_io: fix gcc complaints about potentially uninitialized variables
xfs_io: print sysfs paths of mounted filesystems
mkfs: use libxfs to create symlinks

io/bulkstat.c            |  4 +--
io/fsuuid.c              | 68 +++++++++++++++++++++++++++++++++++++++++++++
libxfs/libxfs_api_defs.h |  1 +
man/man8/xfs_io.8        |  7 +++++
mkfs/proto.c             | 72 +++++++++++++++++++++++++-----------------------
quota/quota.c            |  2 ++
quota/quota.h            | 20 ++++++++++++++
quota/util.c             | 26 +++++++++++++++++
8 files changed, 164 insertions(+), 36 deletions(-)