[PRBOMB] xfs: online repair patches for 6.9

[PRBOMB] xfs: online repair patches for 6.9

[Darrick J. Wong]

Hi Chandan,

Please pull these multiple pull requests containing all the online
repair functionality that Christoph and I managed to get reviewed in
time for 6.9.

--D

PS: Has anyone out there observed the following crash on arm64?

run fstests xfs/570 at 2024-02-22 20:32:17
spectre-v4 mitigation disabled by command-line option
XFS (sda2): Mounting V5 Filesystem 2fd78ebc-692d-46e1-bd8a-9f1591c007f6
XFS (sda2): Ending clean mount
XFS (sda2): EXPERIMENTAL online scrub feature in use. Use at your own risk!
XFS (sda3): Mounting V5 Filesystem ac0b0a07-294f-418e-b4d0-c14cc345fcd4
XFS (sda3): Ending clean mount
XFS (sda3): EXPERIMENTAL online scrub feature in use. Use at your own risk!
Unable to handle kernel paging request at virtual address ffffffff80206388
Mem abort info:
  ESR = 0x0000000096000006
  EC = 0x25: DABT (current EL), IL = 32 bits
  SET = 0, FnV = 0
  EA = 0, S1PTW = 0
  FSC = 0x06: level 2 translation fault
Data abort info:
  ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
  CM = 0, WnR = 0, TnD = 0, TagAccess = 0
  GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
swapper pgtable: 64k pages, 42-bit VAs, pgdp=0000000040d40000
[ffffffff80206388] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000, pmd=0000000000000000
Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in: xfs rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_REDIRECT nf_nat nf_conntrack nf_defra
sch_fq_codel fuse configfs efivarfs ip_tables x_tables overlay nfsv4
CPU: 0 PID: 38 Comm: 0:1H Not tainted 6.8.0-rc4-djwa #rc4 892edfc98307d2cdb226d4164dfd6775c2b3b52c
Hardware name: QEMU KVM Virtual Machine, BIOS 1.6.6 08/22/2023
Workqueue: xfs-log/sda3 xlog_ioend_work [xfs]
pstate: a0401005 (NzCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)
pc : kfree+0x54/0x2d8
lr : xlog_cil_committed+0x11c/0x1d8 [xfs]
sp : fffffe00830cfbe0
x29: fffffe00830cfbe0 x28: fffffc00240e0c80 x27: fffffc00240e0c00
x26: 00000005000037a8 x25: 0000000000000000 x24: 0000000000000000
x23: fffffc0021e68d40 x22: fffffe00818e0000 x21: fffffc0021e68d88
x20: fffffe007a6b93cc x19: ffffffff80206380 x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000 x15: fffffe008840f620
x14: 0000000000000000 x13: 0000000000000020 x12: 0101010101010101
x11: 0000000000000040 x10: fffffc0025b0d5d8 x9 : fffffe007a6b93cc
x8 : fffffe00830cfbe0 x7 : 0000000000000000 x6 : 0000000000000000
x5 : d230261c49c51c03 x4 : fffffc00e1357340 x3 : dead000000000122
x2 : fffffc00ea7fa000 x1 : fffffc0021e68d88 x0 : ffffffff00000000
Call trace:
 kfree+0x54/0x2d8
 xlog_cil_committed+0x11c/0x1d8 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
 xlog_cil_process_committed+0x6c/0xa8 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
 xlog_state_do_callback+0x1e0/0x3e0 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
 xlog_state_done_syncing+0x8c/0x158 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
 xlog_ioend_work+0x70/0xd8 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
 process_one_work+0x174/0x3e8
 worker_thread+0x2c4/0x3e8
 kthread+0x110/0x128
 ret_from_fork+0x10/0x20
Code: 8b1302d3 b2607fe0 d350fe73 8b131813 (f9400660) 
---[ end trace 0000000000000000 ]---

[GIT PULL 1/18] xfs: repair inode mode by scanning dirs

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 4b2f459d86252619448455013f581836c8b1b7da:

xfs: fix SEEK_HOLE/DATA for regions with active COW extents (2024-02-21 12:31:12 +0530)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-inode-mode-6.9_2024-02-23

for you to fetch changes up to 5385f1a60d4e5b73e8ecd2757865352b68f54fb9:

xfs: repair file modes by scanning for a dirent pointing to us (2024-02-22 12:30:51 -0800)

----------------------------------------------------------------
xfs: repair inode mode by scanning dirs [v29.3 01/18]

One missing piece of functionality in the inode record repair code is
figuring out what to do with a file whose mode is so corrupt that we
cannot tell us the type of the file.  Originally this was done by
guessing the mode from the ondisk inode contents, but Christoph didn't
like that because it read from data fork block 0, which could be user
controlled data.

Therefore, I've replaced all that with a directory scanner that looks
for any dirents that point to the file with the garbage mode.  If so,
the ftype in the dirent will tell us exactly what mode to set on the
file.  Since users cannot directly write to the ftype field of a dirent,
this should be safe.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (10):
xfs: speed up xfs_iwalk_adjust_start a little bit
xfs: implement live inode scan for scrub
xfs: allow scrub to hook metadata updates in other writers
xfs: stagger the starting AG of scrub iscans to reduce contention
xfs: cache a bunch of inodes for repair scans
xfs: iscan batching should handle unallocated inodes too
xfs: create a static name for the dot entry too
xfs: create a predicate to determine if two xfs_names are the same
xfs: create a macro for decoding ftypes in tracepoints
xfs: repair file modes by scanning for a dirent pointing to us

fs/xfs/Kconfig                |   5 +
fs/xfs/Makefile               |   2 +
fs/xfs/libxfs/xfs_da_format.h |  11 +
fs/xfs/libxfs/xfs_dir2.c      |   6 +
fs/xfs/libxfs/xfs_dir2.h      |  13 +
fs/xfs/scrub/dir.c            |   4 +-
fs/xfs/scrub/inode_repair.c   | 236 ++++++++++++-
fs/xfs/scrub/iscan.c          | 767 ++++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/iscan.h          |  84 +++++
fs/xfs/scrub/trace.c          |   2 +
fs/xfs/scrub/trace.h          | 194 +++++++++++
fs/xfs/xfs_hooks.c            |  52 +++
fs/xfs/xfs_hooks.h            |  65 ++++
fs/xfs/xfs_iwalk.c            |  13 +-
fs/xfs/xfs_linux.h            |   1 +
15 files changed, 1436 insertions(+), 19 deletions(-)
create mode 100644 fs/xfs/scrub/iscan.c
create mode 100644 fs/xfs/scrub/iscan.h
create mode 100644 fs/xfs/xfs_hooks.c
create mode 100644 fs/xfs/xfs_hooks.h

[GIT PULL 2/18] xfs: online repair of quota counters

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 5385f1a60d4e5b73e8ecd2757865352b68f54fb9:

xfs: repair file modes by scanning for a dirent pointing to us (2024-02-22 12:30:51 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-quotacheck-6.9_2024-02-23

for you to fetch changes up to 96ed2ae4a9b06b417e1c20c086c77755a43284bf:

xfs: repair dquots based on live quotacheck results (2024-02-22 12:30:57 -0800)

----------------------------------------------------------------
xfs: online repair of quota counters [v29.3 02/18]

This series uses the inode scanner and live update hook functionality
introduced in the last patchset to implement quotacheck on a live
filesystem.  The quotacheck scrubber builds an incore copy of the
dquot resource usage counters and compares it to the live dquots to
report discrepancies.

If the user chooses to repair the quota counters, the repair function
visits each incore dquot to update the counts from the live information.
The live update hooks are key to keeping the incore copy up to date.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (8):
xfs: report the health of quota counts
xfs: create a xchk_trans_alloc_empty helper for scrub
xfs: create a helper to count per-device inode block usage
xfs: create a sparse load xfarray function
xfs: implement live quotacheck inode scan
xfs: track quota updates during live quotacheck
xfs: repair cannot update the summary counters when logging quota flags
xfs: repair dquots based on live quotacheck results

fs/xfs/Makefile                  |   2 +
fs/xfs/libxfs/xfs_fs.h           |   4 +-
fs/xfs/libxfs/xfs_health.h       |   4 +-
fs/xfs/scrub/common.c            |  49 ++-
fs/xfs/scrub/common.h            |  11 +
fs/xfs/scrub/fscounters.c        |   2 +-
fs/xfs/scrub/health.c            |   1 +
fs/xfs/scrub/quotacheck.c        | 867 +++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/quotacheck.h        |  76 ++++
fs/xfs/scrub/quotacheck_repair.c | 261 ++++++++++++
fs/xfs/scrub/repair.c            |  46 ++-
fs/xfs/scrub/repair.h            |   5 +
fs/xfs/scrub/scrub.c             |   9 +
fs/xfs/scrub/scrub.h             |  10 +-
fs/xfs/scrub/stats.c             |   1 +
fs/xfs/scrub/trace.h             |  30 +-
fs/xfs/scrub/xfarray.h           |  19 +
fs/xfs/xfs_health.c              |   1 +
fs/xfs/xfs_inode.c               |  16 +
fs/xfs/xfs_inode.h               |   2 +
fs/xfs/xfs_qm.c                  |  23 +-
fs/xfs/xfs_qm.h                  |  16 +
fs/xfs/xfs_qm_bhv.c              |   1 +
fs/xfs/xfs_quota.h               |  46 +++
fs/xfs/xfs_trans_dquot.c         | 169 +++++++-
25 files changed, 1645 insertions(+), 26 deletions(-)
create mode 100644 fs/xfs/scrub/quotacheck.c
create mode 100644 fs/xfs/scrub/quotacheck.h
create mode 100644 fs/xfs/scrub/quotacheck_repair.c

[GIT PULL 3/18] xfs: online repair of file link counts

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 96ed2ae4a9b06b417e1c20c086c77755a43284bf:

xfs: repair dquots based on live quotacheck results (2024-02-22 12:30:57 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-nlinks-6.9_2024-02-23

for you to fetch changes up to 6b631c60c90a1e5264bc9bcdca2c0adb492d7a62:

xfs: teach repair to fix file nlinks (2024-02-22 12:31:00 -0800)

----------------------------------------------------------------
xfs: online repair of file link counts [v29.3 03/18]

Now that we've created the infrastructure to perform live scans of every
file in the filesystem and the necessary hook infrastructure to observe
live updates, use it to scan directories to compute the correct link
counts for files in the filesystem, and reset those link counts.

This patchset creates a tailored readdir implementation for scrub
because the regular version has to cycle ILOCKs to copy information to
userspace.  We can't cycle the ILOCK during the nlink scan and we don't
need all the other VFS support code (maintaining a readdir cursor and
translating XFS structures to VFS structures and back) so it was easier
to duplicate the code.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (4):
xfs: report health of inode link counts
xfs: teach scrub to check file nlinks
xfs: track directory entry updates during live nlinks fsck
xfs: teach repair to fix file nlinks

fs/xfs/Makefile              |   2 +
fs/xfs/libxfs/xfs_fs.h       |   4 +-
fs/xfs/libxfs/xfs_health.h   |   4 +-
fs/xfs/scrub/common.c        |   3 +
fs/xfs/scrub/common.h        |   1 +
fs/xfs/scrub/health.c        |   1 +
fs/xfs/scrub/nlinks.c        | 930 +++++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/nlinks.h        | 102 +++++
fs/xfs/scrub/nlinks_repair.c | 223 +++++++++++
fs/xfs/scrub/repair.h        |   2 +
fs/xfs/scrub/scrub.c         |   9 +
fs/xfs/scrub/scrub.h         |   5 +-
fs/xfs/scrub/stats.c         |   1 +
fs/xfs/scrub/trace.c         |   2 +
fs/xfs/scrub/trace.h         | 183 ++++++++-
fs/xfs/xfs_health.c          |   1 +
fs/xfs/xfs_inode.c           | 117 ++++++
fs/xfs/xfs_inode.h           |  31 ++
fs/xfs/xfs_mount.h           |   3 +
fs/xfs/xfs_super.c           |   2 +
fs/xfs/xfs_symlink.c         |   1 +
21 files changed, 1623 insertions(+), 4 deletions(-)
create mode 100644 fs/xfs/scrub/nlinks.c
create mode 100644 fs/xfs/scrub/nlinks.h
create mode 100644 fs/xfs/scrub/nlinks_repair.c

[GIT PULL 4/18] xfs: report corruption to the health trackers

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 6b631c60c90a1e5264bc9bcdca2c0adb492d7a62:

xfs: teach repair to fix file nlinks (2024-02-22 12:31:00 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/corruption-health-reports-6.9_2024-02-23

for you to fetch changes up to 989d5ec3175be7c0012d7744c667ae6a266fab06:

xfs: report XFS_IS_CORRUPT errors to the health system (2024-02-22 12:32:55 -0800)

----------------------------------------------------------------
xfs: report corruption to the health trackers [v29.3 04/18]

Any time that the runtime code thinks it has found corrupt metadata, it
should tell the health tracking subsystem that the corresponding part of
the filesystem is sick.  These reports come primarily from two places --
code that is reading a buffer that fails validation, and higher level
pieces that observe a conflict involving multiple buffers.  This
patchset uses automated scanning to update all such callsites with a
mark_sick call.

Doing this enables the health system to record problem observed at
runtime, which (for now) can prompt the sysadmin to run xfs_scrub, and
(later) may enable more targetted fixing of the filesystem.

Note: Earlier reviewers of this patchset suggested that the verifier
functions themselves should be responsible for calling _mark_sick.  In a
higher level language this would be easily accomplished with lambda
functions and closures.  For the kernel, however, we'd have to create
the necessary closures by hand, pass them to the buf_read calls, and
then implement necessary state tracking to detach the xfs_buf from the
closure at the necessary time.  This is far too much work and complexity
and will not be pursued further.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (11):
xfs: separate the marking of sick and checked metadata
xfs: report fs corruption errors to the health tracking system
xfs: report ag header corruption errors to the health tracking system
xfs: report block map corruption errors to the health tracking system
xfs: report btree block corruption errors to the health system
xfs: report dir/attr block corruption errors to the health system
xfs: report symlink block corruption errors to the health system
xfs: report inode corruption errors to the health system
xfs: report quota block corruption errors to the health system
xfs: report realtime metadata corruption errors to the health system
xfs: report XFS_IS_CORRUPT errors to the health system

fs/xfs/libxfs/xfs_ag.c          |   5 +-
fs/xfs/libxfs/xfs_alloc.c       | 105 +++++++++++++++++++----
fs/xfs/libxfs/xfs_attr_leaf.c   |   4 +
fs/xfs/libxfs/xfs_attr_remote.c |  35 +++++---
fs/xfs/libxfs/xfs_bmap.c        | 135 +++++++++++++++++++++++++----
fs/xfs/libxfs/xfs_btree.c       |  39 ++++++++-
fs/xfs/libxfs/xfs_da_btree.c    |  37 ++++++--
fs/xfs/libxfs/xfs_dir2.c        |   5 +-
fs/xfs/libxfs/xfs_dir2_block.c  |   2 +
fs/xfs/libxfs/xfs_dir2_data.c   |   3 +
fs/xfs/libxfs/xfs_dir2_leaf.c   |   3 +
fs/xfs/libxfs/xfs_dir2_node.c   |   7 ++
fs/xfs/libxfs/xfs_health.h      |  42 ++++++++-
fs/xfs/libxfs/xfs_ialloc.c      |  57 ++++++++++--
fs/xfs/libxfs/xfs_inode_buf.c   |  12 ++-
fs/xfs/libxfs/xfs_inode_fork.c  |   8 ++
fs/xfs/libxfs/xfs_refcount.c    |  43 +++++++++-
fs/xfs/libxfs/xfs_rmap.c        |  83 ++++++++++++++++--
fs/xfs/libxfs/xfs_rtbitmap.c    |   9 +-
fs/xfs/libxfs/xfs_sb.c          |   2 +
fs/xfs/scrub/health.c           |   8 +-
fs/xfs/scrub/refcount_repair.c  |   9 +-
fs/xfs/xfs_attr_inactive.c      |   4 +
fs/xfs/xfs_attr_list.c          |  18 +++-
fs/xfs/xfs_dir2_readdir.c       |   6 +-
fs/xfs/xfs_discard.c            |   2 +
fs/xfs/xfs_dquot.c              |  30 +++++++
fs/xfs/xfs_health.c             | 186 ++++++++++++++++++++++++++++++++++++++++
fs/xfs/xfs_icache.c             |   9 ++
fs/xfs/xfs_inode.c              |  16 +++-
fs/xfs/xfs_iomap.c              |  15 +++-
fs/xfs/xfs_iwalk.c              |   5 +-
fs/xfs/xfs_qm.c                 |   8 +-
fs/xfs/xfs_reflink.c            |   6 +-
fs/xfs/xfs_rtalloc.c            |   6 ++
fs/xfs/xfs_symlink.c            |  17 ++--
fs/xfs/xfs_trace.h              |   4 +
37 files changed, 881 insertions(+), 104 deletions(-)

[GIT PULL 5/18] xfs: indirect health reporting

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 989d5ec3175be7c0012d7744c667ae6a266fab06:

xfs: report XFS_IS_CORRUPT errors to the health system (2024-02-22 12:32:55 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/indirect-health-reporting-6.9_2024-02-23

for you to fetch changes up to a1f3e0cca41036c3c66abb6a2ed8fedc214e9a4c:

xfs: update health status if we get a clean bill of health (2024-02-22 12:33:04 -0800)

----------------------------------------------------------------
xfs: indirect health reporting [v29.3 05/18]

This series enables the XFS health reporting infrastructure to remember
indirect health concerns when resources are scarce.  For example, if a
scrub notices that there's something wrong with an inode's metadata but
memory reclaim needs to free the incore inode, we want to record in the
perag data the fact that there was some inode somewhere with an error.
The perag structures never go away.

The first two patches in this series set that up, and the third one
provides a means for xfs_scrub to tell the kernel that it can forget the
indirect problem report.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: add secondary and indirect classes to the health tracking system
xfs: remember sick inodes that get inactivated
xfs: update health status if we get a clean bill of health

fs/xfs/libxfs/xfs_fs.h        |  4 ++-
fs/xfs/libxfs/xfs_health.h    | 47 +++++++++++++++++++++++++
fs/xfs/libxfs/xfs_inode_buf.c |  2 +-
fs/xfs/scrub/health.c         | 80 +++++++++++++++++++++++++++++++++++++++++--
fs/xfs/scrub/health.h         |  1 +
fs/xfs/scrub/repair.c         |  1 +
fs/xfs/scrub/scrub.c          |  6 ++++
fs/xfs/scrub/trace.h          |  4 ++-
fs/xfs/xfs_health.c           | 33 +++++++++++-------
fs/xfs/xfs_inode.c            | 35 +++++++++++++++++++
fs/xfs/xfs_trace.h            |  1 +
11 files changed, 196 insertions(+), 18 deletions(-)

[GIT PULL 6/18] xfs: online repair for fs summary counters

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit a1f3e0cca41036c3c66abb6a2ed8fedc214e9a4c:

xfs: update health status if we get a clean bill of health (2024-02-22 12:33:04 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-fscounters-6.9_2024-02-23

for you to fetch changes up to 4ed080cd7cb077bbb4b64f0712be1618c9d55a0d:

xfs: repair summary counters (2024-02-22 12:33:05 -0800)

----------------------------------------------------------------
xfs: online repair for fs summary counters [v29.3 06/18]

A longstanding deficiency in the online fs summary counter scrubbing
code is that it hasn't any means to quiesce the incore percpu counters
while it's running.  There is no way to coordinate with other threads
are reserving or freeing free space simultaneously, which leads to false
error reports.  Right now, if the discrepancy is large, we just sort of
shrug and bail out with an incomplete flag, but this is lame.

For repair activity, we actually /do/ need to stabilize the counters to
get an accurate reading and install it in the percpu counter.  To
improve the former and enable the latter, allow the fscounters online
fsck code to perform an exclusive mini-freeze on the filesystem.  The
exclusivity prevents userspace from thawing while we're running, and the
mini-freeze means that we don't wait for the log to quiesce, which will
make both speedier.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (1):
xfs: repair summary counters

fs/xfs/Makefile                  |  1 +
fs/xfs/scrub/fscounters.c        | 27 +++++++--------
fs/xfs/scrub/fscounters.h        | 20 +++++++++++
fs/xfs/scrub/fscounters_repair.c | 72 ++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/repair.h            |  2 ++
fs/xfs/scrub/scrub.c             |  2 +-
fs/xfs/scrub/trace.c             |  1 +
fs/xfs/scrub/trace.h             | 21 +++++++++---
8 files changed, 128 insertions(+), 18 deletions(-)
create mode 100644 fs/xfs/scrub/fscounters.h
create mode 100644 fs/xfs/scrub/fscounters_repair.c

[GIT PULL 7/18] xfs: move btree geometry to ops struct

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 4ed080cd7cb077bbb4b64f0712be1618c9d55a0d:

xfs: repair summary counters (2024-02-22 12:33:05 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/btree-geometry-in-ops-6.9_2024-02-23

for you to fetch changes up to f73def90a7cd24a32a42f689efba6a7a35edeb7b:

xfs: create predicate to determine if cursor is at inode root level (2024-02-22 12:37:24 -0800)

----------------------------------------------------------------
xfs: move btree geometry to ops struct [v29.3 07/18]

This patchset prepares the generic btree code to allow for the creation
of new btree types outside of libxfs.  The end goal here is for online
fsck to be able to create its own in-memory btrees that will be used to
improve the performance (and reduce the memory requirements of) the
refcount btree.

To enable this, I decided that the btree ops structure is the ideal
place to encode all of the geometry information about a btree. The btree
ops struture already contains the buffer ops (and hence the btree block
magic numbers) as well as the key and record sizes, so it doesn't seem
all that farfetched to encode the XFS_BTREE_ flags that determine the
geometry (ROOT_IN_INODE, LONG_PTRS, etc).

The rest of the patchset cleans up the btree functions that initialize
btree blocks and btree buffers.  The bulk of this work is to replace
btree geometry related function call arguments with a single pointer to
the ops structure, and then clean up everything else around that.  As a
side effect, we rename the functions.

Later, Christoph Hellwig and I merged together a bunch more cleanups
that he wanted to do for a while.  All the btree geometry information is
now in the btree ops structure, we've created an explicit btree type
(ag, inode, mem) and moved the per-btree type information to a separate
union.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (6):
xfs: remove bc_ino.flags
xfs: consolidate the xfs_alloc_lookup_* helpers
xfs: turn the allocbt cursor active field into a btree flag
xfs: move the btree stats offset into struct btree_ops
xfs: split out a btree type from the btree ops geometry flags
xfs: split the per-btree union in struct xfs_btree_cur

Darrick J. Wong (17):
xfs: consolidate btree block freeing tracepoints
xfs: consolidate btree block allocation tracepoints
xfs: set the btree cursor bc_ops in xfs_btree_alloc_cursor
xfs: drop XFS_BTREE_CRC_BLOCKS
xfs: fix imprecise logic in xchk_btree_check_block_owner
xfs: encode the btree geometry flags in the btree ops structure
xfs: extern some btree ops structures
xfs: initialize btree blocks using btree_ops structure
xfs: rename btree block/buffer init functions
xfs: btree convert xfs_btree_init_block to xfs_btree_init_buf calls
xfs: remove the unnecessary daddr paramter to _init_block
xfs: set btree block buffer ops in _init_buf
xfs: move lru refs to the btree ops structure
xfs: factor out a xfs_btree_owner helper
xfs: factor out a btree block owner check
xfs: store the btree pointer length in struct xfs_btree_ops
xfs: create predicate to determine if cursor is at inode root level

fs/xfs/libxfs/xfs_ag.c             |  33 ++--
fs/xfs/libxfs/xfs_ag.h             |   2 +-
fs/xfs/libxfs/xfs_alloc.c          |  54 +++---
fs/xfs/libxfs/xfs_alloc_btree.c    |  39 ++--
fs/xfs/libxfs/xfs_bmap.c           |  58 +++---
fs/xfs/libxfs/xfs_bmap_btree.c     |  59 +++---
fs/xfs/libxfs/xfs_bmap_btree.h     |   3 +
fs/xfs/libxfs/xfs_btree.c          | 365 +++++++++++++++++++------------------
fs/xfs/libxfs/xfs_btree.h          | 165 +++++++++--------
fs/xfs/libxfs/xfs_btree_staging.c  |  20 +-
fs/xfs/libxfs/xfs_btree_staging.h  |   3 +-
fs/xfs/libxfs/xfs_ialloc_btree.c   |  35 ++--
fs/xfs/libxfs/xfs_refcount.c       |  24 +--
fs/xfs/libxfs/xfs_refcount_btree.c |  24 ++-
fs/xfs/libxfs/xfs_rmap_btree.c     |  19 +-
fs/xfs/libxfs/xfs_shared.h         |   9 +
fs/xfs/scrub/btree.c               |  29 +--
fs/xfs/scrub/newbt.c               |   2 +-
fs/xfs/scrub/trace.c               |   2 +-
fs/xfs/xfs_trace.h                 |  83 ++++++++-
20 files changed, 566 insertions(+), 462 deletions(-)

[GIT PULL 8/18] xfs: remove bc_btnum from btree cursors

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit f73def90a7cd24a32a42f689efba6a7a35edeb7b:

xfs: create predicate to determine if cursor is at inode root level (2024-02-22 12:37:24 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/btree-remove-btnum-6.9_2024-02-23

for you to fetch changes up to ec793e690f801d97a7ae2a0d429fea1fee4d44aa:

xfs: remove xfs_btnum_t (2024-02-22 12:40:51 -0800)

----------------------------------------------------------------
xfs: remove bc_btnum from btree cursors [v29.3 08/18]

From Christoph Hellwig,

This series continues the migration of btree geometry information out of
the cursor structure and into the ops structure.  This time around, we
replace the btree type enumeration (btnum) with an explicit name string
in the btree ops structure.  This enables easy creation of /any/ new
btree type without having to mess with libxfs.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (26):
xfs: move comment about two 2 keys per pointer in the rmap btree
xfs: add a xfs_btree_init_ptr_from_cur
xfs: don't override bc_ops for staging btrees
xfs: fold xfs_allocbt_init_common into xfs_allocbt_init_cursor
xfs: remove xfs_allocbt_stage_cursor
xfs: fold xfs_inobt_init_common into xfs_inobt_init_cursor
xfs: remove xfs_inobt_stage_cursor
xfs: fold xfs_refcountbt_init_common into xfs_refcountbt_init_cursor
xfs: remove xfs_refcountbt_stage_cursor
xfs: fold xfs_rmapbt_init_common into xfs_rmapbt_init_cursor
xfs: remove xfs_rmapbt_stage_cursor
xfs: make full use of xfs_btree_stage_ifakeroot in xfs_bmbt_stage_cursor
xfs: fold xfs_bmbt_init_common into xfs_bmbt_init_cursor
xfs: remove xfs_bmbt_stage_cursor
xfs: split the agf_roots and agf_levels arrays
xfs: add a name field to struct xfs_btree_ops
xfs: add a sick_mask to struct xfs_btree_ops
xfs: refactor the btree cursor allocation logic in xchk_ag_btcur_init
xfs: split xfs_allocbt_init_cursor
xfs: remove xfs_inobt_cur
xfs: remove the btnum argument to xfs_inobt_count_blocks
xfs: remove the which variable in xchk_iallocbt
xfs: split xfs_inobt_insert_sprec
xfs: split xfs_inobt_init_cursor
xfs: pass a 'bool is_finobt' to xfs_inobt_insert
xfs: remove xfs_btnum_t

Darrick J. Wong (1):
xfs: make staging file forks explicit

fs/xfs/libxfs/xfs_ag.c             |  13 ++-
fs/xfs/libxfs/xfs_ag.h             |   8 +-
fs/xfs/libxfs/xfs_alloc.c          |  99 +++++++++-----------
fs/xfs/libxfs/xfs_alloc_btree.c    | 156 ++++++++++++++++++--------------
fs/xfs/libxfs/xfs_alloc_btree.h    |  10 +-
fs/xfs/libxfs/xfs_bmap_btree.c     |  89 ++++++------------
fs/xfs/libxfs/xfs_bmap_btree.h     |   2 -
fs/xfs/libxfs/xfs_btree.c          | 114 +++++++++++++++++------
fs/xfs/libxfs/xfs_btree.h          |  18 ++--
fs/xfs/libxfs/xfs_btree_staging.c  | 111 +----------------------
fs/xfs/libxfs/xfs_btree_staging.h  |   7 +-
fs/xfs/libxfs/xfs_format.h         |  21 ++---
fs/xfs/libxfs/xfs_ialloc.c         | 181 +++++++++++++++++++++++--------------
fs/xfs/libxfs/xfs_ialloc_btree.c   | 144 +++++++++++++----------------
fs/xfs/libxfs/xfs_ialloc_btree.h   |  11 +--
fs/xfs/libxfs/xfs_refcount_btree.c |  52 ++++-------
fs/xfs/libxfs/xfs_refcount_btree.h |   2 -
fs/xfs/libxfs/xfs_rmap_btree.c     |  65 +++++--------
fs/xfs/libxfs/xfs_rmap_btree.h     |   2 -
fs/xfs/libxfs/xfs_shared.h         |  35 +++++++
fs/xfs/libxfs/xfs_types.h          |  26 +-----
fs/xfs/scrub/agheader.c            |  12 +--
fs/xfs/scrub/agheader_repair.c     |  47 ++++------
fs/xfs/scrub/alloc_repair.c        |  27 +++---
fs/xfs/scrub/bmap_repair.c         |   8 +-
fs/xfs/scrub/btree.c               |  12 +--
fs/xfs/scrub/common.c              |  72 ++++++++-------
fs/xfs/scrub/health.c              |  54 ++---------
fs/xfs/scrub/health.h              |   4 +-
fs/xfs/scrub/ialloc.c              |  20 ++--
fs/xfs/scrub/ialloc_repair.c       |   8 +-
fs/xfs/scrub/iscan.c               |   2 +-
fs/xfs/scrub/refcount_repair.c     |   4 +-
fs/xfs/scrub/repair.c              |  14 +--
fs/xfs/scrub/rmap.c                |  15 ++-
fs/xfs/scrub/trace.h               |  48 ++++------
fs/xfs/xfs_discard.c               |   2 +-
fs/xfs/xfs_fsmap.c                 |   4 +-
fs/xfs/xfs_health.c                |  36 +++-----
fs/xfs/xfs_iwalk.c                 |   8 +-
fs/xfs/xfs_trace.h                 |  89 ++++++++----------
41 files changed, 748 insertions(+), 904 deletions(-)

[GIT PULL 9/18] xfs: btree check cleanups

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit ec793e690f801d97a7ae2a0d429fea1fee4d44aa:

xfs: remove xfs_btnum_t (2024-02-22 12:40:51 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/btree-check-cleanups-6.9_2024-02-23

for you to fetch changes up to 79e72304dcba471e5c0dea2f3c67fe1a0558c140:

xfs: factor out a __xfs_btree_check_lblock_hdr helper (2024-02-22 12:40:59 -0800)

----------------------------------------------------------------
xfs: btree check cleanups [v29.3 09/18]

Minor cleanups for the btree block pointer checking code.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (10):
xfs: simplify xfs_btree_check_sblock_siblings
xfs: simplify xfs_btree_check_lblock_siblings
xfs: open code xfs_btree_check_lptr in xfs_bmap_btree_to_extents
xfs: consolidate btree ptr checking
xfs: misc cleanups for __xfs_btree_check_sblock
xfs: remove the crc variable in __xfs_btree_check_lblock
xfs: tighten up validation of root block in inode forks
xfs: consolidate btree block verification
xfs: rename btree helpers that depends on the block number representation
xfs: factor out a __xfs_btree_check_lblock_hdr helper

fs/xfs/libxfs/xfs_alloc_btree.c    |   8 +-
fs/xfs/libxfs/xfs_bmap.c           |   2 +-
fs/xfs/libxfs/xfs_bmap_btree.c     |   8 +-
fs/xfs/libxfs/xfs_btree.c          | 252 ++++++++++++++++++-------------------
fs/xfs/libxfs/xfs_btree.h          |  44 ++-----
fs/xfs/libxfs/xfs_ialloc_btree.c   |   8 +-
fs/xfs/libxfs/xfs_refcount_btree.c |   8 +-
fs/xfs/libxfs/xfs_rmap_btree.c     |   8 +-
fs/xfs/scrub/btree.c               |  21 +---
9 files changed, 159 insertions(+), 200 deletions(-)

[GIT PULL 10/18] xfs: btree readahead cleanups

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 79e72304dcba471e5c0dea2f3c67fe1a0558c140:

xfs: factor out a __xfs_btree_check_lblock_hdr helper (2024-02-22 12:40:59 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/btree-readahead-cleanups-6.9_2024-02-23

for you to fetch changes up to 24f755e4854e0fddb78d18f610bf1b5cb61db520:

xfs: split xfs_buf_rele for cached vs uncached buffers (2024-02-22 12:41:02 -0800)

----------------------------------------------------------------
xfs: btree readahead cleanups [v29.3 10/18]

Minor cleanups for the btree block readahead code.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (4):
xfs: remove xfs_btree_reada_bufl
xfs: remove xfs_btree_reada_bufs
xfs: move and rename xfs_btree_read_bufl
xfs: split xfs_buf_rele for cached vs uncached buffers

fs/xfs/libxfs/xfs_bmap.c  | 33 ++++++++++++----
fs/xfs/libxfs/xfs_btree.c | 98 ++++++++---------------------------------------
fs/xfs/libxfs/xfs_btree.h | 36 -----------------
fs/xfs/xfs_buf.c          | 46 ++++++++++++++--------
fs/xfs/xfs_iwalk.c        |  6 ++-
5 files changed, 76 insertions(+), 143 deletions(-)

[GIT PULL 11/18] xfs: buftarg cleanups

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 24f755e4854e0fddb78d18f610bf1b5cb61db520:

xfs: split xfs_buf_rele for cached vs uncached buffers (2024-02-22 12:41:02 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/buftarg-cleanups-6.9_2024-02-23

for you to fetch changes up to 1c51ac0998ed9baaca3ac75c0083b4c3b4d993ef:

xfs: move setting bt_logical_sectorsize out of xfs_setsize_buftarg (2024-02-22 12:42:45 -0800)

----------------------------------------------------------------
xfs: buftarg cleanups [v29.3 11/18]

Clean up the buffer target code in preparation for adding the ability to
target tmpfs files.  That will enable the creation of in memory btrees.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (3):
xfs: remove the xfs_buftarg_t typedef
xfs: remove xfs_setsize_buftarg_early
xfs: move setting bt_logical_sectorsize out of xfs_setsize_buftarg

fs/xfs/xfs_buf.c   | 34 +++++++++++++---------------------
fs/xfs/xfs_buf.h   |  4 ++--
fs/xfs/xfs_log.c   | 14 +++++++-------
fs/xfs/xfs_mount.h |  6 +++---
4 files changed, 25 insertions(+), 33 deletions(-)

[GIT PULL 12/18] xfs: support in-memory btrees

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 1c51ac0998ed9baaca3ac75c0083b4c3b4d993ef:

xfs: move setting bt_logical_sectorsize out of xfs_setsize_buftarg (2024-02-22 12:42:45 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/in-memory-btrees-6.9_2024-02-23

for you to fetch changes up to 0dc63c8a1ce39c1ac7da536ee9174cdc714afae2:

xfs: launder in-memory btree buffers before transaction commit (2024-02-22 12:43:36 -0800)

----------------------------------------------------------------
xfs: support in-memory btrees [v29.3 12/18]

Online repair of the reverse-mapping btrees presens some unique
challenges.  To construct a new reverse mapping btree, we must scan the
entire filesystem, but we cannot afford to quiesce the entire filesystem
for the potentially lengthy scan.

For rmap btrees, therefore, we relax our requirements of totally atomic
repairs.  Instead, repairs will scan all inodes, construct a new reverse
mapping dataset, format a new btree, and commit it before anyone trips
over the corruption.  This is exactly the same strategy as was used in
the quotacheck and nlink scanners.

Unfortunately, the xfarray cannot perform key-based lookups and is
therefore unsuitable for supporting live updates.  Luckily, we already a
data structure that maintains an indexed rmap recordset -- the existing
rmap btree code!  Hence we port the existing btree and buffer target
code to be able to create a btree using the xfile we developed earlier.
Live hooks keep the in-memory btree up to date for any resources that
have already been scanned.

This approach is not maximally memory efficient, but we can use the same
rmap code that we do everywhere else, which provides improved stability
without growing the code base even more.  Note that in-memory btree
blocks are always page sized.

This patchset modifies the kernel xfs buffer cache to be capable of
using a xfile (aka a shmem file) as a backing device.  It then augments
the btree code to support creating btree cursors with buffers that come
from a buftarg other than the data device (namely an xfile-backed
buftarg).  For the userspace xfs buffer cache, we instead use a memfd or
an O_TMPFILE file as a backing device.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Christoph Hellwig (1):
xfs: add a xfs_btree_ptrs_equal helper

Darrick J. Wong (4):
xfs: teach buftargs to maintain their own buffer hashtable
xfs: support in-memory buffer cache targets
xfs: support in-memory btrees
xfs: launder in-memory btree buffers before transaction commit

.../filesystems/xfs/xfs-online-fsck-design.rst     |   5 +-
fs/xfs/Kconfig                                     |   8 +
fs/xfs/Makefile                                    |   2 +
fs/xfs/libxfs/xfs_ag.c                             |   6 +-
fs/xfs/libxfs/xfs_ag.h                             |   4 +-
fs/xfs/libxfs/xfs_btree.c                          | 286 ++++++++++++++---
fs/xfs/libxfs/xfs_btree.h                          |   7 +
fs/xfs/libxfs/xfs_btree_mem.c                      | 347 +++++++++++++++++++++
fs/xfs/libxfs/xfs_btree_mem.h                      |  75 +++++
fs/xfs/scrub/scrub.c                               |   5 +
fs/xfs/scrub/scrub.h                               |   3 +
fs/xfs/xfs_buf.c                                   | 214 ++++++++-----
fs/xfs/xfs_buf.h                                   |  17 +
fs/xfs/xfs_buf_mem.c                               | 270 ++++++++++++++++
fs/xfs/xfs_buf_mem.h                               |  34 ++
fs/xfs/xfs_health.c                                |   3 +
fs/xfs/xfs_mount.h                                 |   3 -
fs/xfs/xfs_trace.c                                 |   2 +
fs/xfs/xfs_trace.h                                 | 167 +++++++++-
fs/xfs/xfs_trans.h                                 |   1 +
fs/xfs/xfs_trans_buf.c                             |  42 +++
21 files changed, 1363 insertions(+), 138 deletions(-)
create mode 100644 fs/xfs/libxfs/xfs_btree_mem.c
create mode 100644 fs/xfs/libxfs/xfs_btree_mem.h
create mode 100644 fs/xfs/xfs_buf_mem.c
create mode 100644 fs/xfs/xfs_buf_mem.h

[GIT PULL 13/18] xfs: online repair of rmap btrees

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 0dc63c8a1ce39c1ac7da536ee9174cdc714afae2:

xfs: launder in-memory btree buffers before transaction commit (2024-02-22 12:43:36 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-rmap-btree-6.9_2024-02-23

for you to fetch changes up to 7e1b84b24d257700e417bc9cd724c1efdff653d7:

xfs: hook live rmap operations during a repair operation (2024-02-22 12:43:40 -0800)

----------------------------------------------------------------
xfs: online repair of rmap btrees [v29.3 13/18]

We have now constructed the four tools that we need to scan the
filesystem looking for reverse mappings: an inode scanner, hooks to
receive live updates from other writer threads, the ability to construct
btrees in memory, and a btree bulk loader.

This series glues those three together, enabling us to scan the
filesystem for mappings and keep it up to date while other writers run,
and then commit the new btree to disk atomically.

To reduce the size of each patch, the functionality is left disabled
until the end of the series and broken up into three patches: one to
create the mechanics of scanning the filesystem, a second to transition
to in-memory btrees, and a third to set up the live hooks.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (5):
xfs: create a helper to decide if a file mapping targets the rt volume
xfs: create agblock bitmap helper to count the number of set regions
xfs: repair the rmapbt
xfs: create a shadow rmap btree during rmap repair
xfs: hook live rmap operations during a repair operation

fs/xfs/Makefile                |    1 +
fs/xfs/libxfs/xfs_ag.c         |    1 +
fs/xfs/libxfs/xfs_ag.h         |    4 +
fs/xfs/libxfs/xfs_bmap.c       |   49 +-
fs/xfs/libxfs/xfs_bmap.h       |    8 +
fs/xfs/libxfs/xfs_inode_fork.c |    9 +
fs/xfs/libxfs/xfs_inode_fork.h |    1 +
fs/xfs/libxfs/xfs_rmap.c       |  199 +++--
fs/xfs/libxfs/xfs_rmap.h       |   31 +-
fs/xfs/libxfs/xfs_rmap_btree.c |  163 +++-
fs/xfs/libxfs/xfs_rmap_btree.h |    6 +
fs/xfs/libxfs/xfs_shared.h     |   10 +
fs/xfs/scrub/agb_bitmap.h      |    5 +
fs/xfs/scrub/bitmap.c          |   14 +
fs/xfs/scrub/bitmap.h          |    2 +
fs/xfs/scrub/bmap.c            |    2 +-
fs/xfs/scrub/common.c          |    5 +-
fs/xfs/scrub/common.h          |    1 +
fs/xfs/scrub/newbt.c           |   12 +-
fs/xfs/scrub/newbt.h           |    7 +
fs/xfs/scrub/reap.c            |    2 +-
fs/xfs/scrub/repair.c          |   59 +-
fs/xfs/scrub/repair.h          |   12 +-
fs/xfs/scrub/rmap.c            |   11 +-
fs/xfs/scrub/rmap_repair.c     | 1697 ++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/scrub.c           |    6 +-
fs/xfs/scrub/scrub.h           |    4 +-
fs/xfs/scrub/trace.c           |    1 +
fs/xfs/scrub/trace.h           |   80 +-
fs/xfs/xfs_stats.c             |    3 +-
fs/xfs/xfs_stats.h             |    1 +
31 files changed, 2336 insertions(+), 70 deletions(-)
create mode 100644 fs/xfs/scrub/rmap_repair.c

[GIT PULL 14/18] xfs: reduce refcount repair memory usage

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 7e1b84b24d257700e417bc9cd724c1efdff653d7:

xfs: hook live rmap operations during a repair operation (2024-02-22 12:43:40 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-refcount-scalability-6.9_2024-02-23

for you to fetch changes up to 7fbaab57a80f1639add1c7d02adeb9d17bd50206:

xfs: port refcount repair to the new refcount bag structure (2024-02-22 12:43:42 -0800)

----------------------------------------------------------------
xfs: reduce refcount repair memory usage [v29.3 14/18]

The refcountbt repair code has serious memory usage problems when the
block sharing factor of the filesystem is very high.  This can happen if
a deduplication tool has been run against the filesystem, or if the fs
stores reflinked VM images that have been aging for a long time.

Recall that the original reference counting algorithm walks the reverse
mapping records of the filesystem to generate reference counts.  For any
given block in the AG, the rmap bag structure contains the all rmap
records that cover that block; the refcount is the size of that bag.

For online repair, the bag doesn't need the owner, offset, or state flag
information, so it discards those.  This halves the record size, but the
bag structure still stores one excerpted record for each reverse
mapping.  If the sharing count is high, this will use a LOT of memory
storing redundant records.  In the extreme case, 100k mappings to the
same piece of space will consume 100k*16 bytes = 1.6M of memory.

For offline repair, the bag stores the owner values so that we know
which inodes need to be marked as being reflink inodes.  If a
deduplication tool has been run and there are many blocks within a file
pointing to the same physical space, this will stll use a lot of memory
to store redundant records.

The solution to this problem is to deduplicate the bag records when
possible by adding a reference count to the bag record, and changing the
bag add function to detect an existing record to bump the refcount.  In
the above example, the 100k mappings will now use 24 bytes of memory.
These lookups can be done efficiently with a btree, so we create a new
refcount bag btree type (inside of online repair).  This is why we
refactored the btree code in the previous patchset.

The btree conversion also dramatically reduces the runtime of the
refcount generation algorithm, because the code to delete all bag
records that end at a given agblock now only has to delete one record
instead of (using the example above) 100k records.  As an added benefit,
record deletion now gives back the unused xfile space, which it did not
do previously.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: define an in-memory btree for storing refcount bag info during repairs
xfs: create refcount bag structure for btree repairs
xfs: port refcount repair to the new refcount bag structure

fs/xfs/Makefile                |   2 +
fs/xfs/scrub/rcbag.c           | 307 ++++++++++++++++++++++++++++++++++
fs/xfs/scrub/rcbag.h           |  28 ++++
fs/xfs/scrub/rcbag_btree.c     | 370 +++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/rcbag_btree.h     |  81 +++++++++
fs/xfs/scrub/refcount.c        |  12 ++
fs/xfs/scrub/refcount_repair.c | 164 +++++++-----------
fs/xfs/scrub/repair.h          |   2 +
fs/xfs/xfs_stats.c             |   3 +-
fs/xfs/xfs_stats.h             |   1 +
fs/xfs/xfs_super.c             |  10 +-
11 files changed, 872 insertions(+), 108 deletions(-)
create mode 100644 fs/xfs/scrub/rcbag.c
create mode 100644 fs/xfs/scrub/rcbag.h
create mode 100644 fs/xfs/scrub/rcbag_btree.c
create mode 100644 fs/xfs/scrub/rcbag_btree.h

[GIT PULL 15/18] xfs: bmap log intent cleanups

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 7fbaab57a80f1639add1c7d02adeb9d17bd50206:

xfs: port refcount repair to the new refcount bag structure (2024-02-22 12:43:42 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/bmap-intent-cleanups-6.9_2024-02-23

for you to fetch changes up to c75f1a2c154979287ee12c336e2b8c3122832bf7:

xfs: add a xattr_entry helper (2024-02-22 12:44:22 -0800)

----------------------------------------------------------------
xfs: bmap log intent cleanups [v29.3 15/18]

The next major target of online repair are metadata that are persisted
in blocks mapped by a file fork.  In other words, we want to repair
directories, extended attributes, symbolic links, and the realtime free
space information.  For file-based metadata, we assume that the space
metadata is correct, which enables repair to construct new versions of
the metadata in a temporary file.  We then need to swap the file fork
mappings of the two files atomically.  With this patchset, we begin
constructing such a facility based on the existing bmap log items and a
new extent swap log item.

This series cleans up a few parts of the file block mapping log intent
code before we start adding support for realtime bmap intents.  Most of
it involves cleaning up tracepoints so that more of the data extraction
logic ends up in the tracepoint code and not the tracepoint call site,
which should reduce overhead further when tracepoints are disabled.
There is also a change to pass bmap intents all the way back to the bmap
code instead of unboxing the intent values and re-boxing them after the
_finish_one function completes.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (7):
xfs: split tracepoint classes for deferred items
xfs: clean up bmap log intent item tracepoint callsites
xfs: remove xfs_trans_set_bmap_flags
xfs: add a bi_entry helper
xfs: reuse xfs_bmap_update_cancel_item
xfs: move xfs_bmap_defer_add to xfs_bmap_item.c
xfs: add a xattr_entry helper

fs/xfs/libxfs/xfs_bmap.c |  21 +---
fs/xfs/libxfs/xfs_bmap.h |   7 +-
fs/xfs/xfs_attr_item.c   |  11 +-
fs/xfs/xfs_bmap_item.c   |  95 +++++++++--------
fs/xfs/xfs_bmap_item.h   |   4 +
fs/xfs/xfs_trace.c       |   1 +
fs/xfs/xfs_trace.h       | 267 ++++++++++++++++++++++++++++++-----------------
7 files changed, 237 insertions(+), 169 deletions(-)

[GIT PULL 16/18] xfs: widen BUI formats to support realtime

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit c75f1a2c154979287ee12c336e2b8c3122832bf7:

xfs: add a xattr_entry helper (2024-02-22 12:44:22 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/realtime-bmap-intents-6.9_2024-02-23

for you to fetch changes up to 1b5453baed3a43dd4726eda0e8a5618c56a4f3f7:

xfs: support recovering bmap intent items targetting realtime extents (2024-02-22 12:44:24 -0800)

----------------------------------------------------------------
xfs: widen BUI formats to support realtime [v29.3 16/18]

Atomic extent swapping (and later, reverse mapping and reflink) on the
realtime device needs to be able to defer file mapping and extent
freeing work in much the same manner as is required on the data volume.
Make the BUI log items operate on rt extents in preparation for atomic
swapping and realtime rmap.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: fix xfs_bunmapi to allow unmapping of partial rt extents
xfs: add a realtime flag to the bmap update log redo items
xfs: support recovering bmap intent items targetting realtime extents

fs/xfs/libxfs/xfs_bmap.c       |  4 ++--
fs/xfs/libxfs/xfs_log_format.h |  4 +++-
fs/xfs/xfs_bmap_item.c         | 17 +++++++++++++++++
fs/xfs/xfs_trace.h             | 23 ++++++++++++++++++-----
4 files changed, 40 insertions(+), 8 deletions(-)

[GIT PULL 17/18] xfs: support attrfork and unwritten BUIs

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 1b5453baed3a43dd4726eda0e8a5618c56a4f3f7:

xfs: support recovering bmap intent items targetting realtime extents (2024-02-22 12:44:24 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/expand-bmap-intent-usage_2024-02-23

for you to fetch changes up to 6c8127e93e3ac9c2cf6a13b885dd2d057b7e7d50:

xfs: xfs_bmap_finish_one should map unwritten extents properly (2024-02-22 12:45:00 -0800)

----------------------------------------------------------------
xfs: support attrfork and unwritten BUIs [v29.3 17/18]

In preparation for atomic extent swapping and the online repair
functionality that wants atomic extent swaps, enhance the BUI code so
that we can support deferred work on the extended attribute fork and on
unwritten extents.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (2):
xfs: support deferred bmap updates on the attr fork
xfs: xfs_bmap_finish_one should map unwritten extents properly

fs/xfs/libxfs/xfs_bmap.c | 49 +++++++++++++++++++++---------------------------
fs/xfs/libxfs/xfs_bmap.h |  4 ++--
fs/xfs/xfs_bmap_util.c   |  8 ++++----
fs/xfs/xfs_reflink.c     |  8 ++++----
4 files changed, 31 insertions(+), 38 deletions(-)

[GIT PULL 18/18] xfs: clean up symbolic link code

[Darrick J. Wong]

Hi Chandan,

Please pull this branch with changes for xfs for 6.9-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

--D

The following changes since commit 6c8127e93e3ac9c2cf6a13b885dd2d057b7e7d50:

xfs: xfs_bmap_finish_one should map unwritten extents properly (2024-02-22 12:45:00 -0800)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/symlink-cleanups-6.9_2024-02-23

for you to fetch changes up to b8102b61f7b8929ad8043e4574a1e26276398041:

xfs: move symlink target write function to libxfs (2024-02-22 12:52:37 -0800)

----------------------------------------------------------------
xfs: clean up symbolic link code [v29.3 18/18]

This series cleans up a few bits of the symbolic link code as needed for
future projects.  Online repair requires the ability to commit fixed
fork-based filesystem metadata such as directories, xattrs, and symbolic
links atomically, so we need to rearrange the symlink code before we
land the atomic extent swapping.

Accomplish this by moving the remote symlink target block code and
declarations to xfs_symlink_remote.[ch].

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (3):
xfs: move xfs_symlink_remote.c declarations to xfs_symlink_remote.h
xfs: move remote symlink target read function to libxfs
xfs: move symlink target write function to libxfs

fs/xfs/libxfs/xfs_bmap.c           |   1 +
fs/xfs/libxfs/xfs_inode_fork.c     |   1 +
fs/xfs/libxfs/xfs_shared.h         |  13 ----
fs/xfs/libxfs/xfs_symlink_remote.c | 155 ++++++++++++++++++++++++++++++++++++-
fs/xfs/libxfs/xfs_symlink_remote.h |  26 +++++++
fs/xfs/scrub/inode_repair.c        |   1 +
fs/xfs/scrub/symlink.c             |   3 +-
fs/xfs/xfs_symlink.c               | 146 ++--------------------------------
fs/xfs/xfs_symlink.h               |   1 -
9 files changed, 192 insertions(+), 155 deletions(-)
create mode 100644 fs/xfs/libxfs/xfs_symlink_remote.h

Re: [PRBOMB] xfs: online repair patches for 6.9

[Chandan Babu R]

On Fri, Feb 23, 2024 at 05:02:20 PM -0800, Darrick J. Wong wrote:
> Hi Chandan,
>
> Please pull these multiple pull requests containing all the online
> repair functionality that Christoph and I managed to get reviewed in
> time for 6.9.
>
> --D
>
> PS: Has anyone out there observed the following crash on arm64?
>
> run fstests xfs/570 at 2024-02-22 20:32:17
> spectre-v4 mitigation disabled by command-line option
> XFS (sda2): Mounting V5 Filesystem 2fd78ebc-692d-46e1-bd8a-9f1591c007f6
> XFS (sda2): Ending clean mount
> XFS (sda2): EXPERIMENTAL online scrub feature in use. Use at your own risk!
> XFS (sda3): Mounting V5 Filesystem ac0b0a07-294f-418e-b4d0-c14cc345fcd4
> XFS (sda3): Ending clean mount
> XFS (sda3): EXPERIMENTAL online scrub feature in use. Use at your own risk!
> Unable to handle kernel paging request at virtual address ffffffff80206388
> Mem abort info:
>   ESR = 0x0000000096000006
>   EC = 0x25: DABT (current EL), IL = 32 bits
>   SET = 0, FnV = 0
>   EA = 0, S1PTW = 0
>   FSC = 0x06: level 2 translation fault
> Data abort info:
>   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
>   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
>   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
> swapper pgtable: 64k pages, 42-bit VAs, pgdp=0000000040d40000
> [ffffffff80206388] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000, pmd=0000000000000000
> Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP
> Dumping ftrace buffer:
>    (ftrace buffer empty)
> Modules linked in: xfs rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_REDIRECT nf_nat nf_conntrack nf_defra
> sch_fq_codel fuse configfs efivarfs ip_tables x_tables overlay nfsv4
> CPU: 0 PID: 38 Comm: 0:1H Not tainted 6.8.0-rc4-djwa #rc4 892edfc98307d2cdb226d4164dfd6775c2b3b52c
> Hardware name: QEMU KVM Virtual Machine, BIOS 1.6.6 08/22/2023
> Workqueue: xfs-log/sda3 xlog_ioend_work [xfs]
> pstate: a0401005 (NzCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)
> pc : kfree+0x54/0x2d8
> lr : xlog_cil_committed+0x11c/0x1d8 [xfs]
> sp : fffffe00830cfbe0
> x29: fffffe00830cfbe0 x28: fffffc00240e0c80 x27: fffffc00240e0c00
> x26: 00000005000037a8 x25: 0000000000000000 x24: 0000000000000000
> x23: fffffc0021e68d40 x22: fffffe00818e0000 x21: fffffc0021e68d88
> x20: fffffe007a6b93cc x19: ffffffff80206380 x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: fffffe008840f620
> x14: 0000000000000000 x13: 0000000000000020 x12: 0101010101010101
> x11: 0000000000000040 x10: fffffc0025b0d5d8 x9 : fffffe007a6b93cc
> x8 : fffffe00830cfbe0 x7 : 0000000000000000 x6 : 0000000000000000
> x5 : d230261c49c51c03 x4 : fffffc00e1357340 x3 : dead000000000122
> x2 : fffffc00ea7fa000 x1 : fffffc0021e68d88 x0 : ffffffff00000000
> Call trace:
>  kfree+0x54/0x2d8
>  xlog_cil_committed+0x11c/0x1d8 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
>  xlog_cil_process_committed+0x6c/0xa8 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
>  xlog_state_do_callback+0x1e0/0x3e0 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
>  xlog_state_done_syncing+0x8c/0x158 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
>  xlog_ioend_work+0x70/0xd8 [xfs 6eb07a1ebfe13a228ea62c550c04c138eaa0de6a]
>  process_one_work+0x174/0x3e8
>  worker_thread+0x2c4/0x3e8
>  kthread+0x110/0x128
>  ret_from_fork+0x10/0x20
> Code: 8b1302d3 b2607fe0 d350fe73 8b131813 (f9400660) 
> ---[ end trace 0000000000000000 ]---

I just now noticed that generic/019 has failed with the same call trace as
above on an x86_64 machine. I will try to root cause the problem.

-- 
Chandan

Re: [PRBOMB] xfs: online repair patches for 6.9

[Christoph Hellwig]

On Fri, Feb 23, 2024 at 05:02:20PM -0800, Darrick J. Wong wrote:
> pc : kfree+0x54/0x2d8
> lr : xlog_cil_committed+0x11c/0x1d8 [xfs]

This looks a lot like the bug I found in getbmap.  Maybe try changing
that kfree to a kvfree?

Re: [PRBOMB] xfs: online repair patches for 6.9

[Dave Chinner]

On Mon, Feb 26, 2024 at 02:26:46AM -0800, Christoph Hellwig wrote:
> On Fri, Feb 23, 2024 at 05:02:20PM -0800, Darrick J. Wong wrote:
> > pc : kfree+0x54/0x2d8
> > lr : xlog_cil_committed+0x11c/0x1d8 [xfs]
> 
> This looks a lot like the bug I found in getbmap.  Maybe try changing
> that kfree to a kvfree?

yup, I think it is - it's the freeing of the logvec chain that does
it, I think. That memory comes from xlog_kvmalloc().

I have no idea how I haven't tripped over that in my testing as I've
done plenty of 64kB directory block size testing in the past few
weeks...

I'll get a patch out for it and the other kfree issue that was
reported late last week tomorrow...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PRBOMB] xfs: online repair patches for 6.9

[Chandan Babu R]

On Mon, Feb 26, 2024 at 02:26:46 AM -0800, Christoph Hellwig wrote:
> On Fri, Feb 23, 2024 at 05:02:20PM -0800, Darrick J. Wong wrote:
>> pc : kfree+0x54/0x2d8
>> lr : xlog_cil_committed+0x11c/0x1d8 [xfs]
>
> This looks a lot like the bug I found in getbmap.  Maybe try changing
> that kfree to a kvfree?

CIL context structures are allocated using kzalloc() (i.e. kmalloc() with
__GFP_ZERO flag appended to flags argument). So kfree() should work right?

Darrick, I have not been able to recreate the bug after several attempts.

-- 
Chandan

Re: [PRBOMB] xfs: online repair patches for 6.9

[Chandan Babu R]

On Mon, Feb 26, 2024 at 06:29:43 PM +0530, Chandan Babu R wrote:
> On Mon, Feb 26, 2024 at 02:26:46 AM -0800, Christoph Hellwig wrote:
>> On Fri, Feb 23, 2024 at 05:02:20PM -0800, Darrick J. Wong wrote:
>>> pc : kfree+0x54/0x2d8
>>> lr : xlog_cil_committed+0x11c/0x1d8 [xfs]
>>
>> This looks a lot like the bug I found in getbmap.  Maybe try changing
>> that kfree to a kvfree?
>
> CIL context structures are allocated using kzalloc() (i.e. kmalloc() with
> __GFP_ZERO flag appended to flags argument). So kfree() should work right?
>

Sorry, I read Dave's email now. I had mistakenly thought that the call to
kfree() is the one which frees CIL context structure.

-- 
Chandan