From: Dave Chinner <david@fromorbit.com>
To: "Darrick J. Wong" <djwong@kernel.org>
Cc: Kent Overstreet <kent.overstreet@linux.dev>,
linux-xfs@vger.kernel.org, willy@infradead.org,
linux-fsdevel@vger.kernel.org
Subject: Re: [PATCHSET v25.0 0/7] xfs: stage repair information in pageable memory
Date: Thu, 22 Jun 2023 12:58:08 +1000 [thread overview]
Message-ID: <ZJO4wDMYEdaA0zc4@dread.disaster.area> (raw)
In-Reply-To: <168506056447.3729324.13624212283929857624.stgit@frogsfrogsfrogs>
On Thu, May 25, 2023 at 05:28:55PM -0700, Darrick J. Wong wrote:
> Hi all,
>
> In general, online repair of an indexed record set walks the filesystem
> looking for records. These records are sorted and bulk-loaded into a
> new btree. To make this happen without pinning gigabytes of metadata in
> memory, first create an abstraction ('xfile') of memfd files so that
> kernel code can access paged memory, and then an array abstraction
> ('xfarray') based on xfiles so that online repair can create an array of
> new records without pinning memory.
>
> These two data storage abstractions are critical for repair of space
> metadata -- the memory used is pageable, which helps us avoid pinning
> kernel memory and driving OOM problems; and they are byte-accessible
> enough that we can use them like (very slow and programmatic) memory
> buffers.
>
> Later patchsets will build on this functionality to provide blob storage
> and btrees.
Apart from the need for a struct xfs-mount just for the xfile name
at creation time, this all looks OK.
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Is there any specific test harness for this infrastructure, or is it
just validated by having other functions built on top of it "work
correctly"?
-Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2023-06-22 2:58 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-26 0:00 [MEGAPATCHSET v25 1/2] xfs: online repair, part 1 Darrick J. Wong
2023-05-26 0:28 ` [PATCHSET v25.0 0/7] xfs: stage repair information in pageable memory Darrick J. Wong
2023-05-26 0:47 ` [PATCH 1/7] xfs: create a big array data structure Darrick J. Wong
2023-05-26 1:34 ` Kent Overstreet
2023-05-26 3:19 ` Darrick J. Wong
2023-06-22 2:55 ` Dave Chinner
2023-07-05 23:48 ` Darrick J. Wong
2023-05-26 0:47 ` [PATCH 2/7] xfs: enable sorting of xfile-backed arrays Darrick J. Wong
2023-05-26 0:47 ` [PATCH 3/7] xfs: convert xfarray insertion sort to heapsort using scratchpad memory Darrick J. Wong
2023-05-26 0:47 ` [PATCH 4/7] xfs: teach xfile to pass back direct-map pages to caller Darrick J. Wong
2023-05-26 0:48 ` [PATCH 5/7] xfs: speed up xfarray sort by sorting xfile page contents directly Darrick J. Wong
2023-05-26 0:48 ` [PATCH 6/7] xfs: cache pages used for xfarray quicksort convergence Darrick J. Wong
2023-05-26 0:48 ` [PATCH 7/7] xfs: improve xfarray quicksort pivot Darrick J. Wong
2023-06-22 2:58 ` Dave Chinner [this message]
2023-05-26 0:32 ` [PATCHSET v25.0 0/9] xfs: support in-memory btrees Darrick J. Wong
2023-05-26 1:04 ` [PATCH 1/9] xfs: dump xfiles for debugging purposes Darrick J. Wong
2023-05-26 1:05 ` [PATCH 2/9] xfs: teach buftargs to maintain their own buffer hashtable Darrick J. Wong
2023-05-26 1:05 ` [PATCH 3/9] xfs: create buftarg helpers to abstract block_device operations Darrick J. Wong
2023-05-26 1:05 ` [PATCH 4/9] xfs: make GFP_ usage consistent when allocating buftargs Darrick J. Wong
2023-05-26 1:05 ` [PATCH 5/9] xfs: support in-memory buffer cache targets Darrick J. Wong
2023-05-26 1:06 ` [PATCH 6/9] xfs: consolidate btree block freeing tracepoints Darrick J. Wong
2023-05-26 1:06 ` [PATCH 7/9] xfs: consolidate btree block allocation tracepoints Darrick J. Wong
2023-05-26 1:06 ` [PATCH 8/9] xfs: support in-memory btrees Darrick J. Wong
2023-05-26 1:06 ` [PATCH 9/9] xfs: connect in-memory btrees to xfiles Darrick J. Wong
2023-05-26 0:34 ` [PATCHSET v25.0 00/25] xfs: atomic file updates Darrick J. Wong
2023-05-26 1:14 ` [PATCH 01/25] xfs: add a libxfs header file for staging new ioctls Darrick J. Wong
2023-05-26 1:14 ` [PATCH 02/25] xfs: introduce new file range exchange ioctl Darrick J. Wong
2023-05-26 1:15 ` [PATCH 03/25] xfs: move inode lease breaking functions to xfs_inode.c Darrick J. Wong
2023-05-26 1:15 ` [PATCH 04/25] xfs: move xfs_iops.c declarations out of xfs_inode.h Darrick J. Wong
2023-05-26 1:15 ` [PATCH 05/25] xfs: declare xfs_file.c symbols in xfs_file.h Darrick J. Wong
2023-05-26 1:16 ` [PATCH 06/25] xfs: create a new helper to return a file's allocation unit Darrick J. Wong
2023-05-26 1:16 ` [PATCH 07/25] xfs: refactor non-power-of-two alignment checks Darrick J. Wong
2023-05-26 1:16 ` [PATCH 08/25] xfs: parameterize all the incompat log feature helpers Darrick J. Wong
2023-05-26 1:16 ` [PATCH 09/25] xfs: create a log incompat flag for atomic extent swapping Darrick J. Wong
2023-05-26 1:17 ` [PATCH 10/25] xfs: introduce a swap-extent log intent item Darrick J. Wong
2023-05-26 1:17 ` [PATCH 11/25] xfs: create deferred log items for extent swapping Darrick J. Wong
2023-05-26 1:17 ` [PATCH 12/25] xfs: enable xlog users to toggle atomic " Darrick J. Wong
2023-05-26 1:17 ` [PATCH 13/25] xfs: bind the xfs-specific extent swape code to the vfs-generic file exchange code Darrick J. Wong
2023-05-26 1:18 ` [PATCH 14/25] xfs: add error injection to test swapext recovery Darrick J. Wong
2023-05-26 1:18 ` [PATCH 15/25] xfs: port xfs_swap_extents_rmap to our new code Darrick J. Wong
2023-05-26 1:18 ` [PATCH 16/25] xfs: consolidate all of the xfs_swap_extent_forks code Darrick J. Wong
2023-05-26 1:19 ` [PATCH 17/25] xfs: port xfs_swap_extent_forks to use xfs_swapext_req Darrick J. Wong
2023-05-26 1:26 ` [PATCH 18/25] xfs: allow xfs_swap_range to use older extent swap algorithms Darrick J. Wong
2023-05-26 1:26 ` [PATCH 19/25] xfs: remove old swap extents implementation Darrick J. Wong
2023-05-26 1:27 ` [PATCH 20/25] xfs: condense extended attributes after an atomic swap Darrick J. Wong
2023-05-26 1:27 ` [PATCH 21/25] xfs: condense directories " Darrick J. Wong
2023-05-26 1:27 ` [PATCH 22/25] xfs: condense symbolic links " Darrick J. Wong
2023-05-26 1:28 ` [PATCH 23/25] xfs: make atomic extent swapping support realtime files Darrick J. Wong
2023-05-26 1:28 ` [PATCH 24/25] xfs: support non-power-of-two rtextsize with exchange-range Darrick J. Wong
2023-05-26 1:28 ` [PATCH 25/25] xfs: enable atomic swapext feature Darrick J. Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZJO4wDMYEdaA0zc4@dread.disaster.area \
--to=david@fromorbit.com \
--cc=djwong@kernel.org \
--cc=kent.overstreet@linux.dev \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).