From: "Darrick J. Wong" <djwong@kernel.org>
To: Carlos Maiolino <cmaiolino@redhat.com>,
Zorro Lang <zlang@redhat.com>,
Andrey Albershteyn <aalbersh@redhat.com>,
Christoph Hellwig <hch@infradead.org>
Cc: xfs <linux-xfs@vger.kernel.org>,
greg.marsden@oracle.com, shirley.ma@oracle.com,
konrad.wilk@oracle.com, fstests <fstests@vger.kernel.org>
Subject: [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing
Date: Tue, 31 Dec 2024 15:25:03 -0800 [thread overview]
Message-ID: <20241231232503.GU6174@frogsfrogsfrogs> (raw)
Hi everyone,
Thank you all for helping get online repair, parent pointers, and
metadata directory trees, and realtime allocation groups merged this
year! We got a lot done in 2024.
Having sent pull requests to Carlos for the last pieces of the realtime
modernization project, I have exactly two worthwhile projects left in my
development trees! The stuff here isn't necessarily in mergeable state
yet, but I still believe everyone ought to know what I'm up to.
The first project implements (somewhat buggily; I never quite got back
to dealing with moving eof blocks) free space defragmentation so that we
can meaningfully shrink filesystems; garbage collect regions of the
filesystem; or prepare for large allocations. There's not much new
kernel code other than exporting refcounts and gaining the ability to
map free space.
The second project initiates filesystem self healing routines whenever
problems start to crop up, which means that it can run fully
autonomously in the background. The monitoring system uses some
pseudo-file and seqbuf tricks that I lifted from kmo last winter.
Both of these projects are largely userspace code.
Also I threw in some xfs_repair code to do dangerous fs upgrades.
Nobody should use these, ever.
Maintainers: please do not merge, this is a dog-and-pony show to attract
developer attention.
--D
PS: I'll be back after the holidays to look at the zoned/atomic/fsverity
patches. And finally rebase fstests to 2024-12-08.
next reply other threads:[~2024-12-31 23:25 UTC|newest]
Thread overview: 110+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-12-31 23:25 Darrick J. Wong [this message]
2024-12-31 23:32 ` [PATCHSET 1/5] xfs: improve post-close eofblocks gc behavior Darrick J. Wong
2024-12-31 23:36 ` [PATCH 1/1] xfs: Don't free EOF blocks on close when extent size hints are set Darrick J. Wong
2024-12-31 23:32 ` [PATCHSET RFC 2/5] xfs: noalloc allocation groups Darrick J. Wong
2024-12-31 23:36 ` [PATCH 1/5] xfs: track deferred ops statistics Darrick J. Wong
2024-12-31 23:36 ` [PATCH 2/5] xfs: whine to dmesg when we encounter errors Darrick J. Wong
2024-12-31 23:37 ` [PATCH 3/5] xfs: create a noalloc mode for allocation groups Darrick J. Wong
2024-12-31 23:37 ` [PATCH 4/5] xfs: enable userspace to hide an AG from allocation Darrick J. Wong
2024-12-31 23:37 ` [PATCH 5/5] xfs: apply noalloc mode to inode allocations too Darrick J. Wong
2024-12-31 23:32 ` [PATCHSET 3/5] xfs: report refcount information to userspace Darrick J. Wong
2024-12-31 23:37 ` [PATCH 1/1] xfs: export reference count " Darrick J. Wong
2024-12-31 23:33 ` [PATCHSET 4/5] xfs: defragment free space Darrick J. Wong
2024-12-31 23:38 ` [PATCH 1/4] xfs: export realtime refcount information Darrick J. Wong
2024-12-31 23:38 ` [PATCH 2/4] xfs: capture the offset and length in fallocate tracepoints Darrick J. Wong
2024-12-31 23:38 ` [PATCH 3/4] xfs: add an ioctl to map free space into a file Darrick J. Wong
2024-12-31 23:38 ` [PATCH 4/4] xfs: implement FALLOC_FL_MAP_FREE for realtime files Darrick J. Wong
2024-12-31 23:33 ` [PATCHSET 5/5] xfs: live health monitoring of filesystems Darrick J. Wong
2024-12-31 23:39 ` [PATCH 01/16] xfs: create debugfs uuid aliases Darrick J. Wong
2024-12-31 23:39 ` [PATCH 02/16] xfs: create hooks for monitoring health updates Darrick J. Wong
2024-12-31 23:39 ` [PATCH 03/16] xfs: create a filesystem shutdown hook Darrick J. Wong
2024-12-31 23:39 ` [PATCH 04/16] xfs: create hooks for media errors Darrick J. Wong
2024-12-31 23:40 ` [PATCH 05/16] iomap, filemap: report buffered read and write io errors to the filesystem Darrick J. Wong
2024-12-31 23:40 ` [PATCH 06/16] iomap: report directio read and write errors to callers Darrick J. Wong
2024-12-31 23:40 ` [PATCH 07/16] xfs: create file io error hooks Darrick J. Wong
2024-12-31 23:40 ` [PATCH 08/16] xfs: create a special file to pass filesystem health to userspace Darrick J. Wong
2024-12-31 23:41 ` [PATCH 09/16] xfs: create event queuing, formatting, and discovery infrastructure Darrick J. Wong
2024-12-31 23:41 ` [PATCH 10/16] xfs: report metadata health events through healthmon Darrick J. Wong
2024-12-31 23:41 ` [PATCH 11/16] xfs: report shutdown " Darrick J. Wong
2024-12-31 23:41 ` [PATCH 12/16] xfs: report media errors " Darrick J. Wong
2024-12-31 23:42 ` [PATCH 13/16] xfs: report file io " Darrick J. Wong
2024-12-31 23:42 ` [PATCH 14/16] xfs: allow reconfiguration of the health monitoring device Darrick J. Wong
2024-12-31 23:42 ` [PATCH 15/16] xfs: add media error reporting ioctl Darrick J. Wong
2024-12-31 23:43 ` [PATCH 16/16] xfs: send uevents when mounting and unmounting a filesystem Darrick J. Wong
2024-12-31 23:33 ` [PATCHSET RFC 1/5] xfsprogs: noalloc allocation groups Darrick J. Wong
2024-12-31 23:43 ` [PATCH 1/5] xfs: track deferred ops statistics Darrick J. Wong
2024-12-31 23:43 ` [PATCH 2/5] xfs: create a noalloc mode for allocation groups Darrick J. Wong
2024-12-31 23:43 ` [PATCH 3/5] xfs: enable userspace to hide an AG from allocation Darrick J. Wong
2024-12-31 23:44 ` [PATCH 4/5] xfs: apply noalloc mode to inode allocations too Darrick J. Wong
2024-12-31 23:44 ` [PATCH 5/5] xfs_io: enhance the aginfo command to control the noalloc flag Darrick J. Wong
2024-12-31 23:33 ` [PATCHSET 2/5] xfsprogs: report refcount information to userspace Darrick J. Wong
2024-12-31 23:44 ` [PATCH 1/2] xfs: export reference count " Darrick J. Wong
2024-12-31 23:44 ` [PATCH 2/2] xfs_io: dump reference count information Darrick J. Wong
2024-12-31 23:34 ` [PATCHSET 3/5] xfsprogs: defragment free space Darrick J. Wong
2024-12-31 23:45 ` [PATCH 01/11] xfs_io: display rtgroup number in verbose fsrefs output Darrick J. Wong
2024-12-31 23:45 ` [PATCH 02/11] xfs: add an ioctl to map free space into a file Darrick J. Wong
2024-12-31 23:45 ` [PATCH 03/11] xfs_io: support using XFS_IOC_MAP_FREESP to map free space Darrick J. Wong
2024-12-31 23:45 ` [PATCH 04/11] xfs_db: get and put blocks on the AGFL Darrick J. Wong
2024-12-31 23:46 ` [PATCH 05/11] xfs_spaceman: implement clearing free space Darrick J. Wong
2024-12-31 23:46 ` [PATCH 06/11] spaceman: physically move a regular inode Darrick J. Wong
2024-12-31 23:46 ` [PATCH 07/11] spaceman: find owners of space in an AG Darrick J. Wong
2024-12-31 23:46 ` [PATCH 08/11] xfs_spaceman: wrap radix tree accesses in find_owner.c Darrick J. Wong
2024-12-31 23:47 ` [PATCH 09/11] xfs_spaceman: port relocation structure to 32-bit systems Darrick J. Wong
2024-12-31 23:47 ` [PATCH 10/11] spaceman: relocate the contents of an AG Darrick J. Wong
2024-12-31 23:47 ` [PATCH 11/11] spaceman: move inodes with hardlinks Darrick J. Wong
2024-12-31 23:34 ` [PATCHSET 4/5] xfsprogs: live health monitoring of filesystems Darrick J. Wong
2024-12-31 23:47 ` [PATCH 01/21] xfs: create hooks for monitoring health updates Darrick J. Wong
2024-12-31 23:48 ` [PATCH 02/21] xfs: create a special file to pass filesystem health to userspace Darrick J. Wong
2024-12-31 23:48 ` [PATCH 03/21] xfs: create event queuing, formatting, and discovery infrastructure Darrick J. Wong
2024-12-31 23:48 ` [PATCH 04/21] xfs: report metadata health events through healthmon Darrick J. Wong
2024-12-31 23:49 ` [PATCH 05/21] xfs: report shutdown " Darrick J. Wong
2024-12-31 23:49 ` [PATCH 06/21] xfs: report media errors " Darrick J. Wong
2024-12-31 23:49 ` [PATCH 07/21] xfs: report file io " Darrick J. Wong
2024-12-31 23:49 ` [PATCH 08/21] xfs: add media error reporting ioctl Darrick J. Wong
2024-12-31 23:50 ` [PATCH 09/21] xfs_io: monitor filesystem health events Darrick J. Wong
2024-12-31 23:50 ` [PATCH 10/21] xfs_io: add a media error reporting command Darrick J. Wong
2024-12-31 23:50 ` [PATCH 11/21] xfs_scrubbed: create daemon to listen for health events Darrick J. Wong
2024-12-31 23:50 ` [PATCH 12/21] xfs_scrubbed: check events against schema Darrick J. Wong
2024-12-31 23:51 ` [PATCH 13/21] xfs_scrubbed: enable repairing filesystems Darrick J. Wong
2024-12-31 23:51 ` [PATCH 14/21] xfs_scrubbed: check for fs features needed for effective repairs Darrick J. Wong
2024-12-31 23:51 ` [PATCH 15/21] xfs_scrubbed: use getparents to look up file names Darrick J. Wong
2024-12-31 23:51 ` [PATCH 16/21] builddefs: refactor udev directory specification Darrick J. Wong
2024-12-31 23:52 ` [PATCH 17/21] xfs_scrubbed: create a background monitoring service Darrick J. Wong
2024-12-31 23:52 ` [PATCH 18/21] xfs_scrubbed: don't start service if kernel support unavailable Darrick J. Wong
2024-12-31 23:52 ` [PATCH 19/21] xfs_scrubbed: use the autofsck fsproperty to select mode Darrick J. Wong
2024-12-31 23:52 ` [PATCH 20/21] xfs_scrub: report media scrub failures to the kernel Darrick J. Wong
2024-12-31 23:53 ` [PATCH 21/21] debian: enable xfs_scrubbed on the root filesystem by default Darrick J. Wong
2024-12-31 23:34 ` [PATCHSET 5/5] xfs_repair: add difficult V5 features to filesystems Darrick J. Wong
2024-12-31 23:53 ` [PATCH 01/10] xfs_repair: allow sysadmins to add free inode btree indexes Darrick J. Wong
2024-12-31 23:53 ` [PATCH 02/10] xfs_repair: allow sysadmins to add reflink Darrick J. Wong
2024-12-31 23:53 ` [PATCH 03/10] xfs_repair: allow sysadmins to add reverse mapping indexes Darrick J. Wong
2024-12-31 23:54 ` [PATCH 04/10] xfs_repair: upgrade an existing filesystem to have parent pointers Darrick J. Wong
2024-12-31 23:54 ` [PATCH 05/10] xfs_repair: allow sysadmins to add metadata directories Darrick J. Wong
2024-12-31 23:54 ` [PATCH 06/10] xfs_repair: upgrade filesystems to support rtgroups when adding metadir Darrick J. Wong
2024-12-31 23:55 ` [PATCH 07/10] xfs_repair: allow sysadmins to add realtime reverse mapping indexes Darrick J. Wong
2024-12-31 23:55 ` [PATCH 08/10] xfs_repair: allow sysadmins to add realtime reflink Darrick J. Wong
2024-12-31 23:55 ` [PATCH 09/10] xfs_repair: skip free space checks when upgrading Darrick J. Wong
2024-12-31 23:55 ` [PATCH 10/10] xfs_repair: allow adding rmapbt to reflink filesystems Darrick J. Wong
2024-12-31 23:34 ` [PATCHSET 1/5] fstests: functional test for refcount reporting Darrick J. Wong
2024-12-31 23:56 ` [PATCH 1/1] xfs: test output of new FSREFCOUNTS ioctl Darrick J. Wong
2024-12-31 23:35 ` [PATCHSET 2/5] fstests: defragment free space Darrick J. Wong
2024-12-31 23:56 ` [PATCH 1/1] xfs: test clearing of " Darrick J. Wong
2024-12-31 23:35 ` [PATCHSET 3/5] fstests: capture logs from mount failures Darrick J. Wong
2024-12-31 23:56 ` [PATCH 1/2] treewide: convert all $MOUNT_PROG to _mount Darrick J. Wong
2024-12-31 23:56 ` [PATCH 2/2] check: capture dmesg of mount failures if test fails Darrick J. Wong
2025-01-06 11:18 ` Nirjhar Roy
2025-01-06 23:52 ` Darrick J. Wong
2025-01-13 5:55 ` Nirjhar Roy
2024-12-31 23:35 ` [PATCHSET 4/5] fstests: live health monitoring of filesystems Darrick J. Wong
2024-12-31 23:57 ` [PATCH 1/6] misc: convert all $UMOUNT_PROG to a _umount helper Darrick J. Wong
2024-12-31 23:57 ` [PATCH 2/6] misc: convert all umount(1) invocations to _umount Darrick J. Wong
2024-12-31 23:57 ` [PATCH 3/6] xfs: test health monitoring code Darrick J. Wong
2024-12-31 23:57 ` [PATCH 4/6] xfs: test for metadata corruption error reporting via healthmon Darrick J. Wong
2024-12-31 23:58 ` [PATCH 5/6] xfs: test io " Darrick J. Wong
2024-12-31 23:58 ` [PATCH 6/6] xfs: test new xfs_scrubbed daemon Darrick J. Wong
2024-12-31 23:35 ` [PATCHSET 5/5] fstests: add difficult V5 features to filesystems Darrick J. Wong
2024-12-31 23:58 ` [PATCH 1/3] xfs/1856: add metadir upgrade to test matrix Darrick J. Wong
2024-12-31 23:58 ` [PATCH 2/3] xfs/1856: add rtrmapbt " Darrick J. Wong
2024-12-31 23:59 ` [PATCH 3/3] xfs/1856: add rtreflink " Darrick J. Wong
2025-01-02 1:37 ` [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing Stephen Zhang
2025-01-07 0:26 ` Darrick J. Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241231232503.GU6174@frogsfrogsfrogs \
--to=djwong@kernel.org \
--cc=aalbersh@redhat.com \
--cc=cmaiolino@redhat.com \
--cc=fstests@vger.kernel.org \
--cc=greg.marsden@oracle.com \
--cc=hch@infradead.org \
--cc=konrad.wilk@oracle.com \
--cc=linux-xfs@vger.kernel.org \
--cc=shirley.ma@oracle.com \
--cc=zlang@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox