Re: [PATCH V10 00/10] famfs: port into fuse

John Groves
2026-04-10T14:46:23Z

On 26/04/06 10:43AM, Joanne Koong wrote:
> On Tue, Mar 31, 2026 at 5:37 AM John Groves <john@jagalactic.com> wrote:
> >
> > From: John Groves <john@groves.net>
> >
> > NOTE: this series depends on the famfs dax series in Ira's for-7.1/dax-famfs
> > branch [0]
> >
> > Changes v9 -> v10
> > - Rebased to Ira's for-7.1/dax-famfs branch [0], which contains the required
> > dax patches
> > - Add parentheses to FUSE_IS_VIRTIO_DAX() macro, in case something bad is
> > passed in as fuse_inode (thanks Jonathan's AI)
> >
> > Description:
> >
> > This patch series introduces famfs into the fuse file system framework.
> > Famfs depends on the bundled dax patch set.
> >
> > The famfs user space code can be found at [1].
> >
> > Fuse Overview:
> >
> > Famfs started as a standalone file system, but this series is intended to
> > permanently supersede that implementation. At a high level, famfs adds
> > two new fuse server messages:
> >
> > GET_FMAP - Retrieves a famfs fmap (the file-to-dax map for a famfs
> > file)
> > GET_DAXDEV - Retrieves the details of a particular daxdev that was
> > referenced by an fmap
> >
> > Famfs Overview
> >
> > Famfs exposes shared memory as a file system. Famfs consumes shared
> > memory from dax devices, and provides memory-mappable files that map
> > directly to the memory - no page cache involvement. Famfs differs from
> > conventional file systems in fs-dax mode, in that it handles in-memory
> > metadata in a sharable way (which begins with never caching dirty shared
> > metadata).
> >
> > Famfs started as a standalone file system [2,3], but the consensus at
> > LSFMM was that it should be ported into fuse [4,5].
> >
> > The key performance requirement is that famfs must resolve mapping faults
> > without upcalls. This is achieved by fully caching the file-to-devdax
> > metadata for all active files. This is done via two fuse client/server
> > message/response pairs: GET_FMAP and GET_DAXDEV.
> >
> > Famfs remains the first fs-dax file system that is backed by devdax
> > rather than pmem in fs-dax mode (hence the need for the new dax mode).
> >
> > Notes
> >
> > - When a file is opened in a famfs mount, the OPEN is followed by a
> > GET_FMAP message and response. The "fmap" is the full file-to-dax
> > mapping, allowing the fuse/famfs kernel code to handle
> > read/write/fault without any upcalls.
> >
> > - After each GET_FMAP, the fmap is checked for extents that reference
> > previously-unknown daxdevs. Each such occurrence is handled with a
> > GET_DAXDEV message and response.
> >
> > - Daxdevs are stored in a table (which might become an xarray at some
> > point). When entries are added to the table, we acquire exclusive
> > access to the daxdev via the fs_dax_get() call (modeled after how
> > fs-dax handles this with pmem devices). Famfs provides
> > holder_operations to devdax, providing a notification path in the
> > event of memory errors or forced reconfiguration.
> >
> > - If devdax notifies famfs of memory errors on a dax device, famfs
> > currently blocks all subsequent accesses to data on that device. The
> > recovery is to re-initialize the memory and file system. Famfs is
> > memory, not storage...
> >
> > - Because famfs uses backing (devdax) devices, only privileged mounts are
> > supported (i.e. the fuse server requires CAP_SYS_RAWIO).
> >
> > - The famfs kernel code never accesses the memory directly - it only
> > facilitates read, write and mmap on behalf of user processes, using
> > fmap metadata provided by its privileged fuse server. As such, the
> > RAS of the shared memory affects applications, but not the kernel.
> >
> > - Famfs has backing device(s), but they are devdax (char) rather than
> > block. Right now there is no way to tell the vfs layer that famfs has a
> > char backing device (unless we say it's block, but it's not). Currently
> > we use the standard anonymous fuse fs_type - but I'm not sure that's
> > ultimately optimal (thoughts?)
> >
> > Changes v8 -> v9
> > - Kconfig: fs/fuse/Kconfig:CONFIG_FUSE_FAMFS_DAX now depends on the
> > new CONFIG_DEV_DAX_FSDEV (from drivers/dax/Kconfig) rather than
> > just CONFIG_DEV_DAX and CONFIG_FS_DAX. (CONFIG_FUSE_FAMFS_DAX
> > depends on those...)
> >
> > Changes v7 -> v8
> > - Moved to inline __free declaration in fuse_get_fmap() and
> > famfs_fuse_meta_alloc(), famfs_teardown()
> > - Adopted FIELD_PREP() macro rather than manual bitfield manipulation
> > - Minor doc edits
> > - I dropped adding magic numbers to include/uapi/linux/magic.h. That
> > can be done later if appropriate
> >
> > Changes v6 -> v7
> > - Fixed a regression in famfs_interleave_fileofs_to_daxofs() that
> > was reported by Intel's kernel test robot
> > - Added a check in __fsdev_dax_direct_access() for negative return
> > from pgoff_to_phys(), which would indicate an out-of-range offset
> > - Fixed a bug in __famfs_meta_free(), where not all interleaved
> > extents were freed
> > - Added chunksize alignment checks in famfs_fuse_meta_alloc() and
> > famfs_interleave_fileofs_to_daxofs() as interleaved chunks must
> > be PTE or PMD aligned
> > - Simplified famfs_file_init_dax() a bit
> > - Re-ran CM's kernel code review prompts on the entire series and
> > fixed several minor issues
> >
> > Changes v4 -> v5 -> v6
> > - None. Re-sending due to technical difficulties
> >
> > Changes v3 [9] -> v4
> > - The patch "dax: prevent driver unbind while filesystem holds device"
> > has been dropped. Dan Williams indicated that the favored behavior is
> > for a file system to stop working if an underlying driver is unbound,
> > rather than preventing the unbind.
> > - The patch "famfs_fuse: Famfs mount opt: -o shadow=<shadowpath>" has
> > been dropped. Found a way for the famfs user space to do without the
> > -o opt (via getxattr).
> > - Squashed the fs/fuse/Kconfig patch into the first subsequent patch
> > that needed the change
> > ("famfs_fuse: Basic fuse kernel ABI enablement for famfs")
> > - Many review comments addressed.
> > - Addressed minor kerneldoc infractions reported by test robot.
> >
> > Changes v2 [7] -> v3
> > - Dax: Completely new fsdev driver (drivers/dax/fsdev.c) replaces the
> > dev_dax_iomap modifications to bus.c/device.c. Devdax devices can now
> > be switched among 'devdax', 'famfs' and 'system-ram' modes via daxctl
> > or sysfs.
> > - Dax: fsdev uses MEMORY_DEVICE_FS_DAX type and leaves folios at order-0
> > (no vmemmap_shift), allowing fs-dax to manage folio lifecycles
> > dynamically like pmem does.
> > - Dax: The "poisoned page" problem is properly fixed via
> > fsdev_clear_folio_state(), which clears stale mapping/compound state
> > when fsdev binds. The temporary WARN_ON_ONCE workaround in fs/dax.c
> > has been removed.
> > - Dax: Added dax_set_ops() so fsdev can set dax_operations at bind time
> > (and clear them on unbind), since the dax_device is created before we
> > know which driver will bind.
> > - Dax: Added custom bind/unbind sysfs handlers; unbind return -EBUSY if a
> > filesystem holds the device, preventing unbind while famfs is mounted.
> > - Fuse: Famfs mounts now require that the fuse server/daemon has
> > CAP_SYS_RAWIO because they expose raw memory devices.
> > - Fuse: Added DAX address_space_operations with noop_dirty_folio since
> > famfs is memory-backed with no writeback required.
> > - Rebased to latest kernels, fully compatible with Alistair Popple
> > et. al's recent dax refactoring.
> > - Ran this series through Chris Mason's code review AI prompts to check
> > for issues - several subtle problems found and fixed.
> > - Dropped RFC status - this version is intended to be mergeable.
> >
> > Changes v1 [8] -> v2:
> >
> > - The GET_FMAP message/response has been moved from LOOKUP to OPEN, as
> > was the pretty much unanimous consensus.
> > - Made the response payload to GET_FMAP variable sized (patch 12)
> > - Dodgy kerneldoc comments cleaned up or removed.
> > - Fixed memory leak of fc->shadow in patch 11 (thanks Joanne)
> > - Dropped many pr_debug and pr_notice calls
> >
> >
> > References
> >
> > [0] - https://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git/
> > [1] - https://famfs.org (famfs user space)
> > [2] - https://lore.kernel.org/linux-cxl/cover.1708709155.git.john@groves.net/
> > [3] - https://lore.kernel.org/linux-cxl/cover.1714409084.git.john@groves.net/
> > [4] - https://lwn.net/Articles/983105/ (lsfmm 2024)
> > [5] - https://lwn.net/Articles/1020170/ (lsfmm 2025)
> > [6] - https://lore.kernel.org/linux-cxl/cover.8068ad144a7eea4a813670301f4d2a86a8e68ec4.1740713401.git-series.apopple@nvidia.com/
> > [7] - https://lore.kernel.org/linux-fsdevel/20250703185032.46568-1-john@groves.net/ (famfs fuse v2)
> > [8] - https://lore.kernel.org/linux-fsdevel/20250421013346.32530-1-john@groves.net/ (famfs fuse v1)
> > [9] - https://lore.kernel.org/linux-fsdevel/20260107153244.64703-1-john@groves.net/T/#mb2c868801be16eca82dab239a1d201628534aea7 (famfs fuse v3)
> >
> >
> > John Groves (10):
> > famfs_fuse: Update macro s/FUSE_IS_DAX/FUSE_IS_VIRTIO_DAX/
> > famfs_fuse: Basic fuse kernel ABI enablement for famfs
> > famfs_fuse: Plumb the GET_FMAP message/response
> > famfs_fuse: Create files with famfs fmaps
> > famfs_fuse: GET_DAXDEV message and daxdev_table
> > famfs_fuse: Plumb dax iomap and fuse read/write/mmap
> > famfs_fuse: Add holder_operations for dax notify_failure()
> > famfs_fuse: Add DAX address_space_operations with noop_dirty_folio
> > famfs_fuse: Add famfs fmap metadata documentation
> > famfs_fuse: Add documentation
> >
> > Documentation/filesystems/famfs.rst | 142 ++++
> > Documentation/filesystems/index.rst | 1 +
> > MAINTAINERS | 10 +
> > fs/fuse/Kconfig | 13 +
> > fs/fuse/Makefile | 1 +
> > fs/fuse/dir.c | 2 +-
> > fs/fuse/famfs.c | 1180 +++++++++++++++++++++++++++
> > fs/fuse/famfs_kfmap.h | 167 ++++
> > fs/fuse/file.c | 45 +-
> > fs/fuse/fuse_i.h | 116 ++-
> > fs/fuse/inode.c | 35 +-
> > fs/fuse/iomode.c | 2 +-
> > fs/namei.c | 1 +
> > include/uapi/linux/fuse.h | 88 ++
> > 14 files changed, 1790 insertions(+), 13 deletions(-)
> > create mode 100644 Documentation/filesystems/famfs.rst
> > create mode 100644 fs/fuse/famfs.c
> > create mode 100644 fs/fuse/famfs_kfmap.h
> >
> >
> > base-commit: 2ae624d5a555d47a735fb3f4d850402859a4db77
> > --
> > 2.53.0
> >
>
> Hi John,
>
> I’m curious to hear your thoughts on whether you think it makes sense
> for the famfs-specific logic in this series to be moved to a bpf
> program that goes through a generic fuse iomap dax layer.
>
> Based on [1], this gives feature-parity with the famfs logic in this
> series. In my opinion, having famfs go through a generic fuse iomap
> dax layer makes the fuse kernel code more extensible for future
> servers that will also want to use dax iomap, and keeps the fuse code
> cleaner by not having famfs-specific logic hardcoded in and having to
> introduce new fuse uapis for something famfs-specific. In my
> understanding of it, fuse is meant to be generic and it feels like
> adding server-specific logic goes against that design philosophy and
> sets a precedent for other servers wanting similar special-casing in
> the future. I'd like to explore whether the bpf and generic fuse iomap
> dax layer approach can preserve that philosophy while still giving
> famfs the flexibility it needs.
>
> I think moving the famfs logic to bpf benefits famfs as well:
> - Instead of needing to issue a FUSE_GET_FMAP request after a file is
> opened, the server can directly populate the metadata map from
> userspace with the mapping info when it processes the FUSE_OPEN
> request, which gets rid of the roundtrip cost
> - The server can dynamically update the metadata / bpf maps during
> runtime from userspace if any mapping info needs to change
> - Future code changes / updates for famfs are all server-side and can
> be deployed immediately instead of needing to go through the upstream
> kernel mailing list process
> - Famfs updates / new releases can ship independently of kernel releases
>
> I'd appreciate the chance to discuss tradeoffs or if you'd rather
> discuss this at the fuse BoF at lsf, that sounds great too.
>
> Thanks,
> Joanne
>

Hi Joanne,

I'm definitely up for discussing it, and talking before LSFMM would be
good because then I'd have some time to think about before we discuss
at LSFMM.

I have not had a chance to really study this, in part since I've never even
written a "hello world" BPF program.

I'll ping off-list about times to talk.

However...

I would object vehemently to this sort of re-write prior to going upstream,
as would users and vendors who need famfs now that the memory products it
enables have started to ship.

This work started over 3 years ago, initial patches over 2 years ago,
community decision that it should go into fuse 2 years ago, first fuse
patches a year ago.

This implementation is pretty much exactly in line with expectation-setting
starting two years ago. Famfs is a complicated orchestration between dax,
fuse, ndctl (for daxctl), libfuse and the extensive famfs user space. Famfs
has a fairly small kernel footprint, but its user space is much larger.
This could set it back a year if we re-write now.

Two things are true at once: I think this is a serious idea worth
considering, and I think it's too late to make this sort of change before
going upstream. Products need this enablement, and quite a long process has
run in order to make it available in a timely fashion (which means soon
now). I hope we can avoid making the perfect the enemy of the good.

I believe the risk of merging famfs soon is quite low, because famfs will
not affect anybody who doesn't use it. I hope we can run this discussion and
analysis in parallel with merging the current implementation of famfs soon.

Thank you,
John