Re: [PATCH V12 12/12] famfs: Add documentation

[John Groves <john@groves.net>]

On 26/08/05 10:38PM, Darrick J. Wong wrote:
> On Mon, Aug 03, 2026 at 02:30:18AM +0000, John Groves wrote:
> > From: John Groves <John@Groves.net>
> > 
> > Add Documentation/filesystems/famfs.rst and update MAINTAINERS
> > 
> > Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
> > Tested-by: Randy Dunlap <rdunlap@infradead.org>
> > Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com>
> > Signed-off-by: John Groves <john@groves.net>
> > ---
> >  Documentation/filesystems/famfs.rst | 142 ++++++++++++++++++++++++++++
> >  Documentation/filesystems/index.rst |   1 +
> >  MAINTAINERS                         |   1 +
> >  3 files changed, 144 insertions(+)
> >  create mode 100644 Documentation/filesystems/famfs.rst
> > 
> > diff --git a/Documentation/filesystems/famfs.rst b/Documentation/filesystems/famfs.rst
> > new file mode 100644
> > index 000000000000..048791bcbcfd
> > --- /dev/null
> > +++ b/Documentation/filesystems/famfs.rst
> > @@ -0,0 +1,142 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +.. _famfs_index:
> > +
> > +==================================================================
> > +famfs: The fabric-attached memory file system
> > +==================================================================
> > +
> > +- Copyright (C) 2024-2026 Micron Technology, Inc.
> > +
> > +Introduction
> > +============
> > +Compute Express Link (CXL) provides a mechanism for disaggregated or
> > +fabric-attached memory (FAM). This creates opportunities for data sharing;
> > +clustered apps that would otherwise have to shard or replicate data can
> > +share one copy in disaggregated memory.
> > +
> > +Famfs, which is not CXL-specific in any way, provides a mechanism for
> > +multiple hosts to concurrently access data in shared memory, by giving it
> > +a file system interface. With famfs, any app that understands files can
> > +access data sets in shared memory. Although famfs supports read and write,
> > +the real point is to support mmap, which provides direct (dax) access to
> > +the memory - either writable or read-only.
> > +
> > +Shared memory can pose complex coherency and synchronization issues, but
> > +there are also simple cases. Two simple and eminently useful patterns that
> > +occur frequently in data analytics and AI are:
> > +
> > +* Serial Sharing - Only one host or process at a time has access to a file
> > +* Read-only Sharing - Multiple hosts or processes share read-only access
> > +  to a file
> > +
> > +The famfs file system is part of the famfs framework; user space
> > +components [1] handle metadata allocation and distribution, and populate
> > +the in-kernel files (via ioctls) so that they map directly to shared
> > +memory.
> 
> Ok, so maybe it is the famfs server that's supposed to go modify the
> directory tree after the metadata log updates.

Right, pretty much. With standalone famfs, there is currently no stateful
server per se (which is a benefit!). There is state in the log, and state in 
the kernel (dirs and files with fmaps). Periodically we replay the log to 
push up any new state into the kernel - at which point we add dirs and files 
if the log has been appended.  Since we don't delete or truncate stuff, 
logplay either adds files/dirs or does nothing.

> 
> > +The famfs framework manages coherency of its own metadata and structures,
> > +but does not attempt to manage coherency for applications.
> > +
> > +Famfs also provides data isolation between files. That is, even though
> > +the host has access to an entire memory "device" (as a devdax device), apps
> > +cannot write to memory for which the file is read-only, and mapping one
> > +file provides isolation from the memory of all other files. This is pretty
> > +basic, but some experimental shared memory usage patterns provide no such
> > +isolation.
> > +
> > +Principles of Operation
> > +=======================
> > +
> > +Famfs is a file system with one or more devdax devices as a first-class
> > +backing device(s). Metadata maintenance and query operations happen
> > +entirely in user space.
> > +
> > +The famfs user space provides each file's map (fmap) and its backing
> > +devdax devices to the kernel through ioctls (FAMFSIOC_MAP_CREATE and
> > +FAMFSIOC_DAXDEV_OPEN), after which read/write/mapping faults are handled
> > +entirely in the kernel with no up-calls for all active files.
> > +
> > +The famfs user space is responsible for maintaining and distributing
> > +consistent metadata. This is currently handled via an append-only
> > +metadata log within the memory, but this is orthogonal to the famfs
> > +kernel code.
> > +
> > +Once instantiated, "the same file" on each host points to the same shared
> > +memory, but in-memory metadata (inodes, etc.) is ephemeral on each host
> > +that has a famfs instance mounted. Use cases are free to allow or not
> > +allow mutations to data on a file-by-file basis.
> > +
> > +When an app accesses a data object in a famfs file, there is no page cache
> > +involvement. The CPU cache is loaded directly from the shared memory. In
> > +some use cases, this is an enormous reduction in read amplification
> > +compared to loading an entire page into the page cache.
> > +
> > +
> > +Famfs is Not a Conventional File System
> > +---------------------------------------
> > +
> > +Famfs files can be accessed by conventional means, but there are
> > +limitations. The famfs kernel module is not involved in the allocation of
> > +backing memory for files at all; the famfs user space creates files and
> > +supplies their fmaps and devdax device info to the kernel via ioctls.
> 
> ...and just to be clear, regular programs *do not* modify the directory
> tree, create files, or delete them?

Right, though I'm hesitant to say it should *never* be allowed. One recent
use case needed to create local-only symlinks within famfs, to make some 
app or benchmark happy...

But any mutability other than by the logplay machinery is severely or
completely restricted.

> 
> > +Famfs differs in some important ways from conventional file systems:
> > +
> > +* Files must be pre-allocated by the famfs framework; allocation is never
> > +  performed on (or after) write.
> > +* Any operation that changes a file's size is considered to put the file
> > +  in an invalid state, disabling access to the data. It may be possible to
> > +  revisit this in the future. (Typically the famfs user space can restore
> > +  files to a valid state by replaying the famfs metadata log.)
> > +
> > +Famfs exists to apply the existing file system abstractions to shared
> > +memory so applications and workflows can more easily adapt to an
> > +environment with disaggregated shared memory.
> > +
> > +Memory Error Handling
> > +=====================
> > +
> > +Possible memory errors include timeouts, poison, and unexpected
> > +reconfiguration of an underlying dax device. In all of these cases, famfs
> > +receives a call from the devdax layer via its
> > +dax_holder_operations->notify_failure() function. If any memory errors have
> > +been detected, access to the affected
> > +daxdev is disabled to avoid further errors or corruption.
> > +
> > +In all known cases, famfs can be unmounted cleanly. In most cases errors
> > +can be cleared by re-initializing the memory - at which point a new famfs
> > +file system can be created.
> 
> Should media failure notifications get passed to the new fserror
> framework (see xfs_verify_media.c)?
> 
> --D

I didn't know about fserror; Initial impression is I'm hesitant to jump in
immediately, but we should consider adding it later.

Famfs currently does not attempt to determine which files are affected by
a memory error - it just shuts down the mount. It's possible we'll attempt
this in the future, but I'm not yet convinced it's needed.

Famfs is not storage, and we advise users that it's a mistake to put your 
only copy of valuable data in famfs. But it's great for stuff like kv-cache 
blocks, which can be regenerated. So errors famfs memory, if you're using
it right, are not lost data in the same sense as with xfs.

Push back if you disagree. Otherwise I've noted it in our internal roadmap
docs...

Thanks!
John

<snip>