Re: [RFC PATCH v2 00/12] Introduce the famfs shared-memory file system

[Matthew Wilcox <willy@infradead.org>]

On Mon, Apr 29, 2024 at 09:11:52PM -0500, John Groves wrote:
> On 24/04/29 07:32PM, Matthew Wilcox wrote:
> > On Mon, Apr 29, 2024 at 12:04:16PM -0500, John Groves wrote:
> > > This patch set introduces famfs[1] - a special-purpose fs-dax file system
> > > for sharable disaggregated or fabric-attached memory (FAM). Famfs is not
> > > CXL-specific in anyway way.
> > > 
> > > * Famfs creates a simple access method for storing and sharing data in
> > >   sharable memory. The memory is exposed and accessed as memory-mappable
> > >   dax files.
> > > * Famfs supports multiple hosts mounting the same file system from the
> > >   same memory (something existing fs-dax file systems don't do).
> > 
> > Yes, but we do already have two filesystems that support shared storage,
> > and are rather more advanced than famfs -- GFS2 and OCFS2.  What are
> > the pros and cons of improving either of those to support DAX rather
> > than starting again with a new filesystem?
> > 
> 
> Thanks for paying attention to this Willy.

Well, don't mistake this for an endorsement!  I remain convinced that
this is a science project, not a product.  I am hugely sceptical of
disaggregated systems, mostly because I've seen so many fail.  And they
rarely attempt to answer the "janitor tripped over the cable" problem,
the "we need to upgrade the firmware on the switch" problem, or a bunch
of other problems I've outlined in the past on this list.

So I am not supportive of any changes you want to make to the core kernel
to support this kind of adventure.  Play in your own sandbox all you
like, but not one line of code change in the core.  Unless it's something
generally beneficial, of course; you mentioned refactoring DAX and that
might be a good thing for everybody.

> * Famfs is not, not, not a general purpose file system.
> * One can think of famfs as a shared memory allocator where allocations can be
>   accessed as files. For certain data analytics work flows (especially 
>   involving Apache Arrow data frames) this is really powerful. Consumers of
>   data frames commonly use mmap(MAP_SHARED), and can benefit from the memory
>   de-duplication of shared memory and don't need any new abstractions.

... and are OK with the extra latency?

> * Famfs is not really a data storage tool. It's more of a shared-memroy 
>   allocation tool that has the benefit of allocations being accesssible 
>   (and memory-mappable) as files. So a lot of software can automatically use 
>   it.
> * Famfs is oriented to dumping sharable data into files and then allowing a
>   scale-out cluster to share it (often read-only) to access a single copy in
>   shared memory.

Depending on the exact workload, I can see this being more efficient
than replicating the data to each member of the cluster.  In other
workloads, it'll be a loss, of course.

> * I'm no expert on GFS2 or OCFS2, but I've been around memory, file systems 
>   and storage since well before the turn of the century...
> * If you had brought up the existing fs-dax file systems, I would have pointed
>   that they use write-back metadata, which does not reconcile with shared
>   access to media - but these file systems do handle that.
> * The shared media file systems are still oriented to block devices that
>   provide durable storage and page-oriented access. CXL DRAM is a character 

I'd say "block oriented" rather than page oriented, but I agree.

>   dax (devdax) device and does not provide durable storage.
> * fs-dax-style memory mapping for volatile cxl memory requires the 
>   dev_dax_iomap portion of this patch set - or something similar. 
> * A scale-out shared media file system presumably requires some commitment to
>   configure and manage some complexity in a distributed environment; whether
>   that should be mandatory for enablement of shared memory is worthy of
>   discussion.
> * Adding memory to the storage tier for GFS2/OCFS2 would add non-persistent
>   media to the storage tier; whether this makes sense would be a topic that
>   GFS2/OCFS2 developers/architects should get involved in if they're 
>   interested.
> 
> Although disaggregated shared memory is not commercially available yet, famfs 
> is being actively tested by multiple companies for several use cases and 
> patterns with real and simulated shared memory. Demonstrations will start to
> surface in the coming weeks & months.

I guess we'll see.  SGI died for a reason.