Re: [LSF/MM/BPF TOPIC] Design challenges for a new file system that needs to support multiple billions of file

[Ric Wheeler <ricwheeler@gmail.com>]

On 2/2/25 10:39 PM, RIc Wheeler wrote:
>
> I have always been super interested in how much we can push the 
> scalability limits of file systems and for the workloads we need to 
> support, we need to scale up to supporting absolutely ridiculously 
> large numbers of files (a few billion files doesn't meet the need of 
> the largest customers we support).
>
> Zach Brown is leading a new project on ngnfs (FOSDEM talk this year 
> gave a good background on this - 
> https://www.fosdem.org/2025/schedule/speaker/zach_brown/).  We are 
> looking at taking advantage of modern low latency NVME devices and 
> today's networks to implement a distributed file system that provides  
> better concurrency that high object counts need and still have the 
> bandwidth needed to support the backend archival systems we feed.
>
> ngnfs as a topic would go into the coherence design (and code) that 
> underpins the increased concurrency it aims to deliver.
>
> Clear that the project is in early days compared to most of the 
> proposed content, but it can be useful to spend some of the time on 
> new ideas.
>
Just adding that all of this work is GPL'ed and we aspire to getting it 
upstream.

This is planned to be a core part of future shipping products, so we 
intend to fully maintain it going forward.