Re: [LSF/MM/BPF TOPIC] bcachefs

[Kent Overstreet <kent.overstreet@linux.dev>]

On Tue, Mar 07, 2023 at 07:25:15AM +0000, Matthew Wilcox wrote:
> On Wed, Feb 22, 2023 at 02:46:31PM -0500, Kent Overstreet wrote:
> > I'd like to talk more about where things are at, long term goals, and
> > finally upstreaming this beast.
> 
> We don't have any rules about when we decide to upstream a new filesystem.
> There are at least four filesystems considering inclusion right now;
> bcachefs, SSDFS, composefs and nvfs.  Every new filesystem imposes
> certain costs on other developers (eg those who make sweeping API changes,
> *cough*).  I don't think we've ever articulated a clear set of criteria,
> and maybe we can learn from the recent pain of accepting ntfs3 in the
> upstream kernel.

I've been thinking about what's good for the filesystem.

I've been leery about upstreaming too soon, with unfinished important
features or major design work still to be done. When it's upstream, I'll
have to spend a lot more of my time in a maintainer role, and it's going
to be harder to find time for multi-month projects that require deep
focus; like snapshots, or the allocator rewrite, or backpointers, or
right now erasure coding.

And once it's upstream the pressure is going to be to keep things
stable, to not break things that will affect users. Whereas right now,
I've got a testing community that's smaller, more forgiving of temporary
bugs, and people work with me and give me feedback. That feedback is
important and guides what I work on; it's driven a lot of the
scalability work over the past few years. We've got people running it on
100 TB arrays right now (I want to hear from the first person to test it
on a 1 PB array!) - it took a lot of work to get there.

It's still not _quite_ where I want it to be. Snapshots needs a bit more
work - the deletion path needs more torture testing, and I'm adamant on
not shipping without solid working erasure coding. Right now I'm working
on getting the erasure coding copygc torture test passing; the
fundamentals are there but I've probably got another month of work to to
get it all polished and thoroughly debugged.

But, you gotta ship someday :) and the feedback from users has
increasingly been "yeah, it's been solid and trouble free", so... maybe
it is finally just about time :)