Re: [GIT PULL] bcachefs fixes for 6.16-rc4

Re: [GIT PULL] bcachefs fixes for 6.16-rc4

[Kyle Sanderson]

On 6/27/2025 12:07 PM, Kyle Sanderson wrote:
> On 6/26/2025 8:21 PM, Linus Torvalds wrote:
>> On Thu, 26 Jun 2025 at 19:23, Kent Overstreet 
>> <kent.overstreet@linux.dev> wrote:
>>>
>>> per the maintainer thread discussion and precedent in xfs and btrfs
>>> for repair code in RCs, journal_rewind is again included
>>
>> I have pulled this, but also as per that discussion, I think we'll be
>> parting ways in the 6.17 merge window.
>>
>> You made it very clear that I can't even question any bug-fixes and I
>> should just pull anything and everything.
>>
>> Honestly, at that point, I don't really feel comfortable being
>> involved at all, and the only thing we both seemed to really
>> fundamentally agree on in that discussion was "we're done".
>>
>>                Linus
> 
> Linus,
> 
> The pushback on rewind makes sense, it wasn’t fully integrated and was 
> fsck code written to fix the problems with the retail 6.15 release - 
> this looks like it slipped through Kents CI and there were indeed 
> multiple people hit by it (myself included).
> 
> Quoting someone back to themselves is not cool, however I believe it 
> highlights what has gone on here which is why I am breaking my own rule:
> 
> "One of the things I liked about the Rust side of the kernel was that 
> there was one maintainer who was clearly much younger than most of the 
> maintainers and that was the Rust maintainer.
> 
> We can clearly see that certain areas in the kernel bring in more young 
> people.
> 
> At the Maintainer Summit, we had this clear division between the 
> filesystem people, who were very careful and very staid, and cared 
> deeply about their code being 100% correct - because if you have a bug 
> in a filesystem, the data on your disk may be gone - so these people 
> take themselves and their code very seriously.
> 
> And then you have the driver people who are a bit more 'okay', 
> especially the GPU folks, 'where anything goes'.
> You notice that on the driver side it’s much easier to find young 
> people, and that is traditionally how we’ve grown a lot of maintainers.
> " (1)
> 
> Kent is moving like the older days of rapid development - fast and 
> driven - and this style clashes with the mature stable filesystem 
> culture that demands extreme caution today. Almost every single patch 
> has been in response to reported issues, the primary issue here is 
> that’s on IRC where his younger users are (not so young, anymore - it is 
> not tiktok), and not on lkml. The pace of development has kept up, and 
> the "new feature" part of it like changing out the entire hash table in 
> rc6 seems to have stopped. This is still experimental, and he's moving 
> that way now with care and continuing to improve his testing coverage 
> with each bug.
> 
> Kent has deep technical experience here, much earlier in the 
> interview(1) regarding the 6.7 merge window this filesystem has been in 
> the works for a decade. Maintainership means adapting to kernel process 
> as much as code quality, that may be closer to the issue here.
> 
> If direct pulls aren’t working, maybe a co-maintainer or routing changes 
> through a senior fs maintainer can help. If you're open to it, maybe 
> that is even you.
> 
> Dropping bcachefs now would be a monumental step backward from the 
> filesystems we have today. Enterprises simply do not use them for true 
> storage at scale which is why vendors have largely taken over this 
> space. The question is how to balance rigor with supporting new 
> maintainers in the ecosystem. Everything Kent has written around 
> supporting users is true, and publicly visible, if only to the 260 users 
> on irc, and however many more are on matrix. There are plenty more that 
> are offline, and while this is experimental there are a number of public 
> sector agencies testing this now (I have seen reference to a number of 
> emergency service providers, which isn’t great, but for whatever reason 
> they are doing that).
> 
> (1) https://youtu.be/OvuEYtkOH88?t=1044
> 
> Kyle.

Re-sending as this thread seems to have typo'd lkml (removing the bad 
entry).

Kyle.

Re: [GIT PULL] bcachefs fixes for 6.16-rc4

[Kent Overstreet]

On Fri, Jun 27, 2025 at 12:16:09PM -0700, Kyle Sanderson wrote:
> On 6/27/2025 12:07 PM, Kyle Sanderson wrote:
> > On 6/26/2025 8:21 PM, Linus Torvalds wrote:
> > > On Thu, 26 Jun 2025 at 19:23, Kent Overstreet
> > > <kent.overstreet@linux.dev> wrote:
> > > > 
> > > > per the maintainer thread discussion and precedent in xfs and btrfs
> > > > for repair code in RCs, journal_rewind is again included
> > > 
> > > I have pulled this, but also as per that discussion, I think we'll be
> > > parting ways in the 6.17 merge window.
> > > 
> > > You made it very clear that I can't even question any bug-fixes and I
> > > should just pull anything and everything.
> > > 
> > > Honestly, at that point, I don't really feel comfortable being
> > > involved at all, and the only thing we both seemed to really
> > > fundamentally agree on in that discussion was "we're done".
> > > 
> > >                Linus
> > 
> > Linus,
> > 
> > The pushback on rewind makes sense, it wasn’t fully integrated and was
> > fsck code written to fix the problems with the retail 6.15 release -
> > this looks like it slipped through Kents CI and there were indeed
> > multiple people hit by it (myself included).
> > 
> > Quoting someone back to themselves is not cool, however I believe it
> > highlights what has gone on here which is why I am breaking my own rule:
> > 
> > "One of the things I liked about the Rust side of the kernel was that
> > there was one maintainer who was clearly much younger than most of the
> > maintainers and that was the Rust maintainer.
> > 
> > We can clearly see that certain areas in the kernel bring in more young
> > people.
> > 
> > At the Maintainer Summit, we had this clear division between the
> > filesystem people, who were very careful and very staid, and cared
> > deeply about their code being 100% correct - because if you have a bug
> > in a filesystem, the data on your disk may be gone - so these people
> > take themselves and their code very seriously.
> > 
> > And then you have the driver people who are a bit more 'okay',
> > especially the GPU folks, 'where anything goes'.
> > You notice that on the driver side it’s much easier to find young
> > people, and that is traditionally how we’ve grown a lot of maintainers.
> > " (1)
> > 
> > Kent is moving like the older days of rapid development - fast and
> > driven - and this style clashes with the mature stable filesystem
> > culture that demands extreme caution today. Almost every single patch
> > has been in response to reported issues, the primary issue here is
> > that’s on IRC where his younger users are (not so young, anymore - it is
> > not tiktok), and not on lkml. The pace of development has kept up, and
> > the "new feature" part of it like changing out the entire hash table in
> > rc6 seems to have stopped. This is still experimental, and he's moving
> > that way now with care and continuing to improve his testing coverage
> > with each bug.
> > 
> > Kent has deep technical experience here, much earlier in the
> > interview(1) regarding the 6.7 merge window this filesystem has been in
> > the works for a decade. Maintainership means adapting to kernel process
> > as much as code quality, that may be closer to the issue here.
> > 
> > If direct pulls aren’t working, maybe a co-maintainer or routing changes
> > through a senior fs maintainer can help. If you're open to it, maybe
> > that is even you.
> > 
> > Dropping bcachefs now would be a monumental step backward from the
> > filesystems we have today. Enterprises simply do not use them for true
> > storage at scale which is why vendors have largely taken over this
> > space. The question is how to balance rigor with supporting new
> > maintainers in the ecosystem. Everything Kent has written around
> > supporting users is true, and publicly visible, if only to the 260 users
> > on irc, and however many more are on matrix. There are plenty more that
> > are offline, and while this is experimental there are a number of public
> > sector agencies testing this now (I have seen reference to a number of
> > emergency service providers, which isn’t great, but for whatever reason
> > they are doing that).
> > 
> > (1) https://youtu.be/OvuEYtkOH88?t=1044
> > 
> > Kyle.
> 
> Re-sending as this thread seems to have typo'd lkml (removing the bad
> entry).

Thanks.

Also, I think I should add, in case my words in the private conversation
were misinterpreted:

I don't think bcachefs should be dropped from the kernel, I think it
would be better for this to be worked out.

I firstly want to reassure people that: if bcachefs has to be shipped as
a DKMS module, that will not kill the project. It will be a giant hassle
(especially if distributions have to scramble), but life will continue.
I remain committed as ever to getting this done - one way or the other.

And I think it is safe to say that going that route would be the better
option for the sanity of myself and Linus, but it wouldn't be the better
option for the users or the rest of the development community.

With that, I am going to take a breather.

Re: [GIT PULL] bcachefs fixes for 6.16-rc4

[Kent Overstreet]

On Wed, Jul 02, 2025 at 10:41:34AM -0700, Carl E. Thompson wrote:
> Kent, at this point in bcachefs' development you want complete control
> over your development processes and timetable that you simply can't
> get in the mainline kernel. It's in your own best interest for you to
> develop out-of-tree for now.

Carl, all I'm doing is stating up front what it's going to take to get
this done right.

I'm not particularly pushing one way or the other for bcachefs to stay
in; there are pros and cons either way. It'll be disruptive for it to be
out, but if the alternative is disrupting process too much and driving
Linus and I completely completely nuts, that's ok.

Everyone please be patient. This is a 10+ year process, no one thing is
make or break.

Re: [GIT PULL] bcachefs fixes for 6.16-rc4

[Malte Schröder]

On 02.07.25 19:53, Kent Overstreet wrote:
> On Wed, Jul 02, 2025 at 10:41:34AM -0700, Carl E. Thompson wrote:
>> Kent, at this point in bcachefs' development you want complete control
>> over your development processes and timetable that you simply can't
>> get in the mainline kernel. It's in your own best interest for you to
>> develop out-of-tree for now.
> Carl, all I'm doing is stating up front what it's going to take to get
> this done right.
>
> I'm not particularly pushing one way or the other for bcachefs to stay
> in; there are pros and cons either way. It'll be disruptive for it to be
> out, but if the alternative is disrupting process too much and driving
> Linus and I completely completely nuts, that's ok.
>
> Everyone please be patient. This is a 10+ year process, no one thing is
> make or break.
>
So as a user usually hanging out on IRC and running Kent's trees:

I think most of those people actually testing bcachefs are either
running bcachefs-master, -rc or some outdated distro kernel. From my
perspective I'd think it would be good enough to push for-upstream
during the merge window and then only provide further patches if there
where regressions or some really bad bug appears that actually eats data
(like the one that bit me). If it's "just" stability fixes, well, if
people running a distro kernel hit those bugs they'll need to build a
-rc kernel anyways to get fixes, those could just build bcachefs-master.
When running Linus' tree I am aware and accept that I am not running the
absolutely latest code.

I've had some pretty bad experience with amdgpu requiring out of tree
patches to get my system running free of glitches, which took months to
get into upstream. It's annoying, but I accept that.

I'd rather have a slightly outdated bcachefs in kernel than not at all.
It is good to have a distro kernel I can fall back to if I mess up my
own kernel building ;)


my 0.02€

/Malte