[GIT PULL] bcachefs fixes for 6.16-rc3

[GIT PULL] bcachefs fixes for 6.16-rc3

[Kent Overstreet]

Entirely too many patches, but mostly check/repair, and related.

6.16 looks to be shaping up well, knock on wood.

The following changes since commit e04c78d86a9699d136910cfc0bdcf01087e3267e:

  Linux 6.16-rc2 (2025-06-15 13:49:41 -0700)

are available in the Git repository at:

  git://evilpiepirate.org/bcachefs.git tags/bcachefs-2025-06-19

for you to fetch changes up to b2e2bed119809a5ca384241e0631f04c6142ae08:

  bcachefs: Add missing key type checks to check_snapshot_exists() (2025-06-19 14:37:04 -0400)

----------------------------------------------------------------
bcachefs fixes for 6.16-rc3

- Lots of small check/repair fixes, primarily in subvol loop and
  directory structure loop (when involving snapshots).

- Fix a few 6.16 regressions: rare UAF in the foreground allocator path
  when taking a transaction restart from the transaction bump allocator,
  and some small fallout from the change to log the error being
  corrected in the journal when repairing errors, also some fallout from
  the btree node read error logging improvements.

  (Alan, Bharadwaj)

- New option: journal_rewind

  This lets the entire filesystem be reset to an earlier point in time.

  Note that this is only a disaster recovery tool, and right now there
  are major caveats to using it (discards should be disabled, in
  particular), but it successfully restored the filesystem of one of the
  users who was bit by the subvolume deletion bug and didn't have
  backups. I'll likely be making some changes to the discard path in the
  future to make this a reliable recovery tool.

- Some new btree iterator tracepoints, for tracking down some
  livelock-ish behaviour we've been seeing in the main data write path.

----------------------------------------------------------------
Alan Huang (6):
      bcachefs: Don't allocate new memory when mempool is exhausted
      bcachefs: Fix alloc_req use after free
      bcachefs: Add missing EBUG_ON
      bcachefs: Delay calculation of trans->journal_u64s
      bcachefs: Move bset size check before csum check
      bcachefs: Fix pool->alloc NULL pointer dereference

Bharadwaj Raju (1):
      bcachefs: don't return fsck_fix for unfixable node errors in __btree_err

Kent Overstreet (33):
      bcachefs: trace_extent_trim_atomic
      bcachefs: btree iter tracepoints
      bcachefs: Fix bch2_journal_keys_peek_prev_min()
      bcachefs: btree_iter: fix updates, journal overlay
      bcachefs: better __bch2_snapshot_is_ancestor() assert
      bcachefs: pass last_seq into fs_journal_start()
      bcachefs: Fix "now allowing incompatible features" message
      bcachefs: Fix snapshot_key_missing_inode_snapshot repair
      bcachefs: fsck: fix add_inode()
      bcachefs: fsck: fix extent past end of inode repair
      bcachefs: opts.journal_rewind
      bcachefs: Kill unused tracepoints
      bcachefs: mark more errors autofix
      bcachefs: fsck: Improve check_key_has_inode()
      bcachefs: Call bch2_fs_init_rw() early if we'll be going rw
      bcachefs: Fix __bch2_inum_to_path() when crossing subvol boundaries
      bcachefs: fsck: Print path when we find a subvol loop
      bcachefs: fsck: Fix remove_backpointer() for subvol roots
      bcachefs: fsck: Fix reattach_inode() for subvol roots
      bcachefs: fsck: check_directory_structure runs in reverse order
      bcachefs: fsck: additional diagnostics for reattach_inode()
      bcachefs: fsck: check_subdir_count logs path
      bcachefs: fsck: Fix check_path_loop() + snapshots
      bcachefs: Fix bch2_read_bio_to_text()
      bcachefs: Fix restart handling in btree_node_scrub_work()
      bcachefs: fsck: Fix check_directory_structure when no check_dirents
      bcachefs: fsck: fix unhandled restart in topology repair
      bcachefs: fsck: Fix oops in key_visible_in_snapshot()
      bcachefs: fix spurious error in read_btree_roots()
      bcachefs: Fix missing newlines before ero
      bcachefs: Fix *__bch2_trans_subbuf_alloc() error path
      bcachefs: Don't log fsck err in the journal if doing repair elsewhere
      bcachefs: Add missing key type checks to check_snapshot_exists()

 fs/bcachefs/alloc_background.c         |  13 +-
 fs/bcachefs/bcachefs.h                 |   3 +-
 fs/bcachefs/btree_gc.c                 |   8 +-
 fs/bcachefs/btree_io.c                 |  74 ++++----
 fs/bcachefs/btree_iter.c               | 173 ++++++++++++------
 fs/bcachefs/btree_journal_iter.c       |  64 ++++---
 fs/bcachefs/btree_journal_iter_types.h |   5 +-
 fs/bcachefs/btree_trans_commit.c       |  18 +-
 fs/bcachefs/btree_types.h              |   1 +
 fs/bcachefs/btree_update.c             |  16 +-
 fs/bcachefs/btree_update_interior.c    |  11 +-
 fs/bcachefs/btree_update_interior.h    |   3 +
 fs/bcachefs/btree_write_buffer.c       |   3 +
 fs/bcachefs/chardev.c                  |  29 ++-
 fs/bcachefs/data_update.c              |   1 +
 fs/bcachefs/errcode.h                  |   5 -
 fs/bcachefs/error.c                    |   4 +-
 fs/bcachefs/extent_update.c            |  13 +-
 fs/bcachefs/fsck.c                     | 317 +++++++++++++++++++++++----------
 fs/bcachefs/inode.h                    |   5 +
 fs/bcachefs/io_read.c                  |   7 +-
 fs/bcachefs/journal.c                  |  18 +-
 fs/bcachefs/journal.h                  |   2 +-
 fs/bcachefs/journal_io.c               |  26 ++-
 fs/bcachefs/namei.c                    |  30 +++-
 fs/bcachefs/opts.h                     |   5 +
 fs/bcachefs/recovery.c                 |  24 ++-
 fs/bcachefs/recovery_passes.c          |   6 +-
 fs/bcachefs/recovery_passes.h          |   9 +
 fs/bcachefs/sb-errors_format.h         |  17 +-
 fs/bcachefs/snapshot.c                 |  14 +-
 fs/bcachefs/super.c                    |  13 +-
 fs/bcachefs/super.h                    |   1 +
 fs/bcachefs/trace.h                    | 125 +++----------
 34 files changed, 657 insertions(+), 406 deletions(-)

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Linus Torvalds]

On Thu, 19 Jun 2025 at 16:06, Kent Overstreet <kent.overstreet@linux.dev> wrote:
>
> - New option: journal_rewind
>
>   This lets the entire filesystem be reset to an earlier point in time.
>
>   Note that this is only a disaster recovery tool, and right now there
>   are major caveats to using it (discards should be disabled, in
>   particular), but it successfully restored the filesystem of one of the
>   users who was bit by the subvolume deletion bug and didn't have
>   backups. I'll likely be making some changes to the discard path in the
>   future to make this a reliable recovery tool.

You seem to have forgotten what the point of the merge window was again.

We don't start adding new features just because you found other bugs.

I remain steadfastly convinced that anybody who uses bcachefs is
expecting it to be experimental. They had better.

Make the -rc fixes be pure fixes.

                Linus

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Kent Overstreet]

On Thu, Jun 19, 2025 at 05:51:17PM -0700, Linus Torvalds wrote:
> On Thu, 19 Jun 2025 at 16:06, Kent Overstreet <kent.overstreet@linux.dev> wrote:
> >
> > - New option: journal_rewind
> >
> >   This lets the entire filesystem be reset to an earlier point in time.
> >
> >   Note that this is only a disaster recovery tool, and right now there
> >   are major caveats to using it (discards should be disabled, in
> >   particular), but it successfully restored the filesystem of one of the
> >   users who was bit by the subvolume deletion bug and didn't have
> >   backups. I'll likely be making some changes to the discard path in the
> >   future to make this a reliable recovery tool.
> 
> You seem to have forgotten what the point of the merge window was again.

The goal is to get users _code that works_, is it not?

> We don't start adding new features just because you found other bugs.
> 
> I remain steadfastly convinced that anybody who uses bcachefs is
> expecting it to be experimental. They had better.

Honestly, most of the people using bcachefs from what I've seen just
want something that works.

There are a _lot_ of people who've been burned by btrfs. I've even been
seeing more and more people in recent discussions talking about
unrecoverable filesystems with XFS (!).

That last one has been a surprise to me (and I don't think it's anything
to do with the quality of the code), but it honestly should serve as a
wakeup call as to how much is falling through the cracks and how badly
we've been failing.

There are still a lot of people who don't want to move off ext4... and I
can't really blame them.

If you go looking, you won't find those stories about bcachefs - except
from me, when I'm telling people what to watch out for.

And that's because of a lot of hard work, and because I'm dead set on
not repeating past mistakes; I actively hunt down bug reports and I
frequently tell people - "I don't care if you think it's a hardware
issue or pebcak, it's the filesystem's job to not lose data; get me the
info I need and I'll get it sorted and get it working again".

That's the goal here, delivering something that users can trust and rely
on.

I'm not seeing that _you_ get that.

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Jani Partanen]

On 20/06/2025 4.09, Kent Overstreet wrote:
> I'm not seeing that _you_ get that.


How hard it is?

New feature window for 6.16 was 2 weeks ago.

rc<insert number here> is purely for fixing bugs, not adding new 
features and potential new bugs.


But I am not a coder, thats how I see it just by following.

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Kent Overstreet]

On Fri, Jun 20, 2025 at 04:25:58AM +0300, Jani Partanen wrote:
> 
> On 20/06/2025 4.09, Kent Overstreet wrote:
> > I'm not seeing that _you_ get that.
> 
> 
> How hard it is?
> 
> New feature window for 6.16 was 2 weeks ago.
> 
> rc<insert number here> is purely for fixing bugs, not adding new features
> and potential new bugs.

That's an easy rule for the rest of the kernel, where all your mistakes
are erased at a reboot. Filesystems don't have that luxury.

In the past, I've had to rush entire new on disk format features in
response to issues I saw starting to arise - I think more than once, but
the btree bitmap in the member info section was the big one that sticks
in my mind; that one was very hectic, but 100% proved its worth.

Thankfully, we're well past that. This time, we're just talking about a
~70 line patch that just picks overwrites instead of updates from the
journal and sorts them in reverse order.

So your next question might be - why not leave that in a branch for the
users that need it until the next merge window?

For a lot of users, compiling a kernel from some random git repository
is a lot to ask. I spend a lot of time doing what amounts to support;
that's just how it is these days. But rc kernels are packaged by most
kernels, and we absolutely do not want to wait an additional 3 months
for it to show up in a release kernel -

For something that might be the difference between losing a filesystem
and getting it back.

There's also a couple patches for tracepoints and introspection
improvements; I don't know if Linus was referring to those. But those
are important too.

I think at least as much about "how do I make this codebase easy to
debug; how do I make it _practical_ to support and QA when it's running
on random user machines in the wild" as I do about the debugging itself.
That's at least as important as the debugging; making it maintainable.

Partly that's about maintaining a quick feedback cycle between myself
and the users reporting issues; that builds trust, brings people into
the community, turns into opportunities to teach them more about testing
and QA and bug reporting.

I also never cease to be amazed how often I add some bit of logging or
improve a tracepoint or some introspection - and then a week later I'm
working on something else and it's exactly the thing I need.

IOW - it's not just about fixing the bugs, it's about how we fix the
bugs.

Tools to repair, tools to debug, it's all just tools, all the way
down...

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Martin Steigerwald]

Hi Kent, hi,

Kent Overstreet - 20.06.25, 03:51:45 CEST:
> On Fri, Jun 20, 2025 at 04:25:58AM +0300, Jani Partanen wrote:
> > On 20/06/2025 4.09, Kent Overstreet wrote:
> > > I'm not seeing that _you_ get that.
> > 
> > How hard it is?
> > 
> > New feature window for 6.16 was 2 weeks ago.
> > 
> > rc<insert number here> is purely for fixing bugs, not adding new
> > features and potential new bugs.
> 
> That's an easy rule for the rest of the kernel, where all your mistakes
> are erased at a reboot. Filesystems don't have that luxury.
> 
> In the past, I've had to rush entire new on disk format features in
> response to issues I saw starting to arise - I think more than once, but
> the btree bitmap in the member info section was the big one that sticks
> in my mind; that one was very hectic, but 100% proved its worth.

Kent, from what I gathered, you'd like to change some window rules – at 
least for filesystems or new-in-kernel filesystems.

And from what I observed: You try to do so by just not adhering to at 
least one of those rules at least once in a while. Maybe for a good 
reason, but still, you just do not adhere to the rule of only fixes. Or at 
the very least, you define a fix to be something that is (way) more than a 
fix for other kernel developers.

It may make sense to change some merge window rules or it may not, I don't 
know. However… from what I observe:

Just unilaterally changing a rule or redefining a word in that rule may 
not be a sustainable and working (!) approach to go about it. Just from 
observing the communication pattern here I conclude that.

I'd rather recommend to bring this up the next opportunity you can discuss 
it with fellow kernel developers, ideally while meeting face to face. 
Cause let's face it: The Linux kernel is a team effort.

If you stick to your approach about merge window rules and many other 
kernel developers including Linus stick to their rules this will go in 
circles.

Indefinitely so.

Not sure whether that is a good use of your time. But your call really.

So how about adhering to the current rules for now and bringing up the 
topic in a meeting you will have at one point in the future?

Just my two cents from observing the repeated (!) communication pattern 
here. Cause it is not at all the first time the discussion arrives exactly 
at this point.

> For a lot of users, compiling a kernel from some random git repository
> is a lot to ask. I spend a lot of time doing what amounts to support;
> that's just how it is these days. But rc kernels are packaged by most
> kernels, and we absolutely do not want to wait an additional 3 months
> for it to show up in a release kernel -

Those users should probably not use BCacheFS right now already to begin 
with but wait for it to be marked as stable?

It reminds my about Debian Unstable users wanting the newest and greatest 
and then complaining that at times some things are not stable. Which is 
the very definition of what can happen in Debian Unstable.

I meanwhile use BCacheFS. On Devuan. But for now that means I have to 
compile BCacheFS tools myself, and better also the kernel to have the 
latest and probably greatest during the hard freeze of Debian.

And I use it for data that is backed up or that I can afford to loose.


And about what you wrote in your previous mail:

Kent Overstreet - 20.06.25, 03:09:07 CEST:
> There are a _lot_ of people who've been burned by btrfs. I've even been
> seeing more and more people in recent discussions talking about
> unrecoverable filesystems with XFS (!).

And I have seen a lot of threads on XFS over the years where XFS 
developers went great lengths to help users recover their data. I have 
seen those threads also on the BTRFS mailing list. Those users had no 
support contract, they did not pay anything for that free service either.

So I am not sure it is wise or even just accurate to implicitly imply that 
other than BCacheFS filesystem developers do not care about user data. 
From what I have seen I conclude: They do!

But maybe that is not even the point: Other filesystems are maintained by 
other people and they do what they think is best. If you like to see a 
change in some merge window rules for what you try to achieve with 
BCacheFS you can independently from what other filesystem developers do 
ask for a slot to talk in one of the next face to face kernel developer 
meetings.

Best,
-- 
Martin

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Martin Steigerwald]

Martin Steigerwald - 20.06.25, 09:12:21 CEST:
> If you stick to your approach about merge window rules and many other
> kernel developers including Linus stick to their rules this will go in
> circles.
> 
> Indefinitely so.

Also it has even been quite predictable.

The moment I saw your merge request I would have been very surprised if 
Linus would just have pulled it.

-- 
Martin

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Kent Overstreet]

On Fri, Jun 20, 2025 at 09:12:21AM +0200, Martin Steigerwald wrote:
> Hi Kent, hi,
> 
> Kent Overstreet - 20.06.25, 03:51:45 CEST:
> > On Fri, Jun 20, 2025 at 04:25:58AM +0300, Jani Partanen wrote:
> > > On 20/06/2025 4.09, Kent Overstreet wrote:
> > > > I'm not seeing that _you_ get that.
> > > 
> > > How hard it is?
> > > 
> > > New feature window for 6.16 was 2 weeks ago.
> > > 
> > > rc<insert number here> is purely for fixing bugs, not adding new
> > > features and potential new bugs.
> > 
> > That's an easy rule for the rest of the kernel, where all your mistakes
> > are erased at a reboot. Filesystems don't have that luxury.
> > 
> > In the past, I've had to rush entire new on disk format features in
> > response to issues I saw starting to arise - I think more than once, but
> > the btree bitmap in the member info section was the big one that sticks
> > in my mind; that one was very hectic, but 100% proved its worth.
> 
> Kent, from what I gathered, you'd like to change some window rules – at 
> least for filesystems or new-in-kernel filesystems.

There is a time and a place for rules, and there is a time and a place
for using your head and exercising some common sense and judgement.

I'm the one who's responsible for making sure that bcachefs users have a
working filesystem. That means reading and responding to every bug
report and keeping track of what's working and what's not in
fs/bcachefs/. Not you, and not Linus.

There's no need for any of this micromanaging, which is what this has
turned into. All it's been doing is generating conflict and drama.

> > For a lot of users, compiling a kernel from some random git repository
> > is a lot to ask. I spend a lot of time doing what amounts to support;
> > that's just how it is these days. But rc kernels are packaged by most
> > kernels, and we absolutely do not want to wait an additional 3 months
> > for it to show up in a release kernel -
> 
> Those users should probably not use BCacheFS right now already to begin 
> with but wait for it to be marked as stable?

I've often told people that they should probably wait before switching,
particularly when they need features that aren't ready yet (erasure
coding), and things are not yet so trouble free that I would recommend
normal users switch.

Besides the most basic "will it eat your data", there's a lot of other
things to consider, like "is it providing stable backports yet" (I'm
explicitly not) or "is there wonky behaviour or missing APIs still to be
worked out" (yes, there definitely is, and I'm still triaging so there
will be awhile).

None of that changes my most basic responsibility.

> Kent Overstreet - 20.06.25, 03:09:07 CEST:
> > There are a _lot_ of people who've been burned by btrfs. I've even been
> > seeing more and more people in recent discussions talking about
> > unrecoverable filesystems with XFS (!).
> 
> And I have seen a lot of threads on XFS over the years where XFS 
> developers went great lengths to help users recover their data. I have 
> seen those threads also on the BTRFS mailing list. Those users had no 
> support contract, they did not pay anything for that free service either.
> 
> So I am not sure it is wise or even just accurate to implicitly imply that 
> other than BCacheFS filesystem developers do not care about user data. 
> From what I have seen I conclude: They do!

No, I'm not saying that they don't care about user data.

I know most of the other filesystem developers, particularly the XFS
ones; I'm not trying to disparage their work.

But track records do matter, and bcachefs has a good one, and I intend
to keep it that way.

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Theodore Ts'o]

On Fri, Jun 20, 2025 at 04:14:24AM -0400, Kent Overstreet wrote:
> 
> There is a time and a place for rules, and there is a time and a place
> for using your head and exercising some common sense and judgement.
> 
> I'm the one who's responsible for making sure that bcachefs users have a
> working filesystem. That means reading and responding to every bug
> report and keeping track of what's working and what's not in
> fs/bcachefs/. Not you, and not Linus.

Kent, the risk as always of adding features after the merge window is
that you might introduce regressions.  This is especially true if you
are making changes to relatively sensitive portions of any file system
--- such as journalling.

The rules around the merge window is something which the vast majority
of the kernel developers have agreeded upon for well over a decade.
And it is Linus's responsibility to *enforce* those rules.

If, as you say, bcachefs is experimental, and no sane person should be
trusting their data on it, then perhaps this shouldn't be urgent.  On
the flip side, perhaps if you are claiming that people should be using
it for critical data, then perhaps your care for user's data safety
should be.... revisted.

Cheers,

						- Ted

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Christoph Heinrich]

> On Fri, Jun 20, 2025 at 04:14:24AM -0400, Kent Overstreet wrote:
>> 
>> There is a time and a place for rules, and there is a time and a place
>> for using your head and exercising some common sense and judgement.
>> 
>> I'm the one who's responsible for making sure that bcachefs users have a
>> working filesystem. That means reading and responding to every bug
>> report and keeping track of what's working and what's not in
>> fs/bcachefs/. Not you, and not Linus.
> 
> Kent, the risk as always of adding features after the merge window is
> that you might introduce regressions.  This is especially true if you
> are making changes to relatively sensitive portions of any file system
> --- such as journalling.
> > The rules around the merge window is something which the vast majority
> of the kernel developers have agreeded upon for well over a decade.
> And it is Linus's responsibility to *enforce* those rules.
While bcachefs is marked as experimental, perhaps the rules should be
somewhat relaxed. After all those rules were made in the context of
"stable" parts of the kernel and thus might not be the best strategy
for parts explicitly marked as experimental.

After following bcachefs development for a while now, I'd be totally
fine with him pushing new features up to rc5 or so.
Of course such a relaxed rule set should be agreed upon _before_
sending something.

> If, as you say, bcachefs is experimental, and no sane person should be
> trusting their data on it, then perhaps this shouldn't be urgent.  On
> the flip side, perhaps if you are claiming that people should be using
> it for critical data, then perhaps your care for user's data safety
> should be.... revisted.

Considering bcachefs's track record of not loosing data, it shouldn't
be surprising that some people start trusting it, despite being marked
experimental. With that one fs being saved by journal rewind, I guess
we're back to nobody ever loosing any data to bcachefs, which is quite
impressive.

FWIW I'm running two multi device filesystems with bcachefs right
now. They are purely for bulk storage so far, so I'm not the best
advocate for daily use stability. However I've been lurking in IRC ever
since Kent saved my ass after I screwed up one of those filesystems
(100% user error, wouldn't have blamed it on the fs for loosing it),
and after watching him work his magic for other users, I'd trust
bcachefs more to not permanently eat my data then other filesystems.

- Christoph

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Kent Overstreet]

On Fri, Jun 20, 2025 at 08:43:46AM -0400, Theodore Ts'o wrote:
> On Fri, Jun 20, 2025 at 04:14:24AM -0400, Kent Overstreet wrote:
> > 
> > There is a time and a place for rules, and there is a time and a place
> > for using your head and exercising some common sense and judgement.
> > 
> > I'm the one who's responsible for making sure that bcachefs users have a
> > working filesystem. That means reading and responding to every bug
> > report and keeping track of what's working and what's not in
> > fs/bcachefs/. Not you, and not Linus.
> 
> Kent, the risk as always of adding features after the merge window is
> that you might introduce regressions.  This is especially true if you
> are making changes to relatively sensitive portions of any file system
> --- such as journalling.

Ted, you know better than most how long I've been shipping storage code.

You also know just as well as I do that when to include a patch is very
much a judgement call, so I'm quite curious why you and Linus think
yourselves better able to make that judgement call than I.

> The rules around the merge window is something which the vast majority
> of the kernel developers have agreeded upon for well over a decade.
> And it is Linus's responsibility to *enforce* those rules.

And those rules are _always_ flexible when there's a reason for them to
be, and here we do have a very good reason for including that code -
making sure people don't lose filesystems.

> If, as you say, bcachefs is experimental, and no sane person should be
> trusting their data on it, then perhaps this shouldn't be urgent. 

I don't see how that follows.

Firstly, we're not defining what the "experimental" label, and since I
am the one who put that label on bcachefs, I'll define that now.

Primarily:

- Bugs are still outstanding, so users should not expect the same level
  of polish as other filesystems - but that does _not_ mean that it's
  not supported to the same degree as other filesystems. Bug reports are
  triaged, and anything that effects the filesystem as a whole will be
  fixed with great immediacy: "polish" issues or bugs that only affect a
  specific application, or a new feature, might have to wait.

- Stable backports are only happening for the most critical bugs, due to
  the volume of bugfixing. Users are expected to stick close to the
  latest release if they want a fix, and often run rc kernels if there's
  something specific they need. This has been working well, because the
  rate of _regressions_ has been quite low.

And that's basically it.

This isn't btrfs, which took off the experimental label, as I recall,
when the on disk format was frozen. bcachefs's on disk format hasn't
been making incompatible changes in years, and as of 6.15 the last of
the required on disk format upgrades is done.

I'll be taking off the experimental label when I believe bcachefs ready
for widespread deployment by the distributions, not before.

Again, that _does not mean_ that I'm not supporting it like any other
production filesystem is supported. I think user reports should have
made that clear by now.

The talk about "no sane person should trust their data to it" is
entirely baseless - I honestly don't know where you and Linus are
getting that from. It's certainly not based on user reports or anything
I've seen.

bcachefs has been used in production for years, by a small but growing
number of users, and at this point 100+tb filesystems in production are
commonplace. I have one user I work with closely who can rattle off a
long list of workloads he's got on bcachefs, and the only reason he
isn't moving more stuff off of btrfs is send/recv - it'll be a long time
before we have that.

> On the flip side, perhaps if you are claiming that people should be
> using it for critical data, then perhaps your care for user's data
> safety should be.... revisted.

This seems entirely backwards. Whatever I'm claiming is besides the
point: bcachefs has users, and I actively support them.

Now, onwards:

It's _entirely_ unclear why you and Linus are bringing up the
experimental label of bcachefs.

If bcachefs were seen as a widely deployed, non-experimental filesystem
(despite the fact that for all purposes it is), I have to think - I have
to hope - that you'd be taking a more pragmatic approach here.

But I think I and myself would assume that having an experimental label
would mean _looser_ rules, so that development can move more quickly.

So it's hard to fathom what's going on here.

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Kent Overstreet]

On Fri, Jun 20, 2025 at 07:35:04PM -0400, Kent Overstreet wrote:
> So it's hard to fathom what's going on here.

I also need to add that this kind of drama, and these responses to pull
requests - second guessing technical decisions, outright trash talk -
have done an incredible amount of damage, and I think it's time to make
you guys aware of that since it's directly relevant to the story of this
pull request.

I've put a lot of work into building a real community around bcachefs,
because that's critical to making it the rock solid, dependable
filesystem, for eeryone, that I intend it to be: building a community
where people feel free to share observations, bug reports, and where
people trust that those will be acted on responsibly.

That all gets set back whenever drama like this happens. Last time, the
casefolding bugfix pull request, ignited a whole vi. vs. emacs holy war.
Every time this happens, the calm, thoughtful people pull back, and all
I hear from are the angry, dramatic voices.

More than that, I lost a hire because of Linus's constant,
every-other-pull-request "I'm thinking about removing bcachefs from the
kernel". It turns out, smart, thoughtful engineers with stable jobs
become very hesitant about leaving those jobs when that happens, and
that's all their co-workers are seeing.

And the first thing that got cancelled/put aside because of that - work
that was in progress, and hasn't been completed - was tooling for
comprehensive programatic fault injection for on disk format errors.
IOW - the tooling and test coverage that would have caught the subvolume
deletion bug.

That's a really painful loss right now.

Even despite that, bcachefs development has been going incredibly
smoothly, and it's shaping up fast. Like I mentioned before, 100+ TB
filesystems are commonplace, users are commenting every release on how
much smoother is getting. We are, I hope, only a year or less from being
able to take the experimental label off, based on the decline in
critical bug reports I'm seeing.

The only area that gives me cause for concern - and it causes a _lot_ of
concern - is upstream.

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Jérôme Poulin]

As a bcachefs user who has been following this discussion, I'd like to
share my perspective on the current state of the filesystem and the
path forward.

I'm currently using this filesystem for a backup staging server so it
is easy for me to make sure data isn't getting lost and can verify
checksums at the application levels from time to time. The solution
uses snapshots extensively as well as replication, reflinks and
background compression.

I really like this filesystem for multiple reasons, it fills the gap
for missing features of traditional filesystems, it allows integrating
cache devices almost seamlessly, it allows having metadata on local
devices while still pushing to slow HDD, SMR or network devices
without having to setup something like Ceph or a stack like Btrfs,
mdadm and nbd/iSCSI.  I've seen all the features appear one by one on
Bcachefs and it is growing fast.

I migrated from Btrfs after an incident with a RAID controller losing
its cache that caused the filesystem to be unmountable and
unrepairable.  Btrfs restore was able to recover *most* of the files
on that server except a couple subvolumes which had to be recreated by
the backup system.  And again, since this is a staging area for
backups, I don't need 100% uptime or a guarantee that my files won't
be lost so I felt pretty confident in using Bcachefs to speed up
operations there.

Bcachefs was able to triple the speed of the backup system by having
metadata stored in NVMe + passively caching all writes to NVMe.  The
last part of the backup is now blazing fast since everything is in
NVMe.

At this point in time, I do believe Bcachefs has solid foundations, as
of now, the only data corruption that lost me some files were related
to a snapshot deletion bug for a feature that was not yet published to
mainline.

It hasn't been without its downsides, many times I had to take the
filesystem for offline repair and Kent was always able to figure out
the root cause of issues causing the FS not to mount read-write and
issue a patch for the FS and for fsck.  We found many weird bugs
together, ARM specific bugs, reflink causing corruption, resize not
allocating buckets, many races and lock ups, upgrade not finishing
correctly, corruption from weird interactions, data not staying cached
when there's no promote_target.  All of this was fixed without much
more damage than the last operations being lost and most were fixed
really quickly from cat'ing a couple diagnostic files, using perf or
worst case metadata image.

The filesystem is very resilient at being rebooted anywhere, anytime.
It went through many random resets during any of..  fsck repairs, fsck
rebuilding the btree from scratch, upgrades, in the middle of snapshot
operations, while replaying journal.  It just always recovers at
places I wouldn't expect to be able to hit the power switch. Worst
case, it mounted read-only and needed fsck but could always be mounted
read-only.

It also went through losing 6 devices and the write-back cache (that
defective controller, again).  Fsck could repair it with minimal loss
related to recent data. A lot of scary messages in fsck, but it
finished and I could run scrub+rereplicate to finish it off (which
fixed a couple more files).

Where things get a bit more touchy is when combining all those
features together;  operations tend to be a bit "racy" between each
other and tend to lock up when there's multiple features running/being
used in parallel.  I think this is where we get to the "move fast
break things" part of the filesystem.  The foundation is solid, read,
write, inode creations/suppression, bucket management, all basic posix
operations, checksums, scrub, device addition. Many of the
bcachefs-specific operations are stable, being able to set compression
and replication level and data target per folder is awesome stuff and
works well.

From my experience, what is less polished are; snapshots and snapshot
operations, reflink, nocow, multiprocess heavy workloads, those seem
to be where the "experimental" part of the filesystem goes into the
spotlight.  I've been running rotating snapshots on many machines, it
works well until it doesn't and I need to reboot or fsck. Reflink
before 6.14 seemed a bit hacky and can result in errors. Nocow tends
to lock up but isn't really useful with bcachefs anyway. Maybe
casefolding which might not be fully tested yet. Those are the true
experimental features and aren't really labelled as such.

We can always say "yes, this is fixed in master, this is fixed in
6.XX-rc4" but it is still experimental and tends to be what causes the
most pain right now.  I think this needs to be communicated more
clearly. If the filesystem goes off experimental, I think a subset of
features should be gated by filesystem options to reduce the need for
big and urgent rc patches.

The problem is...  when the experimental label is removed, it needs to
be very clear that users aren't expected to be running the latest rc
and master branch.  All the features marked as stable should have
settled enough that there won't be 6 users requiring a developer to
mount their filesystem read-write or recover files from a catastrophic
race condition.

This is where communication needs to be clear, bcachefs website,
tools, options; should all clearly label features that might require
someone to ask a developer's help or to run the latest release
candidate or a debug version of the kernel.

Bcachefs has very nice unit and integration testing with ktest, but it
isn't enough to represent real-world usage yet and that's why I think
some features should still be marked just as experimental as erasure
coding.  Bcachefs filesystem where I do not use reflink, snapshot or
anything wild, only multiple devices with foreground/promote_target,
replication, compression, never experience weird issues or lockups for
many kernel versions now.  Mind you, I'm not using bcachefs on any
rootfs yet, only specific use-case and patterns that can be
documented.

I care about the future and success of bcachefs to be my go-to
filesystem for anything that requires CoW features, robust repair
tools, caching and flexible RAID-like features.  I just don't want it
to get kicked out of the kernel because of huge changesets to fix bugs
on features that shouldn't be used by someone who expects the
filesystem to behave.

It might slow down development a bit to mark some features as
experimental, but it'll remove the pressure of having to push so many
bug fixes that are critical to make sure users don't experience
critical failures or blindly try to repair their FS using fsck -y
without reporting issues. It reduces the experimental surface to a
subset of features, it also makes the user aware of what they should
do if enabled, eg.: contact dev before fsck -y, run a recent kernel at
all time, etc.

One more thing that I think is missing, many patches submitted, even
if it doesn't show up, should have a Reported-By and Tested-By tag to
help show how many people in the community are working and helping
make Bcachefs great, it would also make people on the ML aware that
patches aren't just thrown in there; it usually has been a reported
bug from a community member which had to test the resulting patch.

Anyway, that message is bigger than I expected and I hope brings some
light on how I perceive bcachefs from a user standpoint.

Have a great weekend!

On Fri, Jun 20, 2025 at 8:15 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Fri, Jun 20, 2025 at 07:35:04PM -0400, Kent Overstreet wrote:
> > So it's hard to fathom what's going on here.
>
> I also need to add that this kind of drama, and these responses to pull
> requests - second guessing technical decisions, outright trash talk -
> have done an incredible amount of damage, and I think it's time to make
> you guys aware of that since it's directly relevant to the story of this
> pull request.
>
> I've put a lot of work into building a real community around bcachefs,
> because that's critical to making it the rock solid, dependable
> filesystem, for eeryone, that I intend it to be: building a community
> where people feel free to share observations, bug reports, and where
> people trust that those will be acted on responsibly.
>
> That all gets set back whenever drama like this happens. Last time, the
> casefolding bugfix pull request, ignited a whole vi. vs. emacs holy war.
> Every time this happens, the calm, thoughtful people pull back, and all
> I hear from are the angry, dramatic voices.
>
> More than that, I lost a hire because of Linus's constant,
> every-other-pull-request "I'm thinking about removing bcachefs from the
> kernel". It turns out, smart, thoughtful engineers with stable jobs
> become very hesitant about leaving those jobs when that happens, and
> that's all their co-workers are seeing.
>
> And the first thing that got cancelled/put aside because of that - work
> that was in progress, and hasn't been completed - was tooling for
> comprehensive programatic fault injection for on disk format errors.
> IOW - the tooling and test coverage that would have caught the subvolume
> deletion bug.
>
> That's a really painful loss right now.
>
> Even despite that, bcachefs development has been going incredibly
> smoothly, and it's shaping up fast. Like I mentioned before, 100+ TB
> filesystems are commonplace, users are commenting every release on how
> much smoother is getting. We are, I hope, only a year or less from being
> able to take the experimental label off, based on the decline in
> critical bug reports I'm seeing.
>
> The only area that gives me cause for concern - and it causes a _lot_ of
> concern - is upstream.
>

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Jérôme Poulin]

Apologies for bottom quoting, Gmail issues, just clarifying this was
meant to be a standalone message,

On Sat, Jun 21, 2025 at 5:07 PM Jérôme Poulin <jeromepoulin@gmail.com> wrote:
>
> As a bcachefs user who has been following this discussion, I'd like to
> share my perspective on the current state of the filesystem and the
> path forward.

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[Kent Overstreet]

Thanks for the report.

I'd like to add, bcachefs is the _lot_ of people investing a ton of time
QAing this thing. There are a lot of users like Jérôme that I've worked
with for extended periods tracking down all sorts of crazy stuff, and I
thank them for their patience and all their help.

It's been a true community effort.

On Sat, Jun 21, 2025 at 05:07:51PM -0400, Jérôme Poulin wrote:
> The filesystem is very resilient at being rebooted anywhere, anytime.
> It went through many random resets during any of..  fsck repairs, fsck
> rebuilding the btree from scratch, upgrades, in the middle of snapshot
> operations, while replaying journal.  It just always recovers at
> places I wouldn't expect to be able to hit the power switch. Worst
> case, it mounted read-only and needed fsck but could always be mounted
> read-only.

That's the dream :)

I don't think the filesystem should ever fail in a way that leads to
data loss, and I think this is a more than achievable goal.

> Where things get a bit more touchy is when combining all those
> features together;  operations tend to be a bit "racy" between each
> other and tend to lock up when there's multiple features running/being
> used in parallel.  I think this is where we get to the "move fast
> break things" part of the filesystem.  The foundation is solid, read,
> write, inode creations/suppression, bucket management, all basic posix
> operations, checksums, scrub, device addition. Many of the
> bcachefs-specific operations are stable, being able to set compression
> and replication level and data target per folder is awesome stuff and
> works well.

It's not "move fast and break things", we haven't had a problem with
regressions that I've seen.

It's just a project with massive scope, and it takes awhile to find all
the corner cases and make sure there's no pathalogical behaviour in any
scenario.

> From my experience, what is less polished are; snapshots and snapshot
> operations, reflink, nocow, multiprocess heavy workloads, those seem
> to be where the "experimental" part of the filesystem goes into the
> spotlight.

This mostly fits with what I've been seeing; exception being that I
haven't seen any major issues with reflink in ages (you mentioned a
reflink corruption earlier, are you sure that was reflink?).

And rebalance (background data movement) has taken awhile to make
polished, and we're still not done - I think as of 6.16 we've got all
the outright bugs I know of fixed, but there's still behaviour that's
less than ideal (charitably) - if you ask it to move more data to a
target than fits, it'll spin (no longer wasting IO, though). That one
needs some real work to fix properly - another auxiliary index of
"pending" extents, extents that rebalance would like to move but can't
until something changes.

Re: multiprocess workloads, those livelock-ish behaviour have been the
most problematic to track down - but we made some recent progress on
understanding where they're coming from, and the new btree iterator
tracepoints should help.

The new error_throw tracepoint is also already proving useful for
tracking down wonky behaviour (just not the one you're talking about).

> I've been running rotating snapshots on many machines, it
> works well until it doesn't and I need to reboot or fsck. Reflink
> before 6.14 seemed a bit hacky and can result in errors. Nocow tends
> to lock up but isn't really useful with bcachefs anyway. Maybe
> casefolding which might not be fully tested yet. Those are the true
> experimental features and aren't really labelled as such.

Casefolding still has a strange rename bug. Some of the recent self
healing work was partly to make it easier to track down - we now will
notice that something went wrong and pop a fsck error on the first 'ls'
of an improperly renamed dirent.

> We can always say "yes, this is fixed in master, this is fixed in
> 6.XX-rc4" but it is still experimental and tends to be what causes the
> most pain right now.  I think this needs to be communicated more
> clearly. If the filesystem goes off experimental, I think a subset of
> features should be gated by filesystem options to reduce the need for
> big and urgent rc patches.

Yeah, this is coming up more as the userbase grows.

For the moment doing more backports is infeasible due to sheer volume,
but I expect this to be changing soon - 6.17 is when I expect to start
doing more backports.

> The problem is...  when the experimental label is removed, it needs to
> be very clear that users aren't expected to be running the latest rc
> and master branch.  All the features marked as stable should have
> settled enough that there won't be 6 users requiring a developer to
> mount their filesystem read-write or recover files from a catastrophic
> race condition.

Correct. Stable backports will start happening _before_ the experimental
is lifted

> This is where communication needs to be clear, bcachefs website,
> tools, options; should all clearly label features that might require
> someone to ask a developer's help or to run the latest release
> candidate or a debug version of the kernel.

Everything just needs to be solid before the experimental label is
lifted. I don't want users to have to check a website to know what's
safe to use.

From the trends I've been seeing - bug report frequency, severity, and
where in the code we're located - I think we should be in good shape to
lift the experimental label in a year.

Of the ~2 years that bcachefs has been in, a good chunk of that has been
scalabilitiy and feature work, as we hit wider deployment and started
finding out more about what was needed in actual usage.

That's largely done, we don't need any more major development before
lifting experimental (there are some smaller things, like subvolume
walking APIs, but nothing like the disk accounting rewrite or all the
backpointers scalability work). That's why rate of patches has been
going up, now we're down to just fixing user bugs.

> Bcachefs has very nice unit and integration testing with ktest, but it
> isn't enough to represent real-world usage yet and that's why I think
> some features should still be marked just as experimental as erasure
> coding.  Bcachefs filesystem where I do not use reflink, snapshot or
> anything wild, only multiple devices with foreground/promote_target,
> replication, compression, never experience weird issues or lockups for
> many kernel versions now.  Mind you, I'm not using bcachefs on any
> rootfs yet, only specific use-case and patterns that can be
> documented.

Better automated testing would _always_ be nice :)

But realistically there's always going to be only so much we can do with
automated testing, there's always going to be things that only show up
in the wild when users start coming up with crazy usage scenarios and
doing all sorts of wild and wonderful things to break it. There's no
substitute for real world battle testing, and lots of it.

So the big thing on my mind right now isn't improving the automated
testing (besides, the server bill is already $1.5k/month and I run those
machines hard) - it's improving all the debugging tools to make sure we
can quickly and easy debug anything that might happen in the wild.

Later on, I hope we do add more automated testing - especially fault
inejection. _Lots_ of fault injection.

> One more thing that I think is missing, many patches submitted, even
> if it doesn't show up, should have a Reported-By and Tested-By tag to
> help show how many people in the community are working and helping
> make Bcachefs great, it would also make people on the ML aware that
> patches aren't just thrown in there; it usually has been a reported
> bug from a community member which had to test the resulting patch.

Yeah, for sure. You guys have been keeping be busy lately, so when I'm
debugging sunup to sundown for multiple people (a lot of days,
literalyl!) things do get lost :) But I will do more of that.

Re: [GIT PULL] bcachefs fixes for 6.16-rc3

[pr-tracker-bot]

The pull request you sent on Thu, 19 Jun 2025 19:06:01 -0400:

> git://evilpiepirate.org/bcachefs.git tags/bcachefs-2025-06-19

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/bb378314ceee4d181e26bfe180deca852ae80c5c

Thank you!

-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html