Re: fstests generic/441 -- occasional bcachefs failure

[Kent Overstreet <kent.overstreet@linux.dev>]

On Thu, Feb 09, 2023 at 07:57:31AM -0500, Brian Foster wrote:
> If you have any other open bugs that you think might be useful to look
> into, feel free to send them along..

https://evilpiepirate.org/~testdashboard/c/f99e51d956a74e1cd78f88648b27b0dac7c13a02/xfstests.generic.388/log.br

This might be a good one to start with: a file got truncated but still
has extents in it. I'd start by running it in a loop locally in ktest,
and seeing how easily it reproduces and what else turns up.

It's starting to look like there might be a btree ordering issue -
there's been a number of unusual bug reports of filesystem
inconsistencies that generally are repaired fine by fsck - but aren't
showing up in our tests either, this is the closest to those bugs that
I've come across in the tests.

truncate calls bch2_fpunch() which deletes extents using
bch2_extent_update(); it failing to delete some extents shouldn't be
possible and if i_sie/i_sectors somehow got out of sync I'd expect it to
be in the other direction; still, there might be some asserts we can
add.

'bcachefs list_journal -a' should be one of the first things you look
at, you can grep through the output to find the most recent transaction
commit(s) that updated that inode. See if anything looks funny - was
this one of the most recent updates when the test stopped, or was it
further back?

If it's a lower level btree inconsistency, that might show up by
verifying what the journal says we have against what's in the btree -
'bcachefs list' command.

If it's a btree ordering issue, there's two main possibilites:
 - after a crash, the btree has updates it shouldn't have, because
   they're newer than the newest update that made it into the journal
 - the btree failed to persist some updates that the journal didn't
   think it needed to replay anymore

Keys have a version number field: we can use it for distinguishing
between these two possibilities. In the transaction commit path, you'll
see code for a debug mode where we initialize it based on the sequence
number of the journal reservation we obtained. There may still be some
other related code for that debug mode; the idea is to check for keys
newer than the newest journal sequence number we replayed after a crash.

We should also check for keys with blacklisted journal sequence numbers.

I think one of those approaches will turn up something :)