Re: Freezing (was: Re: fstests generic/441 -- occasional bcachefs failure)

[Dave Chinner <dchinner@redhat.com>]

On Thu, Feb 02, 2023 at 12:09:20PM -0500, Kent Overstreet wrote:
> On Thu, Feb 02, 2023 at 10:50:23AM -0500, Brian Foster wrote:
> > I don't have a public repo atm but I've posted the patch if you have
> > somewhere to land it for CI testing..? It survived my regression tests
> > so far, FWIW. (I also had posted that random cleanup patch a bit ago if
> > you hadn't noticed..).
> 
> Must have missed it, sorry. I can host a git repo for you on my server,
> or github works fine - I generally prefer git repo links, git am is
> always a bit of a hassle.
> 
> > Is there a reporting dashboard or something available for the test
> > infrastruture for bcachefs?
> 
> https://evilpiepirate.org/~testdashboard/ci
> 
> I've got a small server farm that watches git branches and runs the
> entire test suite on every commit starting from the recent - once you've
> got a git branch up I'll add yours to the list it watches.
> 
> You'll probably want to get acquainted with ktest, it's what both the CI
> uses for running tests, and what we use for local development:
> 
> https://evilpiepirate.org/git/ktest.git
> 
> > > Freeze definitely needs to happen. It's been _ages_ since I was looking
> > > at it so I couldn't say offhand where we'd need to start, but if you're
> > > interested I'd be happy to look at what it'd take.
> > > 
> > 
> > Yeah, that would be interesting. Thanks.
> 
> Maybe we could get Dave to give us a brief rundown of freezing? It's
> been ages since I was thinking about that and it's all fallen out of my
> brain, but Dave was the one who was able to explain it to me before :)

What do you need to know? The vast majority of the freeze
infrastructure is generic and the filesystem doesn't need to do
anything special. The only thing it needs to implement is
->freeze_fs to essentially quiesce the filesystem - by this stage
all the data has been written back and all user-driven operations
have either been stalled or drained at either the VFS or transaction
reservation points.

This requires the filesystem transaction start point to call
sb_start_intwrite() in a location the transaction start can block
safely forever, and to call sb_end_intwrite() when the transaction
is complete and being torn down. 

[Note that bcachefs might also require
sb_{start/end}_{write/pagefault} calls in paths that it has custom
handlers for and to protect against ioctl operations triggering
modifications during freezes.]

This allows freeze_super() to set a barrier to prevent new
transactions from starting, and to wait on transactions in flight to
drain. Once all transactions have drained, it will then call
->freeze_fs if it is defined so the filesystem can flush it's
journal and dirty in-memory metadata so that it becomes consistent
on disk without requiring journal recovery to be run.

This basically means that once ->fs_freeze completes, the filesystem
should be in the same state on-disk as if it were unmounted cleanly,
and the fs will not issue any more IO until the filesystem is
thawed. Thaw will call ->unfreeze_fs if defined before unblocking
tasks so that the filesystem can restart things that may be needed
for normal operation that were stopped during the freeze.

It's not all that complex anymore - a few hooks to enable
modification barriers to be placed and running the
writeback part of unmount in ->freeze_fs is the main component
of the work that needs to be done....

Cheers,

Dave.
-- 
Dave Chinner
dchinner@redhat.com