Re: Freezing (was: Re: fstests generic/441 -- occasional bcachefs failure)

[Kent Overstreet <kent.overstreet@linux.dev>]

On Tue, Feb 21, 2023 at 09:19:37AM +1100, Dave Chinner wrote:
> > Wait, so are you saying that XFS does not commit dirty buffers for 
> > sleeping, only on remount_ro?
> 
> Yup. But this is not unique to XFS - every journalling filesystem
> (ext3, ext4, jfs, etc) have exactly the same problem:
> sync_filesystem() only guarantees that the filesystem is consistent
> on disk, not that it is clean in memory.
> 
> And by "consistent on disk", that means all dirty metadata has been
> written -to the journal- so that if a crash occurs immeditately
> afterwards, journal recovery on the next mount will ensure that the
> filesystem is consistent.
> 
> IOWs, after sync_filesystem(), the filesystem is most definitely
> *not idle* and *not clean in memory*, and that's where all the
> issues with suspend end up coming from - it assumes sync() is all
> that is needed to put a filesystem in an idle state....
> 
> > In an ideal case I suppose the laptop ram is 
> > still hot... But sometimes (ahem, far too often) I close my laptop and 
> > forget about it, in which case the battery dies and of course then any 
> > dirty pages are lost.  IMHO sleep should always be crash-safe.
> 
> suspend is generally considered crash safe. The problems with
> suspend stem from inconsistent in-memory vs on-disk filesystem state
> in the suspend image - this causes problems on resume of the
> suspended image, not on the next cold boot of the system.

Ok, so that means we'll want to do things differently.

I don't think we _quite_ have a straightforward mechanism for quiescing
things without completely flushing the btree (which is actually driven
by journal reclaim).

There is bch2_journal_block()/bch2_journal_unblock() which blocks new
journal reservations - I think that can be our starting point.
bch2_journal_block() also waits for all in flight journal entries to be
written, so that's what we want here.

If we want all dirty metadata to be written _and visible_ in the
journal, we need to make sure the last journal entry written is a flush
entry - in general we do this by setting journal_buf->must_flush for the
entry to be written; see bch2_journal_meta() for a simple example.

We'll also need to prevent new btree node writes from being issued, and
wait for in flight ones to be finished. We don't have a mechanism for
that currently, we can probably create something like
bch2_journal_block() for that.

Other IO sources:
 - superblock writes (rare in normal operation, they happen when e.g. we
   start writing data to a new set of devices... we need that in the
   superblock so that mount knows which devices we need to mount).

 - data reads/writes

 - erasure coding writes (parity blocks when creating stripes,
   reconstruct reads).

I expect we'll have to block reads from being submitted as well? Then
we'll want to block data reads/btree node reads/ec reconstruct reads
after blocking all the sources of writes...

(this is starting to feel more like the old freezer stuff.. blech)