journal replay at each boot problematic?

journal replay at each boot problematic?

[Clemens Eisserer]

Hi,

I am using bcache in writeback-mode for a very slow (~100kb/s write
speed) backing device.

At each boot I see messages like the following:
> bcache: bch_journal_replay() journal replay done, 2297 keys in 86 entries, seq 71136

Are these messages indicating an unclean shutdown / some kind of
rollback operation - or is it part of "normal" bcache initialization?

Thank you in advance, Clemens

Re: journal replay at each boot problematic?

[Kai Krakow]

Am Mon, 27 Mar 2017 13:33:57 +0200
schrieb Clemens Eisserer <linuxhippy@gmail.com>:

> Hi,
> 
> I am using bcache in writeback-mode for a very slow (~100kb/s write
> speed) backing device.
> 
> At each boot I see messages like the following:
> > bcache: bch_journal_replay() journal replay done, 2297 keys in 86
> > entries, seq 71136  
> 
> Are these messages indicating an unclean shutdown / some kind of
> rollback operation - or is it part of "normal" bcache initialization?

As far as the docs tell, bcache is designed to shut down unclean. So
this is expected behavior in write-back mode.

To cleanly detach a bcache, you would need to first put it in
write-through or write-around mode and wait for write-back to finish.
In turn, this mode should also avoid these messages at boot.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: journal replay at each boot problematic?

[Clemens Eisserer]

Hi Kai,

> As far as the docs tell, bcache is designed to shut down unclean. So
> this is expected behavior in write-back mode.

Yes, I also read it - but since I got a few btrfs fs errors when
running btrfs check (mostly space-cache related) I became a bit
alarmed.
As all csums are ok, so I don't think the backing/caching devices are
corrupt in any way.

> To cleanly detach a bcache, you would need to first put it in
> write-through or write-around mode and wait for write-back to finish.
> In turn, this mode should also avoid these messages at boot.

This is exactly what I do now before each shutdown: switch to
writethrough and wait in a bash-loop until state="clean" - sorrounded
by two calls to sync.
However, I still get the journal replay messages at bootup:

[    2.590615] bcache: bch_journal_replay() journal replay done, 827
keys in 37 entries, seq 145717


Br, Clemens

Re: journal replay at each boot problematic?

[Kai Krakow]

Am Wed, 29 Mar 2017 14:27:11 +0200
schrieb Clemens Eisserer <linuxhippy@gmail.com>:

> Hi Kai,
> 
> > As far as the docs tell, bcache is designed to shut down unclean. So
> > this is expected behavior in write-back mode.  
> 
> Yes, I also read it - but since I got a few btrfs fs errors when
> running btrfs check (mostly space-cache related) I became a bit
> alarmed.
> As all csums are ok, so I don't think the backing/caching devices are
> corrupt in any way.

This is probably related to btrfs not closed properly or some bug in an
older kernel. Space cache problems can usually be safely ignored as
long as the kernel detects them. But maybe ask about this in the btrfs
list. I'm pretty sure that is related to your shutdown procedure and
not to bcache. I don't see such messages here without special
workarounds in the shutdown procedure, however, I'm using initramfs
created by dracut. It properly unmounts btrfs.

> > To cleanly detach a bcache, you would need to first put it in
> > write-through or write-around mode and wait for write-back to
> > finish. In turn, this mode should also avoid these messages at
> > boot.  
> 
> This is exactly what I do now before each shutdown: switch to
> writethrough and wait in a bash-loop until state="clean" - sorrounded
> by two calls to sync.

I don't think this is necessary. Just ensure that the FS is umounted or
at least RO mounted, sync, and ensure that all storage buffers are
flushed (SCSI, SATA, whatever). Btrfs may lazily still write data even
in the RO mounted mode. This may be your problem. Bcache may even
prevent further evil here because of its always-unclean writeback
behavior.

> However, I still get the journal replay messages at bootup:
> 
> [    2.590615] bcache: bch_journal_replay() journal replay done, 827
> keys in 37 entries, seq 145717

Do you properly close the FS before shutting down, i.e. unmount (and
not only remount RO), and flush the storage layer?

-- 
Regards,
Kai

Replies to list-only preferred.

Re: journal replay at each boot problematic?

[Kai Krakow]

Am Wed, 29 Mar 2017 19:52:57 +0200
schrieb Kai Krakow <hurikhan77@gmail.com>:

> Am Wed, 29 Mar 2017 14:27:11 +0200
> schrieb Clemens Eisserer <linuxhippy@gmail.com>:
> 
> > Hi Kai,
> >   
> > > As far as the docs tell, bcache is designed to shut down unclean.
> > > So this is expected behavior in write-back mode.    
> > 
> > Yes, I also read it - but since I got a few btrfs fs errors when
> > running btrfs check (mostly space-cache related) I became a bit
> > alarmed.
> > As all csums are ok, so I don't think the backing/caching devices
> > are corrupt in any way.  
> 
> This is probably related to btrfs not closed properly or some bug in
> an older kernel. Space cache problems can usually be safely ignored as
> long as the kernel detects them. But maybe ask about this in the btrfs
> list. I'm pretty sure that is related to your shutdown procedure and
> not to bcache. I don't see such messages here without special
> workarounds in the shutdown procedure, however, I'm using initramfs
> created by dracut. It properly unmounts btrfs.
> 
> > > To cleanly detach a bcache, you would need to first put it in
> > > write-through or write-around mode and wait for write-back to
> > > finish. In turn, this mode should also avoid these messages at
> > > boot.    
> > 
> > This is exactly what I do now before each shutdown: switch to
> > writethrough and wait in a bash-loop until state="clean" -
> > sorrounded by two calls to sync.  
> 
> I don't think this is necessary. Just ensure that the FS is umounted
> or at least RO mounted, sync, and ensure that all storage buffers are
> flushed (SCSI, SATA, whatever). Btrfs may lazily still write data even
> in the RO mounted mode. This may be your problem. Bcache may even
> prevent further evil here because of its always-unclean writeback
> behavior.

Addendum: Maybe try "btrfs sync"...

> > However, I still get the journal replay messages at bootup:
> > 
> > [    2.590615] bcache: bch_journal_replay() journal replay done, 827
> > keys in 37 entries, seq 145717  
> 
> Do you properly close the FS before shutting down, i.e. unmount (and
> not only remount RO), and flush the storage layer?
> 



-- 
Regards,
Kai

Replies to list-only preferred.