Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Dave Chinner <david@fromorbit.com>]

On Tue, Mar 21, 2017 at 01:47:12PM -0600, Chris Murphy wrote:
> On Thu, Mar 16, 2017 at 10:07 PM, Chris Murphy <lists@colorremedies.com> wrote:
> 
> > Yuck. I'll ask on systemd list if there's a way to get more verbose
> > information about this without that. It really ought to be in
> > systemd.log_level=debug anyway.
> 
> I've gotten one response so far, which did not answer the question how
> to get more verbose debug information.
> https://lists.freedesktop.org/archives/systemd-devel/2017-March/038502.html
> 
> The assertion there is that it's virtually certainly an EBUSY exit code.
> 
> Additionally:
> a. Systemd only ever does a remount to read-only with root fs. It
> never umounts it.
> https://github.com/systemd/systemd/blob/master/src/core/umount.c line 413
> 
> b. The three messages suggests that it's retrying to remount ro.
> 
> c. The failure to remount ro is predicted by the plymouth message "It
> is running from the root file system, and thus likely to block
> re-mounting of the root file system to read-only."
> 
> d. All of this happens on ext4, XFS, and Btrfs. But only XFS manifests
> by being unbootable.

Because, in this case, both ext4 and XFS require a successful
remount-ro to guarantee that the metadata grub is relying on is
written to disk. In the case of ext4, you've just been lucky,
probably because it has a faster background journal flush cycle.

....

> https://github.com/systemd/systemd/blob/master/src/core/shutdown.c
> line 213 suggests a sync happens. This sync is after pk offline update
> has finished, so why is this sync not syncing with XFS and ext4? Why
> does the kernel permit a reboot before root fs has been cleanly
> umounted?

sync is /not sufficient/ to force metadata to disk. All that is
required during sync for a journalling filesystem is to ensure that
data is flushed and the journal is committed to disk. If the system
crashes, then log recovery is run and no metadata or data is lost.

Hence, if grub is trying to find the new kernel that was written to
disk then sync()d but not unmounted/remounted before rebooting, the
metadata that grub needs to find the new kernel image is in the
journal, not resting on disk as grub is assuming it will be.

Hence, boot fails because grub/systemd did not correctly
sync/unmount/remount the filesystem before boot. And, no, having
grub replay the log is not the answer - that's a recipe for endless
filesystem corruption problems that filesystem developers will
disown with "grub screwed up your filesystem, go shout at them".

FYI, I've ranted previously over many years about how broken grub's
kernel update and retreival process is fundamentally broken, but
it's never been fixed(*). As a result, I don't use grub on any of my
systems, nor do I recommend that anyone else use it.

(*) The simple fix for grub to freeze/unfreeze the filesystem rather
than/after calling sync() - this does the same thing as remount-ro,
but unlike remount-ro it does not fail if there are writable file
descriptors open.

> Meanwhile, retesting on Btrfs, offline check reports no error after
> the pk offline update reboot; nor when mounting the fs.

of course - it's the nature of btrfs structure that the superblock
is updated to only point at a valid tree. If the superblock is not
updated, then none of the update in progress is even known to
exist...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com