boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

Hi,

A user reports a curious OS update bug with Fedora 22, 23, 24, and 25,
where after a kernel update, upon reboot there's only a grub> prompt,
no menu.

I've reproduced this using a qemu-kvm VM, with device SATA and unsafe
caching, pointed to an LVM LV. If I do 'dnf update' to update the
kernel, the problem doesn't happen, if I use gnome-software the
problem always happens.

gnome-software leverages PackageKit for downloading and staging, then
reboots to a systemd offline update target to do the update. So it's a
joint effort. The detailed reproduce steps I've come up with are in
comment 39 of the original bug report:

https://bugzilla.redhat.com/show_bug.cgi?id=1227736#c39

The gist of the analysis in that comment is that the modified grub.cfg
is a 0 length file, and that's why GRUB arrives at a grub> prompt
instead of a menu. If I boot from alternate media and run xfs_repair
-n, I get tons of errors - that log is attached to the bug report and
is here

https://bugzilla.redhat.com/attachment.cgi?id=1263176

If all I do is do a normal mount, kernel reports log replay, no
errors, and then I can boot the previously unbootable system just
fine, and xfs_repair -n reports no errors. So it seems like the volume
is left dirty at reboot time following these offline updates, and of
course GRUB doesn't read the XFS log so it has no idea how to recover
and find grub.cfg or any other affected files.

But with the available information I can't tell why the fs is left
dirty at reboot. Using the same configuration, the problem doesn't
happen with ext4 or Btrfs, but it may be normal behavior and what
really needs to be done is teach GRUB how to do log replay.



-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Darrick J. Wong]

On Wed, Mar 15, 2017 at 12:53:10AM -0600, Chris Murphy wrote:
> Hi,
> 
> A user reports a curious OS update bug with Fedora 22, 23, 24, and 25,
> where after a kernel update, upon reboot there's only a grub> prompt,
> no menu.
> 
> I've reproduced this using a qemu-kvm VM, with device SATA and unsafe
> caching, pointed to an LVM LV. If I do 'dnf update' to update the
> kernel, the problem doesn't happen, if I use gnome-software the
> problem always happens.
> 
> gnome-software leverages PackageKit for downloading and staging, then
> reboots to a systemd offline update target to do the update. So it's a
> joint effort. The detailed reproduce steps I've come up with are in
> comment 39 of the original bug report:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1227736#c39
> 
> The gist of the analysis in that comment is that the modified grub.cfg
> is a 0 length file, and that's why GRUB arrives at a grub> prompt
> instead of a menu. If I boot from alternate media and run xfs_repair
> -n, I get tons of errors - that log is attached to the bug report and
> is here
> 
> https://bugzilla.redhat.com/attachment.cgi?id=1263176
> 
> If all I do is do a normal mount, kernel reports log replay, no
> errors, and then I can boot the previously unbootable system just
> fine, and xfs_repair -n reports no errors. So it seems like the volume
> is left dirty at reboot time following these offline updates, and of
> course GRUB doesn't read the XFS log so it has no idea how to recover
> and find grub.cfg or any other affected files.

I'm not sure you'd want to replay the log from grub anyway. :)

So... you fire up gnome-software, click "Update", it downloads rpms to
the hard disk, reboots, (presumably) installs the rpms, reboots again,
and after the /second/ reboot grub just barfs up a grub> prompt?

Now I'm curious how systemd actually triggers the reboot.  Unmounting
would seem to force the log and push the AIL, which (you'd think) would
be enough.  But maybe systemd is doing something weird?  Or maybe
post-update we fail to unmount / ?

(Tempting to, uh, "delegate" to Eric Sandeen. :))

--D

> But with the available information I can't tell why the fs is left
> dirty at reboot. Using the same configuration, the problem doesn't
> happen with ext4 or Btrfs, but it may be normal behavior and what
> really needs to be done is teach GRUB how to do log replay.
> 
> 
> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

On Wed, Mar 15, 2017 at 1:23 AM, Darrick J. Wong
<darrick.wong@oracle.com> wrote:

>
> So... you fire up gnome-software, click "Update", it downloads rpms to
> the hard disk, reboots, (presumably) installs the rpms, reboots again,
> and after the /second/ reboot grub just barfs up a grub> prompt?

Correct.


> Now I'm curious how systemd actually triggers the reboot.  Unmounting
> would seem to force the log and push the AIL, which (you'd think) would
> be enough.  But maybe systemd is doing something weird?  Or maybe
> post-update we fail to unmount / ?

If I boot with the following parameters, and use virsh console to
track what's going on right after the updating,

systemd.log_level=debug systemd.log_target=console console=ttyS0,38400



Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Process 304 (plymouthd) has been marked to be excluded from killing.
It is running from the root file system, and thus likely to block
re-mounting of the root file system to read-only. Please consider
moving it into an initrd file system instead.
Unmounting file systems.
Remounting '/tmp' read-only with options 'seclabel'.
Unmounting /tmp.
Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
All filesystems unmounted.
Deactivating swaps.
All swaps deactivated.
Detaching loop devices.
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
  device-enumerator: scanning /sys/class
All loop devices detached.
Detaching DM devices.
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
  device-enumerator: scanning /sys/class
All DM devices detached.
Spawned /usr/lib/systemd/system-shutdown/mdadm.shutdown as 8408.
/usr/lib/systemd/system-shutdown/mdadm.shutdown succeeded.
system-shutdown succeeded.
Failed to read reboot parameter file: No such file or directory
Rebooting.
[   52.963598] Unregister pv shared memory for cpu 0
[   52.965736] Unregister pv shared memory for cpu 1
[   52.966795] sd 1:0:0:0: [sda] Synchronizing SCSI cache
[   52.991220] reboot: Restarting system
[   52.993119] reboot: machine restart
<no further entries, VM shuts down>

However, I get a different result. This time I get a stale grub menu
with a single kernel option (same as if the update hadn't happened).
If I boot, kernel messages reports log replay. When I reboot again,
now I have a two kernel grub menu. I'm guessing the slowness of
systemd debug is



-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

On Wed, Mar 15, 2017 at 12:11 PM, Chris Murphy <lists@colorremedies.com> wrote:
> I'm guessing the slowness of
> systemd debug is

resulting in the difference; more time to commit the changes?

Anyway, it looks / is remounted read-only before reboot. Note, this is
a single partition/volume setup, so /boot/grub2/grub.cfg being
modified by grubby is on the fs being remounted read-only and reboot
time.

-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Darrick J. Wong]

On Wed, Mar 15, 2017 at 12:11:22PM -0600, Chris Murphy wrote:
> On Wed, Mar 15, 2017 at 1:23 AM, Darrick J. Wong
> <darrick.wong@oracle.com> wrote:
> 
> >
> > So... you fire up gnome-software, click "Update", it downloads rpms to
> > the hard disk, reboots, (presumably) installs the rpms, reboots again,
> > and after the /second/ reboot grub just barfs up a grub> prompt?
> 
> Correct.
> 
> 
> > Now I'm curious how systemd actually triggers the reboot.  Unmounting
> > would seem to force the log and push the AIL, which (you'd think) would
> > be enough.  But maybe systemd is doing something weird?  Or maybe
> > post-update we fail to unmount / ?
> 
> If I boot with the following parameters, and use virsh console to
> track what's going on right after the updating,
> 
> systemd.log_level=debug systemd.log_target=console console=ttyS0,38400
> 
> 
> 
> Sending SIGTERM to remaining processes...
> Sending SIGKILL to remaining processes...
> Process 304 (plymouthd) has been marked to be excluded from killing.
> It is running from the root file system, and thus likely to block
> re-mounting of the root file system to read-only. Please consider
> moving it into an initrd file system instead.

Heh.  So the splash screen is still running and we can't unmount /.
Consequently systemd remounts the fs readonly...

> Unmounting file systems.
> Remounting '/tmp' read-only with options 'seclabel'.
> Unmounting /tmp.
> Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.

(I wonder why it does this three times?)

> All filesystems unmounted.
> Deactivating swaps.
> All swaps deactivated.
> Detaching loop devices.
> device-enumerator: scan all dirs
>   device-enumerator: scanning /sys/bus
>   device-enumerator: scanning /sys/class
> All loop devices detached.
> Detaching DM devices.
> device-enumerator: scan all dirs
>   device-enumerator: scanning /sys/bus
>   device-enumerator: scanning /sys/class
> All DM devices detached.
> Spawned /usr/lib/systemd/system-shutdown/mdadm.shutdown as 8408.
> /usr/lib/systemd/system-shutdown/mdadm.shutdown succeeded.
> system-shutdown succeeded.
> Failed to read reboot parameter file: No such file or directory
> Rebooting.
> [   52.963598] Unregister pv shared memory for cpu 0
> [   52.965736] Unregister pv shared memory for cpu 1
> [   52.966795] sd 1:0:0:0: [sda] Synchronizing SCSI cache
> [   52.991220] reboot: Restarting system
> [   52.993119] reboot: machine restart

...and reboots, having not unmounted, and (I guess?) without forcing the
log to write all of the logged changes back to the metadata.  They
changes are safely recorded /in/ the log....

> <no further entries, VM shuts down>
> 
> However, I get a different result. This time I get a stale grub menu
> with a single kernel option (same as if the update hadn't happened).
> If I boot, kernel messages reports log replay. When I reboot again,

...because mounting replays them and after that the metadata is up to
date.

Strange, though, because afaict we xfs_attr_quiesce seems to do the
right things when we remount rw->ro, so you wouldn't think that there
would be a need to recover.  Maybe there's something funny going on
w.r.t. the three messages about remounting readonly?

> now I have a two kernel grub menu. I'm guessing the slowness of
> systemd debug is resulting in the difference; more time to commit the
> changes?
>
> Anyway, it looks / is remounted read-only before reboot. Note, this is
> a single partition/volume setup, so /boot/grub2/grub.cfg being
> modified by grubby is on the fs being remounted read-only and reboot
> time.

--D

> 
> 
> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Darrick J. Wong]

On Wed, Mar 15, 2017 at 12:13:55PM -0700, Darrick J. Wong wrote:
> On Wed, Mar 15, 2017 at 12:11:22PM -0600, Chris Murphy wrote:
> > On Wed, Mar 15, 2017 at 1:23 AM, Darrick J. Wong
> > <darrick.wong@oracle.com> wrote:
> > 
> > >
> > > So... you fire up gnome-software, click "Update", it downloads rpms to
> > > the hard disk, reboots, (presumably) installs the rpms, reboots again,
> > > and after the /second/ reboot grub just barfs up a grub> prompt?
> > 
> > Correct.
> > 
> > 
> > > Now I'm curious how systemd actually triggers the reboot.  Unmounting
> > > would seem to force the log and push the AIL, which (you'd think) would
> > > be enough.  But maybe systemd is doing something weird?  Or maybe
> > > post-update we fail to unmount / ?
> > 
> > If I boot with the following parameters, and use virsh console to
> > track what's going on right after the updating,
> > 
> > systemd.log_level=debug systemd.log_target=console console=ttyS0,38400
> > 
> > 
> > 
> > Sending SIGTERM to remaining processes...
> > Sending SIGKILL to remaining processes...
> > Process 304 (plymouthd) has been marked to be excluded from killing.
> > It is running from the root file system, and thus likely to block
> > re-mounting of the root file system to read-only. Please consider
> > moving it into an initrd file system instead.
> 
> Heh.  So the splash screen is still running and we can't unmount /.
> Consequently systemd remounts the fs readonly...
> 
> > Unmounting file systems.
> > Remounting '/tmp' read-only with options 'seclabel'.
> > Unmounting /tmp.
> > Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> > Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> > Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> 
> (I wonder why it does this three times?)

Because systemd (v229, anyway) doesn't ever unmount /; it merely tries
to remount it readonly.  Regrettably it /also/ throws away the return
code from the mount call so there's no way to positively confirm my
current theory, which is that we don't actually succeed at the
remount-ro request and that's why the log needs recovery after the
reboot.

So maybe plymouthd, or journald, or systemd, or something else entirely
still holds a writable file descriptor to somewhere on the rootfs.  I
guess you could build a custom systemd that actually logs the return
value....

> > All filesystems unmounted.
> > Deactivating swaps.
> > All swaps deactivated.
> > Detaching loop devices.
> > device-enumerator: scan all dirs
> >   device-enumerator: scanning /sys/bus
> >   device-enumerator: scanning /sys/class
> > All loop devices detached.
> > Detaching DM devices.
> > device-enumerator: scan all dirs
> >   device-enumerator: scanning /sys/bus
> >   device-enumerator: scanning /sys/class
> > All DM devices detached.
> > Spawned /usr/lib/systemd/system-shutdown/mdadm.shutdown as 8408.
> > /usr/lib/systemd/system-shutdown/mdadm.shutdown succeeded.
> > system-shutdown succeeded.
> > Failed to read reboot parameter file: No such file or directory
> > Rebooting.
> > [   52.963598] Unregister pv shared memory for cpu 0
> > [   52.965736] Unregister pv shared memory for cpu 1
> > [   52.966795] sd 1:0:0:0: [sda] Synchronizing SCSI cache
> > [   52.991220] reboot: Restarting system
> > [   52.993119] reboot: machine restart
> 
> ...and reboots, having not unmounted, and (I guess?) without forcing the
> log to write all of the logged changes back to the metadata.  They
> changes are safely recorded /in/ the log....
> 
> > <no further entries, VM shuts down>
> > 
> > However, I get a different result. This time I get a stale grub menu
> > with a single kernel option (same as if the update hadn't happened).
> > If I boot, kernel messages reports log replay. When I reboot again,
> 
> ...because mounting replays them and after that the metadata is up to
> date.
> 
> Strange, though, because afaict we xfs_attr_quiesce seems to do the
> right things when we remount rw->ro, so you wouldn't think that there
> would be a need to recover.  Maybe there's something funny going on
> w.r.t. the three messages about remounting readonly?

Or maybe we're simply /not/ remounting readonly despite the log
messages, as mentioned above.

--D

> > now I have a two kernel grub menu. I'm guessing the slowness of
> > systemd debug is resulting in the difference; more time to commit the
> > changes?
> >
> > Anyway, it looks / is remounted read-only before reboot. Note, this is
> > a single partition/volume setup, so /boot/grub2/grub.cfg being
> > modified by grubby is on the fs being remounted read-only and reboot
> > time.
> 
> --D
> 
> > 
> > 
> > 
> > -- 
> > Chris Murphy
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

On Wed, Mar 15, 2017 at 1:30 PM, Darrick J. Wong
<darrick.wong@oracle.com> wrote:
> On Wed, Mar 15, 2017 at 12:13:55PM -0700, Darrick J. Wong wrote:
>> On Wed, Mar 15, 2017 at 12:11:22PM -0600, Chris Murphy wrote:
>> > On Wed, Mar 15, 2017 at 1:23 AM, Darrick J. Wong
>> > <darrick.wong@oracle.com> wrote:

>> > Unmounting file systems.
>> > Remounting '/tmp' read-only with options 'seclabel'.
>> > Unmounting /tmp.
>> > Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
>> > Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
>> > Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
>>
>> (I wonder why it does this three times?)
>
> Because systemd (v229, anyway) doesn't ever unmount /; it merely tries
> to remount it readonly.  Regrettably it /also/ throws away the return
> code from the mount call so there's no way to positively confirm my
> current theory, which is that we don't actually succeed at the
> remount-ro request and that's why the log needs recovery after the
> reboot.

The last line after those three is "All filesystems unmounted." So if
it isn't really unmounting, then that's a bogus message.

Also, on Btrfs I'm getting the same three remounting messages, and the
unmounted message.

Remounting '/' read-only with options
'seclabel,space_cache,subvolid=5,subvol=/'.
Remounting '/' read-only with options
'seclabel,space_cache,subvolid=5,subvol=/'.
Remounting '/' read-only with options
'seclabel,space_cache,subvolid=5,subvol=/'.
All filesystems unmounted.


> So maybe plymouthd, or journald, or systemd, or something else entirely
> still holds a writable file descriptor to somewhere on the rootfs.  I
> guess you could build a custom systemd that actually logs the return
> value....

Yuck. I'll ask on systemd list if there's a way to get more verbose
information about this without that. It really ought to be in
systemd.log_level=debug anyway.


-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

On Thu, Mar 16, 2017 at 10:07 PM, Chris Murphy <lists@colorremedies.com> wrote:

> Yuck. I'll ask on systemd list if there's a way to get more verbose
> information about this without that. It really ought to be in
> systemd.log_level=debug anyway.

I've gotten one response so far, which did not answer the question how
to get more verbose debug information.
https://lists.freedesktop.org/archives/systemd-devel/2017-March/038502.html

The assertion there is that it's virtually certainly an EBUSY exit code.

Additionally:
a. Systemd only ever does a remount to read-only with root fs. It
never umounts it.
https://github.com/systemd/systemd/blob/master/src/core/umount.c line 413

b. The three messages suggests that it's retrying to remount ro.

c. The failure to remount ro is predicted by the plymouth message "It
is running from the root file system, and thus likely to block
re-mounting of the root file system to read-only."

d. All of this happens on ext4, XFS, and Btrfs. But only XFS manifests
by being unbootable.

Ostensibly all file system (volume formats) could be dirty. I didn't
previously check this. But repeating the test with ext4 I find this
after the offline update reboot

[root@f25h ~]# e2fsck -nfv /dev/mapper/vg-f25ext1
e2fsck 1.43.3 (04-Sep-2016)
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (7195921, counted=7185813).
Fix? no

Free inodes count wrong (1961935, counted=1957897).
Fix? no

OK so what about mounting?

[25247.672099] EXT4-fs (dm-11): recovery complete
[25247.676051] EXT4-fs (dm-11): mounted filesystem with ordered data
mode. Opts: (null)

OK sure enough. ext4 is dirty too. It just doesn't manifest by being
unbootable. It boots after offline update, and gets fixed by journal
replay.


https://github.com/systemd/systemd/blob/master/src/core/shutdown.c
line 213 suggests a sync happens. This sync is after pk offline update
has finished, so why is this sync not syncing with XFS and ext4? Why
does the kernel permit a reboot before root fs has been cleanly
umounted?

Meanwhile, retesting on Btrfs, offline check reports no error after
the pk offline update reboot; nor when mounting the fs.



-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Dave Chinner]

On Tue, Mar 21, 2017 at 01:47:12PM -0600, Chris Murphy wrote:
> On Thu, Mar 16, 2017 at 10:07 PM, Chris Murphy <lists@colorremedies.com> wrote:
> 
> > Yuck. I'll ask on systemd list if there's a way to get more verbose
> > information about this without that. It really ought to be in
> > systemd.log_level=debug anyway.
> 
> I've gotten one response so far, which did not answer the question how
> to get more verbose debug information.
> https://lists.freedesktop.org/archives/systemd-devel/2017-March/038502.html
> 
> The assertion there is that it's virtually certainly an EBUSY exit code.
> 
> Additionally:
> a. Systemd only ever does a remount to read-only with root fs. It
> never umounts it.
> https://github.com/systemd/systemd/blob/master/src/core/umount.c line 413
> 
> b. The three messages suggests that it's retrying to remount ro.
> 
> c. The failure to remount ro is predicted by the plymouth message "It
> is running from the root file system, and thus likely to block
> re-mounting of the root file system to read-only."
> 
> d. All of this happens on ext4, XFS, and Btrfs. But only XFS manifests
> by being unbootable.

Because, in this case, both ext4 and XFS require a successful
remount-ro to guarantee that the metadata grub is relying on is
written to disk. In the case of ext4, you've just been lucky,
probably because it has a faster background journal flush cycle.

....

> https://github.com/systemd/systemd/blob/master/src/core/shutdown.c
> line 213 suggests a sync happens. This sync is after pk offline update
> has finished, so why is this sync not syncing with XFS and ext4? Why
> does the kernel permit a reboot before root fs has been cleanly
> umounted?

sync is /not sufficient/ to force metadata to disk. All that is
required during sync for a journalling filesystem is to ensure that
data is flushed and the journal is committed to disk. If the system
crashes, then log recovery is run and no metadata or data is lost.

Hence, if grub is trying to find the new kernel that was written to
disk then sync()d but not unmounted/remounted before rebooting, the
metadata that grub needs to find the new kernel image is in the
journal, not resting on disk as grub is assuming it will be.

Hence, boot fails because grub/systemd did not correctly
sync/unmount/remount the filesystem before boot. And, no, having
grub replay the log is not the answer - that's a recipe for endless
filesystem corruption problems that filesystem developers will
disown with "grub screwed up your filesystem, go shout at them".

FYI, I've ranted previously over many years about how broken grub's
kernel update and retreival process is fundamentally broken, but
it's never been fixed(*). As a result, I don't use grub on any of my
systems, nor do I recommend that anyone else use it.

(*) The simple fix for grub to freeze/unfreeze the filesystem rather
than/after calling sync() - this does the same thing as remount-ro,
but unlike remount-ro it does not fail if there are writable file
descriptors open.

> Meanwhile, retesting on Btrfs, offline check reports no error after
> the pk offline update reboot; nor when mounting the fs.

of course - it's the nature of btrfs structure that the superblock
is updated to only point at a valid tree. If the superblock is not
updated, then none of the update in progress is even known to
exist...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Darrick J. Wong]

On Wed, Mar 22, 2017 at 03:34:38PM +1100, Dave Chinner wrote:
> On Tue, Mar 21, 2017 at 01:47:12PM -0600, Chris Murphy wrote:
> > On Thu, Mar 16, 2017 at 10:07 PM, Chris Murphy <lists@colorremedies.com> wrote:
> > 
> > > Yuck. I'll ask on systemd list if there's a way to get more verbose
> > > information about this without that. It really ought to be in
> > > systemd.log_level=debug anyway.
> > 
> > I've gotten one response so far, which did not answer the question how
> > to get more verbose debug information.
> > https://lists.freedesktop.org/archives/systemd-devel/2017-March/038502.html
> > 
> > The assertion there is that it's virtually certainly an EBUSY exit code.
> > 
> > Additionally:
> > a. Systemd only ever does a remount to read-only with root fs. It
> > never umounts it.
> > https://github.com/systemd/systemd/blob/master/src/core/umount.c line 413
> > 
> > b. The three messages suggests that it's retrying to remount ro.
> > 
> > c. The failure to remount ro is predicted by the plymouth message "It
> > is running from the root file system, and thus likely to block
> > re-mounting of the root file system to read-only."
> > 
> > d. All of this happens on ext4, XFS, and Btrfs. But only XFS manifests
> > by being unbootable.
> 
> Because, in this case, both ext4 and XFS require a successful
> remount-ro to guarantee that the metadata grub is relying on is
> written to disk. In the case of ext4, you've just been lucky,
> probably because it has a faster background journal flush cycle.
> 
> ....
> 
> > https://github.com/systemd/systemd/blob/master/src/core/shutdown.c
> > line 213 suggests a sync happens. This sync is after pk offline update
> > has finished, so why is this sync not syncing with XFS and ext4? Why
> > does the kernel permit a reboot before root fs has been cleanly
> > umounted?
> 
> sync is /not sufficient/ to force metadata to disk. All that is
> required during sync for a journalling filesystem is to ensure that
> data is flushed and the journal is committed to disk. If the system
> crashes, then log recovery is run and no metadata or data is lost.
> 
> Hence, if grub is trying to find the new kernel that was written to
> disk then sync()d but not unmounted/remounted before rebooting, the
> metadata that grub needs to find the new kernel image is in the
> journal, not resting on disk as grub is assuming it will be.

Yep, that's exactly what I was getting at in my last mail.

> Hence, boot fails because grub/systemd did not correctly
> sync/unmount/remount the filesystem before boot. And, no, having
> grub replay the log is not the answer - that's a recipe for endless
> filesystem corruption problems that filesystem developers will
> disown with "grub screwed up your filesystem, go shout at them".

The prospect of replaying the log in 640K of memory is comical to me!

:P

> FYI, I've ranted previously over many years about how broken grub's
> kernel update and retreival process is fundamentally broken, but
> it's never been fixed(*). As a result, I don't use grub on any of my
> systems, nor do I recommend that anyone else use it.
> 
> (*) The simple fix for grub to freeze/unfreeze the filesystem rather
> than/after calling sync() - this does the same thing as remount-ro,
> but unlike remount-ro it does not fail if there are writable file
> descriptors open.

You mean systemd here, right?  AFAIK the grub scripts (at least on
Debian) don't actually retrieve or update kernels; apt/dpkg still do
that, and they don't use this weird systemd update mode.  So it's really
systemd that ought to be freezing the rootfs as the last thing it does
before triggering the kernel reboot.

(Then again I have separate /boot so it probably gets umounted
successfully bootsplash or otherwise. :P)

--D

> > Meanwhile, retesting on Btrfs, offline check reports no error after
> > the pk offline update reboot; nor when mounting the fs.
> 
> of course - it's the nature of btrfs structure that the superblock
> is updated to only point at a valid tree. If the superblock is not
> updated, then none of the update in progress is even known to
> exist...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

On Tue, Mar 21, 2017 at 10:34 PM, Dave Chinner <david@fromorbit.com> wrote:

> FYI, I've ranted previously over many years about how broken grub's
> kernel update and retreival process is fundamentally broken, but
> it's never been fixed(*). As a result, I don't use grub on any of my
> systems, nor do I recommend that anyone else use it.

OK I follow all the previous comments, thanks, but here is where
there's a departure.

On Ubuntu systems, maybe all Debian systems, they are using
grub-mkconfig script, possibly in the form of update-grub, to create a
completely new grub.cfg. That script writes out grub.cfg.new and then
renames it to grub.cfg.

Fedora/RH systems don't use that at all. They don't use anything from
upstream GRUB to do updates in fact. The kernel RPM uses
new-kernel-pkg which in turn calls things like dracut, and eventually
it has grubby modify the grub.cfg in place. The grubby package has no
relationship at all with upstream grub. Directly modifying grub.cfg
and overwriting it seems risky, but I'm pretty sure that's how it
works.


> (*) The simple fix for grub to freeze/unfreeze the filesystem rather
> than/after calling sync() - this does the same thing as remount-ro,
> but unlike remount-ro it does not fail if there are writable file
> descriptors open.


OK. It looks to me like systemd eventually gives up on the remount-ro,
and then just reboots. That strikes me as a flawed design. Systemd
needs to wait longer (which is vague advice), or maybe after the 3rd
failed remount-ro, insert a freeze/unfreeze, then reboot. How does
that sound?


-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Dave Chinner]

On Wed, Mar 22, 2017 at 09:36:21AM -0600, Chris Murphy wrote:
> On Tue, Mar 21, 2017 at 10:34 PM, Dave Chinner <david@fromorbit.com> wrote:
> 
> > FYI, I've ranted previously over many years about how broken grub's
> > kernel update and retreival process is fundamentally broken, but
> > it's never been fixed(*). As a result, I don't use grub on any of my
> > systems, nor do I recommend that anyone else use it.
> 
> OK I follow all the previous comments, thanks, but here is where
> there's a departure.
> 
> On Ubuntu systems, maybe all Debian systems, they are using
> grub-mkconfig script, possibly in the form of update-grub, to create a
> completely new grub.cfg. That script writes out grub.cfg.new and then
> renames it to grub.cfg.
> 
> Fedora/RH systems don't use that at all. They don't use anything from
> upstream GRUB to do updates in fact. The kernel RPM uses
> new-kernel-pkg which in turn calls things like dracut, and eventually
> it has grubby modify the grub.cfg in place. The grubby package has no
> relationship at all with upstream grub. Directly modifying grub.cfg
> and overwriting it seems risky, but I'm pretty sure that's how it
> works.
> 
> 
> > (*) The simple fix for grub to freeze/unfreeze the filesystem rather
> > than/after calling sync() - this does the same thing as remount-ro,
> > but unlike remount-ro it does not fail if there are writable file
> > descriptors open.
> 
> 
> OK. It looks to me like systemd eventually gives up on the remount-ro,
> and then just reboots. That strikes me as a flawed design. Systemd
> needs to wait longer (which is vague advice), or maybe after the 3rd
> failed remount-ro, insert a freeze/unfreeze, then reboot. How does
> that sound?

Sounds supremely fragile and fucked up. Sure, lets add another
layers of hacks to systemd to work around the problem that systemd
can't remount-ro the root filesystem because systemd holds open
write fds on the root fs. Fix the problem that causes open write fds
on the root fs at the end of shutdown? Nah, that's stupid talk -
just hack a bandaid over the top....

And let's also ignore the fact that, in this case, the remount-ro is
effectively working around bootloader updates not providing
durability guarantees.  i.e. the bootloader doesn't do guaranteed
safe updates and requires 3rd party action to provide update
integrity.  The distro update scripts are all different because the
bootloader doesn't provide the necessary update infrastructure the
distros require, and it's extremely naive to expect the developers
of those scripts to understand that they need to provide integrity
guarantees that the bootloader itself doesn't provide.

Really, the bootloader needs fixing, then we can go back to
blissfully ignoring all the stupid bugs in the steaming pile we know
as systemd...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

On Wed, Mar 22, 2017 at 4:19 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Wed, Mar 22, 2017 at 09:36:21AM -0600, Chris Murphy wrote:

>> OK. It looks to me like systemd eventually gives up on the remount-ro,
>> and then just reboots. That strikes me as a flawed design. Systemd
>> needs to wait longer (which is vague advice), or maybe after the 3rd
>> failed remount-ro, insert a freeze/unfreeze, then reboot. How does
>> that sound?
>
> Sounds supremely fragile and fucked up. Sure, lets add another
> layers of hacks to systemd to work around the problem that systemd
> can't remount-ro the root filesystem because systemd holds open
> write fds on the root fs. Fix the problem that causes open write fds
> on the root fs at the end of shutdown? Nah, that's stupid talk -
> just hack a bandaid over the top....

That's adorable, no really. I thought you were on vacation?

Anyway, there isn't enough information from systemd's debugging to
know exactly why it's failing to remount-ro. The trail of breadcrumbs
we have implicates plymouth, not systemd which I reported in the 2nd
message of this thread.

Process 304 (plymouthd) has been marked to be excluded from killing.
It is running from the root file system, and thus likely to block
re-mounting of the root file system to read-only. Please consider
moving it into an initrd file system instead.

I already suggested elsewhere that this is a distribution problem, and
dracut needs to always stuff plymouth into the initramfs. When I do
this manually, the reported problem does not happen; where if plymouth
is running from root fs *and* /boot is a directory on that same root
fs, this problem always happens.

So plymouth blames itself and dracut, pretty much. Is there some
additional technical reason to blame systemd for this problem?



> And let's also ignore the fact that, in this case, the remount-ro is
> effectively working around bootloader updates not providing
> durability guarantees.  i.e. the bootloader doesn't do guaranteed
> safe updates and requires 3rd party action to provide update
> integrity.  The distro update scripts are all different because the
> bootloader doesn't provide the necessary update infrastructure the
> distros require, and it's extremely naive to expect the developers
> of those scripts to understand that they need to provide integrity
> guarantees that the bootloader itself doesn't provide.

The only bootloader that has a script for updating its configuration
file is GRUB, and has I previously explained that's grub-mkconfig and
it's not used at all on Fedora (or CentOS or RHEL). For at least 10
years it has been the job of grubby, going back to grub legacy. And it
doesn't matter what the bootloader being used is, grubby directly
modifies (read-modify-write) the configuration file for grub,
systemd-boot, extlinux, lilo, u-boot, and a pile of others.

> Really, the bootloader needs fixing, then we can go back to
> blissfully ignoring all the stupid bugs in the steaming pile we know
> as systemd...

If I thought that bitching about the deplorable state of bootloaders
on Linux, the total lack of cooperation among the distributions when
it comes to the basic task of booting a fucking computer in 2017, when
this shit was solved fucking 25 goddamn years ago by Microsoft and
Apple, and they don't have the myriad problems and bugs I'm constantly
finding on just Fedora release to release? I would have done it a long
time ago.

But I sincerely doubt anyone on this list gives a flying fuck about
bootloader problems. Actually, characterizing the distros as
uncooperative is too polite. They actively sabotage each other. It's
so bad, they even sabotage themselves (try doing back to back side by
side installations of Fedora). Each distro's grub is effectively a
fork. Fedora ostensibly uses mostly upstream stuff, but with grub the
upstream copy has 200+ hacks that are Fedora specific, SUSE has more
and more invasive hacks, both of them are mutually incompatible. The
distros don't agree. They're different operating systems that just so
happen to share a kernel (or an init service). This fact isn't going
to get solved on this list.

So if you have something else for me to actually act on, I'm happy to
try and push this forward so a teeny tiny handful of users can
actually do single volume XFS booting and not end up face planted when
they do offline updates. Thanks.


-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Dave Chinner]

On Wed, Mar 22, 2017 at 06:15:42PM -0600, Chris Murphy wrote:
> On Wed, Mar 22, 2017 at 4:19 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Wed, Mar 22, 2017 at 09:36:21AM -0600, Chris Murphy wrote:
> 
> >> OK. It looks to me like systemd eventually gives up on the remount-ro,
> >> and then just reboots. That strikes me as a flawed design. Systemd
> >> needs to wait longer (which is vague advice), or maybe after the 3rd
> >> failed remount-ro, insert a freeze/unfreeze, then reboot. How does
> >> that sound?
> >
> > Sounds supremely fragile and fucked up. Sure, lets add another
> > layers of hacks to systemd to work around the problem that systemd
> > can't remount-ro the root filesystem because systemd holds open
                                                 ^^^^^^^
Meant to say "something" here, not "systemd".

> > write fds on the root fs. Fix the problem that causes open write fds
> > on the root fs at the end of shutdown? Nah, that's stupid talk -
> > just hack a bandaid over the top....

<snip misdirected rant about it not being systemd's fault>

> > Really, the bootloader needs fixing, then we can go back to
> > blissfully ignoring all the stupid bugs in the steaming pile we know
> > as systemd...
> 
> If I thought that bitching about the deplorable state of bootloaders
> on Linux, the total lack of cooperation among the distributions when
> it comes to the basic task of booting a fucking computer in 2017, when
> this shit was solved fucking 25 goddamn years ago by Microsoft and
> Apple, and they don't have the myriad problems and bugs I'm constantly
> finding on just Fedora release to release? I would have done it a long
> time ago.

We solved reliable booting 20 years ago - it's a simple as "set up
separate boot partition and use Lilo". Been working reliably for me
and all the systems I've installed and managed since 1995.... :/

<feel free to insert you own "rocking chairs and lawn" comments :P>

> But I sincerely doubt anyone on this list gives a flying fuck about
> bootloader problems.

Nobody should care, because in future we should be replacing the
bootloader with "copy kernels into EFI partition and boot from EFI".
i.e. bypassing the need for a bootloader altogether. Worked
perfectly well for booting ia64 machines 10-15 years ago....

> Actually, characterizing the distros as
> uncooperative is too polite. They actively sabotage each other. It's
> so bad, they even sabotage themselves (try doing back to back side by
> side installations of Fedora). Each distro's grub is effectively a
> fork. Fedora ostensibly uses mostly upstream stuff, but with grub the
> upstream copy has 200+ hacks that are Fedora specific, SUSE has more
> and more invasive hacks, both of them are mutually incompatible. The
> distros don't agree. They're different operating systems that just so
> happen to share a kernel (or an init service). This fact isn't going
> to get solved on this list.

Yup, that's exactly what I was implying is the problem - everyone
has their own special snowflake rather than working together to
produce one simple, robust, reliable solution to a well known problem...

> So if you have something else for me to actually act on, I'm happy to
> try and push this forward so a teeny tiny handful of users can
> actually do single volume XFS booting and not end up face planted when
> they do offline updates. Thanks.

*As I always say*: fix the root cause of the problem, don't hack
around it because that almost always makes things worse in the long
run. Hacking code into systemd to work around a distro-specific
and/or bootloader bug is simply not a viable long term solution to
the collaboration problems that have been pointed out here...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Chris Murphy]

On Thu, Mar 23, 2017 at 5:01 PM, Dave Chinner <david@fromorbit.com> wrote:

>
> We solved reliable booting 20 years ago - it's a simple as "set up
> separate boot partition and use Lilo". Been working reliably for me
> and all the systems I've installed and managed since 1995.... :/

The problem doesn't happen with a separate boot volume; different
bootloader isn't necessary. No one is actively supporting LILO
anymore. There are apparently so many uses cases people want to
support that it doesn't support that no one cares to bring it up to
speed. It's like saying the old ice box works just fine since 1895,
just that it's someone else's fault they don't deliver ice blocks
anymore.



>
> <feel free to insert you own "rocking chairs and lawn" comments :P>
>
>> But I sincerely doubt anyone on this list gives a flying fuck about
>> bootloader problems.
>
> Nobody should care, because in future we should be replacing the
> bootloader with "copy kernels into EFI partition and boot from EFI".
> i.e. bypassing the need for a bootloader altogether. Worked
> perfectly well for booting ia64 machines 10-15 years ago....

Unworkable for dual boot, let alone multibooting. The ESP is too
small, no installer team is going to take responsibility for growing
it, Windows gets confused when there's more than one on a disk,  and
the UEFI spec has no standardization for firmware boot managers, so
there's no way to provide generic documentation for the built-in boot
manager: and thus the bootloader needs to also be a boot manager in
order to document it. And that leaves us right back to each distro
owning it's own bootloader even on UEFI systems, where they have no
idea WTF other distros have been installed (yes they will all see
Windows, but it won't discover each other because Windows is not
really a competitor, again leaving us with distros eating each others
babies, etc.)



>
>> Actually, characterizing the distros as
>> uncooperative is too polite. They actively sabotage each other. It's
>> so bad, they even sabotage themselves (try doing back to back side by
>> side installations of Fedora). Each distro's grub is effectively a
>> fork. Fedora ostensibly uses mostly upstream stuff, but with grub the
>> upstream copy has 200+ hacks that are Fedora specific, SUSE has more
>> and more invasive hacks, both of them are mutually incompatible. The
>> distros don't agree. They're different operating systems that just so
>> happen to share a kernel (or an init service). This fact isn't going
>> to get solved on this list.
>
> Yup, that's exactly what I was implying is the problem - everyone
> has their own special snowflake rather than working together to
> produce one simple, robust, reliable solution to a well known problem...

Unsolvable. Systemd folks wrote up the bootloaderspec, and about 10
minutes later there was a more sane fork by Matthew Garrett, and 10
minutes later there were three implementations, all of which are
mutually incompatible with each other. Still no cross distro
cooperation either.


>> So if you have something else for me to actually act on, I'm happy to
>> try and push this forward so a teeny tiny handful of users can
>> actually do single volume XFS booting and not end up face planted when
>> they do offline updates. Thanks.
>
> *As I always say*: fix the root cause of the problem, don't hack
> around it because that almost always makes things worse in the long
> run. Hacking code into systemd to work around a distro-specific
> and/or bootloader bug is simply not a viable long term solution to
> the collaboration problems that have been pointed out here...

OK well in three replies you're still vague on precisely how the
bootloader is even involved in this problem, seeing as it doesn't make
any changes to the disk during kernel update, at all. If you want to
implicate grubby for doing the wrong thing, that'll need more detailed
explanation. From what I can tell the problem is systemd initiates a
reboot, while root fs is still mounted rw, the kernel tolerates this
and leaves the fs dirty, and the bootloader can't handle the fallout
and can't be expected to handle it.

It's either systemd needs to not do reboots when root fs can't be
remounted-ro; and therefore must not have indefinite exemptions from
killing processes that don't hold rw mount on volumes. LIterally any
bad program could exist somewhere at sometime, not exit, and thus
cause this problem. So even if plymouth is an instigator in this case,
the central problem seems to be either systemd for the indefinite kill
exemption, or it's the kernel for doing a reboot when a file system is
still dirty. And I don't know nearly enough about the separation of
responsibilities between systemd and the kernel to have a clue whether
the root cause is systemd or the kernel.

But I'm unconvinced it's plymouth, dracut, grubby, or the bootloader,
even though all of them are instigators.

And by the way, the freeze/unfreeze idea was your idea. So to bring
that up, and then in the very next email say it's fucked up to
actually insert it somewhere if multiple remount attempts fail is
bizarre. Since it's sufficiently fucked up to get you to actually
write out the words fucked up, maybe the idea shouldn't have been
mentioned in the first place? Why mention a bag of dicks as a solution
if you're just doing to say oh shit that's a bag of dicks are you
fucking crazy? Don't do that!


-- 
Chris Murphy

Re: boot failure after kernel update, imap claims in-use inode 661690 is free, would correct imap

[Eric Sandeen]

On 3/15/17 1:53 AM, Chris Murphy wrote:
> Hi,
> 
> A user reports a curious OS update bug with Fedora 22, 23, 24, and 25,
> where after a kernel update, upon reboot there's only a grub> prompt,
> no menu.
> 
> I've reproduced this using a qemu-kvm VM, with device SATA and unsafe
> caching, pointed to an LVM LV. If I do 'dnf update' to update the
> kernel, the problem doesn't happen, if I use gnome-software the
> problem always happens.
> 
> gnome-software leverages PackageKit for downloading and staging, then
> reboots to a systemd offline update target to do the update. So it's a
> joint effort. The detailed reproduce steps I've come up with are in
> comment 39 of the original bug report:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1227736#c39
> 
> The gist of the analysis in that comment is that the modified grub.cfg
> is a 0 length file, and that's why GRUB arrives at a grub> prompt
> instead of a menu. If I boot from alternate media and run xfs_repair
> -n, I get tons of errors - that log is attached to the bug report and
> is here
> 
> https://bugzilla.redhat.com/attachment.cgi?id=1263176

Ok, those errors are probably just due to a dirty log.  (I just sent
a patch to make this more clear in xfs_repair output with -n)

> If all I do is do a normal mount, kernel reports log replay,

Ok, so I think the question is: why after a clean reboot is the log
still dirty?  That's not supposed to happen.

systemd is still doing remount,ro before shooting the system in the
head, right?

Does a normal reboot also come up w/ a dirty log?

/me goes to read the bug...

-Eric