[4.3-rc1, regression] ext2 vs ext3/ext4 fs probing issues

[Dave Chinner <david@fromorbit.com>]

Hi folks,

The first of my test VMs that I upgraded to 4.3-rc1 from 4.2 has
been behaving rather strangely w.r.t. boot hangs and ext3
filesystems.

One the first cold boot of a new kernel, the boot appears to hang.
What i've discovered (which took a long time thanks to the shitpile
that is systemd) is that it appears to be doing a e2fsck on the root
device, and that is failing resulting in systemd outputing:

[FAILED] Failed to start File System Check on Root Device.

systemd then goes to shutting down the system and reboot but that
fail because it's still starting other stuff up and simultaneously
shutting shit down tha the stuff starting up depends on. It fucks up
badly, with teh last concole entries looking like it has hung
waiting for lvm/md devices to appear.

I can't tell you what the e2fsck failure is, because systemd oh so
helpfully overwrites the console output from fsck with all the other
shit that it is doing concurrently. it also fails to log it anywhere
because this has happened before A) the console logging has been
started and b) the the root fs is still only mounted RO at this
point.

Hence I have absolutely zero output apart from seeing the "checking
xxx%" update on the bottom line of the screen occasionally before it
is immediately overwritten by systemd logging something else.

So, warm reboot the VM (via system_reset in the qemu console), and
the system comes up. No fsck is run, but the filesystem is mounted
as ext2, not ext3.  What I see is this in dmesg:

[    2.322798] EXT2-fs (sda1): warning: mounting ext3 filesystem as ext2
[    2.325123] VFS: Mounted root (ext2 filesystem) readonly on device 8:1.

It's definitely an ext3 filesystem, but the interesting point is
that it has a clean journal so can be mounted as ext2:

$ sudo blkid /dev/sda1
/dev/sda1: UUID="b21615e5-fe8a-4ffc-ab80-c24cdc8b740a" SEC_TYPE="ext2" TYPE="ext3" PARTUUID="000efa91-01"
$ sudo dumpe2fs -h /dev/sda1 | grep -i feature
dumpe2fs 1.42.13 (17-May-2015)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Journal features:	journal_incompat_revoke
$

So, I change the kernel config from:

CONFIG_EXT2_FS=y
# CONFIG_EXT4_USE_FOR_EXT2 is not set

to

# CONFIG_EXT2_FS is not set
CONFIG_EXT4_USE_FOR_EXT2=y

And what I see is this:

[    2.228894] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[    2.238832] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)

Which tells me that there's a problem with fstype probe ordering
regressions w.r.t ext2 and ext3 as a result of removing the ext3
module. It also doesn't fail fsck checks now, so boots successfully
every time. I suspect the "boot hang" problem is that e2fsck sees a
dirty journal, fixes everything and then asks for a reboot, which
fails.

So, suspecting this, I switched back to the original
CONFIG_EXT2_FS=y build:

[    2.254213] EXT2-fs (sda1): error: couldn't mount because of unsupported optional features (4)
[    2.257076] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[    2.259712] EXT4-fs (sda1): INFO: recovery required on readonly filesystem
[    2.261740] EXT4-fs (sda1): write access will be enabled during recovery
[    2.336798] EXT4-fs (sda1): orphan cleanup on readonly fs
[    2.338492] EXT4-fs (sda1): 2 orphan inodes deleted
[    2.339922] EXT4-fs (sda1): recovery complete
[    2.346063] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    2.348349] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.

Which makes it pretty clean that if the journal was clean it would
have mounted as an ext2 filesystem, not ext3.  Basically, we need to
ensure that the ext4 module probes filesystems before the ext2
module when CONFIG_EXT2_FS=y is set so that ext3 filesystems are
correctly mounted....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com