ext4 confusion

ext4 confusion

[Randy Dunlap]

Hi,

With the following kernel config entries on Linux 4.16-rc3:

CONFIG_EXT2_FS=m
# CONFIG_EXT2_FS_XATTR is not set
CONFIG_EXT3_FS=m
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_ENCRYPTION is not set
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set

ext4_fill_super() tells me:

[    3.033174] EXT4-fs (sda5): couldn't mount as ext3 due to feature incompatibilities
[    3.100186] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
[    3.102683] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.


This is a new install, new filesystem. It has never been ext2 or ext3.

After bootup and before I do anything else, I can remount /dev/sda5 on / as
rw and everything is OK.

What am I doing wrong?

thanks,
-- 
~Randy

Re: ext4 confusion

[Eric Sandeen]

On 3/5/18 10:17 PM, Randy Dunlap wrote:
> Hi,
> 
> With the following kernel config entries on Linux 4.16-rc3:
> 
> CONFIG_EXT2_FS=m
> # CONFIG_EXT2_FS_XATTR is not set
> CONFIG_EXT3_FS=m
> # CONFIG_EXT3_FS_POSIX_ACL is not set
> # CONFIG_EXT3_FS_SECURITY is not set
> CONFIG_EXT4_FS=y
> CONFIG_EXT4_FS_POSIX_ACL=y
> CONFIG_EXT4_FS_SECURITY=y
> # CONFIG_EXT4_ENCRYPTION is not set
> # CONFIG_EXT4_DEBUG is not set
> CONFIG_JBD2=y
> # CONFIG_JBD2_DEBUG is not set
> 
> ext4_fill_super() tells me:
> 
> [    3.033174] EXT4-fs (sda5): couldn't mount as ext3 due to feature incompatibilities
> [    3.100186] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
> [    3.102683] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.
> 
> 
> This is a new install, new filesystem. It has never been ext2 or ext3.
> 
> After bootup and before I do anything else, I can remount /dev/sda5 on / as
> rw and everything is OK.
> 
> What am I doing wrong?

Hm you're the second person to report this in a week.  
(see EXT4-fs (mmcblk0p1): couldn't mount as ext3 due to feature incompatibilities)

It sounds like filesystem probing behavior; it tried to mount as ext3 and failed
(as it should fail) then moved on to ext4 and succeeded.

I sent:

[PATCH V2] ext4: don't complain about incorrect features when probing

which should fix it, but I really wonder why this is just now showing up for 
people; it could be a combination of /etc/filesystems, fstab entries, blkid
handling in mount, etc...

Is this a new OS/installer release where this is showing up, or did a simple
kernel upgrade trigger it?  If the latter, what kernel version didn't have
the above kernel message?

-Eric

Re: ext4 confusion

[Randy Dunlap]

On 03/05/2018 08:34 PM, Eric Sandeen wrote:
> On 3/5/18 10:17 PM, Randy Dunlap wrote:
>> Hi,
>>
>> With the following kernel config entries on Linux 4.16-rc3:
>>
>> CONFIG_EXT2_FS=m
>> # CONFIG_EXT2_FS_XATTR is not set
>> CONFIG_EXT3_FS=m
>> # CONFIG_EXT3_FS_POSIX_ACL is not set
>> # CONFIG_EXT3_FS_SECURITY is not set
>> CONFIG_EXT4_FS=y
>> CONFIG_EXT4_FS_POSIX_ACL=y
>> CONFIG_EXT4_FS_SECURITY=y
>> # CONFIG_EXT4_ENCRYPTION is not set
>> # CONFIG_EXT4_DEBUG is not set
>> CONFIG_JBD2=y
>> # CONFIG_JBD2_DEBUG is not set
>>
>> ext4_fill_super() tells me:
>>
>> [    3.033174] EXT4-fs (sda5): couldn't mount as ext3 due to feature incompatibilities
>> [    3.100186] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
>> [    3.102683] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.
>>
>>
>> This is a new install, new filesystem. It has never been ext2 or ext3.
>>
>> After bootup and before I do anything else, I can remount /dev/sda5 on / as
>> rw and everything is OK.
>>
>> What am I doing wrong?
> 
> Hm you're the second person to report this in a week.  
> (see EXT4-fs (mmcblk0p1): couldn't mount as ext3 due to feature incompatibilities)
> 
> It sounds like filesystem probing behavior; it tried to mount as ext3 and failed
> (as it should fail) then moved on to ext4 and succeeded.

Mostly succeeded, but why did it leave the filesystem as read-only?

> I sent:
> 
> [PATCH V2] ext4: don't complain about incorrect features when probing
> 
> which should fix it, but I really wonder why this is just now showing up for 
> people; it could be a combination of /etc/filesystems, fstab entries, blkid
> handling in mount, etc...
> 
> Is this a new OS/installer release where this is showing up, or did a simple
> kernel upgrade trigger it?  If the latter, what kernel version didn't have
> the above kernel message?

It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
rolling updates release.

thanks,
-- 
~Randy

Re: ext4 confusion

[Eric Sandeen]

On 3/5/18 10:38 PM, Randy Dunlap wrote:
> On 03/05/2018 08:34 PM, Eric Sandeen wrote:
>> On 3/5/18 10:17 PM, Randy Dunlap wrote:
>>> Hi,
>>>
>>> With the following kernel config entries on Linux 4.16-rc3:
>>>
>>> CONFIG_EXT2_FS=m
>>> # CONFIG_EXT2_FS_XATTR is not set
>>> CONFIG_EXT3_FS=m
>>> # CONFIG_EXT3_FS_POSIX_ACL is not set
>>> # CONFIG_EXT3_FS_SECURITY is not set
>>> CONFIG_EXT4_FS=y
>>> CONFIG_EXT4_FS_POSIX_ACL=y
>>> CONFIG_EXT4_FS_SECURITY=y
>>> # CONFIG_EXT4_ENCRYPTION is not set
>>> # CONFIG_EXT4_DEBUG is not set
>>> CONFIG_JBD2=y
>>> # CONFIG_JBD2_DEBUG is not set
>>>
>>> ext4_fill_super() tells me:
>>>
>>> [    3.033174] EXT4-fs (sda5): couldn't mount as ext3 due to feature incompatibilities
>>> [    3.100186] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
>>> [    3.102683] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.
>>>
>>>
>>> This is a new install, new filesystem. It has never been ext2 or ext3.
>>>
>>> After bootup and before I do anything else, I can remount /dev/sda5 on / as
>>> rw and everything is OK.
>>>
>>> What am I doing wrong?
>>
>> Hm you're the second person to report this in a week.  
>> (see EXT4-fs (mmcblk0p1): couldn't mount as ext3 due to feature incompatibilities)
>>
>> It sounds like filesystem probing behavior; it tried to mount as ext3 and failed
>> (as it should fail) then moved on to ext4 and succeeded.
> 
> Mostly succeeded, but why did it leave the filesystem as read-only?

oh it left it there?  Root usually mounts readonly then transitions to RW.
That didn't eventually happen?

> 
>> I sent:
>>
>> [PATCH V2] ext4: don't complain about incorrect features when probing
>>
>> which should fix it, but I really wonder why this is just now showing up for 
>> people; it could be a combination of /etc/filesystems, fstab entries, blkid
>> handling in mount, etc...
>>
>> Is this a new OS/installer release where this is showing up, or did a simple
>> kernel upgrade trigger it?  If the latter, what kernel version didn't have
>> the above kernel message?
> 
> It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
> rolling updates release.

Hrmph.  A lot of things go into this behavior, it may not be a kernel change at
all that has made it show up now...

-Eric

Re: ext4 confusion

[Randy Dunlap]

On 03/05/2018 08:40 PM, Eric Sandeen wrote:
> On 3/5/18 10:38 PM, Randy Dunlap wrote:
>> On 03/05/2018 08:34 PM, Eric Sandeen wrote:
>>> On 3/5/18 10:17 PM, Randy Dunlap wrote:
>>>> Hi,
>>>>
>>>> With the following kernel config entries on Linux 4.16-rc3:
>>>>
>>>> CONFIG_EXT2_FS=m
>>>> # CONFIG_EXT2_FS_XATTR is not set
>>>> CONFIG_EXT3_FS=m
>>>> # CONFIG_EXT3_FS_POSIX_ACL is not set
>>>> # CONFIG_EXT3_FS_SECURITY is not set
>>>> CONFIG_EXT4_FS=y
>>>> CONFIG_EXT4_FS_POSIX_ACL=y
>>>> CONFIG_EXT4_FS_SECURITY=y
>>>> # CONFIG_EXT4_ENCRYPTION is not set
>>>> # CONFIG_EXT4_DEBUG is not set
>>>> CONFIG_JBD2=y
>>>> # CONFIG_JBD2_DEBUG is not set
>>>>
>>>> ext4_fill_super() tells me:
>>>>
>>>> [    3.033174] EXT4-fs (sda5): couldn't mount as ext3 due to feature incompatibilities
>>>> [    3.100186] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
>>>> [    3.102683] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.
>>>>
>>>>
>>>> This is a new install, new filesystem. It has never been ext2 or ext3.
>>>>
>>>> After bootup and before I do anything else, I can remount /dev/sda5 on / as
>>>> rw and everything is OK.
>>>>
>>>> What am I doing wrong?
>>>
>>> Hm you're the second person to report this in a week.  
>>> (see EXT4-fs (mmcblk0p1): couldn't mount as ext3 due to feature incompatibilities)
>>>
>>> It sounds like filesystem probing behavior; it tried to mount as ext3 and failed
>>> (as it should fail) then moved on to ext4 and succeeded.
>>
>> Mostly succeeded, but why did it leave the filesystem as read-only?
> 
> oh it left it there?  Root usually mounts readonly then transitions to RW.
> That didn't eventually happen?

Nope, I had to do that manually and then everything was OK.

>>
>>> I sent:
>>>
>>> [PATCH V2] ext4: don't complain about incorrect features when probing
>>>
>>> which should fix it, but I really wonder why this is just now showing up for 
>>> people; it could be a combination of /etc/filesystems, fstab entries, blkid
>>> handling in mount, etc...
>>>
>>> Is this a new OS/installer release where this is showing up, or did a simple
>>> kernel upgrade trigger it?  If the latter, what kernel version didn't have
>>> the above kernel message?
>>
>> It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
>> rolling updates release.
> 
> Hrmph.  A lot of things go into this behavior, it may not be a kernel change at
> all that has made it show up now...

Yes, it could be that wonderful systemd or something else.

-- 
~Randy

Re: ext4 confusion

[Eric Sandeen]

On 3/5/18 10:42 PM, Randy Dunlap wrote:
>>> It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
>>> rolling updates release.
>> Hrmph.  A lot of things go into this behavior, it may not be a kernel change at
>> all that has made it show up now...
> Yes, it could be that wonderful systemd or something else.

I think I'd pursue a parallel track of bugging SUSE about the issue... ;)

(I don't think the kernel will ever just downgrade an rw mount request to
ro, or skip an ro->rw transition silently... leaving it ro does seem
like an init bug, but *shrug* init long ago transitioned into deep magic.)

-Eric

Re: ext4 confusion

[Randy Dunlap]

On 03/05/2018 08:45 PM, Eric Sandeen wrote:
> On 3/5/18 10:42 PM, Randy Dunlap wrote:
>>>> It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
>>>> rolling updates release.
>>> Hrmph.  A lot of things go into this behavior, it may not be a kernel change at
>>> all that has made it show up now...
>> Yes, it could be that wonderful systemd or something else.
> 
> I think I'd pursue a parallel track of bugging SUSE about the issue... ;)
> 
> (I don't think the kernel will ever just downgrade an rw mount request to
> ro, or skip an ro->rw transition silently... leaving it ro does seem
> like an init bug, but *shrug* init long ago transitioned into deep magic.)

More info:  :(

This problem happens when booting my own custom 4.16-rc3 kernel.
If I boot the OpenSUSE-supplied (4.15.7) kernel, the / fs is remounted rw later on.

So I'm more or less back to "what am I doing wrong"?

thanks,
-- 
~Randy

Re: ext4 confusion

[Dave Chinner]

On Mon, Mar 05, 2018 at 09:12:03PM -0800, Randy Dunlap wrote:
> On 03/05/2018 08:45 PM, Eric Sandeen wrote:
> > On 3/5/18 10:42 PM, Randy Dunlap wrote:
> >>>> It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
> >>>> rolling updates release.
> >>> Hrmph.  A lot of things go into this behavior, it may not be a kernel change at
> >>> all that has made it show up now...
> >> Yes, it could be that wonderful systemd or something else.
> > 
> > I think I'd pursue a parallel track of bugging SUSE about the issue... ;)
> > 
> > (I don't think the kernel will ever just downgrade an rw mount request to
> > ro, or skip an ro->rw transition silently... leaving it ro does seem
> > like an init bug, but *shrug* init long ago transitioned into deep magic.)
> 
> More info:  :(
> 
> This problem happens when booting my own custom 4.16-rc3 kernel.
> If I boot the OpenSUSE-supplied (4.15.7) kernel, the / fs is remounted rw later on.
> 
> So I'm more or less back to "what am I doing wrong"?

The filesystem probing order has probably changed. mount tries to
use blkid to determine the filesytem type to use, and if that
doesn't find a known type it will fall back to trying mounts with
explicit types as per the filesystem type order listed in
/proc/filesystems. (it's in the man mount page) Maybe the device
module hasn't been loaded when blkid runs to probe existing block
devices?

These sorts of whacky behaviours have occurred for me in the past
when either userspace behaviour changed, the order of filesystems
listed in /proc/filesystems or module load order changed. Typically
it's a difference in kernel config that causes such shenanigans.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: ext4 confusion

[Randy Dunlap]

On 03/05/2018 10:47 PM, Dave Chinner wrote:
> On Mon, Mar 05, 2018 at 09:12:03PM -0800, Randy Dunlap wrote:
>> On 03/05/2018 08:45 PM, Eric Sandeen wrote:
>>> On 3/5/18 10:42 PM, Randy Dunlap wrote:
>>>>>> It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
>>>>>> rolling updates release.
>>>>> Hrmph.  A lot of things go into this behavior, it may not be a kernel change at
>>>>> all that has made it show up now...
>>>> Yes, it could be that wonderful systemd or something else.
>>>
>>> I think I'd pursue a parallel track of bugging SUSE about the issue... ;)
>>>
>>> (I don't think the kernel will ever just downgrade an rw mount request to
>>> ro, or skip an ro->rw transition silently... leaving it ro does seem
>>> like an init bug, but *shrug* init long ago transitioned into deep magic.)
>>
>> More info:  :(
>>
>> This problem happens when booting my own custom 4.16-rc3 kernel.
>> If I boot the OpenSUSE-supplied (4.15.7) kernel, the / fs is remounted rw later on.
>>
>> So I'm more or less back to "what am I doing wrong"?
> 
> The filesystem probing order has probably changed. mount tries to
> use blkid to determine the filesytem type to use, and if that
> doesn't find a known type it will fall back to trying mounts with
> explicit types as per the filesystem type order listed in
> /proc/filesystems. (it's in the man mount page) Maybe the device
> module hasn't been loaded when blkid runs to probe existing block
> devices?
> 
> These sorts of whacky behaviours have occurred for me in the past
> when either userspace behaviour changed, the order of filesystems
> listed in /proc/filesystems or module load order changed. Typically
> it's a difference in kernel config that causes such shenanigans.

There is also a (big) difference in the $DISTRO using initramfs and my
custom kernel not using one. But in both cases the SATA/AHCI driver is
loaded/ready before ext4fs, so it's still a mystery to me.

Eric, I tested your patch but it didn't help in my environment.

thanks,

-- 
~Randy

Re: ext4 confusion

[Eric Sandeen]

On 3/6/18 2:07 PM, Randy Dunlap wrote:
> On 03/05/2018 10:47 PM, Dave Chinner wrote:
>> On Mon, Mar 05, 2018 at 09:12:03PM -0800, Randy Dunlap wrote:
>>> On 03/05/2018 08:45 PM, Eric Sandeen wrote:
>>>> On 3/5/18 10:42 PM, Randy Dunlap wrote:
>>>>>>> It's a new OS/installer.  OpenSUSE Tumbleweed, which is their bleeding edge
>>>>>>> rolling updates release.
>>>>>> Hrmph.  A lot of things go into this behavior, it may not be a kernel change at
>>>>>> all that has made it show up now...
>>>>> Yes, it could be that wonderful systemd or something else.
>>>>
>>>> I think I'd pursue a parallel track of bugging SUSE about the issue... ;)
>>>>
>>>> (I don't think the kernel will ever just downgrade an rw mount request to
>>>> ro, or skip an ro->rw transition silently... leaving it ro does seem
>>>> like an init bug, but *shrug* init long ago transitioned into deep magic.)
>>>
>>> More info:  :(
>>>
>>> This problem happens when booting my own custom 4.16-rc3 kernel.
>>> If I boot the OpenSUSE-supplied (4.15.7) kernel, the / fs is remounted rw later on.
>>>
>>> So I'm more or less back to "what am I doing wrong"?
>>
>> The filesystem probing order has probably changed. mount tries to
>> use blkid to determine the filesytem type to use, and if that
>> doesn't find a known type it will fall back to trying mounts with
>> explicit types as per the filesystem type order listed in
>> /proc/filesystems. (it's in the man mount page) Maybe the device
>> module hasn't been loaded when blkid runs to probe existing block
>> devices?
>>
>> These sorts of whacky behaviours have occurred for me in the past
>> when either userspace behaviour changed, the order of filesystems
>> listed in /proc/filesystems or module load order changed. Typically
>> it's a difference in kernel config that causes such shenanigans.
> 
> There is also a (big) difference in the $DISTRO using initramfs and my
> custom kernel not using one. But in both cases the SATA/AHCI driver is
> loaded/ready before ext4fs, so it's still a mystery to me.
> 
> Eric, I tested your patch but it didn't help in my environment.

Odd.  It's supposed to be silent if ext3 doesn't recognize the features,
but ext4 does.  Too many moving parts.  :/

-Eric

Re: ext4 confusion

[Theodore Y. Ts'o]

On Mon, Mar 05, 2018 at 08:17:10PM -0800, Randy Dunlap wrote:
> 
> ext4_fill_super() tells me:
> 
> [    3.033174] EXT4-fs (sda5): couldn't mount as ext3 due to feature incompatibilities
> [    3.100186] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
> [    3.102683] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.
> 
> This is a new install, new filesystem. It has never been ext2 or ext3.
> 
> After bootup and before I do anything else, I can remount /dev/sda5 on / as
> rw and everything is OK.

What is the boot command-line that are you using?  What does
/proc/cmdline say?  Mine says (for example):

% cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-4.15.3-00026-g373ea7d39542 root=/dev/mapper/cwcc-root ro fbcon=font:sun12x22 quiet

The ro mount option is usually what causes the root file system to be
mounted read-only.

I would check and see whether you are using the same init script path
for your custom kernel versus your distro kernel.  In particular, is
the initramfs the same for both?  With Debian, there is an initial
ramdisk which is used:

	linux	/vmlinuz-4.15.3-00026-g373ea7d39542 root=/dev/mapper/cwcc-root ro fbcon=font:sun12x22 quiet
	echo	'Loading initial ramdisk ...'
	initrd	/initrd.img-4.15.3-00026-g373ea7d39542

With the Debian initial ram disk, e2fsck is actually run on the file
system *before* it is mounted, and then it is mounted under userspace
control, and the kernel's default autoprobing isn't used at all.  So
with Debian's initramfs, it *knows* it is an ext4 file system and it
mounts it directly as ext4, so in my kernel logs I just see this:

Mar  5 18:26:11 cwcc kernel: [   15.073579] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)

So I don't see the "VFS: mounting root..." message at all.  I don't
know what your distribution is doing, but you might want to check and
see if the "VFS: mounting root" is showing up when you are booting the
distro-kernel.  This really smells like a problem with how the
initramfs for your custom kernel was set up...

	      	   	  	     - Ted

Re: ext4 confusion

[Randy Dunlap]

On 03/06/2018 02:34 PM, Theodore Y. Ts'o wrote:
> On Mon, Mar 05, 2018 at 08:17:10PM -0800, Randy Dunlap wrote:
>>
>> ext4_fill_super() tells me:
>>
>> [    3.033174] EXT4-fs (sda5): couldn't mount as ext3 due to feature incompatibilities
>> [    3.100186] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
>> [    3.102683] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.
>>
>> This is a new install, new filesystem. It has never been ext2 or ext3.
>>
>> After bootup and before I do anything else, I can remount /dev/sda5 on / as
>> rw and everything is OK.
> 
> What is the boot command-line that are you using?  What does
> /proc/cmdline say?  Mine says (for example):
> 
> % cat /proc/cmdline 
> BOOT_IMAGE=/vmlinuz-4.15.3-00026-g373ea7d39542 root=/dev/mapper/cwcc-root ro fbcon=font:sun12x22 quiet

Mine says:

BOOT_IMAGE=/boot/bzImage-416-3m32 root=/dev/sda5 resume=/dev/sda3 splash=native quiet showopts elevator=cfq selinux=0 apparmor=0 debug initcall_debug ignore_loglevel


> The ro mount option is usually what causes the root file system to be
> mounted read-only.
> 
> I would check and see whether you are using the same init script path
> for your custom kernel versus your distro kernel.  In particular, is
> the initramfs the same for both?  With Debian, there is an initial
> ramdisk which is used:
> 
> 	linux	/vmlinuz-4.15.3-00026-g373ea7d39542 root=/dev/mapper/cwcc-root ro fbcon=font:sun12x22 quiet
> 	echo	'Loading initial ramdisk ...'
> 	initrd	/initrd.img-4.15.3-00026-g373ea7d39542

My custom kernel does not use an initramfs at all.

> With the Debian initial ram disk, e2fsck is actually run on the file
> system *before* it is mounted, and then it is mounted under userspace
> control, and the kernel's default autoprobing isn't used at all.  So
> with Debian's initramfs, it *knows* it is an ext4 file system and it
> mounts it directly as ext4, so in my kernel logs I just see this:
> 
> Mar  5 18:26:11 cwcc kernel: [   15.073579] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
> 
> So I don't see the "VFS: mounting root..." message at all.  I don't
> know what your distribution is doing, but you might want to check and
> see if the "VFS: mounting root" is showing up when you are booting the
> distro-kernel.  This really smells like a problem with how the
> initramfs for your custom kernel was set up...

OK, I don't see the "VFS: mounting root" message with the distro kernel.
With the distro kernel, I see:

[   10.678700] EXT4-fs (sda5): re-mounted. Opts: data=ordered,acl,user_xattr

and with my custom kernel (no initramfs) I see:

[    3.146701] EXT4-fs (sda5): mounted filesystem with ordered data mode. Opts: (null)
[    3.149212] VFS: Mounted root (ext4 filesystem) readonly on device 8:5.


so maybe when I'm not using an initramfs, the fs type probing is confused.

I'll keep looking into it.  Thanks for your time(s).  I'll just continue to remount
it R/W for now.

-- 
~Randy

Re: ext4 confusion

[Theodore Y. Ts'o]

On Tue, Mar 06, 2018 at 03:34:25PM -0800, Randy Dunlap wrote:
> 
> My custom kernel does not use an initramfs at all.

OK, so if you are not using an initramfs at all, it's *normal* for the
file system to be mounted read-only, since it's not safe to run fsck
on the file system unless it is mounted read-only.  So the standard
init scripts (at least in the good old days pre-systemd) expected that
you use the ro option to make sure the root file system is mounted
read-only, and then the init scripts would check the file system if it
was needed, and then the init scripts would remount the file sytsem
read-write afterwards.

As I mentioned, with many modern distro's this is now done in
initramfs, and with Debian it will actually run the fsck *before* the
root file system is even mounted, and then it can just mount it
read/write.  Obviously, if that is what initramfs does, then the init
scripts (or the magic systemd units in the brave new systemd world)
don't need to remount the file system read/write.

To be honest, there is a huge amount of magic these days in the
initramfs and systemd scripts.  I do know that Debian stable supports
using a read-only mount and its systemd setup does the right thing,
because that's how kvm-xfstests works.  But with some of the other
distro's (Red Hat Enterprse Linux especially), it's been magic, and
trying to figure out how it works is something I gave up on a long
time ago.  I was just amazed that it managed to boot over fiber
channel, and I was glad I never had to debug on the freaking thing
worked.  :-/

						- Ted

Re: ext4 confusion

[Randy Dunlap]

On 03/06/2018 08:58 PM, Theodore Y. Ts'o wrote:
> On Tue, Mar 06, 2018 at 03:34:25PM -0800, Randy Dunlap wrote:
>>
>> My custom kernel does not use an initramfs at all.
> 
> OK, so if you are not using an initramfs at all, it's *normal* for the
> file system to be mounted read-only, since it's not safe to run fsck
> on the file system unless it is mounted read-only.  So the standard
> init scripts (at least in the good old days pre-systemd) expected that
> you use the ro option to make sure the root file system is mounted
> read-only, and then the init scripts would check the file system if it
> was needed, and then the init scripts would remount the file sytsem
> read-write afterwards.
> 
> As I mentioned, with many modern distro's this is now done in
> initramfs, and with Debian it will actually run the fsck *before* the
> root file system is even mounted, and then it can just mount it
> read/write.  Obviously, if that is what initramfs does, then the init
> scripts (or the magic systemd units in the brave new systemd world)
> don't need to remount the file system read/write.
> 
> To be honest, there is a huge amount of magic these days in the
> initramfs and systemd scripts.  I do know that Debian stable supports
> using a read-only mount and its systemd setup does the right thing,
> because that's how kvm-xfstests works.  But with some of the other
> distro's (Red Hat Enterprse Linux especially), it's been magic, and
> trying to figure out how it works is something I gave up on a long
> time ago.  I was just amazed that it managed to boot over fiber
> channel, and I was glad I never had to debug on the freaking thing
> worked.  :-/

OK.  Thanks, Ted.

-- 
~Randy