Modification of block device by R/O mount

Modification of block device by R/O mount

[Johannes Bauer]

Dear list,

I'm a little bit puzzled by behavior of ext4 I've been seeing initially 
on aarch64 Linux 6.1 but can reproduce easily on my machine:

Linux reliant 6.5.0-28-generic #29-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 28 
23:46:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

The behavior I'm seeing is that a R/O mount modifies a device mapper 
block device (or loopback device), which is unsettling. That change is 
not propagated back to the original source, but it is causing massive 
problems nevertheless. For example:

reliant [/tmp/ext4test]: dd if=/dev/null bs=1M seek=64 count=0 of=image.img
0+0 records in
0+0 records out
0 bytes copied, 2,2172e-05 s, 0,0 kB/s


reliant [/tmp/ext4test]: mkfs.ext4 image.img
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 16384 4k blocks and 16384 inodes

Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done



reliant [/tmp/ext4test]: losetup --read-only --show -f image.img
/dev/loop28


reliant [/tmp/ext4test]: md5sum /dev/loop28 image.img
34d7cd8eb4abb1943aabe078b8fb3c74  /dev/loop28
34d7cd8eb4abb1943aabe078b8fb3c74  image.img


reliant [/tmp/ext4test]: mkdir mnt; mount -o ro /dev/loop28 mnt


reliant [/tmp/ext4test]: md5sum /dev/loop28 image.img
9145654c1e6a5855c1db239815a05198  /dev/loop28
34d7cd8eb4abb1943aabe078b8fb3c74  image.img


reliant [/tmp/ext4test]: cmp /dev/loop28 image.img
/dev/loop28 image.img differ: byte 61484, line 5


reliant [/tmp/ext4test]: umount mnt; losetup -d /dev/loop28


reliant [/tmp/ext4test]: md5sum image.img
34d7cd8eb4abb1943aabe078b8fb3c74  image.img


As you can see, the original image was never modified (MD5 34d7... 
remains the same) but /dev/loop28 changes once R/O mounted. This 
behavior is shocking to me and I've not observed this with other file 
systems (I tried btrfs) so this seems to be ext4-specific.

Is this expected behavior? Is there a way to mitigate it?

Thanks and best regards,
Johannes

-- 
"A PC without Windows is like a chocolate cake without mustard."

Re: Modification of block device by R/O mount

[Dave Chinner]

On Wed, Jul 31, 2024 at 01:16:26PM +0200, Johannes Bauer wrote:
> Is this expected behavior?

Yes. A read-only mount doesn't mean the filesystem cannot write to
the block device. All it means is that users cannot make
modifications to the filesystem contents once the filesystem is
mounted.

The filesystem may still have to do things like journal recovery on
a read-only mount which requires writing to the block device, or
maybe other house-keeping things that happen at mount time which
require writing updates to the superblock or other metadata.

This is not ext4 specific. XFS behaves the same way on read-only
mounts and, IIRC, JFS, Reiser and most other journalling filesystems
will also behave the same way.

> Is there a way to mitigate it?

If you want to stop the filesystem writing to the block device, you
have to set the -block device- to be read only. At this point, the
filesystem will refuse to mount if it needs to write to the block
device during mount.

Hence you may need to use "-ro,norecovery" on journalled filesystems
when the block device is read only to get them to mount. However,
this can expose inconsistent metadata to userspace and weird stuff
can happen because the journal will not have been recovered...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Modification of block device by R/O mount

[Johannes Bauer]

Hey Dave,

thanks for your response!

Am 01.08.24 um 03:53 schrieb Dave Chinner:

>> Is there a way to mitigate it?
> 
> If you want to stop the filesystem writing to the block device, you
> have to set the -block device- to be read only. At this point, the
> filesystem will refuse to mount if it needs to write to the block
> device during mount.

But my point is, that is what I am doing -- creating the losetup mapping 
R/O:

# losetup --read-only --show -f image.img
/dev/loop35

# echo foo >/dev/loop35
bash: echo: write error: Operation not permitted

I.e., the block device is write protected and *yet* it changes content. 
This is what I find so extremely puzzling, that the file system should 
not have the capability to change the underlying block device, yet it does.

Cheers,
Johannes

-- 
"A PC without Windows is like a chocolate cake without mustard."

Re: Modification of block device by R/O mount

[Johannes Bauer]

Am 01.08.24 um 08:18 schrieb Johannes Bauer:

> But my point is, that is what I am doing -- creating the losetup mapping 
> R/O:
> 
> # losetup --read-only --show -f image.img
> /dev/loop35
> 
> # echo foo >/dev/loop35
> bash: echo: write error: Operation not permitted
> 
> I.e., the block device is write protected and *yet* it changes content. 
> This is what I find so extremely puzzling, that the file system should 
> not have the capability to change the underlying block device, yet it does.

To expand on this, it also happens when I create a dmsetup linear 
mapping that has been explicitly marked as read-only:

# dmsetup create --concise "linear,,2,ro,0 131072 linear /dev/loop28 0"

# md5sum image.img /dev/loop28 /dev/mapper/linear
56f56801923108d241947024926fea53  image.img
56f56801923108d241947024926fea53  /dev/loop28
56f56801923108d241947024926fea53  /dev/mapper/linear

# mount -o ro /dev/mapper/linear mnt

# md5sum image.img /dev/loop28 /dev/mapper/linear
56f56801923108d241947024926fea53  image.img
56f56801923108d241947024926fea53  /dev/loop28
d804867eab0106f9659c02a6add2f5da  /dev/mapper/linear

It is my understanding that while mounting (even -o ro) might modify the 
block device, a R/O block device should never be modifiable my anything. 
This does not seem to be the case?

Best regards,
Johannes

-- 
"A PC without Windows is like a chocolate cake without mustard."

Re: Modification of block device by R/O mount

[Dave Chinner]

On Thu, Aug 01, 2024 at 08:32:18AM +0200, Johannes Bauer wrote:
> Am 01.08.24 um 08:18 schrieb Johannes Bauer:
> 
> > But my point is, that is what I am doing -- creating the losetup mapping
> > R/O:
> > 
> > # losetup --read-only --show -f image.img
> > /dev/loop35
> > 
> > # echo foo >/dev/loop35
> > bash: echo: write error: Operation not permitted

Oh, I missed that critical detail buried in the example code.  Most
people who report "writes occuring on RO mounts" have no clue that
writes are actually allowed on RO mounts, and you made no mention
that you were also setting the block device read only and that's why
you were expecting writes to fail....

As it is, I don't think the filesystem is actually writing anything
to disk. There's a couple of individual metadata fields that have
been changed in the block device page cache in some metadata at
offset 0x8000:

$ diff -u img.t bdev.t
--- img.t	2024-08-09 11:41:20.358217508 +1000
+++ bdev.t	2024-08-09 11:41:29.966289011 +1000
@@ -103,10 +103,11 @@
 007ff0 0000 0000 0000 0000 000c de00 0eb9 b0b0
 008000 3bc0 9839 0000 0400 0000 0000 0000 0004
 008010 0000 0004 0000 0100 0000 0100 0000 0000
-008020 0000 0000 0000 0000 0000 0000 0000 0000
+008020 0000 0000 0000 0000 0000 1200 0000 0000
 008030 9f03 d923 77f8 9748 64af 1176 1b28 9959
 008040 0000 0100 0000 0000 0000 0000 0000 0000
-008050 0000 0000 0000 0000 0000 0000 0000 0000
+008050 0004 0000 0000 0000 0000 0000 0000 0000
+008060 0000 0000 0000 0000 0000 0000 0000 0000
 *
 008800 0fff 0000 0000 0000 0000 0000 0000 0000
 008810 0000 0000 0000 0000 0000 0000 0000 0000

This does not mean a write was done, however. The filesystem may
have changed metadata directly in the page cache so that it is
correct for a RO mount on a RO block device without actually writing
anything to disk. There's nothing wrong with doing this - it isn't a
bug at all even though such modifications will be visible to
userspace via block device page cache reads.

You'll need someone who actually knows what metadata ext4 needs to
modify directly on read-only mounts to tell you exactly what the
mount is modifying and whether or not it should actually be doing
that or not. However, I personally wouldn't consider this behaviour
a bug if it necessary to allow read-only mounts on read-only block
devices to work reliably...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Modification of block device by R/O mount

[Christoph Hellwig]

On Fri, Aug 09, 2024 at 11:50:30AM +1000, Dave Chinner wrote:
> You'll need someone who actually knows what metadata ext4 needs to
> modify directly on read-only mounts to tell you exactly what the
> mount is modifying and whether or not it should actually be doing
> that or not. However, I personally wouldn't consider this behaviour
> a bug if it necessary to allow read-only mounts on read-only block
> devices to work reliably...

Modifying page cache without dirtying is in general a bug.  On a
block device we'll mostly get away with it, but it still is asking
for trouble later.  So this probably still warrants fixing, but as
Dave said it isn't exactly a grave bug.