ext4: spurious "orphan cleanup on readonly fs" due to remount,ro failing to reset orphan_file structures

ext4: spurious "orphan cleanup on readonly fs" due to remount,ro failing to reset orphan_file structures

[Tigran Aivazian]

Hello,

I have been experimenting with optimising rootfs for custom builds of
Ubuntu 26 with various options like fast_commit, sparse_super2,
orphan_file, inline_data, metadata_csum_seed, etc. And I noticed
something strange: if my root filesystem is made with "orphan_file"
feature then on every reboot I get this message:

EXT4-fs (sda): orphan cleanup on readonly fs

(tested with SSD, NVMe and even old HDD -- because initially I thought
that maybe it was NVMe controller failing to flush its volatile DRAM
cache to the NAND flash, but no, it wasn't the reason)

I think what happens here is that when a filesystem with the
"orphan_file" feature is fully unmounted, ext4_put_super()
successfully clears the "orphan_present" superblock flag AND properly
collapses/resets the physical orphan file headers. However, during a
read-only remount (which systemd must do for / in order to halt the
system) ext4_reconfigure() clears the "orphan_present" superblock flag
but skips resetting the physical orphan file structures. Consequently,
on the next boot (which starts with a read-only mount of /),
ext4_fill_super() observes that while the superblock flag is clear,
the orphan file itself appears non-empty. This unconditionally
triggers ext4_orphan_cleanup(), which prints the spurious warning,
scans the file, finds 0 orphans, and completes silently.

Let's test this theory on the kernel 7.0.0-28-generic of Ubuntu 26:

# 1. Create a filesystem with the orphan_file feature
mkfs.ext4 -O orphan_file /dev/sda

# 2. Mount read-write
mount -o rw /dev/sda /mnt

# 3. Remount read-only (Simulating systemd shutdown)
# This clears the orphan_present flag in the superblock, but leaves
the orphan file structures intact.
mount -o remount,ro /mnt

# 4. Unmount and mount read-only (Simulating the next boot)
umount /mnt
mount -o ro /dev/sda /mnt

Expected result in dmesg: nothing (well, except the usual
mount/remount messages about ordered data mode, etc).

Actual result in dmesg:

EXT4-fs (sda): orphan cleanup on readonly fs

Since this affects the default shutdown path for any modern Linux
distribution using systemd and ext4 with the "orphan_file" feature, it
would be great to get the remount,ro teardown path aligned with the
full unmount teardown path in this aspect.

Kind regards,
Tigran

Re: ext4: spurious "orphan cleanup on readonly fs" due to remount,ro failing to reset orphan_file structures

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 2689 bytes --]

[Cc'ing ext4 folks with full reply context]

On Sun, Aug 02, 2026 at 11:18:39PM +0100, Tigran Aivazian wrote:
> Hello,
> 
> I have been experimenting with optimising rootfs for custom builds of
> Ubuntu 26 with various options like fast_commit, sparse_super2,
> orphan_file, inline_data, metadata_csum_seed, etc. And I noticed
> something strange: if my root filesystem is made with "orphan_file"
> feature then on every reboot I get this message:
> 
> EXT4-fs (sda): orphan cleanup on readonly fs
> 
> (tested with SSD, NVMe and even old HDD -- because initially I thought
> that maybe it was NVMe controller failing to flush its volatile DRAM
> cache to the NAND flash, but no, it wasn't the reason)
> 
> I think what happens here is that when a filesystem with the
> "orphan_file" feature is fully unmounted, ext4_put_super()
> successfully clears the "orphan_present" superblock flag AND properly
> collapses/resets the physical orphan file headers. However, during a
> read-only remount (which systemd must do for / in order to halt the
> system) ext4_reconfigure() clears the "orphan_present" superblock flag
> but skips resetting the physical orphan file structures. Consequently,
> on the next boot (which starts with a read-only mount of /),
> ext4_fill_super() observes that while the superblock flag is clear,
> the orphan file itself appears non-empty. This unconditionally
> triggers ext4_orphan_cleanup(), which prints the spurious warning,
> scans the file, finds 0 orphans, and completes silently.
> 
> Let's test this theory on the kernel 7.0.0-28-generic of Ubuntu 26:

Can you also confirm this on latest mainline (7.2-rc6)?

> 
> # 1. Create a filesystem with the orphan_file feature
> mkfs.ext4 -O orphan_file /dev/sda
> 
> # 2. Mount read-write
> mount -o rw /dev/sda /mnt
> 
> # 3. Remount read-only (Simulating systemd shutdown)
> # This clears the orphan_present flag in the superblock, but leaves
> the orphan file structures intact.
> mount -o remount,ro /mnt
> 
> # 4. Unmount and mount read-only (Simulating the next boot)
> umount /mnt
> mount -o ro /dev/sda /mnt
> 
> Expected result in dmesg: nothing (well, except the usual
> mount/remount messages about ordered data mode, etc).
> 
> Actual result in dmesg:
> 
> EXT4-fs (sda): orphan cleanup on readonly fs
> 
> Since this affects the default shutdown path for any modern Linux
> distribution using systemd and ext4 with the "orphan_file" feature, it
> would be great to get the remount,ro teardown path aligned with the
> full unmount teardown path in this aspect.
> 

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: ext4: spurious "orphan cleanup on readonly fs" due to remount,ro failing to reset orphan_file structures

[Jan Kara]

On Mon 03-08-26 07:28:30, Bagas Sanjaya wrote:
> [Cc'ing ext4 folks with full reply context]

Thanks for CC.
 
> On Sun, Aug 02, 2026 at 11:18:39PM +0100, Tigran Aivazian wrote:
> > Hello,
> > 
> > I have been experimenting with optimising rootfs for custom builds of
> > Ubuntu 26 with various options like fast_commit, sparse_super2,
> > orphan_file, inline_data, metadata_csum_seed, etc. And I noticed
> > something strange: if my root filesystem is made with "orphan_file"
> > feature then on every reboot I get this message:
> > 
> > EXT4-fs (sda): orphan cleanup on readonly fs
> > 
> > (tested with SSD, NVMe and even old HDD -- because initially I thought
> > that maybe it was NVMe controller failing to flush its volatile DRAM
> > cache to the NAND flash, but no, it wasn't the reason)
> > 
> > I think what happens here is that when a filesystem with the
> > "orphan_file" feature is fully unmounted, ext4_put_super()
> > successfully clears the "orphan_present" superblock flag AND properly
> > collapses/resets the physical orphan file headers. However, during a
> > read-only remount (which systemd must do for / in order to halt the
> > system) ext4_reconfigure() clears the "orphan_present" superblock flag
> > but skips resetting the physical orphan file structures. Consequently,
> > on the next boot (which starts with a read-only mount of /),
> > ext4_fill_super() observes that while the superblock flag is clear,
> > the orphan file itself appears non-empty. This unconditionally
> > triggers ext4_orphan_cleanup(), which prints the spurious warning,
> > scans the file, finds 0 orphans, and completes silently.
> > 
> > Let's test this theory on the kernel 7.0.0-28-generic of Ubuntu 26:
> 
> Can you also confirm this on latest mainline (7.2-rc6)?

No need to, the behavior didn't change for a long time. In fact, any
read-only mount with orphan_file feature enabled will result in the
spurious "orphan cleanup on readonly fs" message. I've just checked that.
It is only a cosmetic problem but nobody noticed so far.

> > # 1. Create a filesystem with the orphan_file feature
> > mkfs.ext4 -O orphan_file /dev/sda
> > 
> > # 2. Mount read-write
> > mount -o rw /dev/sda /mnt
> > 
> > # 3. Remount read-only (Simulating systemd shutdown)
> > # This clears the orphan_present flag in the superblock, but leaves
> > the orphan file structures intact.
> > mount -o remount,ro /mnt
> > 
> > # 4. Unmount and mount read-only (Simulating the next boot)
> > umount /mnt
> > mount -o ro /dev/sda /mnt
> > 
> > Expected result in dmesg: nothing (well, except the usual
> > mount/remount messages about ordered data mode, etc).
> > 
> > Actual result in dmesg:
> > 
> > EXT4-fs (sda): orphan cleanup on readonly fs
> > 
> > Since this affects the default shutdown path for any modern Linux
> > distribution using systemd and ext4 with the "orphan_file" feature, it
> > would be great to get the remount,ro teardown path aligned with the
> > full unmount teardown path in this aspect.

The problem really isn't that the remount,ro path would be doing something
differently. Even after standard unmount I get the spurious message if I
mount the fs read only. What we need is to improve ext4_orphan_cleanup() to
really skip orphan recovery if there are no orphans. It will just require
some tweaks to the mount path.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: ext4: spurious "orphan cleanup on readonly fs" due to remount,ro failing to reset orphan_file structures

[Tigran Aivazian]

Hi,

On Mon, 3 Aug 2026 at 13:36, Jan Kara <jack@suse.cz> wrote:
> No need to, the behavior didn't change for a long time. In fact, any
> read-only mount with orphan_file feature enabled will result in the
> spurious "orphan cleanup on readonly fs" message. I've just checked that.
> It is only a cosmetic problem but nobody noticed so far.

You say it is a cosmetic problem, but here I somewhat disagree.
Logically, seeing "orphan cleanup on readonly fs" implies an unclean
shutdown that caused those orphans. Consequently, I spent time
debugging it by instrumenting scripts in
/usr/lib/systemd/system-shutdown/ and even suspecting the NVMe
controller caches, etc. Any sysadmin who doesn't like leaving
"mysteries" on their Linux host would have done the same.

Kind regards,
Tigran

Re: ext4: spurious "orphan cleanup on readonly fs" due to remount,ro failing to reset orphan_file structures

[Jan Kara]

On Mon 03-08-26 13:57:55, Tigran Aivazian wrote:
> Hi,
> 
> On Mon, 3 Aug 2026 at 13:36, Jan Kara <jack@suse.cz> wrote:
> > No need to, the behavior didn't change for a long time. In fact, any
> > read-only mount with orphan_file feature enabled will result in the
> > spurious "orphan cleanup on readonly fs" message. I've just checked that.
> > It is only a cosmetic problem but nobody noticed so far.
> 
> You say it is a cosmetic problem, but here I somewhat disagree.
> Logically, seeing "orphan cleanup on readonly fs" implies an unclean
> shutdown that caused those orphans. Consequently, I spent time
> debugging it by instrumenting scripts in
> /usr/lib/systemd/system-shutdown/ and even suspecting the NVMe
> controller caches, etc. Any sysadmin who doesn't like leaving
> "mysteries" on their Linux host would have done the same.

Fair point and to be clear I think this needs to fixed. I can see how this
"cosmetic problem" misled you and you have wasted time debugging that. I'm
sorry for that.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR