ext4 metadata corruption - snapshot related?

ext4 metadata corruption - snapshot related?

[Jean-Louis Dupond]

Hi,

We have around 200 VM's running on qemu (on a AlmaLinux 9 based hypervisor).
All those VM's are migrated from physical machines recently.

But when we enable backups on those VM's (which triggers snapshots), we 
notice some weird/random ext4 corruption within the VM itself.
The VM itself runs CloudLinux 8 (4.18.0-553.40.1.lve.el8.x86_64 kernel).

This are some examples of corruption we see:
1)
kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1036: inode 
#19280823: comm lsphp: Directory block failed checksum
kernel: EXT4-fs error (device sdc1): ext4_empty_dir:2801: inode 
#19280823: comm lsphp: Directory block failed checksum
kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1036: inode 
#19280820: comm lsphp: Directory block failed checksum
kernel: EXT4-fs error (device sdc1): ext4_empty_dir:2801: inode 
#19280820: comm lsphp: Directory block failed checksum

2)
kernel: EXT4-fs error (device sdc1): ext4_lookup:1645: inode #49419787: 
comm lsphp: deleted inode referenced: 49422454
kernel: EXT4-fs error (device sdc1): ext4_lookup:1645: inode #49419787: 
comm lsphp: deleted inode referenced: 49422454
kernel: EXT4-fs error (device sdc1): ext4_lookup:1645: inode #49419787: 
comm lsphp: deleted inode referenced: 49422454

3)
kernel: EXT4-fs error (device sdb1): ext4_validate_block_bitmap:384: 
comm kworker/u240:3: bg 308: bad block bitmap checksum
kernel: EXT4-fs (sdb1): Delayed block allocation failed for inode 
2513946 at logical offset 2 with max blocks 1 with error 74
kernel: EXT4-fs (sdb1): This should not happen!! Data will be lost
kernel: EXT4-fs (sdb1): Inode 2513946 (00000000265d63ca): 
i_reserved_data_blocks (1) not cleared!
kernel: EXT4-fs (sdb1): error count since last fsck: 1
kernel: EXT4-fs (sdb1): initial error at time 1747923211: 
ext4_validate_block_bitmap:384
kernel: EXT4-fs (sdb1): last error at time 1747923211: 
ext4_validate_block_bitmap:384
kernel: EXT4-fs (sdb1): error count since last fsck: 1
kernel: EXT4-fs (sdb1): initial error at time 1747923211: 
ext4_validate_block_bitmap:384
kernel: EXT4-fs (sdb1): last error at time 1747923211: 
ext4_validate_block_bitmap:384

4)
kernel: EXT4-fs (sdc1): error count since last fsck: 4
kernel: EXT4-fs (sdc1): initial error at time 1746616017: 
ext4_validate_block_bitmap:384
kernel: EXT4-fs (sdc1): last error at time 1746621676: 
ext4_mb_generate_buddy:808


Now as a test we upgraded to some newer (backported) kernel, more 
specificly: 5.14.0-284.1101
And after doing some backups again, we had another error:

kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: inode 
#34752060: comm tar: Directory block failed checksum
kernel: EXT4-fs warning (device sdc1): ext4_dirblock_csum_verify:405: 
inode #34752232: comm tar: No space for directory leaf checksum. Please 
run e2fsck -D.
kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: inode 
#34752232: comm tar: Directory block failed checksum
kernel: EXT4-fs warning (device sdc1): ext4_dirblock_csum_verify:405: 
inode #34752064: comm tar: No space for directory leaf checksum. Please 
run e2fsck -D.
kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: inode 
#34752064: comm tar: Directory block failed checksum
kernel: EXT4-fs warning (device sdc1): ext4_dirblock_csum_verify:405: 
inode #34752167: comm tar: No space for directory leaf checksum. Please 
run e2fsck -D.
kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: inode 
#34752167: comm tar: Directory block failed checksum


So now we are wondering what could cause this corruption here.
- We have more VM's on the same kind of setup, without seeing any 
corruption. The only difference there is that the VM's are running 
Debian, have smaller disks and not doing quota.
- If we disable backups/snapshots, no corruption is observed
- Even if we disable the qemu-guest-agent (so no fsfreeze is executed), 
the corruption still occurs

We (for now at least) only see the corruption on filesystems where quota 
is enabled (both usrjquota and usrquota).
The filesystems are between 600GB and 2TB.
And today I noticed (as the filesystems are resized during setup), the 
journal size is only 64M (could this potentially be an issue?).

The big question in the whole story here is, could it be an in-guest 
(ext4?) bug/issue? Or do we really need to look into the layer below 
(aka qemu/hypervisor).
Or if somebody has other idea's, feel free to share! Also additional 
things that could help to troubleshoot the issue.

Thanks
Jean-Louis

Re: ext4 metadata corruption - snapshot related?

[Jean-Louis Dupond]

We updated a machine to a newer 6.15.2-1.el8.elrepo.x86_64 kernel, and 
the same? bug reoccurred after some time:

The error was the following:
Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): 
ext4_lookup:1791: inode #44962812: comm imap: deleted inode referenced: 
44997932
Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): 
ext4_lookup:1791: inode #44962812: comm imap: deleted inode referenced: 
44997932
Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): 
ext4_lookup:1791: inode #44962812: comm imap: deleted inode referenced: 
44997932
Jul 02 11:04:03 xxxxx kernel: EXT4-fs error (device sdd1): 
ext4_lookup:1791: inode #44962812: comm imap: deleted inode referenced: 
44997932

Any idea's on how this could be debugged further?

Thanks
Jean-Louis

On 12/06/2025 16:43, Jean-Louis Dupond wrote:
> Hi,
>
> We have around 200 VM's running on qemu (on a AlmaLinux 9 based 
> hypervisor).
> All those VM's are migrated from physical machines recently.
>
> But when we enable backups on those VM's (which triggers snapshots), 
> we notice some weird/random ext4 corruption within the VM itself.
> The VM itself runs CloudLinux 8 (4.18.0-553.40.1.lve.el8.x86_64 kernel).
>
> This are some examples of corruption we see:
> 1)
> kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1036: 
> inode #19280823: comm lsphp: Directory block failed checksum
> kernel: EXT4-fs error (device sdc1): ext4_empty_dir:2801: inode 
> #19280823: comm lsphp: Directory block failed checksum
> kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1036: 
> inode #19280820: comm lsphp: Directory block failed checksum
> kernel: EXT4-fs error (device sdc1): ext4_empty_dir:2801: inode 
> #19280820: comm lsphp: Directory block failed checksum
>
> 2)
> kernel: EXT4-fs error (device sdc1): ext4_lookup:1645: inode 
> #49419787: comm lsphp: deleted inode referenced: 49422454
> kernel: EXT4-fs error (device sdc1): ext4_lookup:1645: inode 
> #49419787: comm lsphp: deleted inode referenced: 49422454
> kernel: EXT4-fs error (device sdc1): ext4_lookup:1645: inode 
> #49419787: comm lsphp: deleted inode referenced: 49422454
>
> 3)
> kernel: EXT4-fs error (device sdb1): ext4_validate_block_bitmap:384: 
> comm kworker/u240:3: bg 308: bad block bitmap checksum
> kernel: EXT4-fs (sdb1): Delayed block allocation failed for inode 
> 2513946 at logical offset 2 with max blocks 1 with error 74
> kernel: EXT4-fs (sdb1): This should not happen!! Data will be lost
> kernel: EXT4-fs (sdb1): Inode 2513946 (00000000265d63ca): 
> i_reserved_data_blocks (1) not cleared!
> kernel: EXT4-fs (sdb1): error count since last fsck: 1
> kernel: EXT4-fs (sdb1): initial error at time 1747923211: 
> ext4_validate_block_bitmap:384
> kernel: EXT4-fs (sdb1): last error at time 1747923211: 
> ext4_validate_block_bitmap:384
> kernel: EXT4-fs (sdb1): error count since last fsck: 1
> kernel: EXT4-fs (sdb1): initial error at time 1747923211: 
> ext4_validate_block_bitmap:384
> kernel: EXT4-fs (sdb1): last error at time 1747923211: 
> ext4_validate_block_bitmap:384
>
> 4)
> kernel: EXT4-fs (sdc1): error count since last fsck: 4
> kernel: EXT4-fs (sdc1): initial error at time 1746616017: 
> ext4_validate_block_bitmap:384
> kernel: EXT4-fs (sdc1): last error at time 1746621676: 
> ext4_mb_generate_buddy:808
>
>
> Now as a test we upgraded to some newer (backported) kernel, more 
> specificly: 5.14.0-284.1101
> And after doing some backups again, we had another error:
>
> kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: 
> inode #34752060: comm tar: Directory block failed checksum
> kernel: EXT4-fs warning (device sdc1): ext4_dirblock_csum_verify:405: 
> inode #34752232: comm tar: No space for directory leaf checksum. 
> Please run e2fsck -D.
> kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: 
> inode #34752232: comm tar: Directory block failed checksum
> kernel: EXT4-fs warning (device sdc1): ext4_dirblock_csum_verify:405: 
> inode #34752064: comm tar: No space for directory leaf checksum. 
> Please run e2fsck -D.
> kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: 
> inode #34752064: comm tar: Directory block failed checksum
> kernel: EXT4-fs warning (device sdc1): ext4_dirblock_csum_verify:405: 
> inode #34752167: comm tar: No space for directory leaf checksum. 
> Please run e2fsck -D.
> kernel: EXT4-fs error (device sdc1): htree_dirblock_to_tree:1073: 
> inode #34752167: comm tar: Directory block failed checksum
>
>
> So now we are wondering what could cause this corruption here.
> - We have more VM's on the same kind of setup, without seeing any 
> corruption. The only difference there is that the VM's are running 
> Debian, have smaller disks and not doing quota.
> - If we disable backups/snapshots, no corruption is observed
> - Even if we disable the qemu-guest-agent (so no fsfreeze is 
> executed), the corruption still occurs
>
> We (for now at least) only see the corruption on filesystems where 
> quota is enabled (both usrjquota and usrquota).
> The filesystems are between 600GB and 2TB.
> And today I noticed (as the filesystems are resized during setup), the 
> journal size is only 64M (could this potentially be an issue?).
>
> The big question in the whole story here is, could it be an in-guest 
> (ext4?) bug/issue? Or do we really need to look into the layer below 
> (aka qemu/hypervisor).
> Or if somebody has other idea's, feel free to share! Also additional 
> things that could help to troubleshoot the issue.
>
> Thanks
> Jean-Louis

Re: ext4 metadata corruption - snapshot related?

[Theodore Ts'o]

On Wed, Jul 02, 2025 at 03:43:25PM +0200, Jean-Louis Dupond wrote:
> We updated a machine to a newer 6.15.2-1.el8.elrepo.x86_64 kernel, and the
> same? bug reoccurred after some time:
> 
> The error was the following:
> Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
> inode #44962812: comm imap: deleted inode referenced: 44997932
> Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
> inode #44962812: comm imap: deleted inode referenced: 44997932
> Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
> inode #44962812: comm imap: deleted inode referenced: 44997932
> Jul 02 11:04:03 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
> inode #44962812: comm imap: deleted inode referenced: 44997932
> 
> Any idea's on how this could be debugged further?

If it's correlated to snapshots, then I'd certainly be trying to
looking at potential bugs on the hypervisor.  We've also had a bug
where people were trying to look at bugs on the guest kernel, but the
bug ended up being root caused to a bug on the host kernel.

If moving from 4.18 Cloudlinux 8 kernel to a 6.15.2 RHEL8 kernel shows
the same problem, then it does suggest that the problem isn't with the
guest kernel, but rather in the part of the setup which didn't change
(e.g., the host kernel and hypervisor).

Without a whole lot more details about what your workload might be,
what the host OS software might be, etc., it's really hard to make any
further suggestions.  Are you running this on some kind of cloud
infrastructure (e.g., Microsoft Azure, Amazon AWS, Google Cloud, etc?
Something else?  Have you tried running your workload on some kind of
alternate infrastructure and see if the problem gets solved if you use
a different Cloud provider?

						- Ted

Re: ext4 metadata corruption - snapshot related?

[Jean-Louis Dupond]

On 2/07/2025 16:37, Theodore Ts'o wrote:
> On Wed, Jul 02, 2025 at 03:43:25PM +0200, Jean-Louis Dupond wrote:
>> We updated a machine to a newer 6.15.2-1.el8.elrepo.x86_64 kernel, and the
>> same? bug reoccurred after some time:
>>
>> The error was the following:
>> Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
>> inode #44962812: comm imap: deleted inode referenced: 44997932
>> Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
>> inode #44962812: comm imap: deleted inode referenced: 44997932
>> Jul 02 11:03:35 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
>> inode #44962812: comm imap: deleted inode referenced: 44997932
>> Jul 02 11:04:03 xxxxx kernel: EXT4-fs error (device sdd1): ext4_lookup:1791:
>> inode #44962812: comm imap: deleted inode referenced: 44997932
>>
>> Any idea's on how this could be debugged further?
> If it's correlated to snapshots, then I'd certainly be trying to
> looking at potential bugs on the hypervisor.  We've also had a bug
> where people were trying to look at bugs on the guest kernel, but the
> bug ended up being root caused to a bug on the host kernel.
That is something we are surely investigating.
But for some reason we only see the bug on our CloudLinux machines, not 
on Debian machines (where we have 20 times more of them on the same 
platform).
The fact that it only happens when running snapshots could also be 
related to the small freeze during snapshotting and then causing some 
race somewhere when IO is flushed after the freeze.
>
> If moving from 4.18 Cloudlinux 8 kernel to a 6.15.2 RHEL8 kernel shows
> the same problem, then it does suggest that the problem isn't with the
> guest kernel, but rather in the part of the setup which didn't change
> (e.g., the host kernel and hypervisor).
Well it still shows 'corruption', but the message sometimes differs.
For example previously we also had:
htree_dirblock_to_tree:1036: inode #19280823: comm lsphp: Directory 
block failed checksum

error message etc.

The strange thing here is we only observe ext4 metadata corruption.
For now we didn't had (or see) any corruption within files.
So if it would be a hypervisor issue for example, I would suspect random 
corruption and not only metadata corruption.
>
> Without a whole lot more details about what your workload might be,
> what the host OS software might be, etc., it's really hard to make any
> further suggestions.  Are you running this on some kind of cloud
> infrastructure (e.g., Microsoft Azure, Amazon AWS, Google Cloud, etc?
> Something else?  Have you tried running your workload on some kind of
> alternate infrastructure and see if the problem gets solved if you use
> a different Cloud provider?
The workload is a typical all-in-one webserver with mail and db.
So MySQL/PHP/Apache/Postfix/...
This runs on CloudLinux 8 with the LVE Kernel module (which could be a 
cause also of course).

We are running those VM's on our in-house platform based on Qemu + Libvirt.
The hypervisors are running AlmaLinux 9 with Qemu 9.1.0.

>
> 						- Ted
Thanks for having a look!
Jean-Louis