[Bug 202809] New: ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[Bug 202809] New: ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

            Bug ID: 202809
           Summary: ext4: ext4_xattr_ibody_get:591: comm systemd-journal:
                    corrupted in-inode xattr
           Product: File System
           Version: 2.5
    Kernel Version: 4.19.23-19
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: ext4
          Assignee: fs_ext4@kernel-bugs.osdl.org
          Reporter: feng.tang@intel.com
        Regression: No

Created attachment 281569
  --> https://bugzilla.kernel.org/attachment.cgi?id=281569&action=edit
fsck output

When we run some stress test on our platform, we saw error message like this:

[49074.700194] EXT4-fs error (device mmcblk1p1): ext4_xattr_ibody_get:591:
inode #164242: comm systemd-journal: corrupted in-inode xattr
[49074.701633] EXT4-fs error (device mmcblk1p1): ext4_xattr_ibody_get:591:
inode #164247: comm systemd-journal: corrupted in-inode xattr
[49074.703260] EXT4-fs error (device mmcblk1p1): ext4_xattr_ibody_get:591:
inode #164245: comm systemd-journal: corrupted in-inode xattr
[49079.621065] EXT4-fs error: 39 callbacks suppressed

kernel version is 4.19.23-19.
CPU: Intel(R) Atom(TM) Processor A3960 @ 1.90GHz
rootfs: emmc storage card
Running on top of a hypervisor (ACRN)

When the error message happens, the GUI will stop working, sometimes the serial
console will stop too, but we can still connect to it through SSH connections,
and if we run fsck on that system, will see some error info (full log
attached).

some are:

Multiply-claimed block(s) in inode 164223: 656535
Multiply-claimed block(s) in inode 164224: 673971 757260--758282
Multiply-claimed block(s) in inode 164225: 656531

File
/var/log/journal/cf5c3ff473bd4a2dbfc890565e0439f9/system@e2d41008f0ce4d53849cdc7cec0f8a29-000000000003045c-0005831a9bb56af7.journal
(inode #164224, mod time Sat Mar  2 11:13:48 2019)
  has 1024 multiply-claimed block(s), shared with 1 file(s):
        /var/log/crashlog/vmevent393_98cd58c91b1da34b635e/crashfile (inode
#164240, mod time Sat Mar  2 11:13:48 2019)

Could you please help to check, and provide some hints for debugging? thanks!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

--- Comment #1 from Feng Tang (feng.tang@intel.com) ---
according to our test, when there error happens, the system(including
systemd-journal) is busy writing logs into different log files under different
directories. (mostly under /var/log/)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

--- Comment #2 from Theodore Tso (tytso@mit.edu) ---
Can you try reproducing the problem when using something more robust than an
emmc storage card?   The errors:

Multiply-claimed block(s) in inode 164223: 656535

alongside the xattr errors strongly suggest read or writes being corrupted.

So the first thing I would ask is: (a) can you reproduce this on multiple emmc
cards?   (b) can you use some other storage, say, a USB-attached SSD, as
opposed to crap flash (or to be politically correct, "cost-optimized flash" :-)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

--- Comment #3 from Feng Tang (feng.tang@intel.com) ---
Thanks for your prompt response.

We cannot change the emmc card as it is fixed on the board. One other
interesting thing is, we have 2 labs, each has 10 boards to do the stress test,
and only the boards in one lab can reproduce the issue, so we are doubting some
env setup trigger the issue, and not the emmc hw issue.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

Feng Tang (feng.tang@intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #4 from Feng Tang (feng.tang@intel.com) ---
We has some more findings, if we disable the HW cache feature of the eMMC card
hardened in our board, then the issue cannot be reproduced.

So it seems to be like a HW issue, let's close it. And thanks for your help.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

Hushup (1158340263@qq.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |1158340263@qq.com

--- Comment #5 from Hushup (1158340263@qq.com) ---
Hi ,We meet the same problem too,Can you share us how to reproduce the problem?
Thanks.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

Gerald Boyle (peanutsunless@gmail.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |peanutsunless@gmail.com

--- Comment #6 from Gerald Boyle (peanutsunless@gmail.com) ---
The error messages you're encountering indicate that there is filesystem
corruption on your EXT4 filesystem, particularly related to extended attributes
(xattrs). To run fsck safely, unmount the filesystem. If it's your root
filesystem, you may need to use a live CD/USB or boot into single-user 
mode.umount /dev/mmcblk1p1
fsck.ext4 -f -y /dev/mmcblk1p1
smartctl -a /dev/mmcblk1
Check system logs for additional context around the time the errors occur. Logs
like /var/log/syslog or /var/log/messages might provide more insights. I
noticed that a new driver is available at
http://www.ralinktech.com/ralink/Home/Support/Linux.html
https://geometrydashworld.net
Identify if there is a pattern or specific actions that correlate with the
corruption, which might help isolate the cause.  Persistent filesystem
corruption could indicate failing storage hardware. While eMMC storage is
generally reliable, it can still fail or degrade over time.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

--- Comment #7 from Theodore Tso (tytso@mit.edu) ---
Comment #6 is SPAM, marking private.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 202809] ext4: ext4_xattr_ibody_get:591: comm systemd-journal: corrupted in-inode xattr

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=202809

--- Comment #8 from Theodore Tso (tytso@mit.edu) ---
Comment #6 is SPAM, marking private.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.