XFS metadata CRC errors on zram block device on ppc64le architecture

XFS metadata CRC errors on zram block device on ppc64le architecture

[Dusty Mabe]

In Fedora CoreOS we found an issue with an interaction of an XFS filesystem on a zram block device on ppc64le:

- https://github.com/coreos/fedora-coreos-tracker/issues/1489
- https://bugzilla.redhat.com/show_bug.cgi?id=2221314

The dmesg output shows several errors:

```
[ 3247.206007] XFS (zram0): Mounting V5 Filesystem 0b7d6149-614c-4f4c-9a1f-a80a9810f58f
[ 3247.210781] XFS (zram0): Metadata CRC error detected at xfs_agf_read_verify+0x108/0x150 [xfs], xfs_agf block 0x80008 
[ 3247.211121] XFS (zram0): Unmount and run xfs_repair
[ 3247.211198] XFS (zram0): First 128 bytes of corrupted metadata buffer:
[ 3247.211293] 00000000: fe ed ba be 00 00 00 00 00 00 00 02 00 00 00 00 ................
[ 3247.211405] 00000010: 00 00 00 00 00 00 00 18 00 00 00 01 00 00 00 00  ................
[ 3247.211515] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 3247.211625] 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 3247.211735] 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 3247.211842] 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 3247.211951] 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 3247.212063] 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[ 3247.212171] XFS (zram0): metadata I/O error in "xfs_read_agf+0xb4/0x180 [xfs]" at daddr 0x80008 len 8 error 74
[ 3247.212485] XFS (zram0): Error -117 reserving per-AG metadata reserve pool.
[ 3247.212497] XFS (zram0): Corruption of in-memory data (0x8) detected at xfs_fs_reserve_ag_blocks+0x1e0/0x220 [xfs] (fs/xfs/xfs_fsops.c:587).  Shutting down filesystem.
[ 3247.212828] XFS (zram0): Please unmount the filesystem and rectify the problem(s)
[ 3247.212943] XFS (zram0): Ending clean mount
[ 3247.212970] XFS (zram0): Error -5 reserving per-AG metadata reserve pool.
```

The issue can be reproduced easily with a simple script:

```
[root@p8 ~]# cat test.sh 
#!/bin/bash
set -eux -o pipefail
modprobe zram num_devices=0
read dev < /sys/class/zram-control/hot_add
echo 10G > /sys/block/zram"${dev}"/disksize
mkfs.xfs /dev/zram"${dev}"
mkdir -p /tmp/foo
mount -t xfs /dev/zram"${dev}" /tmp/foo
```

We ran a kernel bisect and narrowed it down to offending commit af8b04c6:

```
[root@ibm-p8-kvm-03-guest-02 linux]# git bisect good
af8b04c63708fa730c0257084fab91fb2a9cecc4 is the first bad commit
commit af8b04c63708fa730c0257084fab91fb2a9cecc4
Author: Christoph Hellwig <hch@lst.de>
Date:   Tue Apr 11 19:14:46 2023 +0200

    zram: simplify bvec iteration in __zram_make_request
    
    bio_for_each_segment synthetize bvecs that never cross page boundaries, so
    don't duplicate that work in an inner loop.
    
    Link: https://lkml.kernel.org/r/20230411171459.567614-5-hch@lst.de
    Signed-off-by: Christoph Hellwig <hch@lst.de>
    Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>
    Acked-by: Minchan Kim <minchan@kernel.org>
    Cc: Jens Axboe <axboe@kernel.dk>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

 drivers/block/zram/zram_drv.c | 42 +++++++++++-------------------------------
 1 file changed, 11 insertions(+), 31 deletions(-)
```

Any ideas on how to fix the problem?

Thanks!
Dusty

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Christoph Hellwig]

On Tue, Aug 01, 2023 at 11:31:37PM -0400, Dusty Mabe wrote:
> We ran a kernel bisect and narrowed it down to offending commit af8b04c6:
> 
> ```
> [root@ibm-p8-kvm-03-guest-02 linux]# git bisect good
> af8b04c63708fa730c0257084fab91fb2a9cecc4 is the first bad commit
> commit af8b04c63708fa730c0257084fab91fb2a9cecc4
> Author: Christoph Hellwig <hch@lst.de>
> Date:   Tue Apr 11 19:14:46 2023 +0200
> 
>     zram: simplify bvec iteration in __zram_make_request
>     
>     bio_for_each_segment synthetize bvecs that never cross page boundaries, so
>     don't duplicate that work in an inner loop.

> Any ideas on how to fix the problem?

So the interesting cases are:

  - ppc64 usually uses 64k page sizes
  - ppc64 is somewhat cache incoherent (compared to say x86)

Let me think of this a bit more.

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Hannes Reinecke]

On 8/2/23 11:41, Christoph Hellwig wrote:
> On Tue, Aug 01, 2023 at 11:31:37PM -0400, Dusty Mabe wrote:
>> We ran a kernel bisect and narrowed it down to offending commit af8b04c6:
>>
>> ```
>> [root@ibm-p8-kvm-03-guest-02 linux]# git bisect good
>> af8b04c63708fa730c0257084fab91fb2a9cecc4 is the first bad commit
>> commit af8b04c63708fa730c0257084fab91fb2a9cecc4
>> Author: Christoph Hellwig <hch@lst.de>
>> Date:   Tue Apr 11 19:14:46 2023 +0200
>>
>>      zram: simplify bvec iteration in __zram_make_request
>>      
>>      bio_for_each_segment synthetize bvecs that never cross page boundaries, so
>>      don't duplicate that work in an inner loop.
> 
>> Any ideas on how to fix the problem?
> 
> So the interesting cases are:
> 
>    - ppc64 usually uses 64k page sizes
>    - ppc64 is somewhat cache incoherent (compared to say x86)
> 
> Let me think of this a bit more.

Would need to be confirmed first that 64k pages really are in use
(eg we compile ppc64le with 4k page sizes ...).
Dusty?
For which page size did you compile your kernel?

Cheers,

Hannes

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Dusty Mabe]

On 8/2/23 07:03, Hannes Reinecke wrote:
> On 8/2/23 11:41, Christoph Hellwig wrote:
>> On Tue, Aug 01, 2023 at 11:31:37PM -0400, Dusty Mabe wrote:
>>> We ran a kernel bisect and narrowed it down to offending commit af8b04c6:
>>>
>>> ```
>>> [root@ibm-p8-kvm-03-guest-02 linux]# git bisect good
>>> af8b04c63708fa730c0257084fab91fb2a9cecc4 is the first bad commit
>>> commit af8b04c63708fa730c0257084fab91fb2a9cecc4
>>> Author: Christoph Hellwig <hch@lst.de>
>>> Date:   Tue Apr 11 19:14:46 2023 +0200
>>>
>>>      zram: simplify bvec iteration in __zram_make_request
>>>      
>>>      bio_for_each_segment synthetize bvecs that never cross page boundaries, so
>>>      don't duplicate that work in an inner loop.
>>
>>> Any ideas on how to fix the problem?
>>
>> So the interesting cases are:
>>
>>    - ppc64 usually uses 64k page sizes
>>    - ppc64 is somewhat cache incoherent (compared to say x86)
>>
>> Let me think of this a bit more.
> 
> Would need to be confirmed first that 64k pages really are in use
> (eg we compile ppc64le with 4k page sizes ...).
> Dusty?
> For which page size did you compile your kernel?


For Fedora the configuration is to enable 64k pages with CONFIG_PPC_64K_PAGES=y
https://src.fedoraproject.org/rpms/kernel/blob/064c1675a16b4d379b42ab6c3397632ca54ad897/f/kernel-ppc64le-fedora.config#_4791

I used the same configuration when running the git bisect.

Dusty

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Dusty Mabe]

On 8/2/23 08:00, Dusty Mabe wrote:
> 
> 
> On 8/2/23 07:03, Hannes Reinecke wrote:
>> On 8/2/23 11:41, Christoph Hellwig wrote:
>>> On Tue, Aug 01, 2023 at 11:31:37PM -0400, Dusty Mabe wrote:
>>>> We ran a kernel bisect and narrowed it down to offending commit af8b04c6:
>>>>
>>>> ```
>>>> [root@ibm-p8-kvm-03-guest-02 linux]# git bisect good
>>>> af8b04c63708fa730c0257084fab91fb2a9cecc4 is the first bad commit
>>>> commit af8b04c63708fa730c0257084fab91fb2a9cecc4
>>>> Author: Christoph Hellwig <hch@lst.de>
>>>> Date:   Tue Apr 11 19:14:46 2023 +0200
>>>>
>>>>      zram: simplify bvec iteration in __zram_make_request
>>>>      
>>>>      bio_for_each_segment synthetize bvecs that never cross page boundaries, so
>>>>      don't duplicate that work in an inner loop.
>>>
>>>> Any ideas on how to fix the problem?
>>>
>>> So the interesting cases are:
>>>
>>>    - ppc64 usually uses 64k page sizes
>>>    - ppc64 is somewhat cache incoherent (compared to say x86)
>>>
>>> Let me think of this a bit more.
>>
>> Would need to be confirmed first that 64k pages really are in use
>> (eg we compile ppc64le with 4k page sizes ...).
>> Dusty?
>> For which page size did you compile your kernel?
> 
> 
> For Fedora the configuration is to enable 64k pages with CONFIG_PPC_64K_PAGES=y
> https://src.fedoraproject.org/rpms/kernel/blob/064c1675a16b4d379b42ab6c3397632ca54ad897/f/kernel-ppc64le-fedora.config#_4791
> 
> I used the same configuration when running the git bisect.

Naive question from my side: would this be a candidate for reverting while we investigate the root cause?

Dusty

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Sergey Senozhatsky]

On (23/08/03 17:32), Dusty Mabe wrote:
> >>>>      zram: simplify bvec iteration in __zram_make_request
> >>>>      
> >>>>      bio_for_each_segment synthetize bvecs that never cross page boundaries, so
> >>>>      don't duplicate that work in an inner loop.
> >>>
> >>>> Any ideas on how to fix the problem?
> >>>
> >>> So the interesting cases are:
> >>>
> >>>    - ppc64 usually uses 64k page sizes
> >>>    - ppc64 is somewhat cache incoherent (compared to say x86)
> >>>
> >>> Let me think of this a bit more.
> >>
> >> Would need to be confirmed first that 64k pages really are in use
> >> (eg we compile ppc64le with 4k page sizes ...).
> >> Dusty?
> >> For which page size did you compile your kernel?
> > 
> > 
> > For Fedora the configuration is to enable 64k pages with CONFIG_PPC_64K_PAGES=y
> > https://src.fedoraproject.org/rpms/kernel/blob/064c1675a16b4d379b42ab6c3397632ca54ad897/f/kernel-ppc64le-fedora.config#_4791
> > 
> > I used the same configuration when running the git bisect.
> 
> Naive question from my side: would this be a candidate for reverting while we investigate the root cause?

That's certainly a possible solution.

But I don't quite understand why af8b04c63708 doesn't work.

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Christoph Hellwig]

FYI, I've found an arm64 system with 16k page size support, and while
I can't reproduce the exact issue, I do see corruption with I/O test
on zram that don't show on the same system with 4k pages.  I'm trying
to understand the details at the moment.

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Hannes Reinecke]

On 8/4/23 15:42, Christoph Hellwig wrote:
> FYI, I've found an arm64 system with 16k page size support, and while
> I can't reproduce the exact issue, I do see corruption with I/O test
> on zram that don't show on the same system with 4k pages.  I'm trying
> to understand the details at the moment.
> 
For some reason zram run with a logical block size of 4k:

#define ZRAM_LOGICAL_BLOCK_SHIFT 12
#define ZRAM_LOGICAL_BLOCK_SIZE (1 << ZRAM_LOGICAL_BLOCK_SHIFT)

so we'll have sub-page accesses for larger page sizes.
My bet is that the issue goes away if we set the logical block size to 
page size ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Linux regression tracking (Thorsten Leemhuis)]

[CCing Linus and the regressions list; fwiw, initial report is here:
https://lore.kernel.org/all/b2d40565-7868-ba15-4bb1-fca6f0df076b@dustymabe.com/
]

On 04.08.23 05:25, Sergey Senozhatsky wrote:
> On (23/08/03 17:32), Dusty Mabe wrote:
>>>>>>      zram: simplify bvec iteration in __zram_make_request
>>>>>>      
>>>>>>      bio_for_each_segment synthetize bvecs that never cross page boundaries, so
>>>>>>      don't duplicate that work in an inner loop.
>>>>>
>>>>>> Any ideas on how to fix the problem?
>>>>>
>>>>> So the interesting cases are:
>>>>>
>>>>>    - ppc64 usually uses 64k page sizes
>>>>>    - ppc64 is somewhat cache incoherent (compared to say x86)
>>>>>
>>>>> Let me think of this a bit more.
>>>>
>>>> Would need to be confirmed first that 64k pages really are in use
>>>> (eg we compile ppc64le with 4k page sizes ...).
>>>> Dusty?
>>>> For which page size did you compile your kernel?
>>>
>>> For Fedora the configuration is to enable 64k pages with CONFIG_PPC_64K_PAGES=y
>>> https://src.fedoraproject.org/rpms/kernel/blob/064c1675a16b4d379b42ab6c3397632ca54ad897/f/kernel-ppc64le-fedora.config#_4791
>>>
>>> I used the same configuration when running the git bisect.
>>
>> Naive question from my side: would this be a candidate for reverting while we investigate the root cause?
> 
> That's certainly a possible solution.
> 
> But I don't quite understand why af8b04c63708 doesn't work.

Seems Christoph and Hannes (thx to both of you) got a bit closer to
that, but as this apparently is causing data corruption and we are close
to -rc5 I'd like to bring the following up now, as it gets harder to
discuss these things on weekends:

Should Linus revert the culprit for -rc5 if no fix is found within the
next 48 hours?

Ciao, Thorsten

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Linux regression tracking #adding (Thorsten Leemhuis)]

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 02.08.23 05:31, Dusty Mabe wrote:
> In Fedora CoreOS we found an issue with an interaction of an XFS filesystem on a zram block device on ppc64le:
> 
> - https://github.com/coreos/fedora-coreos-tracker/issues/1489
> - https://bugzilla.redhat.com/show_bug.cgi?id=2221314
> 
> The dmesg output shows several errors:
> 
> ```
> [ 3247.206007] XFS (zram0): Mounting V5 Filesystem 0b7d6149-614c-4f4c-9a1f-a80a9810f58f
> [ 3247.210781] XFS (zram0): Metadata CRC error detected at xfs_agf_read_verify+0x108/0x150 [xfs], xfs_agf block 0x80008 
> [ 3247.211121] XFS (zram0): Unmount and run xfs_repair
> [...]
> We ran a kernel bisect and narrowed it down to offending commit af8b04c6:
> 
> ```
> [root@ibm-p8-kvm-03-guest-02 linux]# git bisect good
> af8b04c63708fa730c0257084fab91fb2a9cecc4 is the first bad commit
> commit af8b04c63708fa730c0257084fab91fb2a9cecc4
> Author: Christoph Hellwig <hch@lst.de>
> Date:   Tue Apr 11 19:14:46 2023 +0200
> 
>     zram: simplify bvec iteration in __zram_make_request
>     
> [...]

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced af8b04c63708fa730c0257084fab91fb2a9cec
#regzbot title zram: XFS metadata CRC errors on zram block device on ppc64le
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: XFS metadata CRC errors on zram block device on ppc64le architecture

[Linux regression tracking #update (Thorsten Leemhuis)]

[TLDR: This mail in primarily relevant for Linux regression tracking. A
change or fix related to the regression discussed in this thread was
posted or applied, but it did not use a Closes: tag to point to the
report, as Linus and the documentation call for. Things happen, no
worries -- but now the regression tracking bot needs to be told manually
about the fix. See link in footer if these mails annoy you.]

On 04.08.23 18:22, Linux regression tracking #adding (Thorsten Leemhuis)
wrote:

> On 02.08.23 05:31, Dusty Mabe wrote:
>> In Fedora CoreOS we found an issue with an interaction of an XFS filesystem on a zram block device on ppc64le:
>>
>> - https://github.com/coreos/fedora-coreos-tracker/issues/1489
>> - https://bugzilla.redhat.com/show_bug.cgi?id=2221314

#regzbot monitor:
https://lore.kernel.org/all/20230805055537.147835-1-hch@lst.de/
#regzbot fix: zram: take device and not only bvec offset into account
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.