Re: BUG: scrub reports uncorrectable csum errors linked to readable file (data: single)

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2024/6/9 01:45, Lionel Bouton 写道:
> Hi,
>
> To keep this short I've removed most of past exchanges as this is just
> to keep people following this thread informed on my progress.
>
> Le 07/06/2024 à 01:46, Lionel Bouton a écrit :
>>
>> [...]
>>>> I briefly considered doing just that... but then I found out that the
>>>> scrub errors were themselves in error and the on disk data was matching
>>>> the checksums. When I tried to read the file not only didn't the
>>>> filesystem report an IO error (if I'm not mistaken it should if the
>>>> csum
>>>> doesn't match) but the file content matched the original file fetched
>>>> from its source.
>>>
>>> Got it, this is really weird now.
>>>
>>> What scrub doing is read the data from disk (without bothering page
>>> cache), and verify against checksums.
>>>
>>> Would it be possible to run "btrfs check --check-data-csum" on the
>>> unmounted/RO mounted fs?
>>
>> Yes with some caveats as the scrub takes approximately a week to
>> complete and I can't easily stop the services on this system for a week.
>> The block device is RBD on Ceph, so what I can do is take a block
>> level snapshot, map this snapshot to another system and run btrfs
>> check --check-data-csum there. If the IO is the same than btrfs scrub
>> this will probably take between 3 to 5 days to complete. I'll have to
>> run this on another VM with the same kernel and btrfs-progs versions,
>> BTRFS doesn't like having 2 devices showing up with the same internal
>> identifiers...
>>
>>>
>>> That would output the error for each corrupted sector (without
>>> ratelimit), so that you can record them all.
>>> And try to do logical-resolve to find each corrupted location?
>>>
>>> If btrfs check reports no error, it's 100% sure scrub is to blamce.
>>>
>>> If btrfs check reports error, and logical-resolve failed to locate the
>>> file and its position, it means the corruption is in bookend exntets.
>>>
>>> If btrfs check reports error and logical-resolve can locate the file and
>>> position, it's a different problem then.
>>
>> OK. I understand. This is time for me to go to sleep, but I'll work on
>> this tomorrow. I'll report as soon as check-data-sum finds something
>> or at the end in several days if it didn't.
>
> There is a bit of a slowdown. btrfs check was killed a couple hours ago
> (after running more than a day) by the OOM killer. I anticipated that it
> would need large amounts of memory (see below for the number of
> files/dirs/subvolumes) and started it on a VM with 32GB but it wasn't
> enough. It stopped after printing: "[4/7] checking fs roots".
>
> I restarted btrfs check --check-data-csum after giving 64GB of RAM to
> the VM hoping this will be enough.
> If it doesn't work and the oom-killer still is triggered I'll have to
> move other VMs around and the realistic maximum I can give the VM used
> for runing the btrfs check is ~200GB.

That's why we have --mode=lowmem exactly for that reason.

But please be aware that, the low memory usage is traded off by doing a
lot of more IO.

Thanks,
Qu
>
> If someone familiar with btrfs check can estimate how much RAM is
> needed, here is some information that might be relevant:
> - according to the latest estimations there should be a total of around
> 50M files and 2.5M directories in the 3 main subvolumes on this filesystem.
> - for each of these 3 subvolumes there should be approximately 30
> snapshots.
>
> Here is the filesystem usage output:
> Overall:
>      Device size:                  40.00TiB
>      Device allocated:             31.62TiB
>      Device unallocated:            8.38TiB
>      Device missing:                  0.00B
>      Device slack:                 12.00TiB
>      Used:                         18.72TiB
>      Free (estimated):             20.32TiB      (min: 16.13TiB)
>      Free (statfs, df):            20.32TiB
>      Data ratio:                       1.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
>      Multiple profiles:                  no
>
> Data,single: Size:30.51TiB, Used:18.57TiB (60.87%)
>     /dev/sdb       30.51TiB
>
> Metadata,DUP: Size:565.50GiB, Used:75.40GiB (13.33%)
>     /dev/sdb        1.10TiB
>
> System,DUP: Size:8.00MiB, Used:3.53MiB (44.14%)
>     /dev/sdb       16.00MiB
>
> Unallocated:
>     /dev/sdb        8.38TiB
>
>
> Best regards,
> Lionel