Re: BUG: scrub reports uncorrectable csum errors linked to readable file (data: single)

[Lionel Bouton <lionel-subscription@bouton.name>]

Hi,

To keep this short I've removed most of past exchanges as this is just 
to keep people following this thread informed on my progress.

Le 07/06/2024 à 01:46, Lionel Bouton a écrit :
>
> [...]
>>> I briefly considered doing just that... but then I found out that the
>>> scrub errors were themselves in error and the on disk data was matching
>>> the checksums. When I tried to read the file not only didn't the
>>> filesystem report an IO error (if I'm not mistaken it should if the 
>>> csum
>>> doesn't match) but the file content matched the original file fetched
>>> from its source.
>>
>> Got it, this is really weird now.
>>
>> What scrub doing is read the data from disk (without bothering page
>> cache), and verify against checksums.
>>
>> Would it be possible to run "btrfs check --check-data-csum" on the
>> unmounted/RO mounted fs?
>
> Yes with some caveats as the scrub takes approximately a week to 
> complete and I can't easily stop the services on this system for a week.
> The block device is RBD on Ceph, so what I can do is take a block 
> level snapshot, map this snapshot to another system and run btrfs 
> check --check-data-csum there. If the IO is the same than btrfs scrub 
> this will probably take between 3 to 5 days to complete. I'll have to 
> run this on another VM with the same kernel and btrfs-progs versions, 
> BTRFS doesn't like having 2 devices showing up with the same internal 
> identifiers...
>
>>
>> That would output the error for each corrupted sector (without
>> ratelimit), so that you can record them all.
>> And try to do logical-resolve to find each corrupted location?
>>
>> If btrfs check reports no error, it's 100% sure scrub is to blamce.
>>
>> If btrfs check reports error, and logical-resolve failed to locate the
>> file and its position, it means the corruption is in bookend exntets.
>>
>> If btrfs check reports error and logical-resolve can locate the file and
>> position, it's a different problem then.
>
> OK. I understand. This is time for me to go to sleep, but I'll work on 
> this tomorrow. I'll report as soon as check-data-sum finds something 
> or at the end in several days if it didn't.

There is a bit of a slowdown. btrfs check was killed a couple hours ago 
(after running more than a day) by the OOM killer. I anticipated that it 
would need large amounts of memory (see below for the number of 
files/dirs/subvolumes) and started it on a VM with 32GB but it wasn't 
enough. It stopped after printing: "[4/7] checking fs roots".

I restarted btrfs check --check-data-csum after giving 64GB of RAM to 
the VM hoping this will be enough.
If it doesn't work and the oom-killer still is triggered I'll have to 
move other VMs around and the realistic maximum I can give the VM used 
for runing the btrfs check is ~200GB.

If someone familiar with btrfs check can estimate how much RAM is 
needed, here is some information that might be relevant:
- according to the latest estimations there should be a total of around 
50M files and 2.5M directories in the 3 main subvolumes on this filesystem.
- for each of these 3 subvolumes there should be approximately 30 snapshots.

Here is the filesystem usage output:
Overall:
     Device size:                  40.00TiB
     Device allocated:             31.62TiB
     Device unallocated:            8.38TiB
     Device missing:                  0.00B
     Device slack:                 12.00TiB
     Used:                         18.72TiB
     Free (estimated):             20.32TiB      (min: 16.13TiB)
     Free (statfs, df):            20.32TiB
     Data ratio:                       1.00
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 0.00B)
     Multiple profiles:                  no

Data,single: Size:30.51TiB, Used:18.57TiB (60.87%)
    /dev/sdb       30.51TiB

Metadata,DUP: Size:565.50GiB, Used:75.40GiB (13.33%)
    /dev/sdb        1.10TiB

System,DUP: Size:8.00MiB, Used:3.53MiB (44.14%)
    /dev/sdb       16.00MiB

Unallocated:
    /dev/sdb        8.38TiB


Best regards,
Lionel