Re: question to btrfs scrub

[Qu Wenruo <wqu@suse.com>]

On 2023/7/5 00:03, Bernd Lentes wrote:
> 
>> -----Original Message-----
>> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>> Sent: Friday, June 30, 2023 12:17 PM
>> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; linux-btrfs <linux-
>> btrfs@vger.kernel.org>
>> Subject: Re: question to btrfs scrub
>>
>>
>>
>>
>> And I don't want to go that path unless there are enough evidence to indicate
>> that.
>>
>> Have you figured out which file(s) are affected and what's the workload of
>> them?
> 
> Yes. Several images from virtual machines.
> 
> I made a btrfs check on the image and on the volume itself:
> 
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/vg_san/lv_domains
> Opening filesystem to check...
> Checking filesystem on /dev/vg_san/lv_domains
> UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
> [1/7] checking root items                      (0:00:36 elapsed, 6900134 items checked)
> [2/7] checking extents                         (0:01:58 elapsed, 484995 items checked)
> [3/7] checking free space cache                (0:00:13 elapsed, 5184 items checked)
> [4/7] checking fs roots                        (0:02:39 elapsed, 65549 items checked)
> [5/7] checking csums (without verifying data)

Notice the quote "without verifying data".

That's why it doesn't report the csum error, and we need 
"--check-data-csum" to verify data csum.

   (0:00:10 elapsed, 3236328 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 5396770611200 bytes used, no error found
> total csum bytes: 5263001424
> total tree bytes: 7945863168
> total fs tree bytes: 1077493760
> total extent tree bytes: 828391424
> btree space waste bytes: 1124143787
> file data blocks allocated: 127704684556288
>   referenced 8084906622976
> 
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/loop0
> Opening filesystem to check...
> Checking filesystem on /dev/loop0
> UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
> [1/7] checking root items                      (0:01:24 elapsed, 6900131 items checked)
> [2/7] checking extents                         (0:04:28 elapsed, 484992 items checked)
> [3/7] checking free space cache                (0:00:46 elapsed, 5184 items checked)
> [4/7] checking fs roots                        (0:02:45 elapsed, 65549 items checked)
> [5/7] checking csums (without verifying data)  (0:00:10 elapsed, 3236328 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 5396770562048 bytes used, no error found
> total csum bytes: 5263001424
> total tree bytes: 7945814016
> total fs tree bytes: 1077493760
> total extent tree bytes: 828342272
> btree space waste bytes: 1124095211
> file data blocks allocated: 127704684556288
>   referenced 8084906622976
> 
> Strange. No error found. I expected errors because of what btrfs scrub said.
> I checked the logical volume with badblocks - no error:
> ha-idg-1:~ # badblocks -sv -b 4096 /dev/vg_san/lv_domains
> Checking blocks 0 to 1664303103
> Checking for bad blocks (read-only test): ^[[A^[[Bdone
> Pass completed, 0 bad blocks found. (0/0/0 errors)
> 
> Now I'm a bit confused. badblocks and btrfs check no error, but btrfs scrub a lot of errors:
> 
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs scrub start -B /mnt/image/
> scrub done for bbcfa007-fb2b-432a-b513-207d5df35a2a
> Scrub started:    Tue Jun 27 20:47:26 2023
> Status:           finished
> Duration:         35:39:48
> Total to scrub:   5.07TiB
> Rate:             40.16MiB/s
> Error summary:    csum=1052
>    Corrected:      0
>    Uncorrectable:  1052
>    Unverified:     0
> ERROR: there are uncorrectable errors
> 
> What can I do ? I'm afraid I have to reformat.'

Nope, you don't need to reformat at all.

> But what be the culprit for the errors ?

It can be a problem of the VM workload on btrfs.

As a workaround, you can easily disable datacow for the VM directory 
using the following command:

# chattr +C <VM images directory>

And then just delete the VM image file causing the problem.

Thanks,
Qu
> 
> Bernd
> 
> Bernd
> 
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671