question to btrfs scrub

question to btrfs scrub

[Bernd Lentes]

Hi guys,

i have a BTRFS volume which produces a lot of errors in the syslog.
Here I got the recommendation to start a “btrfs scrub” on that volume.
I made an image of that volume (with dd) and started the scrub on that.
That’s the result:

ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs scrub start -B /mnt/image/

scrub done for bbcfa007-fb2b-432a-b513-207d5df35a2a
Scrub started:    Tue Jun 27 20:47:26 2023
Status:           finished
Duration:         35:39:48
Total to scrub:   5.07TiB
Rate:             40.16MiB/s
Error summary:    csum=1052
  Corrected:      0
  Uncorrectable:  1052
  Unverified:     0
ERROR: there are uncorrectable errors

1052 checksum errors on a 5TB volume. Is that much, or is that normal ?
What can I do ?
Start a btrfs check ? First on the image before on the original ?

Thanks.


Bernd Lentes

--
Bernd Lentes
System Administrator
MCD
Helmholtzzentrum München
+49 89 3187 1241
mailto:bernd.lentes@helmholtz-munich.de
https://www.helmholtz-munich.de/en/mcd


Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/6/29 16:26, Bernd Lentes wrote:
> Hi guys,
>
> i have a BTRFS volume which produces a lot of errors in the syslog.
> Here I got the recommendation to start a “btrfs scrub” on that volume.
> I made an image of that volume (with dd) and started the scrub on that.
> That’s the result:
>
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs scrub start -B /mnt/image/
>
> scrub done for bbcfa007-fb2b-432a-b513-207d5df35a2a
> Scrub started:    Tue Jun 27 20:47:26 2023
> Status:           finished
> Duration:         35:39:48
> Total to scrub:   5.07TiB
> Rate:             40.16MiB/s
> Error summary:    csum=1052
>    Corrected:      0
>    Uncorrectable:  1052
>    Unverified:     0
> ERROR: there are uncorrectable errors
>
> 1052 checksum errors on a 5TB volume. Is that much, or is that normal ?

I believe it's not normal, but please also provide the following info:

- Kernel version
- SMART info
- HDD model

> What can I do ?

You can check the dmesg, which should mention the file name of the
offending inode.

IIRC there used to be some bug causing csum mismatch, if it's all in one
file, you can backup and remove the file, then set nodatacow for that file.

> Start a btrfs check ?

It would never hurt, but if it's only csum errors, I believe btrfs check
would not report any major problem.

> First on the image before on the original ?

Yeah, on the image first.

Thanks,
Qu
>
> Thanks.
>
>
> Bernd Lentes
>
> --
> Bernd Lentes
> System Administrator
> MCD
> Helmholtzzentrum München
> +49 89 3187 1241
> mailto:bernd.lentes@helmholtz-munich.de
> https://www.helmholtz-munich.de/en/mcd
>
>
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Forza]

On 2023-06-29 10:26, Bernd Lentes wrote:
> Hi guys,
> 
> i have a BTRFS volume which produces a lot of errors in the syslog.
> Here I got the recommendation to start a “btrfs scrub” on that volume.
> I made an image of that volume (with dd) and started the scrub on that.
> That’s the result:
> 
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs scrub start -B /mnt/image/
> 
> scrub done for bbcfa007-fb2b-432a-b513-207d5df35a2a
> Scrub started:    Tue Jun 27 20:47:26 2023
> Status:           finished
> Duration:         35:39:48
> Total to scrub:   5.07TiB
> Rate:             40.16MiB/s
> Error summary:    csum=1052
>    Corrected:      0
>    Uncorrectable:  1052
>    Unverified:     0
> ERROR: there are uncorrectable errors
> 
> 1052 checksum errors on a 5TB volume. Is that much, or is that normal ?
> What can I do ?
> Start a btrfs check ? First on the image before on the original ?
> 

Uncorrectable errors means there were some corruptions that Btrfs could 
not correct using a good copy (RAID/DUP profiles). Those corruptions 
could be different things, for example media errors on the disk drive.

Is it just a single disk that you have in this filesystem?

What does smartctl -x /dev/xxx show? Especially look at the table 
containing Uncorrectable_Error_Cnt or Reallocated_Sector_Ct.

You can also issue a drive self-test using `smartctl -t long /dev/xxx`

~F

RE: question to btrfs scrub

[Bernd Lentes]

> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Sent: Thursday, June 29, 2023 11:08 AM
> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; linux-btrfs <linux-
> btrfs@vger.kernel.org>
> Subject: Re: question to btrfs scrub

> - Kernel version
5.14.21-150400.24.63-default

> - SMART info
ha-idg-1:/mnt/sdc1/ha-idg-1/image # smartctl -T verypermissive -a /dev/sdc
smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.14.21-150400.24.63-default] (SUSE RPM)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: scsi error unsupported field in scsi command

=== START OF INFORMATION SECTION ===
Device Model:     [No Information Found]
Serial Number:    [No Information Found]
Firmware Version: [No Information Found]
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   [No Information Found]
Local Time is:    Fri Jun 30 11:52:26 2023 CEST
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled.
                  Checking to be sure by trying SMART RETURN STATUS command.
SMART support is: Unknown - Try option -s with argument 'on' to enable it.
Read SMART Data failed: scsi error unsupported field in scsi command

=== START OF READ SMART DATA SECTION ===
SMART Status command failed: scsi error unsupported field in scsi command
SMART overall-health self-assessment test result: UNKNOWN!
SMART Status, Attributes and Thresholds cannot be read.

Read SMART Error Log failed: scsi error unsupported field in scsi command

Read SMART Self-test Log failed: scsi error unsupported field in scsi command

Selective Self-tests/Logging not supported

ha-idg-1:/mnt/sdc1/ha-idg-1/image #
ha-idg-1:/mnt/sdc1/ha-idg-1/image # smartctl -T verypermissive -s on /dev/sdc
smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.14.21-150400.24.63-default] (SUSE RPM)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: scsi error unsupported field in scsi command

SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled.
                  Checking to be sure by trying SMART RETURN STATUS command.
SMART support is: Unknown - Try option -s with argument 'on' to enable it.=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enable failed: scsi error unsupported field in scsi command

SMART not available ?

> - HDD model
external USB Seagate

> IIRC there used to be some bug causing csum mismatch, if it's all in one file,
> you can backup and remove the file, then set nodatacow for that file.
bugs causing csum mismatch - Uaargh !
How do i set nodatacow ?

Bernd

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/6/30 17:59, Bernd Lentes wrote:
>
>> -----Original Message-----
>> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>> Sent: Thursday, June 29, 2023 11:08 AM
>> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; linux-btrfs <linux-
>> btrfs@vger.kernel.org>
>> Subject: Re: question to btrfs scrub
>
>> - Kernel version
> 5.14.21-150400.24.63-default
>
>> - SMART info
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # smartctl -T verypermissive -a /dev/sdc
> smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.14.21-150400.24.63-default] (SUSE RPM)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> Read Device Identity failed: scsi error unsupported field in scsi command
>
> === START OF INFORMATION SECTION ===
> Device Model:     [No Information Found]
> Serial Number:    [No Information Found]
> Firmware Version: [No Information Found]
> Device is:        Not in smartctl database [for details use: -P showall]
> ATA Version is:   [No Information Found]
> Local Time is:    Fri Jun 30 11:52:26 2023 CEST
> SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
> SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled.
>                    Checking to be sure by trying SMART RETURN STATUS command.
> SMART support is: Unknown - Try option -s with argument 'on' to enable it.
> Read SMART Data failed: scsi error unsupported field in scsi command
>
> === START OF READ SMART DATA SECTION ===
> SMART Status command failed: scsi error unsupported field in scsi command
> SMART overall-health self-assessment test result: UNKNOWN!
> SMART Status, Attributes and Thresholds cannot be read.
>
> Read SMART Error Log failed: scsi error unsupported field in scsi command
>
> Read SMART Self-test Log failed: scsi error unsupported field in scsi command
>
> Selective Self-tests/Logging not supported
>
> ha-idg-1:/mnt/sdc1/ha-idg-1/image #
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # smartctl -T verypermissive -s on /dev/sdc
> smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.14.21-150400.24.63-default] (SUSE RPM)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
>
> Read Device Identity failed: scsi error unsupported field in scsi command
>
> SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
> SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled.
>                    Checking to be sure by trying SMART RETURN STATUS command.
> SMART support is: Unknown - Try option -s with argument 'on' to enable it.=== START OF ENABLE/DISABLE COMMANDS SECTION ===
> SMART Enable failed: scsi error unsupported field in scsi command
>
> SMART not available ?

That maybe caused by whatever the controller of the USB enclosure.

>
>> - HDD model
> external USB Seagate
>
>> IIRC there used to be some bug causing csum mismatch, if it's all in one file,
>> you can backup and remove the file, then set nodatacow for that file.
> bugs causing csum mismatch - Uaargh !

Your kernel is not that old AFAIK.

And I don't want to go that path unless there are enough evidence to
indicate that.

Have you figured out which file(s) are affected and what's the workload
of them?

Thanks,
Qu
> How do i set nodatacow ?
>
> Bernd
>
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Sent: Friday, June 30, 2023 12:17 PM
> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; linux-btrfs <linux-
> btrfs@vger.kernel.org>
> Subject: Re: question to btrfs scrub
>
>
>
>
> And I don't want to go that path unless there are enough evidence to indicate
> that.
>
> Have you figured out which file(s) are affected and what's the workload of
> them?

Yes. Several images from virtual machines.

I made a btrfs check on the image and on the volume itself:

ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/vg_san/lv_domains
Opening filesystem to check...
Checking filesystem on /dev/vg_san/lv_domains
UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
[1/7] checking root items                      (0:00:36 elapsed, 6900134 items checked)
[2/7] checking extents                         (0:01:58 elapsed, 484995 items checked)
[3/7] checking free space cache                (0:00:13 elapsed, 5184 items checked)
[4/7] checking fs roots                        (0:02:39 elapsed, 65549 items checked)
[5/7] checking csums (without verifying data)  (0:00:10 elapsed, 3236328 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 5396770611200 bytes used, no error found
total csum bytes: 5263001424
total tree bytes: 7945863168
total fs tree bytes: 1077493760
total extent tree bytes: 828391424
btree space waste bytes: 1124143787
file data blocks allocated: 127704684556288
 referenced 8084906622976

ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/loop0
Opening filesystem to check...
Checking filesystem on /dev/loop0
UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
[1/7] checking root items                      (0:01:24 elapsed, 6900131 items checked)
[2/7] checking extents                         (0:04:28 elapsed, 484992 items checked)
[3/7] checking free space cache                (0:00:46 elapsed, 5184 items checked)
[4/7] checking fs roots                        (0:02:45 elapsed, 65549 items checked)
[5/7] checking csums (without verifying data)  (0:00:10 elapsed, 3236328 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 5396770562048 bytes used, no error found
total csum bytes: 5263001424
total tree bytes: 7945814016
total fs tree bytes: 1077493760
total extent tree bytes: 828342272
btree space waste bytes: 1124095211
file data blocks allocated: 127704684556288
 referenced 8084906622976

Strange. No error found. I expected errors because of what btrfs scrub said.
I checked the logical volume with badblocks - no error:
ha-idg-1:~ # badblocks -sv -b 4096 /dev/vg_san/lv_domains
Checking blocks 0 to 1664303103
Checking for bad blocks (read-only test): ^[[A^[[Bdone
Pass completed, 0 bad blocks found. (0/0/0 errors)

Now I'm a bit confused. badblocks and btrfs check no error, but btrfs scrub a lot of errors:

ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs scrub start -B /mnt/image/
scrub done for bbcfa007-fb2b-432a-b513-207d5df35a2a
Scrub started:    Tue Jun 27 20:47:26 2023
Status:           finished
Duration:         35:39:48
Total to scrub:   5.07TiB
Rate:             40.16MiB/s
Error summary:    csum=1052
  Corrected:      0
  Uncorrectable:  1052
  Unverified:     0
ERROR: there are uncorrectable errors

What can I do ? I'm afraid I have to reformat.
But what be the culprit for the errors ?

Bernd

Bernd

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

> -----Original Message-----
> From: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>
> Sent: Tuesday, July 4, 2023 6:04 PM
> To: Qu Wenruo <quwenruo.btrfs@gmx.com>; linux-btrfs <linux-
> btrfs@vger.kernel.org>
> Subject: RE: question to btrfs scrub
>
>
> I made a btrfs check on the image and on the volume itself:
>
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/vg_san/lv_domains
> Opening filesystem to check...
> Checking filesystem on /dev/vg_san/lv_domains
> UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
> [1/7] checking root items                      (0:00:36 elapsed, 6900134 items
> checked)
> [2/7] checking extents                         (0:01:58 elapsed, 484995 items checked)
> [3/7] checking free space cache                (0:00:13 elapsed, 5184 items checked)
> [4/7] checking fs roots                        (0:02:39 elapsed, 65549 items checked)
> [5/7] checking csums (without verifying data)  (0:00:10 elapsed, 3236328
> items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
> [7/7] checking quota groups skipped (not enabled on this FS) found
> 5396770611200 bytes used, no error found total csum bytes: 5263001424
> total tree bytes: 7945863168 total fs tree bytes: 1077493760 total extent
> tree bytes: 828391424 btree space waste bytes: 1124143787 file data blocks
> allocated: 127704684556288  referenced 8084906622976
>
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/loop0 Opening
> filesystem to check...
> Checking filesystem on /dev/loop0
> UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
> [1/7] checking root items                      (0:01:24 elapsed, 6900131 items
> checked)
> [2/7] checking extents                         (0:04:28 elapsed, 484992 items checked)
> [3/7] checking free space cache                (0:00:46 elapsed, 5184 items checked)
> [4/7] checking fs roots                        (0:02:45 elapsed, 65549 items checked)
> [5/7] checking csums (without verifying data)  (0:00:10 elapsed, 3236328
> items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
> [7/7] checking quota groups skipped (not enabled on this FS) found
> 5396770562048 bytes used, no error found total csum bytes: 5263001424
> total tree bytes: 7945814016 total fs tree bytes: 1077493760 total extent
> tree bytes: 828342272 btree space waste bytes: 1124095211 file data blocks
> allocated: 127704684556288  referenced 8084906622976
>
> Strange. No error found. I expected errors because of what btrfs scrub said.
> I checked the logical volume with badblocks - no error:
> ha-idg-1:~ # badblocks -sv -b 4096 /dev/vg_san/lv_domains Checking blocks
> 0 to 1664303103 Checking for bad blocks (read-only test): ^[[A^[[Bdone Pass
> completed, 0 bad blocks found. (0/0/0 errors)
>
> Now I'm a bit confused. badblocks and btrfs check no error, but btrfs scrub a
> lot of errors:
>
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs scrub start -B /mnt/image/ scrub
> done for bbcfa007-fb2b-432a-b513-207d5df35a2a
> Scrub started:    Tue Jun 27 20:47:26 2023
> Status:           finished
> Duration:         35:39:48
> Total to scrub:   5.07TiB
> Rate:             40.16MiB/s
> Error summary:    csum=1052
>   Corrected:      0
>   Uncorrectable:  1052
>   Unverified:     0
> ERROR: there are uncorrectable errors
>
> What can I do ? I'm afraid I have to reformat.
> But what be the culprit for the errors ?
>
> Bernd

I think this is also informative:
ha-idg-1:~ # btrfs device stats /mnt/domains/
[/dev/mapper/vg_san-lv_domains].write_io_errs    0
[/dev/mapper/vg_san-lv_domains].read_io_errs     0
[/dev/mapper/vg_san-lv_domains].flush_io_errs    0
[/dev/mapper/vg_san-lv_domains].corruption_errs  23134
[/dev/mapper/vg_san-lv_domains].generation_errs  0

Bernd

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/7/5 00:03, Bernd Lentes wrote:
> 
>> -----Original Message-----
>> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>> Sent: Friday, June 30, 2023 12:17 PM
>> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; linux-btrfs <linux-
>> btrfs@vger.kernel.org>
>> Subject: Re: question to btrfs scrub
>>
>>
>>
>>
>> And I don't want to go that path unless there are enough evidence to indicate
>> that.
>>
>> Have you figured out which file(s) are affected and what's the workload of
>> them?
> 
> Yes. Several images from virtual machines.
> 
> I made a btrfs check on the image and on the volume itself:
> 
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/vg_san/lv_domains
> Opening filesystem to check...
> Checking filesystem on /dev/vg_san/lv_domains
> UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
> [1/7] checking root items                      (0:00:36 elapsed, 6900134 items checked)
> [2/7] checking extents                         (0:01:58 elapsed, 484995 items checked)
> [3/7] checking free space cache                (0:00:13 elapsed, 5184 items checked)
> [4/7] checking fs roots                        (0:02:39 elapsed, 65549 items checked)
> [5/7] checking csums (without verifying data)

Notice the quote "without verifying data".

That's why it doesn't report the csum error, and we need 
"--check-data-csum" to verify data csum.

   (0:00:10 elapsed, 3236328 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 5396770611200 bytes used, no error found
> total csum bytes: 5263001424
> total tree bytes: 7945863168
> total fs tree bytes: 1077493760
> total extent tree bytes: 828391424
> btree space waste bytes: 1124143787
> file data blocks allocated: 127704684556288
>   referenced 8084906622976
> 
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs check -p /dev/loop0
> Opening filesystem to check...
> Checking filesystem on /dev/loop0
> UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
> [1/7] checking root items                      (0:01:24 elapsed, 6900131 items checked)
> [2/7] checking extents                         (0:04:28 elapsed, 484992 items checked)
> [3/7] checking free space cache                (0:00:46 elapsed, 5184 items checked)
> [4/7] checking fs roots                        (0:02:45 elapsed, 65549 items checked)
> [5/7] checking csums (without verifying data)  (0:00:10 elapsed, 3236328 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 5396770562048 bytes used, no error found
> total csum bytes: 5263001424
> total tree bytes: 7945814016
> total fs tree bytes: 1077493760
> total extent tree bytes: 828342272
> btree space waste bytes: 1124095211
> file data blocks allocated: 127704684556288
>   referenced 8084906622976
> 
> Strange. No error found. I expected errors because of what btrfs scrub said.
> I checked the logical volume with badblocks - no error:
> ha-idg-1:~ # badblocks -sv -b 4096 /dev/vg_san/lv_domains
> Checking blocks 0 to 1664303103
> Checking for bad blocks (read-only test): ^[[A^[[Bdone
> Pass completed, 0 bad blocks found. (0/0/0 errors)
> 
> Now I'm a bit confused. badblocks and btrfs check no error, but btrfs scrub a lot of errors:
> 
> ha-idg-1:/mnt/sdc1/ha-idg-1/image # btrfs scrub start -B /mnt/image/
> scrub done for bbcfa007-fb2b-432a-b513-207d5df35a2a
> Scrub started:    Tue Jun 27 20:47:26 2023
> Status:           finished
> Duration:         35:39:48
> Total to scrub:   5.07TiB
> Rate:             40.16MiB/s
> Error summary:    csum=1052
>    Corrected:      0
>    Uncorrectable:  1052
>    Unverified:     0
> ERROR: there are uncorrectable errors
> 
> What can I do ? I'm afraid I have to reformat.'

Nope, you don't need to reformat at all.

> But what be the culprit for the errors ?

It can be a problem of the VM workload on btrfs.

As a workaround, you can easily disable datacow for the VM directory 
using the following command:

# chattr +C <VM images directory>

And then just delete the VM image file causing the problem.

Thanks,
Qu
> 
> Bernd
> 
> Bernd
> 
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

> -----Original Message-----
> From: Qu Wenruo <wqu@suse.com>
> Sent: Wednesday, July 5, 2023 12:19 AM
> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo
> <quwenruo.btrfs@gmx.com>; linux-btrfs <linux-btrfs@vger.kernel.org>
> Subject: Re: question to btrfs scrub
>
> That's why it doesn't report the csum error, and we need "--check-data-csum"
> to verify data csum.

OK. I started it. But isn't that the same as "btrfs scrub" ?
The man page gives a hint:
--check-data-csum
verify checksums of data blocks
This expects that the filesystem is otherwise OK, and is basically an offline scrub that does not repair data from spare copies.

>> What can I do ? I'm afraid I have to reformat.'
>
> Nope, you don't need to reformat at all.
>
> > But what be the culprit for the errors ?
>
> It can be a problem of the VM workload on btrfs.
My VM's are not under heavy load.
>
> As a workaround, you can easily disable datacow for the VM directory using
> the following command:
>
> # chattr +C <VM images directory>

No. I use btrfs to make snapshots from the images from the VM's.

Thanks.

Bernd


Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/7/5 16:39, Bernd Lentes wrote:
> 
>> -----Original Message-----
>> From: Qu Wenruo <wqu@suse.com>
>> Sent: Wednesday, July 5, 2023 12:19 AM
>> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo
>> <quwenruo.btrfs@gmx.com>; linux-btrfs <linux-btrfs@vger.kernel.org>
>> Subject: Re: question to btrfs scrub
>>
>> That's why it doesn't report the csum error, and we need "--check-data-csum"
>> to verify data csum.
> 
> OK. I started it. But isn't that the same as "btrfs scrub" ?
> The man page gives a hint:
> --check-data-csum
> verify checksums of data blocks
> This expects that the filesystem is otherwise OK, and is basically an offline scrub that does not repair data from spare copies.

Btrfs check has way more comprehensive checks on metadata, but it by 
default not check data csums.

Which is quite the opposite of btrfs scrub, which only checks data csum, 
and metadata checks are very lightweight.

> 
>>> What can I do ? I'm afraid I have to reformat.'
>>
>> Nope, you don't need to reformat at all.
>>
>>> But what be the culprit for the errors ?
>>
>> It can be a problem of the VM workload on btrfs.
> My VM's are not under heavy load.
>>
>> As a workaround, you can easily disable datacow for the VM directory using
>> the following command:
>>
>> # chattr +C <VM images directory>
> 
> No. I use btrfs to make snapshots from the images from the VM's.

NodataCOW would still work with snapshot.

It would still do COW when there are snapshots involved.

The main thing here is, nodatacow implies nodatacsum, thus it would not 
generate any csum nor verify it.

Thanks,
Qu

> 
> Thanks.
> 
> Bernd
> 
> 
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

> -----Original Message-----
> From: Qu Wenruo <wqu@suse.com>
> Sent: Wednesday, July 5, 2023 10:46 AM
> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo
> <quwenruo.btrfs@gmx.com>; linux-btrfs <linux-btrfs@vger.kernel.org>
> Subject: Re: question to btrfs scrub
> That's why it doesn't report the csum error, and we need "--check-data-
> csum"
> to verify data csum.
> >
> > OK. I started it. But isn't that the same as "btrfs scrub" ?
> > The man page gives a hint:
> > --check-data-csum
> > verify checksums of data blocks
> > This expects that the filesystem is otherwise OK, and is basically an offline
> > scrub that does not repair data from spare copies.

> Btrfs check has way more comprehensive checks on metadata, but it by
> default not check data csums.

> Which is quite the opposite of btrfs scrub, which only checks data csum, and
> metadata checks are very lightweight.

> The main thing here is, nodatacow implies nodatacsum, thus it would not
> generate any csum nor verify it.

But aren't checksums important in case of errors ?
OK. I know which VM images produced checksum errors. I delete them and restore them from the backup.
Then I set the attribute for the directory.

OK ?

Here my output from the btrfs check:
ha-idg-1:~ # btrfs check -p --check-data-csum /dev/vg_san/lv_domains
Opening filesystem to check...
Checking filesystem on /dev/vg_san/lv_domains
UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
[1/7] checking root items                      (0:00:35 elapsed, 6900134 items checked)
[2/7] checking extents                         (0:01:58 elapsed, 484995 items checked)
[3/7] checking free space cache                (0:00:14 elapsed, 5184 items checked)
[4/7] checking fs roots                        (0:02:38 elapsed, 65549 items checked)
mirror 1 bytenr 1489997918208 csum 0x0e45a39c expected csum 0xaa326ac93 items checked)
mirror 1 bytenr 2036881817600 csum 0x714ca3eb expected csum 0x2cf56c3f9 items checked)
mirror 1 bytenr 2708733394944 csum 0xa5bc757d expected csum 0x7055fdf48 items checked)
mirror 1 bytenr 2743035994112 csum 0x743f7516 expected csum 0x2f21def46 items checked)
mirror 1 bytenr 2855677394944 csum 0x50cbb065 expected csum 0xfb923a738 items checked)
mirror 1 bytenr 4354053398528 csum 0x62eba801 expected csum 0x879bde4a0 items checked)
mirror 1 bytenr 4355127242752 csum 0x71594d4c expected csum 0x879bde4a1 items checked)
mirror 1 bytenr 4359422152704 csum 0x71594d4c expected csum 0x879bde4a6 items checked)
 ...
mirror 1 bytenr 5227759489024 csum 0x92051821 expected csum 0xa61efb7f89 items checked)
mirror 1 bytenr 5229552353280 csum 0x26981ed8 expected csum 0xad21497c44 items checked)
mirror 1 bytenr 5229990215680 csum 0x27f66fb8 expected csum 0x746827b001 items checked)
mirror 1 bytenr 5231527952384 csum 0x9cfa691f expected csum 0x2847c53493 items checked)
mirror 1 bytenr 5233822765056 csum 0xe8072b89 expected csum 0xaf60264806 items checked)
mirror 1 bytenr 5234632364032 csum 0xfd83365b expected csum 0x26dc10d424 items checked)
[5/7] checking csums against data              (3:54:58 elapsed, 3236328 items checked)
ERROR: errors found in csum tree
[6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 5396770611200 bytes used, error(s) found
total csum bytes: 5263001424
total tree bytes: 7945863168
total fs tree bytes: 1077493760
total extent tree bytes: 828391424
btree space waste bytes: 1124143787
file data blocks allocated: 127704684556288
 referenced 8084906622976

So it finds errors in the data csum, correct ?
Thanks.

Bernd



Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Remi Gauvin]

On 2023-07-05 11:01 a.m., Bernd Lentes wrote:

>> The main thing here is, nodatacow implies nodatacsum, thus it would not
>> generate any csum nor verify it.
> 
> But aren't checksums important in case of errors ?
> OK. I know which VM images produced checksum errors. I delete them and restore them from the backup.
> Then I set the attribute for the directory.
> 
> OK ?
> 

I'm really not sure how we jumped to this being a bug that you should
work around by disabling Csum.  I know Qu mentioned the possibility (and
I by no means want to question his expertise.)... But unless I missed
something in this thread, there has been no real indication to not
simply take the error at face value,, the drive/controller/usb combo
resulted in corrupt data, BTRFS detected and reported...

I would hope that the error detection working exactly as it is intended
would be the most likely explanation.

As for what you can do if that's the case, delete the corrupted file and
see if it happens again, (in which case, buy more reliable hardware.),
or just skip the wait and go right to (hopefully) better hardware.

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/7/5 23:01, Bernd Lentes wrote:
>> -----Original Message-----
>> From: Qu Wenruo <wqu@suse.com>
>> Sent: Wednesday, July 5, 2023 10:46 AM
>> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo
>> <quwenruo.btrfs@gmx.com>; linux-btrfs <linux-btrfs@vger.kernel.org>
>> Subject: Re: question to btrfs scrub
>> That's why it doesn't report the csum error, and we need "--check-data-
>> csum"
>> to verify data csum.
>>>
>>> OK. I started it. But isn't that the same as "btrfs scrub" ?
>>> The man page gives a hint:
>>> --check-data-csum
>>> verify checksums of data blocks
>>> This expects that the filesystem is otherwise OK, and is basically an offline
>>> scrub that does not repair data from spare copies.
>
>> Btrfs check has way more comprehensive checks on metadata, but it by
>> default not check data csums.
>
>> Which is quite the opposite of btrfs scrub, which only checks data csum, and
>> metadata checks are very lightweight.
>
>> The main thing here is, nodatacow implies nodatacsum, thus it would not
>> generate any csum nor verify it.
>
> But aren't checksums important in case of errors ?

Checksum have two functions:

- Detect errors
   As long as it's not a false alert, then it makes sense.
   But please keep in mind that, a csum mismatch will prevent anyone
   from reading that corrupted data, whether this is the expected
   behavior really depends on your workload.

   E.g. if some archive with built-in error detect and recovery (like RAR
   files), it's definitely not a good idea to return -EIO for the whole
   block, other than reading out the corrupted data and let the software
   to handle them.

- Recover the good data from the extra mirrors
   This only works if you're using profiles with duplication (DUP,
   RAID1*, RAID10, RAID5, RAID6).
   Otherwise btrfs won't be able to recover anything.

In your case, since you're already using LVM, thus I believe the fs is
using default profiles (DUP for meta, SINGLE for data), thus there would
be no extra copy to recover from.

So there is really error detection functionality lost if go nodatasum.

> OK. I know which VM images produced checksum errors. I delete them and restore them from the backup.

You mentioned snapshots are utilized for those images, thus you have to
delete all the involved files, including ones in the snapshots.

> Then I set the attribute for the directory.
>
> OK ?
>
> Here my output from the btrfs check:
> ha-idg-1:~ # btrfs check -p --check-data-csum /dev/vg_san/lv_domains
> Opening filesystem to check...
> Checking filesystem on /dev/vg_san/lv_domains
> UUID: bbcfa007-fb2b-432a-b513-207d5df35a2a
> [1/7] checking root items                      (0:00:35 elapsed, 6900134 items checked)
> [2/7] checking extents                         (0:01:58 elapsed, 484995 items checked)
> [3/7] checking free space cache                (0:00:14 elapsed, 5184 items checked)
> [4/7] checking fs roots                        (0:02:38 elapsed, 65549 items checked)
> mirror 1 bytenr 1489997918208 csum 0x0e45a39c expected csum 0xaa326ac93 items checked)
> mirror 1 bytenr 2036881817600 csum 0x714ca3eb expected csum 0x2cf56c3f9 items checked)
> mirror 1 bytenr 2708733394944 csum 0xa5bc757d expected csum 0x7055fdf48 items checked)
> mirror 1 bytenr 2743035994112 csum 0x743f7516 expected csum 0x2f21def46 items checked)
> mirror 1 bytenr 2855677394944 csum 0x50cbb065 expected csum 0xfb923a738 items checked)
> mirror 1 bytenr 4354053398528 csum 0x62eba801 expected csum 0x879bde4a0 items checked)
> mirror 1 bytenr 4355127242752 csum 0x71594d4c expected csum 0x879bde4a1 items checked)
> mirror 1 bytenr 4359422152704 csum 0x71594d4c expected csum 0x879bde4a6 items checked)
>   ...
> mirror 1 bytenr 5227759489024 csum 0x92051821 expected csum 0xa61efb7f89 items checked)
> mirror 1 bytenr 5229552353280 csum 0x26981ed8 expected csum 0xad21497c44 items checked)
> mirror 1 bytenr 5229990215680 csum 0x27f66fb8 expected csum 0x746827b001 items checked)
> mirror 1 bytenr 5231527952384 csum 0x9cfa691f expected csum 0x2847c53493 items checked)
> mirror 1 bytenr 5233822765056 csum 0xe8072b89 expected csum 0xaf60264806 items checked)
> mirror 1 bytenr 5234632364032 csum 0xfd83365b expected csum 0x26dc10d424 items checked)
> [5/7] checking csums against data              (3:54:58 elapsed, 3236328 items checked)
> ERROR: errors found in csum tree
> [6/7] checking root refs                       (0:00:00 elapsed, 13 items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 5396770611200 bytes used, error(s) found
> total csum bytes: 5263001424
> total tree bytes: 7945863168
> total fs tree bytes: 1077493760
> total extent tree bytes: 828391424
> btree space waste bytes: 1124143787
> file data blocks allocated: 127704684556288
>   referenced 8084906622976
>
> So it finds errors in the data csum, correct ?

Yes, either there are some files not deleted, or the file is snapshoted.

You'll need to resolve the file, either through scrub (and check dmesg
of the file names), or go with "btrfs inspect logical -o <bytenr like
2743035994112> <mount point>" to find the files.

Otherwise your fs is completely fine.

Thanks,
Qu

> Thanks.
>
> Bernd
>
>
>
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/7/6 01:53, Remi Gauvin wrote:
> On 2023-07-05 11:01 a.m., Bernd Lentes wrote:
>
>>> The main thing here is, nodatacow implies nodatacsum, thus it would not
>>> generate any csum nor verify it.
>>
>> But aren't checksums important in case of errors ?
>> OK. I know which VM images produced checksum errors. I delete them and restore them from the backup.
>> Then I set the attribute for the directory.
>>
>> OK ?
>>
>
> I'm really not sure how we jumped to this being a bug that you should
> work around by disabling Csum.  I know Qu mentioned the possibility (and
> I by no means want to question his expertise.)... But unless I missed
> something in this thread, there has been no real indication to not
> simply take the error at face value,, the drive/controller/usb combo
> resulted in corrupt data, BTRFS detected and reported...
>
> I would hope that the error detection working exactly as it is intended
> would be the most likely explanation.

I hate to point my finger to btrfs itself, but I still remember in the
old days some workload can lead to such false alerts.
But I can not recall which commit is causing and which one is fixing the
problem.

Another concern is, the report is using SINGLE for data, which is
completely fine, but it doesn't help us to determine if it's really a
hardware data corruption or btrfs bugs.

If the report is using RAID1, and those corruption are all repairable,
then we're pretty sure it's data corruption on disk.
Or if all mirrors are corrupted in a RAID1* config, then we know it's
definitely btrfs causing the problem.

But with SINGLE profile, it's really hard to say.

Thanks,
Qu

>
> As for what you can do if that's the case, delete the corrupted file and
> see if it happens again, (in which case, buy more reliable hardware.),
> or just skip the wait and go right to (hopefully) better hardware.
>
>

Re: question to btrfs scrub

[Andrei Borzenkov]

On Thu, Jul 6, 2023 at 9:22 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
> In your case, since you're already using LVM, thus I believe the fs is
> using default profiles (DUP for meta, SINGLE for data), thus there would
> be no extra copy to recover from.
>
> So there is really error detection functionality lost if go nodatasum.
>

You probably mean, "error *correction*". Error detection will
certainly be lost. And error detection is certainly useful to detect
data corruption (that is not usually possible using traditional
filesystems). Yes, it will not be possible to correct these errors on
btrfs level without redundancy.

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/7/6 15:04, Andrei Borzenkov wrote:
> On Thu, Jul 6, 2023 at 9:22 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>> In your case, since you're already using LVM, thus I believe the fs is
>> using default profiles (DUP for meta, SINGLE for data), thus there would
>> be no extra copy to recover from.
>>
>> So there is really error detection functionality lost if go nodatasum.
>>
>
> You probably mean, "error *correction*".

Oh, my bad, it's indeed error *correction*.

Thanks,
Qu

> Error detection will
> certainly be lost. And error detection is certainly useful to detect
> data corruption (that is not usually possible using traditional
> filesystems). Yes, it will not be possible to correct these errors on
> btrfs level without redundancy.

Re: question to btrfs scrub

[Remi Gauvin]

On 2023-07-06 2:23 a.m., Qu Wenruo wrote:

> 
> I hate to point my finger to btrfs itself, but I still remember in the
> old days some workload can lead to such false alerts.
> But I can not recall which commit is causing and which one is fixing the
> problem.
> 
> Another concern is, the report is using SINGLE for data, which is
> completely fine, but it doesn't help us to determine if it's really a
> hardware data corruption or btrfs bugs.
> 

Ok, but I think you missed my point here.

Given what we *do* know about *this* particular situation, (recent
kernel, copies of VM snapshots that were not used as live images, usb
device).  Correct me if I'm wrong, but with the caveat there is no way
to know, it is more likely that csum errors are the result of data
corruption on the disk.

The reason I bring this up, is because the user is being led by thread
into disabling error detection, which is the exact opposite of what
should be done in the case of problems caused by malfunctioning storage
device.

RE: question to btrfs scrub

[Bernd Lentes]

>-----Original Message-----
>From: Remi Gauvin <remi@georgianit.com>
>Sent: Wednesday, July 5, 2023 7:54 PM
>To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo
><wqu@suse.com>; linux-btrfs <linux-btrfs@vger.kernel.org>
>Subject: Re: question to btrfs scrub



>As for what you can do if that's the case, delete the corrupted file and see if it
>happens again, (in which case, buy more reliable hardware.), or just skip the
>wait and go right to (hopefully) better hardware.

We are using HP Servers and HP FC SAN. i think this is reliable hardware, in all the years we had very rarely problems.

Bernd


Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

>-----Original Message-----
>From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>Sent: Thursday, July 6, 2023 8:20 AM
>To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo
><wqu@suse.com>; linux-btrfs <linux-btrfs@vger.kernel.org>
>Subject: Re: question to btrfs scrub



>Checksum have two functions:
>
>- Detect errors
>   As long as it's not a false alert, then it makes sense.
>   But please keep in mind that, a csum mismatch will prevent anyone
>   from reading that corrupted data, whether this is the expected
>   behavior really depends on your workload.
>
>   E.g. if some archive with built-in error detect and recovery (like RAR
>   files), it's definitely not a good idea to return -EIO for the whole
>   block, other than reading out the corrupted data and let the software
>   to handle them.
>
>- Recover the good data from the extra mirrors
>   This only works if you're using profiles with duplication (DUP,
>   RAID1*, RAID10, RAID5, RAID6).
>   Otherwise btrfs won't be able to recover anything.
>
>In your case, since you're already using LVM, thus I believe the fs is using
>default profiles (DUP for meta, SINGLE for data), thus there would be no extra
>copy to recover from.
>
>So there is really error detection functionality lost if go nodatasum.
>
>> OK. I know which VM images produced checksum errors. I delete them and
>restore them from the backup.
>
>You mentioned snapshots are utilized for those images, thus you have to
>delete all the involved files, including ones in the snapshots.
>
>> So it finds errors in the data csum, correct ?
>
>Yes, either there are some files not deleted, or the file is snapshoted.

I don't understand what you mean with that.

Thanks.

Bernd

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

>-----Original Message-----
>From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>Sent: Thursday, July 6, 2023 8:23 AM
>To: Remi Gauvin <remi@georgianit.com>; Bernd Lentes
><bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo <wqu@suse.com>;
>linux-btrfs <linux-btrfs@vger.kernel.org>
>Subject: Re: question to btrfs scrub


>I hate to point my finger to btrfs itself, but I still remember in the old days
>some workload can lead to such false alerts.
>But I can not recall which commit is causing and which one is fixing the
>problem.
>
>Another concern is, the report is using SINGLE for data, which is completely
>fine, but it doesn't help us to determine if it's really a hardware data
>corruption or btrfs bugs.
>
>If the report is using RAID1, and those corruption are all repairable, then we're
>pretty sure it's data corruption on disk.
>Or if all mirrors are corrupted in a RAID1* config, then we know it's definitely
>btrfs causing the problem.
>
>But with SINGLE profile, it's really hard to say.

The server we are talking about is still in testing, not in production.
That means we can change the config.
Should we take RAID1 into account ?
We have enough disk space.

Bernd

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

>-----Original Message-----
>From: Remi Gauvin <remi@georgianit.com>
>Sent: Thursday, July 6, 2023 3:18 PM
>To: Qu Wenruo <quwenruo.btrfs@gmx.com>; Bernd Lentes
><bernd.lentes@helmholtz-muenchen.de>; linux-btrfs <linux-
>btrfs@vger.kernel.org>
>Subject: Re: question to btrfs scrub

>Ok, but I think you missed my point here.
>
>Given what we *do* know about *this* particular situation, (recent kernel,
>copies of VM snapshots that were not used as live images, usb device).
>Correct me if I'm wrong, but with the caveat there is no way to know, it is
>more likely that csum errors are the result of data corruption on the disk.

But i ran badblocks without any error. Would malfunction hardware not create many errors with badblocks ?

>The reason I bring this up, is because the user is being led by thread into
>disabling error detection, which is the exact opposite of what should be done
>in the case of problems caused by malfunctioning storage device.

And the user (me) is a bit confused :-).

Bernd

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/7/8 04:48, Bernd Lentes wrote:
>
>> -----Original Message-----
>> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>> Sent: Thursday, July 6, 2023 8:20 AM
>> To: Bernd Lentes <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo
>> <wqu@suse.com>; linux-btrfs <linux-btrfs@vger.kernel.org>
>> Subject: Re: question to btrfs scrub
>
>
>
>> Checksum have two functions:
>>
>> - Detect errors
>>    As long as it's not a false alert, then it makes sense.
>>    But please keep in mind that, a csum mismatch will prevent anyone
>>    from reading that corrupted data, whether this is the expected
>>    behavior really depends on your workload.
>>
>>    E.g. if some archive with built-in error detect and recovery (like RAR
>>    files), it's definitely not a good idea to return -EIO for the whole
>>    block, other than reading out the corrupted data and let the software
>>    to handle them.
>>
>> - Recover the good data from the extra mirrors
>>    This only works if you're using profiles with duplication (DUP,
>>    RAID1*, RAID10, RAID5, RAID6).
>>    Otherwise btrfs won't be able to recover anything.
>>
>> In your case, since you're already using LVM, thus I believe the fs is using
>> default profiles (DUP for meta, SINGLE for data), thus there would be no extra
>> copy to recover from.
>>
>> So there is really error detection functionality lost if go nodatasum.
>>
>>> OK. I know which VM images produced checksum errors. I delete them and
>> restore them from the backup.
>>
>> You mentioned snapshots are utilized for those images, thus you have to
>> delete all the involved files, including ones in the snapshots.
>>
>>> So it finds errors in the data csum, correct ?
>>
>> Yes, either there are some files not deleted, or the file is snapshoted.
>
> I don't understand what you mean with that.

Is the file you deleted also in other snapshots?

And are you sure you deleted every file reported in dmesg by scrub?

Another possibility is, scrub reports can be ratelimited, thus some
files are not properly prompted.

Thanks,
Qu

>
> Thanks.
>
> Bernd
>
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

Re: question to btrfs scrub

[Qu Wenruo]

On 2023/7/8 04:54, Bernd Lentes wrote:
>> -----Original Message-----
>> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
>> Sent: Thursday, July 6, 2023 8:23 AM
>> To: Remi Gauvin <remi@georgianit.com>; Bernd Lentes
>> <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo <wqu@suse.com>;
>> linux-btrfs <linux-btrfs@vger.kernel.org>
>> Subject: Re: question to btrfs scrub
> 
> 
>> I hate to point my finger to btrfs itself, but I still remember in the old days
>> some workload can lead to such false alerts.
>> But I can not recall which commit is causing and which one is fixing the
>> problem.
>>
>> Another concern is, the report is using SINGLE for data, which is completely
>> fine, but it doesn't help us to determine if it's really a hardware data
>> corruption or btrfs bugs.
>>
>> If the report is using RAID1, and those corruption are all repairable, then we're
>> pretty sure it's data corruption on disk.
>> Or if all mirrors are corrupted in a RAID1* config, then we know it's definitely
>> btrfs causing the problem.
>>
>> But with SINGLE profile, it's really hard to say.
> 
> The server we are talking about is still in testing, not in production.
> That means we can change the config.
> Should we take RAID1 into account ?

Personally speaking, with your hardware I already believe it's not 
hardware causing the corruption.

But it can be extra safe if you go RAID1 for data, and re-test the workload.

If the corruption still happens, it's almost sure it's btrfs to blame.

Thanks,
Qu
> We have enough disk space.
> 
> Bernd
> 
> Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
> Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
> Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671

RE: question to btrfs scrub

[Bernd Lentes]

>
> On 2023/7/8 04:54, Bernd Lentes wrote:
> >> -----Original Message-----
> >> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> >> Sent: Thursday, July 6, 2023 8:23 AM
> >> To: Remi Gauvin <remi@georgianit.com>; Bernd Lentes
> >> <bernd.lentes@helmholtz-muenchen.de>; Qu Wenruo <wqu@suse.com>;
> >> linux-btrfs <linux-btrfs@vger.kernel.org>
> >> Subject: Re: question to btrfs scrub


> >> I hate to point my finger to btrfs itself, but I still remember in
> >> the old days some workload can lead to such false alerts.
> >> But I can not recall which commit is causing and which one is fixing
> >> the problem.

Dear Qu,

does that mean the bug is possibly not fixed ?
Uuuh.

Bernd

Helmholtz Zentrum München – Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH)
Ingolstädter Landstraße 1, D-85764 Neuherberg, https://www.helmholtz-munich.de
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Daniela Sommer (komm.) | Aufsichtsratsvorsitzende: MinDir’in Prof. Dr. Veronika von Messling
Registergericht: Amtsgericht München HRB 6466 | USt-IdNr. DE 129521671