clear space cache v1 fails with Unable to find block group for 0

clear space cache v1 fails with Unable to find block group for 0

[j4nn]

Hi,

I am trying to switch 8TB raid1 btrfs from space cache v1 to v2, but
the clear space cache v1 fails as following:

gentoo ~ # btrfs filesystem df /mnt/data
Data, RAID1: total=7.36TiB, used=7.00TiB
System, RAID1: total=64.00MiB, used=1.11MiB
Metadata, RAID1: total=63.00GiB, used=57.37GiB
Metadata, DUP: total=5.00GiB, used=1.18GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'
WARNING:    Metadata: raid1, dup
gentoo ~ # umount /mnt/data

gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
ERROR: failed to clear free space cache
extent buffer leak: start 9587384418304 len 16384

real    7m8.174s
user    0m6.883s
sys     0m9.322s


Here some info:

gentoo ~ # uname -a
Linux gentoo 6.12.3-gentoo-x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec  8
00:12:56 CET 2024 x86_64 AMD Ryzen 9 5950X 16-Core Processor
AuthenticAMD GNU/Linux
gentoo ~ # btrfs --version
btrfs-progs v6.12
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
gentoo ~ # btrfs filesystem show /mnt/data
Label: 'rdata'  uuid: 1dfac20a-3f84-4149-aba0-f160ab633373
       Total devices 2 FS bytes used 7.06TiB
       devid    1 size 8.00TiB used 7.26TiB path /dev/mapper/wdrb-bdata
       devid    2 size 8.00TiB used 7.25TiB path /dev/mapper/wdrc-cdata
gentoo ~ # dmesg | tail -n 6
[31008.980706] BTRFS info (device dm-0): first mount of filesystem
1dfac20a-3f84-4149-aba0-f160ab633373
[31008.980726] BTRFS info (device dm-0): using crc32c (crc32c-intel)
checksum algorithm
[31008.980731] BTRFS info (device dm-0): disk space caching is enabled
[31008.980734] BTRFS warning (device dm-0): space cache v1 is being
deprecated and will be removed in a future release, please use -o
space_cache=v2
[31009.994687] BTRFS info (device dm-0): bdev /dev/mapper/wdrb-bdata
errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[31009.994696] BTRFS info (device dm-0): bdev /dev/mapper/wdrc-cdata
errs: wr 7, rd 0, flush 0, corrupt 0, gen 0

Completed scrub (which corrected 4 errors), btrfs check completed
without errors:

gentoo ~ # btrfs scrub status /mnt/data
UUID:             1dfac20a-3f84-4149-aba0-f160ab633373
Scrub started:    Fri Dec  6 13:12:36 2024
Status:           finished
Duration:         16:11:22
Total to scrub:   14.92TiB
Rate:             268.35MiB/s
Error summary:    verify=4
 Corrected:      4
 Uncorrectable:  0
 Unverified:     0
gentoo ~ # umount /mnt/data

gentoo ~ # time btrfs check -p /dev/mapper/wdrb-bdata
Opening filesystem to check...
Checking filesystem on /dev/mapper/wdrb-bdata
UUID: 1dfac20a-3f84-4149-aba0-f160ab633373
[1/7] checking root items                      (0:06:57 elapsed,
34253945 items checked)
[2/7] checking extents                         (0:23:08 elapsed,
3999596 items checked)
[3/7] checking free space cache                (0:04:25 elapsed, 7868
items checked)
[4/7] checking fs roots                        (1:03:46 elapsed,
3215533 items checked)
[5/7] checking csums (without verifying data)  (0:11:58 elapsed,
15418322 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 52
items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 8199989936128 bytes used, no error found
total csum bytes: 7940889876
total tree bytes: 65528446976
total fs tree bytes: 52856799232
total extent tree bytes: 3578331136
btree space waste bytes: 10797983857
file data blocks allocated: 21632555483136
referenced 9547690319872

real    111m10.370s
user    10m28.442s
sys     6m44.888s

Tried some balance as found example posted, not really sure if that should help:

gentoo ~ # btrfs balance start -dusage=10 /mnt/data
Done, had to relocate 32 out of 7467 chunks

gentoo ~ # btrfs filesystem df /mnt/data
Data, RAID1: total=7.19TiB, used=7.00TiB
System, RAID1: total=64.00MiB, used=1.08MiB
Metadata, RAID1: total=63.00GiB, used=57.36GiB
Metadata, DUP: total=5.00GiB, used=1.18GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
WARNING: Multiple block group profiles detected, see 'man btrfs(5)'
WARNING:    Metadata: raid1, dup

But it did not help:

gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
Unable to find block group for 0
Unable to find block group for 0
Unable to find block group for 0
ERROR: failed to clear free space cache
extent buffer leak: start 7995086045184 len 16384

real    6m58.515s
user    0m6.270s
sys     0m9.586s

Any idea how to fix this?
Thanks.

Re: clear space cache v1 fails with Unable to find block group for 0

[Qu Wenruo]

在 2024/12/9 02:32, j4nn 写道:
> Hi,
>
> I am trying to switch 8TB raid1 btrfs from space cache v1 to v2, but
> the clear space cache v1 fails as following:
>
> gentoo ~ # btrfs filesystem df /mnt/data
> Data, RAID1: total=7.36TiB, used=7.00TiB
> System, RAID1: total=64.00MiB, used=1.11MiB
> Metadata, RAID1: total=63.00GiB, used=57.37GiB
> Metadata, DUP: total=5.00GiB, used=1.18GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> WARNING: Multiple block group profiles detected, see 'man btrfs(5)'
> WARNING:    Metadata: raid1, dup
> gentoo ~ # umount /mnt/data
>
> gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
> Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0

This is a common indicator of -ENOSPC.

But according to the fi df output, we should have quite a lot of
metadata space left.

The only concern is the DUP metadata, which may cause the space
reservation code not to work in progs.

Have you tried to convert the DUP metadata first?

And `btrfs fi usage` output please.

> ERROR: failed to clear free space cache
> extent buffer leak: start 9587384418304 len 16384
>
> real    7m8.174s
> user    0m6.883s
> sys     0m9.322s
>
>
> Here some info:
>
> gentoo ~ # uname -a
> Linux gentoo 6.12.3-gentoo-x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec  8
> 00:12:56 CET 2024 x86_64 AMD Ryzen 9 5950X 16-Core Processor
> AuthenticAMD GNU/Linux
> gentoo ~ # btrfs --version
> btrfs-progs v6.12
> -EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
> gentoo ~ # btrfs filesystem show /mnt/data
> Label: 'rdata'  uuid: 1dfac20a-3f84-4149-aba0-f160ab633373
>         Total devices 2 FS bytes used 7.06TiB
>         devid    1 size 8.00TiB used 7.26TiB path /dev/mapper/wdrb-bdata
>         devid    2 size 8.00TiB used 7.25TiB path /dev/mapper/wdrc-cdata
> gentoo ~ # dmesg | tail -n 6
> [31008.980706] BTRFS info (device dm-0): first mount of filesystem
> 1dfac20a-3f84-4149-aba0-f160ab633373
> [31008.980726] BTRFS info (device dm-0): using crc32c (crc32c-intel)
> checksum algorithm
> [31008.980731] BTRFS info (device dm-0): disk space caching is enabled
> [31008.980734] BTRFS warning (device dm-0): space cache v1 is being
> deprecated and will be removed in a future release, please use -o
> space_cache=v2
> [31009.994687] BTRFS info (device dm-0): bdev /dev/mapper/wdrb-bdata
> errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
> [31009.994696] BTRFS info (device dm-0): bdev /dev/mapper/wdrc-cdata
> errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
>
> Completed scrub (which corrected 4 errors), btrfs check completed
> without errors:
>
> gentoo ~ # btrfs scrub status /mnt/data
> UUID:             1dfac20a-3f84-4149-aba0-f160ab633373
> Scrub started:    Fri Dec  6 13:12:36 2024
> Status:           finished
> Duration:         16:11:22
> Total to scrub:   14.92TiB
> Rate:             268.35MiB/s
> Error summary:    verify=4
>   Corrected:      4
>   Uncorrectable:  0
>   Unverified:     0
> gentoo ~ # umount /mnt/data
>
> gentoo ~ # time btrfs check -p /dev/mapper/wdrb-bdata
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/wdrb-bdata
> UUID: 1dfac20a-3f84-4149-aba0-f160ab633373
> [1/7] checking root items                      (0:06:57 elapsed,
> 34253945 items checked)
> [2/7] checking extents                         (0:23:08 elapsed,
> 3999596 items checked)
> [3/7] checking free space cache                (0:04:25 elapsed, 7868
> items checked)
> [4/7] checking fs roots                        (1:03:46 elapsed,
> 3215533 items checked)
> [5/7] checking csums (without verifying data)  (0:11:58 elapsed,
> 15418322 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 52
> items checked)
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 8199989936128 bytes used, no error found
> total csum bytes: 7940889876
> total tree bytes: 65528446976
> total fs tree bytes: 52856799232
> total extent tree bytes: 3578331136
> btree space waste bytes: 10797983857
> file data blocks allocated: 21632555483136
> referenced 9547690319872
>
> real    111m10.370s
> user    10m28.442s
> sys     6m44.888s
>
> Tried some balance as found example posted, not really sure if that should help:
>
> gentoo ~ # btrfs balance start -dusage=10 /mnt/data
> Done, had to relocate 32 out of 7467 chunks

The balance doesn't do much, the overall chunk layout is still the same.
>
> gentoo ~ # btrfs filesystem df /mnt/data
> Data, RAID1: total=7.19TiB, used=7.00TiB
> System, RAID1: total=64.00MiB, used=1.08MiB
> Metadata, RAID1: total=63.00GiB, used=57.36GiB
> Metadata, DUP: total=5.00GiB, used=1.18GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> WARNING: Multiple block group profiles detected, see 'man btrfs(5)'
> WARNING:    Metadata: raid1, dup
>
> But it did not help:
>
> gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
> Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0
> ERROR: failed to clear free space cache
> extent buffer leak: start 7995086045184 len 16384
>
> real    6m58.515s
> user    0m6.270s
> sys     0m9.586s

Migrating to v2 cache doesn't really need to manually clear the v1 cache.

Just mounting with "space_cache=v2" option will automatically purge the
v1 cache, just as explained in the man page:

   If v2 is enabled, and v1 space cache will be cleared (at the first
   mount)

If you want to dig deeper, the implementation is done in
btrfs_set_free_space_cache_v1_active() which calls
cleanup_free_space_cache_v1() if @active is false.

Thanks,
Qu
>
> Any idea how to fix this?
> Thanks.
>

Re: clear space cache v1 fails with Unable to find block group for 0

[j4nn]

On Sun, 8 Dec 2024 at 21:26, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 在 2024/12/9 02:32, j4nn 写道:
> > gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
> > Unable to find block group for 0
> > Unable to find block group for 0
> > Unable to find block group for 0
>
> This is a common indicator of -ENOSPC.
>
> But according to the fi df output, we should have quite a lot of
> metadata space left.
>
> The only concern is the DUP metadata, which may cause the space
> reservation code not to work in progs.
>
> Have you tried to convert the DUP metadata first?

I am not sure how to do that.
I see the "Multiple block group profiles detected" warning, assumed it
is about metadata in RAID1 and DUP.
But I am not sure how that got created or if it has any benefit or not.
And what that DUP should be converted into?

> And `btrfs fi usage` output please.

gentoo ~ # btrfs fi usage /mnt/data
Overall:
   Device size:                  16.00TiB
   Device allocated:             14.51TiB
   Device unallocated:            1.48TiB
   Device missing:                  0.00B
   Device slack:                    0.00B
   Used:                         14.18TiB
   Free (estimated):            923.95GiB      (min: 923.95GiB)
   Free (statfs, df):           918.95GiB
   Data ratio:                       2.00
   Metadata ratio:                   2.00
   Global reserve:              512.00MiB      (used: 0.00B)
   Multiple profiles:                 yes      (metadata)

Data,RAID1: Size:7.19TiB, Used:7.03TiB (97.77%)
  /dev/mapper/wdrb-bdata          7.19TiB
  /dev/mapper/wdrc-cdata          7.19TiB

Metadata,RAID1: Size:63.00GiB, Used:58.56GiB (92.95%)
  /dev/mapper/wdrb-bdata         63.00GiB
  /dev/mapper/wdrc-cdata         63.00GiB

Metadata,DUP: Size:5.00GiB, Used:1.18GiB (23.60%)
  /dev/mapper/wdrb-bdata         10.00GiB

System,RAID1: Size:32.00MiB, Used:1.08MiB (3.37%)
  /dev/mapper/wdrb-bdata         32.00MiB
  /dev/mapper/wdrc-cdata         32.00MiB

Unallocated:
  /dev/mapper/wdrb-bdata        755.00GiB
  /dev/mapper/wdrc-cdata        765.00GiB

gentoo ~ # lvs
 LV     VG     Attr       LSize    Pool Origin Data%  Meta%  Move Log
Cpy%Sync Convert
 bdata  wdrb   -wi-ao----    8.00t
 cdata  wdrc   -wi-ao----    8.00t
gentoo ~ # vgs
 VG     #PV #LV #SN Attr   VSize    VFree
 wdrb     1   1   0 wz--n-   <9.10t <1.10t
 wdrc     1   3   0 wz--n-    9.09t     0
gentoo ~ # pvs
 PV         VG     Fmt  Attr PSize    PFree
 /dev/sdb1  wdrc   lvm2 a--     9.09t     0
 /dev/sdd1  wdrb   lvm2 a--    <9.10t <1.10t


> > Tried some balance as found example posted, not really sure if that should help:
> >
> > gentoo ~ # btrfs balance start -dusage=10 /mnt/data
> > Done, had to relocate 32 out of 7467 chunks
>
> The balance doesn't do much, the overall chunk layout is still the same.
> >
> > gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
> > Unable to find block group for 0
> > Unable to find block group for 0
> > Unable to find block group for 0
> > ERROR: failed to clear free space cache
> > extent buffer leak: start 7995086045184 len 16384
>
> Migrating to v2 cache doesn't really need to manually clear the v1 cache.
>
> Just mounting with "space_cache=v2" option will automatically purge the
> v1 cache, just as explained in the man page:
>
>    If v2 is enabled, and v1 space cache will be cleared (at the first
>    mount)
>
> If you want to dig deeper, the implementation is done in
> btrfs_set_free_space_cache_v1_active() which calls
> cleanup_free_space_cache_v1() if @active is false.

Ok, I just followed a howto for the switch.
Did not know it is ok just with the mount option.
Should it be safe to try it if I get the errors with the "btrfs rescue
clear-space-cache v1"?

Thank you.

Re: clear space cache v1 fails with Unable to find block group for 0

[Qu Wenruo]

在 2024/12/9 07:55, j4nn 写道:
> On Sun, 8 Dec 2024 at 21:26, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>> 在 2024/12/9 02:32, j4nn 写道:
>>> gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
>>> Unable to find block group for 0
>>> Unable to find block group for 0
>>> Unable to find block group for 0
>>
>> This is a common indicator of -ENOSPC.
>>
>> But according to the fi df output, we should have quite a lot of
>> metadata space left.
>>
>> The only concern is the DUP metadata, which may cause the space
>> reservation code not to work in progs.
>>
>> Have you tried to convert the DUP metadata first?
>
> I am not sure how to do that.
> I see the "Multiple block group profiles detected" warning, assumed it
> is about metadata in RAID1 and DUP.
> But I am not sure how that got created or if it has any benefit or not.
> And what that DUP should be converted into?

Not sure either. But I guess in the past you mounted the device with one
disk missing, and did some writes.
And those writes by incident created a new chunk, and in that case the
new chunk are only seeing one writable disk, so it went DUP.


To remove it, you need specific balance filter, e.g

  # btrfs balance start -mprofiles=dup,convert=raid1 /mnt/data

>
>> And `btrfs fi usage` output please.
>
> gentoo ~ # btrfs fi usage /mnt/data
> Overall:
>     Device size:                  16.00TiB
>     Device allocated:             14.51TiB
>     Device unallocated:            1.48TiB
>     Device missing:                  0.00B
>     Device slack:                    0.00B
>     Used:                         14.18TiB
>     Free (estimated):            923.95GiB      (min: 923.95GiB)
>     Free (statfs, df):           918.95GiB
>     Data ratio:                       2.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                 yes      (metadata)
>
> Data,RAID1: Size:7.19TiB, Used:7.03TiB (97.77%)
>    /dev/mapper/wdrb-bdata          7.19TiB
>    /dev/mapper/wdrc-cdata          7.19TiB
>
> Metadata,RAID1: Size:63.00GiB, Used:58.56GiB (92.95%)
>    /dev/mapper/wdrb-bdata         63.00GiB
>    /dev/mapper/wdrc-cdata         63.00GiB
>
> Metadata,DUP: Size:5.00GiB, Used:1.18GiB (23.60%)
>    /dev/mapper/wdrb-bdata         10.00GiB
>
> System,RAID1: Size:32.00MiB, Used:1.08MiB (3.37%)
>    /dev/mapper/wdrb-bdata         32.00MiB
>    /dev/mapper/wdrc-cdata         32.00MiB
>
> Unallocated:
>    /dev/mapper/wdrb-bdata        755.00GiB
>    /dev/mapper/wdrc-cdata        765.00GiB

You have more than enough space to remove the DUP chunks.

>
> gentoo ~ # lvs
>   LV     VG     Attr       LSize    Pool Origin Data%  Meta%  Move Log
> Cpy%Sync Convert
>   bdata  wdrb   -wi-ao----    8.00t
>   cdata  wdrc   -wi-ao----    8.00t
> gentoo ~ # vgs
>   VG     #PV #LV #SN Attr   VSize    VFree
>   wdrb     1   1   0 wz--n-   <9.10t <1.10t
>   wdrc     1   3   0 wz--n-    9.09t     0
> gentoo ~ # pvs
>   PV         VG     Fmt  Attr PSize    PFree
>   /dev/sdb1  wdrc   lvm2 a--     9.09t     0
>   /dev/sdd1  wdrb   lvm2 a--    <9.10t <1.10t
>
>
>>> Tried some balance as found example posted, not really sure if that should help:
>>>
>>> gentoo ~ # btrfs balance start -dusage=10 /mnt/data
>>> Done, had to relocate 32 out of 7467 chunks
>>
>> The balance doesn't do much, the overall chunk layout is still the same.
>>>
>>> gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
>>> Unable to find block group for 0
>>> Unable to find block group for 0
>>> Unable to find block group for 0
>>> ERROR: failed to clear free space cache
>>> extent buffer leak: start 7995086045184 len 16384
>>
>> Migrating to v2 cache doesn't really need to manually clear the v1 cache.
>>
>> Just mounting with "space_cache=v2" option will automatically purge the
>> v1 cache, just as explained in the man page:
>>
>>     If v2 is enabled, and v1 space cache will be cleared (at the first
>>     mount)
>>
>> If you want to dig deeper, the implementation is done in
>> btrfs_set_free_space_cache_v1_active() which calls
>> cleanup_free_space_cache_v1() if @active is false.
>
> Ok, I just followed a howto for the switch.
> Did not know it is ok just with the mount option.
> Should it be safe to try it if I get the errors with the "btrfs rescue
> clear-space-cache v1"?

Since progs and kernel have different implementations on the space
reservation code, it's not that rare to hit cases where btrfs-progs hits
some false alerts.

If you balanced removed the DUP profile, then you can try "btrfs rescue"
again, just to see if it works and I really appreciate the extra
feedback to help debugging the progs bug.

Otherwise I believe it should be pretty safe just using "space_cache=v2"
mount option.

Thanks,
Qu
>
> Thank you.

Re: clear space cache v1 fails with Unable to find block group for 0

[j4nn]

On Sun, 8 Dec 2024 at 22:36, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 在 2024/12/9 07:55, j4nn 写道:
> > On Sun, 8 Dec 2024 at 21:26, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >> 在 2024/12/9 02:32, j4nn 写道:
> >>> gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
> >>> Unable to find block group for 0
> >>> Unable to find block group for 0
> >>> Unable to find block group for 0
> >>
> >> This is a common indicator of -ENOSPC.
> >>
> >> But according to the fi df output, we should have quite a lot of
> >> metadata space left.
> >>
> >> The only concern is the DUP metadata, which may cause the space
> >> reservation code not to work in progs.
> >>
> >> Have you tried to convert the DUP metadata first?
> >
> > I am not sure how to do that.
> > I see the "Multiple block group profiles detected" warning, assumed it
> > is about metadata in RAID1 and DUP.
> > But I am not sure how that got created or if it has any benefit or not.
> > And what that DUP should be converted into?
>
> Not sure either. But I guess in the past you mounted the device with one
> disk missing, and did some writes.
> And those writes by incident created a new chunk, and in that case the
> new chunk are only seeing one writable disk, so it went DUP.
>
>
> To remove it, you need specific balance filter, e.g
>
>   # btrfs balance start -mprofiles=dup,convert=raid1 /mnt/data

Thank you very much - it worked!

gentoo ~ # btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
Free space cache cleared

[55392.407421] BTRFS info (device dm-0): balance: start
-mconvert=raid1,profiles=dup -sconvert=raid1,profiles=dup
[55392.480051] BTRFS info (device dm-0): relocating block group
3216960913408 flags metadata|dup
[55431.499765] BTRFS info (device dm-0): found 27769 extents, stage:
move data extents
[55434.081160] BTRFS info (device dm-0): relocating block group
1142491709440 flags metadata|dup
[55464.555860] BTRFS info (device dm-0): found 9152 extents, stage:
move data extents
[55466.614881] BTRFS info (device dm-0): relocating block group
1112426938368 flags metadata|dup
[55497.322148] BTRFS info (device dm-0): found 9925 extents, stage:
move data extents
[55499.938966] BTRFS info (device dm-0): relocating block group
1079140941824 flags metadata|dup
[55520.415801] BTRFS info (device dm-0): found 14986 extents, stage:
move data extents
[55521.796855] BTRFS info (device dm-0): relocating block group
746280976384 flags metadata|dup
[55548.800335] BTRFS info (device dm-0): found 15430 extents, stage:
move data extents
[55550.291120] BTRFS info (device dm-0): balance: ended with status: 0
[55849.661892] BTRFS info (device dm-0): last unmount of filesystem
1dfac20a-3f84-4149-aba0-f160ab633373

gentoo ~ # vi /etc/fstab
gentoo ~ # systemctl daemon-reload
gentoo ~ # mount /mnt/data

[56068.187456] BTRFS info (device dm-0): first mount of filesystem
1dfac20a-3f84-4149-aba0-f160ab633373
[56068.187473] BTRFS info (device dm-0): using crc32c (crc32c-intel)
checksum algorithm
[56068.187477] BTRFS info (device dm-0): using free-space-tree
[56068.823725] BTRFS info (device dm-0): bdev /dev/mapper/wdrb-bdata
errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[56068.823733] BTRFS info (device dm-0): bdev /dev/mapper/wdrc-cdata
errs: wr 7, rd 0, flush 0, corrupt 0, gen 0

Just wondering what the "errs: wr 8" and "errs: wr 7" mean here?


>
> You have more than enough space to remove the DUP chunks.
>
> > Ok, I just followed a howto for the switch.
> > Did not know it is ok just with the mount option.
> > Should it be safe to try it if I get the errors with the "btrfs rescue
> > clear-space-cache v1"?
>
> Since progs and kernel have different implementations on the space
> reservation code, it's not that rare to hit cases where btrfs-progs hits
> some false alerts.
>
> If you balanced removed the DUP profile, then you can try "btrfs rescue"
> again, just to see if it works and I really appreciate the extra
> feedback to help debugging the progs bug.

Yes, it worked - thanks again.
Here some extra feedback:

gentoo ~ # btrfs fi usage /mnt/data
Overall:
   Device size:                  16.00TiB
   Device allocated:             14.55TiB
   Device unallocated:            1.45TiB
   Device missing:                  0.00B
   Device slack:                    0.00B
   Used:                         14.18TiB
   Free (estimated):            906.41GiB      (min: 906.41GiB)
   Free (statfs, df):           906.41GiB
   Data ratio:                       2.00
   Metadata ratio:                   2.00
   Global reserve:              512.00MiB      (used: 0.00B)
   Multiple profiles:                  no

Data,RAID1: Size:7.19TiB, Used:7.03TiB (97.77%)
  /dev/mapper/wdrb-bdata          7.19TiB
  /dev/mapper/wdrc-cdata          7.19TiB

Metadata,RAID1: Size:86.00GiB, Used:59.74GiB (69.46%)
  /dev/mapper/wdrb-bdata         86.00GiB
  /dev/mapper/wdrc-cdata         86.00GiB

System,RAID1: Size:32.00MiB, Used:1.08MiB (3.37%)
  /dev/mapper/wdrb-bdata         32.00MiB
  /dev/mapper/wdrc-cdata         32.00MiB

Unallocated:
  /dev/mapper/wdrb-bdata        742.00GiB
  /dev/mapper/wdrc-cdata        742.00GiB

gentoo ~ # btrfs filesystem df /mnt/data
Data, RAID1: total=7.19TiB, used=7.03TiB
System, RAID1: total=32.00MiB, used=1.08MiB
Metadata, RAID1: total=86.00GiB, used=59.74GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

gentoo ~ # mount -o rw,remount /mnt/data

[56709.279574] BTRFS info (device dm-0 state M): creating free space tree
[56893.200005] INFO: task btrfs-transacti:2510362 blocked for more
than 122 seconds.
[56893.200011]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
...
[57061.541085] BTRFS info (device dm-0 state M): setting compat-ro
feature flag for FREE_SPACE_TREE (0x1)
[57061.541091] BTRFS info (device dm-0 state M): setting compat-ro
feature flag for FREE_SPACE_TREE_VALID (0x2)
[57062.642266] BTRFS info (device dm-0 state M): cleaning free space cache v1

gentoo ~ # btrfs filesystem df /mnt/data
Data, RAID1: total=7.19TiB, used=7.03TiB
System, RAID1: total=32.00MiB, used=1.08MiB
Metadata, RAID1: total=64.00GiB, used=59.75GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

gentoo ~ # btrfs fi usage /mnt/data
Overall:
   Device size:                  16.00TiB
   Device allocated:             14.51TiB
   Device unallocated:            1.49TiB
   Device missing:                  0.00B
   Device slack:                    0.00B
   Used:                         14.18TiB
   Free (estimated):            928.41GiB      (min: 928.41GiB)
   Free (statfs, df):           928.41GiB
   Data ratio:                       2.00
   Metadata ratio:                   2.00
   Global reserve:              512.00MiB      (used: 0.00B)
   Multiple profiles:                  no

Data,RAID1: Size:7.19TiB, Used:7.03TiB (97.77%)
  /dev/mapper/wdrb-bdata          7.19TiB
  /dev/mapper/wdrc-cdata          7.19TiB

Metadata,RAID1: Size:64.00GiB, Used:59.75GiB (93.35%)
  /dev/mapper/wdrb-bdata         64.00GiB
  /dev/mapper/wdrc-cdata         64.00GiB

System,RAID1: Size:32.00MiB, Used:1.08MiB (3.37%)
  /dev/mapper/wdrb-bdata         32.00MiB
  /dev/mapper/wdrc-cdata         32.00MiB

Unallocated:
  /dev/mapper/wdrb-bdata        764.00GiB
  /dev/mapper/wdrc-cdata        764.00GiB

>
> Otherwise I believe it should be pretty safe just using "space_cache=v2"
> mount option.
>
> Thanks,
> Qu
> >
> > Thank you.
>

Re: clear space cache v1 fails with Unable to find block group for 0

[Qu Wenruo]

在 2024/12/9 08:49, j4nn 写道:
> On Sun, 8 Dec 2024 at 22:36, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>> 在 2024/12/9 07:55, j4nn 写道:
>>> On Sun, 8 Dec 2024 at 21:26, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>> 在 2024/12/9 02:32, j4nn 写道:
>>>>> gentoo ~ # time btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
>>>>> Unable to find block group for 0
>>>>> Unable to find block group for 0
>>>>> Unable to find block group for 0
>>>>
>>>> This is a common indicator of -ENOSPC.
>>>>
>>>> But according to the fi df output, we should have quite a lot of
>>>> metadata space left.
>>>>
>>>> The only concern is the DUP metadata, which may cause the space
>>>> reservation code not to work in progs.
>>>>
>>>> Have you tried to convert the DUP metadata first?
>>>
>>> I am not sure how to do that.
>>> I see the "Multiple block group profiles detected" warning, assumed it
>>> is about metadata in RAID1 and DUP.
>>> But I am not sure how that got created or if it has any benefit or not.
>>> And what that DUP should be converted into?
>>
>> Not sure either. But I guess in the past you mounted the device with one
>> disk missing, and did some writes.
>> And those writes by incident created a new chunk, and in that case the
>> new chunk are only seeing one writable disk, so it went DUP.
>>
>>
>> To remove it, you need specific balance filter, e.g
>>
>>    # btrfs balance start -mprofiles=dup,convert=raid1 /mnt/data
>
> Thank you very much - it worked!
>
> gentoo ~ # btrfs rescue clear-space-cache v1 /dev/mapper/wdrb-bdata
> Free space cache cleared
>
> [55392.407421] BTRFS info (device dm-0): balance: start
> -mconvert=raid1,profiles=dup -sconvert=raid1,profiles=dup
> [55392.480051] BTRFS info (device dm-0): relocating block group
> 3216960913408 flags metadata|dup
> [55431.499765] BTRFS info (device dm-0): found 27769 extents, stage:
> move data extents
> [55434.081160] BTRFS info (device dm-0): relocating block group
> 1142491709440 flags metadata|dup
> [55464.555860] BTRFS info (device dm-0): found 9152 extents, stage:
> move data extents
> [55466.614881] BTRFS info (device dm-0): relocating block group
> 1112426938368 flags metadata|dup
> [55497.322148] BTRFS info (device dm-0): found 9925 extents, stage:
> move data extents
> [55499.938966] BTRFS info (device dm-0): relocating block group
> 1079140941824 flags metadata|dup
> [55520.415801] BTRFS info (device dm-0): found 14986 extents, stage:
> move data extents
> [55521.796855] BTRFS info (device dm-0): relocating block group
> 746280976384 flags metadata|dup
> [55548.800335] BTRFS info (device dm-0): found 15430 extents, stage:
> move data extents
> [55550.291120] BTRFS info (device dm-0): balance: ended with status: 0
> [55849.661892] BTRFS info (device dm-0): last unmount of filesystem
> 1dfac20a-3f84-4149-aba0-f160ab633373
>
> gentoo ~ # vi /etc/fstab
> gentoo ~ # systemctl daemon-reload
> gentoo ~ # mount /mnt/data
>
> [56068.187456] BTRFS info (device dm-0): first mount of filesystem
> 1dfac20a-3f84-4149-aba0-f160ab633373
> [56068.187473] BTRFS info (device dm-0): using crc32c (crc32c-intel)
> checksum algorithm
> [56068.187477] BTRFS info (device dm-0): using free-space-tree
> [56068.823725] BTRFS info (device dm-0): bdev /dev/mapper/wdrb-bdata
> errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
> [56068.823733] BTRFS info (device dm-0): bdev /dev/mapper/wdrc-cdata
> errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
>
> Just wondering what the "errs: wr 8" and "errs: wr 7" mean here?

It means in the past, there are 8 write failures for the device
wdrb-bdata, and 7 write failures for device wdrc-cdata.

The error is from the lower layer, thus btrfs has no idea what happened.
You may need to check the log to find out why.

>
>
>>
>> You have more than enough space to remove the DUP chunks.
>>
>>> Ok, I just followed a howto for the switch.
>>> Did not know it is ok just with the mount option.
>>> Should it be safe to try it if I get the errors with the "btrfs rescue
>>> clear-space-cache v1"?
>>
>> Since progs and kernel have different implementations on the space
>> reservation code, it's not that rare to hit cases where btrfs-progs hits
>> some false alerts.
>>
>> If you balanced removed the DUP profile, then you can try "btrfs rescue"
>> again, just to see if it works and I really appreciate the extra
>> feedback to help debugging the progs bug.
>
> Yes, it worked - thanks again.
> Here some extra feedback:
>
> gentoo ~ # btrfs fi usage /mnt/data
> Overall:
>     Device size:                  16.00TiB
>     Device allocated:             14.55TiB
>     Device unallocated:            1.45TiB
>     Device missing:                  0.00B
>     Device slack:                    0.00B
>     Used:                         14.18TiB
>     Free (estimated):            906.41GiB      (min: 906.41GiB)
>     Free (statfs, df):           906.41GiB
>     Data ratio:                       2.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
>
> Data,RAID1: Size:7.19TiB, Used:7.03TiB (97.77%)
>    /dev/mapper/wdrb-bdata          7.19TiB
>    /dev/mapper/wdrc-cdata          7.19TiB
>
> Metadata,RAID1: Size:86.00GiB, Used:59.74GiB (69.46%)
>    /dev/mapper/wdrb-bdata         86.00GiB
>    /dev/mapper/wdrc-cdata         86.00GiB
>
> System,RAID1: Size:32.00MiB, Used:1.08MiB (3.37%)
>    /dev/mapper/wdrb-bdata         32.00MiB
>    /dev/mapper/wdrc-cdata         32.00MiB
>
> Unallocated:
>    /dev/mapper/wdrb-bdata        742.00GiB
>    /dev/mapper/wdrc-cdata        742.00GiB
>
> gentoo ~ # btrfs filesystem df /mnt/data
> Data, RAID1: total=7.19TiB, used=7.03TiB
> System, RAID1: total=32.00MiB, used=1.08MiB
> Metadata, RAID1: total=86.00GiB, used=59.74GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> gentoo ~ # mount -o rw,remount /mnt/data
>
> [56709.279574] BTRFS info (device dm-0 state M): creating free space tree
> [56893.200005] INFO: task btrfs-transacti:2510362 blocked for more
> than 122 seconds.
> [56893.200011]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1

I'm a little concerned about this.

Mind to share the full dmesg?

My current guess is, the v1 cache clearing is taking too much time for
the initial mount.
We should handle it properly or it may cause false alerts for future
migration.

Thanks,
Qu

> ...
> [57061.541085] BTRFS info (device dm-0 state M): setting compat-ro
> feature flag for FREE_SPACE_TREE (0x1)
> [57061.541091] BTRFS info (device dm-0 state M): setting compat-ro
> feature flag for FREE_SPACE_TREE_VALID (0x2)
> [57062.642266] BTRFS info (device dm-0 state M): cleaning free space cache v1
>
> gentoo ~ # btrfs filesystem df /mnt/data
> Data, RAID1: total=7.19TiB, used=7.03TiB
> System, RAID1: total=32.00MiB, used=1.08MiB
> Metadata, RAID1: total=64.00GiB, used=59.75GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> gentoo ~ # btrfs fi usage /mnt/data
> Overall:
>     Device size:                  16.00TiB
>     Device allocated:             14.51TiB
>     Device unallocated:            1.49TiB
>     Device missing:                  0.00B
>     Device slack:                    0.00B
>     Used:                         14.18TiB
>     Free (estimated):            928.41GiB      (min: 928.41GiB)
>     Free (statfs, df):           928.41GiB
>     Data ratio:                       2.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
>
> Data,RAID1: Size:7.19TiB, Used:7.03TiB (97.77%)
>    /dev/mapper/wdrb-bdata          7.19TiB
>    /dev/mapper/wdrc-cdata          7.19TiB
>
> Metadata,RAID1: Size:64.00GiB, Used:59.75GiB (93.35%)
>    /dev/mapper/wdrb-bdata         64.00GiB
>    /dev/mapper/wdrc-cdata         64.00GiB
>
> System,RAID1: Size:32.00MiB, Used:1.08MiB (3.37%)
>    /dev/mapper/wdrb-bdata         32.00MiB
>    /dev/mapper/wdrc-cdata         32.00MiB
>
> Unallocated:
>    /dev/mapper/wdrb-bdata        764.00GiB
>    /dev/mapper/wdrc-cdata        764.00GiB
>
>>
>> Otherwise I believe it should be pretty safe just using "space_cache=v2"
>> mount option.
>>
>> Thanks,
>> Qu
>>>
>>> Thank you.
>>
>

Re: clear space cache v1 fails with Unable to find block group for 0

[j4nn]

On Sun, 8 Dec 2024 at 23:32, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 在 2024/12/9 08:49, j4nn 写道:
> > On Sun, 8 Dec 2024 at 22:36, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >> 在 2024/12/9 07:55, j4nn 写道:
> >>> On Sun, 8 Dec 2024 at 21:26, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>> 在 2024/12/9 02:32, j4nn 写道:
> > [56068.823725] BTRFS info (device dm-0): bdev /dev/mapper/wdrb-bdata
> > errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
> > [56068.823733] BTRFS info (device dm-0): bdev /dev/mapper/wdrc-cdata
> > errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
> >
> > Just wondering what the "errs: wr 8" and "errs: wr 7" mean here?
>
> It means in the past, there are 8 write failures for the device
> wdrb-bdata, and 7 write failures for device wdrc-cdata.
>
> The error is from the lower layer, thus btrfs has no idea what happened.
> You may need to check the log to find out why.

Thank you for the explanation.
I guess that must have been from the time when replacing cpu - my
motherboard has flaky sata connectors and it's really a pain to
"position" sata cables in a way that does not cause sata errors in
kernel logs.


> > gentoo ~ # mount -o rw,remount /mnt/data
> >
> > [56709.279574] BTRFS info (device dm-0 state M): creating free space tree
> > [56893.200005] INFO: task btrfs-transacti:2510362 blocked for more
> > than 122 seconds.
> > [56893.200011]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
>
> I'm a little concerned about this.
>
> Mind to share the full dmesg?
>

[56068.187456] BTRFS info (device dm-0): first mount of filesystem
1dfac20a-3f84-4149-aba0-f160ab633373
[56068.187473] BTRFS info (device dm-0): using crc32c (crc32c-intel)
checksum algorithm
[56068.187477] BTRFS info (device dm-0): using free-space-tree
[56068.823725] BTRFS info (device dm-0): bdev /dev/mapper/wdrb-bdata
errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[56068.823733] BTRFS info (device dm-0): bdev /dev/mapper/wdrc-cdata
errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
[56709.279574] BTRFS info (device dm-0 state M): creating free space tree
[56893.200005] INFO: task btrfs-transacti:2510362 blocked for more
than 122 seconds.
[56893.200011]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[56893.200014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[56893.200016] task:btrfs-transacti state:D stack:0     pid:2510362
tgid:2510362 ppid:2      flags:0x00004000
[56893.200021] Call Trace:
[56893.200023]  <TASK>
[56893.200028]  __schedule+0x3f0/0xbd0
[56893.200037]  schedule+0x27/0xf0
[56893.200043]  btrfs_commit_transaction+0xc27/0xe80 [btrfs]
[56893.200086]  ? start_transaction+0xc0/0x820 [btrfs]
[56893.200120]  ? __pfx_autoremove_wake_function+0x10/0x10
[56893.200126]  transaction_kthread+0x159/0x1c0 [btrfs]
[56893.200160]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[56893.200198]  kthread+0xd2/0x100
[56893.200204]  ? __pfx_kthread+0x10/0x10
[56893.200209]  ret_from_fork+0x34/0x50
[56893.200214]  ? __pfx_kthread+0x10/0x10
[56893.200217]  ret_from_fork_asm+0x1a/0x30
[56893.200224]  </TASK>
[57061.541085] BTRFS info (device dm-0 state M): setting compat-ro
feature flag for FREE_SPACE_TREE (0x1)
[57061.541091] BTRFS info (device dm-0 state M): setting compat-ro
feature flag for FREE_SPACE_TREE_VALID (0x2)
[57062.642266] BTRFS info (device dm-0 state M): cleaning free space cache v1

> My current guess is, the v1 cache clearing is taking too much time for
> the initial mount.

I assumed the same, so I used the suggested echo to avoid repeating
that backtrace.
The initial mount after removal of v1 has been read only, the
backtrace happened when remounting rw, which started to create the
free space tree.

> We should handle it properly or it may cause false alerts for future
> migration.

Hope that helps...
Thank you.

Re: clear space cache v1 fails with Unable to find block group for 0

[j4nn]

This is unrelated, but as you have been interested in the hung task
backtrace, I got two more when using "btrfs send ... | btrfs receive
..." to copy 7TB of data from one btrfs disk to another one (still in
progress, both rotational hard drives):
[81837.347137] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[81837.347144]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[81837.347147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[81837.347149] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[81837.347154] Call Trace:
[81837.347156]  <TASK>
[81837.347161]  __schedule+0x3f0/0xbd0
[81837.347170]  schedule+0x27/0xf0
[81837.347174]  io_schedule+0x46/0x70
[81837.347177]  folio_wait_bit_common+0x123/0x340
[81837.347184]  ? __pfx_wake_page_function+0x10/0x10
[81837.347189]  folio_wait_writeback+0x2b/0x80
[81837.347193]  __filemap_fdatawait_range+0x7d/0xd0
[81837.347201]  filemap_fdatawait_range+0x12/0x20
[81837.347206]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[81837.347250]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[81837.347286]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[81837.347319]  ? start_transaction+0xc0/0x820 [btrfs]
[81837.347353]  transaction_kthread+0x159/0x1c0 [btrfs]
[81837.347386]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[81837.347418]  kthread+0xd2/0x100
[81837.347423]  ? __pfx_kthread+0x10/0x10
[81837.347427]  ret_from_fork+0x34/0x50
[81837.347430]  ? __pfx_kthread+0x10/0x10
[81837.347434]  ret_from_fork_asm+0x1a/0x30
[81837.347440]  </TASK>
[82205.983491] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[82205.983498]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[82205.983501] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[82205.983503] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[82205.983508] Call Trace:
[82205.983511]  <TASK>
[82205.983515]  __schedule+0x3f0/0xbd0
[82205.983524]  schedule+0x27/0xf0
[82205.983528]  io_schedule+0x46/0x70
[82205.983531]  folio_wait_bit_common+0x123/0x340
[82205.983538]  ? __pfx_wake_page_function+0x10/0x10
[82205.983543]  folio_wait_writeback+0x2b/0x80
[82205.983546]  __filemap_fdatawait_range+0x7d/0xd0
[82205.983551]  ? srso_alias_return_thunk+0x5/0xfbef5
[82205.983555]  ? __slab_free+0xbf/0x2c0
[82205.983560]  ? srso_alias_return_thunk+0x5/0xfbef5
[82205.983563]  ? kmem_cache_alloc_noprof+0x201/0x2a0
[82205.983567]  ? clear_state_bit+0xfc/0x160 [btrfs]
[82205.983609]  ? srso_alias_return_thunk+0x5/0xfbef5
[82205.983612]  ? __clear_extent_bit+0x160/0x490 [btrfs]
[82205.983646]  filemap_fdatawait_range+0x12/0x20
[82205.983650]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[82205.983689]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[82205.983725]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[82205.983759]  ? start_transaction+0xc0/0x820 [btrfs]
[82205.983793]  transaction_kthread+0x159/0x1c0 [btrfs]
[82205.983827]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[82205.983859]  kthread+0xd2/0x100
[82205.983864]  ? __pfx_kthread+0x10/0x10
[82205.983867]  ret_from_fork+0x34/0x50
[82205.983871]  ? __pfx_kthread+0x10/0x10
[82205.983874]  ret_from_fork_asm+0x1a/0x30
[82205.983881]  </TASK>

Let me know please if you need some more info.

Re: clear space cache v1 fails with Unable to find block group for 0

[j4nn]

On Mon, 16 Dec 2024 at 19:52, j4nn <j4nn.xda@gmail.com> wrote:
>
> This is unrelated, but as you have been interested in the hung task
> backtrace, I got two more when using "btrfs send ... | btrfs receive
> ..." to copy 7TB of data from one btrfs disk to another one (still in
> progress, both rotational hard drives):
> [81837.347137] INFO: task btrfs-transacti:29385 blocked for more than
> 122 seconds.
> [81837.347144]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
> [81837.347147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [81837.347149] task:btrfs-transacti state:D stack:0     pid:29385
> tgid:29385 ppid:2      flags:0x00004000
> [81837.347154] Call Trace:
> [81837.347156]  <TASK>
> [81837.347161]  __schedule+0x3f0/0xbd0
> [81837.347170]  schedule+0x27/0xf0
> [81837.347174]  io_schedule+0x46/0x70
> [81837.347177]  folio_wait_bit_common+0x123/0x340
> [81837.347184]  ? __pfx_wake_page_function+0x10/0x10
> [81837.347189]  folio_wait_writeback+0x2b/0x80
> [81837.347193]  __filemap_fdatawait_range+0x7d/0xd0
> [81837.347201]  filemap_fdatawait_range+0x12/0x20
> [81837.347206]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
> [81837.347250]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
> [81837.347286]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
> [81837.347319]  ? start_transaction+0xc0/0x820 [btrfs]
> [81837.347353]  transaction_kthread+0x159/0x1c0 [btrfs]
> [81837.347386]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
> [81837.347418]  kthread+0xd2/0x100
> [81837.347423]  ? __pfx_kthread+0x10/0x10
> [81837.347427]  ret_from_fork+0x34/0x50
> [81837.347430]  ? __pfx_kthread+0x10/0x10
> [81837.347434]  ret_from_fork_asm+0x1a/0x30
> [81837.347440]  </TASK>
> [82205.983491] INFO: task btrfs-transacti:29385 blocked for more than
> 122 seconds.
> [82205.983498]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
> [82205.983501] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [82205.983503] task:btrfs-transacti state:D stack:0     pid:29385
> tgid:29385 ppid:2      flags:0x00004000
> [82205.983508] Call Trace:
> [82205.983511]  <TASK>
> [82205.983515]  __schedule+0x3f0/0xbd0
> [82205.983524]  schedule+0x27/0xf0
> [82205.983528]  io_schedule+0x46/0x70
> [82205.983531]  folio_wait_bit_common+0x123/0x340
> [82205.983538]  ? __pfx_wake_page_function+0x10/0x10
> [82205.983543]  folio_wait_writeback+0x2b/0x80
> [82205.983546]  __filemap_fdatawait_range+0x7d/0xd0
> [82205.983551]  ? srso_alias_return_thunk+0x5/0xfbef5
> [82205.983555]  ? __slab_free+0xbf/0x2c0
> [82205.983560]  ? srso_alias_return_thunk+0x5/0xfbef5
> [82205.983563]  ? kmem_cache_alloc_noprof+0x201/0x2a0
> [82205.983567]  ? clear_state_bit+0xfc/0x160 [btrfs]
> [82205.983609]  ? srso_alias_return_thunk+0x5/0xfbef5
> [82205.983612]  ? __clear_extent_bit+0x160/0x490 [btrfs]
> [82205.983646]  filemap_fdatawait_range+0x12/0x20
> [82205.983650]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
> [82205.983689]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
> [82205.983725]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
> [82205.983759]  ? start_transaction+0xc0/0x820 [btrfs]
> [82205.983793]  transaction_kthread+0x159/0x1c0 [btrfs]
> [82205.983827]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
> [82205.983859]  kthread+0xd2/0x100
> [82205.983864]  ? __pfx_kthread+0x10/0x10
> [82205.983867]  ret_from_fork+0x34/0x50
> [82205.983871]  ? __pfx_kthread+0x10/0x10
> [82205.983874]  ret_from_fork_asm+0x1a/0x30
> [82205.983881]  </TASK>
>
> Let me know please if you need some more info.

Got few more of those during the above mentioned transfer:
[101497.950425] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[101497.950432]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[101497.950435] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[101497.950437] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[101497.950442] Call Trace:
[101497.950444]  <TASK>
[101497.950449]  __schedule+0x3f0/0xbd0
[101497.950458]  schedule+0x27/0xf0
[101497.950461]  io_schedule+0x46/0x70
[101497.950465]  folio_wait_bit_common+0x123/0x340
[101497.950472]  ? __pfx_wake_page_function+0x10/0x10
[101497.950477]  folio_wait_writeback+0x2b/0x80
[101497.950480]  __filemap_fdatawait_range+0x7d/0xd0
[101497.950489]  filemap_fdatawait_range+0x12/0x20
[101497.950493]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[101497.950536]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[101497.950572]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[101497.950606]  ? start_transaction+0xc0/0x820 [btrfs]
[101497.950640]  transaction_kthread+0x159/0x1c0 [btrfs]
[101497.950674]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[101497.950705]  kthread+0xd2/0x100
[101497.950710]  ? __pfx_kthread+0x10/0x10
[101497.950714]  ret_from_fork+0x34/0x50
[101497.950718]  ? __pfx_kthread+0x10/0x10
[101497.950721]  ret_from_fork_asm+0x1a/0x30
[101497.950727]  </TASK>
[102358.101888] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[102358.101895]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[102358.101898] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[102358.101899] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[102358.101904] Call Trace:
[102358.101907]  <TASK>
[102358.101912]  __schedule+0x3f0/0xbd0
[102358.101920]  schedule+0x27/0xf0
[102358.101924]  io_schedule+0x46/0x70
[102358.101928]  folio_wait_bit_common+0x123/0x340
[102358.101934]  ? __pfx_wake_page_function+0x10/0x10
[102358.101939]  folio_wait_writeback+0x2b/0x80
[102358.101943]  __filemap_fdatawait_range+0x7d/0xd0
[102358.101951]  filemap_fdatawait_range+0x12/0x20
[102358.101956]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[102358.102000]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[102358.102035]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[102358.102069]  ? start_transaction+0xc0/0x820 [btrfs]
[102358.102107]  transaction_kthread+0x159/0x1c0 [btrfs]
[102358.102142]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[102358.102174]  kthread+0xd2/0x100
[102358.102178]  ? __pfx_kthread+0x10/0x10
[102358.102182]  ret_from_fork+0x34/0x50
[102358.102186]  ? __pfx_kthread+0x10/0x10
[102358.102189]  ret_from_fork_asm+0x1a/0x30
[102358.102195]  </TASK>
[102849.617090] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[102849.617097]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[102849.617100] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[102849.617102] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[102849.617107] Call Trace:
[102849.617109]  <TASK>
[102849.617113]  __schedule+0x3f0/0xbd0
[102849.617122]  schedule+0x27/0xf0
[102849.617126]  io_schedule+0x46/0x70
[102849.617130]  folio_wait_bit_common+0x123/0x340
[102849.617137]  ? __pfx_wake_page_function+0x10/0x10
[102849.617141]  folio_wait_writeback+0x2b/0x80
[102849.617145]  __filemap_fdatawait_range+0x7d/0xd0
[102849.617151]  ? srso_alias_return_thunk+0x5/0xfbef5
[102849.617155]  ? kmem_cache_alloc_noprof+0x201/0x2a0
[102849.617160]  ? clear_state_bit+0xfc/0x160 [btrfs]
[102849.617202]  ? srso_alias_return_thunk+0x5/0xfbef5
[102849.617206]  ? __clear_extent_bit+0x160/0x490 [btrfs]
[102849.617240]  filemap_fdatawait_range+0x12/0x20
[102849.617244]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[102849.617283]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[102849.617319]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[102849.617353]  ? start_transaction+0xc0/0x820 [btrfs]
[102849.617387]  transaction_kthread+0x159/0x1c0 [btrfs]
[102849.617421]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[102849.617452]  kthread+0xd2/0x100
[102849.617457]  ? __pfx_kthread+0x10/0x10
[102849.617461]  ret_from_fork+0x34/0x50
[102849.617465]  ? __pfx_kthread+0x10/0x10
[102849.617468]  ret_from_fork_asm+0x1a/0x30
[102849.617474]  </TASK>
[103464.011016] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[103464.011023]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[103464.011025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[103464.011027] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[103464.011031] Call Trace:
[103464.011033]  <TASK>
[103464.011037]  __schedule+0x3f0/0xbd0
[103464.011045]  schedule+0x27/0xf0
[103464.011048]  io_schedule+0x46/0x70
[103464.011051]  folio_wait_bit_common+0x123/0x340
[103464.011057]  ? __pfx_wake_page_function+0x10/0x10
[103464.011061]  folio_wait_writeback+0x2b/0x80
[103464.011064]  __filemap_fdatawait_range+0x7d/0xd0
[103464.011072]  filemap_fdatawait_range+0x12/0x20
[103464.011076]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[103464.011113]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[103464.011143]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[103464.011172]  ? start_transaction+0xc0/0x820 [btrfs]
[103464.011200]  transaction_kthread+0x159/0x1c0 [btrfs]
[103464.011228]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[103464.011255]  kthread+0xd2/0x100
[103464.011259]  ? __pfx_kthread+0x10/0x10
[103464.011262]  ret_from_fork+0x34/0x50
[103464.011266]  ? __pfx_kthread+0x10/0x10
[103464.011268]  ret_from_fork_asm+0x1a/0x30
[103464.011274]  </TASK>
[103832.647363] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[103832.647370]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[103832.647372] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[103832.647374] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[103832.647378] Call Trace:
[103832.647380]  <TASK>
[103832.647384]  __schedule+0x3f0/0xbd0
[103832.647392]  schedule+0x27/0xf0
[103832.647396]  io_schedule+0x46/0x70
[103832.647399]  folio_wait_bit_common+0x123/0x340
[103832.647405]  ? __pfx_wake_page_function+0x10/0x10
[103832.647410]  folio_wait_writeback+0x2b/0x80
[103832.647413]  __filemap_fdatawait_range+0x7d/0xd0
[103832.647420]  ? srso_alias_return_thunk+0x5/0xfbef5
[103832.647425]  ? __clear_extent_bit+0x160/0x490 [btrfs]
[103832.647466]  filemap_fdatawait_range+0x12/0x20
[103832.647470]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[103832.647509]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[103832.647545]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[103832.647579]  ? start_transaction+0xc0/0x820 [btrfs]
[103832.647613]  transaction_kthread+0x159/0x1c0 [btrfs]
[103832.647648]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[103832.647680]  kthread+0xd2/0x100
[103832.647685]  ? __pfx_kthread+0x10/0x10
[103832.647688]  ret_from_fork+0x34/0x50
[103832.647692]  ? __pfx_kthread+0x10/0x10
[103832.647695]  ret_from_fork_asm+0x1a/0x30
[103832.647700]  </TASK>
[104078.404940] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[104078.404947]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[104078.404950] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[104078.404952] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[104078.404957] Call Trace:
[104078.404959]  <TASK>
[104078.404964]  __schedule+0x3f0/0xbd0
[104078.404973]  schedule+0x27/0xf0
[104078.404976]  io_schedule+0x46/0x70
[104078.404980]  folio_wait_bit_common+0x123/0x340
[104078.404987]  ? __pfx_wake_page_function+0x10/0x10
[104078.404992]  folio_wait_writeback+0x2b/0x80
[104078.404995]  __filemap_fdatawait_range+0x7d/0xd0
[104078.405001]  ? srso_alias_return_thunk+0x5/0xfbef5
[104078.405005]  ? kmem_cache_alloc_noprof+0x201/0x2a0
[104078.405010]  ? clear_state_bit+0xfc/0x160 [btrfs]
[104078.405051]  ? srso_alias_return_thunk+0x5/0xfbef5
[104078.405055]  ? __clear_extent_bit+0x160/0x490 [btrfs]
[104078.405089]  filemap_fdatawait_range+0x12/0x20
[104078.405093]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[104078.405132]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[104078.405168]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[104078.405202]  ? start_transaction+0xc0/0x820 [btrfs]
[104078.405236]  transaction_kthread+0x159/0x1c0 [btrfs]
[104078.405270]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[104078.405301]  kthread+0xd2/0x100
[104078.405306]  ? __pfx_kthread+0x10/0x10
[104078.405310]  ret_from_fork+0x34/0x50
[104078.405313]  ? __pfx_kthread+0x10/0x10
[104078.405317]  ret_from_fork_asm+0x1a/0x30
[104078.405323]  </TASK>
[104324.162492] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[104324.162499]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[104324.162502] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[104324.162504] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[104324.162509] Call Trace:
[104324.162511]  <TASK>
[104324.162515]  __schedule+0x3f0/0xbd0
[104324.162525]  schedule+0x27/0xf0
[104324.162528]  io_schedule+0x46/0x70
[104324.162532]  folio_wait_bit_common+0x123/0x340
[104324.162539]  ? __pfx_wake_page_function+0x10/0x10
[104324.162544]  folio_wait_writeback+0x2b/0x80
[104324.162547]  __filemap_fdatawait_range+0x7d/0xd0
[104324.162553]  ? srso_alias_return_thunk+0x5/0xfbef5
[104324.162557]  ? kmem_cache_alloc_noprof+0x201/0x2a0
[104324.162562]  ? clear_state_bit+0xfc/0x160 [btrfs]
[104324.162604]  ? srso_alias_return_thunk+0x5/0xfbef5
[104324.162608]  ? __clear_extent_bit+0x160/0x490 [btrfs]
[104324.162641]  filemap_fdatawait_range+0x12/0x20
[104324.162645]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[104324.162685]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[104324.162721]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[104324.162755]  ? start_transaction+0xc0/0x820 [btrfs]
[104324.162789]  transaction_kthread+0x159/0x1c0 [btrfs]
[104324.162823]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[104324.162854]  kthread+0xd2/0x100
[104324.162859]  ? __pfx_kthread+0x10/0x10
[104324.162863]  ret_from_fork+0x34/0x50
[104324.162866]  ? __pfx_kthread+0x10/0x10
[104324.162870]  ret_from_fork_asm+0x1a/0x30
[104324.162876]  </TASK>
[105798.707833] INFO: task btrfs-transacti:29385 blocked for more than
122 seconds.
[105798.707840]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
[105798.707843] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[105798.707845] task:btrfs-transacti state:D stack:0     pid:29385
tgid:29385 ppid:2      flags:0x00004000
[105798.707850] Call Trace:
[105798.707853]  <TASK>
[105798.707857]  __schedule+0x3f0/0xbd0
[105798.707866]  schedule+0x27/0xf0
[105798.707870]  io_schedule+0x46/0x70
[105798.707874]  folio_wait_bit_common+0x123/0x340
[105798.707881]  ? __pfx_wake_page_function+0x10/0x10
[105798.707886]  folio_wait_writeback+0x2b/0x80
[105798.707890]  __filemap_fdatawait_range+0x7d/0xd0
[105798.707896]  ? srso_alias_return_thunk+0x5/0xfbef5
[105798.707900]  ? kmem_cache_alloc_noprof+0x201/0x2a0
[105798.707906]  ? clear_state_bit+0xfc/0x160 [btrfs]
[105798.707949]  ? srso_alias_return_thunk+0x5/0xfbef5
[105798.707953]  ? __clear_extent_bit+0x160/0x490 [btrfs]
[105798.707989]  filemap_fdatawait_range+0x12/0x20
[105798.707993]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
[105798.708034]  btrfs_write_and_wait_transaction+0x5e/0xd0 [btrfs]
[105798.708071]  btrfs_commit_transaction+0x8d9/0xe80 [btrfs]
[105798.708107]  ? start_transaction+0xc0/0x820 [btrfs]
[105798.708142]  transaction_kthread+0x159/0x1c0 [btrfs]
[105798.708177]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[105798.708211]  kthread+0xd2/0x100
[105798.708215]  ? __pfx_kthread+0x10/0x10
[105798.708219]  ret_from_fork+0x34/0x50
[105798.708223]  ? __pfx_kthread+0x10/0x10
[105798.708227]  ret_from_fork_asm+0x1a/0x30
[105798.708233]  </TASK>
[105798.708235] Future hung task reports are suppressed, see sysctl
kernel.hung_task_warnings

The transfer has been completed, took following 'time':
real    812m30.231s
user    6m7.214s
sys     169m1.938s

The destination btrfs filesystem has been freshly created (single
device, no raid) and thus empty before starting the transfer.
This is output of destination 'btrfs filesystem df' after the transfer:
Data, single: total=7.10TiB, used=7.06TiB
System, DUP: total=8.00MiB, used=768.00KiB
Metadata, DUP: total=8.00GiB, used=7.53GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

The btrfs is on a luks device (cryptsetup -h sha512 -c aes-xts-plain64
-s 256), but cpu has been idle the whole time (ryzen 5950x).

Hope that helps.
Thank you.

Re: clear space cache v1 fails with Unable to find block group for 0

[Qu Wenruo]

在 2024/12/17 16:01, j4nn 写道:
> On Mon, 16 Dec 2024 at 19:52, j4nn <j4nn.xda@gmail.com> wrote:
>>
>> This is unrelated, but as you have been interested in the hung task
>> backtrace, I got two more when using "btrfs send ... | btrfs receive
>> ..." to copy 7TB of data from one btrfs disk to another one (still in
>> progress, both rotational hard drives):
>> [81837.347137] INFO: task btrfs-transacti:29385 blocked for more than
>> 122 seconds.
>> [81837.347144]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
>> [81837.347147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [81837.347149] task:btrfs-transacti state:D stack:0     pid:29385
>> tgid:29385 ppid:2      flags:0x00004000
>> [81837.347154] Call Trace:
>> [81837.347156]  <TASK>
>> [81837.347161]  __schedule+0x3f0/0xbd0
>> [81837.347170]  schedule+0x27/0xf0
>> [81837.347174]  io_schedule+0x46/0x70
>> [81837.347177]  folio_wait_bit_common+0x123/0x340
>> [81837.347184]  ? __pfx_wake_page_function+0x10/0x10
>> [81837.347189]  folio_wait_writeback+0x2b/0x80
>> [81837.347193]  __filemap_fdatawait_range+0x7d/0xd0
>> [81837.347201]  filemap_fdatawait_range+0x12/0x20
>> [81837.347206]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]

This is from the metadata writeback.

My guess is, since you're transfering a lot of data, the metadata also 
go very large very fast.

And the default commit interval is 30s, and considering your hardware, 
your hardware memory may also be pretty large (at least 32G I believe?).

That means we can have several giga bytes of metadata waiting to be 
written back.

What makes things worse may be the fact that, metadata writeback 
nowadays are always in nodesize, no merging at all.

Furthermore since your storage device is HDD, the low IOPS performance 
is making it even worse.


Combining all those things together, we're writing several giga bytes of 
metadata, all in 16K nodesize no merging, resulting a very bad write 
performance on rotating disks...

IIRC there are some ways to limit how many bytes can be utilized by page 
cache (btrfs metadata is also utilizing page cache), thus it may improve 
the situation by not writing too much metadata in one go.

[...]
> 
> The destination btrfs filesystem has been freshly created (single
> device, no raid) and thus empty before starting the transfer.
> This is output of destination 'btrfs filesystem df' after the transfer:
> Data, single: total=7.10TiB, used=7.06TiB
> System, DUP: total=8.00MiB, used=768.00KiB
> Metadata, DUP: total=8.00GiB, used=7.53GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> The btrfs is on a luks device (cryptsetup -h sha512 -c aes-xts-plain64
> -s 256), but cpu has been idle the whole time (ryzen 5950x).


Considering the transfer finished, and you can unmount the fs, it should 
really be a false alert, mind to share how large your RAM is?
32G or even 64G?

Thanks,
Qu
> 
> Hope that helps.
> Thank you.
> 

Re: clear space cache v1 fails with Unable to find block group for 0

[j4nn]

On Tue, 17 Dec 2024 at 06:50, Qu Wenruo <wqu@suse.com> wrote:
> 在 2024/12/17 16:01, j4nn 写道:
> > On Mon, 16 Dec 2024 at 19:52, j4nn <j4nn.xda@gmail.com> wrote:
> >>
> >> This is unrelated, but as you have been interested in the hung task
> >> backtrace, I got two more when using "btrfs send ... | btrfs receive
> >> ..." to copy 7TB of data from one btrfs disk to another one (still in
> >> progress, both rotational hard drives):
> >> [81837.347137] INFO: task btrfs-transacti:29385 blocked for more than
> >> 122 seconds.
> >> [81837.347144]       Tainted: G        W  O       6.12.3-gentoo-x86_64 #1
> >> [81837.347147] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >> disables this message.
> >> [81837.347149] task:btrfs-transacti state:D stack:0     pid:29385
> >> tgid:29385 ppid:2      flags:0x00004000
> >> [81837.347154] Call Trace:
> >> [81837.347156]  <TASK>
> >> [81837.347161]  __schedule+0x3f0/0xbd0
> >> [81837.347170]  schedule+0x27/0xf0
> >> [81837.347174]  io_schedule+0x46/0x70
> >> [81837.347177]  folio_wait_bit_common+0x123/0x340
> >> [81837.347184]  ? __pfx_wake_page_function+0x10/0x10
> >> [81837.347189]  folio_wait_writeback+0x2b/0x80
> >> [81837.347193]  __filemap_fdatawait_range+0x7d/0xd0
> >> [81837.347201]  filemap_fdatawait_range+0x12/0x20
> >> [81837.347206]  __btrfs_wait_marked_extents.isra.0+0xb8/0xf0 [btrfs]
>
> This is from the metadata writeback.
>
> My guess is, since you're transfering a lot of data, the metadata also
> go very large very fast.
>
> And the default commit interval is 30s, and considering your hardware,
> your hardware memory may also be pretty large (at least 32G I believe?).
>
> That means we can have several giga bytes of metadata waiting to be
> written back.
>
> What makes things worse may be the fact that, metadata writeback
> nowadays are always in nodesize, no merging at all.
>
> Furthermore since your storage device is HDD, the low IOPS performance
> is making it even worse.
>
>
> Combining all those things together, we're writing several giga bytes of
> metadata, all in 16K nodesize no merging, resulting a very bad write
> performance on rotating disks...
>
> IIRC there are some ways to limit how many bytes can be utilized by page
> cache (btrfs metadata is also utilizing page cache), thus it may improve
> the situation by not writing too much metadata in one go.
>
> [...]
> >
> > The destination btrfs filesystem has been freshly created (single
> > device, no raid) and thus empty before starting the transfer.
> > This is output of destination 'btrfs filesystem df' after the transfer:
> > Data, single: total=7.10TiB, used=7.06TiB
> > System, DUP: total=8.00MiB, used=768.00KiB
> > Metadata, DUP: total=8.00GiB, used=7.53GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> >
> > The btrfs is on a luks device (cryptsetup -h sha512 -c aes-xts-plain64
> > -s 256), but cpu has been idle the whole time (ryzen 5950x).
>
>
> Considering the transfer finished, and you can unmount the fs, it should
> really be a false alert, mind to share how large your RAM is?
> 32G or even 64G?

Yes, you are right, using both :-)
That is 96GB of RAM...

Thank you for the explanations, particularly the "16K nodesize no
merging" metadata chunks.
The reported 'time' of the transfer included a 'sync' command after
the btrfs receive.

Re: clear space cache v1 fails with Unable to find block group for 0

[j4nn]

On Tue, 17 Dec 2024 at 07:10, j4nn <j4nn.xda@gmail.com> wrote:
>
> On Tue, 17 Dec 2024 at 06:50, Qu Wenruo <wqu@suse.com> wrote:
> > 在 2024/12/17 16:01, j4nn 写道:
> > > On Mon, 16 Dec 2024 at 19:52, j4nn <j4nn.xda@gmail.com> wrote:
> > >>
> >
> > This is from the metadata writeback.
> >
> > My guess is, since you're transfering a lot of data, the metadata also
> > go very large very fast.
> >
> > And the default commit interval is 30s, and considering your hardware,
> > your hardware memory may also be pretty large (at least 32G I believe?).
> >
> > That means we can have several giga bytes of metadata waiting to be
> > written back.
> >
> > What makes things worse may be the fact that, metadata writeback
> > nowadays are always in nodesize, no merging at all.
> >
> > Furthermore since your storage device is HDD, the low IOPS performance
> > is making it even worse.
> >
> >
> > Combining all those things together, we're writing several giga bytes of
> > metadata, all in 16K nodesize no merging, resulting a very bad write
> > performance on rotating disks...
> >
> > IIRC there are some ways to limit how many bytes can be utilized by page
> > cache (btrfs metadata is also utilizing page cache), thus it may improve
> > the situation by not writing too much metadata in one go.
> >
> > [...]
> > Considering the transfer finished, and you can unmount the fs, it should
> > really be a false alert, mind to share how large your RAM is?
> > 32G or even 64G?
>
> Yes, you are right, using both :-)
> That is 96GB of RAM...
>
> Thank you for the explanations, particularly the "16K nodesize no
> merging" metadata chunks.
> The reported 'time' of the transfer included a 'sync' command after
> the btrfs receive.

I guess the hung task backtrace that appeared during creating of free
space tree cache had the same cause as this simple btrfs send and
receive?

Re: clear space cache v1 fails with Unable to find block group for 0

[Qu Wenruo]

在 2024/12/17 17:04, j4nn 写道:
> On Tue, 17 Dec 2024 at 07:10, j4nn <j4nn.xda@gmail.com> wrote:
>>
>> On Tue, 17 Dec 2024 at 06:50, Qu Wenruo <wqu@suse.com> wrote:
>>> 在 2024/12/17 16:01, j4nn 写道:
>>>> On Mon, 16 Dec 2024 at 19:52, j4nn <j4nn.xda@gmail.com> wrote:
>>>>>
>>>
>>> This is from the metadata writeback.
>>>
>>> My guess is, since you're transfering a lot of data, the metadata also
>>> go very large very fast.
>>>
>>> And the default commit interval is 30s, and considering your hardware,
>>> your hardware memory may also be pretty large (at least 32G I believe?).
>>>
>>> That means we can have several giga bytes of metadata waiting to be
>>> written back.
>>>
>>> What makes things worse may be the fact that, metadata writeback
>>> nowadays are always in nodesize, no merging at all.
>>>
>>> Furthermore since your storage device is HDD, the low IOPS performance
>>> is making it even worse.
>>>
>>>
>>> Combining all those things together, we're writing several giga bytes of
>>> metadata, all in 16K nodesize no merging, resulting a very bad write
>>> performance on rotating disks...
>>>
>>> IIRC there are some ways to limit how many bytes can be utilized by page
>>> cache (btrfs metadata is also utilizing page cache), thus it may improve
>>> the situation by not writing too much metadata in one go.
>>>
>>> [...]
>>> Considering the transfer finished, and you can unmount the fs, it should
>>> really be a false alert, mind to share how large your RAM is?
>>> 32G or even 64G?
>>
>> Yes, you are right, using both :-)
>> That is 96GB of RAM...
>>
>> Thank you for the explanations, particularly the "16K nodesize no
>> merging" metadata chunks.
>> The reported 'time' of the transfer included a 'sync' command after
>> the btrfs receive.
>
> I guess the hung task backtrace that appeared during creating of free
> space tree cache had the same cause as this simple btrfs send and
> receive?

I can only be more or less certain on the receive end. (receive is
mostly just writing data into the fs, as most of the work is done in
user space with buffered write).

The v2 cache rebuilding process is indeed problematic, but for a
different reason.

When rebuilding v2 cache can cause a huge hang, that's for sure.
We are using a single transaction to build new v2 cache for each block
group, no wonder it will hang.

Anyway I'll change the rebuilding process to at least do
multi-transactional update, to avoid holding one transaction too long.

But the rebuilding itself can still be very time consuming anyway.

Thanks,
Qu