btrfs with huge numbers of hardlinks is extremely slow

Andrey Melnikov
2021-11-25T21:59:24Z

Every night a new backup is stored on this fs with 'rsync
--link-dest=$yestoday/ $today/' - 1402700 hardlinks and 23000
directories are created, 50-100 normal files transferred.
Now, FS contains 351 copies of backup data with 486086495 hardlinks
and ANY operations on this FS take significant time. For example -
simple count hardlinks with
"time find . -type f -links +1 | wc -l" take:
real 28567m33.611s
user 31m33.395s
sys 506m28.576s

19 days 20 hours 10 mins with constant reads from storage 2-4Mb/s.

- BTRFS not suitable for this workload?
- using reflinks helps speedup FS operations?
- readed metadata not cached at all? What BTRFS read 19 days from disks???

Hardware: dell r300 with 2 WD Purple 1Tb disk on LSISAS1068E RAID 1
(without cache).
Linux nlxc 5.14-51412-generic #0~lch11 SMP Wed Oct 13 15:57:07 UTC
2021 x86_64 GNU/Linux
btrfs-progs v5.14.1

# btrfs fi show
Label: none uuid: a840a2ca-bf05-4074-8895-60d993cb5bdd
Total devices 1 FS bytes used 474.26GiB
devid 1 size 931.00GiB used 502.23GiB path /dev/sdb1

# btrfs fi df /srv
Data, single: total=367.19GiB, used=343.92GiB
System, single: total=32.00MiB, used=128.00KiB
Metadata, single: total=135.00GiB, used=130.34GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

# btrfs fi us /srv
Overall:
Device size: 931.00GiB
Device allocated: 502.23GiB
Device unallocated: 428.77GiB
Device missing: 0.00B
Used: 474.26GiB
Free (estimated): 452.04GiB (min: 452.04GiB)
Free (statfs, df): 452.04GiB
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no

Data,single: Size:367.19GiB, Used:343.92GiB (93.66%)
/dev/sdb1 367.19GiB

Metadata,single: Size:135.00GiB, Used:130.34GiB (96.55%)
/dev/sdb1 135.00GiB

System,single: Size:32.00MiB, Used:128.00KiB (0.39%)
/dev/sdb1 32.00MiB

Unallocated:
/dev/sdb1 428.77GiB