Re: btrfs with huge numbers of hardlinks is extremely slow

Forza
2021-11-26T08:25:24Z

---- From: Zygo Blaxell <ce3g8jdj@umail.furryterror.org> -- Sent: 2021-11-26 - 06:15 ----

> On Fri, Nov 26, 2021 at 12:56:25AM +0300, Andrey Melnikov wrote:
>> Every night a new backup is stored on this fs with 'rsync
>> --link-dest=$yestoday/ $today/' - 1402700 hardlinks and 23000
>> directories are created, 50-100 normal files transferred.
>> Now, FS contains 351 copies of backup data with 486086495 hardlinks
>> and ANY operations on this FS take significant time. For example -
>> simple count hardlinks with
>> "time find . -type f -links +1 | wc -l" take:
>> real 28567m33.611s
>> user 31m33.395s
>> sys 506m28.576s
>>
>> 19 days 20 hours 10 mins with constant reads from storage 2-4Mb/s.
>
> That works out to reading the entire drive 4x in 20 days, or all of the
> metadata 30x. Certainly hardlinks will not result in optimal object
> placement, and you probably don't have enough RAM to cache the entire
> metadata tree, and you're using WD Purple drives in a fileserver for
> some reason, so those numbers seem plausible.

Defragmenting the subvolume and extent tree could help reducing the amount and distance of seeks to metadata which should improve performance of find.

# btrfs fi defrag /path/to/subvol

If you cannot change drive model, breaking up your HW raid and create a btrfs raid1 should also improve performance, as well as fault tolerance.

Apart from this, I second Zygo's suggestions below.

>
>> - BTRFS not suitable for this workload?
>
> There are definitely better ways to do this on btrfs, e.g.
>
> btrfs sub snap $yesterday $today
> rsync ... (no link-dest) ... $today
>
> This will avoid duplicating the entire file tree every time. It will also
> store historical file attributes correctly, which --link-dest sometimes
> does not.
>
> You might also consider doing it differently:
>
> rsync ... (no link-dest) ... working-dir/. &&
> btrfs sub snap -r working-dir $today
>
> so that your $today directory doesn't exist until it is complete with
> no rsync errors. 'working-dir' will have to be a subvol, but you only
> have to create it once and you can keep reusing it afterwards.
>
>> - using reflinks helps speedup FS operations?
>
> Snapshots are lazy reflink copies, so they'll do a little better than
> reflinks. You'll only modify the metadata for the 50-100 files that you
> transfer each day, instead of completely rewriting all of the metadata
> in the filesystem every day with hardlinks.
>
> Hardlinks put the inodes further and further away from their directory
> nodes each day, and add some extra searching overhead within directories
> as well. You'll need more and more RAM to cache the same amount of
> each filesystem tree, because they're all in the same metadata tree.
> With snapshots they'll end up in separate metadata trees.
>
>> - readed metadata not cached at all?
>
> If you have less than about 640GB of RAM (4x the size of your metadata)
> then you're going to be rereading metadata pages at some point. Because
> you're using hardlinks, the metadata pages from different days are all
> mashed together, and 'find' will flood the cache chasing references to
> them.
>
> Other recommendations:
>
> - Use the right drive model for your workload. WD Purple drives are for
> continuous video streaming, they are not for seeky millions-of-tiny-files
> rsync workloads. Almost any other model will outperform them, and
> better drives for this workload (e.g. CMR WD Red models) are cheaper.
> Your WD Purple drives are getting 283 links/s. Compare that with some
> other drive models:
>
> 1665 links/s: WD Green (2x1TB + 1x2TB btrfs raid1)
>
> 6850 links/s: Sandisk Extreme MicroSD (1x256GB btrfs single/dup)
>
> 12511 links/s: WD Red (2x1TB btrfs raid1)
>
> 13371 links/s: WD Red SSD + Seagate Ironwolf SSD (6x1TB btrfs raid1)
>
> 14872 links/s: WD Black (1x1TB btrfs single/dup, 8 years old)
>
> 25498 links/s: WD Gold + Seagate Exos (3x16TB btrfs raid1)
>
> 27341 links/s: Toshiba NVME (1x2TB btrfs single/dup)
>
> 311284 links/s: Sabrent Rocket 4 NVME (2x1TB btrfs raid1)
> (1344748222 links, 111 snapshots)
>
> Some of these numbers are lower than they should be, because I ran
> 'find' commands on some machines that were busy doing other work.
> The point is that even if some of these numbers are too low, all of
> these numbers are higher what we can expect from a WD Purple.
>
> - Use btrfs raid1 instead of hardware RAID1, i.e. expose each disk
> separately through the RAID interface to btrfs. This will enable btrfs
> to correct errors and isolate faults if one of your drives goes bad.
> You can also use iostat to see if one of the drives is running much
> slower than the other, which might be an early indication of failure
> (and it might be the only indication of failure you get, if your drive's
> firmware doesn't support SCTERC and hides failures).
>
>> What BTRFS read 19 days from disks???
>>
>> Hardware: dell r300 with 2 WD Purple 1Tb disk on LSISAS1068E RAID 1
>> (without cache).
>> Linux nlxc 5.14-51412-generic #0~lch11 SMP Wed Oct 13 15:57:07 UTC
>> 2021 x86_64 GNU/Linux
>> btrfs-progs v5.14.1
>>
>> # btrfs fi show
>> Label: none uuid: a840a2ca-bf05-4074-8895-60d993cb5bdd
>> Total devices 1 FS bytes used 474.26GiB
>> devid 1 size 931.00GiB used 502.23GiB path /dev/sdb1
>>
>> # btrfs fi df /srv
>> Data, single: total=367.19GiB, used=343.92GiB
>> System, single: total=32.00MiB, used=128.00KiB
>> Metadata, single: total=135.00GiB, used=130.34GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> # btrfs fi us /srv
>> Overall:
>> Device size: 931.00GiB
>> Device allocated: 502.23GiB
>> Device unallocated: 428.77GiB
>> Device missing: 0.00B
>> Used: 474.26GiB
>> Free (estimated): 452.04GiB (min: 452.04GiB)
>> Free (statfs, df): 452.04GiB
>> Data ratio: 1.00
>> Metadata ratio: 1.00
>> Global reserve: 512.00MiB (used: 0.00B)
>> Multiple profiles: no
>>
>> Data,single: Size:367.19GiB, Used:343.92GiB (93.66%)
>> /dev/sdb1 367.19GiB
>>
>> Metadata,single: Size:135.00GiB, Used:130.34GiB (96.55%)
>> /dev/sdb1 135.00GiB
>>
>> System,single: Size:32.00MiB, Used:128.00KiB (0.39%)
>> /dev/sdb1 32.00MiB
>>
>> Unallocated:
>> /dev/sdb1 428.77GiB