Re: [BUG] 500-2000% performance regression w/ 5.10

[Rene Rebe <rene@exactcode.com>]

Dear Qu,

On 12/22/20 9:28 AM, Qu Wenruo wrote:
> On 2020/12/22 上午3:45, René Rebe wrote:
>> Hey there,
>>
>> as a long time btrfs user I noticed some some things became very slow
>> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
>> a huge tarball like:
>>
>>    tar xf 
>> /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>>
>> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
>> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>>
>> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
>> a similar, albeit not as shocking regression from 5.2 seconds
>> to ~34 seconds or∫~650%.
>>
>> Somehow testing that in a VM did over virtio did not produce
>> as different results, although it was already 35 seconds slow
>> with 5.9.
>>
>> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
>
> This means metadata space is not enough and we go shrink_delalloc() to
> free some metadata space.
>
> My concern is, why we need to go shrink_delalloc() in the first place.
>
> Normally either the fs has enough unallocated space (thus we can
> over-commit) or has enough unused metadata space.
>
> We only need to shrink delalloc if we have no unallocated space, and not
> enough space for the over-estimated metadata reserve.
>
>
> Would you please try to provide the `btrfs fi usage` output of your test
> drive?
> My initial guess is, this is related to fs usage/layout.


Thank you for looking into this and your reply.

That was my initial thoguht, too, and thus I already had run the test on 
a brand new, previously unused 1TB SSD with a fresh mkfs.btrfs with 
similar results to the long used drive:

# btrfs fi usage /mnt/
Overall:
     Device size:                 931.51GiB
     Device allocated:              4.01GiB
     Device unallocated:          927.50GiB
     Device missing:                  0.00B
     Used:                        640.00KiB
     Free (estimated):            930.50GiB      (min: 930.50GiB)
     Data ratio:                       1.00
     Metadata ratio:                   1.00
     Global reserve:                3.25MiB      (used: 0.00B)

Data,single: Size:3.00GiB, Used:512.00KiB
    /dev/nvme0n1    3.00GiB

Metadata,single: Size:1.01GiB, Used:112.00KiB
    /dev/nvme0n1    1.01GiB

System,single: Size:4.00MiB, Used:16.00KiB
    /dev/nvme0n1    4.00MiB

Unallocated:
    /dev/nvme0n1  927.50GiB


# mkfs.btrfs /dev/nvme0n1 -f

# mount /dev/nvme0n1 /mnt

# cat /t2/download/mirror/f/firefox-84.0.source.tar.zst > /dev/null
# time tar -x -C /mnt -f /t2/download/mirror/f/firefox-84.0.source.tar.zst

I hope this helps,

   René

> Thanks,
> Qu
>>
>> Now just this single commit does obviously not revert cleanly,
>> and I did not have the time today to look into the rather more
>> complex code today.
>>
>> I hope this helps improve this for the next release, maybe you
>> want to test on bare metal, too.
>>
>> Greetings,
>>     René    https://youtu.be/NhUMdvLyKJc
>>