Re: [BUG] 500-2000% performance regression w/ 5.10

[Rene Rebe <rene@exactcode.com>]

Hey there,

On 12/26/20 1:24 PM, Nikolay Borisov wrote:
>
> On 24.12.20 г. 23:11 ч., René Rebe wrote:
>> Hi Josef,
>>
>>> On 24. Dec 2020, at 19:09, Josef Bacik <josef@toxicpanda.com> wrote:
>>>
>>> On 12/21/20 2:45 PM, René Rebe wrote:
>>>> Hey there,
>>>> as a long time btrfs user I noticed some some things became very slow
>>>> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
>>>> a huge tarball like:
>>>>    tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>>>> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
>>>> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>>>> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
>>>> a similar, albeit not as shocking regression from 5.2 seconds
>>>> to ~34 seconds or∫~650%.
>>>> Somehow testing that in a VM did over virtio did not produce
>>>> as different results, although it was already 35 seconds slow
>>>> with 5.9.
>>>> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>>>>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
>>>> Now just this single commit does obviously not revert cleanly,
>>>> and I did not have the time today to look into the rather more
>>>> complex code today.
>>>> I hope this helps improve this for the next release, maybe you
>>>> want to test on bare metal, too.
>>> Alright to close the loop with this, this slipped through the cracks because I was doing a lot of performance related work, and specifically had been testing with these patches on top of everything
>>>
>>> https://lore.kernel.org/linux-btrfs/cover.1602249928.git.josef@toxicpanda.com/
>>>
>>> These patches bring the performance up to around 40% higher than baseline
>> I indeed tested the linux-btrfs for-5.11 and found the performance some 50% better. I would hope that can be brought back to 5.9 levels sometime soon ;-)
> Do you mean 50% better as compared to 5.9?


Sorry for any confusion, I meant 50% better than the bisected regression 
as found with 5.10, not yet as good as 5.9 has been before.


     René

>>> .  In the meantime we'll probably push this partial revert into 5.10 stable so performance isn't sucking in the meantime.  Thanks,
>> That certainly makes sense for the LTS kernel series.
>>
>> Thanks for looking into this,
>> Merry Christmas,
>> 	René Rebe
>>