Re: Scrub of my nvme SSD has slowed by about 2/3

Re: Scrub of my nvme SSD has slowed by about 2/3

[Linux regression tracking #adding (Thorsten Leemhuis)]

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 03.07.23 22:19, Tim Cuthbertson wrote:
> Yesterday, I noticed that a scrub of my main system filesystem has
> slowed from about 2.9 gb/sec to about 949 mb/sec. My scrub used to run
> in about 12 seconds, now it is taking 51 seconds. I had just installed
> Linux kernel 6.4.1 on Arch Linux, upgrading from 6.3.9. At first I
> suspected the new kernel, but now I am not so sure.

Thanks for the report. It seems it will take some work to address this,
so to be sure the issue doesn't fall through the cracks unnoticed, I'm
adding it to regzbot, the Linux kernel regression tracking bot:

#regzbot ^introduced e02ee89baa66
#regzbot title btrfs: scrub nvme SSD has slowed by about 2/3 due to csum
#regzbot monitor:
https://lore.kernel.org/all/6c1ffe48e93fee9aa975ecc22dc2e7a1f3d7a0de.1688539673.git.wqu@suse.com/
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: Scrub of my nvme SSD has slowed by about 2/3

[Martin Steigerwald]

Hi Thorsten.

#regzbot monitor: https://lore.kernel.org/linux-btrfs/cover.1689744163.git.wqu@suse.com/

I hope that does it.

Above links to a patch of Qu about asking for other ideas to fix the
regression. It however has been superseded by a later patch of him.

So feel free to drop this.

Best,
Martin

Linux regression tracking #adding (Thorsten Leemhuis) - 12.07.23, 13:02:18 CEST:
> [CCing the regression list, as it should be in the loop for
> regressions:
> https://docs.kernel.org/admin-guide/reporting-regressions.html]
> 
> [TLDR: I'm adding this report to the list of tracked Linux kernel
> regressions; the text you find below is based on a few templates
> paragraphs you might have encountered already in similar form.
> See link in footer if these mails annoy you.]
> 
> On 03.07.23 22:19, Tim Cuthbertson wrote:
> > Yesterday, I noticed that a scrub of my main system filesystem has
> > slowed from about 2.9 gb/sec to about 949 mb/sec. My scrub used to
> > run in about 12 seconds, now it is taking 51 seconds. I had just
> > installed Linux kernel 6.4.1 on Arch Linux, upgrading from 6.3.9.
> > At first I suspected the new kernel, but now I am not so sure.
> 
> Thanks for the report. It seems it will take some work to address
> this, so to be sure the issue doesn't fall through the cracks
> unnoticed, I'm adding it to regzbot, the Linux kernel regression
> tracking bot:
> 
> #regzbot ^introduced e02ee89baa66
> #regzbot title btrfs: scrub nvme SSD has slowed by about 2/3 due to
> csum #regzbot monitor:
> https://lore.kernel.org/all/6c1ffe48e93fee9aa975ecc22dc2e7a1f3d7a0de.1
> 688539673.git.wqu@suse.com/ #regzbot ignore-activity
> 
> This isn't a regression? This issue or a fix for it are already
> discussed somewhere else? It was fixed already? You want to clarify
> when the regression started to happen? Or point out I got the title
> or something else totally wrong? Then just reply and tell me --
> ideally while also telling regzbot about it, as explained by the page
> listed in the footer of this mail.
> 
> Developers: When fixing the issue, remember to add 'Link:' tags
> pointing to the report (the parent of this mail). See page linked in
> footer for details.
> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker'
> hat) --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> That page also explains what to do if mails like this annoy you.


-- 
Martin

Re: Scrub of my nvme SSD has slowed by about 2/3

[Martin Steigerwald]

Martin Steigerwald - 19.07.23, 08:42:58 CEST:
> Hi Thorsten.
> 
> #regzbot monitor:
> https://lore.kernel.org/linux-btrfs/cover.1689744163.git.wqu@suse.com
> /
> 
> I hope that does it.
> 
> Above links to a patch of Qu about asking for other ideas to fix the
> regression. It however has been superseded by a later patch of him.

FWIW time-wise it was the other way around. The other patch was slightly 
earlier, but it not fully recover the original speed of scrubbing of 
NVME devices in 6.3¹. Thus Qu was asking for additional ideas.

[1] https://lore.kernel.org/linux-btrfs/4f48e79c-93f7-b473-648d-4c995070c8ac@gmx.com/T/#t

Best,
-- 
Martin

Re: Scrub of my nvme SSD has slowed by about 2/3

[Linux regression tracking #update (Thorsten Leemhuis)]

On 12.07.23 13:02, Linux regression tracking #adding (Thorsten Leemhuis)
wrote:

> On 03.07.23 22:19, Tim Cuthbertson wrote:
>> Yesterday, I noticed that a scrub of my main system filesystem has
>> slowed from about 2.9 gb/sec to about 949 mb/sec. My scrub used to run
>> in about 12 seconds, now it is taking 51 seconds. I had just installed
>> Linux kernel 6.4.1 on Arch Linux, upgrading from 6.3.9. At first I
>> suspected the new kernel, but now I am not so sure.
> 
> Thanks for the report. It seems it will take some work to address this,
> so to be sure the issue doesn't fall through the cracks unnoticed, I'm
> adding it to regzbot, the Linux kernel regression tracking bot:
> 
> #regzbot ^introduced e02ee89baa66

#regzbot resolve: various changes merged for 6.6 improve things again;
more planned; backporting is planned, too;
#regzbot ignore-activity

(yes, that is not idea, but that's how it is sometimes)

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: Scrub of my nvme SSD has slowed by about 2/3

[Martin Steigerwald]

Hi Thorsten, Qu, Tim, everyone,

Linux regression tracking #update (Thorsten Leemhuis) - 29.08.23, 14:17:33 
CEST:
> On 12.07.23 13:02, Linux regression tracking #adding (Thorsten Leemhuis)
> wrote:
> > On 03.07.23 22:19, Tim Cuthbertson wrote:
> >> Yesterday, I noticed that a scrub of my main system filesystem has
> >> slowed from about 2.9 gb/sec to about 949 mb/sec. My scrub used to
> >> run
> >> in about 12 seconds, now it is taking 51 seconds. I had just
> >> installed
> >> Linux kernel 6.4.1 on Arch Linux, upgrading from 6.3.9. At first I
> >> suspected the new kernel, but now I am not so sure.
> > 
> > Thanks for the report. It seems it will take some work to address
> > this,
> > so to be sure the issue doesn't fall through the cracks unnoticed, I'm
> > adding it to regzbot, the Linux kernel regression tracking bot:
> > 
> > #regzbot ^introduced e02ee89baa66
> 
> #regzbot resolve: various changes merged for 6.6 improve things again;
> more planned; backporting is planned, too;
> #regzbot ignore-activity
> 
> (yes, that is not idea, but that's how it is sometimes)

ideal?

Scrubbing "/home" with 304.61GiB (interestingly both back then with 6.4 
and now with 6.5.2):

- 6.4: 966.84MiB/s
- 6.5.2:  748.02MiB/s

I expected an improvement.

Same Lenovo ThinkPad T14 AMD Gen 1 with AMD Ryzen 7 PRO 4750U, 32 GiB RAM 
and 2TB Samsung 980 Pro NVME SSD as before.

Ciao,
-- 
Martin

Re: Scrub of my nvme SSD has slowed by about 2/3

[Qu Wenruo]

On 2023/9/8 19:54, Martin Steigerwald wrote:
> Hi Thorsten, Qu, Tim, everyone,
>
> Linux regression tracking #update (Thorsten Leemhuis) - 29.08.23, 14:17:33
> CEST:
>> On 12.07.23 13:02, Linux regression tracking #adding (Thorsten Leemhuis)
>> wrote:
>>> On 03.07.23 22:19, Tim Cuthbertson wrote:
>>>> Yesterday, I noticed that a scrub of my main system filesystem has
>>>> slowed from about 2.9 gb/sec to about 949 mb/sec. My scrub used to
>>>> run
>>>> in about 12 seconds, now it is taking 51 seconds. I had just
>>>> installed
>>>> Linux kernel 6.4.1 on Arch Linux, upgrading from 6.3.9. At first I
>>>> suspected the new kernel, but now I am not so sure.
>>>
>>> Thanks for the report. It seems it will take some work to address
>>> this,
>>> so to be sure the issue doesn't fall through the cracks unnoticed, I'm
>>> adding it to regzbot, the Linux kernel regression tracking bot:
>>>
>>> #regzbot ^introduced e02ee89baa66
>>
>> #regzbot resolve: various changes merged for 6.6 improve things again;
>> more planned; backporting is planned, too;
>> #regzbot ignore-activity
>>
>> (yes, that is not idea, but that's how it is sometimes)
>
> ideal?
>
> Scrubbing "/home" with 304.61GiB (interestingly both back then with 6.4
> and now with 6.5.2):
>
> - 6.4: 966.84MiB/s
> - 6.5.2:  748.02MiB/s
>
> I expected an improvement.
>
> Same Lenovo ThinkPad T14 AMD Gen 1 with AMD Ryzen 7 PRO 4750U, 32 GiB RAM
> and 2TB Samsung 980 Pro NVME SSD as before.

The fixes didn't arrive until v6.6.

Thanks,
Qu

>
> Ciao,

Re: Scrub of my nvme SSD has slowed by about 2/3

[Martin Steigerwald]

Qu Wenruo - 09.09.23, 00:03:38 CEST:
> > Scrubbing "/home" with 304.61GiB (interestingly both back then with
> > 6.4
> > and now with 6.5.2):
> > 
> > - 6.4: 966.84MiB/s
> > - 6.5.2:  748.02MiB/s
> > 
> > I expected an improvement.
> > 
> > Same Lenovo ThinkPad T14 AMD Gen 1 with AMD Ryzen 7 PRO 4750U, 32 GiB
> > RAM and 2TB Samsung 980 Pro NVME SSD as before.
> 
> The fixes didn't arrive until v6.6.

Ah, okay. That explains it.

Well I can wait.

Best,
-- 
Martin

Re: Scrub of my nvme SSD has slowed by about 2/3

[Martin Steigerwald]

Hello Qu, hello,

Qu Wenruo - 09.09.23, 00:03:38 CEST:
> > Scrubbing "/home" with 304.61GiB (interestingly both back then with
> > 6.4
> > and now with 6.5.2):
> > 
> > - 6.4: 966.84MiB/s
> > - 6.5.2:  748.02MiB/s
> > 
> > I expected an improvement.
> > 
> > Same Lenovo ThinkPad T14 AMD Gen 1 with AMD Ryzen 7 PRO 4750U, 32 GiB
> > RAM and 2TB Samsung 980 Pro NVME SSD as before.
> 
> The fixes didn't arrive until v6.6.

Thank you for making scrub fast again: /home with 1.88GiB/s ;) on 6.5.6.

Best,
-- 
Martin