Re: Scrub of my nvme SSD has slowed by about 2/3

[Martin Steigerwald <martin@lichtvoll.de>]

Qu Wenruo - 11.07.23, 11:57:52 CEST:
> On 2023/7/11 17:25, Martin Steigerwald wrote:
> > Qu Wenruo - 11.07.23, 10:59:55 CEST:
> >> On 2023/7/11 13:52, Martin Steigerwald wrote:
> >>> Martin Steigerwald - 11.07.23, 07:49:43 CEST:
> >>>> I see about 180000 reads in 10 seconds in atop. I have seen
> >>>> latency
> >>>> values from 55 to 85 µs which is highly unusual for NVME SSD
> >>>> ("avio"
> >>>> in atop¹).
> >>> 
> >>> Well I did not compare to a base line during scrub with 6.3. So
> >>> not actually sure about the unusual bit. But at least during daily
> >>> activity I do not see those values.
> >>> 
> >>> Anyway, I am willing to test a patch.
> >> 
> >> Mind to try the following branch?
> >> 
> >> https://github.com/adam900710/linux/tree/scrub_multi_thread
> >> 
> >> Or you can grab the commit on top and backport to any kernel >=
> >> 6.4.
> > 
> > Cherry picking the commit on top of v6.4.3 lead to a merge conflict.
> > Since this is a production machine and I am no kernel developer with
> > insight to the inner workings of BTRFS, I'd prefer a patch that
> > applies cleanly on top of v6.4.3. I'd rather not try out a tree,
> > unless I know its a stable kernel version or at least rc3/4 or
> > later. Again this is a production machine.
> 
> Well, I have only tested that patch on that development branch, thus I
> can not ensure the result of the backport.
> 
> But still, here you go the backported patch.
> 
> I'd recommend to test the functionality of scrub on some less
> important machine first then on your production latptop though.

I took this calculated risk.

However, while with the patch applied there seem to be more kworker 
threads doing work using 500-600% of CPU time in system (8 cores with 
hyper threading, so 16 logical cores) the result is even less 
performance. Latency values got even worse going up to 0,2 ms. An 
unrelated BTRFS filesystem in another logical volume is even stalled to 
almost a second for (mostly) write accesses.

Scrubbing about 650 to 750 MiB/s for a volume with about 1,2 TiB of 
data, mostly in larger files. Now on second attempt even only 620 MiB/s. 
Which is less than before. Before it reaches about 1 GiB/s. I made sure 
that no desktop search indexing was interfering.

Oh, I forgot to mention, BTRFS uses xxhash here. However it was easily 
scrubbing at 1,5 to 2,5 GiB/s with 5.3. The filesystem uses zstd 
compression and free space tree (free space cache v2).

So from what I can see here, your patch made it worse.

Best,
-- 
Martin