Re: Scrub of my nvme SSD has slowed by about 2/3

[Martin Steigerwald <martin@lichtvoll.de>]

Qu Wenruo - 11.07.23, 13:05:42 CEST:
> On 2023/7/11 18:56, Martin Steigerwald wrote:
> > Qu Wenruo - 11.07.23, 11:57:52 CEST:
> >> On 2023/7/11 17:25, Martin Steigerwald wrote:
> >>> Qu Wenruo - 11.07.23, 10:59:55 CEST:
> >>>> On 2023/7/11 13:52, Martin Steigerwald wrote:
> >>>>> Martin Steigerwald - 11.07.23, 07:49:43 CEST:
> >>>>>> I see about 180000 reads in 10 seconds in atop. I have seen
> >>>>>> latency
> >>>>>> values from 55 to 85 µs which is highly unusual for NVME SSD
> >>>>>> ("avio"
> >>>>>> in atop¹).
[…]
> >>>> Mind to try the following branch?
> >>>> 
> >>>> https://github.com/adam900710/linux/tree/scrub_multi_thread
> >>>> 
> >>>> Or you can grab the commit on top and backport to any kernel >=
> >>>> 6.4.
> >>> 
> >>> Cherry picking the commit on top of v6.4.3 lead to a merge
> >>> conflict.
[…]
> >> Well, I have only tested that patch on that development branch,
> >> thus I can not ensure the result of the backport.
> >> 
> >> But still, here you go the backported patch.
> >> 
> >> I'd recommend to test the functionality of scrub on some less
> >> important machine first then on your production latptop though.
> > 
> > I took this calculated risk.
> > 
> > However, while with the patch applied there seem to be more kworker
> > threads doing work using 500-600% of CPU time in system (8 cores
> > with
> > hyper threading, so 16 logical cores) the result is even less
> > performance. Latency values got even worse going up to 0,2 ms. An
> > unrelated BTRFS filesystem in another logical volume is even stalled
> > to almost a second for (mostly) write accesses.
> > 
> > Scrubbing about 650 to 750 MiB/s for a volume with about 1,2 TiB of
> > data, mostly in larger files. Now on second attempt even only 620
> > MiB/s. Which is less than before. Before it reaches about 1 GiB/s.
> > I made sure that no desktop search indexing was interfering.
> > 
> > Oh, I forgot to mention, BTRFS uses xxhash here. However it was
> > easily scrubbing at 1,5 to 2,5 GiB/s with 5.3. The filesystem uses
> > zstd compression and free space tree (free space cache v2).
> > 
> > So from what I can see here, your patch made it worse.
> 
> Thanks for the confirming, this at least prove it's not the hashing
> threads limit causing the regression.
> 
> Which is pretty weird, the read pattern is in fact better than the
> original behavior, all read are in 64K (even if there are some holes,
> we are fine reading the garbage, this should reduce IOPS workload),
> and we submit a batch of 8 of such read in one go.
> 
> BTW, what's the CPU usage of v6.3 kernel? Is it higher or lower?
> And what about the latency?

CPU usage is between 600-700% on 6.3.9, Latency between 50-70 µs. And 
scrubbing speed is above 2 GiB/s, peaking at 2,27 GiB/s. Later it went 
down a bit to 1,7 GiB/s, maybe due to background activity.

I'd say the CPU usage to result (=scrubbing speed) ratio is much, much 
better with 6.3. However the latencies during scrubbing are pretty much 
the same. I even seen up to 0.2 ms.

> Currently I'm out of ideas, for now you can revert that testing patch.
> 
> If you're interested in more testing, you can apply the following
> small diff, which changed the batch number of scrub.
> 
> You can try either double or half the number to see which change helps
> more.

No time for further testing at the moment. Maybe at a later time.

It might be good you put together a test setup yourself. Any computer 
with NVME SSD should do I think. Unless there is something very special 
about my laptop, but I doubt this. This reduces greatly on the turn-
around time.

I think for now I am back at 6.3. It works. :)

-- 
Martin