From: yajun.deng@linux.dev
To: "Song Liu" <song@kernel.org>
Cc: stockhausen@collogia.de,
"open list" <linux-kernel@vger.kernel.org>,
linux-rt-users@vger.kernel.org,
"linux-raid" <linux-raid@vger.kernel.org>,
masahiroy@kernel.org, williams@redhat.com
Subject: Re: [PATCH] lib/raid6: fix abnormally high latency
Date: Wed, 15 Dec 2021 02:14:44 +0000 [thread overview]
Message-ID: <0d07e13a5454dfb03b22e5223d101a1b@linux.dev> (raw)
In-Reply-To: <CAPhsuW5X+zewpKoJLjMMGOUeSiJ1EYqD+0i1bA8y7SFtJLkMeg@mail.gmail.com>
December 15, 2021 1:27 AM, "Song Liu" <song@kernel.org> wrote:
> On Mon, Dec 13, 2021 at 7:17 PM Yajun Deng <yajun.deng@linux.dev> wrote:
>
>> We found an abnormally high latency when executing modprobe raid6_pq, the
>> latency is greater than 1.2s when CONFIG_PREEMPT_VOLUNTARY=y, greater than
>> 67ms when CONFIG_PREEMPT=y, and greater than 16ms when CONFIG_PREEMPT_RT=y.
>> This is caused by disable the preemption, this time is too long and
>> unreasonable. We just need to disable migration. so used migrate_disable()/
>> migrate_enable() instead of preempt_disable()/preempt_enable(). This is
>> beneficial for CONFIG_PREEMPT=y or CONFIG_PREEMPT_RT=y, but no effect for
>> CONFIG_PREEMPT_VOLUNTARY=y.
>>
>> Fixes: fe5cbc6e06c7 ("md/raid6 algorithms: delta syndrome functions")
>> Signed-off-by: Yajun Deng <yajun.deng@linux.dev>
>
> We measure the speed of different RAID algorithms.If we don't disable
> preempt, the result may be inaccurate, right? IIUC, we only disable preempt
> for 16 jiffies. Why do we see 1.2 second delay?
Here are the command of my test：
Execute "sudo cyclictest -S -p 95 -d 0 -i 1000 -D 24h -m" in one terminal and "sudo modprobe raid6_pq" in the other terminal.
Here are the results of my test：
CONFIG_PREEMPT_VOLUNTARY=y，CONFIG_HZ_250=y
T: 0 ( 3092) P:95 I:1000 C: 8514 Min: 1 Act: 2 Avg: 1 Max: 6
T: 1 ( 3093) P:95 I:1000 C: 8511 Min: 1 Act: 2 Avg: 1 Max: 14
T: 2 ( 3094) P:95 I:1000 C: 8509 Min: 1 Act: 2 Avg: 1 Max: 5
T: 3 ( 3095) P:95 I:1000 C: 8506 Min: 1 Act: 2 Avg: 1 Max: 2
T: 4 ( 3096) P:95 I:1000 C: 8504 Min: 1 Act: 2 Avg: 1 Max: 3
T: 5 ( 3097) P:95 I:1000 C: 8502 Min: 1 Act: 2 Avg: 1 Max: 3
T: 6 ( 3098) P:95 I:1000 C: 8499 Min: 1 Act: 2 Avg: 1 Max: 3
T: 7 ( 3099) P:95 I:1000 C: 8497 Min: 1 Act: 1 Avg: 1 Max: 16
T: 8 ( 3100) P:95 I:1000 C: 8495 Min: 1 Act: 2 Avg: 1 Max: 5
T: 9 ( 3101) P:95 I:1000 C: 8493 Min: 1 Act: 2 Avg: 1 Max: 4
T:10 ( 3102) P:95 I:1000 C: 8490 Min: 1 Act: 2 Avg: 1 Max: 13
T:11 ( 3103) P:95 I:1000 C: 8488 Min: 1 Act: 2 Avg: 1 Max: 8
T:12 ( 3104) P:95 I:1000 C: 8486 Min: 1 Act: 2 Avg: 1 Max: 10
T:13 ( 3105) P:95 I:1000 C: 8484 Min: 1 Act: 2 Avg: 2 Max: 6
T:14 ( 3106) P:95 I:1000 C: 8482 Min: 1 Act: 2 Avg: 1 Max: 3
T:15 ( 3107) P:95 I:1000 C: 8479 Min: 1 Act: 2 Avg: 1 Max: 5
T:16 ( 3108) P:95 I:1000 C: 8477 Min: 1 Act: 2 Avg: 2 Max: 137
T:17 ( 3109) P:95 I:1000 C: 7255 Min: 1 Act: 2 Avg: 170 Max: 1220832
T:18 ( 3110) P:95 I:1000 C: 8473 Min: 1 Act: 2 Avg: 1 Max: 5
T:19 ( 3111) P:95 I:1000 C: 8471 Min: 1 Act: 2 Avg: 1 Max: 4
T:20 ( 3112) P:95 I:1000 C: 8469 Min: 1 Act: 2 Avg: 1 Max: 4
T:21 ( 3113) P:95 I:1000 C: 8466 Min: 1 Act: 2 Avg: 1 Max: 3
T:22 ( 3114) P:95 I:1000 C: 8464 Min: 1 Act: 2 Avg: 1 Max: 8
T:23 ( 3115) P:95 I:1000 C: 8462 Min: 1 Act: 2 Avg: 1 Max: 10
T:24 ( 3116) P:95 I:1000 C: 8457 Min: 1 Act: 2 Avg: 1 Max: 9
T:25 ( 3117) P:95 I:1000 C: 8458 Min: 1 Act: 2 Avg: 1 Max: 4
T:26 ( 3118) P:95 I:1000 C: 8456 Min: 1 Act: 2 Avg: 1 Max: 5
T:27 ( 3119) P:95 I:1000 C: 8454 Min: 1 Act: 2 Avg: 1 Max: 2
T:28 ( 3120) P:95 I:1000 C: 8452 Min: 1 Act: 2 Avg: 1 Max: 3
T:29 ( 3121) P:95 I:1000 C: 8450 Min: 2 Act: 2 Avg: 2 Max: 3
T:30 ( 3122) P:95 I:1000 C: 8447 Min: 1 Act: 2 Avg: 1 Max: 2
T:31 ( 3123) P:95 I:1000 C: 8445 Min: 1 Act: 2 Avg: 1 Max: 12
CONFIG_PREEMPT=y,CONFIG_HZ_250=y
T: 0 ( 2460) P:95 I:1000 C: 49161 Min: 1 Act: 2 Avg: 2 Max: 8
T: 1 ( 2461) P:95 I:1000 C: 49158 Min: 1 Act: 2 Avg: 2 Max: 9
T: 2 ( 2462) P:95 I:1000 C: 49024 Min: 1 Act: 2 Avg: 4 Max: 67888
T: 3 ( 2463) P:95 I:1000 C: 49153 Min: 1 Act: 2 Avg: 2 Max: 4
T: 4 ( 2464) P:95 I:1000 C: 48882 Min: 1 Act: 2 Avg: 7 Max: 67726
T: 5 ( 2465) P:95 I:1000 C: 49148 Min: 1 Act: 2 Avg: 2 Max: 5
T: 6 ( 2466) P:95 I:1000 C: 49146 Min: 1 Act: 2 Avg: 1 Max: 6
T: 7 ( 2467) P:95 I:1000 C: 49143 Min: 2 Act: 2 Avg: 2 Max: 6
T: 8 ( 2468) P:95 I:1000 C: 49140 Min: 1 Act: 2 Avg: 2 Max: 929
T: 9 ( 2469) P:95 I:1000 C: 49139 Min: 1 Act: 2 Avg: 2 Max: 8
T:10 ( 2470) P:95 I:1000 C: 49136 Min: 1 Act: 2 Avg: 2 Max: 8
T:11 ( 2471) P:95 I:1000 C: 48866 Min: 1 Act: 2 Avg: 7 Max: 67417
T:12 ( 2472) P:95 I:1000 C: 49132 Min: 1 Act: 2 Avg: 2 Max: 6
T:13 ( 2473) P:95 I:1000 C: 49063 Min: 1 Act: 2 Avg: 3 Max: 67854
T:14 ( 2474) P:95 I:1000 C: 49127 Min: 1 Act: 2 Avg: 2 Max: 5
T:15 ( 2475) P:95 I:1000 C: 49125 Min: 1 Act: 2 Avg: 2 Max: 7
T:16 ( 2476) P:95 I:1000 C: 49123 Min: 1 Act: 2 Avg: 2 Max: 5
T:17 ( 2477) P:95 I:1000 C: 49121 Min: 1 Act: 2 Avg: 2 Max: 6
T:18 ( 2478) P:95 I:1000 C: 49052 Min: 1 Act: 2 Avg: 3 Max: 67717
T:19 ( 2479) P:95 I:1000 C: 49116 Min: 1 Act: 2 Avg: 2 Max: 6
T:20 ( 2480) P:95 I:1000 C: 48913 Min: 1 Act: 2 Avg: 6 Max: 67291
T:21 ( 2481) P:95 I:1000 C: 49112 Min: 1 Act: 2 Avg: 1 Max: 4
T:22 ( 2482) P:95 I:1000 C: 49110 Min: 1 Act: 2 Avg: 2 Max: 5
T:23 ( 2483) P:95 I:1000 C: 49108 Min: 1 Act: 2 Avg: 1 Max: 4
T:24 ( 2484) P:95 I:1000 C: 49105 Min: 1 Act: 2 Avg: 2 Max: 6
T:25 ( 2485) P:95 I:1000 C: 49103 Min: 1 Act: 2 Avg: 2 Max: 7
T:26 ( 2486) P:95 I:1000 C: 49101 Min: 1 Act: 2 Avg: 2 Max: 6
T:27 ( 2487) P:95 I:1000 C: 48898 Min: 1 Act: 2 Avg: 6 Max: 67004
T:28 ( 2488) P:95 I:1000 C: 49097 Min: 1 Act: 2 Avg: 2 Max: 7
T:29 ( 2489) P:95 I:1000 C: 49095 Min: 1 Act: 2 Avg: 1 Max: 3
T:30 ( 2490) P:95 I:1000 C: 49092 Min: 1 Act: 2 Avg: 1 Max: 4
T:31 ( 2491) P:95 I:1000 C: 49090 Min: 1 Act: 2 Avg: 2 Max: 6
CONFIG_PREEMPT_RT=y,CONFIG_HZ_1000=y
T: 0 ( 2555) P:95 I:1000 C: 25336 Min: 2 Act: 2 Avg: 2 Max: 11
T: 1 ( 2556) P:95 I:1000 C: 25332 Min: 2 Act: 4 Avg: 3 Max: 10
T: 2 ( 2557) P:95 I:1000 C: 25329 Min: 3 Act: 3 Avg: 3 Max: 7
T: 3 ( 2558) P:95 I:1000 C: 25326 Min: 3 Act: 3 Avg: 3 Max: 9
T: 4 ( 2559) P:95 I:1000 C: 25322 Min: 3 Act: 3 Avg: 3 Max: 19
T: 5 ( 2560) P:95 I:1000 C: 25319 Min: 3 Act: 3 Avg: 3 Max: 9
T: 6 ( 2561) P:95 I:1000 C: 25284 Min: 3 Act: 3 Avg: 4 Max: 16593
T: 7 ( 2562) P:95 I:1000 C: 25265 Min: 3 Act: 3 Avg: 5 Max: 16375
T: 8 ( 2563) P:95 I:1000 C: 25293 Min: 3 Act: 3 Avg: 3 Max: 16138
T: 9 ( 2564) P:95 I:1000 C: 25290 Min: 3 Act: 5 Avg: 5 Max: 16902
T:10 ( 2565) P:95 I:1000 C: 25287 Min: 3 Act: 3 Avg: 4 Max: 16696
T:11 ( 2566) P:95 I:1000 C: 25300 Min: 3 Act: 3 Avg: 3 Max: 7
T:12 ( 2567) P:95 I:1000 C: 25297 Min: 2 Act: 2 Avg: 2 Max: 9
T:13 ( 2568) P:95 I:1000 C: 25277 Min: 3 Act: 3 Avg: 3 Max: 16179
T:14 ( 2569) P:95 I:1000 C: 25274 Min: 3 Act: 3 Avg: 3 Max: 16043
T:15 ( 2570) P:95 I:1000 C: 25287 Min: 3 Act: 3 Avg: 3 Max: 8
T:16 ( 2571) P:95 I:1000 C: 25284 Min: 2 Act: 2 Avg: 2 Max: 5
T:17 ( 2572) P:95 I:1000 C: 25281 Min: 3 Act: 3 Avg: 3 Max: 7
T:18 ( 2573) P:95 I:1000 C: 25278 Min: 3 Act: 3 Avg: 3 Max: 7
T:19 ( 2574) P:95 I:1000 C: 25275 Min: 3 Act: 3 Avg: 3 Max: 6
T:20 ( 2575) P:95 I:1000 C: 25272 Min: 3 Act: 3 Avg: 3 Max: 7
T:21 ( 2576) P:95 I:1000 C: 25269 Min: 3 Act: 4 Avg: 3 Max: 10
T:22 ( 2577) P:95 I:1000 C: 25234 Min: 2 Act: 3 Avg: 4 Max: 16244
T:23 ( 2578) P:95 I:1000 C: 25231 Min: 2 Act: 4 Avg: 4 Max: 16200
T:24 ( 2579) P:95 I:1000 C: 25259 Min: 3 Act: 3 Avg: 3 Max: 6
T:25 ( 2580) P:95 I:1000 C: 25256 Min: 3 Act: 3 Avg: 3 Max: 8
T:26 ( 2581) P:95 I:1000 C: 25237 Min: 3 Act: 3 Avg: 3 Max: 16078
T:27 ( 2582) P:95 I:1000 C: 25250 Min: 3 Act: 3 Avg: 3 Max: 7
T:28 ( 2583) P:95 I:1000 C: 25247 Min: 3 Act: 3 Avg: 3 Max: 8
T:29 ( 2584) P:95 I:1000 C: 25214 Min: 2 Act: 3 Avg: 4 Max: 15985
T:30 ( 2585) P:95 I:1000 C: 25225 Min: 2 Act: 2 Avg: 3 Max: 16988
T:31 ( 2586) P:95 I:1000 C: 25238 Min: 3 Act: 3 Avg: 3 Max: 12
>
> Thanks,
> Song
>
>> ---
>> lib/raid6/algos.c | 8 ++++----
>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
>> index 6d5e5000fdd7..21611d05c34c 100644
>> --- a/lib/raid6/algos.c
>> +++ b/lib/raid6/algos.c
>> @@ -162,7 +162,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>>
>> perf = 0;
>>
>> - preempt_disable();
>> + migrate_disable();
>> j0 = jiffies;
>> while ((j1 = jiffies) == j0)
>> cpu_relax();
>> @@ -171,7 +171,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>> (*algo)->gen_syndrome(disks, PAGE_SIZE, *dptrs);
>> perf++;
>> }
>> - preempt_enable();
>> + migrate_enable();
>>
>> if (perf > bestgenperf) {
>> bestgenperf = perf;
>> @@ -186,7 +186,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>>
>> perf = 0;
>>
>> - preempt_disable();
>> + migrate_disable();
>> j0 = jiffies;
>> while ((j1 = jiffies) == j0)
>> cpu_relax();
>> @@ -196,7 +196,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>> PAGE_SIZE, *dptrs);
>> perf++;
>> }
>> - preempt_enable();
>> + migrate_enable();
>>
>> if (best == *algo)
>> bestxorperf = perf;
>> --
>> 2.32.0
next prev parent reply other threads:[~2021-12-15 2:14 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-14 3:15 [PATCH] lib/raid6: fix abnormally high latency Yajun Deng
2021-12-14 17:27 ` Song Liu
2021-12-15 2:14 ` yajun.deng [this message]
2021-12-15 16:52 ` Song Liu
2021-12-16 2:15 ` yajun.deng
2021-12-16 6:39 ` Song Liu
2021-12-16 8:09 ` Paul Menzel
2021-12-16 8:27 ` yajun.deng
2021-12-17 21:51 ` Daniel Vacek
2021-12-16 7:04 ` yajun.deng
[not found] ` <CAA7rmPEjdV32_2A8zR0OnbG+kG8EpB3XEnBP7dGSJUm0hpnABg@mail.gmail.com>
2021-12-17 21:43 ` Fwd: " Daniel Vacek
2021-12-18 9:01 ` Song Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0d07e13a5454dfb03b22e5223d101a1b@linux.dev \
--to=yajun.deng@linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=masahiroy@kernel.org \
--cc=song@kernel.org \
--cc=stockhausen@collogia.de \
--cc=williams@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).