From: yajun.deng@linux.dev
To: "Song Liu" <song@kernel.org>
Cc: stockhausen@collogia.de,
"open list" <linux-kernel@vger.kernel.org>,
linux-rt-users@vger.kernel.org,
"linux-raid" <linux-raid@vger.kernel.org>,
masahiroy@kernel.org, williams@redhat.com
Subject: Re: [PATCH] lib/raid6: fix abnormally high latency
Date: Wed, 15 Dec 2021 02:14:44 +0000 [thread overview]
Message-ID: <0d07e13a5454dfb03b22e5223d101a1b@linux.dev> (raw)
In-Reply-To: <CAPhsuW5X+zewpKoJLjMMGOUeSiJ1EYqD+0i1bA8y7SFtJLkMeg@mail.gmail.com>
December 15, 2021 1:27 AM, "Song Liu" <song@kernel.org> wrote:
> On Mon, Dec 13, 2021 at 7:17 PM Yajun Deng <yajun.deng@linux.dev> wrote:
>
>> We found an abnormally high latency when executing modprobe raid6_pq, the
>> latency is greater than 1.2s when CONFIG_PREEMPT_VOLUNTARY=y, greater than
>> 67ms when CONFIG_PREEMPT=y, and greater than 16ms when CONFIG_PREEMPT_RT=y.
>> This is caused by disable the preemption, this time is too long and
>> unreasonable. We just need to disable migration. so used migrate_disable()/
>> migrate_enable() instead of preempt_disable()/preempt_enable(). This is
>> beneficial for CONFIG_PREEMPT=y or CONFIG_PREEMPT_RT=y, but no effect for
>> CONFIG_PREEMPT_VOLUNTARY=y.
>>
>> Fixes: fe5cbc6e06c7 ("md/raid6 algorithms: delta syndrome functions")
>> Signed-off-by: Yajun Deng <yajun.deng@linux.dev>
>
> We measure the speed of different RAID algorithms.If we don't disable
> preempt, the result may be inaccurate, right? IIUC, we only disable preempt
> for 16 jiffies. Why do we see 1.2 second delay?
Here are the command of my test：
Execute "sudo cyclictest -S -p 95 -d 0 -i 1000 -D 24h -m" in one terminal and "sudo modprobe raid6_pq" in the other terminal.
Here are the results of my test：
CONFIG_PREEMPT_VOLUNTARY=y，CONFIG_HZ_250=y
T: 0 ( 3092) P:95 I:1000 C: 8514 Min: 1 Act: 2 Avg: 1 Max: 6
T: 1 ( 3093) P:95 I:1000 C: 8511 Min: 1 Act: 2 Avg: 1 Max: 14
T: 2 ( 3094) P:95 I:1000 C: 8509 Min: 1 Act: 2 Avg: 1 Max: 5
T: 3 ( 3095) P:95 I:1000 C: 8506 Min: 1 Act: 2 Avg: 1 Max: 2
T: 4 ( 3096) P:95 I:1000 C: 8504 Min: 1 Act: 2 Avg: 1 Max: 3
T: 5 ( 3097) P:95 I:1000 C: 8502 Min: 1 Act: 2 Avg: 1 Max: 3
T: 6 ( 3098) P:95 I:1000 C: 8499 Min: 1 Act: 2 Avg: 1 Max: 3
T: 7 ( 3099) P:95 I:1000 C: 8497 Min: 1 Act: 1 Avg: 1 Max: 16
T: 8 ( 3100) P:95 I:1000 C: 8495 Min: 1 Act: 2 Avg: 1 Max: 5
T: 9 ( 3101) P:95 I:1000 C: 8493 Min: 1 Act: 2 Avg: 1 Max: 4
T:10 ( 3102) P:95 I:1000 C: 8490 Min: 1 Act: 2 Avg: 1 Max: 13
T:11 ( 3103) P:95 I:1000 C: 8488 Min: 1 Act: 2 Avg: 1 Max: 8
T:12 ( 3104) P:95 I:1000 C: 8486 Min: 1 Act: 2 Avg: 1 Max: 10
T:13 ( 3105) P:95 I:1000 C: 8484 Min: 1 Act: 2 Avg: 2 Max: 6
T:14 ( 3106) P:95 I:1000 C: 8482 Min: 1 Act: 2 Avg: 1 Max: 3
T:15 ( 3107) P:95 I:1000 C: 8479 Min: 1 Act: 2 Avg: 1 Max: 5
T:16 ( 3108) P:95 I:1000 C: 8477 Min: 1 Act: 2 Avg: 2 Max: 137
T:17 ( 3109) P:95 I:1000 C: 7255 Min: 1 Act: 2 Avg: 170 Max: 1220832
T:18 ( 3110) P:95 I:1000 C: 8473 Min: 1 Act: 2 Avg: 1 Max: 5
T:19 ( 3111) P:95 I:1000 C: 8471 Min: 1 Act: 2 Avg: 1 Max: 4
T:20 ( 3112) P:95 I:1000 C: 8469 Min: 1 Act: 2 Avg: 1 Max: 4
T:21 ( 3113) P:95 I:1000 C: 8466 Min: 1 Act: 2 Avg: 1 Max: 3
T:22 ( 3114) P:95 I:1000 C: 8464 Min: 1 Act: 2 Avg: 1 Max: 8
T:23 ( 3115) P:95 I:1000 C: 8462 Min: 1 Act: 2 Avg: 1 Max: 10
T:24 ( 3116) P:95 I:1000 C: 8457 Min: 1 Act: 2 Avg: 1 Max: 9
T:25 ( 3117) P:95 I:1000 C: 8458 Min: 1 Act: 2 Avg: 1 Max: 4
T:26 ( 3118) P:95 I:1000 C: 8456 Min: 1 Act: 2 Avg: 1 Max: 5
T:27 ( 3119) P:95 I:1000 C: 8454 Min: 1 Act: 2 Avg: 1 Max: 2
T:28 ( 3120) P:95 I:1000 C: 8452 Min: 1 Act: 2 Avg: 1 Max: 3
T:29 ( 3121) P:95 I:1000 C: 8450 Min: 2 Act: 2 Avg: 2 Max: 3
T:30 ( 3122) P:95 I:1000 C: 8447 Min: 1 Act: 2 Avg: 1 Max: 2
T:31 ( 3123) P:95 I:1000 C: 8445 Min: 1 Act: 2 Avg: 1 Max: 12
CONFIG_PREEMPT=y,CONFIG_HZ_250=y
T: 0 ( 2460) P:95 I:1000 C: 49161 Min: 1 Act: 2 Avg: 2 Max: 8
T: 1 ( 2461) P:95 I:1000 C: 49158 Min: 1 Act: 2 Avg: 2 Max: 9
T: 2 ( 2462) P:95 I:1000 C: 49024 Min: 1 Act: 2 Avg: 4 Max: 67888
T: 3 ( 2463) P:95 I:1000 C: 49153 Min: 1 Act: 2 Avg: 2 Max: 4
T: 4 ( 2464) P:95 I:1000 C: 48882 Min: 1 Act: 2 Avg: 7 Max: 67726
T: 5 ( 2465) P:95 I:1000 C: 49148 Min: 1 Act: 2 Avg: 2 Max: 5
T: 6 ( 2466) P:95 I:1000 C: 49146 Min: 1 Act: 2 Avg: 1 Max: 6
T: 7 ( 2467) P:95 I:1000 C: 49143 Min: 2 Act: 2 Avg: 2 Max: 6
T: 8 ( 2468) P:95 I:1000 C: 49140 Min: 1 Act: 2 Avg: 2 Max: 929
T: 9 ( 2469) P:95 I:1000 C: 49139 Min: 1 Act: 2 Avg: 2 Max: 8
T:10 ( 2470) P:95 I:1000 C: 49136 Min: 1 Act: 2 Avg: 2 Max: 8
T:11 ( 2471) P:95 I:1000 C: 48866 Min: 1 Act: 2 Avg: 7 Max: 67417
T:12 ( 2472) P:95 I:1000 C: 49132 Min: 1 Act: 2 Avg: 2 Max: 6
T:13 ( 2473) P:95 I:1000 C: 49063 Min: 1 Act: 2 Avg: 3 Max: 67854
T:14 ( 2474) P:95 I:1000 C: 49127 Min: 1 Act: 2 Avg: 2 Max: 5
T:15 ( 2475) P:95 I:1000 C: 49125 Min: 1 Act: 2 Avg: 2 Max: 7
T:16 ( 2476) P:95 I:1000 C: 49123 Min: 1 Act: 2 Avg: 2 Max: 5
T:17 ( 2477) P:95 I:1000 C: 49121 Min: 1 Act: 2 Avg: 2 Max: 6
T:18 ( 2478) P:95 I:1000 C: 49052 Min: 1 Act: 2 Avg: 3 Max: 67717
T:19 ( 2479) P:95 I:1000 C: 49116 Min: 1 Act: 2 Avg: 2 Max: 6
T:20 ( 2480) P:95 I:1000 C: 48913 Min: 1 Act: 2 Avg: 6 Max: 67291
T:21 ( 2481) P:95 I:1000 C: 49112 Min: 1 Act: 2 Avg: 1 Max: 4
T:22 ( 2482) P:95 I:1000 C: 49110 Min: 1 Act: 2 Avg: 2 Max: 5
T:23 ( 2483) P:95 I:1000 C: 49108 Min: 1 Act: 2 Avg: 1 Max: 4
T:24 ( 2484) P:95 I:1000 C: 49105 Min: 1 Act: 2 Avg: 2 Max: 6
T:25 ( 2485) P:95 I:1000 C: 49103 Min: 1 Act: 2 Avg: 2 Max: 7
T:26 ( 2486) P:95 I:1000 C: 49101 Min: 1 Act: 2 Avg: 2 Max: 6
T:27 ( 2487) P:95 I:1000 C: 48898 Min: 1 Act: 2 Avg: 6 Max: 67004
T:28 ( 2488) P:95 I:1000 C: 49097 Min: 1 Act: 2 Avg: 2 Max: 7
T:29 ( 2489) P:95 I:1000 C: 49095 Min: 1 Act: 2 Avg: 1 Max: 3
T:30 ( 2490) P:95 I:1000 C: 49092 Min: 1 Act: 2 Avg: 1 Max: 4
T:31 ( 2491) P:95 I:1000 C: 49090 Min: 1 Act: 2 Avg: 2 Max: 6
CONFIG_PREEMPT_RT=y,CONFIG_HZ_1000=y
T: 0 ( 2555) P:95 I:1000 C: 25336 Min: 2 Act: 2 Avg: 2 Max: 11
T: 1 ( 2556) P:95 I:1000 C: 25332 Min: 2 Act: 4 Avg: 3 Max: 10
T: 2 ( 2557) P:95 I:1000 C: 25329 Min: 3 Act: 3 Avg: 3 Max: 7
T: 3 ( 2558) P:95 I:1000 C: 25326 Min: 3 Act: 3 Avg: 3 Max: 9
T: 4 ( 2559) P:95 I:1000 C: 25322 Min: 3 Act: 3 Avg: 3 Max: 19
T: 5 ( 2560) P:95 I:1000 C: 25319 Min: 3 Act: 3 Avg: 3 Max: 9
T: 6 ( 2561) P:95 I:1000 C: 25284 Min: 3 Act: 3 Avg: 4 Max: 16593
T: 7 ( 2562) P:95 I:1000 C: 25265 Min: 3 Act: 3 Avg: 5 Max: 16375
T: 8 ( 2563) P:95 I:1000 C: 25293 Min: 3 Act: 3 Avg: 3 Max: 16138
T: 9 ( 2564) P:95 I:1000 C: 25290 Min: 3 Act: 5 Avg: 5 Max: 16902
T:10 ( 2565) P:95 I:1000 C: 25287 Min: 3 Act: 3 Avg: 4 Max: 16696
T:11 ( 2566) P:95 I:1000 C: 25300 Min: 3 Act: 3 Avg: 3 Max: 7
T:12 ( 2567) P:95 I:1000 C: 25297 Min: 2 Act: 2 Avg: 2 Max: 9
T:13 ( 2568) P:95 I:1000 C: 25277 Min: 3 Act: 3 Avg: 3 Max: 16179
T:14 ( 2569) P:95 I:1000 C: 25274 Min: 3 Act: 3 Avg: 3 Max: 16043
T:15 ( 2570) P:95 I:1000 C: 25287 Min: 3 Act: 3 Avg: 3 Max: 8
T:16 ( 2571) P:95 I:1000 C: 25284 Min: 2 Act: 2 Avg: 2 Max: 5
T:17 ( 2572) P:95 I:1000 C: 25281 Min: 3 Act: 3 Avg: 3 Max: 7
T:18 ( 2573) P:95 I:1000 C: 25278 Min: 3 Act: 3 Avg: 3 Max: 7
T:19 ( 2574) P:95 I:1000 C: 25275 Min: 3 Act: 3 Avg: 3 Max: 6
T:20 ( 2575) P:95 I:1000 C: 25272 Min: 3 Act: 3 Avg: 3 Max: 7
T:21 ( 2576) P:95 I:1000 C: 25269 Min: 3 Act: 4 Avg: 3 Max: 10
T:22 ( 2577) P:95 I:1000 C: 25234 Min: 2 Act: 3 Avg: 4 Max: 16244
T:23 ( 2578) P:95 I:1000 C: 25231 Min: 2 Act: 4 Avg: 4 Max: 16200
T:24 ( 2579) P:95 I:1000 C: 25259 Min: 3 Act: 3 Avg: 3 Max: 6
T:25 ( 2580) P:95 I:1000 C: 25256 Min: 3 Act: 3 Avg: 3 Max: 8
T:26 ( 2581) P:95 I:1000 C: 25237 Min: 3 Act: 3 Avg: 3 Max: 16078
T:27 ( 2582) P:95 I:1000 C: 25250 Min: 3 Act: 3 Avg: 3 Max: 7
T:28 ( 2583) P:95 I:1000 C: 25247 Min: 3 Act: 3 Avg: 3 Max: 8
T:29 ( 2584) P:95 I:1000 C: 25214 Min: 2 Act: 3 Avg: 4 Max: 15985
T:30 ( 2585) P:95 I:1000 C: 25225 Min: 2 Act: 2 Avg: 3 Max: 16988
T:31 ( 2586) P:95 I:1000 C: 25238 Min: 3 Act: 3 Avg: 3 Max: 12
>
> Thanks,
> Song
>
>> ---
>> lib/raid6/algos.c | 8 ++++----
>> 1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
>> index 6d5e5000fdd7..21611d05c34c 100644
>> --- a/lib/raid6/algos.c
>> +++ b/lib/raid6/algos.c
>> @@ -162,7 +162,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>>
>> perf = 0;
>>
>> - preempt_disable();
>> + migrate_disable();
>> j0 = jiffies;
>> while ((j1 = jiffies) == j0)
>> cpu_relax();
>> @@ -171,7 +171,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>> (*algo)->gen_syndrome(disks, PAGE_SIZE, *dptrs);
>> perf++;
>> }
>> - preempt_enable();
>> + migrate_enable();
>>
>> if (perf > bestgenperf) {
>> bestgenperf = perf;
>> @@ -186,7 +186,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>>
>> perf = 0;
>>
>> - preempt_disable();
>> + migrate_disable();
>> j0 = jiffies;
>> while ((j1 = jiffies) == j0)
>> cpu_relax();
>> @@ -196,7 +196,7 @@ static inline const struct raid6_calls *raid6_choose_gen(
>> PAGE_SIZE, *dptrs);
>> perf++;
>> }
>> - preempt_enable();
>> + migrate_enable();
>>
>> if (best == *algo)
>> bestxorperf = perf;
>> --
>> 2.32.0
next prev parent reply other threads:[~2021-12-15 2:14 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-14 3:15 [PATCH] lib/raid6: fix abnormally high latency Yajun Deng
2021-12-14 17:27 ` Song Liu
2021-12-15 2:14 ` yajun.deng [this message]
2021-12-15 16:52 ` Song Liu
2021-12-16 2:15 ` yajun.deng
2021-12-16 6:39 ` Song Liu
2021-12-16 7:04 ` yajun.deng
2021-12-16 8:09 ` Paul Menzel
2021-12-16 8:27 ` yajun.deng
2021-12-17 21:51 ` Daniel Vacek
[not found] ` <CAA7rmPEjdV32_2A8zR0OnbG+kG8EpB3XEnBP7dGSJUm0hpnABg@mail.gmail.com>
2021-12-17 21:43 ` Fwd: " Daniel Vacek
2021-12-18 9:01 ` Song Liu
-- strict thread matches above, loose matches on Subject: below --
2021-12-14 2:59 Yajun Deng
2021-12-14 2:55 Yajun Deng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0d07e13a5454dfb03b22e5223d101a1b@linux.dev \
--to=yajun.deng@linux.dev \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=masahiroy@kernel.org \
--cc=song@kernel.org \
--cc=stockhausen@collogia.de \
--cc=williams@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.