disk initialize priority

disk initialize priority

[d tbsky]

Hi:
     I just tried to install rhel 7.3 on a 4 disk raid6. it's a
minimal install but took 3 hours to finish. I found the disk
initialize speed is around 150MB/sec. in past the disk initialize
speed will down to slow speed when there are other disk IO, but it
seems not the case in rhel 7.3.

    I have check the below parameter:
    cat /proc/sys/dev/raid/speed_limit_min  -> 1000
    cat /proc/sys/dev/raid/speed_limit_max -> 200000

    when the system is idle, the init speed is also about 150M/sec, so
the system is use all the power to init when there are other disk IO.

     is it normal? where should I check?

     thanks a lot for help!!


Regards,
tbskyd

Re: disk initialize priority

[Reindl Harald]

Am 13.04.2017 um 10:13 schrieb d tbsky:
> Hi:
>       I just tried to install rhel 7.3 on a 4 disk raid6. it's a
> minimal install but took 3 hours to finish. I found the disk
> initialize speed is around 150MB/sec. in past the disk initialize
> speed will down to slow speed when there are other disk IO, but it
> seems not the case in rhel 7.3.
> 
>      I have check the below parameter:
>      cat /proc/sys/dev/raid/speed_limit_min  -> 1000
>      cat /proc/sys/dev/raid/speed_limit_max -> 200000
> 
>      when the system is idle, the init speed is also about 150M/sec, so
> the system is use all the power to init when there are other disk IO.
> 
>       is it normal? where should I check?

sysctl.conf:
dev.raid.speed_limit_min = 10000
dev.raid.speed_limit_max = 50000

sysctl -p

and it will adjust itself within seconds - the same works if that values 
are too low and raid-check would take forever

Re: disk initialize priority

[d tbsky]

2017-04-13 17:09 GMT+08:00 Reindl Harald <h.reindl@thelounge.net>:
>
>
> Am 13.04.2017 um 10:13 schrieb d tbsky:
>>
>> Hi:
>>       I just tried to install rhel 7.3 on a 4 disk raid6. it's a
>> minimal install but took 3 hours to finish. I found the disk
>> initialize speed is around 150MB/sec. in past the disk initialize
>> speed will down to slow speed when there are other disk IO, but it
>> seems not the case in rhel 7.3.
>>
>>      I have check the below parameter:
>>      cat /proc/sys/dev/raid/speed_limit_min  -> 1000
>>      cat /proc/sys/dev/raid/speed_limit_max -> 200000
>>
>>      when the system is idle, the init speed is also about 150M/sec, so
>> the system is use all the power to init when there are other disk IO.
>>
>>       is it normal? where should I check?
>
>
> sysctl.conf:
> dev.raid.speed_limit_min = 10000
> dev.raid.speed_limit_max = 50000
>
> sysctl -p
>
> and it will adjust itself within seconds - the same works if that values are
> too low and raid-check would take forever

hi:
    my problem is the speed is too high, not too low...

Regards,
tbskyd

Re: disk initialize priority

[Reindl Harald]

Am 13.04.2017 um 11:44 schrieb d tbsky:
> 2017-04-13 17:09 GMT+08:00 Reindl Harald <h.reindl@thelounge.net>:
>>
>>
>> Am 13.04.2017 um 10:13 schrieb d tbsky:
>>>
>>> Hi:
>>>        I just tried to install rhel 7.3 on a 4 disk raid6. it's a
>>> minimal install but took 3 hours to finish. I found the disk
>>> initialize speed is around 150MB/sec. in past the disk initialize
>>> speed will down to slow speed when there are other disk IO, but it
>>> seems not the case in rhel 7.3.
>>>
>>>       I have check the below parameter:
>>>       cat /proc/sys/dev/raid/speed_limit_min  -> 1000
>>>       cat /proc/sys/dev/raid/speed_limit_max -> 200000
>>>
>>>       when the system is idle, the init speed is also about 150M/sec, so
>>> the system is use all the power to init when there are other disk IO.
>>>
>>>        is it normal? where should I check?
>>
>>
>> sysctl.conf:
>> dev.raid.speed_limit_min = 10000
>> dev.raid.speed_limit_max = 50000
>>
>> sysctl -p
>>
>> and it will adjust itself within seconds - the same works if that values are
>> too low and raid-check would take forever
> 
> hi: my problem is the speed is too high, not too low...

and?

you said it is at 200000
i showed you how to reduce it to 50000 or whatever you want

Re: disk initialize priority

[d tbsky]

2017-04-13 17:47 GMT+08:00 Reindl Harald <h.reindl@thelounge.net>:
>
>
> Am 13.04.2017 um 11:44 schrieb d tbsky:
>>
>> 2017-04-13 17:09 GMT+08:00 Reindl Harald <h.reindl@thelounge.net>:
>>>
>>>
>>>
>>> Am 13.04.2017 um 10:13 schrieb d tbsky:
>>>>
>>>>
>>>> Hi:
>>>>        I just tried to install rhel 7.3 on a 4 disk raid6. it's a
>>>> minimal install but took 3 hours to finish. I found the disk
>>>> initialize speed is around 150MB/sec. in past the disk initialize
>>>> speed will down to slow speed when there are other disk IO, but it
>>>> seems not the case in rhel 7.3.
>>>>
>>>>       I have check the below parameter:
>>>>       cat /proc/sys/dev/raid/speed_limit_min  -> 1000
>>>>       cat /proc/sys/dev/raid/speed_limit_max -> 200000
>>>>
>>>>       when the system is idle, the init speed is also about 150M/sec, so
>>>> the system is use all the power to init when there are other disk IO.
>>>>
>>>>        is it normal? where should I check?
>>>
>>>
>>>
>>> sysctl.conf:
>>> dev.raid.speed_limit_min = 10000
>>> dev.raid.speed_limit_max = 50000
>>>
>>> sysctl -p
>>>
>>> and it will adjust itself within seconds - the same works if that values
>>> are
>>> too low and raid-check would take forever
>>
>>
>> hi: my problem is the speed is too high, not too low...
>
>
> and?
>
> you said it is at 200000
> i showed you how to reduce it to 50000 or whatever you want

hi:
   sorry I didn't understand you correctly. thanks for the hint. I
will try to resolve the root cause of the problem.

Regards,
tbskyd

Re: disk initialize priority

[Reindl Harald]

Am 13.04.2017 um 11:58 schrieb d tbsky:
> 2017-04-13 17:47 GMT+08:00 Reindl Harald <h.reindl@thelounge.net>:
>>
>>
>> Am 13.04.2017 um 11:44 schrieb d tbsky:
>>>
>>> 2017-04-13 17:09 GMT+08:00 Reindl Harald <h.reindl@thelounge.net>:
>>>>
>>>>
>>>>
>>>> Am 13.04.2017 um 10:13 schrieb d tbsky:
>>>>>
>>>>>
>>>>> Hi:
>>>>>         I just tried to install rhel 7.3 on a 4 disk raid6. it's a
>>>>> minimal install but took 3 hours to finish. I found the disk
>>>>> initialize speed is around 150MB/sec. in past the disk initialize
>>>>> speed will down to slow speed when there are other disk IO, but it
>>>>> seems not the case in rhel 7.3.
>>>>>
>>>>>        I have check the below parameter:
>>>>>        cat /proc/sys/dev/raid/speed_limit_min  -> 1000
>>>>>        cat /proc/sys/dev/raid/speed_limit_max -> 200000
>>>>>
>>>>>        when the system is idle, the init speed is also about 150M/sec, so
>>>>> the system is use all the power to init when there are other disk IO.
>>>>>
>>>>>         is it normal? where should I check?
>>>>
>>>>
>>>>
>>>> sysctl.conf:
>>>> dev.raid.speed_limit_min = 10000
>>>> dev.raid.speed_limit_max = 50000
>>>>
>>>> sysctl -p
>>>>
>>>> and it will adjust itself within seconds - the same works if that values
>>>> are
>>>> too low and raid-check would take forever
>>>
>>>
>>> hi: my problem is the speed is too high, not too low...
>>
>> and?
>>
>> you said it is at 200000
>> i showed you how to reduce it to 50000 or whatever you want
> 
> hi:
> sorry I didn't understand you correctly. thanks for the hint. I
> will try to resolve the root cause of the problem

the root cause is a default which is too large for your disks wqhile on 
the other hand if someone has a RAID10 with SSD disks it's even too low 
and you should just configure that for your needs and workload

it's not only about how fast are your disks, it's also about your 
workload, there are times where i set it to 500000 to get the weekly 
raid-scrub as fast as possible when i don't heavily use the machine and 
on other days i slow it down dramatically because i need performance and 
don't care when the background task is finished

it's also a difference between RAID1 or RAID10 when concurrency reads 
can be served by the other disk and the impact is not so big than on a RAID1

Re: disk initialize priority

[Roman Mamedov]

On Thu, 13 Apr 2017 11:47:29 +0200
Reindl Harald <h.reindl@thelounge.net> wrote:

> and?
> 
> you said it is at 200000
> i showed you how to reduce it to 50000 or whatever you want

The point is that this should automatically adjust between 1000 and 200000 (on
default settings) according to other IO in the system. But since April 2015
this automatic tuning got broken and has worsened significantly for some
scenarios. The solution should be not that "well you're SOL and now always
have to hand-tune it", but to keep reporting your experience and try out the
proposed fixes so the prior automated adjustment is reimplemented, or baring
that, to revert the behavior change on your systems (already tried that just
now on my kernel build, it still reverts cleanly).

-- 
With respect,
Roman

Re: disk initialize priority

[d tbsky]

2017-04-13 18:05 GMT+08:00 Roman Mamedov <rm@romanrm.net>:
> On Thu, 13 Apr 2017 11:47:29 +0200
> Reindl Harald <h.reindl@thelounge.net> wrote:
>
>> and?
>>
>> you said it is at 200000
>> i showed you how to reduce it to 50000 or whatever you want
>
> The point is that this should automatically adjust between 1000 and 200000 (on
> default settings) according to other IO in the system. But since April 2015
> this automatic tuning got broken and has worsened significantly for some
> scenarios. The solution should be not that "well you're SOL and now always
> have to hand-tune it", but to keep reporting your experience and try out the
> proposed fixes so the prior automated adjustment is reimplemented, or baring
> that, to revert the behavior change on your systems (already tried that just
> now on my kernel build, it still reverts cleanly).
>
> --
> With respect,
> Roman

hi:
     RHEL 7 use kernel 3.10 so it should not be affected. at least
RHEL 7.2 I didn't notice this kind of problem. maybe RHEL 7.3 pull
some feature/patch from upstream kernel. I will try to report to
redhat bugzilla to see if they can fix these.

     thanks again for all the hints and help!!

Regards,
tbskyd

Re: disk initialize priority

[Roman Mamedov]

On Thu, 13 Apr 2017 18:25:44 +0800
d tbsky <tbskyd@gmail.com> wrote:

>      RHEL 7 use kernel 3.10 so it should not be affected. at least
> RHEL 7.2 I didn't notice this kind of problem. maybe RHEL 7.3 pull
> some feature/patch from upstream kernel. I will try to report to
> redhat bugzilla to see if they can fix these.

Out of curiousity I checked the RHEL 7.3 kernel 3.10 source as available at:
http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.10.2.el7.src.rpm
and yes indeed they have included that patch from the upstream 4.x kernel.

-- 
With respect,
Roman

Re: disk initialize priority

[d tbsky]

2017-04-13 18:34 GMT+08:00 Roman Mamedov <rm@romanrm.net>:
> On Thu, 13 Apr 2017 18:25:44 +0800
> d tbsky <tbskyd@gmail.com> wrote:
>
>>      RHEL 7 use kernel 3.10 so it should not be affected. at least
>> RHEL 7.2 I didn't notice this kind of problem. maybe RHEL 7.3 pull
>> some feature/patch from upstream kernel. I will try to report to
>> redhat bugzilla to see if they can fix these.
>
> Out of curiousity I checked the RHEL 7.3 kernel 3.10 source as available at:
> http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.10.2.el7.src.rpm
> and yes indeed they have included that patch from the upstream 4.x kernel.
>
> --
> With respect,
> Roman

hi:
     you are really life saver. now I can easily report the bugs to redhat.
     thanks again for your patient and help!!

Regards,
tbskyd

Re: disk initialize priority

[Stephane Thiell]

> On Apr 13, 2017, at 3:34 AM, Roman Mamedov <rm@romanrm.net> wrote:
> 
> On Thu, 13 Apr 2017 18:25:44 +0800
> d tbsky <tbskyd@gmail.com> wrote:
> 
>>     RHEL 7 use kernel 3.10 so it should not be affected. at least
>> RHEL 7.2 I didn't notice this kind of problem. maybe RHEL 7.3 pull
>> some feature/patch from upstream kernel. I will try to report to
>> redhat bugzilla to see if they can fix these.
> 
> Out of curiousity I checked the RHEL 7.3 kernel 3.10 source as available at:
> http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.10.2.el7.src.rpm
> and yes indeed they have included that patch from the upstream 4.x kernel.
> 

Interesting.

Just to say that I have noticed a similar issue on CentOS 7.3 (using 3.10.0-514.10.2.el7_lustre.x86_64). I am using multiple raid6 md (8+2) and I had to deploy a cron script that checks if each raid is degraded or not, and set speed_limit_max accordingly (eg. higher “priority” for rebuild if array degraded) but always below the max throughput… or other user I/Os become too slow. Not ideal but works fine this way.

Best regards,

Stéphane

Re: disk initialize priority

[Roman Mamedov]

On Thu, 13 Apr 2017 16:13:41 +0800
d tbsky <tbskyd@gmail.com> wrote:

> in past the disk initialize
> speed will down to slow speed when there are other disk IO, but it
> seems not the case in rhel 7.3.

It was changed some time ago with:

  "md: allow resync to go faster when there is competing IO"
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ac8fa4196d205ac8fff3f8932bddbad4f16e4110

and a problem like you describe has been reported:

  https://www.spinics.net/lists/raid/msg51363.html

Some patches have been discussed, but apparently there is no fix for it up to
this day, as the current code in question looks the same as in the above patch.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/md/md.c#n8138

-- 
With respect,
Roman

Re: disk initialize priority

[d tbsky]

2017-04-13 17:37 GMT+08:00 Roman Mamedov <rm@romanrm.net>:
> On Thu, 13 Apr 2017 16:13:41 +0800
> d tbsky <tbskyd@gmail.com> wrote:
>
>> in past the disk initialize
>> speed will down to slow speed when there are other disk IO, but it
>> seems not the case in rhel 7.3.
>
> It was changed some time ago with:
>
>   "md: allow resync to go faster when there is competing IO"
>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ac8fa4196d205ac8fff3f8932bddbad4f16e4110
>
> and a problem like you describe has been reported:
>
>   https://www.spinics.net/lists/raid/msg51363.html
>
> Some patches have been discussed, but apparently there is no fix for it up to
> this day, as the current code in question looks the same as in the above patch.
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/md/md.c#n8138
>
> --
> With respect,
> Roman

hi:
    thanks for the hint. I will try to report RHEL to see what
happened. I didn't notice these in previous versions.

Regards,
tbskyd