RAID10 performance with 20 drives

RAID10 performance with 20 drives

[CoolCold]

Hello!
Got a new box, for image storage, playing around, created raid10 array
with 20 1.8TB SATA drives, and found that we hit the cpu limit,
details below.

Creation (disable write intent bitmap, with bitmap all is much worse):
mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
/dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
/dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
/dev/sdu /dev/sdv /dev/sdw /dev/sdx

/proc/mdstat output:
[root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid10 sdx[19] sdw[18] sdv[17] sdu[16] sdt[15] sds[14]
sdr[13] sdq[12] sdp[11] sdo[10] sdn[9] sdm[8] sdl[7] sdk[6] sdj[5]
sdi[4] sdh[3] sdg[2] sdf[1] sde[0]
      17580330880 blocks super 1.2 64K chunks 2 near-copies [20/20]
[UUUUUUUUUUUUUUUUUUUU]
      [=>...................]  resync =  6.4% (1133170368/17580330880)
finish=192.6min speed=1423140K/sec

top stat:
top - 12:09:03 up  4:55,  2 users,  load average: 3.33, 3.18, 2.88
Tasks: 487 total,   4 running, 483 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  4.5 sy,  0.0 ni, 95.3 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 13174918+total, 13005539+free,  1191212 used,   502584 buff/cache
KiB Swap:  9764860 total,  9764860 free,        0 used. 13020440+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
22275 root      20   0       0      0      0 R  99.0  0.0   7:01.01 md1_raid10

this cpu usage 99-100% is constant.

iostat shows drives have much more potential:
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.48    0.24    0.24    0.00   0.13  27.00
sdf               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.38    0.19    0.19    0.00   0.12  24.76
sds               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.26    0.13    0.13    0.00   0.13  25.76
sde               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.35    0.17    0.17    0.00   0.12  24.08
sdh               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.31    0.15    0.15    0.00   0.14  29.32
sdj               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.35    0.17    0.17    0.00   0.13  26.46
sdt               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.31    0.15    0.15    0.00   0.13  26.36
sdn               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.56    0.28    0.28    0.00   0.14  28.40
sdq               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.60    0.29    0.29    0.00   0.13  26.28
sdl               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.43    0.21    0.21    0.00   0.15  29.94
sdm               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.31    0.15    0.15    0.00   0.13  26.02
sdk               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.33    0.16    0.16    0.00   0.14  28.74
sdv               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.38    0.19    0.19    0.00   0.11  23.32
sdo               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.48    0.24    0.24    0.00   0.15  29.80
sdr               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.34    0.16    0.16    0.00   0.12  25.10
sdg               0.00     0.00 2038.60    0.00 130470.40     0.00
128.00     0.56    0.27    0.27    0.00   0.15  30.66
sdu               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.27    0.13    0.13    0.00   0.11  22.02
sdp               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.32    0.16    0.16    0.00   0.14  28.70
sdw               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.30    0.15    0.15    0.00   0.12  25.38
sdx               0.00     0.00 2038.40    0.00 130457.60     0.00
128.00     0.40    0.20    0.20    0.00   0.13  26.48

kernel:
[root@spare-a17484327407661 rovchinnikov]# cat /proc/version
Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org)
(gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
19 22:10:57 UTC 2015

So, the question is - why cpu usage is so high and I suppose is a limit here?

-- 
Best regards,
[COOLCOLD-RIPN]

Re: RAID10 performance with 20 drives

[Joe Landman]

On 05/31/2017 08:20 AM, CoolCold wrote:
> Hello!
> Got a new box, for image storage, playing around, created raid10 array
> with 20 1.8TB SATA drives, and found that we hit the cpu limit,
> details below.
>

[...]

> /proc/mdstat output:
> [root@spare-a17484327407661 rovchinnikov]# cat /proc/mdstat
> Personalities : [raid1] [raid10] [raid6] [raid5] [raid4]
> md1 : active raid10 sdx[19] sdw[18] sdv[17] sdu[16] sdt[15] sds[14]
> sdr[13] sdq[12] sdp[11] sdo[10] sdn[9] sdm[8] sdl[7] sdk[6] sdj[5]
> sdi[4] sdh[3] sdg[2] sdf[1] sde[0]
>        17580330880 blocks super 1.2 64K chunks 2 near-copies [20/20]
> [UUUUUUUUUUUUUUUUUUUU]
>        [=>...................]  resync =  6.4% (1133170368/17580330880)
> finish=192.6min speed=1423140K/sec

Note:  you are syncing the drives at 1.4 GB/s.

[...]

> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
> Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org)
> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
> 19 22:10:57 UTC 2015

And you have an ancient kernel.

>
> So, the question is - why cpu usage is so high and I suppose is a limit here?
>

Without seeing 'vmstat 1' or 'dstat' output, I think all that is 
possible is a guess.

If you have 20 drives, all connected over a single HBA going into an 
expander, it is possible that this is one of the rate limiting factors 
(and its around the same speed limit I've seen in other contexts for 
expander based systems).  Unfortunately, without more info, this is 
going to be pure speculation.

1.4GB/s / 20 drives -> 70 MB/s.  Without knowing what make/model drives 
you have there, it would be hard to speculate what fraction of the 
actual bandwidth you are getting.  Most modern (e.g. new) drives can do 
between 150-220 MB/s, so you could be anywhere from 33% to 50% of bandwidth.

Your HBA ... this matters tremendously to performance.  Not all HBAs are 
equivalent, and some are not very good at all.  Which make and model, 
how is it connected to the drives (direct or via expander), firmware 
revs, etc.   Since your kernel is ancient, chances are your HBA driver 
is as well, so ...

Closely related are how many context switches and interrupts per second 
you are seeing (hence the vmstat question).  Also quite related is how 
the irqs are being distributed for the HBA, or, as I've found many 
times, "if" they are being distributed.

Also, something I've found quite often has to do with how the PCIe 
devices actually negotiate their speeds.  This has bitten me more many 
times ... and I've written a tool to help answer that question:  
https://github.com/joelandman/pcilist

Then there are questions on the disk config, such as "is the write cache 
enabled", "is the read cache disabled".

     sdparm | grep WCE
     sdparm | grep RCD

And then the SD subsystem (read-ahead, ncq, etc.)

Basically you need to report far more information for anyone to give you 
anything more than pure speculation.


--
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman

Re: RAID10 performance with 20 drives

[Adam Goryachev]

On 31/5/17 22:20, CoolCold wrote:
> top stat:
> top - 12:09:03 up  4:55,  2 users,  load average: 3.33, 3.18, 2.88
> Tasks: 487 total,   4 running, 483 sleeping,   0 stopped,   0 zombie
> %Cpu(s):  0.0 us,  4.5 sy,  0.0 ni, 95.3 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
> KiB Mem : 13174918+total, 13005539+free,  1191212 used,   502584 buff/cache
> KiB Swap:  9764860 total,  9764860 free,        0 used. 13020440+avail Mem
>
>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 22275 root      20   0       0      0      0 R  99.0  0.0   7:01.01 md1_raid10
>
> this cpu usage 99-100% is constant.
>
Sorry, but doesn't that say 95.3% idle?

Do you have a multi core CPU? Is it multi threaded? What type of CPU is it?

When running top, press 1, it will then show you each individual core 
and the stats for it.

You might find that creating 10 RAID1 devices, and then using linear 
raid to join them together will perform better, from hearsay and memory, 
this will allow you to use a CPU for each RAID1, and another CPU for the 
linear, so if you had 11 CPU's (or more) then this should get you the 
best possible outcome (from a CPU point of view). In fact, if you have 
more than one CPU it would help.

Also, you might want to run a newer kernel, I think there was a lot of 
work done on the resync parts to optimise that. You might also prefer to 
focus on performance measurements *after* the resync has completed, 
since that would be your "normal" status. Though in addition, you should 
test performance with one lost disk, and while replacing that disk to 
ensure that you are still able to sustain the required load during those 
events.

Regards,
Adam

Re: RAID10 performance with 20 drives

[David Brown]

On 31/05/17 15:14, Adam Goryachev wrote:
> 
> 
> On 31/5/17 22:20, CoolCold wrote:
>> top stat:
>> top - 12:09:03 up  4:55,  2 users,  load average: 3.33, 3.18, 2.88
>> Tasks: 487 total,   4 running, 483 sleeping,   0 stopped,   0 zombie
>> %Cpu(s):  0.0 us,  4.5 sy,  0.0 ni, 95.3 id,  0.0 wa,  0.0 hi,  0.1
>> si,  0.0 st
>> KiB Mem : 13174918+total, 13005539+free,  1191212 used,   502584
>> buff/cache
>> KiB Swap:  9764860 total,  9764860 free,        0 used. 13020440+avail
>> Mem
>>
>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
>> COMMAND
>> 22275 root      20   0       0      0      0 R  99.0  0.0   7:01.01
>> md1_raid10
>>
>> this cpu usage 99-100% is constant.
>>
> Sorry, but doesn't that say 95.3% idle?
> 
> Do you have a multi core CPU? Is it multi threaded? What type of CPU is it?
> 
> When running top, press 1, it will then show you each individual core
> and the stats for it.
> 
> You might find that creating 10 RAID1 devices, and then using linear
> raid to join them together will perform better, from hearsay and memory,
> this will allow you to use a CPU for each RAID1, and another CPU for the
> linear, so if you had 11 CPU's (or more) then this should get you the
> best possible outcome (from a CPU point of view). In fact, if you have
> more than one CPU it would help.
> 

For some workloads, a linear concat (which takes no cpu work) of raid1
pairs will be much faster than a raid0 stripe of raid1 pairs.  Maybe I
am lacking the imagination, but I can't see a use-case where a 20 disk
raid10 setup is going to be the most efficient.  Raid 0 is good for
striped performance, but you would need /massive/ single file streamed
reads or writes to cover the loses of latencies on all these disks.  And
what would you do with all that data?  You would quickly saturate 10 Gb
network links - there is no point in trying to get data on or off disks
much faster than you can use it.

> Also, you might want to run a newer kernel, I think there was a lot of
> work done on the resync parts to optimise that. You might also prefer to
> focus on performance measurements *after* the resync has completed,
> since that would be your "normal" status. Though in addition, you should
> test performance with one lost disk, and while replacing that disk to
> ensure that you are still able to sustain the required load during those
> events.
> 
> Regards,
> Adam

Re: RAID10 performance with 20 drives

[CoolCold]

Hello!
thanks for reply, tried to answer questions below and with help of
gists of github as Gmail forcibly cuts the lines.
https://gist.github.com/CoolCold/676a6f9df0478c1c2d8ac8f3e6f9e22a

Added vmstat output as well.


On Wed, May 31, 2017 at 8:14 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>
>
> On 31/5/17 22:20, CoolCold wrote:
>>
>> top stat:
>> top - 12:09:03 up  4:55,  2 users,  load average: 3.33, 3.18, 2.88
>> Tasks: 487 total,   4 running, 483 sleeping,   0 stopped,   0 zombie
>> %Cpu(s):  0.0 us,  4.5 sy,  0.0 ni, 95.3 id,  0.0 wa,  0.0 hi,  0.1 si,
>> 0.0 st
>> KiB Mem : 13174918+total, 13005539+free,  1191212 used,   502584
>> buff/cache
>> KiB Swap:  9764860 total,  9764860 free,        0 used. 13020440+avail Mem
>>
>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
>> COMMAND
>> 22275 root      20   0       0      0      0 R  99.0  0.0   7:01.01
>> md1_raid10
>>
>> this cpu usage 99-100% is constant.
>>
> Sorry, but doesn't that say 95.3% idle?

No, it says 95.3 busy with right formatting, see
https://gist.github.com/CoolCold/676a6f9df0478c1c2d8ac8f3e6f9e22a#file-gistfile1-txt-L49
>
> Do you have a multi core CPU? Is it multi threaded? What type of CPU is it?
>
> When running top, press 1, it will then show you each individual core and
> the stats for it.
Top provided on the same gist, system has 40 cpu (2x10xhyperthreading) cpus

>
> You might find that creating 10 RAID1 devices, and then using linear raid to
> join them together will perform better, from hearsay and memory, this will
> allow you to use a CPU for each RAID1, and another CPU for the linear, so if
> you had 11 CPU's (or more) then this should get you the best possible
> outcome (from a CPU point of view). In fact, if you have more than one CPU
> it would help.
So, basically you are saying that one should avoid using raid10 for 20 drives?
>
> Also, you might want to run a newer kernel, I think there was a lot of work
> done on the resync parts to optimise that.
Can you please provide keywords/commits to look onto? I've did fast
search through archive, not much found, except may be intent bitmap,
which is not the case here.

 You might also prefer to focus on
> performance measurements *after* the resync has completed, since that would
> be your "normal" status. Though in addition, you should test performance
> with one lost disk, and while replacing that disk to ensure that you are
> still able to sustain the required load during those events.
That's would be my next step with FIO & lvmcache, but just limiting
speed to 1.4GB for resync makes me worry.

>
> Regards,
> Adam



-- 
Best regards,
[COOLCOLD-RIPN]

Re: RAID10 performance with 20 drives

[Joe Landman]

On 05/31/2017 09:57 AM, CoolCold wrote:
> Hello!
> thanks for reply, tried to answer questions below and with help of
> gists of github as Gmail forcibly cuts the lines.
> https://gist.github.com/CoolCold/676a6f9df0478c1c2d8ac8f3e6f9e22a
>
> Added vmstat output as well.

[...]

> Sorry, but doesn't that say 95.3% idle?
> No, it says 95.3 busy with right formatting, see
> https://gist.github.com/CoolCold/676a6f9df0478c1c2d8ac8f3e6f9e22a#file-gistfile1-txt-L49

Look at the vmstat output.

procs -----------memory---------- ---swap-- -----io---- -system-- 
------cpu-----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy 
id wa st
  3  0      0 130058176   2412 500660    0    0     0     0    3 17  0  
2 98  0  0
  1  0      0 130057352   2412 501012    0    0     0     0 28827 69339  
0  3 97  0  0

3rd from right is % idle.

This is 95-98% idle.  Is the rebuild done?

You are doing about 29k interrupts per second, and about 69k context 
switches.  Not heavily loaded unless this is a lower end CPU/desktop system.

This isn't a CPU bound problem if the rebuild is not done.  You'll need 
to look elsewhere.

How are the drives connected to the HBA, which HBA are you using, and 
what driver for it?  Also

     cat /proc/interrupts | grep $HBA

for the driver named $HBA (your driver),  could be helpful here ... 
though ... since you aren't throwing interrupts hard, chances are that 
this won't be as useful.

Likely ... lower end SATA HBA or motherboard based unit, or a weak expander.

--
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman

Re: RAID10 performance with 20 drives

[Roman Mamedov]

On Wed, 31 May 2017 19:20:10 +0700
CoolCold <coolthecold@gmail.com> wrote:

> Creation (disable write intent bitmap, with bitmap all is much worse):
> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
>
> kernel:
> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
> Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org)
> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
> 19 22:10:57 UTC 2015
> 
> So, the question is - why cpu usage is so high and I suppose is a limit here?

Definitely try a newer kernel, 4.4 at the very least; if no changes then 4.11.

Also I would suggest to try out larger chunk sizes, such as 512 and 1024 KB.

If you plan to use this long-term in production, also read up on the various
RAID10 data layouts and their benefits and downsides (man md, search for
"layout"; and search the Internet for benchmarks of all three).

-- 
With respect,
Roman

Re: RAID10 performance with 20 drives

[Roman Mamedov]

On Wed, 31 May 2017 10:07:50 -0400
Joe Landman <joe.landman@gmail.com> wrote:

> procs -----------memory---------- ---swap-- -----io---- -system-- 
> ------cpu-----
>   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy 
> id wa st
>   3  0      0 130058176   2412 500660    0    0     0     0    3 17  0  
> 2 98  0  0
>   1  0      0 130057352   2412 501012    0    0     0     0 28827 69339  
> 0  3 97  0  0
> 
> 3rd from right is % idle.
> 
> This is 95-98% idle.  Is the rebuild done?

It's a 40-core CPU with one core completely maxed out into 100% use with some
non-multithreaded load from md. Yes, 100% use of one core on a 40-core CPU
will show up as ~97% idle overall. Take a closer look at all the data
presented.

-- 
With respect,
Roman

Re: RAID10 performance with 20 drives

[Joe Landman]

On 05/31/2017 10:18 AM, Roman Mamedov wrote:
> On Wed, 31 May 2017 10:07:50 -0400
> Joe Landman <joe.landman@gmail.com> wrote:
>
>> procs -----------memory---------- ---swap-- -----io---- -system--
>> ------cpu-----
>>    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
>> id wa st
>>    3  0      0 130058176   2412 500660    0    0     0     0    3 17  0
>> 2 98  0  0
>>    1  0      0 130057352   2412 501012    0    0     0     0 28827 69339
>> 0  3 97  0  0
>>
>> 3rd from right is % idle.
>>
>> This is 95-98% idle.  Is the rebuild done?
> It's a 40-core CPU with one core completely maxed out into 100% use with some
> non-multithreaded load from md. Yes, 100% use of one core on a 40-core CPU
> will show up as ~97% idle overall. Take a closer look at all the data
> presented.
>

Hmmm... Methinks thou dost protesteth to much.

The system is effectively idle apart from 1 CPU.  20 physical, 40 with 
SMT.  1 fully loaded CPU in either context is between 2.5 and 5% 
loaded.  In no scenario that I've seen, would I (or anyone else) call 
this "loaded".

Moreover ... and this is the important part ... the interrupt rate and 
CSW rate was low.  Which means that the CPU is not struggling with 
overhead of handling the "calculations" which for RAID10 are ... well 
... trivial (effectively buffer copies).

This means a single CPU was "loaded", but in the context of bio 
submissions that were queued and being waited for.  Not because of 
calculations.  That is, if you understand how linux actually calculates 
load, you understand that IOs queued play a (significant) factor.  You 
would see queued reads in the vmstat line, as well as blocked reads.  
This is one of the reasons I asked for this output ... vmstat is 
suprisingly simple, and incredibly informative.  You can get similar 
information from dstat, or glances -t 1 if you have that installed.

So, the information we have is

1) interrupts are not wildly inappropriate
2) context switches are also reasonable
3) CPU (a single core) is not doing much calculation

Whats left.

1) driver
2) hardware (HBA and/or expander)
3) disk configuration (WCE,RCD)
4) ncq
5) read-ahead  (what does 'blockdev --getra /dev/sd*' report?)

In a Holmesian manner, we simple remove the impossible (based upon our 
observation), and what remains, no matter how improbably, is likely a 
factor.

The system has very low actual computational load, interrupt and context 
switch load.   So  ... its not ... loaded.  Then what comes next?  The 
list I gave.

Feel free to suggest other things.

--
Joe Landman
e: joe.landman@gmail.com
t: @hpcjoe
w: https://scalability.org
g: https://github.com/joelandman

Re: RAID10 performance with 20 drives

[CoolCold]

Hello!
Roman, i've updated the kernel to 4.11 and started "check" action,
results are basically the same, output on github
https://gist.github.com/CoolCold/663de7c006490d7fd0ac7cc98b7a6844
1 cpu is overloaded, not more than 1.3 - 1.4GB/sec

On Wed, May 31, 2017 at 9:14 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Wed, 31 May 2017 19:20:10 +0700
> CoolCold <coolthecold@gmail.com> wrote:
>
>> Creation (disable write intent bitmap, with bitmap all is much worse):
>> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
>> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
>> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
>> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
>>
>> kernel:
>> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
>> Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org)
>> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
>> 19 22:10:57 UTC 2015
>>
>> So, the question is - why cpu usage is so high and I suppose is a limit here?
>
> Definitely try a newer kernel, 4.4 at the very least; if no changes then 4.11.
>
> Also I would suggest to try out larger chunk sizes, such as 512 and 1024 KB.
>
> If you plan to use this long-term in production, also read up on the various
> RAID10 data layouts and their benefits and downsides (man md, search for
> "layout"; and search the Internet for benchmarks of all three).
>
> --
> With respect,
> Roman



-- 
Best regards,
[COOLCOLD-RIPN]

Re: RAID10 performance with 20 drives

[Pasi Kärkkäinen]

On Thu, Jun 01, 2017 at 12:59:01PM +0700, CoolCold wrote:
> Hello!
> Roman, i've updated the kernel to 4.11 and started "check" action,
> results are basically the same, output on github
> https://gist.github.com/CoolCold/663de7c006490d7fd0ac7cc98b7a6844
> 1 cpu is overloaded, not more than 1.3 - 1.4GB/sec
> 

You need to provide more details about the actual storage setup.

Like already said/asked for:

- Which HBA are you using?
- Which PCIe link speed are you using for the HBA?
- Which driver version for the HBA?
- Which HBA firmware version?

- How are the disks connected to the HBA ? Direct-connect, or via an Expander? 
- If you have an expander, what's the (SAS) link speed/count between the HBA(s) and the Expander? 


-- Pasi

> On Wed, May 31, 2017 at 9:14 PM, Roman Mamedov <rm@romanrm.net> wrote:
> > On Wed, 31 May 2017 19:20:10 +0700
> > CoolCold <coolthecold@gmail.com> wrote:
> >
> >> Creation (disable write intent bitmap, with bitmap all is much worse):
> >> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
> >> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
> >> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
> >> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
> >>
> >> kernel:
> >> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
> >> Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org)
> >> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
> >> 19 22:10:57 UTC 2015
> >>
> >> So, the question is - why cpu usage is so high and I suppose is a limit here?
> >
> > Definitely try a newer kernel, 4.4 at the very least; if no changes then 4.11.
> >
> > Also I would suggest to try out larger chunk sizes, such as 512 and 1024 KB.
> >
> > If you plan to use this long-term in production, also read up on the various
> > RAID10 data layouts and their benefits and downsides (man md, search for
> > "layout"; and search the Internet for benchmarks of all three).
> >
> > --
> > With respect,
> > Roman
> 
> 
> 
> -- 
> Best regards,
> [COOLCOLD-RIPN]
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 performance with 20 drives

[Roger Heflin]

Also supply an "iostat -x 1 5" since that will show each disks usage.

vmstat in my experience does not appear to show internal MD disk traffic.

On Thu, Jun 1, 2017 at 1:33 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Thu, Jun 01, 2017 at 12:59:01PM +0700, CoolCold wrote:
>> Hello!
>> Roman, i've updated the kernel to 4.11 and started "check" action,
>> results are basically the same, output on github
>> https://gist.github.com/CoolCold/663de7c006490d7fd0ac7cc98b7a6844
>> 1 cpu is overloaded, not more than 1.3 - 1.4GB/sec
>>
>
> You need to provide more details about the actual storage setup.
>
> Like already said/asked for:
>
> - Which HBA are you using?
> - Which PCIe link speed are you using for the HBA?
> - Which driver version for the HBA?
> - Which HBA firmware version?
>
> - How are the disks connected to the HBA ? Direct-connect, or via an Expander?
> - If you have an expander, what's the (SAS) link speed/count between the HBA(s) and the Expander?
>
>
> -- Pasi
>
>> On Wed, May 31, 2017 at 9:14 PM, Roman Mamedov <rm@romanrm.net> wrote:
>> > On Wed, 31 May 2017 19:20:10 +0700
>> > CoolCold <coolthecold@gmail.com> wrote:
>> >
>> >> Creation (disable write intent bitmap, with bitmap all is much worse):
>> >> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
>> >> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
>> >> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
>> >> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
>> >>
>> >> kernel:
>> >> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
>> >> Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org)
>> >> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
>> >> 19 22:10:57 UTC 2015
>> >>
>> >> So, the question is - why cpu usage is so high and I suppose is a limit here?
>> >
>> > Definitely try a newer kernel, 4.4 at the very least; if no changes then 4.11.
>> >
>> > Also I would suggest to try out larger chunk sizes, such as 512 and 1024 KB.
>> >
>> > If you plan to use this long-term in production, also read up on the various
>> > RAID10 data layouts and their benefits and downsides (man md, search for
>> > "layout"; and search the Internet for benchmarks of all three).
>> >
>> > --
>> > With respect,
>> > Roman
>>
>>
>>
>> --
>> Best regards,
>> [COOLCOLD-RIPN]
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID10 performance with 20 drives

[CoolCold]

Hello!
The data i was able to find with lsscsi:
Enclosure:
[0:0:24:0]   enclosu LSI      SAS3x40          0601  -
  state=running queue_depth=254 scsi_level=6 type=13 device_blocked=0 timeout=0

Server https://www.supermicro.com/products/system/2u/2028/ssg-2028r-e1cr24l.cfm

Drives are:
=== START OF INFORMATION SECTION ===
Vendor:               TOSHIBA
Product:              AL14SEB18EQ
Revision:             0101
User Capacity:        1,800,360,124,416 bytes [1.80 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Lowest aligned LBA:   0
Rotation Rate:        10500 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x500003975840f759
Serial number:        X6K0A0D5FZRC
Device type:          disk
Transport protocol:   SAS
Local Time is:        Mon Jun  5 09:51:56 2017 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

(don't think it matters though)

On Thu, Jun 1, 2017 at 6:31 PM, Nikhil Kshirsagar <nkshirsa@redhat.com> wrote:
> Did you check max_sectors_kb ? We found once, in some setups with raid, in
> particular with HP raid controllers, larger number of drives seemed to
> reduce the value of max_sectors_kb. Sorry if its already mentioned, I
> haven't read the entire thread in detail.
>
> On Thu, Jun 1, 2017 at 4:50 PM, Roger Heflin <rogerheflin@gmail.com> wrote:
>>
>> Also supply an "iostat -x 1 5" since that will show each disks usage.
>>
>> vmstat in my experience does not appear to show internal MD disk traffic.
>>
>> On Thu, Jun 1, 2017 at 1:33 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>> > On Thu, Jun 01, 2017 at 12:59:01PM +0700, CoolCold wrote:
>> >> Hello!
>> >> Roman, i've updated the kernel to 4.11 and started "check" action,
>> >> results are basically the same, output on github
>> >> https://gist.github.com/CoolCold/663de7c006490d7fd0ac7cc98b7a6844
>> >> 1 cpu is overloaded, not more than 1.3 - 1.4GB/sec
>> >>
>> >
>> > You need to provide more details about the actual storage setup.
>> >
>> > Like already said/asked for:
>> >
>> > - Which HBA are you using?
>> > - Which PCIe link speed are you using for the HBA?
>> > - Which driver version for the HBA?
>> > - Which HBA firmware version?
>> >
>> > - How are the disks connected to the HBA ? Direct-connect, or via an
>> > Expander?
>> > - If you have an expander, what's the (SAS) link speed/count between the
>> > HBA(s) and the Expander?
>> >
>> >
>> > -- Pasi
>> >
>> >> On Wed, May 31, 2017 at 9:14 PM, Roman Mamedov <rm@romanrm.net> wrote:
>> >> > On Wed, 31 May 2017 19:20:10 +0700
>> >> > CoolCold <coolthecold@gmail.com> wrote:
>> >> >
>> >> >> Creation (disable write intent bitmap, with bitmap all is much
>> >> >> worse):
>> >> >> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
>> >> >> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
>> >> >> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
>> >> >> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
>> >> >>
>> >> >> kernel:
>> >> >> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
>> >> >> Linux version 3.10.0-327.el7.x86_64
>> >> >> (builder@kbuilder.dev.centos.org)
>> >> >> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
>> >> >> 19 22:10:57 UTC 2015
>> >> >>
>> >> >> So, the question is - why cpu usage is so high and I suppose is a
>> >> >> limit here?
>> >> >
>> >> > Definitely try a newer kernel, 4.4 at the very least; if no changes
>> >> > then 4.11.
>> >> >
>> >> > Also I would suggest to try out larger chunk sizes, such as 512 and
>> >> > 1024 KB.
>> >> >
>> >> > If you plan to use this long-term in production, also read up on the
>> >> > various
>> >> > RAID10 data layouts and their benefits and downsides (man md, search
>> >> > for
>> >> > "layout"; and search the Internet for benchmarks of all three).
>> >> >
>> >> > --
>> >> > With respect,
>> >> > Roman
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >> [COOLCOLD-RIPN]
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-raid"
>> >> in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>



-- 
Best regards,
[COOLCOLD-RIPN]

Re: RAID10 performance with 20 drives

[Pasi Kärkkäinen]

On Mon, Jun 05, 2017 at 04:52:27PM +0700, CoolCold wrote:
> Hello!
> The data i was able to find with lsscsi:
> Enclosure:
> [0:0:24:0]   enclosu LSI      SAS3x40          0601  -
>   state=running queue_depth=254 scsi_level=6 type=13 device_blocked=0 timeout=0
> 
> Server https://www.supermicro.com/products/system/2u/2028/ssg-2028r-e1cr24l.cfm
>

OK. So you have a single expander SAS3 backplane. And a Broadcom/LSI SAS 3008 IT-mode HBA, which should be good. Well assuming you're running up-to-date firmware with it.

 
> Drives are:
> === START OF INFORMATION SECTION ===
> Vendor:               TOSHIBA
> Product:              AL14SEB18EQ
> Revision:             0101
> User Capacity:        1,800,360,124,416 bytes [1.80 TB]
> Logical block size:   512 bytes
> Physical block size:  4096 bytes
> Lowest aligned LBA:   0
> Rotation Rate:        10500 rpm
> Form Factor:          2.5 inches
> Logical Unit id:      0x500003975840f759
> Serial number:        X6K0A0D5FZRC
> Device type:          disk
> Transport protocol:   SAS
> Local Time is:        Mon Jun  5 09:51:56 2017 UTC
> SMART support is:     Available - device has SMART capability.
> SMART support is:     Enabled
> Temperature Warning:  Enabled
> 
> (don't think it matters though)
> 

Have you tried for example reading from all the disks simultaneously? Just to verify the link(s) between the HBA and the expander backplane is/are working properly, and you get good throughput with simultaneous reads from all disks? 


-- Pasi

> On Thu, Jun 1, 2017 at 6:31 PM, Nikhil Kshirsagar <nkshirsa@redhat.com> wrote:
> > Did you check max_sectors_kb ? We found once, in some setups with raid, in
> > particular with HP raid controllers, larger number of drives seemed to
> > reduce the value of max_sectors_kb. Sorry if its already mentioned, I
> > haven't read the entire thread in detail.
> >
> > On Thu, Jun 1, 2017 at 4:50 PM, Roger Heflin <rogerheflin@gmail.com> wrote:
> >>
> >> Also supply an "iostat -x 1 5" since that will show each disks usage.
> >>
> >> vmstat in my experience does not appear to show internal MD disk traffic.
> >>
> >> On Thu, Jun 1, 2017 at 1:33 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> >> > On Thu, Jun 01, 2017 at 12:59:01PM +0700, CoolCold wrote:
> >> >> Hello!
> >> >> Roman, i've updated the kernel to 4.11 and started "check" action,
> >> >> results are basically the same, output on github
> >> >> https://gist.github.com/CoolCold/663de7c006490d7fd0ac7cc98b7a6844
> >> >> 1 cpu is overloaded, not more than 1.3 - 1.4GB/sec
> >> >>
> >> >
> >> > You need to provide more details about the actual storage setup.
> >> >
> >> > Like already said/asked for:
> >> >
> >> > - Which HBA are you using?
> >> > - Which PCIe link speed are you using for the HBA?
> >> > - Which driver version for the HBA?
> >> > - Which HBA firmware version?
> >> >
> >> > - How are the disks connected to the HBA ? Direct-connect, or via an
> >> > Expander?
> >> > - If you have an expander, what's the (SAS) link speed/count between the
> >> > HBA(s) and the Expander?
> >> >
> >> >
> >> > -- Pasi
> >> >
> >> >> On Wed, May 31, 2017 at 9:14 PM, Roman Mamedov <rm@romanrm.net> wrote:
> >> >> > On Wed, 31 May 2017 19:20:10 +0700
> >> >> > CoolCold <coolthecold@gmail.com> wrote:
> >> >> >
> >> >> >> Creation (disable write intent bitmap, with bitmap all is much
> >> >> >> worse):
> >> >> >> mdadm --create -c 64 -b none -n 20 -l 10 /dev/md1 /dev/sde /dev/sdf
> >> >> >> /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
> >> >> >> /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
> >> >> >> /dev/sdu /dev/sdv /dev/sdw /dev/sdx
> >> >> >>
> >> >> >> kernel:
> >> >> >> [root@spare-a17484327407661 rovchinnikov]# cat /proc/version
> >> >> >> Linux version 3.10.0-327.el7.x86_64
> >> >> >> (builder@kbuilder.dev.centos.org)
> >> >> >> (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov
> >> >> >> 19 22:10:57 UTC 2015
> >> >> >>
> >> >> >> So, the question is - why cpu usage is so high and I suppose is a
> >> >> >> limit here?
> >> >> >
> >> >> > Definitely try a newer kernel, 4.4 at the very least; if no changes
> >> >> > then 4.11.
> >> >> >
> >> >> > Also I would suggest to try out larger chunk sizes, such as 512 and
> >> >> > 1024 KB.
> >> >> >
> >> >> > If you plan to use this long-term in production, also read up on the
> >> >> > various
> >> >> > RAID10 data layouts and their benefits and downsides (man md, search
> >> >> > for
> >> >> > "layout"; and search the Internet for benchmarks of all three).
> >> >> >
> >> >> > --
> >> >> > With respect,
> >> >> > Roman
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best regards,
> >> >> [COOLCOLD-RIPN]
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> >> >> in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> 
> 
> 
> -- 
> Best regards,
> [COOLCOLD-RIPN]