From: Stan Hoeppner <stan@hardwarefreak.com>
To: lilofile <lilofile@aliyun.com>, linux-raid@vger.kernel.org
Subject: Re: md raid5 random performace 6x SSD RAID5
Date: Sun, 01 Dec 2013 20:37:33 -0600 [thread overview]
Message-ID: <529BF26D.8020107@hardwarefreak.com> (raw)
In-Reply-To: <efed564c-afcf-4871-b6eb-65c0814709d8@aliyun.com>
Again, please post the result output from the streaming read/write fio
runs, not random. After I see those we can discuss your random performance.
On 12/1/2013 10:33 AM, lilofile wrote:
> six ssd disk ,raid5 cpu:Intel(R) Xeon(R) CPU X5650 @ 2.67GHz memory:32G
> sTEC SSD disk: single disk iops=35973
> root@host0:/sys/block/md127/md# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md127 : active raid5 sdg[6] sdl[4] sdk[3] sdj[2] sdi[1] sdh[0]
> 3906404480 blocks super 1.2 level 5, 128k chunk, algorithm 2 [6/6] [UUUUUU]
>
> unused devices: <none>
>
>
> ramdom write iops is as follows:
> stripe_cache_size==2048 iops= 59617
> stripe_cache_size==4096 iops=61623
> stripe_cache_size==8192 iops= 59877
>
>
> why the random write iops is so low,while single disk write IOPS reach to 3.6W?
>
>
> fio parameter is as follows:
>
> the test result shows: stripe_cache_size==2048
> root@sc0:~# fio -filename=/dev/md/md0 -iodepth 16 -thread -rw=randwrite -ioengine=libaio -bs=4k -size=30G -numjobs=16 -runtime=1000 -group_reporting -name=mytest
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> ...
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> fio 1.59
> Starting 16 threads
> Jobs: 7 (f=7): [www__w____w_w__w] [47.3% done] [0K/186.6M /s] [0 /46.7K iops] [eta 18m:35s]s]
> mytest: (groupid=0, jobs=16): err= 0: pid=5208
> write: io=232889MB, bw=238470KB/s, iops=59617 , runt=1000036msec
> slat (usec): min=1 , max=65595 , avg=264.91, stdev=3322.66
> clat (usec): min=4 , max=111435 , avg=3992.16, stdev=12317.14
> lat (usec): min=40 , max=111439 , avg=4257.19, stdev=12679.23
> bw (KB/s) : min= 0, max=350792, per=6.31%, avg=15039.33, stdev=6492.82
> cpu : usr=1.45%, sys=31.90%, ctx=7766821, majf=136, minf=3585068
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued r/w/d: total=0/59619701/0, short=0/0/0
> lat (usec): 10=0.01%, 50=19.28%, 100=70.12%, 250=1.14%, 500=0.01%
> lat (usec): 750=0.01%, 1000=0.01%
> lat (msec): 2=0.01%, 4=0.02%, 10=0.05%, 20=0.09%, 50=9.14%
> lat (msec): 100=0.13%, 250=0.01%
>
> Run status group 0 (all jobs):
> WRITE: io=232889MB, aggrb=238470KB/s, minb=244193KB/s, maxb=244193KB/s, mint=1000036msec, maxt=1000036msec
> root@host0:~#
>
>
>
> the test result shows: stripe_cache_size==4096
> root@host0:~# fio -filename=/dev/md/md0 -iodepth 16 -thread -rw=randwrite -ioengine=libaio -bs=4k -size=30G -numjobs=16 -runtime=1000 -group_reporting -name=mytest
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> ...
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> fio 1.59
> Starting 16 threads
> Jobs: 7 (f=7): [ww_ww_ww_______w] [48.3% done] [0K/224.8M /s] [0 /56.2K iops] [eta 17m:58s]s]
> mytest: (groupid=0, jobs=16): err= 0: pid=4851
> write: io=240727MB, bw=246495KB/s, iops=61623 , runt=1000037msec
> slat (usec): min=1 , max=837996 , avg=257.06, stdev=3387.21
> clat (usec): min=4 , max=838074 , avg=3873.92, stdev=12967.09
> lat (usec): min=41 , max=838077 , avg=4131.10, stdev=13376.14
> bw (KB/s) : min= 0, max=449685, per=6.28%, avg=15490.34, stdev=5760.87
> cpu : usr=6.16%, sys=18.83%, ctx=15818324, majf=181, minf=3591162
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued r/w/d: total=0/61626113/0, short=0/0/0
> lat (usec): 10=0.01%, 50=20.21%, 100=70.72%, 250=0.21%, 500=0.01%
> lat (usec): 750=0.01%, 1000=0.01%
> lat (msec): 2=0.01%, 4=0.02%, 10=0.06%, 20=0.10%, 50=7.87%
> lat (msec): 100=0.75%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
>
> Run status group 0 (all jobs):
> WRITE: io=240727MB, aggrb=246495KB/s, minb=252411KB/s, maxb=252411KB/s, mint=1000037msec, maxt=1000037msec
> root@host0:~#
>
> the test result shows: stripe_cache_size==8192
> root@host0:~# fio -filename=/dev/md/md0 -iodepth 16 -thread -rw=randwrite -ioengine=libaio -bs=4k -size=30G -numjobs=16 -runtime=1000 -group_reporting -name=mytest
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> ...
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> fio 1.59
> Starting 16 threads
> Jobs: 6 (f=6): [__w_w__ww__w___w] [47.6% done] [0K/178.6M /s] [0 /44.7K iops] [eta 18m:24s]s]
> mytest: (groupid=0, jobs=16): err= 0: pid=5047
> write: io=233924MB, bw=239511KB/s, iops=59877 , runt=1000114msec
> slat (usec): min=1 , max=235194 , avg=263.80, stdev=4435.78
> clat (usec): min=2 , max=391878 , avg=3974.23, stdev=16930.35
> lat (usec): min=4 , max=391885 , avg=4238.15, stdev=17467.30
> bw (KB/s) : min= 0, max=303248, per=6.34%, avg=15180.71, stdev=5877.14
> cpu : usr=4.93%, sys=27.37%, ctx=6335719, majf=103, minf=3591206
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued r/w/d: total=0/59884454/0, short=0/0/0
> lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=36.26%, 100=55.83%
> lat (usec): 250=0.78%, 500=0.01%, 750=0.01%, 1000=0.01%
> lat (msec): 2=0.01%, 4=0.02%, 10=0.05%, 20=0.09%, 50=5.38%
> lat (msec): 100=0.75%, 250=0.80%, 500=0.01%
>
> Run status group 0 (all jobs):
> WRITE: io=233924MB, aggrb=239510KB/s, minb=245258KB/s, maxb=245258KB/s, mint=1000114msec, maxt=1000114msec
> root@host0:~#
>
> // single ssd disk
> root@host0:~# fio -filename=/dev/sdb -iodepth 16 -thread -rw=randwrite -ioengine=libaio -bs=4k -size=30G -numjobs=16 -runtime=1000 -group_reporting -name=mytest
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> ...
> mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
> fio 1.59
> Starting 16 threads
> Jobs: 1 (f=1): [___w____________] [28.5% done] [0K/0K /s] [0 /0 iops] [eta 43m:08s] s]
> mytest: (groupid=0, jobs=16): err= 0: pid=5308
> write: io=140528MB, bw=143894KB/s, iops=35973 , runt=1000046msec
> slat (usec): min=1 , max=159802 , avg=443.06, stdev=4487.35
> clat (usec): min=4 , max=159916 , avg=6665.26, stdev=16174.17
> lat (usec): min=40 , max=159922 , avg=7108.46, stdev=16611.67
> bw (KB/s) : min= 3, max=892696, per=6.26%, avg=9008.49, stdev=8706.58
> cpu : usr=2.61%, sys=13.09%, ctx=7436836, majf=58, minf=782937
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued r/w/d: total=0/35975210/0, short=0/0/0
> lat (usec): 10=0.01%, 50=16.00%, 100=67.45%, 250=1.81%, 500=0.05%
> lat (usec): 750=0.01%, 1000=0.01%
> lat (msec): 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=13.33%
> lat (msec): 100=1.28%, 250=0.04%
>
> Run status group 0 (all jobs):
> WRITE: io=140528MB, aggrb=143894KB/s, minb=147347KB/s, maxb=147347KB/s, mint=1000046msec, maxt=1000046msec
>
> Disk stats (read/write):
> sdb: ios=261/27342034, merge=0/5212609, ticks=48/143752312, in_queue=143721596, util=100.00%
> root@host0:~#
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2013-12-02 2:37 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-11-22 11:13 ARC-1120 and MD very sloooow Jimmy Thrasibule
2013-11-22 11:17 ` Mikael Abrahamsson
2013-11-22 20:17 ` Stan Hoeppner
2013-11-22 20:17 ` Stan Hoeppner
2013-11-25 8:56 ` Jimmy Thrasibule
2013-11-26 0:45 ` Stan Hoeppner
2013-11-26 0:45 ` Stan Hoeppner
2013-11-26 2:52 ` Dave Chinner
2013-11-26 2:52 ` Dave Chinner
2013-11-26 3:58 ` Stan Hoeppner
2013-11-26 3:58 ` Stan Hoeppner
2013-11-26 6:14 ` Dave Chinner
2013-11-26 8:03 ` Stan Hoeppner
2013-11-26 8:03 ` Stan Hoeppner
2013-11-28 15:59 ` Jimmy Thrasibule
2013-11-28 15:59 ` Jimmy Thrasibule
2013-11-28 19:59 ` Stan Hoeppner
2013-11-27 13:48 ` md raid5 performace 6x SSD RAID5 lilofile
2013-11-27 13:51 ` 答复：md " lilofile
2013-11-28 4:41 ` Stan Hoeppner
2013-11-28 4:46 ` Roman Mamedov
2013-11-28 6:24 ` Stan Hoeppner
2013-11-28 10:02 ` 答复：答复：md " lilofile
2013-11-29 2:38 ` Stan Hoeppner
2013-11-29 6:23 ` Stan Hoeppner
2013-11-30 14:12 ` 答复：答复：答复：md raid5 random " lilofile
2013-12-01 14:14 ` Stan Hoeppner
2013-12-01 16:33 ` md " lilofile
2013-12-02 2:37 ` Stan Hoeppner [this message]
2013-11-28 11:54 ` 答复：答复：md raid5 " lilofile
2013-12-02 3:48 ` md " lilofile
2013-12-02 5:51 ` Stan Hoeppner
2014-09-23 3:34 ` raid sync speed lilofile
2014-09-23 5:11 ` behind_writes lilofile
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=529BF26D.8020107@hardwarefreak.com \
--to=stan@hardwarefreak.com \
--cc=lilofile@aliyun.com \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.