* Strange aggregate BW
@ 2014-01-20 21:23 grant
2014-01-20 21:28 ` Jeff Johnson
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: grant @ 2014-01-20 21:23 UTC (permalink / raw)
To: fio
Hello,
I�ve got a HWRAID with 10 NL-SAS HDDs.
All of them used in logical volume RAID6.
I�ve tried to benchmark created volume with FIO v2.1.4:
# cat seqwrite.fio
[write]
blocksize=512k
filename=/dev/sda
rw=write
direct=1
buffered=0
ioengine=libaio
iodepth=1
runtime=10m
numjobs=16
#./fio seqwrite.fio
FIO returns:
...
Run status group 0 (all jobs):
WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s..
2132 / 10 ~ 200 MB/s per HDD, is that possible???
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Strange aggregate BW 2014-01-20 21:23 Strange aggregate BW grant @ 2014-01-20 21:28 ` Jeff Johnson 2014-01-22 20:52 ` grant 2014-01-20 21:29 ` David Nellans 2014-01-20 21:47 ` John Williams 2 siblings, 1 reply; 6+ messages in thread From: Jeff Johnson @ 2014-01-20 21:28 UTC (permalink / raw) To: fio What make/model of NL-SAS disk drive? The 4TB/7200RPM drives are pushing sequential media rates of >150MB/sec. On 1/20/14, 1:23 PM, grant wrote: > Hello, > > I�ve got a HWRAID with 10 NL-SAS HDDs. > All of them used in logical volume RAID6. > I�ve tried to benchmark created volume with FIO v2.1.4: > > # cat seqwrite.fio > [write] > blocksize=512k > filename=/dev/sda > rw=write > direct=1 > buffered=0 > ioengine=libaio > iodepth=1 > runtime=10m > numjobs=16 > > #./fio seqwrite.fio > > FIO returns: > ... > Run status group 0 (all jobs): > WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s.. > > 2132 / 10 ~ 200 MB/s per HDD, is that possible??? -- ------------------------------ Jeff Johnson Co-Founder Aeon Computing jeff.johnson "at" aeoncomputing dot com www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 4170 Morena Boulevard, Suite D - San Diego, CA 92117 High-performance Computing / Lustre Filesystems / Scale-out Storage ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Strange aggregate BW 2014-01-20 21:28 ` Jeff Johnson @ 2014-01-22 20:52 ` grant 0 siblings, 0 replies; 6+ messages in thread From: grant @ 2014-01-22 20:52 UTC (permalink / raw) To: Jeff Johnson; +Cc: fio As I know 3TB/7200RPM drive used. I suppose sustained data transfer for it <= 150 MB/s. On 21 янв. 2014 г., at 1:28, Jeff Johnson <jeff.johnson@aeoncomputing.com> wrote: > What make/model of NL-SAS disk drive? The 4TB/7200RPM drives are pushing sequential media rates of >150MB/sec. > > On 1/20/14, 1:23 PM, grant wrote: >> Hello, >> >> I’ve got a HWRAID with 10 NL-SAS HDDs. >> All of them used in logical volume RAID6. >> I’ve tried to benchmark created volume with FIO v2.1.4: >> >> # cat seqwrite.fio >> [write] >> blocksize=512k >> filename=/dev/sda >> rw=write >> direct=1 >> buffered=0 >> ioengine=libaio >> iodepth=1 >> runtime=10m >> numjobs=16 >> >> #./fio seqwrite.fio >> >> FIO returns: >> ... >> Run status group 0 (all jobs): >> WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s.. >> >> 2132 / 10 ~ 200 MB/s per HDD, is that possible??? > > -- > ------------------------------ > Jeff Johnson > Co-Founder > Aeon Computing > > jeff.johnson "at" aeoncomputing dot com > www.aeoncomputing.com > t: 858-412-3810 x1001 f: 858-412-3845 > > 4170 Morena Boulevard, Suite D - San Diego, CA 92117 > > High-performance Computing / Lustre Filesystems / Scale-out Storage > > -- > To unsubscribe from this list: send the line "unsubscribe fio" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Strange aggregate BW 2014-01-20 21:23 Strange aggregate BW grant 2014-01-20 21:28 ` Jeff Johnson @ 2014-01-20 21:29 ` David Nellans 2014-01-20 21:47 ` John Williams 2 siblings, 0 replies; 6+ messages in thread From: David Nellans @ 2014-01-20 21:29 UTC (permalink / raw) To: grant, fio On 01/20/2014 03:23 PM, grant wrote: > Hello, > > I�ve got a HWRAID with 10 NL-SAS HDDs. > All of them used in logical volume RAID6. > I�ve tried to benchmark created volume with FIO v2.1.4: > > # cat seqwrite.fio > [write] > blocksize=512k > filename=/dev/sda > rw=write > direct=1 > buffered=0 > ioengine=libaio > iodepth=1 > runtime=10m > numjobs=16 > > #./fio seqwrite.fio > > FIO returns: > ... > Run status group 0 (all jobs): > WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s.. Between some hardware raid buffering, large block transfers, and 100% sequential write you'll certainly be near optimal for sequential write I/O seen at each disk. 200MB/s/disk is higher than most folks would probably ballpark, but its not off by an order of magnitude or anything. I assume ~150MB/s sequential performance for most "good" consumer 7200RPM drives. Higher RPM alone could push you to ~200MB/s. Try turning it into random I/O and if it goes to shit, then the results are probably as-expected ;) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Strange aggregate BW 2014-01-20 21:23 Strange aggregate BW grant 2014-01-20 21:28 ` Jeff Johnson 2014-01-20 21:29 ` David Nellans @ 2014-01-20 21:47 ` John Williams 2014-01-22 21:11 ` grant 2 siblings, 1 reply; 6+ messages in thread From: John Williams @ 2014-01-20 21:47 UTC (permalink / raw) To: grant; +Cc: fio On Mon, Jan 20, 2014 at 1:23 PM, grant <626542@gmail.com> wrote: > I�ve got a HWRAID with 10 NL-SAS HDDs. If it is HW RAID6 with 10 total drives, then the aggregrate write throughput to the HDDs is about 25% ( 10 / 8 - 1 ) higher than the write throughput that fio measures as going into the HW RAID device. So you'd be looking at about 266MB/s per drive. Which seems unlikely. I wonder if your HW RAID has a large cache. Did you let the fio job run for the full 10 minutes? Because I would expect the cache to fill well before 10 minutes, unless the cache is huge. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Strange aggregate BW 2014-01-20 21:47 ` John Williams @ 2014-01-22 21:11 ` grant 0 siblings, 0 replies; 6+ messages in thread From: grant @ 2014-01-22 21:11 UTC (permalink / raw) To: John Williams; +Cc: fio HW RAID has got large cache, 12 GB per controller. 60 total drives in array, but in logical volume (controller A) used only 10. I’ll tryed increase runtime of benchmark to bypass cache effect, and run batch of tests with cfgs: # cat searead.fio [read] blocksize=${BSIZE} filename=/dev/sda rw=read direct=1 bufferred=0 ioengine=libaio iodepth=48 runtime=1h numjobs=16 group_reporting thread disable_lat=1 disable_slat=1 disable_bw_measurement=1 flat_percentiles=0 and # cat seqwrite.fio [write] … rw=write … BSIZE = 4K, 8K, 32K, 64K, 256K, 512K, 1M, 8M, 10M, 64M I’ve got results: bs=4K, bw_read=1764, iops_read=451670, bw_write=1055, iops=write=270084 bs=8K, bw_read=1909, iops_read=244354, bw_write=1516, iops=write=194136 bs=32K, bw_read=2003, iops_read=64120, bw_write=1093, iops=write=34983 bs=64K, bw_read=383, iops_read=6131, bw_write=2102, iops=write=33639 bs=256K, bw_read=281, iops_read=1125, bw_write=1134, iops=write=4539 bs=512K, bw_read=295, iops_read=590, bw_write=1721, iops=write=3443 bs=1M, bw_read=287, iops_read=287, bw_write=1825, iops=write=1825 bs=8M, bw_read=330, iops_read=41, bw_write=1954, iops=write=244 bs=10M, bw_read=337, iops_read=33, bw_write=1614, iops=write=161 bs=64M, bw_read=317, iops_read=4, bw_write=1657, iops=write=25 bw’s in MB/s bw_read in bs range 4K-32K under cache effect? bw_write in all bss cache effect? Pls help me to describe results. On 21 янв. 2014 г., at 1:47, John Williams <jwilliams4200@gmail.com> wrote: > On Mon, Jan 20, 2014 at 1:23 PM, grant <626542@gmail.com> wrote: > >> I’ve got a HWRAID with 10 NL-SAS HDDs. > > If it is HW RAID6 with 10 total drives, then the aggregrate write > throughput to the HDDs is about 25% ( 10 / 8 - 1 ) higher than the > write throughput that fio measures as going into the HW RAID device. > > So you'd be looking at about 266MB/s per drive. Which seems unlikely. > I wonder if your HW RAID has a large cache. Did you let the fio job > run for the full 10 minutes? Because I would expect the cache to fill > well before 10 minutes, unless the cache is huge. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-01-22 21:11 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-01-20 21:23 Strange aggregate BW grant 2014-01-20 21:28 ` Jeff Johnson 2014-01-22 20:52 ` grant 2014-01-20 21:29 ` David Nellans 2014-01-20 21:47 ` John Williams 2014-01-22 21:11 ` grant
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.