Strange aggregate BW

Strange aggregate BW

[grant]

Hello,

I�ve got a HWRAID with 10 NL-SAS HDDs.
All of them used in logical volume RAID6.
I�ve tried to benchmark created volume with FIO v2.1.4:

# cat seqwrite.fio
[write]
blocksize=512k
filename=/dev/sda
rw=write
direct=1
buffered=0
ioengine=libaio
iodepth=1
runtime=10m
numjobs=16

#./fio seqwrite.fio

FIO returns:
...
Run status group 0 (all jobs):
  WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s..

2132 / 10 ~ 200 MB/s per HDD, is that possible???

Re: Strange aggregate BW

[Jeff Johnson]

What make/model of NL-SAS disk drive? The 4TB/7200RPM drives are pushing 
sequential media rates of >150MB/sec.

On 1/20/14, 1:23 PM, grant wrote:
> Hello,
>
> I�ve got a HWRAID with 10 NL-SAS HDDs.
> All of them used in logical volume RAID6.
> I�ve tried to benchmark created volume with FIO v2.1.4:
>
> # cat seqwrite.fio
> [write]
> blocksize=512k
> filename=/dev/sda
> rw=write
> direct=1
> buffered=0
> ioengine=libaio
> iodepth=1
> runtime=10m
> numjobs=16
>
> #./fio seqwrite.fio
>
> FIO returns:
> ...
> Run status group 0 (all jobs):
>    WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s..
>
> 2132 / 10 ~ 200 MB/s per HDD, is that possible???

-- 
------------------------------
Jeff Johnson
Co-Founder
Aeon Computing

jeff.johnson "at" aeoncomputing dot com
www.aeoncomputing.com
t: 858-412-3810 x1001   f: 858-412-3845

4170 Morena Boulevard, Suite D - San Diego, CA 92117

High-performance Computing / Lustre Filesystems / Scale-out Storage

Re: Strange aggregate BW

[David Nellans]

On 01/20/2014 03:23 PM, grant wrote:
> Hello,
>
> I�ve got a HWRAID with 10 NL-SAS HDDs.
> All of them used in logical volume RAID6.
> I�ve tried to benchmark created volume with FIO v2.1.4:
>
> # cat seqwrite.fio
> [write]
> blocksize=512k
> filename=/dev/sda
> rw=write
> direct=1
> buffered=0
> ioengine=libaio
> iodepth=1
> runtime=10m
> numjobs=16
>
> #./fio seqwrite.fio
>
> FIO returns:
> ...
> Run status group 0 (all jobs):
>    WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s..


Between some hardware raid buffering, large block transfers, and 100% 
sequential write you'll certainly be near optimal for sequential write 
I/O seen at each disk.  200MB/s/disk is higher than most folks would 
probably ballpark, but its not off by an order of magnitude or anything. 
I assume ~150MB/s sequential performance for most "good" consumer 
7200RPM drives.  Higher RPM alone could push you to ~200MB/s.
Try turning it into random I/O and if it goes to shit, then the results
are probably as-expected ;)

Re: Strange aggregate BW

[John Williams]

On Mon, Jan 20, 2014 at 1:23 PM, grant <626542@gmail.com> wrote:

> I�ve got a HWRAID with 10 NL-SAS HDDs.

If it is HW RAID6 with 10 total drives, then the aggregrate write
throughput to the HDDs is about 25% ( 10 / 8 - 1 ) higher than the
write throughput that fio measures as going into the HW RAID device.

So you'd be looking at about 266MB/s per drive. Which seems unlikely.
I wonder if your HW RAID has a large cache. Did you let the fio job
run for the full 10 minutes? Because I would expect the cache to fill
well before 10 minutes, unless the cache is huge.

Re: Strange aggregate BW

[grant]

As I know 3TB/7200RPM drive used. I suppose sustained data transfer for it <= 150 MB/s.

On 21 янв. 2014 г., at 1:28, Jeff Johnson <jeff.johnson@aeoncomputing.com> wrote:

> What make/model of NL-SAS disk drive? The 4TB/7200RPM drives are pushing sequential media rates of >150MB/sec.
> 
> On 1/20/14, 1:23 PM, grant wrote:
>> Hello,
>> 
>> I’ve got a HWRAID with 10 NL-SAS HDDs.
>> All of them used in logical volume RAID6.
>> I’ve tried to benchmark created volume with FIO v2.1.4:
>> 
>> # cat seqwrite.fio
>> [write]
>> blocksize=512k
>> filename=/dev/sda
>> rw=write
>> direct=1
>> buffered=0
>> ioengine=libaio
>> iodepth=1
>> runtime=10m
>> numjobs=16
>> 
>> #./fio seqwrite.fio
>> 
>> FIO returns:
>> ...
>> Run status group 0 (all jobs):
>>   WRITE: io=127958MB, aggrb=2132.5MB/s, minb=56530KB/s, maxb=229141KB/s..
>> 
>> 2132 / 10 ~ 200 MB/s per HDD, is that possible???
> 
> -- 
> ------------------------------
> Jeff Johnson
> Co-Founder
> Aeon Computing
> 
> jeff.johnson "at" aeoncomputing dot com
> www.aeoncomputing.com
> t: 858-412-3810 x1001   f: 858-412-3845
> 
> 4170 Morena Boulevard, Suite D - San Diego, CA 92117
> 
> High-performance Computing / Lustre Filesystems / Scale-out Storage
> 
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Strange aggregate BW

[grant]

HW RAID has got large cache, 12 GB per controller.
60 total drives in array, but in logical volume (controller A) used only 10.
I’ll tryed increase runtime of benchmark to bypass cache effect, and run batch of tests with cfgs:

# cat searead.fio
[read]
blocksize=${BSIZE}
filename=/dev/sda
rw=read
direct=1
bufferred=0
ioengine=libaio
iodepth=48
runtime=1h
numjobs=16
group_reporting
thread
disable_lat=1
disable_slat=1
disable_bw_measurement=1
flat_percentiles=0

and

# cat seqwrite.fio
[write]
…
rw=write
…

BSIZE = 4K, 8K, 32K, 64K, 256K, 512K, 1M, 8M, 10M, 64M

I’ve got results:

bs=4K, bw_read=1764, iops_read=451670, bw_write=1055, iops=write=270084
bs=8K, bw_read=1909, iops_read=244354, bw_write=1516, iops=write=194136
bs=32K, bw_read=2003, iops_read=64120, bw_write=1093, iops=write=34983
bs=64K, bw_read=383, iops_read=6131, bw_write=2102, iops=write=33639
bs=256K, bw_read=281, iops_read=1125, bw_write=1134, iops=write=4539
bs=512K, bw_read=295, iops_read=590, bw_write=1721, iops=write=3443
bs=1M, bw_read=287, iops_read=287, bw_write=1825, iops=write=1825
bs=8M, bw_read=330, iops_read=41, bw_write=1954, iops=write=244
bs=10M, bw_read=337, iops_read=33, bw_write=1614, iops=write=161
bs=64M, bw_read=317, iops_read=4, bw_write=1657, iops=write=25

bw’s in MB/s

bw_read in bs range 4K-32K under cache effect?
bw_write in all bss cache effect?

Pls help me to describe results.

On 21 янв. 2014 г., at 1:47, John Williams <jwilliams4200@gmail.com> wrote:

> On Mon, Jan 20, 2014 at 1:23 PM, grant <626542@gmail.com> wrote:
> 
>> I’ve got a HWRAID with 10 NL-SAS HDDs.
> 
> If it is HW RAID6 with 10 total drives, then the aggregrate write
> throughput to the HDDs is about 25% ( 10 / 8 - 1 ) higher than the
> write throughput that fio measures as going into the HW RAID device.
> 
> So you'd be looking at about 266MB/s per drive. Which seems unlikely.
> I wonder if your HW RAID has a large cache. Did you let the fio job
> run for the full 10 minutes? Because I would expect the cache to fill
> well before 10 minutes, unless the cache is huge.