OCZ Z-drive p84 read performance

OCZ Z-drive p84 read performance

[Ryousei Takano]

Dear Eric Moore and all,

I am installing ubuntu 9.10 (kernel 2.6.31-14-server) on my PC, which
has OCZ Z-Drive p84.
This drive consists of LSI logic's SAS1068E controller and 4 MLC SSDs.
I am using mptsas driver instead of the official binary driver
(megasar).  I ran simple read & write
benchmark using dd(1). The result shows the throughput of sequential
read achieves 260 MB/s,
which is the half of sequential write performance (520 MB/s):

block size  write (MB/s)  read (MB/s)
1024	201	241
4096	472	262
16384	486	267
65536	502	266
262144	472	259
524288	525	251
1048576	528	245
2097152	517	253
4194304	433	247
8388608	406	246
16777216	405	243
33554432	405	242
67108864	405	242

I wonder why read performance is worse than write performance.
What is the bottleneck of read performance?

For more information, please see my page:
http://code.google.com/p/pspacer/wiki/ZdrivePerformance

Any comments and suggestions will be welcome.

Best regards,
Ryousei

RE: OCZ Z-drive p84 read performance

[Desai, Kashyap]

Ryousei,

Replied inlined.

Thanks,
Kashyap


> -----Original Message-----
> From: Ryousei Takano [mailto:ryousei@gmail.com]
> Sent: Monday, November 30, 2009 3:08 PM
> To: Moore, Eric
> Cc: linux-scsi@vger.kernel.org; DL-MPT Fusion Linux; Support, Software;
> takano-ryousei@aist.go.jp
> Subject: OCZ Z-drive p84 read performance
> 
> Dear Eric Moore and all,
> 
> I am installing ubuntu 9.10 (kernel 2.6.31-14-server) on my PC, which
> has OCZ Z-Drive p84.
> This drive consists of LSI logic's SAS1068E controller and 4 MLC SSDs.
> I am using mptsas driver instead of the official binary driver
> (megasar). 
Is this megaraid driver or mpt fusion? 
Driver version?

 I ran simple read & write
> benchmark using dd(1). The result shows the throughput of sequential
> read achieves 260 MB/s,
> which is the half of sequential write performance (520 MB/s):
Setup information? 
> 
> block size  write (MB/s)  read (MB/s)
> 1024	201	241
> 4096	472	262
> 16384	486	267
> 65536	502	266
> 262144	472	259
> 524288	525	251
> 1048576	528	245
> 2097152	517	253
> 4194304	433	247
> 8388608	406	246
> 16777216	405	243
> 33554432	405	242
> 67108864	405	242
> 
> I wonder why read performance is worse than write performance.
> What is the bottleneck of read performance?
> 
> For more information, please see my page:
> http://code.google.com/p/pspacer/wiki/ZdrivePerformance
> 
> Any comments and suggestions will be welcome.
> 
> Best regards,
> Ryousei

Re: OCZ Z-drive p84 read performance

[Ryousei Takano]

Hi Kashyap,

Thanks for your reply, and I am sorry for lack of information.

I am using mpt fusion driver version 3.04.10.

What setup information do you need?

My PC consists of two Intel Quad-core Xeon E5430 2.66~GHz, Intel 5100 chipset,
and 4 GB memory (DDR2-667). OCZ Z-Drive p84 is plugged into a
PCI-Express x8 lane.

Here is my test script:

#!/bin/bash

BS=$((1024))
COUNT=$((1024*1024*10))

for i in 1 4 16 64 256 512 1024 2048 4096 8192 16384 32768 65536; do
        bs=$((BS * i))
        count=$((COUNT / i))

        echo bs=$bs count=$count
        sudo mount /dev/sdb1 /media/test
        dd if=/dev/zero of=/media/test/foo bs=$bs count=$count
        sudo umount /media/test
        sleep 1
        sudo mount /dev/sdb1 /media/test
        dd if=/media/test/foo of=/dev/null bs=$bs count=$count
        rm /media/test/foo
        sudo umount /media/test
done

This script assumes the Z-drive is /dev/sdb1 and it is mounted on /media/test.

Best regards,
Ryousei

On Mon, Nov 30, 2009 at 10:25 PM, Desai, Kashyap <Kashyap.Desai@lsi.com> wrote:
> Ryousei,
>
> Replied inlined.
>
> Thanks,
> Kashyap
>
>
>> -----Original Message-----
>> From: Ryousei Takano [mailto:ryousei@gmail.com]
>> Sent: Monday, November 30, 2009 3:08 PM
>> To: Moore, Eric
>> Cc: linux-scsi@vger.kernel.org; DL-MPT Fusion Linux; Support, Software;
>> takano-ryousei@aist.go.jp
>> Subject: OCZ Z-drive p84 read performance
>>
>> Dear Eric Moore and all,
>>
>> I am installing ubuntu 9.10 (kernel 2.6.31-14-server) on my PC, which
>> has OCZ Z-Drive p84.
>> This drive consists of LSI logic's SAS1068E controller and 4 MLC SSDs.
>> I am using mptsas driver instead of the official binary driver
>> (megasar).
> Is this megaraid driver or mpt fusion?
> Driver version?
>
>  I ran simple read & write
>> benchmark using dd(1). The result shows the throughput of sequential
>> read achieves 260 MB/s,
>> which is the half of sequential write performance (520 MB/s):
> Setup information?
>>
>> block size  write (MB/s)  read (MB/s)
>> 1024  201     241
>> 4096  472     262
>> 16384 486     267
>> 65536 502     266
>> 262144        472     259
>> 524288        525     251
>> 1048576       528     245
>> 2097152       517     253
>> 4194304       433     247
>> 8388608       406     246
>> 16777216      405     243
>> 33554432      405     242
>> 67108864      405     242
>>
>> I wonder why read performance is worse than write performance.
>> What is the bottleneck of read performance?
>>
>> For more information, please see my page:
>> http://code.google.com/p/pspacer/wiki/ZdrivePerformance
>>
>> Any comments and suggestions will be welcome.
>>
>> Best regards,
>> Ryousei
>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: OCZ Z-drive p84 read performance

[Desai, Kashyap]

Ryousei,

See my comments inlined.

Thanks,
Kashyap

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-
> owner@vger.kernel.org] On Behalf Of Ryousei Takano
> Sent: Monday, November 30, 2009 7:38 PM
> To: Desai, Kashyap
> Cc: Moore, Eric; linux-scsi@vger.kernel.org; DL-MPT Fusion Linux; Support,
> Software; takano-ryousei@aist.go.jp; Yang, Bo
> Subject: Re: OCZ Z-drive p84 read performance
> 
> Hi Kashyap,
> 
> Thanks for your reply, and I am sorry for lack of information.
> 
> I am using mpt fusion driver version 3.04.10.
> 
> What setup information do you need?
> 
> My PC consists of two Intel Quad-core Xeon E5430 2.66~GHz, Intel 5100
> chipset,
> and 4 GB memory (DDR2-667). OCZ Z-Drive p84 is plugged into a
> PCI-Express x8 lane.
> 
> Here is my test script:
> 
> #!/bin/bash
> 
> BS=$((1024))
> COUNT=$((1024*1024*10))
> 
> for i in 1 4 16 64 256 512 1024 2048 4096 8192 16384 32768 65536; do
>         bs=$((BS * i))
>         count=$((COUNT / i))
> 
>         echo bs=$bs count=$count
>         sudo mount /dev/sdb1 /media/test
>         dd if=/dev/zero of=/media/test/foo bs=$bs count=$count
>         sudo umount /media/test
>         sleep 1
>         sudo mount /dev/sdb1 /media/test
>         dd if=/media/test/foo of=/dev/null bs=$bs count=$count
Replace /media/test/foo with /dev/sdb1, you will see raw read
>         rm /media/test/foo
>         sudo umount /media/test
> done
> 
This test is not purely RAW read/write test. In you test File system performance is also included. While read operation, (sequential read) File system buffering will give huge advantage to data transfer.


> This script assumes the Z-drive is /dev/sdb1 and it is mounted on
> /media/test.
> 
> Best regards,
> Ryousei
> 
> On Mon, Nov 30, 2009 at 10:25 PM, Desai, Kashyap <Kashyap.Desai@lsi.com>
> wrote:
> > Ryousei,
> >
> > Replied inlined.
> >
> > Thanks,
> > Kashyap
> >
> >
> >> -----Original Message-----
> >> From: Ryousei Takano [mailto:ryousei@gmail.com]
> >> Sent: Monday, November 30, 2009 3:08 PM
> >> To: Moore, Eric
> >> Cc: linux-scsi@vger.kernel.org; DL-MPT Fusion Linux; Support, Software;
> >> takano-ryousei@aist.go.jp
> >> Subject: OCZ Z-drive p84 read performance
> >>
> >> Dear Eric Moore and all,
> >>
> >> I am installing ubuntu 9.10 (kernel 2.6.31-14-server) on my PC, which
> >> has OCZ Z-Drive p84.
> >> This drive consists of LSI logic's SAS1068E controller and 4 MLC SSDs.
> >> I am using mptsas driver instead of the official binary driver
> >> (megasar).
> > Is this megaraid driver or mpt fusion?
> > Driver version?
> >
> >  I ran simple read & write
> >> benchmark using dd(1). The result shows the throughput of sequential
> >> read achieves 260 MB/s,
> >> which is the half of sequential write performance (520 MB/s):
> > Setup information?
> >>
> >> block size  write (MB/s)  read (MB/s)
> >> 1024  201     241
> >> 4096  472     262
> >> 16384 486     267
> >> 65536 502     266
> >> 262144        472     259
> >> 524288        525     251
> >> 1048576       528     245
> >> 2097152       517     253
> >> 4194304       433     247
> >> 8388608       406     246
> >> 16777216      405     243
> >> 33554432      405     242
> >> 67108864      405     242
> >>
> >> I wonder why read performance is worse than write performance.
> >> What is the bottleneck of read performance?
> >>
> >> For more information, please see my page:
> >> http://code.google.com/p/pspacer/wiki/ZdrivePerformance
> >>
> >> Any comments and suggestions will be welcome.
> >>
> >> Best regards,
> >> Ryousei
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OCZ Z-drive p84 read performance

[Matthew Wilcox]

On Mon, Nov 30, 2009 at 08:07:58PM +0530, Desai, Kashyap wrote:
> > for i in 1 4 16 64 256 512 1024 2048 4096 8192 16384 32768 65536; do
> >         bs=$((BS * i))
> >         count=$((COUNT / i))
> > 
> >         echo bs=$bs count=$count
> >         sudo mount /dev/sdb1 /media/test
> >         dd if=/dev/zero of=/media/test/foo bs=$bs count=$count
> >         sudo umount /media/test
> >         sleep 1
> >         sudo mount /dev/sdb1 /media/test
> >         dd if=/media/test/foo of=/dev/null bs=$bs count=$count
> Replace /media/test/foo with /dev/sdb1, you will see raw read
> >         rm /media/test/foo
> >         sudo umount /media/test
> > done
> > 
> This test is not purely RAW read/write test. In you test File system performance is also included. While read operation, (sequential read) File system buffering will give huge advantage to data transfer.

Both filesystem and block access will use the page cache.  You should
use iflag=direct (or oflag=direct as appropriate) in order to bypass
the page cache.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: OCZ Z-drive p84 read performance

[Ryousei Takano]

Hi Matthew and Kashyap,

Thanks for your comments!

On Tue, Dec 1, 2009 at 1:11 AM, Matthew Wilcox <matthew@wil.cx> wrote:
> On Mon, Nov 30, 2009 at 08:07:58PM +0530, Desai, Kashyap wrote:
>> > for i in 1 4 16 64 256 512 1024 2048 4096 8192 16384 32768 65536; do
>> >         bs=$((BS * i))
>> >         count=$((COUNT / i))
>> >
>> >         echo bs=$bs count=$count
>> >         sudo mount /dev/sdb1 /media/test
>> >         dd if=/dev/zero of=/media/test/foo bs=$bs count=$count
>> >         sudo umount /media/test
>> >         sleep 1
>> >         sudo mount /dev/sdb1 /media/test
>> >         dd if=/media/test/foo of=/dev/null bs=$bs count=$count
>> Replace /media/test/foo with /dev/sdb1, you will see raw read
>> >         rm /media/test/foo
>> >         sudo umount /media/test
>> > done
>> >
>> This test is not purely RAW read/write test. In you test File system performance is also included. While read operation, (sequential read) File system buffering will give huge advantage to data transfer.
>
> Both filesystem and block access will use the page cache.  You should
> use iflag=direct (or oflag=direct as appropriate) in order to bypass
> the page cache.
>
> --
> Matthew Wilcox                          Intel Open Source Technology Centre
> "Bill, look, we understand that you're interested in selling us this
> operating system, but compare it to ours.  We can't possibly take such
> a retrograde step."
>

The bottleneck is in the file system.
I retried dd with the direct I/O option.  The performance improves
with large block sizes.
The cross point is about 256 KB.

bs  write (MB/s)  read (MB/s)
1024	9.5	9.7	
4096	34.4	28.3	
16384	95.8	47.0	
65536	186	121	
262144	382	307	
524288	417	366	
1048576	449	380	
2097152	497	467	
4194304	511	532	
8388608	498	560
16777216	523	545	
33554432	555	541	
67108864	554	543	

My page is also updated.

Best regards,
Ryousei
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OCZ Z-drive p84 read performance

[Ryousei Takano]

Hi,

On Tue, Dec 1, 2009 at 10:30 AM, Ryousei Takano <ryousei@gmail.com> wrote:
> Hi Matthew and Kashyap,
>
> Thanks for your comments!
>
> On Tue, Dec 1, 2009 at 1:11 AM, Matthew Wilcox <matthew@wil.cx> wrote:
>> On Mon, Nov 30, 2009 at 08:07:58PM +0530, Desai, Kashyap wrote:
>>> > for i in 1 4 16 64 256 512 1024 2048 4096 8192 16384 32768 65536; do
>>> >         bs=$((BS * i))
>>> >         count=$((COUNT / i))
>>> >
>>> >         echo bs=$bs count=$count
>>> >         sudo mount /dev/sdb1 /media/test
>>> >         dd if=/dev/zero of=/media/test/foo bs=$bs count=$count
>>> >         sudo umount /media/test
>>> >         sleep 1
>>> >         sudo mount /dev/sdb1 /media/test
>>> >         dd if=/media/test/foo of=/dev/null bs=$bs count=$count
>>> Replace /media/test/foo with /dev/sdb1, you will see raw read
>>> >         rm /media/test/foo
>>> >         sudo umount /media/test
>>> > done
>>> >
>>> This test is not purely RAW read/write test. In you test File system performance is also included. While read operation, (sequential read) File system buffering will give huge advantage to data transfer.
>>
>> Both filesystem and block access will use the page cache.  You should
>> use iflag=direct (or oflag=direct as appropriate) in order to bypass
>> the page cache.
>>
>> --
>> Matthew Wilcox                          Intel Open Source Technology Centre
>> "Bill, look, we understand that you're interested in selling us this
>> operating system, but compare it to ours.  We can't possibly take such
>> a retrograde step."
>>
>
> The bottleneck is in the file system.
> I retried dd with the direct I/O option.  The performance improves
> with large block sizes.
> The cross point is about 256 KB.
>
> bs  write (MB/s)  read (MB/s)
> 1024    9.5     9.7
> 4096    34.4    28.3
> 16384   95.8    47.0
> 65536   186     121
> 262144  382     307
> 524288  417     366
> 1048576 449     380
> 2097152 497     467
> 4194304 511     532
> 8388608 498     560
> 16777216        523     545
> 33554432        555     541
> 67108864        554     543
>
> My page is also updated.
>
> Best regards,
> Ryousei
>

Here is the result on btrfs without direct I/O:

bs  write (MB/s)  read (MB/s)
1024	176	605	
4096	435	614	
16384	641	620	
65536	664	624	
262144	676	618	
524288	677	620	
1048576	674	625	
2097152	666	615	
4194304	652	600	
8388608	625	599	
16777216	633	598	
33554432	629	601	
67108864	624	603

I got good performance. However, the continued usage (read&write)
causes the write performance
decrease independently of the block size.
Anyway, my first question is resolved.

Thanks,
Ryousei
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html