Random Seek on Array as slow as on single disk

Random Seek on Array as slow as on single disk

[A. Liemen]

Hi,

I'm having random seek performance problems on our Raid 5 and Raid 10 
arrays. The random seek performance of the whole 14 disk array is just 
as slow as a single disk. This has been tested with 4 identical boxes, 
Single Disk, Raid1, Raid5, Raid10 configurations. Raid5 / Raid10 has 
been tested with 14 disks arrays.

System:

Debian 2.6.15-1-amd64-k8-smp dual opteron 275
Controller: Areca ARC 1160 PCI-X 1GB Cache
Disks: 14x WD400YR - 400GB SATA
Raid: Raid5 / Raid 10 - 64k / 128k stripe
Filesystems: XFS, Reiser, ext3
Ram: 16gb
Schedulers: deadline, anti, cfq


Benchmark:

./bonnie++ -uroot -s40000M

1-4 threads, overall random seeks stay the same (~360 seeks / sec)

Sample bonnie Raid10 output can be found under:

http://alexander.liemen.net/bonnie_sample.txt


No matter if I benchmark a single disk, Raid1 2 disks, Raid5/10 14 disks 
, single thread / multiple threads,
random seek performance always stays at ~ 340-360 seeks per second.

I have also observed overall bad random seek performance on an identical 
production box so it's not just bonnie producing strange results.


Can anyone help me on this?


Thank you a lot
Alex

Re: Random Seek on Array as slow as on single disk

[Jeff Breidenbach]

> Controller: Areca ARC 1160 PCI-X 1GB Cache

Those numbers are for Arica hardware raid or linux software raid?
--Jeff

Re: Random Seek on Array as slow as on single disk

[A. Liemen]

Hardware Raid.

http://www.areca.com.tw/products/html/pcix-sata.htm

Alex

Jeff Breidenbach schrieb:
>> Controller: Areca ARC 1160 PCI-X 1GB Cache
> 
> Those numbers are for Arica hardware raid or linux software raid?
> --Jeff
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Random Seek on Array as slow as on single disk

[Jeff Breidenbach]

The linux-raid mailing list is predominantly covers linux software
RAID, so you may get a lot of software raid centric comments.
Like this one.

Have you tried setting the Areca to JBOD then using mdadm to
set up a software? That might help isolate where the problem is.
Just pick any configuration and try it. As a sanity comparison, I'm
seeing good seek performance with a 5 disk RAID1 (e.g. 5X
redundancy) with resiserfs on Linux software RAID. That's on a
production system, so I can't run bonnie++, but I''m definitely seeing
significantly better seek performance than you are reporting.

-Jeff

Re: Random Seek on Array as slow as on single disk

[Mattias Wadenstein]

On Sun, 16 Jul 2006, A. Liemen wrote:

> Hardware Raid.
>
> http://www.areca.com.tw/products/html/pcix-sata.htm

If you want to see even worse performance with bonnie, try running several 
parallel in sync, somewhere around 6-15 simultaneous read sessions[*] 
should give you rather horrible performance numbers.

That said, synthetic benchmarks isn't everything. Check if it is an issue 
with realistic load too. In some cases there can be quite a difference.

/Mattias Wadenstein

[*] Exact numbers may vary, for me it was as low as 6 on a 12-disk raid6 
where the throughput dropped from ~300M/s to ~30M/s, but smaller raidsets 
had a breakoff point up close to 8-10 reading procseses.

> Jeff Breidenbach schrieb:
>>> Controller: Areca ARC 1160 PCI-X 1GB Cache
>> 
>> Those numbers are for Arica hardware raid or linux software raid?
>> --Jeff
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Random Seek on Array as slow as on single disk

[Bill Davidsen]

A. Liemen wrote:

> Hardware Raid.
>
> http://www.areca.com.tw/products/html/pcix-sata.htm


You should ask the vendor, this isn't a software RAID issue, and the 
usual path to improving bad hardware is in upgrading. You may be able to 
get better firmware if you're lucky.

>
> Alex
>
> Jeff Breidenbach schrieb:
>
>>> Controller: Areca ARC 1160 PCI-X 1GB Cache
>>
>>
>> Those numbers are for Arica hardware raid or linux software raid?
>> --Jeff
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979