3ware disk latency?

3ware disk latency?

[Jan Kasprzak]

	Hi all,

I have upgraded my machine from 3ware 7508 with 8x 250GB ATA drives
to 3ware 9550SX-8LP with 8x 500GB SATA-II drives, and I have found that
while the overall throughput of the disks is higher than before,
the latency - measured as the number of messages per time unit Qmail can
deliver - is far worse than before[1]. I have tried to use deadline
and cfq schedulers, but it made no visible speedup.

	I have found the following two years old mail by Jens Axboe:
http://www.uwsg.iu.edu/hypermail/linux/kernel/0409.0/1330.html
- the problem at that time was that the device had its internal
command queue deeper than the nr_requests of the block device layer,
so the I/O scheduler couldn't do anything.  I was surprised
that the situation is still the same: I have

/sys/block/sd[a-h]/queue/nr_requests == 128, and
/sys/devices/pci0000:00/0000:00:0b.0/0000:01:03.0/host0/target*/*/queue_depth == 254

I have verified that even on older 3ware drivers (7508) the situation
is the same. On the newer hardware it is probably more visible, because
the controller actually has bigger on-board cache (128MB vs. 32MB, I think).

	I have tried to lower the /sys/devices/.../queue_depth to 4
and disable the NCQ, and the latency got a bit better. But it is still
nowhere near to where it was on the older HW.

	Does anybody experience the similar latency problems with
3ware 95xx controllers? Thanks,

-Yenya

[1] the old configuration peaked at ~2k-4k messages per 5 minutes,
	with 1k-2k messages/5min being pretty normal, while the new one
	has maximum throughput of 1k messages per 5 minutes, but the normal
	speed is much lower - some low hundreds messages per 5 minutes.
	The new system runs about the same load as the previous one,
	and the layout of disks is also the same (just the newer drives
	are bigger, of course).


-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
> I will never go to meetings again because I think  face to face meetings <
> are the biggest waste of time you can ever have.        --Linus Torvalds <

Re: 3ware disk latency?

[dean gaudet]

On Mon, 10 Jul 2006, Jan Kasprzak wrote:

> 	Does anybody experience the similar latency problems with
> 3ware 95xx controllers? Thanks,

i think the 95xx is really bad at sharing its cache fairly amongst 
multiple devices... i have a 9550sx-8lp with 3 exported units: a raid1, a 
raid10 and a jbod.  i was zeroing the jbod with dd and it absolutely 
destroyed the latency of the other two units.

to make things sane i had to add oflag=direct to dd to avoid dirtying all 
of memory... and i had to tell the 3ware to disable write caching on the 
jbod device.

the result is a fairly slow throughput on the dd (20MB/s) ... but at least 
the latency of the raid1 and raid10 aren't being affected any more.

i suspect that telling the 9550sx to not use nvram limits the amount of 
its tiny 128MiB of RAM which can be consumed by that unit.

having to use oflag=direct indicates linux is at least partially at fault 
here... maybe there's some tunable i need to learn about... the system has 
8GiB of ram, maybe the default dirty ratio of 40% is too high.  but i 
suspect the problem is more that with dd dirtying memory immediately it 
causes the system to be stuck at the 40% dirty watermark which causes 
excessive write traffic even to the raid1/raid10 units because they too 
are contributing to the 40% dirty (temporary files and whatnot would hit 
disk when they wouldn't in normal operation).

i think since i'm using 3ware hw raid on this setup with only two units 
normally active it isn't as painful as what you're seeing with what looks 
like 8 JBOD units.

yeah i'm a bit disappointed... i neglected to benchmark this sort of 
concurrency thing when i was experimenting with the 9550sx before turning 
it live.  i'm curious if areca controllers handle this better.

-dean

Re: 3ware disk latency?

[Alan Cox]

Ar Mer, 2006-07-26 am 09:52 -0700, ysgrifennodd dean gaudet:
> On Mon, 10 Jul 2006, Jan Kasprzak wrote:
> 
> > 	Does anybody experience the similar latency problems with
> > 3ware 95xx controllers? Thanks,
> 
> i think the 95xx is really bad at sharing its cache fairly amongst 
> multiple devices... i have a 9550sx-8lp with 3 exported units: a raid1, a 
> raid10 and a jbod.  i was zeroing the jbod with dd and it absolutely 
> destroyed the latency of the other two units.

Even on the older 3ware I found it neccessary to bump up the queue sizes
for the 3ware. There's a thread from about 6 months ago (maybe a bit
more) with Jeff Merkey included where it made a huge difference to his
performance playing with the parameters. Might be worth a scan through
the archive.

Alan

Re: 3ware disk latency?

[dean gaudet]

On Wed, 26 Jul 2006, Alan Cox wrote:

> Ar Mer, 2006-07-26 am 09:52 -0700, ysgrifennodd dean gaudet:
> > On Mon, 10 Jul 2006, Jan Kasprzak wrote:
> > 
> > > 	Does anybody experience the similar latency problems with
> > > 3ware 95xx controllers? Thanks,
> > 
> > i think the 95xx is really bad at sharing its cache fairly amongst 
> > multiple devices... i have a 9550sx-8lp with 3 exported units: a raid1, a 
> > raid10 and a jbod.  i was zeroing the jbod with dd and it absolutely 
> > destroyed the latency of the other two units.
> 
> Even on the older 3ware I found it neccessary to bump up the queue sizes
> for the 3ware. There's a thread from about 6 months ago (maybe a bit
> more) with Jeff Merkey included where it made a huge difference to his
> performance playing with the parameters. Might be worth a scan through
> the archive.

yeah i've done some experiments with a multithreaded random O_DIRECT i/o 
generator where i showed the 9550sx peaking at around 192 threads... where 
"peak" means the latency was still reasonable (measured by iostat).  
obviously when going past 128 threads i found it necessary to increase 
nr_requests to see an improvement in io/s.  there was some magic number 
past 256 where i found the 3ware must have choked its own intnernal queue 
and latency became awful (istr it being 384ish but it probably depends on 
the io workload).

unfortunately when i did the experiment i neglected to perform 
simultaneous tests on more than one 3ware unit on the same controller.  i 
got great results from a raid1 or from a raid10 (even a raid5)... but 
never i only tested one unit at a time.

my production config has a raid1 and a raid10 unit on the same controller 
-- with 2 spare ports.  i popped a disk into one of the spare ports, told 
3ware it was JBOD and then proceeded to "dd if=/dev/zero of=/dev/sdX 
bs=1M"... and wham my system load avg shot up to 80+ (from ~4), all in D 
state.

i tried a bunch of things... lowering and raising nr_requests on all the 
devices, changing between cfq, deadline, etc.  the only solution i found 
to be effective in retaining acceptable latencies on the raid1/raid10 was 
to use "oflag=direct" for dd, and tell the 3ware to not use nvram for the 
jbod disk.

my suspicion is the 3ware lacks any sort of "fairness" in its sharing of 
buffer space between multiple units on the same controller.  and by 
disabling the write caching it limits the amount of controller memory that 
the busy disk can consume.

-dean

Re: 3ware disk latency?

[adam radford]

On 7/26/06, dean gaudet <dean@arctic.org> wrote:
>
> unfortunately when i did the experiment i neglected to perform
> simultaneous tests on more than one 3ware unit on the same controller.  i
> got great results from a raid1 or from a raid10 (even a raid5)... but
> never i only tested one unit at a time.
>

Did you try setting /sys/class/scsi_host/hostX/cmd_per_lun to 256 / num_units ?

-Adam

Re: 3ware disk latency?

[Justin Piszcz]

On Wed, 26 Jul 2006, adam radford wrote:

> On 7/26/06, dean gaudet <dean@arctic.org> wrote:
>> 
>> unfortunately when i did the experiment i neglected to perform
>> simultaneous tests on more than one 3ware unit on the same controller.  i
>> got great results from a raid1 or from a raid10 (even a raid5)... but
>> never i only tested one unit at a time.
>> 
>
> Did you try setting /sys/class/scsi_host/hostX/cmd_per_lun to 256 / num_units 
> ?
>
> -Adam
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Speaking of this does anyone having any tuning guides for running 10 disk 
raid 5's on this controller? And as far as ext3 goes, what are the best 
optimizations?

Re: 3ware disk latency?

[Jeff V. Merkey]

adam radford wrote:

> On 7/26/06, dean gaudet <dean@arctic.org> wrote:
>
>>
>> unfortunately when i did the experiment i neglected to perform
>> simultaneous tests on more than one 3ware unit on the same 
>> controller.  i
>> got great results from a raid1 or from a raid10 (even a raid5)... but
>> never i only tested one unit at a time.
>>
>
> Did you try setting /sys/class/scsi_host/hostX/cmd_per_lun to 256 / 
> num_units ?
>
> -Adam
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
Adam,

Isn't it set to this by default or is the default 128?

Jeff

Re: 3ware disk latency?

[adam radford]

On 7/26/06, Jeff V. Merkey <jmerkey@wolfmountaingroup.com> wrote:
>
> Isn't it set to this by default or is the default 128?
>

Default is 254.

-Adam

Re: 3ware disk latency?

[dean gaudet]

On Wed, 26 Jul 2006, adam radford wrote:

> On 7/26/06, dean gaudet <dean@arctic.org> wrote:
> > 
> > unfortunately when i did the experiment i neglected to perform
> > simultaneous tests on more than one 3ware unit on the same controller.  i
> > got great results from a raid1 or from a raid10 (even a raid5)... but
> > never i only tested one unit at a time.
> > 
> 
> Did you try setting /sys/class/scsi_host/hostX/cmd_per_lun to 256 / num_units
> ?

hmm doesn't look like i can do this in 2.6.16.27:

# ls -l /sys/class/scsi_host/host0/cmd_per_lun
-r--r--r-- 1 root root 0 Jul 26 19:58 /sys/class/scsi_host/host0/cmd_per_lun
# echo 85 >!$
echo 85 >/sys/class/scsi_host/host0/cmd_per_lun
zsh: permission denied: /sys/class/scsi_host/host0/cmd_per_lun

it'll probably be sometime until i update this box past 2.6.16.x ... but 
i'll keep it in mind and retry the experiment... thanks.

-dean

Re: 3ware disk latency?

[Jan Kasprzak]

dean gaudet wrote:
: On Wed, 26 Jul 2006, adam radford wrote:
: > 
: > Did you try setting /sys/class/scsi_host/hostX/cmd_per_lun to 256 / num_units
: > ?
: 
: hmm doesn't look like i can do this in 2.6.16.27:
: 
: # ls -l /sys/class/scsi_host/host0/cmd_per_lun
: -r--r--r-- 1 root root 0 Jul 26 19:58 /sys/class/scsi_host/host0/cmd_per_lun
: # echo 85 >!$
: echo 85 >/sys/class/scsi_host/host0/cmd_per_lun
: zsh: permission denied: /sys/class/scsi_host/host0/cmd_per_lun
: 
: it'll probably be sometime until i update this box past 2.6.16.x ... but 
: i'll keep it in mind and retry the experiment... thanks.

	2.6.17.7 also does not allow cmd_per_lun to be written.

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
> I will never go to meetings again because I think  face to face meetings <
> are the biggest waste of time you can ever have.        --Linus Torvalds <

Re: 3ware disk latency?

[Jan Kasprzak]

dean gaudet wrote:
: my suspicion is the 3ware lacks any sort of "fairness" in its sharing of 
: buffer space between multiple units on the same controller.  and by 
: disabling the write caching it limits the amount of controller memory that 
: the busy disk can consume.

	Hmm, do you have a battery backup unit for 9550sx? I don't,
and the 3ware BIOS does not even allow me to enable write caching without it.
So I don't think the write caching on the controller side is related
to my problem.

	I have been able to improve the latency by upgrading the firmware
to the newest release (wow, they even have a firmware updating utility
for Linux!).

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
> I will never go to meetings again because I think  face to face meetings <
> are the biggest waste of time you can ever have.        --Linus Torvalds <

Re: 3ware disk latency?

[David Lang]

On Thu, 27 Jul 2006, Jan Kasprzak wrote:

> dean gaudet wrote:
> : my suspicion is the 3ware lacks any sort of "fairness" in its sharing of
> : buffer space between multiple units on the same controller.  and by
> : disabling the write caching it limits the amount of controller memory that
> : the busy disk can consume.
>
> 	Hmm, do you have a battery backup unit for 9550sx? I don't,
> and the 3ware BIOS does not even allow me to enable write caching without it.
> So I don't think the write caching on the controller side is related
> to my problem.

interesting, on the 9500 series cards I have it pops up a 'are you really sure, 
this is dangerous' warning, but it did allow me to turn on write caching (I've 
since received the batteries, but I did test it before that)

David Lang

> 	I have been able to improve the latency by upgrading the firmware
> to the newest release (wow, they even have a firmware updating utility
> for Linux!).
>
> -Yenya
>
>

Re: 3ware disk latency?

[Dieter Stüken]

Jan Kasprzak wrote: 
> dean gaudet wrote:
> 
> : my suspicion is the 3ware lacks any sort of "fairness" in its sharing of
> : buffer space between multiple units on the same controller.  and by
> : disabling the write caching it limits the amount of controller memory that
> : the busy disk can consume.
> 
>         Hmm, do you have a battery backup unit for 9550sx? I don't,
> and the 3ware BIOS does not even allow me to enable write caching without it.
> So I don't think the write caching on the controller side is related
> to my problem.
> 
>         I have been able to improve the latency by upgrading the firmware
> to the newest release (wow, they even have a firmware updating utility
> for Linux!).
> 

I just discovered this thread, as I have similar problems
with my 9550sx. I will perform a firmware upgrade tomorrow,
hope that helps. I have a BBU installed, thus my controller
should be able to handle concurrent requests for read and write.

But I also want to understand, what causes this problems. I have
several VMWare servers running on a RAID5 setup. Several times
a day I observe periods of excessive IO-waits and the guest systems
become unusable for a while. I may use "top", but all I see is
the high iowait and 4 idling processors. I not even found which
processes are actually waiting. Is there a way to trace,
which IO requests are pending, their block numbers and the
process ids that issued the requests? I found there are different
IO scheduler available, which have access to all these informations.
Is it possible to plug in some debug module to trace the IO
traffic similar to "strace" for processes?

Dieter.