bad command responsiveness Proliant DL 585

[David Osojnik <david@dworf.com>]

Hello,

We have a problem with four HP Proliant DL 585 servers with 4 AMD
Opteron processors and 16Gb of memory and 3x 300Gb U320 SCSI disks and
with all the latest firmware. we noticed bad command responsiveness in
an production environment and poor performance (web server and mysql)
The problem is good reproducible if creating a large file 30Gb in size with:

time dd if=/dev/zero of=test.file bs=3072 count=10240000

on the root partition no matter which one reiserfs, ext3

what happens is I open three more console windows and do random commands
like: "ls, w, route -n, ifconfig" but the commands freezes for random
time (this time is from 1 minute to 15minutes!! per command execution
time) when the command starts working (after 5minutes) i try it again
and the command freezes again for a random time... the strange thing is
that if one command freezes all other commands freeze too when one
starts to work others work too. (if running top it stops refreshing for
the lockup period)

we tried raid0, raid1, raid5 its the same and we even tried some other
raid controllers other then default onboard Smart Array 5i we tried
Smart Array 6402 and MegaRAID SCSI 320-2X. The problem is still there
the only difference is writing speed but the lockups are still there!

and we tried this on four different DL 585 servers with all the same
problem!

load when creating 32Gb file was around 10 (starting creation of the
file with load 0.05) and the system did not run any services except the
default (after a fresh minimum install of more then one distribution) we
tried it too with all services shutdown and as much modules unloaded as
we could remove.

we tried lots of kernels and cciss modules but all have the problem
including 2.6.16.20

only the kernel 2.4 was working better there were still the lockups but
were less often and commands executed faster around 50%.

why am i writing to kernel list because it looks like a kernel problem
to me since the kernel 2.4 looks much better.

hope someone can help because we are getting really desperate

thanks