More XFS resource starvation?

["J.H." <warthog9@kernel.org>]

So apparently I'm having fun tripping over all kinds of bugs lately.
I've seen this a couple of times now on the box in question.  Usually
happens after a few days, or after particularly heavy rsync traffic on
the box.

http://pastebin.osuosl.org/36014

Christoph seemed to think it's a memory exhaustion problem, so I've
included the /proc/meminfo and as you can see there's plenty of memory
around on the system.

Loads have, expectedly, climbed currently around 1250.05 but growing slowly.

Quick overview of the underlying storage:

xfs -> md (raid 0) -+--> P812 hardware raid6 (cciss driver)
                    |
                    +--> P812 hardware raid6 (cciss driver)

This is running on an HP DL380 G7.

I saw this both on an older 2.6.30.10-105.2.23.fc11.x86_64, and
currently on 2.6.34.7-61.fc13.x86_64 (both being Fedora stock kernels)

I have not seen this on a very similar DL380 G6, with the same storage
setup and it is currently running the 2.6.30 kernel from above.

Christoph suggest increasing the nr_request values for each of the
underlying devices, but this didn't seem to change anything
significantly on the system.

Anyone have any ideas on what's going on?

- John 'Warthog9' Hawley