Re: More XFS resource starvation?

[Simon Kirby <sim@hostway.ca>]

On Mon, Nov 15, 2010 at 10:30:38AM -0800, J.H. wrote:

> So apparently I'm having fun tripping over all kinds of bugs lately.
> I've seen this a couple of times now on the box in question.  Usually
> happens after a few days, or after particularly heavy rsync traffic on
> the box.
> 
> http://pastebin.osuosl.org/36014
> 
> Christoph seemed to think it's a memory exhaustion problem, so I've
> included the /proc/meminfo and as you can see there's plenty of memory
> around on the system.
> 
> Loads have, expectedly, climbed currently around 1250.05 but growing slowly.
> 
> Quick overview of the underlying storage:
> 
> xfs -> md (raid 0) -+--> P812 hardware raid6 (cciss driver)
>                     |
>                     +--> P812 hardware raid6 (cciss driver)
> 
> This is running on an HP DL380 G7.
> 
> I saw this both on an older 2.6.30.10-105.2.23.fc11.x86_64, and
> currently on 2.6.34.7-61.fc13.x86_64 (both being Fedora stock kernels)
> 
> I have not seen this on a very similar DL380 G6, with the same storage
> setup and it is currently running the 2.6.30 kernel from above.
> 
> Christoph suggest increasing the nr_request values for each of the
> underlying devices, but this didn't seem to change anything
> significantly on the system.
> 
> Anyone have any ideas on what's going on?

What does this show?

	iostat -x -k 1

In particular, "avgqu-sz" aka "average queue size" would be non-zero if
there are requests pending.  If r/s and w/s are zero over a long time
with the queue size being non-zero, the issuing of commands to the
hardware raid controller is stuck for some reason.

Since your Dirty and Writeback is pretty high, it sounds like this is the
issue.  Not sure where to go from there.

Simon-