Re: help with bad performing raid6

[Jon Nelson <jnelson-suse@jamponi.net>]

On Wed, Jul 29, 2009 at 10:08 AM, Bill Davidsen<davidsen@tmr.com> wrote:
> Jon Nelson wrote:
..

>> When I say "bad performance" I mean writes that vary down to 100KB/s
>> or less, as reported by rsync. The "average" end-to-end speed for
>> writing large (500MB to 5GB) files hovers around 3-4MB/s. This is over
>> 100 MBit.
>>
>> Often times while stracing rsync I will see rsync not make a single
>> system call for sometimes more than a minute. Sometimes well in excess
>> of that. If I look at the load on the server the top process is
>> md0_raid5 (the raid6 process for md0, despite the raid5 in the name).
>> The load hovers around 8 or 9 at this time.
>>
>>
>
> I really suspect disk errors, I assume nothing in /var/log/messages?

Nope. Nothing in /v/l/m. I'm rather strongly beginning to suspect some
sort of weird NFS issue.

> Perhaps iostat looking at the underlying drives would tell you something.

> You might also run iostat with a test write load to see if something is
> unusual:
>  dd if=/dev/zero bs=1024k count=1024k of=BigJunk.File conv=fdatasync

During this test, vmstat reports blocks out of (infrequent) lows of
25000 to about 70000. The values seem to hover in the mid 60K (65MB/s
give or take). That seems very reasonable.

> Of course if it runs like a bat out hell, it tells you the problem is
> elsewhere.

> Other possible causes are a poor chunk size, bad alignment of the whole
> filesystem, and many other things too ugly to name. The fact that you use
> LVM make alignment issue more likely (in the sense of "one more level which
> could mess up"). Checked the error count on the array?

Well, since I can write some 25-30MB/s (actual underlying I/O much
higher obviously) to the same filesystem, and load hovers around 2.5
I'm suspecting some weird NFS issue.

The md0_raid5 process is in R or S most of the time, with about 30% of the CPU.

Summary: writing large files over NFS causes huge load and really
awful performance. Writing similarly large files directly (same
underlying filesystem, ext3) performs as expected without huge load.
Therefore, I am going to assume this is an NFS issue. I've more than
my fair share of NFS issues lately. :-(

PS. I'm running 2.6.27.25 stock openSUSE kernel. I just checked and it
does not apper to have the "NFS packet storm" patches which seems to
cause 2.6.27.X NFS performance to really suck.

Sorry for wasting everybody's time.

-- 
Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html