domU NFS performance

domU NFS performance

[Nicholas Lee]

I've written up a short review of the past few months experimenting
with a Xen based Ubuntu NX thin client server.

http://stateless.geek.nz/2005/08/29/xen-disk-performance/

I'm wondering if other people have experience similar stalling issues
with  NFS services in a domU.  Is it a known issue?

-- 
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

Re: domU NFS performance

[Keir Fraser]

On 29 Aug 2005, at 10:29, Nicholas Lee wrote:

> I've written up a short review of the past few months experimenting
> with a Xen based Ubuntu NX thin client server.
>
> http://stateless.geek.nz/2005/08/29/xen-disk-performance/
>
> I'm wondering if other people have experience similar stalling issues
> with  NFS services in a domU.  Is it a known issue?

No, I would expect that to work just fine. We certainly run our own 
servers with most networked services running in domUs, but I don't 
think that includes an NFS server (i.e., that configuration is likely 
untested).

What version of Xen are you using? Anything special about your network 
setup (or is it the default bridged setup)? Any special NFS 
server/client settings?

  -- Keir

Re: domU NFS performance

[Nicholas Lee]

On 8/29/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> No, I would expect that to work just fine. We certainly run our own
> servers with most networked services running in domUs, but I don't
> think that includes an NFS server (i.e., that configuration is likely
> untested).
> 
> What version of Xen are you using? Anything special about your network
> setup (or is it the default bridged setup)? Any special NFS
> server/client settings?

Pretty standard Debian sarge installs.
nic@bowl:~$ xm dmesg  | head -15 | grep versio
  Xen version 2.0.6 (root@plumtree.co.nz) (gcc version 3.3.5 (Debian
1:3.3.5-13)) Fri Jun 24 01:18:07 NZST 2005

Xen bridged defaults for networking.

Bulk standard NFS. rsize and wsize tuning don't seem to make much
difference with the IO stalling issue.

In three configurations I've tested now with exactly the same domU.

NFS home-dir from domU
NFS home-dir from xen0 (host xen)
LVM home-dir

Only the first had these issues.

I haven't tested NFS home-dir from a dom0 (not the host xen).

NFS home-dir from domU over a network seemed to work fine.  

NFS/domU on the same machine is really the only major issue I've
faced. I have a couple very busy domUs.  One with a disk active 4GL
accounting application server on a LVM slice, and that has no
problems.

One of the things I've noted when using mutt is particular problems
with 'c' opening a new folder.  I'm using courier-imap/maildir, and my
mail spool has almost a 1000 folders.

-- 
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

Re: domU NFS performance

[Keir Fraser]

On 29 Aug 2005, at 11:23, Nicholas Lee wrote:

> One of the things I've noted when using mutt is particular problems
> with 'c' opening a new folder.  I'm using courier-imap/maildir, and my
> mail spool has almost a 1000 folders.

We certainly have seen inter-domain network problems in the past, but 
usually they crop up when running services in dom0, and moving them to 
domU fixes things. So this is odd because the behaviour appears to be 
reversed.

A few things to try: Use TCP rather than UDP as the transport protocol. 
Also, try setting rsize/wsize to less than the MTU (e.g. 1024). If the 
latter fixes things, try setting just one of rsize/wsize to the lower 
value.

UDP and IP fragmentation have been the biggest sources of problems for 
us. And I'm not totally sure how well Linux NFS implementation recovers 
from packet loss when it runs over UDP.

  -- Keir

Re: domU NFS performance

[Nicholas Lee]

On 8/29/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> A few things to try: Use TCP rather than UDP as the transport protocol.
> Also, try setting rsize/wsize to less than the MTU (e.g. 1024). If the
> latter fixes things, try setting just one of rsize/wsize to the lower
> value.
> 
> UDP and IP fragmentation have been the biggest sources of problems for
> us. And I'm not totally sure how well Linux NFS implementation recovers
> from packet loss when it runs over UDP.

Just quickly remounting in TCP mode for both the mail and shell server
didn't seem to help the mutt issue. I can reliably get this issue to
reoccur by simply 'R' entering the Drafts folder. Then 'e' exiting
back to the INBOX.  I'll see if I can put together a test account for
this at the end of the week.

I'll look at trying NFS with TCP or smaller UDP on the desktop domU tomorrow.

-- 
Nicholas Lee
http://stateless.geek.nz
gpg 8072 4F86 EDCD 4FC1 18EF  5BDD 07B0 9597 6D58 D70C

Re: domU NFS performance

[Karsten M. Self]

Nicholas Lee wrote:

>I've written up a short review of the past few months experimenting
>with a Xen based Ubuntu NX thin client server.
>
>http://stateless.geek.nz/2005/08/29/xen-disk-performance/
>
>I'm wondering if other people have experience similar stalling issues
>with  NFS services in a domU.  Is it a known issue?
>  
>
Nicholas:  I'm with Keir, this shouldn't be happening.  I'm with 
XenSource QA, this looks like an interesting situation.

Could you provide us with some additional information (some you've 
already given).

There's a script I've written to gather system information called, oddly 
enough, system-info.  Find a copy at 
http://linuxmafia.com/~karsten/Download/system-info (I'll see about 
getting this to the Cambridge site, we're looking at using this here).

Try running that on both the DomU and physical system as this will give 
us an idea of your system state as well.

You might also want to:

  - Provide specific NFS version info.
  - Check system logs on client and server for any possible related info.
  - Monitor traffic (tcpdump, ethereal, iptraf) to determine if latency 
is visible at network / protocol level.
  - Provide specific network hardware specs.  We *are* seeing issues 
with specific cards, and have seen some quirks with hub/switch 
configurations.

My own experience with mutt is that it can be notoriously laggy 
particularly dealing with large or frequently updated mailboxes.  It may 
not be the best general purpose network performance test tool.  Perhaps 
netperf (packaged for Debian) would be more useful.

As Kier indicates, we're largely dogfooding internally with services run 
via Xen DomUs, so this is a bit of a puzzler.

Cheers.

-- 
Karsten M. Self <karsten@xensource.com>
XenSource, Inc.
2300 Geng Road #250                                +1 650.798.5900 x259
Palo Alto, CA 94303                                +1 650.493.1579 fax