Re: NFS performance ...

Re: NFS performance ...

[Marc-Christian Petersen]

Hi Peter,

> I have a very simple NFS setup over a siwtched 100Mbit/s network.
> client is Celeron 400MHz/256M RAM, using XFS
> server is dual Pentium Pro 200MHz/1G RAM, using XFS
> server is running Linux 2.4.19-pre8aa3.
>
> Network bandwith can be utilized, because ICMP flooding the
> server results in ~20000 kbit/s network traffic (as of
> iptraf), but NFS (v3,udp) write performance is unacceptably
> slow (around 300 KiB/sec), same results with the following
> kernels:
> Linux 2.4.18-WOLK3.1
> Linux 2.4.18-wolk3.7.1
> Linux 2.4.20-pre8aa2
> However, with 2.4.19-rmap14b-xfs the very same NFS
> performance tops out at 2.54 MiB/sec.  What's the catch?
I think Andrea and me have something in our kernels that may cause it. For me 
I don't know what that can be. I even have no idea what it can be :(

Andrea, you?

Peter, have you also tested v3 over tcp?

ciao, Marc

Re: NFS performance ...

[Andrea Arcangeli]

On Sun, Nov 24, 2002 at 03:23:01PM +0100, Marc-Christian Petersen wrote:
> Hi Peter,
> 
> > I have a very simple NFS setup over a siwtched 100Mbit/s network.
> > client is Celeron 400MHz/256M RAM, using XFS
> > server is dual Pentium Pro 200MHz/1G RAM, using XFS
> > server is running Linux 2.4.19-pre8aa3.
> >
> > Network bandwith can be utilized, because ICMP flooding the
> > server results in ~20000 kbit/s network traffic (as of
> > iptraf), but NFS (v3,udp) write performance is unacceptably
> > slow (around 300 KiB/sec), same results with the following
> > kernels:
> > Linux 2.4.18-WOLK3.1
> > Linux 2.4.18-wolk3.7.1
> > Linux 2.4.20-pre8aa2
> > However, with 2.4.19-rmap14b-xfs the very same NFS
> > performance tops out at 2.54 MiB/sec.  What's the catch?
> I think Andrea and me have something in our kernels that may cause it. For me 
> I don't know what that can be. I even have no idea what it can be :(
> 
> Andrea, you?

nfs runs at 10mbyte/sec both directions for me, not on xfs if that
matters. can you try if you can reproduce with ext2 on both sides just
in case, also please try with 2.4.20rc2aa1 (the elevator-lowlatency will
make the system slower in some workload like dbench compared to rc1aa1,
but it doesn't matter for you since your bottleneck must be in the
network/fs layer not the blkdev layer).

> 
> Peter, have you also tested v3 over tcp?
> 
> ciao, Marc
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


Andrea

Re: NFS performance ...

[KELEMEN Peter]

* Marc-Christian Petersen (m.c.p@wolk-project.de) [20021124 15:23]:

Marc, Andrea,

> I think Andrea and me have something in our kernels that may
> cause it. For me I don't know what that can be. I even have no
> idea what it can be :(

The culprit turned out to be an inherited CONFIG_NFS_DIRECTIO
setting.  Having the client kernel (2.4.20rc2aa1) this option
turned off, performance is stable 4 MB/sec (server hasn't
changed).  This is almost twice as good as with 2.4.19-rmap14b.

I understand the CONFIG_NFS_DIRECTIO warning in Configure.help,
but why does it affect 1) client performance and 2) shouldn't it
only matter for files opened with open(..., O_DIRECT)?

> Peter, have you also tested v3 over tcp?

No, for various administrative reasons.

Peter

-- 
    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \       /      fuji@elte.hu
.+'         `+...+'         `+...+'         `+...+'         `+...+'

Re: NFS performance ...

[Trond Myklebust]

>>>>> " " == KELEMEN Peter <fuji@elte.hu> writes:

     > * Marc-Christian Petersen (m.c.p@wolk-project.de) [20021124
     >   15:23]:
     > Marc, Andrea,

    >> I think Andrea and me have something in our kernels that may
    >> cause it. For me I don't know what that can be. I even have no
    >> idea what it can be :(

     > The culprit turned out to be an inherited CONFIG_NFS_DIRECTIO
     > setting.  Having the client kernel (2.4.20rc2aa1) this option
     > turned off, performance is stable 4 MB/sec (server hasn't
     > changed).  This is almost twice as good as with 2.4.19-rmap14b.

Huh? Sounds like something is seriously screwed up in your kernel
build then. CONFIG_NFS_DIRECTIO should should result in 2 things only:

  - direct.c gets compiled.

  - the 'direct_IO' address space operation gets defined, so that the
    VFS knows what to do with files that get opened with the O_DIRECT
    flag.

None of the ordinary NFS read and write code paths should be affected
by the above.

Cheers,
  Trond

Re: NFS performance ...

[KELEMEN Peter]

* Trond Myklebust (trond.myklebust@fys.uio.no) [20021128 17:40]:

> > The culprit turned out to be an inherited CONFIG_NFS_DIRECTIO
> > setting.  Having the client kernel (2.4.20rc2aa1) this option
> > turned off, performance is stable 4 MB/sec (server hasn't
> > changed).  This is almost twice as good as with 2.4.19-rmap14b.

> Huh? Sounds like something is seriously screwed up in your
> kernel build then. CONFIG_NFS_DIRECTIO should should result in 2
> things only: [...] None of the ordinary NFS read and write code
> paths should be affected by the above.

Well, weird as it may seem, this is what happened.  I compiled
2.4.20rc2aa1 with my .config, NFS sucked.  make menuconfig, turned
off CONFIG_NFS_DIRECTIO, make -j2 bzImage modules modules_install
(no compiler errors), install kernel, lilo, reboot, NFS flies.
Confirmed on other machine as well.  gcc is 3.2.1 (Debian sid).
Wish to seek more input on the case?

Peter

-- 
    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \       /      fuji@elte.hu
.+'         `+...+'         `+...+'         `+...+'         `+...+'

Re: NFS performance ...

[Trond Myklebust]

>>>>> " " == KELEMEN Peter <fuji@elte.hu> writes:


     > 2.4.20rc2aa1 with my .config, NFS sucked.  make menuconfig,
     > turned off CONFIG_NFS_DIRECTIO, make -j2 bzImage modules
     > modules_install (no compiler errors), install kernel, lilo,
     > reboot, NFS flies.  Confirmed on other machine as well.  gcc is
     > 3.2.1 (Debian sid).  Wish to seek more input on the case?

I'd rather see if you can reproduce it on stock 2.4.20-pre4 + the
NFS_ALL patch. I have a strong feeling that this is something that is
particular to the aa kernels...

Cheers,
  Trond

Re: NFS performance ...

[KELEMEN Peter]

* Trond Myklebust (trond.myklebust@fys.uio.no) [20021129 07:51]:

> >>>>> " " == KELEMEN Peter <fuji@elte.hu> writes:
> > 2.4.20rc2aa1 with my .config, NFS sucked.  make menuconfig,
> > turned off CONFIG_NFS_DIRECTIO, make -j2 bzImage modules
> > modules_install (no compiler errors), install kernel, lilo,
> > reboot, NFS flies.  Confirmed on other machine as well.  gcc
> > is 3.2.1 (Debian sid).  Wish to seek more input on the case?

> I'd rather see if you can reproduce it on stock 2.4.20-pre4 +
> the NFS_ALL patch. I have a strong feeling that this is
> something that is particular to the aa kernels...

I was unable to reproduce it with your patches.

Peter

-- 
    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \     Peter.Kelemen@cern.ch
.+'         `+...+'         `+...+'         `+...+'         `+...+'

NFS performance ...

[KELEMEN Peter]

Hello,

I have a very simple NFS setup over a siwtched 100Mbit/s network.

client is Celeron 400MHz/256M RAM, using XFS
server is dual Pentium Pro 200MHz/1G RAM, using XFS
server is running Linux 2.4.19-pre8aa3.

Network bandwith can be utilized, because ICMP flooding the
server results in ~20000 kbit/s network traffic (as of
iptraf), but NFS (v3,udp) write performance is unacceptably
slow (around 300 KiB/sec), same results with the following
kernels:
Linux 2.4.18-WOLK3.1
Linux 2.4.18-wolk3.7.1
Linux 2.4.20-pre8aa2

However, with 2.4.19-rmap14b-xfs the very same NFS
performance tops out at 2.54 MiB/sec.  What's the catch?

TIA,
Peter

-- 
    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \       /      fuji@elte.hu
.+'         `+...+'         `+...+'         `+...+'         `+...+'