poor nfs server performance with 2.4.19-preX kernels vs. 2.4.17. Due to XFS and VM?

poor nfs server performance with 2.4.19-preX kernels vs. 2.4.17. Due to XFS and VM?

[HABBINGA,ERIK (HP-Loveland,ex1)]

Hi,
   I'm working on improving NFS server performance.  I've been able to get
satisfactory performance with the 2.4.17 kernel, NFSD BKL removal code, and
3GB kernel address space patches.  However, when I try to migrate to the
2.4.19-preX kernels, I get nothing but pain and suffering.  I'm not allowed
to post SPEC numbers, so I'm showing percentages versus my highest
performing test run.  I haven't had the opportunity to watch every test as
it runs, but have seen the 2.4.19-preX kernel tests spend a lot of time in
shrink_cache schedule() call right before the tests timeout.

All of the tests were run on the following hardware:

4 PIII Xeon processors
6GB RAM
SPEC SFS NFS test v3.0
2 Gigabit network connections to SPEC clients
many Fiber channel hard drives

I tested the following kernels:

2.4.17
2.4.19-pre9
2.4.19-pre10
2.4.19-pre10aa2 (config'd to use 3GB kernel address space)

I patch the kernel with a collection of the following patches:

- nfsd_bkl_removal 022702:
http://marc.theaimsgroup.com/?l=linux-nfs&m=101485118003322&w=2 (remove the
BKL from the nfsd code, add TCP support to nfsd, clean up the RPC stack
code)

- nfsd_bkl_removal 020702:
http://marc.theaimsgroup.com/?l=linux-nfs&m=100888008825015&w=2  (same
intention as nfsd_bkl_removal 022702, but doesn't include the NFSD TCP code
or RPC stack cleanup code)

- 3GB kernel address space:
http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.18pre7
aa2/00_3.5G-address-space-4
   http://www.linuxhq.com/kernel/v2.4/unofficial/patch200202/89.html (give
the kernel 3GB of the virtual address space)

- akpm_nuke_buffers:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102226904021069&w=2 (hunt
down buffer_heads and kill them)

- XFS taken from the XFS CVS tree
(http://oss.sgi.com/cgi-bin/cvsweb.cgi/linux-2.4-xfs/) on Feb 7 2002, June 3
2002, and June 11 2002.  2.4.19-pre10aa2 already includes XFS, but I don't
know its vintage.

The only modifications to any /proc tunables were to increase the UDP stack
size (/proc/sys/net/core/rmem_[default|max]  to 512K


And here are the results:

2.4.17 nfsd_bkl_removal 022702, 3GB kernel address space, XFS 020702
- baseline

2.4.17 nfsd_bkl_removal 022702, XFS 020702
- 50% of baseline

2.4.19-pre9 nfsd_bkl_removal 022702, XFS 060302 
- 0% of baseline, would not run

2.4.19-pre10 xfs 020702
- 45% of baseline, probably would have kept running, but I stopped the test
after max throughput was reached so I could try another configuration

2.4.19-pre10 xfs 061102
- 20% of baseline before timeout, lots of processes stuck in shrink_cache
schedule() call

2.4.19-pre10 xfs 061102, akpm_nuke_buffers
- 30% of baseline before timeout

2.4.19-pre10 xfs 061102, nfsd_bkl_removal 020702, akpm_nuke_buffers
- 10% of baseline before timeout

2.4.19-pre10 xfs 061102, nfsd_bkl_removal 020702
- 20% of baseline before timeout

2.4.19-pre10aa2 (3GB kernel address space)
- 10% of baseline before timeout

2.4.19-pre10aa2 (3GB kernel address space), nfsd_bkl_removal 020702
- wouldn't run

Conclusions:
- using the 3GB kernel address space substantially helps 2.4.17 nfsd
performance
- 2.4.19-pre10 with XFS from Feb 7, 2002 was the most stable of the
2.4.19-preX runs

I will be running 2.4.19-pre10 xfs 061102, 3GB kernel address space later
today or tomorrow.

I will try any patches, kernel CONFIG options, or  /proc values anyone might
suggest to get the 2.4.19-preX kernel nfsd performance at or near the 2.4.17
nfsd performance.

Thanks,
Erik

Re: poor nfs server performance with 2.4.19-preX kernels vs. 2.4.17. Due to XFS and VM?

[kwijibo]

Why not try another benchmark that you can post the
results from?  Like Bonnie++, Postmark, Iozone, etc.
These are the benchmarks I use for my NFS benchmarking.

Steve


HABBINGA,ERIK (HP-Loveland,ex1) wrote:

>Hi,
>   I'm working on improving NFS server performance.  I've been able to get
>satisfactory performance with the 2.4.17 kernel, NFSD BKL removal code, and
>3GB kernel address space patches.  However, when I try to migrate to the
>2.4.19-preX kernels, I get nothing but pain and suffering.  I'm not allowed
>to post SPEC numbers, so I'm showing percentages versus my highest
>performing test run.  I haven't had the opportunity to watch every test as
>it runs, but have seen the 2.4.19-preX kernel tests spend a lot of time in
>shrink_cache schedule() call right before the tests timeout.
>
>All of the tests were run on the following hardware:
>
>4 PIII Xeon processors
>6GB RAM
>SPEC SFS NFS test v3.0
>2 Gigabit network connections to SPEC clients
>many Fiber channel hard drives
>
>I tested the following kernels:
>
>2.4.17
>2.4.19-pre9
>2.4.19-pre10
>2.4.19-pre10aa2 (config'd to use 3GB kernel address space)
>
>I patch the kernel with a collection of the following patches:
>
>- nfsd_bkl_removal 022702:
>http://marc.theaimsgroup.com/?l=linux-nfs&m=101485118003322&w=2 (remove the
>BKL from the nfsd code, add TCP support to nfsd, clean up the RPC stack
>code)
>
>- nfsd_bkl_removal 020702:
>http://marc.theaimsgroup.com/?l=linux-nfs&m=100888008825015&w=2  (same
>intention as nfsd_bkl_removal 022702, but doesn't include the NFSD TCP code
>or RPC stack cleanup code)
>
>- 3GB kernel address space:
>http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.18pre7
>aa2/00_3.5G-address-space-4
>   http://www.linuxhq.com/kernel/v2.4/unofficial/patch200202/89.html (give
>the kernel 3GB of the virtual address space)
>
>- akpm_nuke_buffers:
>http://marc.theaimsgroup.com/?l=linux-kernel&m=102226904021069&w=2 (hunt
>down buffer_heads and kill them)
>
>- XFS taken from the XFS CVS tree
>(http://oss.sgi.com/cgi-bin/cvsweb.cgi/linux-2.4-xfs/) on Feb 7 2002, June 3
>2002, and June 11 2002.  2.4.19-pre10aa2 already includes XFS, but I don't
>know its vintage.
>
>The only modifications to any /proc tunables were to increase the UDP stack
>size (/proc/sys/net/core/rmem_[default|max]  to 512K
>
>
>And here are the results:
>
>2.4.17 nfsd_bkl_removal 022702, 3GB kernel address space, XFS 020702
>- baseline
>
>2.4.17 nfsd_bkl_removal 022702, XFS 020702
>- 50% of baseline
>
>2.4.19-pre9 nfsd_bkl_removal 022702, XFS 060302 
>- 0% of baseline, would not run
>
>2.4.19-pre10 xfs 020702
>- 45% of baseline, probably would have kept running, but I stopped the test
>after max throughput was reached so I could try another configuration
>
>2.4.19-pre10 xfs 061102
>- 20% of baseline before timeout, lots of processes stuck in shrink_cache
>schedule() call
>
>2.4.19-pre10 xfs 061102, akpm_nuke_buffers
>- 30% of baseline before timeout
>
>2.4.19-pre10 xfs 061102, nfsd_bkl_removal 020702, akpm_nuke_buffers
>- 10% of baseline before timeout
>
>2.4.19-pre10 xfs 061102, nfsd_bkl_removal 020702
>- 20% of baseline before timeout
>
>2.4.19-pre10aa2 (3GB kernel address space)
>- 10% of baseline before timeout
>
>2.4.19-pre10aa2 (3GB kernel address space), nfsd_bkl_removal 020702
>- wouldn't run
>
>Conclusions:
>- using the 3GB kernel address space substantially helps 2.4.17 nfsd
>performance
>- 2.4.19-pre10 with XFS from Feb 7, 2002 was the most stable of the
>2.4.19-preX runs
>
>I will be running 2.4.19-pre10 xfs 061102, 3GB kernel address space later
>today or tomorrow.
>
>I will try any patches, kernel CONFIG options, or  /proc values anyone might
>suggest to get the 2.4.19-preX kernel nfsd performance at or near the 2.4.17
>nfsd performance.
>
>Thanks,
>Erik
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>