NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

Hi guys,

I just found out... Installed 2.6.16.1 (32-bit) on a spanking new dual
opteron 275 (dual-core) machine, and saw that my link jobs were taking
ages.

I narrowed it down a bit - these are the kernels I have tested:
2.6.13.5:  Good
2.6.14.7:  Good
2.6.15:    Poor
2.6.15.7:  Poor
2.6.16.1:  Poor

Sequential NFS I/O is good on all kernels. Only "ld" shows the problem.

On 2.6.14.7, I can run a large link job creating a 60 MB executable in
15.6 seconds wall-clock time.

On 2.6.15, the same link job takes 2 minutes 28 seconds.

This is almost 10 *times* longer.

Testing with tiobench, I can see no notable difference between the
kernels (!)   It seems that this is very specific to ld.  I am using GNU
ld version 2.15.

The NFS client mounts the working directory using NFS v3 over UDP with
default (32k) rsize/wsize.

Since this machine is not in production yet, I can experiment with
kernel patches on it - I would like to try and narrow this down even
further - any suggestions as to which patches to exclude/include will be
greatly appreciated.

Note; I did double-check all the common problems (ethernet problems,
mount options, ...).

-- 

 / jakob

NFS client regression, simple test program

[Jakob Oestergaard]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]


Attached is a small test program that exposes the regression.

Kernel LEADING_EMPTY_SPACE wall-clock time to run
2.6.15    0                0.2s
2.6.15    1                29.3s
2.6.14.7  0                0.2s
2.6.14.7  1                0.2s

So it seems that if a file is repeatedly read from and written to, it
matters a lot for later kernels whether or not the blocks that are
worked on, are offset a single byte or not...

Any ideas?

-- 

 / jakob


[-- Attachment #2: nfsbench.c --]
[-- Type: text/x-csrc, Size: 1227 bytes --]

/*
 * Tiny benchmark for Linux NFS client problem 2.6.14.7->2.6.15
 *
 * By Jakob Oestergaard, joe@unthought.net / joe@evalesco.com
 *
 * Compile as:
 *  gcc -o nfsbench -O3 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 nfsbench.c
 */

#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <fcntl.h>
#include <stdint.h>

#define MYBLOCKSIZE 32768
#define MYSUBSIZE 1024

/* Set this define to non-zero to expose the regression */
#define LEADING_EMPTY_SPACE 1

int main(int argc, char **argv)
{
  int testfd = open("testfile", O_RDWR|O_CREAT|O_TRUNC, 0666);
  off_t blockoff;
  uint8_t garbage[MYBLOCKSIZE];

  memset(garbage, 42, sizeof garbage);

  for (blockoff = LEADING_EMPTY_SPACE;
       blockoff < 10 * 1024 * 1024;
       blockoff += MYBLOCKSIZE) {
    off_t inblockoff;
    lseek(testfd, blockoff, SEEK_SET);
    write(testfd, garbage, MYBLOCKSIZE);

    for (inblockoff = MYBLOCKSIZE / 4;
	 inblockoff < MYBLOCKSIZE * 3 / 4;
	 inblockoff += MYSUBSIZE * 2) {
      uint8_t gdata[MYBLOCKSIZE];
      lseek(testfd, blockoff, SEEK_SET);
      read(testfd, gdata, MYBLOCKSIZE);
      lseek(testfd, -inblockoff, SEEK_CUR);
      write(testfd, gdata + inblockoff, MYSUBSIZE);
    }
  }

  close(testfd);
}

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust]

On Fri, 2006-03-31 at 11:48 +0200, Jakob Oestergaard wrote:
> Hi guys,
> 
> I just found out... Installed 2.6.16.1 (32-bit) on a spanking new dual
> opteron 275 (dual-core) machine, and saw that my link jobs were taking
> ages.
> 
> I narrowed it down a bit - these are the kernels I have tested:
> 2.6.13.5:  Good
> 2.6.14.7:  Good
> 2.6.15:    Poor
> 2.6.15.7:  Poor
> 2.6.16.1:  Poor
> 
> Sequential NFS I/O is good on all kernels. Only "ld" shows the problem.
> 
> On 2.6.14.7, I can run a large link job creating a 60 MB executable in
> 15.6 seconds wall-clock time.
> 
> On 2.6.15, the same link job takes 2 minutes 28 seconds.
> 
> This is almost 10 *times* longer.
> 
> Testing with tiobench, I can see no notable difference between the
> kernels (!)   It seems that this is very specific to ld.  I am using GNU
> ld version 2.15.
> 
> The NFS client mounts the working directory using NFS v3 over UDP with
> default (32k) rsize/wsize.
> 
> Since this machine is not in production yet, I can experiment with
> kernel patches on it - I would like to try and narrow this down even
> further - any suggestions as to which patches to exclude/include will be
> greatly appreciated.

Some nfsstat output comparing the good and bad cases would help.

Cheers,
  Trond

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Fri, Mar 31, 2006 at 07:22:50AM -0500, Trond Myklebust wrote:
...
> 
> Some nfsstat output comparing the good and bad cases would help.

Clean boot on 2.6.15 and 2.6.14.7, one run of nfsbench with
LEADING_EMPTY_SPACE=1.  I've skipped the NFS v2 stats because they're
all 0.

--- Run on bad kernel ---

[puffin:joe] $ uname -a
Linux puffin 2.6.15 #1 SMP Fri Mar 31 11:10:28 CEST 2006 i686 GNU/Linux
[puffin:joe] $ nfsstat 
Client rpc stats:
calls      retrans    authrefrsh
63         0          0       

<snip v2 stats>

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 11     18% 0       0% 26     42% 14     22% 0       0% 
read       write      create     mkdir      symlink    mknod      
4       6% 0       0% 0       0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
0       0% 0       0% 0       0% 0       0% 0       0% 4       6% 
fsstat     fsinfo     pathconf   commit     
0       0% 2       3% 0       0% 0       0% 
[puffin:joe] $ time ./nfsbench 

real    0m29.242s
user    0m0.000s
sys     0m0.160s
[puffin:joe] $ nfsstat 
Client rpc stats:
calls      retrans    authrefrsh
13240      0          0       

<snip v2 stats>

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 2583   19% 0       0% 30      0% 24      0% 0       0% 
read       write      create     mkdir      symlink    mknod      
7045   53% 3200   24% 1       0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
0       0% 0       0% 0       0% 0       0% 0       0% 34      0% 
fsstat     fsinfo     pathconf   commit     
0       0% 2       0% 0       0% 319     2% 

[puffin:joe] $

--- Run on good kernel ---

[puffin:joe] $ uname -a
Linux puffin 2.6.14.7 #1 SMP Fri Mar 31 10:41:59 CEST 2006 i686
GNU/Linux
[puffin:joe] $ nfsstat 
Client rpc stats:
calls      retrans    authrefrsh
52         0          0       

<snip v2 stats>

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 7      14% 0       0% 24     48% 13     26% 0       0% 
read       write      create     mkdir      symlink    mknod      
4       8% 0       0% 0       0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
0       0% 0       0% 0       0% 0       0% 0       0% 0       0% 
fsstat     fsinfo     pathconf   commit     
0       0% 2       4% 0       0% 0       0% 

[puffin:joe] $ time ./nfsbench 

real    0m0.224s
user    0m0.000s
sys     0m0.072s
[puffin:joe] $ nfsstat 
Client rpc stats:
calls      retrans    authrefrsh
384        0          0       

<snip v2 stats>

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 10      2% 1       0% 26      6% 15      3% 0       0% 
read       write      create     mkdir      symlink    mknod      
6       1% 321    84% 0       0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
0       0% 0       0% 0       0% 0       0% 0       0% 0       0% 
fsstat     fsinfo     pathconf   commit     
0       0% 2       0% 0       0% 1       0% 

[puffin:joe] $


-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust]

On Fri, 2006-03-31 at 14:45 +0200, Jakob Oestergaard wrote:
> On Fri, Mar 31, 2006 at 07:22:50AM -0500, Trond Myklebust wrote:
> ...
> > 
> > Some nfsstat output comparing the good and bad cases would help.
> 
> Clean boot on 2.6.15 and 2.6.14.7, one run of nfsbench with
> LEADING_EMPTY_SPACE=1.  I've skipped the NFS v2 stats because they're
> all 0.

Why all the GETATTR calls? Are you running with 'noac' set?

I don't have a 2.6.15 kernel to run with, but on a recent git pull, I
get a total of 6 GETATTR calls when I run your nfsbench program.

The number of READ calls is 1, and the number of WRITE calls is 161 (I'm
running with 64k wsize).

Cheers,
  Trond

> --- Run on bad kernel ---
> 
> [puffin:joe] $ uname -a
> Linux puffin 2.6.15 #1 SMP Fri Mar 31 11:10:28 CEST 2006 i686 GNU/Linux
> [puffin:joe] $ nfsstat 
> Client rpc stats:
> calls      retrans    authrefrsh
> 63         0          0       
> 
> <snip v2 stats>
> 
> Client nfs v3:
> null       getattr    setattr    lookup     access     readlink   
> 0       0% 11     18% 0       0% 26     42% 14     22% 0       0% 
> read       write      create     mkdir      symlink    mknod      
> 4       6% 0       0% 0       0% 0       0% 0       0% 0       0% 
> remove     rmdir      rename     link       readdir    readdirplus
> 0       0% 0       0% 0       0% 0       0% 0       0% 4       6% 
> fsstat     fsinfo     pathconf   commit     
> 0       0% 2       3% 0       0% 0       0% 
> [puffin:joe] $ time ./nfsbench 
> 
> real    0m29.242s
> user    0m0.000s
> sys     0m0.160s
> [puffin:joe] $ nfsstat 
> Client rpc stats:
> calls      retrans    authrefrsh
> 13240      0          0       
> 
> <snip v2 stats>
> 
> Client nfs v3:
> null       getattr    setattr    lookup     access     readlink   
> 0       0% 2583   19% 0       0% 30      0% 24      0% 0       0% 
> read       write      create     mkdir      symlink    mknod      
> 7045   53% 3200   24% 1       0% 0       0% 0       0% 0       0% 
> remove     rmdir      rename     link       readdir    readdirplus
> 0       0% 0       0% 0       0% 0       0% 0       0% 34      0% 
> fsstat     fsinfo     pathconf   commit     
> 0       0% 2       0% 0       0% 319     2% 
> 
> [puffin:joe] $
> 
> --- Run on good kernel ---
> 
> [puffin:joe] $ uname -a
> Linux puffin 2.6.14.7 #1 SMP Fri Mar 31 10:41:59 CEST 2006 i686
> GNU/Linux
> [puffin:joe] $ nfsstat 
> Client rpc stats:
> calls      retrans    authrefrsh
> 52         0          0       
> 
> <snip v2 stats>
> 
> Client nfs v3:
> null       getattr    setattr    lookup     access     readlink   
> 0       0% 7      14% 0       0% 24     48% 13     26% 0       0% 
> read       write      create     mkdir      symlink    mknod      
> 4       8% 0       0% 0       0% 0       0% 0       0% 0       0% 
> remove     rmdir      rename     link       readdir    readdirplus
> 0       0% 0       0% 0       0% 0       0% 0       0% 0       0% 
> fsstat     fsinfo     pathconf   commit     
> 0       0% 2       4% 0       0% 0       0% 
> 
> [puffin:joe] $ time ./nfsbench 
> 
> real    0m0.224s
> user    0m0.000s
> sys     0m0.072s
> [puffin:joe] $ nfsstat 
> Client rpc stats:
> calls      retrans    authrefrsh
> 384        0          0       
> 
> <snip v2 stats>
> 
> Client nfs v3:
> null       getattr    setattr    lookup     access     readlink   
> 0       0% 10      2% 1       0% 26      6% 15      3% 0       0% 
> read       write      create     mkdir      symlink    mknod      
> 6       1% 321    84% 0       0% 0       0% 0       0% 0       0% 
> remove     rmdir      rename     link       readdir    readdirplus
> 0       0% 0       0% 0       0% 0       0% 0       0% 0       0% 
> fsstat     fsinfo     pathconf   commit     
> 0       0% 2       0% 0       0% 1       0% 
> 
> [puffin:joe] $
> 
> 

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Fri, Mar 31, 2006 at 08:06:32AM -0500, Trond Myklebust wrote:
> On Fri, 2006-03-31 at 14:45 +0200, Jakob Oestergaard wrote:
> > On Fri, Mar 31, 2006 at 07:22:50AM -0500, Trond Myklebust wrote:
> > ...
> > > 
> > > Some nfsstat output comparing the good and bad cases would help.
> > 
> > Clean boot on 2.6.15 and 2.6.14.7, one run of nfsbench with
> > LEADING_EMPTY_SPACE=1.  I've skipped the NFS v2 stats because they're
> > all 0.
> 
> Why all the GETATTR calls?

That's the $1000 question I guess :)

> Are you running with 'noac' set?

I didn't set it, that's for sure :)  I got the option lines from
/proc/mounts, if there is any way I can get more/other information
please advise.

2.6.14.7:
 rw,v3,rsize=32768,wsize=32768,hard,intr,udp,lock,addr=...

2.6.15:
 rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=udp,addr=...

Except for some formatting (proto=udp insted of udp) I fail to see any
difference.

The way I see it, it seems like caching on the client side fails
completely if the read requests are not aligned (to some
buffer/block/page/whatever).

> I don't have a 2.6.15 kernel to run with, but on a recent git pull, I
> get a total of 6 GETATTR calls when I run your nfsbench program.

The performance regression is present on 2.6.16.1 too.

Do you have a released kernel you can test with (2.6.1[56].*), or can I
somehow get the kernel you're testing with, just so that we test on the
same kernel?

> The number of READ calls is 1, and the number of WRITE calls is 161 (I'm
> running with 64k wsize).

I can't set 64k wsize:

puffin:~# mount -o rw,wsize=65536,udp sparrow:/exported/joe /mnt
puffin:~# cat /proc/mounts 
...
sparrow:/exported/joe /mnt nfs rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=udp,addr=sparrow 0 0
puffin:~#

Server is running a patched 2.6.11.11 kernel - could that be what's
preventing me from 64k wsize?

What happens if you run with 32k rsize/wsize?

-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust]

On Fri, 2006-03-31 at 15:21 +0200, Jakob Oestergaard wrote:
> On Fri, Mar 31, 2006 at 08:06:32AM -0500, Trond Myklebust wrote:
> > On Fri, 2006-03-31 at 14:45 +0200, Jakob Oestergaard wrote:
> > > On Fri, Mar 31, 2006 at 07:22:50AM -0500, Trond Myklebust wrote:
> > > ...
> > > > 
> > > > Some nfsstat output comparing the good and bad cases would help.
> > > 
> > > Clean boot on 2.6.15 and 2.6.14.7, one run of nfsbench with
> > > LEADING_EMPTY_SPACE=1.  I've skipped the NFS v2 stats because they're
> > > all 0.
> > 
> > Why all the GETATTR calls?
> 
> That's the $1000 question I guess :)
> 
> > Are you running with 'noac' set?
> 
> I didn't set it, that's for sure :)  I got the option lines from
> /proc/mounts, if there is any way I can get more/other information
> please advise.
> 
> 2.6.14.7:
>  rw,v3,rsize=32768,wsize=32768,hard,intr,udp,lock,addr=...
> 
> 2.6.15:
>  rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=udp,addr=...
> 
> Except for some formatting (proto=udp insted of udp) I fail to see any
> difference.
> 
> The way I see it, it seems like caching on the client side fails
> completely if the read requests are not aligned (to some
> buffer/block/page/whatever).
> 
> > I don't have a 2.6.15 kernel to run with, but on a recent git pull, I
> > get a total of 6 GETATTR calls when I run your nfsbench program.
> 
> The performance regression is present on 2.6.16.1 too.
> 
> Do you have a released kernel you can test with (2.6.1[56].*), or can I
> somehow get the kernel you're testing with, just so that we test on the
> same kernel?

Just apply
http://client.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-NFS_ALL.dif

to a clean 2.6.16.

> > The number of READ calls is 1, and the number of WRITE calls is 161 (I'm
> > running with 64k wsize).
> 
> I can't set 64k wsize:
> 
> puffin:~# mount -o rw,wsize=65536,udp sparrow:/exported/joe /mnt
> puffin:~# cat /proc/mounts 
> ...
> sparrow:/exported/joe /mnt nfs rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=udp,addr=sparrow 0 0
> puffin:~#
> 
> Server is running a patched 2.6.11.11 kernel - could that be what's
> preventing me from 64k wsize?

Linux servers do not yet support anything larger than a 32k r/wsize (and
in any case, you would have to switch towards using TCP). My home
directory is on a filer.

> What happens if you run with 32k rsize/wsize?

Number of writes balloons to 321.

If I switch to UDP, I get no change.

Cheers,
  Trond

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

On Fri, Mar 31, 2006 at 08:44:18AM -0500, Trond Myklebust wrote:
...
> Just apply
> http://client.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-NFS_ALL.dif
> 
> to a clean 2.6.16.

Done. Plain 2.6.16 plus the patch.

---------------------------------------
[puffin:joe] $ time ./nfsbench

real    0m29.581s
user    0m0.000s
sys     0m0.208s
[puffin:joe] $ nfsstat 
Client rpc stats:
calls      retrans    authrefrsh
13222      0          0       

<snip v2>

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 2577   19% 1       0% 24      0% 18      0% 0       0% 
read       write      create     mkdir      symlink    mknod      
7045   53% 3200   24% 0       0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
0       0% 0       0% 0       0% 0       0% 0       0% 34      0% 
fsstat     fsinfo     pathconf   commit     
0       0% 2       0% 0       0% 319     2% 

---------------------------------------

Lots of getattr and lots of read...

I verified that the patch really got applied.

What could the difference be?

I've attached my .config - if there is anything in there you think might
have an impact, please let me know - again, I can try anything on this
machine...

...
> Linux servers do not yet support anything larger than a 32k r/wsize (and
> in any case, you would have to switch towards using TCP). My home
> directory is on a filer.

Ok.

-- 

 / jakob


[-- Attachment #2: .config --]
[-- Type: text/plain, Size: 23274 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.16
# Fri Mar 31 15:57:40 2006
#
CONFIG_X86_32=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
# CONFIG_IKCONFIG is not set
# CONFIG_CPUSETS is not set
CONFIG_INITRAMFS_SOURCE=""
CONFIG_UID16=y
CONFIG_VM86=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_SLAB=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
# CONFIG_SLOB is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

#
# Block layer
#
# CONFIG_LBD is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
CONFIG_IOSCHED_DEADLINE=y
# CONFIG_IOSCHED_CFQ is not set
# CONFIG_DEFAULT_AS is not set
CONFIG_DEFAULT_DEADLINE=y
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="deadline"

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
CONFIG_MK8=y
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_SMP=y
CONFIG_NR_CPUS=4
# CONFIG_SCHED_SMT is not set
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_BKL is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_NONFATAL is not set
# CONFIG_X86_MCE_P4THERMAL is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_VMSPLIT_3G=y
# CONFIG_VMSPLIT_3G_OPT is not set
# CONFIG_VMSPLIT_2G is not set
# CONFIG_VMSPLIT_1G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_HIGHPTE=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
CONFIG_IRQBALANCE=y
CONFIG_REGPARM=y
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_HOTPLUG_CPU is not set
CONFIG_DOUBLEFAULT=y

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
CONFIG_PM_LEGACY=y
# CONFIG_PM_DEBUG is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=y
# CONFIG_ACPI_HOTKEY is not set
CONFIG_ACPI_FAN=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set

#
# APM (Advanced Power Management) BIOS Support
#
# CONFIG_APM is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_ISA_DMA_API=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
# CONFIG_NETDEBUG is not set
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_BIC=y
# CONFIG_IPV6 is not set
# CONFIG_NETFILTER is not set

#
# DCCP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set

#
# TIPC Configuration (EXPERIMENTAL)
#
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_IEEE80211 is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
# CONFIG_FW_LOADER is not set

#
# Connector - unified userspace <-> kernelspace linker
#
# CONFIG_CONNECTOR is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play support
#
# CONFIG_PNP is not set

#
# Block devices
#
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_BLK_DEV_RAM_COUNT=16
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y

#
# SCSI Transport Attributes
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
CONFIG_SCSI_SATA=y
# CONFIG_SCSI_SATA_AHCI is not set
# CONFIG_SCSI_SATA_SVW is not set
# CONFIG_SCSI_ATA_PIIX is not set
# CONFIG_SCSI_SATA_MV is not set
# CONFIG_SCSI_SATA_NV is not set
# CONFIG_SCSI_PDC_ADMA is not set
# CONFIG_SCSI_SATA_QSTOR is not set
# CONFIG_SCSI_SATA_PROMISE is not set
# CONFIG_SCSI_SATA_SX4 is not set
CONFIG_SCSI_SATA_SIL=y
# CONFIG_SCSI_SATA_SIL24 is not set
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_ULI is not set
# CONFIG_SCSI_SATA_VIA is not set
# CONFIG_SCSI_SATA_VITESSE is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
# CONFIG_MD_LINEAR is not set
# CONFIG_MD_RAID0 is not set
CONFIG_MD_RAID1=y
# CONFIG_MD_RAID10 is not set
# CONFIG_MD_RAID5 is not set
# CONFIG_MD_RAID6 is not set
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_CRYPT is not set
CONFIG_DM_SNAPSHOT=y
# CONFIG_DM_MIRROR is not set
# CONFIG_DM_ZERO is not set
# CONFIG_DM_MULTIPATH is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
# CONFIG_FUSION_SAS is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Network device support
#
CONFIG_NETDEVICES=y
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# PHY device support
#

#
# Ethernet (10 or 100Mbit)
#
# CONFIG_NET_ETHERNET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_SK98LIN is not set
CONFIG_TIGON3=y
# CONFIG_BNX2 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_CHELSIO_T1 is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
# CONFIG_SERIAL_8250_ACPI is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set

#
# IPMI
#
CONFIG_IPMI_HANDLER=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=y
CONFIG_IPMI_SI=y
CONFIG_IPMI_WATCHDOG=y
CONFIG_IPMI_POWEROFF=y

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
CONFIG_HW_RANDOM=y
CONFIG_NVRAM=y
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_HPET_MMAP=y
# CONFIG_HANGCHECK_TIMER is not set

#
# TPM devices
#
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_STUB=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
CONFIG_I2C_PCA_ISA=m

#
# Miscellaneous I2C Chip support
#
CONFIG_SENSORS_DS1337=m
CONFIG_SENSORS_DS1374=m
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_SENSORS_PCA9539=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_RTC8564=m
CONFIG_SENSORS_MAX6875=m
CONFIG_RTC_X1205_I2C=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# SPI support
#
# CONFIG_SPI is not set
# CONFIG_SPI_MASTER is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Hardware Monitoring support
#
CONFIG_HWMON=m
CONFIG_HWMON_VID=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_FSCPOS=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
CONFIG_HWMON_DEBUG_CHIP=y

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia Capabilities Port drivers
#

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
# CONFIG_FB is not set
# CONFIG_VIDEO_SELECT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
# CONFIG_SOUND is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_SPLIT_ISO is not set
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#

#
# may also be needed; see USB_STORAGE Help for more information
#
# CONFIG_USB_STORAGE is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_USB_HIDINPUT=y
# CONFIG_USB_HIDINPUT_POWERBOOK is not set
# CONFIG_HID_FF is not set
# CONFIG_USB_HIDDEV is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_ACECAD is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_ITMTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_YEALINK is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set
# CONFIG_USB_ATI_REMOTE2 is not set
# CONFIG_USB_KEYSPAN_REMOTE is not set
# CONFIG_USB_APPLETOUCH is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set

#
# Video4Linux support is needed for USB Multimedia device support
#

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_MON is not set

#
# USB port drivers
#

#
# USB Serial Converter support
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETKIT is not set
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set

#
# USB DSL modem support
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# EDAC - error detection and reporting (RAS) (EXPERIMENTAL)
#
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_MM_EDAC=y
# CONFIG_EDAC_AMD76X is not set
# CONFIG_EDAC_E7XXX is not set
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82875P is not set
# CONFIG_EDAC_I82860 is not set
# CONFIG_EDAC_R82600 is not set
CONFIG_EDAC_POLL=y

#
# File systems
#
# CONFIG_EXT2_FS is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_XATTR is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_XFS_FS=y
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_SECURITY is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_INOTIFY=y
CONFIG_QUOTA=y
# CONFIG_QFMT_V1 is not set
# CONFIG_QFMT_V2 is not set
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y
# CONFIG_FUSE_FS is not set

#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set

#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
# CONFIG_RELAYFS_FS is not set
# CONFIG_CONFIGFS_FS is not set

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_DIRECTIO is not set
# CONFIG_NFSD is not set
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
# CONFIG_9P_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
# CONFIG_NLS is not set

#
# Instrumentation Support
#
# CONFIG_PROFILING is not set
# CONFIG_KPROBES is not set

#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
# CONFIG_MAGIC_SYSRQ is not set
# CONFIG_DEBUG_KERNEL is not set
CONFIG_LOG_BUF_SHIFT=15
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_EARLY_PRINTK=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
# CONFIG_CRYPTO is not set

#
# Hardware crypto devices
#

#
# Library routines
#
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
# CONFIG_CRC32 is not set
# CONFIG_LIBCRC32C is not set
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust]

On Fri, 2006-03-31 at 16:08 +0200, Jakob Oestergaard wrote:
> On Fri, Mar 31, 2006 at 08:44:18AM -0500, Trond Myklebust wrote:
> ...
> > Just apply
> > http://client.linux-nfs.org/Linux-2.6.x/2.6.16/linux-2.6.16-NFS_ALL.dif
> > 
> > to a clean 2.6.16.
> 
> Done. Plain 2.6.16 plus the patch.
> 
> ---------------------------------------
> [puffin:joe] $ time ./nfsbench
> 
> real    0m29.581s
> user    0m0.000s
> sys     0m0.208s
> [puffin:joe] $ nfsstat 
> Client rpc stats:
> calls      retrans    authrefrsh
> 13222      0          0       
> 
> <snip v2>
> 
> Client nfs v3:
> null       getattr    setattr    lookup     access     readlink   
> 0       0% 2577   19% 1       0% 24      0% 18      0% 0       0% 
> read       write      create     mkdir      symlink    mknod      
> 7045   53% 3200   24% 0       0% 0       0% 0       0% 0       0% 
> remove     rmdir      rename     link       readdir    readdirplus
> 0       0% 0       0% 0       0% 0       0% 0       0% 34      0% 
> fsstat     fsinfo     pathconf   commit     
> 0       0% 2       0% 0       0% 319     2% 
> 
> ---------------------------------------
> 
> Lots of getattr and lots of read...
> 
> I verified that the patch really got applied.
> 
> What could the difference be?
> 
> I've attached my .config - if there is anything in there you think might
> have an impact, please let me know - again, I can try anything on this
> machine...

Hmm... Nothing obvious.

Try catting /proc/self/mountstats and see if the entry for your NFS
mount shows anything interesting.

Cheers,
  Trond

> ...
> > Linux servers do not yet support anything larger than a 32k r/wsize (and
> > in any case, you would have to switch towards using TCP). My home
> > directory is on a filer.
> 
> Ok.
> 

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Fri, Mar 31, 2006 at 09:21:29AM -0500, Trond Myklebust wrote:
...
> 
> Hmm... Nothing obvious.

Ok.

I'm wondering... Can anyone else reproduce this problem?

Just to explain quickly:
 Running nfsbench (on the NFS client) once with LEADING_EMPTY_SPACE set
 to 0 and then once with the option set to 1.  If there's a big change
 in wall-clock execution time, this indicates that the problem exists.

I'd be really interested in knowing whether I'm the only one who sees
this problem.

> Try catting /proc/self/mountstats and see if the entry for your NFS
> mount shows anything interesting.

mountstats doesn't exist on 2.6.15.7 so I can't really compare...

I wonder if any of the following is 'interesting'  :)

device sparrow:/exported/joe mounted on /u/joe with fstype nfs statvers=1.0
 opts: rw,vers=3,rsize=32768,wsize=32768,acregmin=3,acregmax=60,acdirmin=30,
       acdirmax=60,hard,intr,proto=udp,timeo=7,retrans=3
 age:    274
 caps:   caps=0x1,wtmult=4096,dtsize=4096,bsize=0,namelen=255
 sec:    flavor=1
 events: 175 77 3 3 14 15 108 4 0 7 0 4 0 1 2 0 0 14 0 1 4 0 0 0 0 
 bytes:  194733 11746 0 0 37748 15340 13 0 
 RPC iostats version: 1.0  p/v: 100003/3 (nfs)
 xprt:   udp 1023 0 74 74 0 74 0
 per-op statistics
 ... then follows the nfsstat numbers as far as I can see ...

Does the above tell you anything?

-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Bill Rugolsky Jr.]

On Fri, Mar 31, 2006 at 04:35:00PM +0200, Jakob Oestergaard wrote:
> I'm wondering... Can anyone else reproduce this problem?
> 
> Just to explain quickly:
>  Running nfsbench (on the NFS client) once with LEADING_EMPTY_SPACE set
>  to 0 and then once with the option set to 1.  If there's a big change
>  in wall-clock execution time, this indicates that the problem exists.
> 
> I'd be really interested in knowing whether I'm the only one who sees
> this problem.
 
Jakob,

Your NFS setup is specific to your system.  Have you considered trying this
over loopback to narrow down the variables?  If you see similar getattr/write
behavior over loopback, it will make it easy for everyone else to test.

	-Bill

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Fri, Mar 31, 2006 at 09:49:51AM -0500, Bill Rugolsky Jr. wrote:
...
> Jakob,
> 
> Your NFS setup is specific to your system. 

True. But it is my impression that this is a problem isolated on the
client side (am I wrong?)

> Have you considered trying this
> over loopback to narrow down the variables?

Do you mean NFS exporting a local filesystem, NFS mounting it again on
the local host?   Or do you mean something with loopback mounts?

(Sorry if that's a stupid question, but I just need to know what you
mean :)

> If you see similar getattr/write
> behavior over loopback, it will make it easy for everyone else to test.

True.

I've currently tried compiling the kernel with three different gcc
versions to see if that made any change. It didn't. And I tried SMP
versus UP, again no difference. I'm just about to try HIGHMEM vs no
HIGHMEM...

Thanks for your suggestion - I will try it, if you tell me precisely
what you mean in the above  :)

-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Bill Rugolsky Jr.]

On Fri, Mar 31, 2006 at 04:57:26PM +0200, Jakob Oestergaard wrote:
> True. But it is my impression that this is a problem isolated on the
> client side (am I wrong?)

That seems to be the case.

> Do you mean NFS exporting a local filesystem, NFS mounting it again on
> the local host?   Or do you mean something with loopback mounts?
 
The former; just export a local directory to 127.0.0.1 in /etc/exports,
then mount it on /mnt.

> Thanks for your suggestion - I will try it, if you tell me precisely
> what you mean in the above  :)

I'll give it a try here.

	-Bill

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Fri, Mar 31, 2006 at 10:04:53AM -0500, Bill Rugolsky Jr. wrote:
> On Fri, Mar 31, 2006 at 04:57:26PM +0200, Jakob Oestergaard wrote:
> > True. But it is my impression that this is a problem isolated on the
> > client side (am I wrong?)
> 
> That seems to be the case.

Hmm...

> 
> > Do you mean NFS exporting a local filesystem, NFS mounting it again on
> > the local host?   Or do you mean something with loopback mounts?
>  
> The former; just export a local directory to 127.0.0.1 in /etc/exports,
> then mount it on /mnt.

With a local export/mount I see very few reads running the test  :/

Could this be a timing issue?  That pages in the cache are invalidated
after maybe less than 100 microseconds which is roughly the round-trip
time on the gigabit network connecting my server and client?

/proc/self/mountstats on the two filesystems look identical to me:

device sparrow:/exported/joe mounted on /u/joe with fstype nfs statvers=1.0
opts: rw,vers=3,rsize=32768,wsize=32768,acregmin=3,acregmax=60,
      acdirmin=30,acdirmax=60,hard,intr,proto=udp,timeo=7,retrans=3

device localhost:/home/test mounted on /mnt with fstype nfs statvers=1.0
opts: rw,vers=3,rsize=32768,wsize=32768,acregmin=3,acregmax=60,
      acdirmin=30,acdirmax=60,hard,intr,proto=udp,timeo=7,retrans=3


-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Bill Rugolsky Jr.]

On Fri, Mar 31, 2006 at 05:24:01PM +0200, Jakob Oestergaard wrote:
> With a local export/mount I see very few reads running the test  :/
> 
> Could this be a timing issue?  That pages in the cache are invalidated
> after maybe less than 100 microseconds which is roughly the round-trip
> time on the gigabit network connecting my server and client?

I don't see any problem over loopback either.  Perhaps I'll try increasing
delays with tc netem.  That will have to wait until later.

In any case, I tried on our NetApp FAS-250, and I see the same problem:
Mount options are:
netapp:/home /nfs/netapp/home nfs rw,v3,rsize=32768,wsize=32768,hard,intr,tcp,lock,addr=netapp 0 0

I don't have stat zeroing, so I rolled some scripts.   Sorry, these are
not vanilla kernels; if you don't nail this today, I'll try and find time
for git bisection over the weekend.

	-Bill

== 

My production FC1 2.6.10 kernel:

ti64: rm -f testfile ; uname -a ; ( /tmp/nfsstat.sh ; time /tmp/nfsbench ; /tmp/nfsstat.sh ) | /tmp/nfsstatdiff.awk 
Linux ti64 2.6.10-0.ti.4.fc1smp #1 SMP Wed Jan 19 12:15:34 EST 2005 i686 athlon i386 GNU/Linux

real    0m0.215s
user    0m0.000s
sys     0m0.093s
null: 0 getattr: 2 setattr: 0 lookup: 1 access: 0 readlink: 0 read: 1 write: 321 create: 1 mkdir: 0 symlink: 0 mknod: 0 remove: 0 rmdir: 0 rename: 0 link: 0 readdir: 0 readdirplus: 0 fsstat: 0 fsinfo: 0 pathconf: 0 commit: 0 

===

DaveJ's FC4 2.6.15.3-based errata kernel, mounted:

ti63: rm -f testfile ; uname -a ; ( /tmp/nfsstat.sh ; time /tmp/nfsbench ; /tmp/nfsstat.sh ) | /tmp/nfsstatdiff.awk 
Linux ti63 2.6.15-1.1831_FC4smp #1 SMP Tue Feb 7 13:48:31 EST 2006 i686 athlon i386 GNU/Linux

real    0m3.242s
user    0m0.000s
sys     0m0.360s
null: 0 getattr: 2561 setattr: 0 lookup: 1 access: 1 readlink: 0 read: 7040 write: 3200 create: 1 mkdir: 0 symlink: 0 mknod: 0 remove: 0 rmdir: 0 rename: 0 link: 0 readdir: 0 readdirplus: 0 fsstat: 0 fsinfo: 0 pathconf: 0 commit: 0 


	-Bill

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Fri, Mar 31, 2006 at 11:35:42AM -0500, Bill Rugolsky Jr. wrote:
...
> I don't see any problem over loopback either.  Perhaps I'll try increasing
> delays with tc netem.  That will have to wait until later.

That would be my next try.

> In any case, I tried on our NetApp FAS-250, and I see the same problem:
> Mount options are:
> netapp:/home /nfs/netapp/home nfs rw,v3,rsize=32768,wsize=32768,hard,intr,tcp,lock,addr=netapp 0 0

Thanks a lot!  Ah, I thought the computers only hated me  ;)

-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust]

On Fri, 2006-03-31 at 16:35 +0200, Jakob Oestergaard wrote:
> On Fri, Mar 31, 2006 at 09:21:29AM -0500, Trond Myklebust wrote:
> ...
> > 
> > Hmm... Nothing obvious.
> 
> Ok.
> 
> I'm wondering... Can anyone else reproduce this problem?
> 
> Just to explain quickly:
>  Running nfsbench (on the NFS client) once with LEADING_EMPTY_SPACE set
>  to 0 and then once with the option set to 1.  If there's a big change
>  in wall-clock execution time, this indicates that the problem exists.
> 
> I'd be really interested in knowing whether I'm the only one who sees
> this problem.
> 
> > Try catting /proc/self/mountstats and see if the entry for your NFS
> > mount shows anything interesting.
> 
> mountstats doesn't exist on 2.6.15.7 so I can't really compare...
> 
> I wonder if any of the following is 'interesting'  :)
> 
> device sparrow:/exported/joe mounted on /u/joe with fstype nfs statvers=1.0
>  opts: rw,vers=3,rsize=32768,wsize=32768,acregmin=3,acregmax=60,acdirmin=30,
>        acdirmax=60,hard,intr,proto=udp,timeo=7,retrans=3
>  age:    274
>  caps:   caps=0x1,wtmult=4096,dtsize=4096,bsize=0,namelen=255
>  sec:    flavor=1
>  events: 175 77 3 3 14 15 108 4 0 7 0 4 0 1 2 0 0 14 0 1 4 0 0 0 0 
>  bytes:  194733 11746 0 0 37748 15340 13 0 
>  RPC iostats version: 1.0  p/v: 100003/3 (nfs)
>  xprt:   udp 1023 0 74 74 0 74 0
>  per-op statistics
>  ... then follows the nfsstat numbers as far as I can see ...

No. They are the nfsstat numbers + timing information. I was interested
in seeing if the latter can show up something.

Cheers
  Trond

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Fri, Mar 31, 2006 at 10:55:20AM -0500, Trond Myklebust wrote:
...
> No. They are the nfsstat numbers + timing information. I was interested
> in seeing if the latter can show up something.

Oops sorry. Here you go:

device sparrow:/exported/joe mounted on /u/joe with
fstype nfs statvers=1.0
 opts: rw,vers=3,rsize=32768,wsize=32768,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,intr,proto=udp,timeo=7,retrans=3
 age:    2526
 caps: caps=0x1,wtmult=4096,dtsize=4096,bsize=0,namelen=255
 sec:    flavor=1
 events: 854 472 4017 4651 56 74 753 8357 3987 3511 0 4016 20 19 20 2 0 56 0 5 4371 0 0 0 0 
 bytes:  131161418 19867448 0 0 114028223 19862777 31282 4349 
 RPC iostats version: 1.0 p/v: 100003/3 (nfs)
 xprt:   udp 1023 0 560 560 0 1832 176
 per-op statistics
  NULL: 1 1 0 0 24 0 0 0
  GETATTR: 115 115 0 0 12880 0 16 24
  SETATTR: 0 0 0 0 0 0 0 0
  LOOKUP: 149 149 0 0 21540 4 28 32
  ACCESS: 97 97 0 0 11640 0 8 8
  READLINK: 2 2 0 0 272 0 0 0
  READ: 193 193 0 0 5977864 176 424 600
  ... all zeroes ...
  READDIRPLUS: 2 2 0 0 6544 0 0 0
  FSSTAT: 0 0 0 0 0 0 0 0
  FSINFO: 1 1 0 0 80 0 4 4
  PATHCONF: 0 0 0 0 0 0 0 0
  COMMIT: 0 0 0 0 0 0 0 0

I'll be offline for a few hours - will check back later tonight.

-- 
/ jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

I've run a git bisect from 2.6.14 to 2.6.15 to find what broke the NFS
client.

It seems to be the GIT patch: 24aa1fe6779eaddb3e0b1b802585dcf6faf9cc44
that introduces the problem.

Trond, it took me a lot of tries with GIT to narrow this down, because
the problem does not show up consistently. Some times nfsbench would
complete very quickly, but then a second (or third or ...) run would run
slow.

Could I ask you to try: 
 for i in `seq 1 100`; do time ./nfsbench; done
or something like that?

I'm looking through the patch, but to be honest the NFS cache internals
isn't really something I understand... :)

-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust]

On Mon, 2006-04-03 at 17:26 +0200, Jakob Oestergaard wrote:
> I've run a git bisect from 2.6.14 to 2.6.15 to find what broke the NFS
> client.
> 
> It seems to be the GIT patch: 24aa1fe6779eaddb3e0b1b802585dcf6faf9cc44
> that introduces the problem.
> 
> Trond, it took me a lot of tries with GIT to narrow this down, because
> the problem does not show up consistently. Some times nfsbench would
> complete very quickly, but then a second (or third or ...) run would run
> slow.
> 
> Could I ask you to try: 
>  for i in `seq 1 100`; do time ./nfsbench; done
> or something like that?

OK. Thanks for helping narrow this down! I'm travelling right now, so I
can't promise that I'll be able to run any tests until tomorrow. I'll
have a look, though.

Cheers,
  Trond

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Mon, Apr 03, 2006 at 11:41:40AM -0400, Trond Myklebust wrote:
..
> > Could I ask you to try: 
> >  for i in `seq 1 100`; do time ./nfsbench; done
> > or something like that?
> 
> OK. Thanks for helping narrow this down! I'm travelling right now, so I
> can't promise that I'll be able to run any tests until tomorrow. I'll
> have a look, though.

Thanks a lot, and no problem :)

-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Mon, Apr 03, 2006 at 05:45:19PM +0200, Jakob Oestergaard wrote:
> On Mon, Apr 03, 2006 at 11:41:40AM -0400, Trond Myklebust wrote:
> ..
> > OK. Thanks for helping narrow this down! I'm travelling right now, so I
> > can't promise that I'll be able to run any tests until tomorrow. I'll
> > have a look, though.

Just another datapoint;

2.6.16.1 with that one patch reverted does no longer exhibit the
problem.

-- 

 / jakob

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Trond Myklebust]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

On Tue, 2006-04-04 at 11:22 +0200, Jakob Oestergaard wrote:
> On Mon, Apr 03, 2006 at 05:45:19PM +0200, Jakob Oestergaard wrote:
> > On Mon, Apr 03, 2006 at 11:41:40AM -0400, Trond Myklebust wrote:
> > ..
> > > OK. Thanks for helping narrow this down! I'm travelling right now, so I
> > > can't promise that I'll be able to run any tests until tomorrow. I'll
> > > have a look, though.
> 
> Just another datapoint;
> 
> 2.6.16.1 with that one patch reverted does no longer exhibit the
> problem.

Sorry it has taken such a long time to get round to looking into this
problem. Can you see if the attached patch helps in any way?

Cheers,
  Trond

[-- Attachment #2: linux-2.6.17-032-reduce_number_invalidations.dif --]
[-- Type: message/rfc822, Size: 8286 bytes --]

From: Trond Myklebust <Trond.Myklebust@netapp.com>
Subject: No Subject
Date: Mon, 24 Apr 2006 18:03:23 -0400
Message-ID: <1145916203.10974.56.camel@lade.trondhjem.org>

Separate out the function of revalidating the inode metadata, and
revalidating the mapping. The former may be called by lookup(),
and only really needs to check that permissions, ctime, etc haven't changed
whereas the latter needs only done when we want to read data from the page
cache, and may need to sync and then invalidate the mapping.

Fix up a bug in the handling of NFS_INO_REVAL_PAGECACHE: make sure that
nfs_update_inode() clears it when we're sure we're not racing with other
updates.

Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/dir.c           |    2 +-
 fs/nfs/file.c          |   24 +++---------------------
 fs/nfs/inode.c         |   21 ++++++++++++---------
 fs/nfs/symlink.c       |    2 +-
 include/linux/nfs_fs.h |    2 +-
 5 files changed, 18 insertions(+), 33 deletions(-)

diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
index 2ab69cf..3ddda6f 100644
--- a/fs/nfs/dir.c
+++ b/fs/nfs/dir.c
@@ -528,7 +528,7 @@ static int nfs_readdir(struct file *filp
 
 	lock_kernel();
 
-	res = nfs_revalidate_inode(NFS_SERVER(inode), inode);
+	res = nfs_revalidate_mapping(inode, filp->f_mapping);
 	if (res < 0) {
 		unlock_kernel();
 		return res;
diff --git a/fs/nfs/file.c b/fs/nfs/file.c
index fade02c..6315407 100644
--- a/fs/nfs/file.c
+++ b/fs/nfs/file.c
@@ -127,23 +127,6 @@ nfs_file_release(struct inode *inode, st
 }
 
 /**
- * nfs_revalidate_file - Revalidate the page cache & related metadata
- * @inode - pointer to inode struct
- * @file - pointer to file
- */
-static int nfs_revalidate_file(struct inode *inode, struct file *filp)
-{
-	struct nfs_inode *nfsi = NFS_I(inode);
-	int retval = 0;
-
-	if ((nfsi->cache_validity & (NFS_INO_REVAL_PAGECACHE|NFS_INO_INVALID_ATTR))
-			|| nfs_attribute_timeout(inode))
-		retval = __nfs_revalidate_inode(NFS_SERVER(inode), inode);
-	nfs_revalidate_mapping(inode, filp->f_mapping);
-	return 0;
-}
-
-/**
  * nfs_revalidate_size - Revalidate the file size
  * @inode - pointer to inode struct
  * @file - pointer to struct file
@@ -228,7 +211,7 @@ #endif
 		dentry->d_parent->d_name.name, dentry->d_name.name,
 		(unsigned long) count, (unsigned long) pos);
 
-	result = nfs_revalidate_file(inode, iocb->ki_filp);
+	result = nfs_revalidate_mapping(inode, iocb->ki_filp->f_mapping);
 	nfs_add_stats(inode, NFSIOS_NORMALREADBYTES, count);
 	if (!result)
 		result = generic_file_aio_read(iocb, buf, count, pos);
@@ -247,7 +230,7 @@ nfs_file_sendfile(struct file *filp, lof
 		dentry->d_parent->d_name.name, dentry->d_name.name,
 		(unsigned long) count, (unsigned long long) *ppos);
 
-	res = nfs_revalidate_file(inode, filp);
+	res = nfs_revalidate_mapping(inode, filp->f_mapping);
 	if (!res)
 		res = generic_file_sendfile(filp, ppos, count, actor, target);
 	return res;
@@ -263,7 +246,7 @@ nfs_file_mmap(struct file * file, struct
 	dfprintk(VFS, "nfs: mmap(%s/%s)\n",
 		dentry->d_parent->d_name.name, dentry->d_name.name);
 
-	status = nfs_revalidate_file(inode, file);
+	status = nfs_revalidate_mapping(inode, file->f_mapping);
 	if (!status)
 		status = generic_file_mmap(file, vma);
 	return status;
@@ -373,7 +356,6 @@ #endif
 		if (result)
 			goto out;
 	}
-	nfs_revalidate_mapping(inode, iocb->ki_filp->f_mapping);
 
 	result = count;
 	if (!count)
diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index 9a344be..f041b76 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -1271,7 +1271,7 @@ __nfs_revalidate_inode(struct nfs_server
 		status = -ESTALE;
 		/* Do we trust the cached ESTALE? */
 		if (NFS_ATTRTIMEO(inode) != 0) {
-			if (nfsi->cache_validity & (NFS_INO_INVALID_ATTR|NFS_INO_INVALID_DATA|NFS_INO_INVALID_ATIME)) {
+			if (nfsi->cache_validity & (NFS_INO_INVALID_ATTR|NFS_INO_INVALID_ATIME)) {
 				/* no */
 			} else
 				goto out;
@@ -1302,8 +1302,6 @@ __nfs_revalidate_inode(struct nfs_server
 	}
 	spin_unlock(&inode->i_lock);
 
-	nfs_revalidate_mapping(inode, inode->i_mapping);
-
 	if (nfsi->cache_validity & NFS_INO_INVALID_ACL)
 		nfs_zap_acl_cache(inode);
 
@@ -1338,7 +1336,7 @@ int nfs_attribute_timeout(struct inode *
 int nfs_revalidate_inode(struct nfs_server *server, struct inode *inode)
 {
 	nfs_inc_stats(inode, NFSIOS_INODEREVALIDATE);
-	if (!(NFS_I(inode)->cache_validity & (NFS_INO_INVALID_ATTR|NFS_INO_INVALID_DATA))
+	if (!(NFS_I(inode)->cache_validity & NFS_INO_INVALID_ATTR)
 			&& !nfs_attribute_timeout(inode))
 		return NFS_STALE(inode) ? -ESTALE : 0;
 	return __nfs_revalidate_inode(server, inode);
@@ -1349,15 +1347,19 @@ int nfs_revalidate_inode(struct nfs_serv
  * @inode - pointer to host inode
  * @mapping - pointer to mapping
  */
-void nfs_revalidate_mapping(struct inode *inode, struct address_space *mapping)
+int nfs_revalidate_mapping(struct inode *inode, struct address_space *mapping)
 {
 	struct nfs_inode *nfsi = NFS_I(inode);
+	int ret = 0;
+
+	if ((nfsi->cache_validity & NFS_INO_REVAL_PAGECACHE)
+			|| nfs_attribute_timeout(inode))
+		ret = __nfs_revalidate_inode(NFS_SERVER(inode), inode);
 
 	if (nfsi->cache_validity & NFS_INO_INVALID_DATA) {
 		nfs_inc_stats(inode, NFSIOS_DATAINVALIDATE);
 		if (S_ISREG(inode->i_mode))
 			nfs_sync_mapping(mapping);
-		invalidate_inode_pages2(mapping);
 
 		spin_lock(&inode->i_lock);
 		nfsi->cache_validity &= ~NFS_INO_INVALID_DATA;
@@ -1367,11 +1369,13 @@ void nfs_revalidate_mapping(struct inode
 			nfsi->cache_change_attribute = jiffies;
 		}
 		spin_unlock(&inode->i_lock);
+		invalidate_inode_pages2(mapping);
 
 		dfprintk(PAGECACHE, "NFS: (%s/%Ld) data cache invalidated\n",
 				inode->i_sb->s_id,
 				(long long)NFS_FILEID(inode));
 	}
+	return ret;
 }
 
 /**
@@ -1510,7 +1514,6 @@ int nfs_refresh_inode(struct inode *inod
 	if ((fattr->valid & NFS_ATTR_FATTR) == 0)
 		return 0;
 	spin_lock(&inode->i_lock);
-	nfsi->cache_validity &= ~NFS_INO_REVAL_PAGECACHE;
 	if (time_after(fattr->time_start, nfsi->last_updated))
 		status = nfs_update_inode(inode, fattr);
 	else
@@ -1535,7 +1538,7 @@ int nfs_post_op_update_inode(struct inod
 
 	spin_lock(&inode->i_lock);
 	if (unlikely((fattr->valid & NFS_ATTR_FATTR) == 0)) {
-		nfsi->cache_validity |= NFS_INO_INVALID_ATTR | NFS_INO_INVALID_ACCESS;
+		nfsi->cache_validity |= NFS_INO_INVALID_ACCESS|NFS_INO_INVALID_ATTR|NFS_INO_REVAL_PAGECACHE;
 		goto out;
 	}
 	status = nfs_update_inode(inode, fattr);
@@ -1592,7 +1595,7 @@ static int nfs_update_inode(struct inode
 	/* Are we racing with known updates of the metadata on the server? */
 	data_stable = nfs_verify_change_attribute(inode, fattr->time_start);
 	if (data_stable)
-		nfsi->cache_validity &= ~(NFS_INO_INVALID_ATTR|NFS_INO_INVALID_ATIME);
+		nfsi->cache_validity &= ~(NFS_INO_INVALID_ATTR|NFS_INO_REVAL_PAGECACHE|NFS_INO_INVALID_ATIME);
 
 	/* Do atomic weak cache consistency updates */
 	nfs_wcc_update_inode(inode, fattr);
diff --git a/fs/nfs/symlink.c b/fs/nfs/symlink.c
index 18dc95b..636c479 100644
--- a/fs/nfs/symlink.c
+++ b/fs/nfs/symlink.c
@@ -52,7 +52,7 @@ static void *nfs_follow_link(struct dent
 {
 	struct inode *inode = dentry->d_inode;
 	struct page *page;
-	void *err = ERR_PTR(nfs_revalidate_inode(NFS_SERVER(inode), inode));
+	void *err = ERR_PTR(nfs_revalidate_mapping(inode, inode->i_mapping));
 	if (err)
 		goto read_failed;
 	page = read_cache_page(&inode->i_data, 0,
diff --git a/include/linux/nfs_fs.h b/include/linux/nfs_fs.h
index 363eced..2250144 100644
--- a/include/linux/nfs_fs.h
+++ b/include/linux/nfs_fs.h
@@ -298,7 +298,7 @@ extern int nfs_release(struct inode *, s
 extern int nfs_attribute_timeout(struct inode *inode);
 extern int nfs_revalidate_inode(struct nfs_server *server, struct inode *inode);
 extern int __nfs_revalidate_inode(struct nfs_server *, struct inode *);
-extern void nfs_revalidate_mapping(struct inode *inode, struct address_space *mapping);
+extern int nfs_revalidate_mapping(struct inode *inode, struct address_space *mapping);
 extern int nfs_setattr(struct dentry *, struct iattr *);
 extern void nfs_setattr_update_inode(struct inode *inode, struct iattr *attr);
 extern void nfs_begin_attr_update(struct inode *);

Re: NFS client (10x) performance regression 2.6.14.7 -> 2.6.15

[Jakob Oestergaard]

On Mon, Apr 24, 2006 at 06:03:23PM -0400, Trond Myklebust wrote:
...
> 
> Sorry it has taken such a long time to get round to looking into this
> problem. Can you see if the attached patch helps in any way?

No problem.

I will test the patch 'as soon as possible', hopefully today -
unfortunately the test machine is now in pretty heavy use but I should
be able to reproduce the problem and test the patch on another system.

Will let you know.

Thanks,

-- 

 / jakob