Re: Performance 2.4.8 is worse than 2.4.x<8 (SPEC NFS results show this)

[Hans Reiser <reiser@namesys.com>]

We are looking into this.  Elena and Chris, please advise as to whether the
slowdown is ReiserFS code added recently or is due to layers not ReiserFS.

Hans


"HABBINGA,ERIK (HP-Loveland,ex1)" wrote:
> 
> Here are some SPEC SFS NFS testing (http://www.spec.org/osg/sfs97) results
> I've been doing over the past few weeks that shows NFS performance degrading
> since the 2.4.5pre1 kernel.  I've kept the hardware constant, only changing
> the kernel.  I'm prevented by management from releasing our top numbers, but
> have given our results normalized to the 2.4.5pre1 kernel.  I've also shown
> the results from the first three SPEC runs to show the response time trend.
> 
> Normally, response time should start out very low, increasing slowly until
> the maximum load of the system under test is reached.  Starting with
> 2.4.8pre8, the response time starts very high, and then decreases.  Very
> bizarre behaviour.
> 
> The spec results consist of the following data (only the first three numbers
> are significant for this discussion)
> - load.  The load the SPEC prime client will try to get out of the system
> under test.  Measured in I/O's per second (IOPS).
> - throughput.  The load seen from the system under test.  Measured in IOPS
> - response time.  Measured in milliseconds
> - total operations
> - elapsed time.  Measured in seconds
> - NFS version. 2 or 3
> - Protocol. UDP (U) or TCP (T)
> - file set size in megabytes
> - number of clients
> - number of SPEC SFS processes
> - biod reads
> - biod writes
> - SPEC SFS version
> 
> The 2.4.8pre4 and 2.4.8 tests were invalid.  Too many (> 1%) of the RPC
> calls between the SPEC prime client and the system under test failed.  This
> is not a good thing.
> 
> I'm willing to try out any ideas on this system to help find and fix the
> performance degradation.
> 
> Erik Habbinga
> Hewlett Packard
> 
> Hardware:
> 4 processors, 4GB ram
> 45 fibre channel drives, set up in hardware RAID 0/1
> 2 direct Gigabit Ethernet connections between SPEC SFS prime client and
> system under test
> reiserfs
> all NFS filesystems exported with sync,no_wdelay to insure O_SYNC writes to
> storage
> NFS v3 UDP
> 
> Results:
> 2.4.5pre1
>             500     497     0.8   149116  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1004     1.0   300240  299 3 U   10141248   1 48  2  2
> 2.0
>            1500    1501     1.0   448807  299 3 U   15210624   1 48  2  2
> 2.0
> peak IOPS: 100% of 2.4.5pre1
> 
> 2.4.5pre2
>             500     497     1.0   149195  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1005     1.2   300449  299 3 U   10141248   1 48  2  2
> 2.0
>            1500    1502     1.2   449057  299 3 U   15210624   1 48  2  2
> 2.0
> peak IOPS: 91% of 2.4.5pre1
> 
> 2.4.5pre3
>             500     497     1.0   149095  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1004     1.1   300135  299 3 U   10141248   1 48  2  2
> 2.0
>            1500    1502     1.2   449069  299 3 U   15210624   1 48  2  2
> 2.0
> peak IOPS: 91% of 2.4.5pre1
> 
> 2.4.5pre4
>    wouldn't run (stale NFS file handle error)
> 
> 2.4.5pre5
>    wouldn't run (stale NFS file handle error)
> 
> 2.4.5pre6
>    wouldn't run (stale NFS file handle error)
> 
> 2.4.7
>             500     497     1.2   149206  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1005     1.5   300503  299 3 U   10141248   1 48  2  2
> 2.0
>            1500    1502     1.3   449232  299 3 U   15210624   1 48  2  2
> 2.0
> peak IOPS: 65% of 2.4.5pre1
> 
> 2.4.8pre1
>    wouldn't run
> 
> 2.4.8pre4
>             500     497     1.1   149180  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1002     1.2   299465  299 3 U   10141248   1 48  2  2
> 2.0
>            1500    1502     1.3   449190  299 3 U   15210624   1 48  2  2
> 2.0
> INVALID
> peak IOPS: 54% of 2.4.5pre1
> 
> 2.4.8pre6
>             500     497     1.1   149168  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1004     1.3   300246  299 3 U   10141248   1 48  2  2
> 2.0
>            1500    1502     1.3   449135  299 3 U   15210624   1 48  2  2
> 2.0
> peak IOPS 55% of 2.4.5pre1
> 
> 2.4.8pre7
>             500     498     1.5   149367  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1006     2.2   301829  300 3 U   10141248   1 48  2  2
> 2.0
>            1500    1502     2.2   449244  299 3 U   15210624   1 48  2  2
> 2.0
> peak IOPS: 58% of 2.4.5pre1
> 
> 2.4.8pre8
>             500     597     8.3   179030  300 3 U    5070624   1 48  2  2
> 2.0
>            1000    1019     6.5   304614  299 3 U   10141248   1 48  2  2
> 2.0
>            1500    1538     4.5   461335  300 3 U   15210624   1 48  2  2
> 2.0
> peak IOPS: 48% of 2.4.5pre1
> 
> 2.4.8
>             500     607     7.1   181981  300 3 U    5070624   1 48  2  2
> 2.0
>            1000     997     7.0   299243  300 3 U   10141248   1 48  2  2
> 2.0
>            1500    1497     2.9   447475  299 3 U   15210624   1 48  2  2
> 2.0
> INVALID
> peak IOPS: 45% of 2.4.5pre1
> 
> 2.4.9pre2
>    wouldn't run (NFS readdir errors)
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/