Re: Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Jesper Krogh <jesper@krogh.cc>]

Eric Dumazet wrote:
> Jesper Krogh a écrit :
>> Hi.
>>
>> Back in the 2.6.25/26-days .. (around 9 months ago) I had som stuggles
>> getting both performance and stabillity out of a Sun Neptune 10Gbit NIC
>> over fibre. The NIU driver blew up on the system under load but Matheos
>> Worku send me an internal Sun driver (nxge) that performed fairly well.
>> It peaked out around 800-850 MB/s .. it put a fairly high load on the
>> host system and peaked with over 300.000 cs/s (both numbers measured
>> with dstat).
>>
>> Today I got around to tesing the NIU(2.6.27.20) driver again. Having
>> repeated the test I did last summer I couldn't get it to "blow up". But
>> instead of the ~500MB/s i got out of it last summer, it peaks at 940MB/s
>> now and the load on the host is nearly invisible (<2) .. cs rates less
>> than 10K mostly.
>>
>> I'm not using any kind of Jumbo frames in the setup.
>>
>> I'll keep it up on the niu-driver for now and report back if it
>> encounters any problems.
>>
>> In the test .. I do dd over NFS, default exports, default mount options
>> .. dd have "bs" set to either 1M or to 512 (to try to stress the NFS
>> server). I have tried to put cpu-load in the NFS-server while running
>> the NFS-server and pushing some data around on them memory subsystem
>> while doing it.
>>
>> This is just excellent..  (crossing fingers that it can beat the 180
>> days of uptime the nxge-driver got).
>>
>> Link to old struggles:
>> http://thread.gmane.org/gmane.linux.kernel/677545
>>
>> Jesper
> 
> Yes I remember this stuff. David added multi tx queue support in last July
> (for 2.6.27)

Can there be something about "fairness" changed? While producing the 
screenshot to you below I got this one on the server:
do_ypcall: clnt_call: RPC: Unable to send; errno = No buffer space available

So NIS wasn't able to get throgh..

> Could you post more information please, about load on individual cpus
>  for example ? (top snapshot with one line per cpu)

http://krogh.cc/~jesper/top.png

> Is NFS using TCP or UDP on your setup ?

TCP.

> An interesting test would be the reverse path (transmit from clients to
> this server)

Thats harder to setup and not be "disk IO-bound on the server". Perhaps 
exporting a RAM-disk and write into that?

> Also, testing 2.6.29 could be interesting, since UDP receive
> path doesnt need to use a global rwlock anymore.

I'll put that on my todo list..

-- 
Jesper