Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Jesper Krogh]

Hi.

Back in the 2.6.25/26-days .. (around 9 months ago) I had som stuggles 
getting both performance and stabillity out of a Sun Neptune 10Gbit NIC 
over fibre. The NIU driver blew up on the system under load but Matheos 
Worku send me an internal Sun driver (nxge) that performed fairly well. 
It peaked out around 800-850 MB/s .. it put a fairly high load on the
host system and peaked with over 300.000 cs/s (both numbers measured
with dstat).

Today I got around to tesing the NIU(2.6.27.20) driver again. Having
repeated the test I did last summer I couldn't get it to "blow up". But
instead of the ~500MB/s i got out of it last summer, it peaks at 940MB/s
now and the load on the host is nearly invisible (<2) .. cs rates less
than 10K mostly.

I'm not using any kind of Jumbo frames in the setup.

I'll keep it up on the niu-driver for now and report back if it
encounters any problems.

In the test .. I do dd over NFS, default exports, default mount options
.. dd have "bs" set to either 1M or to 512 (to try to stress the NFS
server). I have tried to put cpu-load in the NFS-server while running
the NFS-server and pushing some data around on them memory subsystem
while doing it.

This is just excellent..  (crossing fingers that it can beat the 180 
days of uptime the nxge-driver got).

Link to old struggles:
http://thread.gmane.org/gmane.linux.kernel/677545

Jesper
-- 
Jesper

Re: Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Eric Dumazet]

Jesper Krogh a écrit :
> Hi.
> 
> Back in the 2.6.25/26-days .. (around 9 months ago) I had som stuggles
> getting both performance and stabillity out of a Sun Neptune 10Gbit NIC
> over fibre. The NIU driver blew up on the system under load but Matheos
> Worku send me an internal Sun driver (nxge) that performed fairly well.
> It peaked out around 800-850 MB/s .. it put a fairly high load on the
> host system and peaked with over 300.000 cs/s (both numbers measured
> with dstat).
> 
> Today I got around to tesing the NIU(2.6.27.20) driver again. Having
> repeated the test I did last summer I couldn't get it to "blow up". But
> instead of the ~500MB/s i got out of it last summer, it peaks at 940MB/s
> now and the load on the host is nearly invisible (<2) .. cs rates less
> than 10K mostly.
> 
> I'm not using any kind of Jumbo frames in the setup.
> 
> I'll keep it up on the niu-driver for now and report back if it
> encounters any problems.
> 
> In the test .. I do dd over NFS, default exports, default mount options
> .. dd have "bs" set to either 1M or to 512 (to try to stress the NFS
> server). I have tried to put cpu-load in the NFS-server while running
> the NFS-server and pushing some data around on them memory subsystem
> while doing it.
> 
> This is just excellent..  (crossing fingers that it can beat the 180
> days of uptime the nxge-driver got).
> 
> Link to old struggles:
> http://thread.gmane.org/gmane.linux.kernel/677545
> 
> Jesper

Yes I remember this stuff. David added multi tx queue support in last July
(for 2.6.27)

Could you post more information please, about load on individual cpus
 for example ? (top snapshot with one line per cpu)

Is NFS using TCP or UDP on your setup ?

An interesting test would be the reverse path (transmit from clients to
this server)

Also, testing 2.6.29 could be interesting, since UDP receive
path doesnt need to use a global rwlock anymore.

Re: Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Jesper Krogh]

Eric Dumazet wrote:
> Jesper Krogh a écrit :
>> Hi.
>>
>> Back in the 2.6.25/26-days .. (around 9 months ago) I had som stuggles
>> getting both performance and stabillity out of a Sun Neptune 10Gbit NIC
>> over fibre. The NIU driver blew up on the system under load but Matheos
>> Worku send me an internal Sun driver (nxge) that performed fairly well.
>> It peaked out around 800-850 MB/s .. it put a fairly high load on the
>> host system and peaked with over 300.000 cs/s (both numbers measured
>> with dstat).
>>
>> Today I got around to tesing the NIU(2.6.27.20) driver again. Having
>> repeated the test I did last summer I couldn't get it to "blow up". But
>> instead of the ~500MB/s i got out of it last summer, it peaks at 940MB/s
>> now and the load on the host is nearly invisible (<2) .. cs rates less
>> than 10K mostly.
>>
>> I'm not using any kind of Jumbo frames in the setup.
>>
>> I'll keep it up on the niu-driver for now and report back if it
>> encounters any problems.
>>
>> In the test .. I do dd over NFS, default exports, default mount options
>> .. dd have "bs" set to either 1M or to 512 (to try to stress the NFS
>> server). I have tried to put cpu-load in the NFS-server while running
>> the NFS-server and pushing some data around on them memory subsystem
>> while doing it.
>>
>> This is just excellent..  (crossing fingers that it can beat the 180
>> days of uptime the nxge-driver got).
>>
>> Link to old struggles:
>> http://thread.gmane.org/gmane.linux.kernel/677545
>>
>> Jesper
> 
> Yes I remember this stuff. David added multi tx queue support in last July
> (for 2.6.27)

Can there be something about "fairness" changed? While producing the 
screenshot to you below I got this one on the server:
do_ypcall: clnt_call: RPC: Unable to send; errno = No buffer space available

So NIS wasn't able to get throgh..

> Could you post more information please, about load on individual cpus
>  for example ? (top snapshot with one line per cpu)

http://krogh.cc/~jesper/top.png

> Is NFS using TCP or UDP on your setup ?

TCP.

> An interesting test would be the reverse path (transmit from clients to
> this server)

Thats harder to setup and not be "disk IO-bound on the server". Perhaps 
exporting a RAM-disk and write into that?

> Also, testing 2.6.29 could be interesting, since UDP receive
> path doesnt need to use a global rwlock anymore.

I'll put that on my todo list..

-- 
Jesper

Re: Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Jesper Krogh]

Eric Dumazet wrote:
> Also, testing 2.6.29 could be interesting, since UDP receive
> path doesnt need to use a global rwlock anymore.

... That gained yet another 100MB/s.. now peaking out around 1050-1060MB/s.

Jesper
-- 
Jesper

Re: Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Jesper Krogh]

Jesper Krogh wrote:
> Eric Dumazet wrote:
>> Also, testing 2.6.29 could be interesting, since UDP receive
>> path doesnt need to use a global rwlock anymore.
> 
> ... That gained yet another 100MB/s.. now peaking out around 1050-1060MB/s.

Am I wrong thinking that theoretical max should be around 1150-1200MB/s ?

-- 
Jesper

Re: Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Eric Dumazet]

Jesper Krogh a écrit :
> Eric Dumazet wrote:
>> Also, testing 2.6.29 could be interesting, since UDP receive
>> path doesnt need to use a global rwlock anymore.
> 
> ... That gained yet another 100MB/s.. now peaking out around 1050-1060MB/s.
> 
> Jesper

Wow ! Nice to ear !

Re: Status update on Sun Neptune 10Gbit fibre using the NIU-driver.

[Eric Dumazet]

Jesper Krogh a écrit :
> Jesper Krogh wrote:
>> Eric Dumazet wrote:
>>> Also, testing 2.6.29 could be interesting, since UDP receive
>>> path doesnt need to use a global rwlock anymore.
>>
>> ... That gained yet another 100MB/s.. now peaking out around
>> 1050-1060MB/s.
> 
> Am I wrong thinking that theoretical max should be around 1150-1200MB/s ?
> 

I dont know, what would be the current bottleneck ?