[ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Alex Aizman]

This is our 2nd attempt to submit "xge", the experimental driver for
Neterion, Inc (formerly S2io, Inc) family of 10GbE adapters. 

The "first attempt", along with the still relevant description of features
and motivations, can be looked up at
http://oss.sgi.com/archives/netdev/2005-03/msg00854.html. Note that the xge
driver supports the new 10GbE PCI-X 266MHz adapter.

Changes
======

Removed HAL layering, worked on the coding style to make it compliant with
the kernel style, re-organized the code to accommodate community feedback.

Download
======

A single gzipped patch (filename xge-v.1.1.2870-2.6.11.patch.gz, cksum
2732698831) can be downloaded from: ftp://linuxsrc:opensource@ns1.s2io.com

Signed-off-by: Dmitry Yusupov <dima@neterion.com>
Signed-off-by: Raghavendra Koushik <raghavendra.koushik@neterion.com>
Signed-off-by: Alex Aizman <alex@neterion.com>

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Jeff Garzik]

Alex Aizman wrote:
> This is our 2nd attempt to submit "xge", the experimental driver for
> Neterion, Inc (formerly S2io, Inc) family of 10GbE adapters. 

Can you send a link to the latest version of this driver, for review?

	Jeff

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

Hi Jeff,

Thanks for remembering about the submission. My assumption was that the
submission did not get a consensus and died quietly :-), so we stopped
maintenance on the experimental driver several weeks ago and pulled it
from the ftp site.

I guess the original idea behind developing the "experimental" driver
was to eventually use it as a replacement for the "s2io" driver in the
kernel, mainly for the maintenance reasons - since "experimental" driver
shares hardware-oriented code with other Neterion drivers, our team
could run mature network test suites in other Operating Systems and
indirectly find/fix issues in Linux driver in a very efficient fashion.

This benefit seemed to be outweighed by other concerns that some of you
guys raised on the list. 
In a long run (as 10GbE Xframe cards get shipped in volumes and the
patches start coming in from the community), these concerns may be
valid, so we have decided to stay with "s2io" driver as our production
code.

Leonid  

> -----Original Message-----
> From: netdev-bounce@oss.sgi.com 
> [mailto:netdev-bounce@oss.sgi.com] On Behalf Of Jeff Garzik
> Sent: Thursday, August 11, 2005 10:48 AM
> To: alex@neterion.com
> Cc: netdev@oss.sgi.com; davem@davemloft.net; ak@muc.de
> Subject: Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 
> 10GbE Adapters
> 
> Alex Aizman wrote:
> > This is our 2nd attempt to submit "xge", the experimental 
> driver for 
> > Neterion, Inc (formerly S2io, Inc) family of 10GbE adapters.
> 
> Can you send a link to the latest version of this driver, for review?
> 
> 	Jeff
> 
> 
> 
> 
> 

RE: Intel and TOE in the news

[jamal]

On Mon, 2005-02-21 at 11:52, Leonid Grossman wrote:
>  
> WRT to the burst of packets related to the same flow - we are hoping to be
> able to collapse the burst into a single oversized frame and pass it to the
> stack, this way no or very minimal changes to the stack will be needed. 
> There is enough intelligence on the NIC to do that efficiently, we just need
> to try and see how well this works.

Indeed, would be nice to see what you come up with. I think there may be
value in sending one huge chunk packet which itself is actually a
collection of several independet packets when you have a huge amount of
small packets. The benefit being you amortize the cost of DMA setup.
But then you may need to be able to break them down in the driver; is
this what you are talking about?

cheers,
jamal

[ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Alex Aizman]

This is to announce "xge", an experimental 10GbE driver for Neterion, Inc
(formerly S2io, Inc) family of adapters.

Motivation
==========
The production ("s2io") driver for Xframe-I adapter was accepted in the
kernel starting 2.6.5, and is currently a part of the 2.6 kernel.

Why to re-write an accepted and well-performing production quality driver? 

As Neterion 10GbE ASICs evolve and add more stateless and state-aware TCP
assists, we felt that a forward-looking re-design is warranted. 

Going forward this experimental driver will hopefully serve as a vehicle to
support ASIC features like Large Receive Offload and UDP LSO (based upon
header splitting, multiple logical Tx/Rx paths, MSI/MSI-X usage, etc.) -
both in the driver and in the kernel. Hopefully, some of these TCP assists
will eventually get supported in a de-facto standard way, like checksum
offload and TSO are supported today.

At present, the experimental driver is running on the shipping hardware as
well. Over the last five months it was tested to the same extend as the
"s2io" driver in the kernel, and in some scenarios offers performance
advantage.

When reviewed by the community, we would advocate that "xge" driver is
included in the kernel as an addition, and going forward as a replacement
for the "s2io" driver.

HAL-based
=========
Most Neterion drivers are HAL (Hardware Abstraction Layer) based. This is
always a curse and blessing; in our experience this was the latter by a big
margin. While the current "s2io" driver in the kernel doesn't share HAL code
with other driver, the "xge" driver is HAL-based. 

For these reviewers who consider this a minus, we hope you will find the HAL
code in full compliance with Linux guidelines (in fact, it was written by
our Linux team). Performance-wise, there was no negative impact discovered
either. Testing-wise, this HAL has undergone numerous stress, functional,
and performance tests "under" different drivers on a variety of platforms.

Since the HAL is a "common code", you will find a GPL license on Linux-only
files and a GPL license plus exception on the HAL code.

Going forward, we don't expect community developers to do extensive
modifications to the HAL files - but if it happens, we would appreciate the
changes to be released under the same "GPL plus exception" terms. You are
not required to do so of course, but keeping the default terms on the
hw-specific part of the code will help us to maintain consistency of the
code - for the common benefit.


What about maintenance, support and documentation?
==================================================
Neterion Linux team is committed to maintain the driver going forward;
please post any requests/suggestions on the list and to alex@neterion.com

For developers who intend to work with the hw-dependent code, we will make
available both HAL API guide and ASIC programming manual; please send
requests to leonid@neterion.com

If there is enough interest in modifying the hw-dependent code, please send
the request and we will consider making ASIC programming manual available as
well; please send request to leonid@neterion.com


What about 10GbE setup?
======================
If there will be enough interest in working with the driver; we will put in
place a back-to-back 10GbE setup for remote access, available 24 hrs a day
on a "first come, first served" self-scheduled basis; we realize that 10GbE
gear is still fairly expensive.


Driver Status
=============
The xge driver was done (code complete) by October 2004. Since then it has
accumulated enough mileage in terms of stress, performance, and functional
testing. The driver, although carrying the experimental status, is estimated
to be very close to production.

Kernels: the driver supports both 2.6 and 2.4 kernels.

Platforms: AMD Opteron (TM) (64bit and 32bit), Itanium(TM) based (including
SGI Altix), Xeon 32bit, and PPC(TM) based (in particular, pSeries) systems.

Features:

- Neterion Xframe-I and Xframe-II adapters (the latter adapters were just
recently announced by Neterion, end-user availability is still TBD);

- full checksum offload;

- jumbo frames (up to 9600 bytes);

- there is no limit on a number, size, and alignment of Tx fragments;

- broadcast, multicast, and promiscuous mode;

- TCP Send Offload (TSO);

- ethtool (fully);

- multiple Xframe I and/or Xframe II adapters present in the host;

- multiple receive and transmit rings;

- MSI and MSI-X (the latter Xframe-II only);

- L2, L3, and L4 header splitting;

- Receive Traffic Hashing (Xframe-II only);

- NAPI.

The driver also contains a number of optimization features.

Performance numbers for Xframe-I 10GbE adapter: 7.6 Gbps transmit and 7.6
Gbps receive (jumbo, 2.4GHz dual Opteron), limited by PCI-X 133 bus. Note
that Xframe-II adapter removes the PCI-X bottleneck.

For more information see the readme file: xge.txt.

TODO
=====

(1) Testing so far is 100% successful, but production-grade testing still
remains tbd.

(2) MSI. The driver was tested with a single MSI (see
Documentation/MSI-HOWTO.txt). Multiple MSIs remain tbd.

(3) Large Receive Offload. Unlike most of other features, the LRO retains
its experimental status. It needs to be debugged and fully tested. Note that
to a certain extent the "experimental" status  applies also to MSI and
Receive Traffic Hashing.

(4) NAPI performance. 1500 MTU receive performance is marginally better with
NAPI enabled. We expect to get more than "margin" out of NAPI.


Download
=========

Here's the FTP site to download a single gzipped patch (filename
xge-v.1.1.2816-2.6.11.patch.gz, cksum 3463616385). This patch will replace
the "s2io" driver with the new "xge" driver.

ftp: ns1.s2io.com
user: linuxsrc
password: opensource

Signed-off-by: Dmitry Yusupov <dima@neterion.com>
Signed-off-by: Raghavendra Koushik <raghavendra.koushik@neterion.com>
Signed-off-by: Alex Aizman <alex@neterion.com>


Regards,
Neterion's team.

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[David S. Miller]

On Mon, 14 Mar 2005 12:22:51 -0800
"Alex Aizman" <alex@neterion.com> wrote:

> For these reviewers who consider this a minus, we hope you will find the HAL
> code in full compliance with Linux guidelines (in fact, it was written by
> our Linux team). Performance-wise, there was no negative impact discovered
> either. Testing-wise, this HAL has undergone numerous stress, functional,
> and performance tests "under" different drivers on a variety of platforms.

So you wrote a non-HAL version of this driver and compared the
results?  Simply comparing against the existins s2io driver
does not count.

If you're simply comparing against s2io, and your driver is faster
than s2io is already, imagine how much faster it might be without
that HAL layer.

I totally reject this driver, HAL is unacceptable for in-tree drivers.
We've been over this a thousand times.

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

Hi David,
S2io and Neterion is the same company, we just got our Company name changed
recently. 
So, the non-HAL version of this driver is in fact "s2io" driver in the
kernel - sorry we did not make it clear in the readme file.

I've seen in the past layered architectures for MAC drivers that resulted in
a performance hit, so this was a significant concern for me as well - for
10GbE product performance is an overwriting objective.

We compared performance results of the HAL-based driver vs. monolithic
approach on multiple platforms, and did not see any performance delta that
could be attributed to the approach itself. In fact, in some cases this
experimental driver faster - but I assume this is due to the lessons we
learned not the layered approach itself; performance-wise I see it as a
draw.

Do you have other objections to the submission? We'd like to see if these
could be addressed; going forward we see significant benefits both for
S2io/Neterion (and our customers) and for community to use this driver.

Regards, Leonid

-----Original Message-----
From: David S. Miller [mailto:davem@davemloft.net] 
Sent: Monday, March 14, 2005 12:38 PM
To: alex@neterion.com
Cc: netdev@oss.sgi.com; leonid@neterion.com; jgarzik@pobox.com
Subject: Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

On Mon, 14 Mar 2005 12:22:51 -0800
"Alex Aizman" <alex@neterion.com> wrote:

> For these reviewers who consider this a minus, we hope you will find the
HAL
> code in full compliance with Linux guidelines (in fact, it was written by
> our Linux team). Performance-wise, there was no negative impact discovered
> either. Testing-wise, this HAL has undergone numerous stress, functional,
> and performance tests "under" different drivers on a variety of platforms.

So you wrote a non-HAL version of this driver and compared the
results?  Simply comparing against the existins s2io driver
does not count.

If you're simply comparing against s2io, and your driver is faster
than s2io is already, imagine how much faster it might be without
that HAL layer.

I totally reject this driver, HAL is unacceptable for in-tree drivers.
We've been over this a thousand times.

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Andi Kleen]

"Leonid Grossman" <leonid.grossman@neterion.com> writes:
>
> Do you have other objections to the submission? We'd like to see if these
> could be addressed; going forward we see significant benefits both for
> S2io/Neterion (and our customers) and for community to use this driver.

I guess the main objection to the HAL comes not from performance
issues (Usually the only thing that really counts for performance
is data cache misses and the HAL is unlikely to affect this much), but
the coding style etc.. Indeed it does not look too Linux like. 

One thing that's frowned upon in Linux are lots of wrappers for
standard functions (like spin_lock etc.). I would recommend
to at least replace them with the standard Linux functions.
In principle this can be even done with some kind of preprocessor
Also less ifdefs would be nice.

An possible compromise might be to get rid of all the HAL parts that
wraps Linux functionality, and then only use a leaner low level
library to access the more difficult parts of the hardware.  This
would involve moving more code into the Linux specific layers. This
should be more low level code, nothing like the high level queue
handling functions you currently have etc., with the high level logic
all in Linux code

-Andi

P.S.: The patch would be much easier to read if it created new
files instead of changing the old ones. This makes sense since
the new driver will probably live next to the old one for some time.

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Jeff Garzik]

Andi Kleen wrote:
> "Leonid Grossman" <leonid.grossman@neterion.com> writes:
> 
>>Do you have other objections to the submission? We'd like to see if these
>>could be addressed; going forward we see significant benefits both for
>>S2io/Neterion (and our customers) and for community to use this driver.
> 
> 
> I guess the main objection to the HAL comes not from performance
> issues (Usually the only thing that really counts for performance
> is data cache misses and the HAL is unlikely to affect this much), but
> the coding style etc.. Indeed it does not look too Linux like. 

Well, not coding style, but code analysis and maintenance issues.

HALs are generally type-opaque, breaking checker-style tools and sparse 
checks.

"it looks like Linux code" has implications on bug finding and fixing, 
and long term maintenance of the code.  You want to make it easy for 
someone to make the same change across N net drivers.

Because most ->hard_start_xmit() hooks were written in a similar 
fashion, it was easy and quick to deploy fixes for the skb_padto() 
security bug across many net drivers.

A lot of tiny costs that mostly wind up as noise:  additional branching 
/ derefs.

My biggest objection is that HALs increase the overall "cost" of 
maintaining a piece of code, and serve as a barrier against outside 
(non-primary-author) kernel hacker involvement.

Remember, this driver is going to be with us for -10- years or more.

	Jeff

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

> -----Original Message-----
> From: Jeff Garzik [mailto:jgarzik@pobox.com]
> Sent: Monday, March 14, 2005 3:46 PM
> To: Andi Kleen
> Cc: Leonid Grossman; netdev@oss.sgi.com; leonid@neterion.com;
> alex@neterion.com; David S. Miller
> Subject: Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE
> Adapters
> 
> Andi Kleen wrote:
> > "Leonid Grossman" <leonid.grossman@neterion.com> writes:
> >
> >>Do you have other objections to the submission? We'd like to see if
> these
> >>could be addressed; going forward we see significant benefits both for
> >>S2io/Neterion (and our customers) and for community to use this driver.
> >
> >
> > I guess the main objection to the HAL comes not from performance
> > issues (Usually the only thing that really counts for performance
> > is data cache misses and the HAL is unlikely to affect this much), but
> > the coding style etc.. Indeed it does not look too Linux like.
> 
> Well, not coding style, but code analysis and maintenance issues.
> 
> HALs are generally type-opaque, breaking checker-style tools and sparse
> checks.
> 
> "it looks like Linux code" has implications on bug finding and fixing,
> and long term maintenance of the code.  You want to make it easy for
> someone to make the same change across N net drivers.
> 
> Because most ->hard_start_xmit() hooks were written in a similar
> fashion, it was easy and quick to deploy fixes for the skb_padto()
> security bug across many net drivers.
> 
> A lot of tiny costs that mostly wind up as noise:  additional branching
> / derefs.
> 
> My biggest objection is that HALs increase the overall "cost" of
> maintaining a piece of code, and serve as a barrier against outside
> (non-primary-author) kernel hacker involvement.

There are several valid arguments against HAL-based MAC drivers, but the
maintenance cost is probably one of the main arguments in the "for" column.

My assumption is that the vast majority of the maintenance for the
hw-dependent code will be done by Neterion team - and for us the HAL-based
approach allows to fix a bug or to add a new feature in hw-dependent code
once and for all across all drivers. In our experience, third-party
contribution tends to come to the OS-specific part of the driver not to the
HAL - but if there is a meaningful contribution to the HAL from one of our
Unix driver, it will be useful to pick it up in a transparent fashion. 

Agreed that HAL makes code analysis and changes by the outside hacker
somewhat more difficult - but hopefully not by much, and again I expect most
of the outside contribution to be done above HAL level.
Also, we are trying to ease the pain by releasing HAL API and ASIC
Programming Manual documentation.

> 
> Remember, this driver is going to be with us for -10- years or more.

Yes, and this was actually a second reason for the submission. Present
"s2io" driver is fine for the current and next ASIC, beyond that we will
have to probably re-write it anyways. The new submission is 2+ years
"younger" and has a much better chance to require an incremental upgrade
only. Since 10GbE is just taking off and not a lot of people got familiar
with the code, we feel it's easier to take this hit earlier than later. 

To sum it up, I agree that HAL is a trade-off - but for this hardware it has
been working well for us in multi-platform environment; the hope is that you
will find a (smaller, perhaps) subset of these benefits useful in Linux
environment as well.

Leonid

> 
> 	Jeff
> 

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Alex Aizman]

Andi Kleen writes: 

> I guess the main objection to the HAL comes not from 
> performance issues 

But the second or the third objection comes from there, I guess... As far as
the data path, HAL as a "layer" completely disappears. There is just a few
inline instructions that post descriptors and process completed descriptors.
These same instructions are unavoidable; they'd be present HAL or no-HAL.
There's no HAL locks on the data path (the locks are compiled out), no HAL
(as a "layer") induced overhead. Note that the performance was one
persistent "paranoia" from the very start of this project.

The numbers also tell the tale. We have 7.6Gbps jumbo throughput, the
bottleneck is PCI, not the host. We have 13us 1byte netpipe latency. Here's
for example today's netpipe run:

[root@localhost root]# ./nptcp -a -t -l 256 -u 98304 -i 256 -p 5100 -P - h
17.1.1.227
Latency: 0.000013
Now starting main loop
  0:       256 bytes    7 times -->  131.37 Mbps in 0.000015 sec
  1:       512 bytes   65 times -->  239.75 Mbps in 0.000016 sec
  2:       768 bytes 7701 times -->  181.37 Mbps in 0.000032 sec
  3:      1024 bytes 5168 times -->  212.35 Mbps in 0.000037 sec
  4:      1280 bytes 5102 times -->  209.95 Mbps in 0.000047 sec
  5:      1536 bytes 4303 times -->  211.65 Mbps in 0.000055 sec
  6:      1792 bytes 3765 times -->  238.44 Mbps in 0.000057 sec
  7:      2048 bytes 3739 times -->  267.33 Mbps in 0.000058 sec
  8:      2304 bytes 3744 times -->  297.43 Mbps in 0.000059 sec
  9:      2560 bytes 3761 times -->  319.77 Mbps in 0.000061 sec
 10:      2816 bytes 3685 times -->  349.80 Mbps in 0.000061 sec
 11:      3072 bytes 3701 times -->  344.98 Mbps in 0.000068 sec
 12:      3328 bytes 3374 times -->  372.86 Mbps in 0.000068 sec
 13:      3584 bytes 3389 times -->  400.46 Mbps in 0.000068 sec
...

> (Usually the only thing that really counts 
> for performance is data cache misses and the HAL is unlikely 
> to affect this much), but the coding style etc.. Indeed it 
> does not look too Linux like. 
> 
> One thing that's frowned upon in Linux are lots of wrappers 
> for standard functions (like spin_lock etc.). I would 
> recommend to at least replace them with the standard Linux functions.

There's always a tradeoff, a balancing act. The wrappers is a price to pay
for the reusable and extremely well tested code. Note also that a small
company like Neterion does not have the luxury *not* to re-use the code.. 

Having said that, there's couple ideas that can be worked in relatively
easily.

> In principle this can be even done with some kind of 
> preprocessor Also less ifdefs would be nice.

Exactly. Simple script that'll remove extra #ifdefs and wrappers.

> 
> An possible compromise might be to get rid of all the HAL 
> parts that wraps Linux functionality, and then only use a 
> leaner low level library to access the more difficult parts 
> of the hardware.  This would involve moving more code into 
> the Linux specific layers. This should be more low level 
> code, nothing like the high level queue handling functions 
> you currently have etc., with the high level logic all in Linux code

Well, I guess we can do a lot in that direction. Looking forward to get
specific review comments, etc. However, the main question remains: will the
HAL-based driver (because even after the script-produced "surgery" it'll
continue to be HAL based) ever get accepted?

> 
> -Andi
> 
> P.S.: The patch would be much easier to read if it created 
> new files instead of changing the old ones. 

Can be done, very easy. Obviously, only one of the driver has to be
selected..

> This makes sense 
> since the new driver will probably live next to the old one 
> for some time.
> 
> 

Alex

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Rick Jones]

Alex Aizman wrote:
> Andi Kleen writes: 
> 
> 
>>I guess the main objection to the HAL comes not from 
>>performance issues 
> 
> 
> But the second or the third objection comes from there, I guess... As far as
> the data path, HAL as a "layer" completely disappears. There is just a few
> inline instructions that post descriptors and process completed descriptors.
> These same instructions are unavoidable; they'd be present HAL or no-HAL.
> There's no HAL locks on the data path (the locks are compiled out), no HAL
> (as a "layer") induced overhead. Note that the performance was one
> persistent "paranoia" from the very start of this project.
> 
> The numbers also tell the tale. We have 7.6Gbps jumbo throughput, the
> bottleneck is PCI, not the host.

That would seem to suggest then comparing (using netperf terminology) service 
demands between HAL and no HAL.  JumboFrame can compensate for a host of ills :) 
I really do _not_ mean to imply there are any ills for which compensation is 
required, just suggesting to get folks into the habit of including CPU 
utilization.  And since we cannot count on JumboFrame being there end-to-end, 
performance with 1500 byte frames, while perhaps a bit unpleasant, is still 
important.

> We have 13us 1byte netpipe latency. 

So 76,000 transactions per second on something like single-byte netperf 
TCP_RR?!? Or am I mis-interpreting the netpipe latency figure?

I am of course biased, but netperf (compiled with -DUSE_PROCSTAT under Linux, 
somethign else for other OSes - feel free to contact me about it) tests along 
the lines of:

netperf -c -C -t TCP_STREAM -H <remote> -l <length> -i 10,3 -- -s 256K -S 256K 
-m 32K

and

netperf -c -C -t TCP_RR -H <remote> -l <length> -i 10,3

are generally useful.  If you have the same system type at each end, the -C can 
be dropped from the TCP_RR test since it _should_ be symmetric.  If -C dumps 
core on the TCP_STREAM test, drop it and add a TCP_MAERTS test to get receive 
service demand.

rick jones

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

> -----Original Message-----
> From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com] On
> Behalf Of Rick Jones
> Sent: Monday, March 14, 2005 5:30 PM
> To: netdev@oss.sgi.com
> Subject: Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE
> Adapters
> 
> Alex Aizman wrote:
> > Andi Kleen writes:
> >
> >
> >>I guess the main objection to the HAL comes not from
> >>performance issues
> >
> >
> > But the second or the third objection comes from there, I guess... As
> far as
> > the data path, HAL as a "layer" completely disappears. There is just a
> few
> > inline instructions that post descriptors and process completed
> descriptors.
> > These same instructions are unavoidable; they'd be present HAL or no-
> HAL.
> > There's no HAL locks on the data path (the locks are compiled out), no
> HAL
> > (as a "layer") induced overhead. Note that the performance was one
> > persistent "paranoia" from the very start of this project.
> >
> > The numbers also tell the tale. We have 7.6Gbps jumbo throughput, the
> > bottleneck is PCI, not the host.
> 
> That would seem to suggest then comparing (using netperf terminology)
> service
> demands between HAL and no HAL.  JumboFrame can compensate for a host of
> ills :)
> I really do _not_ mean to imply there are any ills for which compensation
> is
> required, just suggesting to get folks into the habit of including CPU
> utilization.  And since we cannot count on JumboFrame being there end-to-
> end,
> performance with 1500 byte frames, while perhaps a bit unpleasant, is
> still
> important.

Hi Rick,
With Jumbo frames, the performance and %cpu for the two drivers are at par.
With 1500, HAL driver actually shows somewhat better throughput and %cpu -
though the approach has nothing to do with it, the important point is that
there is no performance drop that could be traced to the usage of HAL per
say
Leonid.

> 
> > We have 13us 1byte netpipe latency.
> 
> So 76,000 transactions per second on something like single-byte netperf
> TCP_RR?!? Or am I mis-interpreting the netpipe latency figure?
> 
> I am of course biased, but netperf (compiled with -DUSE_PROCSTAT under
> Linux,
> somethign else for other OSes - feel free to contact me about it) tests
> along
> the lines of:
> 
> netperf -c -C -t TCP_STREAM -H <remote> -l <length> -i 10,3 -- -s 256K -S
> 256K
> -m 32K
> 
> and
> 
> netperf -c -C -t TCP_RR -H <remote> -l <length> -i 10,3
> 
> are generally useful.  If you have the same system type at each end, the -
> C can
> be dropped from the TCP_RR test since it _should_ be symmetric.  If -C
> dumps
> core on the TCP_STREAM test, drop it and add a TCP_MAERTS test to get
> receive
> service demand.
> 
> rick jones

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

> Alex Aizman writes:
> However, the main question remains: will the
> HAL-based driver (because even after the script-produced "surgery" it'll
> continue to be HAL based) ever get accepted?

Hi all, 
We truly appreciate the time spent on looking at the code and the feedback..

I guess Alex is asking the right question - before we start code changes, it
will be great to get a rough consensus on whether this HAL-based driver
(after suggested changes) will be acceptable to the community - or yet
another HAL driver in tree will be still "one too many"?

In particular - after this discussion, does David's statement below still
stand (not sure there was an unconditional rejection of the HAL model from
someone else)? 

>David Miller writes:
>I totally reject this driver, HAL is unacceptable for in-tree drivers.
>We've been over this a thousand times.

If this stands, we are prepared to recall the submission and keep the
current "Linux and everything else" status-quo for 10GbE Xframe drivers.
It's not the best maintenance option (both for us and arguably, even for a
non-primary-author kernel hackers) but it's workable.

Thanks again,
Leonid

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

> 
> > Leonid Grossman writes:
> It's not the best maintenance option (both for us and arguably, even for a
> non-primary-author kernel hackers) but it's workable.

The statement about non-primary-author kernel hacker interests needs some
clarification, before people started to throw things at me. 
This is the last argument before I shut up, I promise :-)

Arguably, a hacker will be much more interested in changing the non-HAL part
of a driver. This part of the code has to look and feel the same as any
other Linux net driver. If it doesn't, then we've done a poor job and need
to go back.

In our experience, the vast majority of code fixes and new features in the
HAL code comes from our team (hey, we planted all these bugs to begin with
:-)), and to a much lesser extend from other OS developers, and only then
from Linux community. 

This is normal - for example, if Large Receive Offload that was discussed
earlier is to be done, I'd expect a hacker to focus on the kernel changes
and generic driver changes (to make a feature common to all net driver that
want it), and a Neterion developer to focus on the hw-specific changes. 

There will be nothing wrong if they trade places, but the first scenario
seems more likely.

So, for a hacker it's harder to understand HAL part of the code - but then
he is not doing the bulk of the maintenance of the ASIC -specific code that
he's arguably least interested in maintaining...
Leonid

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Andi Kleen]

"Leonid Grossman" <leonid.grossman@neterion.com> writes:

>> 
>> > Leonid Grossman writes:
>> It's not the best maintenance option (both for us and arguably, even for a
>> non-primary-author kernel hackers) but it's workable.
>
> The statement about non-primary-author kernel hacker interests needs some
> clarification, before people started to throw things at me. 
> This is the last argument before I shut up, I promise :-)


First you wont like it, but Linux kernel maintainers usually dont give
firm promises to merge anything.  They just say "with that i wont merge it"
and even that is flexible sometimes. It is not like a company who
gives commitments or signs contracts, it is really the final patch that counts.

The best way to get code merged is to write it like the maintainers
wants it, but of course that does not happen always.

However when the code is clean and the biggest issues are fixed
and only some relatively small ones are left I think you have a good
chance to see your code merged.


>
> Arguably, a hacker will be much more interested in changing the non-HAL part
> of a driver. This part of the code has to look and feel the same as any
> other Linux net driver. If it doesn't, then we've done a poor job and need
> to go back.

There are currently some issues, mostly the "LAL" in it (Linux Adaption
Layer that wraps everything) and all the ugly function parameters (IN and 
__STATIC_* and the wrapped list functions ). That is what just poked in my 
eyes at the first look.

I personally dont care that much about the later stuff which
is more cosmetic, but a lot of other people strongly object to 
code that does not look like other Linux code so I would recommend to 
fix it. 


What I also did not like was that you had high level logic in these
HAL parts - like handling packet queues etc. Normally IMHO high level
logic should be directly in the functions that are directly called
from the kernel. This way it is easy to follow the main logic
of the driver if someone is familiar with linux network drivers
in general

LALs are strongly frowned at upon and I personally also dont like them
at all. AFAIK there is one two drivers in the kernel tree that have 
it and it is considered an historic accident by everybody I know  ,=)


> In our experience, the vast majority of code fixes and new features in the
> HAL code comes from our team (hey, we planted all these bugs to begin with
> :-)), and to a much lesser extend from other OS developers, and only then
> from Linux community. 

The problem is usually that when there is some bug in your driver
and someone not in your company wants to debug it because the bug
is causing them problems (that is one of the great advantages of free 
software that they can do it themselves), then they want relatively
clean code. The same happens for the maintainer who is hunting some
issue and needs to change your driver slightly for some infrastructure
change. 

So even though most changes come likely from you there is a need
for other people to understand and change your code.

-Andi

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

> First you wont like it, but Linux kernel maintainers usually dont give
> firm promises to merge anything.  They just say "with that i wont merge
> it"
> and even that is flexible sometimes. It is not like a company who
> gives commitments or signs contracts, it is really the final patch that
> counts.
> 
> The best way to get code merged is to write it like the maintainers
> wants it, but of course that does not happen always.
> 
> However when the code is clean and the biggest issues are fixed
> and only some relatively small ones are left I think you have a good
> chance to see your code merged.

Sure. I was not asking for a promise to merge the code, only for a rough
consensus that a HAL-based approach by itself is not a showstopper. 
Without such a consensus, it doesn't matter if we address other issues,
right?
 
> 
> >
> > Arguably, a hacker will be much more interested in changing the non-HAL
> part
> > of a driver. This part of the code has to look and feel the same as any
> > other Linux net driver. If it doesn't, then we've done a poor job and
> need
> > to go back.
> 
> There are currently some issues, mostly the "LAL" in it (Linux Adaption
> Layer that wraps everything) and all the ugly function parameters (IN and
> __STATIC_* and the wrapped list functions ). That is what just poked in my
> eyes at the first look.
> 
> I personally dont care that much about the later stuff which
> is more cosmetic, but a lot of other people strongly object to
> code that does not look like other Linux code so I would recommend to
> fix it.

Agreed; we are fixing this for the new submission.

> 
> 
> What I also did not like was that you had high level logic in these
> HAL parts - like handling packet queues etc. Normally IMHO high level
> logic should be directly in the functions that are directly called
> from the kernel. This way it is easy to follow the main logic
> of the driver if someone is familiar with linux network drivers
> in general

In general, I agree that low level logic (AKA HAL code) needs to be as small
as possible, and should not include a code that is typically common in Linux
(or any other) network driver. 
Unfortunately MAC driver models for different Operating Systems diverged so
much, the interface for the low level logic ends up higher than anyone would
like...
This is a valid comment though, we will see if there is room for
improvement.

Skip...


> The problem is usually that when there is some bug in your driver
> and someone not in your company wants to debug it because the bug
> is causing them problems (that is one of the great advantages of free
> software that they can do it themselves), then they want relatively
> clean code. The same happens for the maintainer who is hunting some
> issue and needs to change your driver slightly for some infrastructure
> change.
> 
> So even though most changes come likely from you there is a need
> for other people to understand and change your code.

I agree, to a point - if for a hacker or a maintainer this driver is way
hard to understand (comparing to it's "s2io" sibling), then is not a good
thing and has to be addressed as much as practical.

Hopefully follow-up submissions will get to the point when HAL code still
has some non-Linux "accent" but it's sufficiently clean for people to
understand it.
This will be a tradeoff for some, but hopefully a good one - I still think
the majority of low-level code fixes will come from our team (sometimes by
virtue of fixing a problem in different OS).

Thanks for the detailed feedback!
Leonid


> 
> -Andi

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[jamal]

On Sat, 2005-03-19 at 17:19, Leonid Grossman wrote:

> Sure. I was not asking for a promise to merge the code, only for a rough
> consensus that a HAL-based approach by itself is not a showstopper. 
> Without such a consensus, it doesn't matter if we address other issues,
> right?

Typically theres only one motivation for HALs - maintain the same code
base for 20 OSes. Makes it easier for a small company (not sure if yours
fits that category) to maintain. It doesnt matter how differently you
position or spin it (no offense intended), that is the main if not only
motivation. 

OTOH, you shift the burden of the extra maintainance work to the people
on the Linux side who know may have to understand what your layer does.
This is why people hate it.
I dont think it is sufficient to say your company will be updating the
driver:
- Years from now your company may not be around anymore but your
hardware may still be. Who is going to fix the bugs then?  
- APIs, mechanisms etc change on Linux as well. You become a bottleneck
if nobody understands your HAL because they have to wait for you to make
the changes.
Traditionaly whoever makes the changes on dirver schemes ensures all
drivers are updated. Think positively that this is now offloaded from
you.

My suggestion: have a dedicated resource just for Linux - it is big
enough to justify; maintain HAL for other OSes that will never change, I
bet you whatever works for NDIS probably will continue to work for
vxworks for the next 10 years - not much innovation going on there;-> or
you could claim (bah!) theres stability on those OSes.

cheers,
jamal

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

Hi Jamal,
You are right on target, it's all about finding the best driver support
model - for the end users, not for a Neterion developer or even for a Linux
hacker.
At the moment, "s2io" driver is supported in exactly the way you suggest;
the model works well but we think it's possible to do better - again, for
the end user sake.

If we could write a driver, put it in the kernel and let it "heal itself",
then I'd agree that the current model would have been the best. We'd have
been happy to stay with it, rather than to go through with a considerable
extra effort - preserving the status quo is always easier.  

I'd argue that the vast majority of a 10GbE NICs will be initially developed
by a smaller company like Neterion, shipped by large server OEMs and
supported (from end-user prospective) by these two entities, as well as by
Linux community and major Linux distributors.

In our experience, even early adopters of 10GbE adapters in Linux space
rarely attempt to fix hardware-dependent bugs themselves - they ask for
support from one of the three entities above (a NIC vendor, a server vendor
an a Linux distributor), with a lion share of urgent level 2 and level 3
issues ending up with a NIC vendor. As 10GbE adoption accelerates, I expect
this trend will accelerate as well - I wish we could shift the support
burden to someone else, but the chances of that to happen are getting
slimmer over time :-).

BTW this is precisely the reason the NIC vendors favor HAL model. Software
teams are about the same size, no matter whether it's a big or small company
- and no NIC vendor likes spending 30% of their resources on one (out of 10)
device drivers.

So, at least from what we see in our space, open source is a great thing but
it doesn't make driver support (at least for the low level NIC code)
fundamentally different from other Operating Systems, at least not for a
foreseeable future. 
IMHO, nor should it - ability to have source and fix hw-dependent bugs is
great, but I'd argue that much bigger value of the open source model is in
the (more strategic) ability to timely identify and fix existing networking
bottlenecks that are common to most/all NIC drivers. 

An example - at present, there are no Linux UDP TSO APIs.  We see a customer
need for it and we have the ASIC support as well; from an end user (and
about everybody else) prospective it would be great to add these APIs by
people who know Linux best.
Whether the hw-dependent part of the UDP TSO is done by a hacker or it is an
existing and working HAL code from a Neterion developer, is not that
important to the end user - though the latter will be a more likely and more
efficient way to go. 

BTW all arguments above make sense only if a high-level part of the driver
is very much "Linux-like" - we are obviously motivated to make the high
level code suitable to the changes by the community, since this is where the
value for the end user is.

We could talk about some other points that you brought up (for example, the
statement about other OSes not changing is applicable only to a small subset
of "legacy" OSes - I suspect the rest of them will be changing NIC APIs
rather fast and HAL is actually a good way to deal with the change across
the board), but as you pointed out the support model is the key here...

Cheers, Leonid



> -----Original Message-----
> From: netdev-bounce@oss.sgi.com [mailto:netdev-bounce@oss.sgi.com] On
> Behalf Of jamal
> Sent: Sunday, March 20, 2005 5:40 AM
> To: Leonid Grossman
> Cc: 'Andi Kleen'; netdev@oss.sgi.com; leonid@neterion.com;
> jgarzik@pobox.com
> Subject: RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE
> Adapters
> 
> On Sat, 2005-03-19 at 17:19, Leonid Grossman wrote:
> 
> > Sure. I was not asking for a promise to merge the code, only for a rough
> > consensus that a HAL-based approach by itself is not a showstopper.
> > Without such a consensus, it doesn't matter if we address other issues,
> > right?
> 
> Typically theres only one motivation for HALs - maintain the same code
> base for 20 OSes. Makes it easier for a small company (not sure if yours
> fits that category) to maintain. It doesnt matter how differently you
> position or spin it (no offense intended), that is the main if not only
> motivation.
> 
> OTOH, you shift the burden of the extra maintainance work to the people
> on the Linux side who know may have to understand what your layer does.
> This is why people hate it.
> I dont think it is sufficient to say your company will be updating the
> driver:
> - Years from now your company may not be around anymore but your
> hardware may still be. Who is going to fix the bugs then?
> - APIs, mechanisms etc change on Linux as well. You become a bottleneck
> if nobody understands your HAL because they have to wait for you to make
> the changes.
> Traditionaly whoever makes the changes on dirver schemes ensures all
> drivers are updated. Think positively that this is now offloaded from
> you.
> 
> My suggestion: have a dedicated resource just for Linux - it is big
> enough to justify; maintain HAL for other OSes that will never change, I
> bet you whatever works for NDIS probably will continue to work for
> vxworks for the next 10 years - not much innovation going on there;-> or
> you could claim (bah!) theres stability on those OSes.
> 
> cheers,
> jamal
> 
> 

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Scott Feldman]

On Mar 14, 2005, at 12:22 PM, Alex Aizman wrote:

> HAL-based
> =========
> Most Neterion drivers are HAL (Hardware Abstraction Layer) based. This 
> is
> always a curse and blessing; in our experience this was the latter by 
> a big
> margin. While the current "s2io" driver in the kernel doesn't share 
> HAL code
> with other driver, the "xge" driver is HAL-based.

e1000 and ixgb are HAL-based, which is why there is always push back 
when someone in the community modifies *_hw.[ch].  I'd hate to see more 
of this in the kernel, but I can definitely relate to the "testing 
across multiple OSes" gain.

Here's an (old?) idea: remember the NDIS-wrapper project?  I think the 
reverse is much more interesting.  A linux-wrapper takes a plain old 
Linux driver and wraps it with what ever is needed to make it an NDIS 
driver.  Or FreeBSD, or whatever.  Let's pretend this is trivial for a 
second.  What do we gain?  1) one clean Linux driver to maintain, 2) 
testability on other OSes, and 3) access to other OSes' certification 
kits.  Licensing is clean: the Linux driver is GPL and the 
linux-wrapper code is GPL.  Can't the world revolve around Linux and 
let everyone else be burdened with the abstraction layer overhead?

-scott

Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Matt Mackall]

On Mon, Mar 14, 2005 at 09:14:59PM -0800, Scott Feldman wrote:
> 
> On Mar 14, 2005, at 12:22 PM, Alex Aizman wrote:
> 
> >HAL-based
> >=========
> >Most Neterion drivers are HAL (Hardware Abstraction Layer) based. This 
> >is
> >always a curse and blessing; in our experience this was the latter by 
> >a big
> >margin. While the current "s2io" driver in the kernel doesn't share 
> >HAL code
> >with other driver, the "xge" driver is HAL-based.
> 
> e1000 and ixgb are HAL-based, which is why there is always push back 
> when someone in the community modifies *_hw.[ch].  I'd hate to see more 
> of this in the kernel, but I can definitely relate to the "testing 
> across multiple OSes" gain.
> 
> Here's an (old?) idea: remember the NDIS-wrapper project?  I think the 
> reverse is much more interesting.  A linux-wrapper takes a plain old 
> Linux driver and wraps it with what ever is needed to make it an NDIS 
> driver.  Or FreeBSD, or whatever.  Let's pretend this is trivial for a 
> second.  What do we gain?  1) one clean Linux driver to maintain, 2) 
> testability on other OSes, and 3) access to other OSes' certification 
> kits.  Licensing is clean: the Linux driver is GPL and the 
> linux-wrapper code is GPL.  Can't the world revolve around Linux and 
> let everyone else be burdened with the abstraction layer overhead?

Depends. Vendors of non-GPL OSes can't ship such drivers for risk of
their product becoming a derived work to the extent they're relying on
such drivers to make their system useful.

You can't get around the GPL by putting a GPL wrapper with an
exception around something.

-- 
Mathematics is the supreme nostalgia of our time.

RE: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE Adapters

[Leonid Grossman]

> -----Original Message-----
> From: Scott Feldman [mailto:sfeldma@pobox.com]
> Sent: Monday, March 14, 2005 9:15 PM
> To: <alex@neterion.com>
> Cc: <netdev@oss.sgi.com>; <leonid@neterion.com>; 'Jeff Garzik'
> Subject: Re: [ANNOUNCE] Experimental Driver for Neterion/S2io 10GbE
> Adapters
> 
> 
> On Mar 14, 2005, at 12:22 PM, Alex Aizman wrote:
> 
> > HAL-based
> > =========
> > Most Neterion drivers are HAL (Hardware Abstraction Layer) based. This
> > is
> > always a curse and blessing; in our experience this was the latter by
> > a big
> > margin. While the current "s2io" driver in the kernel doesn't share
> > HAL code
> > with other driver, the "xge" driver is HAL-based.
> 
> e1000 and ixgb are HAL-based, which is why there is always push back
> when someone in the community modifies *_hw.[ch].  I'd hate to see more
> of this in the kernel, but I can definitely relate to the "testing
> across multiple OSes" gain.

I'll be thrilled to see meaningful community changes to xge HAL files :-)

> 
> Here's an (old?) idea: remember the NDIS-wrapper project?  I think the
> reverse is much more interesting.  A linux-wrapper takes a plain old
> Linux driver and wraps it with what ever is needed to make it an NDIS
> driver.  Or FreeBSD, or whatever.  Let's pretend this is trivial for a
> second.  What do we gain?  1) one clean Linux driver to maintain, 2)
> testability on other OSes, and 3) access to other OSes' certification
> kits.  Licensing is clean: the Linux driver is GPL and the
> linux-wrapper code is GPL.  Can't the world revolve around Linux and
> let everyone else be burdened with the abstraction layer overhead?

Believe or not, something like this could (and has been) done but did not
stick. People did even spookier things, like a wrapper on top of another OS
binary driver - some of you guys may remember Novell NLM vs. Unixware driver
project :-)

These attempts were always short-lived, my guess is because the teams behind
the project did not have a long-term motivation and/or the expertise to get
it done and (more importantly) to maintain the framework, as OS driver
models evolved and diverged.

On the other hand, the HAL model is more alive than ever - Intel, Neterion
and many/most other modern NICs I'm aware of use HAL approach. 
This is actually much more common between non-Linux Oses, since there are
still significant (warranted or not) GPL-related concerns. 

HAL discussion is an old one indeed, and the truth is still in the eye of a
beholder :-)

People who are sorely responsible for supporting multiple OS drivers for the
same hardware (that is, NIC vendors) have strong bias in favor of HAL
approach, for all the right reasons (mainly, maintainability and support). 

People who are sorely responsible for supporting drivers for multiple NICs
on the same OS have strong bias against HAL approach - also for some pretty
valid reasons.

Linux is one (but not the only) case when the drivers are maintained by both
groups, and the biases/preferences collide...

Interestingly enough, as time goes by and everybody is getting thin on
resources, HAL approach tends to gain "marketshare" rather fast - at
present, Linux "s2io" is one of the very few Neterion drivers that is not
using HAL.


> 
> -scott