Tcp/ip offload card driver

Tcp/ip offload card driver

[chen, xiangping]

Hi,

Is there any TCP offload (TOE) card driver available on Linux?

Thanks for the info!

Xiangping Chen

Re: Tcp/ip offload card driver

[David S. Miller]

   From: "chen, xiangping" <chen_xiangping@emc.com>
   Date: Fri, 10 May 2002 10:48:23 -0400

   Is there any TCP offload (TOE) card driver available on Linux?
   
Why do you want it?  There is no proven performance benefit.

PCI bandwidth is the limiting factor for networking performance.

Re: Tcp/ip offload card driver

[Pedro M. Rodrigues]

   Actually there is. Think iSCSI. Have a look at this article at 
LinuxJournal - http://linuxjournal.com/article.php?sid=4896 .


/Pedro

On 10 May 2002 at 7:43, David S. Miller wrote:

>    From: "chen, xiangping" <chen_xiangping@emc.com>
>    Date: Fri, 10 May 2002 10:48:23 -0400
> 
>    Is there any TCP offload (TOE) card driver available on Linux?
> 
> Why do you want it?  There is no proven performance benefit.
> 
> PCI bandwidth is the limiting factor for networking performance.
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Tcp/ip offload card driver

[David S. Miller]

   From: "Pedro M. Rodrigues" <pmanuel@myrealbox.com>
   Date: Fri, 10 May 2002 17:11:55 +0200

      Actually there is. Think iSCSI. Have a look at this article at 
   LinuxJournal - http://linuxjournal.com/article.php?sid=4896 .

The Linux networking stack need have no hand in any of the IPv4 done
by iSCSI, it can live entirely in the cards firmware and Linux need
not know what the transport looks like at all.

Re: Tcp/ip offload card driver

[Jesse Pollard]

"David S. Miller" <davem@redhat.com>:
>    From: "Pedro M. Rodrigues" <pmanuel@myrealbox.com>
>    Date: Fri, 10 May 2002 17:11:55 +0200
> 
>       Actually there is. Think iSCSI. Have a look at this article at 
>    LinuxJournal - http://linuxjournal.com/article.php?sid=4896 .
> 
> The Linux networking stack need have no hand in any of the IPv4 done
> by iSCSI, it can live entirely in the cards firmware and Linux need
> not know what the transport looks like at all.
> -

Depends on what kind of authentication you also need. I haven't seen anything
on that. So far as I know (and that is limited right now) iSCSI doesn't
perform any kind of authentication beyond IP number.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Tcp/ip offload card driver

[Oliver Xymoron]

On Fri, 10 May 2002, Jesse Pollard wrote:

> Depends on what kind of authentication you also need. I haven't seen anything
> on that. So far as I know (and that is limited right now) iSCSI doesn't
> perform any kind of authentication beyond IP number.

Later drafts of iSCSI do real authentication.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Tcp/ip offload card driver

[Jeff Garzik]

Pedro M. Rodrigues wrote:

>   Actually there is. Think iSCSI. Have a look at this article at 
>LinuxJournal - http://linuxjournal.com/article.php?sid=4896 .
>

Ug...  why bother?  Just buy an SMP system at that point...

Re: Tcp/ip offload card driver

[Nicholas Harring]

And how about when an SMP system isn't enough? Should I have to 
re-engineer my network storage architecture when hardware exists that'll 
increase throughput if a simple device driver gets written? Don't forget 
that with 64 bit PCI that the limit of the bus has been raised, and with 
impending technologies like Infiniband and Hypertransport that limit 
will be raised again. At that point devoting main processor resources to 
something better handled by specialty hardware really stops making 
sense, if that specialty hardware is low-cost (oughta be) and effective 
(still debatable).

Nicholas Harring
Hostway Corporation


Jeff Garzik wrote:
> Pedro M. Rodrigues wrote:
> 
>>   Actually there is. Think iSCSI. Have a look at this article at 
>> LinuxJournal - http://linuxjournal.com/article.php?sid=4896 .
>>
> 
> Ug...  why bother?  Just buy an SMP system at that point...
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Tcp/ip offload card driver

[David S. Miller]

   From: Nicholas Harring <nharring@hostway.net>
   Date: Fri, 10 May 2002 10:51:06 -0500

   And how about when an SMP system isn't enough?

Demonstrate this.

Putting the whole implementation on the cards firmware is feasible,
you don't need SMP.  It's totally doable and Linux needs to see
none of the details.

Franks a lot,
David S. Miller
davem@redhat.com

Re: Tcp/ip offload card driver

[Nicholas Harring]

Obviously some form of driver is necessary to access the device, whether 
or not we're pushing fully formed IP packets or raw payload. Or is that 
a userland problem and I'm just not understanding the flow from 
userspace through the kernel and to the driver properly?

Cheers,
Nicholas Harring

David S. Miller wrote:
>    From: Nicholas Harring <nharring@hostway.net>
>    Date: Fri, 10 May 2002 10:51:06 -0500
> 
>    And how about when an SMP system isn't enough?
> 
> Demonstrate this.
> 
> Putting the whole implementation on the cards firmware is feasible,
> you don't need SMP.  It's totally doable and Linux needs to see
> none of the details.
> 
> Franks a lot,
> David S. Miller
> davem@redhat.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Tcp/ip offload card driver

[Joel Jaeggli]

One would expect that hyper-transport would make using the cpu more 
attractive rather than less since it eleminates many of the shared 
resources that are currently bottlenecks in smp machines.

joelja

 On Fri, 10 May 
2002, Nicholas Harring wrote:

> And how about when an SMP system isn't enough? Should I have to 
> re-engineer my network storage architecture when hardware exists that'll 
> increase throughput if a simple device driver gets written? Don't forget 
> that with 64 bit PCI that the limit of the bus has been raised, and with 
> impending technologies like Infiniband and Hypertransport that limit 
> will be raised again. At that point devoting main processor resources to 
> something better handled by specialty hardware really stops making 
> sense, if that specialty hardware is low-cost (oughta be) and effective 
> (still debatable).
> 
> Nicholas Harring
> Hostway Corporation
> 
> 
> Jeff Garzik wrote:
> > Pedro M. Rodrigues wrote:
> > 
> >>   Actually there is. Think iSCSI. Have a look at this article at 
> >> LinuxJournal - http://linuxjournal.com/article.php?sid=4896 .
> >>
> > 
> > Ug...  why bother?  Just buy an SMP system at that point...
> > 
> > 
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
-------------------------------------------------------------------------- 
Joel Jaeggli	      Academic User Services   joelja@darkwing.uoregon.edu    
--    PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E      --
  In Dr. Johnson's famous dictionary patriotism is defined as the last
  resort of the scoundrel.  With all due respect to an enlightened but
  inferior lexicographer I beg to submit that it is the first.
	   	            -- Ambrose Bierce, "The Devil's Dictionary"

Re: Tcp/ip offload card driver

[john slee]

On Fri, May 10, 2002 at 10:51:06AM -0500, Nicholas Harring wrote:
> And how about when an SMP system isn't enough? Should I have to 
> re-engineer my network storage architecture when hardware exists that'll 
> increase throughput if a simple device driver gets written? Don't forget 
> that with 64 bit PCI that the limit of the bus has been raised, and with 

jeff merkey has already demonstrated 300MiB/sec and higher speeds on x86
linux, with 3ware raid and dolphin sci cards.  how much faster do you
need to go?

j.

-- 
R N G G   "Well, there it goes again... And we just sit 
 I G G G   here without opposable thumbs." -- gary larson

Re: Tcp/ip offload card driver

[Mr. James W. Laferriere]

	Hello John ,  Think about it ;-) .  Maybe an order of magnitude ?
		JimL

On Sun, 12 May 2002, john slee wrote:
> On Fri, May 10, 2002 at 10:51:06AM -0500, Nicholas Harring wrote:
> > And how about when an SMP system isn't enough? Should I have to
> > re-engineer my network storage architecture when hardware exists that'll
> > increase throughput if a simple device driver gets written? Don't forget
> > that with 64 bit PCI that the limit of the bus has been raised, and with
>
> jeff merkey has already demonstrated 300MiB/sec and higher speeds on x86
> linux, with 3ware raid and dolphin sci cards.  how much faster do you
> need to go?
>
> j.
>
> --
> R N G G   "Well, there it goes again... And we just sit
>  I G G G   here without opposable thumbs." -- gary larson
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

       +------------------------------------------------------------------+
       | James   W.   Laferriere | System    Techniques | Give me VMS     |
       | Network        Engineer |     P.O. Box 854     |  Give me Linux  |
       | babydr@baby-dragons.com | Coudersport PA 16915 |   only  on  AXP |
       +------------------------------------------------------------------+

Re: Tcp/ip offload card driver

[Eric W. Biederman]

Jeff Garzik <jgarzik@mandrakesoft.com> writes:

> Pedro M. Rodrigues wrote:
> 
> >   Actually there is. Think iSCSI. Have a look at this article at LinuxJournal
> > - http://linuxjournal.com/article.php?sid=4896 .
> >
> 
> Ug...  why bother?  Just buy an SMP system at that point...

Or equally fix the kernel driver to do interrupt mitigation.  From
the article that looks like all they have managed to achieve.  There
may be a valid argument buried in there for remote DMA as well.

But given in the low contention case the kernel with it's own network
drivers was twice as fast.  tcp/ip offload looks to have some serious
weaknesses even for iSCSI.  The embedded processor kept better
performance going for just a little while but then it's performance
crashed as well.

Plus there is the general rule.  The primary CPU, it's memory, and
it's I/O subsystem improve out of necessity, while IO processors
stagnate, because they can.  The only exception to this I have seen
are graphics coprocessors.

Eric

Re: Tcp/ip offload card driver

[Mark Hahn]

>    Actually there is. Think iSCSI. Have a look at this article at 
> LinuxJournal - http://linuxjournal.com/article.php?sid=4896 .

no mention of ANY aspect of their tested configuration or workload.  
in short: zero content marketing fluff.

Re: Tcp/ip offload card driver

[Jeff Garzik]

David S. Miller wrote:

>   From: "chen, xiangping" <chen_xiangping@emc.com>
>   Date: Fri, 10 May 2002 10:48:23 -0400
>
>   Is there any TCP offload (TOE) card driver available on Linux?
>   
>Why do you want it?  There is no proven performance benefit.
>
>PCI bandwidth is the limiting factor for networking performance.
>


Linux TCP implementation will always be more powerful and more flexible 
than any NIC, too.  I doubt they have netlink and netfilter on NICs, for 
example :)

    Jeff

Re: Tcp/ip offload card driver

[David S. Miller]

   From: Jeff Garzik <jgarzik@mandrakesoft.com>
   Date: Fri, 10 May 2002 11:12:10 -0400
   
   Linux TCP implementation will always be more powerful and more flexible 
   than any NIC, too.  I doubt they have netlink and netfilter on NICs, for 
   example :)

It has the same problem as proprietary implementations of the BSD
stack, same bugs and same enhancements done N-times instead of once.

Anyone who thinks that having a different TCP implementation on each
different kind of network card installed on your system is sane, would
you please pass it on brotha so I can smoke some of it too! :-)

On a more serious note, it might be at some level considerable (the
maintainence nightmare et al.) if there was some real life
demonstrable performance gain with current systems.

For example, do a SpecWEB run with TUX both using on-chip-TCP and
without, same networking card.  Show a demonstrable gain from the
on-chip-TCP implementation.  I bet you can't.  If you can make such a
claim using a setup that other people could reproduce themselves by
buying your card and running the test, I'll eat all of my words.

Re: Tcp/ip offload card driver

[Lincoln Dale]

At 08:07 AM 10/05/2002 -0700, David S. Miller wrote:
>if there was some real life
>demonstrable performance gain with current systems.
>
>For example, do a SpecWEB run with TUX both using on-chip-TCP and
>without, same networking card.  Show a demonstrable gain from the
>on-chip-TCP implementation.  I bet you can't.  If you can make such a
>claim using a setup that other people could reproduce themselves by
>buying your card and running the test, I'll eat all of my words.

i think its conceivable that TOE could be advantagous in some situations.

for example, if TOE was supported in a driver (eg. /dev/toeN) which allowed 
user-space to mmap() into RAM (either on the card or main memory which the 
TOE card DMAs into/out-of).
in that way, a read()-equvalent system-call on a TOE socket could 
potentially just use 1 or 2 memory-accesses versus the 3 that currently occur.
translation: 33% or 50% less traffic across the front-side-bus, no 
pollution of the processor L1/L2 cache as a side-effect of copy_to_user().

in terms of a write()-like system-call, the equivalent holds true also.
right now, a write() means 3 memory-accesses:
  - 2 memory-accesses (memory-read and memory-write) on copy_from_user() 
from userspace to kernel
  - memory-read on NIC DMAing from kernel buffer
its forseeable that something that this could be 1 or 2 memory-accesses in 
a kernel-memory-mmap()ed-to-userspace scenario.

the biggest advantage is the scenario where userspace doesn't care about 
the actual payload of the data.
ie. its reading from-disk and sending to-network, or read-from-socket, 
write-to-another-socket.

of course, if TOE adapters don't work this way and don't have some API of 
their own exposed to allow this, then i'd agree that the benefit is negligible.
and of course, if we did get around to some kind of async i/o scheme in 
linux that allowed the above, then TOE would be of limited value also.
of course, such a scheme would probably be too hard to play fair with the 
page-cache, so it'd only be a "win" for data that is touched once and/or 
the application needs to do its own page-cache-like function.

i guess this is some of the points i'm trying to show with high-speed Fibre 
Channel HBAs -- the overhead of copy_to_user(), copy_from_user() means that 
the front-side-bus/memory-bandwidth ultimately becomes the main bottleneck, 
at least in IA32-architecture machines.


cheers,

lincoln.

Re: Tcp/ip offload card driver

[David S. Miller]

   From: Lincoln Dale <ltd@cisco.com>
   Date: Sat, 11 May 2002 11:53:33 +1000

   for example, if TOE was supported in a driver (eg. /dev/toeN) which allowed 
   user-space to mmap() into RAM (either on the card or main memory which the 
   TOE card DMAs into/out-of).

This is a very old concept, see UNET.

But that is %100 outside the realm of the TCP on a card stuff.
You don't even need firmware to do UNET, you just export the
packet descriptor rings into userspace.  Like I said, it's a
"been there, done that" technology.

Franks a lot,
David S. Miller
davem@redhat.com

Re: Tcp/ip offload card driver

[Nivedita Singhvi]

> Obviously some form of driver is necessary to access the
> device, whether or not we're pushing fully formed IP packets
> or raw payload. Or is that a userland problem and I'm just
> not understanding the flow from userspace through the kernel
> and to the driver properly?

> Cheers,
> Nicholas Harring

Your initial premise seemed to include the offload of TCP
as well. Doesn't that mean:

application -> driver -> card [ creates full TCP/IP pkt ]

TCP is stateful, feature rich and highly configurable.
Do you expect that a fw/hw implementation will provide
an equivalent implementation?  Support for tuning, options,
network taps, diag, bug fixes, feature tweaks, ... ?

Very curious..

thanks,
Nivedita

Re: Tcp/ip offload card driver

[Pete Zaitcev]

>[...]
> For example, do a SpecWEB run with TUX both using on-chip-TCP and
> without, same networking card.  Show a demonstrable gain from the
> on-chip-TCP implementation.  I bet you can't.

NO! Doing such a test sets you up for a failure. If a vendor
of the card provides an on-chip TCP, it is entirely in the
vendor's interest to penalize regular TCP (for example, by
failing to provide checksum offload or sane S/G segments).

I only consider fair a test of on-chip TCP compared to
the best of the normal NICs.

-- Pete

Re: Tcp/ip offload card driver

[David S. Miller]

   From: Pete Zaitcev <zaitcev@redhat.com>
   Date: Fri, 10 May 2002 13:55:52 -0400

   > For example, do a SpecWEB run with TUX both using on-chip-TCP and
   > without, same networking card.  Show a demonstrable gain from the
   > on-chip-TCP implementation.  I bet you can't.
   
   NO! Doing such a test sets you up for a failure. If a vendor
   of the card provides an on-chip TCP, it is entirely in the
   vendor's interest to penalize regular TCP (for example, by
   failing to provide checksum offload or sane S/G segments).
   
   I only consider fair a test of on-chip TCP compared to
   the best of the normal NICs.
   
Sorry, I should have stated this explicitly.  The same card
must have SG/Checksumming capability for the no-TCP-onchip portion of
the test.

re: Tcp/ip offload card driver

[Nivedita Singhvi]

>    > For example, do a SpecWEB run with TUX both using on-chip-TCP and
>    > without, same networking card.  Show a demonstrable gain from the
>    > on-chip-TCP implementation.  I bet you can't.
> 
>    NO! Doing such a test sets you up for a failure. If a vendor
>    of the card provides an on-chip TCP, it is entirely in the
>    vendor's interest to penalize regular TCP (for example, by
>    failing to provide checksum offload or sane S/G segments).
> 
>    I only consider fair a test of on-chip TCP compared to
>    the best of the normal NICs.
 
> Sorry, I should have stated this explicitly.  The same card
> must have SG/Checksumming capability for the no-TCP-onchip portion of
> the test.

Not to belabor the point, but most SpecWeb runs also turn off features
like timestamps, window scaling, sack, etc. ie. things you sometimes
need in the real world. 

That's representative of the problem, really. They could probably
work up a fw/hw implementation for a specific workload, benchmark...

Its much harder to be general purpose, feature rich, fully compliant
in hw than in sw. 

thanks,
Nivedita

RE: Tcp/ip offload card driver

[Woodruff, Robert J]

>Don't forget 
>that with 64 bit PCI that the limit of the bus has been raised, and with 
>impending technologies like Infiniband and Hypertransport that limit 
>will be raised again.

If people are interesting in discussing ideas on how InfiniBand 
networking (sockets direct (SDP) and IP over InfiniBand) should be/could 
be implemented and you are planning on attending the Linux Symposium
in Ottawa in June, there is an InfiniBand BOF session where we could 
discuss and exchange ideas on this topic. 

Re: Tcp/ip offload card driver

[Pete Zaitcev]

>>Don't forget 
>>that with 64 bit PCI that the limit of the bus has been raised, and with 
>>impending technologies like Infiniband and Hypertransport that limit 
>>will be raised again.
> 
> If people are interesting in discussing ideas on how InfiniBand 
> networking (sockets direct (SDP) and IP over InfiniBand) should be/could 
> be implemented and you are planning on attending the Linux Symposium
> in Ottawa in June, there is an InfiniBand BOF session where we could 
> discuss and exchange ideas on this topic. 

The more people, the merrier, but I am afraid that may subvert
the BOF. The problem is that nobody showed any convincing
proof that TOE actually works with benchmarks, let alone
in the real life. It was all a pretty groundless speculation so far.
I was on a conf call a week ago, where a guy said: "We implemented
TOE in the NT stack and achieved 100% speed-up", and I was like
"huh? and it tells me what? That NT stack was crap to begin with?"
I would like to keep the BOF on practical topics of Infiniband
implementation in Linux by any means necessary. TOE people
are welcome to come back when they have something working,
and when we know how fast the regular IP over Infiniband can go.

-- Pete

Re: Tcp/ip offload card driver

[Lincoln Dale]

IA32 isn't going to get you much more.  you need a new architecture.

300mbyte/sec is getting awfully close to real-life PCI limitations on IA32 
and real-world memory throughput
sure, the latest chipsets with DDR memory and memory-interleaving and 
things like HyperTransport, but they have an awful long way to reach your 
"order of magnitude".

i suspect you haven't reaized that "300MiB/sec" below means "300mbyte/sec" 
and not "300mbit/s".

At 01:36 PM 12/05/2002 -0400, Mr. James W. Laferriere wrote:

>         Hello John ,  Think about it ;-) .  Maybe an order of magnitude ?
>                 JimL
>
>On Sun, 12 May 2002, john slee wrote:
> > On Fri, May 10, 2002 at 10:51:06AM -0500, Nicholas Harring wrote:
> > > And how about when an SMP system isn't enough? Should I have to
> > > re-engineer my network storage architecture when hardware exists that'll
> > > increase throughput if a simple device driver gets written? Don't forget
> > > that with 64 bit PCI that the limit of the bus has been raised, and with
> >
> > jeff merkey has already demonstrated 300MiB/sec and higher speeds on x86
> > linux, with 3ware raid and dolphin sci cards.  how much faster do you
> > need to go?
> >
> > j.
> >
> > --
> > R N G G   "Well, there it goes again... And we just sit
> >  I G G G   here without opposable thumbs." -- gary larson
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
>
>        +------------------------------------------------------------------+
>        | James   W.   Laferriere | System    Techniques | Give me VMS     |
>        | Network        Engineer |     P.O. Box 854     |  Give me Linux  |
>        | babydr@baby-dragons.com | Coudersport PA 16915 |   only  on  AXP |
>        +------------------------------------------------------------------+
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

RE: Tcp/ip offload card driver

[Woodruff, Robert J]

>I would like to keep the BOF on practical topics of Infiniband
>implementation in Linux by any means necessary. TOE people
>are welcome to come back when they have something working,
>and when we know how fast the regular IP over Infiniband can go.

I agree that we should stick mainly to discussing InfiniBand 
at the LSM BOF and let the TOE discussion happen elsewhere,
although the issues of implementing InfiniBand SDP and 
TOE in the kernel are similar as they both want a way to
bypass the S/W TCP stack, but with SDP this could simply be
a new address family, where the TOE guys want a way to dynamically
replace the TCP stack with a TOE driver. 


However, perhaps we should start a new email 
thread to discuss topics of interest for the LSM BOF 
rather than highjack this thread to do so.