Re: TOE brain dump

Re: TOE brain dump

[jamal]

Can you please post to netdev? Posting networking related issues to
linux kernel alone is considered rude. Posting them to netdev only
is acceptable.

>  Ihar 'Philips' Filipau wrote:
>
> >Werner Almesberger wrote:
> > Ihar 'Philips' Filipau wrote:
> >
> >| | |  Modern NPUs generally do this.
> >
> >
> > Unfortunately, they don't - they run *some* code, but that
> > is rarely a Linux kernel, or a substantial part of it.
> >
>
>    Embedded CPU we are using is based MIPS, and has a lot of specialized
> instructions.
>    It makes not that much sense to run kernel (especially Linux) on CPU
> which is optimized for handling of network packets. (And has actually
> several co-processors to help in this task).

The coprocessors are useful, but that has nothing to do with the value 
of the NPU. You can add those within a general processor system.
I am also in the camp that to be really useful these things need to run
a real OS - Linux.

>    How much sense it makes to run general purpose OS (optimized for PCs
> and servers) on devices which can make only couple of functions? (and no
> MMU btw)
>
>    It is a whole idea behind this kind of CPUs - to do a few of
> functions - but to do them good.
>
>    If you will start stretching CPUs like this to fit Linux kernel - it
> will generally just increase price. Probably there are some markets
> which can afford this.
>

Actually i believe it will lower the prices.I am waiting for intel to get 
hyperthreading right - then we'll see these things disapear.
The only thing useful about NPUs is their ability to management the
discrepency between memory latency and CPU speeds. Trust me i used to
be in the same camp as you.If you note, a lot of these things appeared
around the height of the .com days. VCs were looking for something
new and exciting.

>    Remeber - "Small is beatiful" (c) - and linux kernel far from it.
>    Our routing code which handles two GE interfaces (actually not pure
> GE, but up to 2.5GB) fits into 3k. 3k of code - and that's it. not 650kb
> of bzip compressed bloat. And it handles two interfaces, handles fast
> data path from siblign interfaces, handles up to 1E6 routes. 3k of code.
> not 650k of bzip.

If all you wanted was to do L3 - why not just buy a $5 chip that can do this 
for a lot more interfaces? Why sweat over optimizing L3 routing in a 
3K space? 
to nit: Its no longer about routing or bridging, friend. Thats like getting 
fries at mcdonalds.

cheers,
jamal 

Re: TOE brain dump

[Ihar 'Philips' Filipau]

jamal wrote:
> to nit: Its no longer about routing or bridging, friend. Thats like getting 
> fries at mcdonalds.
> 

   1GE/10GE - for $5?
   I'm first in the shoping queue!!!-)))

   Since I see no reasonable out-come of this discussion I left it.

   TOE as I see - since my company utilizes several of them - are too 
different and too specialized to application/protocols. And yes - price 
of development/deployment maters too. Linux support for those protocols 
is inmature. It cannot handle or requirements even software-wise. I'm 
not talking about timing requirements - linux network in general is not 
(even soft) real-time.

   My personal flame-meter is out of scale ;-)
   I shall join the discussion back when I will see any real ideas.


 > If all you wanted was to do L3 - why not just buy a $5 chip that
 > can do this for a lot more interfaces? Why sweat over
 > optimizing L3 routing in a 3K space?

   We are doing not a teapot, and high level spec for this code takes 
around 15 pages.
   3k - it is not optimized - we have limit around 2GB ;-)
   It just takes only 3k. And it handles some special (read - 
proprietary) functions too - some bugs of some other pieces of hardware. 
NPU does all stuff by itself, but sometimes we need to extract 
configuration information which is direct to us, for example.

Re: TOE brain dump

[jamal]

On Mon, 2003-08-04 at 14:48, Ihar 'Philips' Filipau wrote:
> jamal wrote:
> > to nit: Its no longer about routing or bridging, friend. Thats like getting 
> > fries at mcdonalds.
> > 
> 
>    1GE/10GE - for $5?
>    I'm first in the shoping queue!!!-)))
> 

I thought you were talking about a 2 Gige interface doing routing, no?
Do the math: Dell will happily sell you a (managed?) switch which has
8Giges on it for about $300. It does wire rate on all 8 interfaces. All
ready to go in a 1U form factor. How much do you think that chip costs?
Lets say it doesnt do L3, how much more do you think it will cost to do
L3 in quantities?

>    Since I see no reasonable out-come of this discussion I left it.
> 
>    TOE as I see - since my company utilizes several of them - are too 
> different and too specialized to application/protocols. And yes - price 
> of development/deployment maters too. Linux support for those protocols 
> is inmature. It cannot handle or requirements even software-wise. I'm 
> not talking about timing requirements - linux network in general is not 
> (even soft) real-time.
> 

Now this is anti-social talk;-> Why do you need to have realtime for any
of this stuff? 

>    My personal flame-meter is out of scale ;-)
>    I shall join the discussion back when I will see any real ideas.
> 

Please dont dissapear, a lot of questions need answers;->

> 
>  > If all you wanted was to do L3 - why not just buy a $5 chip that
>  > can do this for a lot more interfaces? Why sweat over
>  > optimizing L3 routing in a 3K space?
> 
>    We are doing not a teapot, and high level spec for this code takes 
> around 15 pages.
>    3k - it is not optimized - we have limit around 2GB ;-)

I am really confused now. We must be talking about different class of
devices. NPUs as i know them are very limited in how much code you can
stash them. In the 10K ranges is already overkill.
Do you have any URL i can look at on what you are describing?

>    It just takes only 3k. And it handles some special (read - 
> proprietary) functions too - some bugs of some other pieces of hardware. 
> NPU does all stuff by itself, but sometimes we need to extract 
> configuration information which is direct to us, for example.

Please provide me a pointer if you can - I am very interested in the 2G
code space you mention.

cheers,
jamal

> 

Re: TOE brain dump

[Ihar 'Philips' Filipau]

jamal wrote:
> 
>>   It just takes only 3k. And it handles some special (read - 
>>proprietary) functions too - some bugs of some other pieces of hardware. 
>>NPU does all stuff by itself, but sometimes we need to extract 
>>configuration information which is direct to us, for example.
> 
> 
> Please provide me a pointer if you can - I am very interested in the 2G
> code space you mention.
> 

   I'm not sure - actually as I wrote - immediately gone checking specs.
   try: 
http://www.vitesse.com/products/categories.cfm?family_id=5&category_id=16

   ...

   [ Okay I got to docs server. ] You are right - It has limit of 4K 
insns == 16k of executable memory.

   Sorry for confusion :(

   We really can address a lot of memory - we have 32MB for routing info 
and configuration - but for execution only 16kB of memory is available...

RE: TOE brain dump

[Perez-Gonzalez, Inaky]

> From: Larry McVoy [mailto:lm@bitmover.com]
>
> > 2. router nodes that have access to main memory (PCI card running linux
> > acting as a router/firewall/VPN to offload the main CPU's)
> 
> I can get an entire machine, memory, disk, > Ghz CPU, case, power supply,
> cdrom, floppy, onboard enet extra net card for routing, for $250 or less,
> quantity 1, shipped to my door.
> 
> Why would I want to spend money on some silly offload card when I can get
> the whole PC for less than the card?

Because you want to stack 200 of those together in a huge
data center interconnecting whatever you want to interconnect
and you don't want your maintenance costs to go up to the sky?

I see your point, though :)

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

RE: TOE brain dump

[Alan Cox]

On Llu, 2003-08-04 at 19:36, Perez-Gonzalez, Inaky wrote:
> > Why would I want to spend money on some silly offload card when I can get
> > the whole PC for less than the card?
> 
> Because you want to stack 200 of those together in a huge
> data center interconnecting whatever you want to interconnect
> and you don't want your maintenance costs to go up to the sky?

17cm squared, fanless, network booting. Its not as big a cost as
you might think, and TOE cards fail too, the difference being that if
they are now out of production you have a nasty mess on your hands.

TOE brain dump

[Werner Almesberger]

At OLS, there was a bit of discussion on (true and false *) TOEs
(TCP Offload Engines). In the course of this discussion, I've
suggested what might be a novel approach, so in case this is a
good idea, I'd like to dump my thoughts on it, before someone
tries to patent my ideas. (Most likely, some of this has already
been done or tried elsewhere, but it can't hurt to try to err on
the safe side.)

(*) The InfiniBand people unfortunately call also their TCP/IP
    bypass "TOE" (for which they promptly get shouted down,
    every time they use that word). This is misleading, because
    there is no TCP that's getting offloaded, but TCP is simply
    never done. I would consider it to be more accurate to view
    this as a separate networking technology, with semantics
    different from TCP/IP, similar to ATM and AAL5.

While I'm not entirely convinced about the usefulness of TOE in
all the cases it's been suggested for, I can see value in certain
areas, e.g. when TCP per-packet overhead becomes an issue.

However, I consider the approach of putting a new or heavily
modified stack, which duplicates a considerable amount of the
functionality in the main kernel, on a separate piece of hardware
questionable at best. Some of the issues:

 - if this stack is closed source or generally hard to modify,
   security fixes will be slowed down

 - if this stack is closed source or generally hard to modify,
   TOE will not be available to projects modifying the stack,
   e.g. any of the research projects trying to make TCP work at
   gigabit speeds

 - this stack either needs to implement all administrative
   interfaces of the regular kernel, or such a system would have
   non-uniform configuration/monitoring across interfaces

 - in some cases, administrative interfaces will require a
   NIC/TOE-specific switch in the kernel (netlink helps here)

 - route changes on multi-homed hosts (or any similar kind of
   failover) are difficult if the state of TCP connections is
   tied to specific NICs (I've discussed some issues when
   "migrating" TCP connections in the documentation of tcpcp,
   http://www.almesberger.net/tcpcp/)

 - new kernel features will always lag behind on this kind of
   TOE, and different kernels will require different "firmware"

 - last but not least, keeping TOE firmware up to date with the
   TCP/IP stack in the mainstream kernel will require - for each
   such TOE device - a significant and continuous effort over a
   long period of time

In short, I think such a solution is either a pain to use, or
unmaintainable, or - most likely - both.

So, how to do better ? Easy: use the Source, Luke. Here's my
idea:

 - instead of putting a different stack on the TOE, a
   general-purpose processor (probably with some enhancements,
   and certainly with optimized data paths) is added to the NIC

 - that processor runs the same Linux kernel image as the host,
   acting like a NUMA system

 - a selectable part of TCP/IP is handled on the NIC, and the
   rest of the system runs on the host processor

 - instrumentation is added to the mainstream kernel to ensure
   that as little data as possible is shared between the main
   CPU and such peripheral CPUs. Note that such instrumentation
   would be generic, outlining possible boundaries, and not tied
   to a specific TOE design.

 - depending on hardware details (cache coherence, etc.), the
   instrumentation mentioned above may even be necessary for
   correctness. This would have the unfortunate effect of making
   the design very fragile with respect to changes in the
   mainstream kernel. (Performance loss in the case of imperfect
   instrumentation would be preferable.)

 - further instrumentation may be needed to let the kernel switch
   CPUs (i.e. host to NIC, and vice versa) at the right time

 - since the NIC would probably use a CPU design different from
   the host CPU, we'd need "fat" kernel binaries:

   - data structures are the same, i.e. word sizes, byte order,
     bit numbering, etc. are compatible, and alignments are
     chosen such that all CPUs involved are reasonably happy

   - kernels live in the same address space

   - function pointers become arrays, with one pointer per
     architecture. When comparing pointers, the first element is
     used.

 - if one should choose to also run parts of user space on the
   NIC, fat binaries would also be needed for this (along with
   other complications)

Benefits:

 - putting the CPU next to the NIC keeps data paths short, and
   allows for all kinds of optimizations (e.g. a pipelined
   memory architecture)

 - the design is fairly generic, and would equally apply to
   other areas of the kernel than TCP/IP

 - using the same kernel image eliminates most maintenance
   problems, and encourages experimenting with the stack

 - using the same kernel image (and compatible data structures)
   guarantees that administrative interfaces are uniform in the
   entire system

 - such a design is likely to be able to allow TCP state to be
   moved to a different NIC, if necessary

Possible problems, that may kill this idea:

 - it may be too hard to achieve correctness

 - it may be too hard to switch CPUs properly

 - it may not be possible to express copy operations efficiently
   in such a context

 - there may be no way to avoid sharing of hardware-specific
   data structures, such as page tables, or to emulate their use

 - people may consider the instrumentation required for this,
   although fairly generic, too intrusive

 - all this instrumentation may eat too much performance

 - nobody may be interested in building hardware for this

 - nobody may be patient enough to pursue such long-termish
   development, with uncertain outcome

 - something I haven't thought of

I lack the resources (hardware, financial, and otherwise) to
actually do something with these ideas, so please feel free to
put them to some use.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[Nivedita Singhvi]

Werner Almesberger wrote:

> (*) The InfiniBand people unfortunately call also their TCP/IP
>     bypass "TOE" (for which they promptly get shouted down,
>     every time they use that word). This is misleading, because

Thank you! Yes! All in favor say Aye..AYE!!! Motion passes,
the infiniband people don't get to call it TOE anymore..

> While I'm not entirely convinced about the usefulness of TOE in
> all the cases it's been suggested for, I can see value in certain
> areas, e.g. when TCP per-packet overhead becomes an issue.

Ditto, but I see it being used to rollout the idea and process,
rather than anything of value now, and the lessons are being
learned for the future, when we reach 20Gb, 40Gb, even faster
networks of tommorow. The processors might keep up, but nothing
else will, for sure.

> However, I consider the approach of putting a new or heavily
> modified stack, which duplicates a considerable amount of the
> functionality in the main kernel, on a separate piece of hardware
> questionable at best. Some of the issues:
> 
>  - if this stack is closed source or generally hard to modify,
>    security fixes will be slowed down

as will bug fixes, and debugging becomes a right royal pain.

Also, most profiles of networking applications show the
largest blip is essentially the user<->kernel transfer, and
that would still remain the unaddressed bottleneck.

> So, how to do better ? Easy: use the Source, Luke. Here's my
> idea:
> 
>  - instead of putting a different stack on the TOE, a
>    general-purpose processor (probably with some enhancements,
>    and certainly with optimized data paths) is added to the NIC

The thing is, all the TOE efforts are propietary ones, to
my limited knowledge. Thus all the design is occurring in
confidential, vendor internal forums. How will they/we
come up with really the needed, _common_ design approach?

Or is this not so needed?

thanks,
Nivedita

Re: TOE brain dump

[Werner Almesberger]

Nivedita Singhvi wrote:
> Also, most profiles of networking applications show the
> largest blip is essentially the user<->kernel transfer, and
> that would still remain the unaddressed bottleneck.

I have some hope that sendfile plus a NUMA-like approach will be
sufficient for keeping transfers away from buses and memory they
don't need to touch.

> The thing is, all the TOE efforts are propietary ones, to
> my limited knowledge.

Many companies default to "closed" designs if they're not given a
convincing reason for going "open". The approach I've described
may provide that reason.

There are also historicial reasons, e.g. if you want to interface
with the stack of Windows, or any proprietary Unix, you probably
need to obtain some of their source under NDA, and use some of
that information in your own drivers or firmware. Of course, none
of this is an issue here.

Since we're talking about 1-2 years of development time anyway,
legacy hardware (i.e. hardware choices influenced by information
obtained under an NDA) will be quite obsolete by then and doesn't
matter.

> Or is this not so needed?

Exactly. The "NUMA" approach would avoid the "common TOE design"
problem.

All you need is a reasonably well documented "general-purpose"
CPU (that doesn't mean it has to be an off-the-shelf design, but
most likely, the core would be an off-the-shelf one), plus some
NIC hardware. Now, if that NIC in turn has some hidden secrets,
this isn't an issue as long as one can still write a GPLed driver
for it.

Of course, there would be elements in such a system that vendors
would like to keep secret. But then, there always are, and so far,
we've found reasonable compromises most of the time, so I don't
see why this couldn't happen here, too.

Also, if "classical TOE" patches keep getting rejected, but an
open and maintainable approach makes it into the mainstream
kernel, also the business aspects should become fairly clear.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[Jeff Garzik]

My own brain dump:

If one wants to go straight from disk to network, why is anyone 
bothering to involve the host CPU and host memory bus at all?  Memory 
bandwidth and PCI bus bandwidth are still bottlenecks, no much how much 
of the net stack you offload.


Regardless of how fast your network zooms packets, you've gotta keep 
that pipeline full to make use of it.  And you've gotta do something 
intelligent with it, which in TCP's case involves the host CPU quite a 
bit.  TCP is sufficiently complex, for a reason.  It has to handle all 
manner of disturbingly slow and disturbing fast net connections, all 
jabbering at the same time.  TCP is a "one size fits all" solution, but 
it doesn't work well for everyone.

The "TCP Offload Everything" people really need to look at what data 
your users want to push, at such high speeds.  It's obviously not over a 
WAN...  so steer users away from TCP, to an IP protocol that is tuned 
for your LAN needs, and more friendly to some sort of h/w offloading 
solution.

A "foo over ipv6" protocol that was designed for h/w offloading from the 
start, would be a far better idea than full TCP offload will ever be.

In any case, when you approach these high speeds, you really must take a 
good look at the other end of the pipeline:  what are you serving at 
10Gb/s, 20Gb/s, 40Gb/s?  For some time, I think the answer will be 
"highly specialized stuff"  At some point, Intel networking gear will be 
able to transfer more bits per second than there exist atoms on planet 
Earth :)  Garbage in, garbage out.

So, fix the other end of the pipeline too, otherwise this fast network 
stuff is flashly but pointless.  If you want to serve up data from disk, 
then start creating PCI cards that have both Serial ATA and ethernet 
connectors on them :)  Cut out the middleman of the host CPU and host 
memory bus instead of offloading portions of TCP that do not need to be 
offloaded.

	Jeff

Re: TOE brain dump

[Werner Almesberger]

Jeff Garzik wrote:
> jabbering at the same time.  TCP is a "one size fits all" solution, but 
> it doesn't work well for everyone.

But then, ten "optimized xxPs" that work well in two different
scenarios each, but not so good in the 98 others, wouldn't be
much fun either.

It's been tried a number of times. Usually, real life sneaks
in at one point or another, leaving behind a complex mess.
When they've sorted out these problems, regular TCP has caught
up with the great optimized transport protocols. At that point,
they return to their niche, sometimes tail between legs and
muttering curses, sometimes shaking their fist and boldly
proclaiming how badly they'll rub TCP in the dirt in the next
round. Maybe they shed off some of the complexity, and trade it
for even more aggressive optimization, which puts them into
their niche even more firmly. Eventually, they fade away.

There are cases where TCP doesn't work well, like a path of
badly mismatched link layers, but such paths don't treat any
protocol following the end-to-end principle kindly.

Another problem of TCP is that it has grown a bit too many
knobs you need to turn before it works over your really fast
really long pipe. (In one of the OLS after dinner speeches,
this was quite appropriately called the "wizard gap".)

> It's obviously not over a WAN...

That's why NFS turned off UDP checksums ;-) As soon as you put
it on IP, it will crawl to distances you didn't imagine in your
wildest dreams. It always does.

> So, fix the other end of the pipeline too, otherwise this fast network 
> stuff is flashly but pointless.  If you want to serve up data from disk, 
> then start creating PCI cards that have both Serial ATA and ethernet 
> connectors on them :)  Cut out the middleman of the host CPU and host 
> memory bus instead of offloading portions of TCP that do not need to be 
> offloaded.

That's a good point. A hierarchical memory structure can help
here. Moving one end closer to the hardware, and letting it
know (e.g. through sendfile) that also the other end is close
(or can be reached more directly that through some hopelessly
crowded main bus) may help too.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[Jeff Garzik]

Werner Almesberger wrote:
> Jeff Garzik wrote:
> 
>>jabbering at the same time.  TCP is a "one size fits all" solution, but 
>>it doesn't work well for everyone.
> 
> 
> But then, ten "optimized xxPs" that work well in two different
> scenarios each, but not so good in the 98 others, wouldn't be
> much fun either.
> 
> It's been tried a number of times. Usually, real life sneaks
> in at one point or another, leaving behind a complex mess.
> When they've sorted out these problems, regular TCP has caught
> up with the great optimized transport protocols. At that point,
> they return to their niche, sometimes tail between legs and
> muttering curses, sometimes shaking their fist and boldly
> proclaiming how badly they'll rub TCP in the dirt in the next
> round. Maybe they shed off some of the complexity, and trade it
> for even more aggressive optimization, which puts them into
> their niche even more firmly. Eventually, they fade away.
> 
> There are cases where TCP doesn't work well, like a path of
> badly mismatched link layers, but such paths don't treat any
> protocol following the end-to-end principle kindly.
> 
> Another problem of TCP is that it has grown a bit too many
> knobs you need to turn before it works over your really fast
> really long pipe. (In one of the OLS after dinner speeches,
> this was quite appropriately called the "wizard gap".)
> 
> 
>>It's obviously not over a WAN...
> 
> 
> That's why NFS turned off UDP checksums ;-) As soon as you put
> it on IP, it will crawl to distances you didn't imagine in your
> wildest dreams. It always does.

Really fast, really long pipes in practice don't exist for 99.9% of all 
Internet users.


When you approach traffic levels that push you want to offload most of 
the TCP net stack, then TCP isn't the right solution for you anymore, 
all things considered.


The Linux net stack just isn't built to be offloaded.  TOE engines will 
either need to (1) fall back to Linux software for all-but-the-common 
case (otherwise netfilter, etc. break), or, (2) will need to be 
hideously complex beasts themselves.  And I can't see ASIC and firmware 
designers being excited about implementing netfilter on a PCI card :)

Unfortunately some vendors seem to choosing TOE option #3:  TCP offload 
which introduces many limitations (connection limits, netfilter not 
supported, etc.) which Linux never had before.  Vendors don't seem to 
realize TOE has real potential to damage the "good network neighbor" 
image the net stack has.  The Linux net stack's behavior is known, 
documented, predictable.  TOE changes all that.

There is one interesting TOE solution, that I have yet to see created: 
run Linux on an embedded processor, on the NIC.  This stripped-down 
Linux kernel would perform all the header parsing, checksumming, etc. 
into the NIC's local RAM.  The Linux OS driver interface becomes a 
virtual interface with a large MTU, that communicates from host CPU to 
NIC across the PCI bus using jumbo-ethernet-like data frames. 
Management frames would control the ethernet interface on the other side 
of the PCI bus "tunnel".


>>So, fix the other end of the pipeline too, otherwise this fast network 
>>stuff is flashly but pointless.  If you want to serve up data from disk, 
>>then start creating PCI cards that have both Serial ATA and ethernet 
>>connectors on them :)  Cut out the middleman of the host CPU and host 
>>memory bus instead of offloading portions of TCP that do not need to be 
>>offloaded.
> 
> 
> That's a good point. A hierarchical memory structure can help
> here. Moving one end closer to the hardware, and letting it
> know (e.g. through sendfile) that also the other end is close
> (or can be reached more directly that through some hopelessly
> crowded main bus) may help too.

Definitely.

	Jeff

Re: TOE brain dump

[Werner Almesberger]

Jeff Garzik wrote:
> Really fast, really long pipes in practice don't exist for 99.9% of all 
> Internet users.

It matters to some right now, i.e. the ones who are interested
in TOE in the first place. (And there also those who try to
tweak TCP to actually work over such links. Right now, its
congestion control doesn't scale that well.) Also, IT has been
good at making all that elitarian high-performance gear
available to the common people rather quickly, and I don't see
that changing. The Crisis just alters the pace a little.

> When you approach traffic levels that push you want to offload most of 
> the TCP net stack, then TCP isn't the right solution for you anymore, 
> all things considered.

No. Ironically, TCP is almost always the right solution. 
Sometimes people try to use something else. Eventually, their
protocol wants to go over WANs or something that looks
suspiciously like a WAN (MAN or such). At that point, they
usually realize that TCP provides exactly the functionality
they need.

In theory, one could implement the same functionality in other
protocols. There was even talk at IETF to support a generic
congestion control manager for this purpose. That was many
years ago, and I haven't seen anything come out of this.

So it seems that, by the time your protocol grows up to want
to play in the real world, it wants to be so much like TCP
that you're better off using TCP.

The amusing bit here is to watch all the "competitors" pop
up, grow, fail, and eventually die.

> The Linux net stack just isn't built to be offloaded.

Yes ! And that's not a flaw of the stack, but it's simply a
fact of life. I think that no "real life" stack can be
offloaded (in the traditional sense).

> And I can't see ASIC and firmware 
> designers being excited about implementing netfilter on a PCI card :)

And when they're done with netfilter, you can throw IPsec,
IPv6, or traffic control at them. Eventually, you'll wear
them down ;-)

> Unfortunately some vendors seem to choosing TOE option #3:  TCP offload 
> which introduces many limitations (connection limits, netfilter not 
> supported, etc.) which Linux never had before.

That's when that little word "no" comes into play, i.e.
when their modifications to the stack show up on netdev
or linux-kernel. Dave Miller seems to be pretty good at
saying "no". I hope he keeps on being good at this ;-)

> There is one interesting TOE solution, that I have yet to see created: 
> run Linux on an embedded processor, on the NIC.

That's basically what I've been talking about all the
while :-)

> The Linux OS driver interface becomes a virtual interface
> with a large MTU,

Probably not. I think you also want to push some
knowledge of where the data ultimately goes to the NIC.
This could be something like sendfile, something new, or
just a few bytes of user space code.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[Erik Andersen]

On Sun Aug 03, 2003 at 02:57:37PM -0300, Werner Almesberger wrote:
> > There is one interesting TOE solution, that I have yet to see created: 
> > run Linux on an embedded processor, on the NIC.
> 
> That's basically what I've been talking about all the
> while :-)

http://www.snapgear.com/pci630.html

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: TOE brain dump

[Larry McVoy]

On Sun, Aug 03, 2003 at 12:27:55PM -0600, Erik Andersen wrote:
> On Sun Aug 03, 2003 at 02:57:37PM -0300, Werner Almesberger wrote:
> > > There is one interesting TOE solution, that I have yet to see created: 
> > > run Linux on an embedded processor, on the NIC.
> > 
> > That's basically what I've been talking about all the
> > while :-)
> 
> http://www.snapgear.com/pci630.html

ipcop plus a new PC for $200 is way higher performance and does more.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: TOE brain dump

[David Lang]

On Sun, 3 Aug 2003, Larry McVoy wrote:

> On Sun, Aug 03, 2003 at 12:27:55PM -0600, Erik Andersen wrote:
> > On Sun Aug 03, 2003 at 02:57:37PM -0300, Werner Almesberger wrote:
> > > > There is one interesting TOE solution, that I have yet to see created:
> > > > run Linux on an embedded processor, on the NIC.
> > >
> > > That's basically what I've been talking about all the
> > > while :-)
> >
> > http://www.snapgear.com/pci630.html
>
> ipcop plus a new PC for $200 is way higher performance and does more.

however I can see situations where you would put multiple cards in one box
and there could be an advantage to useing PCI (or PCI-X) for you
communications between the different 'nodes' of you routing cluster
instead of gig ethernet.

if this is the approach that the networking guys really want to encourage
how about defining an API that you would be willing to support and you can
even implement it and then any card that is produced would be supported
from day 1.

this interface would not have to cover the configuration of the card (that
can be done with userspace tools that talk to the card over the 'network',
it just needs to cover the ability to do what is effectivly IP over PCI.

Linus has commented that in mahy ways Linux is not designed for any
existing CPU, it's designed for a virtual CPU that implements all the
features we want and those features that aren't implemented in the chips
get emulated as needed (obviously what is actually implemented and the
speed of emulation are serious considerations for performance) why doesn't
the network team define what they thing the ideal NIC interface would be.
I can see three catagories of 'ideal' cards

1. cards that are directly driven by the kernel IP stack (similar to what
we support now, but an ideal version)

2. router nodes that have access to main memory (PCI card running linux
acting as a router/firewall/VPN to offload the main CPU's)

3. router nodes that don't have access to main memory (things like
USB/fibrechannel/infiniband/etc versions of #2, the node can run linux and
deal with the outside world, only sending the data that is needed to/from
the host)

even if nobody makes hardware that supports all the desired features
directly having a 'this is the dieal driver' reference should impruve
furture drivers by letting them use this as the core and implementing code
to simulate the features not in hardware.

they claim they need this sort of performance, you say 'not that way do it
sanely' why not give them a sane way to do it?

David Lang

Re: TOE brain dump

[Larry McVoy]

On Sun, Aug 03, 2003 at 01:13:24PM -0700, David Lang wrote:
> 2. router nodes that have access to main memory (PCI card running linux
> acting as a router/firewall/VPN to offload the main CPU's)

I can get an entire machine, memory, disk, > Ghz CPU, case, power supply,
cdrom, floppy, onboard enet extra net card for routing, for $250 or less,
quantity 1, shipped to my door.

Why would I want to spend money on some silly offload card when I can get 
the whole PC for less than the card?
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: TOE brain dump

[David Lang]

On Sun, 3 Aug 2003, Larry McVoy wrote:

> On Sun, Aug 03, 2003 at 01:13:24PM -0700, David Lang wrote:
> > 2. router nodes that have access to main memory (PCI card running linux
> > acting as a router/firewall/VPN to offload the main CPU's)
>
> I can get an entire machine, memory, disk, > Ghz CPU, case, power supply,
> cdrom, floppy, onboard enet extra net card for routing, for $250 or less,
> quantity 1, shipped to my door.
>
> Why would I want to spend money on some silly offload card when I can get
> the whole PC for less than the card?

you may want to do this for a database box where you want to dedicate your
main processing power to the database task, if you use a seperate box you
still have to talk to that box over a network, if you have it as a card
you can talk to the card much more efficantly then you can talk to the
seperate machine.

if your 8-way opteron database box is already the bottleneck for your
system you will have to spend a LOT of money to get anything that gives
you more available processing power, getting a card to offload any
processing from the main CPU's can be a win.

yes this is somewhat of a niche market, but as you point out adding more
and more processors in a SMP model is not the ideal way to go, either from
performance or from the cost point of view.

on the webserver front there are a lot of companies making a living by
selling cards and boxes to offload processing from the main CPU's of the
webservers (cards to do gzip compression are a relativly new addition, but
cards to do SSL handshakes have been around for a while) used properly
these can be a very worthwhile invenstment for high-volume webserver
companies.

also the cost of an extra box can be considerably higer then just the cost
of the hardware.

I know of one situation where between Linux OS license fees (redhat
advanced server) and security software (intrusion detection, auditing,
privilage management, etc) a company is looking at ~$4000 in licensing
fees for every box they put in their datacenter (and this is for boxes
just running apache, add something like an oracle or J2EE appserver
software and the cost goes up even more). at this point the fact that the
box only cost $200 doesn't really matter, spending an extra $500 each on 4
boxes to eliminate the need for a 5th is easily worth it. (and this
company is re-examining hardwaare raid controllers after having run
software raid for years becouse they are realizing that this is requiring
them to run more servers due to the load on the CPU's)

at the low end you are right, just add another box or add another CPU to
an existing box, but there are conditions that make adding specialized
cards to offload specific functionality a win (for that matter, even at
the low end people routinly offload graphics processing to specialized
cards, simply to make their games run faster)

David Lang

Re: TOE brain dump

[Larry McVoy]

On Sun, Aug 03, 2003 at 02:21:12PM -0700, David Lang wrote:
> if your 8-way opteron database box is already the bottleneck for your
> system you will have to spend a LOT of money to get anything that gives
> you more available processing power, getting a card to offload any
> processing from the main CPU's can be a win.

I'd like to see data which supports this.  CPUs have gotten so fast and
disk I/O still sucks.  All the systems I've seen are CPU rich and I/O
starved.  The smartest thing you could do would be to get a cheap box
with a GB of ram as a disk cache and make it be a SAN device.  Make
N of those and you have tons of disk space and tons of cache and your
8 way opteron database box is going to work just fine.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: TOE brain dump

[Jeff Garzik]

Larry McVoy wrote:
> I can get an entire machine, memory, disk, > Ghz CPU, case, power supply,
> cdrom, floppy, onboard enet extra net card for routing, for $250 or less,
> quantity 1, shipped to my door.
> 
> Why would I want to spend money on some silly offload card when I can get 
> the whole PC for less than the card?


Yep.  I think we are entering the era of what I call RAIC (pronounced 
"rake") -- redundant array of inexpensive computers.  For organizations 
that can handle the space/power/temperature load, a powerful cluster of 
supercheap PCs, the "Wal-Mart Supercomputer", can be built for a 
rock-bottom price.

Re: TOE brain dump

[Timothy Miller]

Larry McVoy wrote:
> On Sun, Aug 03, 2003 at 01:13:24PM -0700, David Lang wrote:
> 
>>2. router nodes that have access to main memory (PCI card running linux
>>acting as a router/firewall/VPN to offload the main CPU's)
> 
> 
> I can get an entire machine, memory, disk, > Ghz CPU, case, power supply,
> cdrom, floppy, onboard enet extra net card for routing, for $250 or less,
> quantity 1, shipped to my door.
> 
> Why would I want to spend money on some silly offload card when I can get 
> the whole PC for less than the card?


Physical space?  Power usage?  Heat dissipation?  Optimization for the 
specific task?  Fast, low latency communication between CPU and device 
(ie. local bus)?  Maintenance?

Lots of reasons why one might pay more for the offload card.  If you're 
cheap, you'll just use the software stack and a $10 NIC and just live 
with the corresponding CPU usage.  If you're a performance freak, you'll 
spend whatever you have to to squeeze out every last bit of performance 
you can.

Mind you, another option is, if you're dealing with the kind of load 
that requires that much network performance, is to use redundant 
servers, like google.  No one server is exceptionally fast, but it not 
many people are using it, it's fast enough.

Re: TOE brain dump

[jamal]

On Sun, 2003-08-03 at 15:40, Larry McVoy wrote:
> On Sun, Aug 03, 2003 at 12:27:55PM -0600, Erik Andersen wrote:
> > On Sun Aug 03, 2003 at 02:57:37PM -0300, Werner Almesberger wrote:
> > > > There is one interesting TOE solution, that I have yet to see created: 
> > > > run Linux on an embedded processor, on the NIC.
> > > 
> > > That's basically what I've been talking about all the
> > > while :-)
> > 
> > http://www.snapgear.com/pci630.html
> 
> ipcop plus a new PC for $200 is way higher performance and does more.

;-> Actually this proves that putting the whole stack on the NIC is the
wrong way to go ;-> That poor piece of NIC was obsoleted before it was
born on pricing alone and not just compute power it was supposed to
liberate us from.

I think the idea of hierachical memories and computation is certainly
interesting. Put a CPU and memory on the NIC but not to do the work that
Linux already does. Instead think of the NIC and its memeory + CPU as a
L1 data and code cache for TCP processing. The idea posed from Davem is
intriguing:
The only thing the NIC should do is TCP fast path processing based on
cached control data generated from the main CPU stack. Any time the fast
path gets violated, the cache gets invalidate and it becomes an
exception handling to be handled by the main CPU stack.

IMO, the only time this will make sense is when the setup cost
(downloading the cache or cookies as Dave calls them) is amortized by
the data that follows. For example, may not make sense to worry about a
HTTP1.0 flow which has 3-4 packets after the SYNack.Bulk transfers make
sense (storage, file serving). I dont remember the Mogul paper details
but i think this is what he was implying.

cheers,
jamal

Re: TOE brain dump

[David S. Miller]

[ This discussion belongs on netdev, not linux-kernel. ]

On Mon, 04 Aug 2003 11:17:23 +0930
Glen Turner <glen.turner@aarnet.edu.au> wrote:

> That's Matt Mathis's phrase.  The Web100 project
> <http://www.web100.org/> has a set of patches to the kernel
> which go a long way to reducing the wizard gap.  It would be
> nice to see those patches eventually appear in the Linux
> mainstream.

The web100 patches aren't in the kernel because 1) they've
never even been submitted and 2) they need a large cleanup.

I sort of get the impression that the web100 folks actually like that
their changes are not in the main sources, it keeps their work
"special".

Re: TOE brain dump

[Andre Hedrick]

Jeff,

Do be sure to check that your data payload is correct.
Everyone knows that a router/gateway/switch with a sticky bit in its
memory will recompute the net crc16 checksum insure it pass the to the nic
regardless.  It is amazing how much data can be corrupted by a network
environment via all the NFS/NBD/etc wantabie storage products out there.

Just a chuckle for you to ponder.

--a

On Sun, 3 Aug 2003, Jeff Garzik wrote:

> Werner Almesberger wrote:
> > Jeff Garzik wrote:
> > 
> >>jabbering at the same time.  TCP is a "one size fits all" solution, but 
> >>it doesn't work well for everyone.
> > 
> > 
> > But then, ten "optimized xxPs" that work well in two different
> > scenarios each, but not so good in the 98 others, wouldn't be
> > much fun either.
> > 
> > It's been tried a number of times. Usually, real life sneaks
> > in at one point or another, leaving behind a complex mess.
> > When they've sorted out these problems, regular TCP has caught
> > up with the great optimized transport protocols. At that point,
> > they return to their niche, sometimes tail between legs and
> > muttering curses, sometimes shaking their fist and boldly
> > proclaiming how badly they'll rub TCP in the dirt in the next
> > round. Maybe they shed off some of the complexity, and trade it
> > for even more aggressive optimization, which puts them into
> > their niche even more firmly. Eventually, they fade away.
> > 
> > There are cases where TCP doesn't work well, like a path of
> > badly mismatched link layers, but such paths don't treat any
> > protocol following the end-to-end principle kindly.
> > 
> > Another problem of TCP is that it has grown a bit too many
> > knobs you need to turn before it works over your really fast
> > really long pipe. (In one of the OLS after dinner speeches,
> > this was quite appropriately called the "wizard gap".)
> > 
> > 
> >>It's obviously not over a WAN...
> > 
> > 
> > That's why NFS turned off UDP checksums ;-) As soon as you put
> > it on IP, it will crawl to distances you didn't imagine in your
> > wildest dreams. It always does.
> 
> Really fast, really long pipes in practice don't exist for 99.9% of all 
> Internet users.
> 
> 
> When you approach traffic levels that push you want to offload most of 
> the TCP net stack, then TCP isn't the right solution for you anymore, 
> all things considered.
> 
> 
> The Linux net stack just isn't built to be offloaded.  TOE engines will 
> either need to (1) fall back to Linux software for all-but-the-common 
> case (otherwise netfilter, etc. break), or, (2) will need to be 
> hideously complex beasts themselves.  And I can't see ASIC and firmware 
> designers being excited about implementing netfilter on a PCI card :)
> 
> Unfortunately some vendors seem to choosing TOE option #3:  TCP offload 
> which introduces many limitations (connection limits, netfilter not 
> supported, etc.) which Linux never had before.  Vendors don't seem to 
> realize TOE has real potential to damage the "good network neighbor" 
> image the net stack has.  The Linux net stack's behavior is known, 
> documented, predictable.  TOE changes all that.
> 
> There is one interesting TOE solution, that I have yet to see created: 
> run Linux on an embedded processor, on the NIC.  This stripped-down 
> Linux kernel would perform all the header parsing, checksumming, etc. 
> into the NIC's local RAM.  The Linux OS driver interface becomes a 
> virtual interface with a large MTU, that communicates from host CPU to 
> NIC across the PCI bus using jumbo-ethernet-like data frames. 
> Management frames would control the ethernet interface on the other side 
> of the PCI bus "tunnel".
> 
> 
> >>So, fix the other end of the pipeline too, otherwise this fast network 
> >>stuff is flashly but pointless.  If you want to serve up data from disk, 
> >>then start creating PCI cards that have both Serial ATA and ethernet 
> >>connectors on them :)  Cut out the middleman of the host CPU and host 
> >>memory bus instead of offloading portions of TCP that do not need to be 
> >>offloaded.
> > 
> > 
> > That's a good point. A hierarchical memory structure can help
> > here. Moving one end closer to the hardware, and letting it
> > know (e.g. through sendfile) that also the other end is close
> > (or can be reached more directly that through some hopelessly
> > crowded main bus) may help too.
> 
> Definitely.
> 
> 	Jeff
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: TOE brain dump

[Lincoln Dale]

At 05:12 PM 6/08/2003, Andre Hedrick wrote:
>Do be sure to check that your data payload is correct.
>Everyone knows that a router/gateway/switch with a sticky bit in its
>memory will recompute the net crc16 checksum insure it pass the to the nic
>regardless.  It is amazing how much data can be corrupted by a network
>environment via all the NFS/NBD/etc wantabie storage products out there.

Andre, you are wrong.

firstly, do you REALLY think that most router(s)/switch(es) out there 
recompute IP checksums because they did a IP TTL decrement when routing an 
IP packet or NAT IP addresses?

no, they don't.  just like netfilter or router-on-linux is smart enough to 
be able to re-code an IP checksum by unmasking and re-masking the old/new 
values in a header, so does the most router vendor's code.

secondly, why would a router or switch even be touching the data at layer-4 
(TCP), let alone recalculating a CRC?

i know you really like your "we do ERL 2 in iSCSI" pitch, but lets stick to 
facts here eh?


cheers,

lincoln.

Re: TOE brain dump

[David S. Miller]

On Wed, 06 Aug 2003 18:20:06 +1000
Lincoln Dale <ltd@cisco.com> wrote:

> secondly, why would a router or switch even be touching the data at layer-4 
> (TCP), let alone recalculating a CRC?

To make sure emails about Falun Gong and other undesirable topics
don't make it into China.

Re: TOE brain dump

[Jesse Pollard]

On Wednesday 06 August 2003 03:22, David S. Miller wrote:
> On Wed, 06 Aug 2003 18:20:06 +1000
>
> Lincoln Dale <ltd@cisco.com> wrote:
> > secondly, why would a router or switch even be touching the data at
> > layer-4 (TCP), let alone recalculating a CRC?
>
> To make sure emails about Falun Gong and other undesirable topics
> don't make it into China.

Thats not a router, or switch... It's a firewall :-)

Re: TOE brain dump

[Eric W. Biederman]

Werner Almesberger <werner@almesberger.net> writes:

> Jeff Garzik wrote:
> > jabbering at the same time.  TCP is a "one size fits all" solution, but 
> > it doesn't work well for everyone.
> 
> But then, ten "optimized xxPs" that work well in two different
> scenarios each, but not so good in the 98 others, wouldn't be
> much fun either.

The optimized for low latency cases seem to have a strong
market in clusters.  And they are currently keeping alive
quite a few technologies.  Myrinet, Infiniband, Quadric's Elan, etc.
Having low latency and switch technologies that scale is quite
rare currently.

> Another problem of TCP is that it has grown a bit too many
> knobs you need to turn before it works over your really fast
> really long pipe. (In one of the OLS after dinner speeches,
> this was quite appropriately called the "wizard gap".)

Does anyone know which knobs to turn to make TCP go fast over
Infiniband.  (A low latency high bandwidth network?)  I get to
deal with them on a regular basis...

There is one place in low latency communications that I can think
of where TCP/IP is not the proper solution.  For low latency
communication the checksum is at the wrong end of the packet.
IB gets this one correct and places the checksum at the tail end of
the packet.  This allows the packet to start transmitting before
the checksum is computed, possibly even having the receive start
at the other end before the tail of the packet is transmitted.

Would it make any sense to do a low latency variation on TCP that
fixes that problem?  For the IP header we are fine as the data
precedes the checksum.  But the problem appears to affect all
of the upper level protocols that ride on IP, UDP, TCP, SCTP...

> > So, fix the other end of the pipeline too, otherwise this fast network 
> > stuff is flashly but pointless.  If you want to serve up data from disk, 
> > then start creating PCI cards that have both Serial ATA and ethernet 
> > connectors on them :)  Cut out the middleman of the host CPU and host 
> > memory bus instead of offloading portions of TCP that do not need to be 
> > offloaded.
> 
> That's a good point. A hierarchical memory structure can help
> here. Moving one end closer to the hardware, and letting it
> know (e.g. through sendfile) that also the other end is close
> (or can be reached more directly that through some hopelessly
> crowded main bus) may help too.

On that score it is worth noting that the next generation of
peripheral busses (Hypertransport, PCI Express, etc) are all switched.
Which means that device to device communication may be more
reasonable.  Going from a bussed interconnect to a switched
interconnect is certainly a dramatic change in infrastructure. How
that will affect the tradeoffs I don't know.  

Eric

Re: TOE brain dump

[Werner Almesberger]

Eric W. Biederman wrote:
> The optimized for low latency cases seem to have a strong
> market in clusters.

Clusters have captive, no, _desperate_ customers ;-) And it
seems that people are just as happy putting MPI as their
transport on top of all those link-layer technologies.

> There is one place in low latency communications that I can think
> of where TCP/IP is not the proper solution.  For low latency
> communication the checksum is at the wrong end of the packet.

That's one of the few things ATM's AAL5 got right. But in the end,
I think it doesn't really matter. At 1 Gbps, an MTU-sized packet
flies by within 13 us. At 10 Gbps, it's only 1.3 us. At that point,
you may well treat it as an atomic unit.

> On that score it is worth noting that the next generation of
> peripheral busses (Hypertransport, PCI Express, etc) are all switched.

And it's about time for that :-)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[David S. Miller]

On Mon, 4 Aug 2003 16:24:33 -0300
Werner Almesberger <werner@almesberger.net> wrote:

> Eric W. Biederman wrote:
> > There is one place in low latency communications that I can think
> > of where TCP/IP is not the proper solution.  For low latency
> > communication the checksum is at the wrong end of the packet.
> 
> That's one of the few things ATM's AAL5 got right.

Let's recall how long the IFF_TRAILERS hack from BSD :-)

> But in the end, I think it doesn't really matter.

I tend to agree on this one.

And on the transmit side if you have more than 1 pending TX frame, you
can always be prefetching the next one into the fifo so that by the
time the medium is ready all the checksum bits have been done.

In fact I'd be surprised if current generation 1g/10g cards are not
doing something like this.

Re: TOE brain dump

[Eric W. Biederman]

"David S. Miller" <davem@redhat.com> writes:

> On Mon, 4 Aug 2003 16:24:33 -0300
> Werner Almesberger <werner@almesberger.net> wrote:
> 
> > Eric W. Biederman wrote:
> > > There is one place in low latency communications that I can think
> > > of where TCP/IP is not the proper solution.  For low latency
> > > communication the checksum is at the wrong end of the packet.
> > 
> > That's one of the few things ATM's AAL5 got right.
> 
> Let's recall how long the IFF_TRAILERS hack from BSD :-)

Putting the variable length headers on the end of a packet?  Or
was that something other than RFC893?

I think IPv6 solves that much more cleanly by simply deleting them.

> > But in the end, I think it doesn't really matter.
> 
> I tend to agree on this one.
> 
> And on the transmit side if you have more than 1 pending TX frame, you
> can always be prefetching the next one into the fifo so that by the
> time the medium is ready all the checksum bits have been done.

For large data transmissions that happens.
 
> In fact I'd be surprised if current generation 1g/10g cards are not
> doing something like this.

Well at this point before I propose anything concrete I suspect I need
to profile some actual application and see how things go.  But from a
very latency sensitive perspective, I would be surprised if the
problem goes away with faster technology.

For now I am happy just to insert the peculiar thought that latency
across the entire cluster/lan is of great importance to some
applications.

Eric

Re: TOE brain dump

[Eric W. Biederman]

Werner Almesberger <werner@almesberger.net> writes:

> Eric W. Biederman wrote:
> > The optimized for low latency cases seem to have a strong
> > market in clusters.
> 
> Clusters have captive, no, _desperate_ customers ;-) And it
> seems that people are just as happy putting MPI as their
> transport on top of all those link-layer technologies.

MPI is not a transport.  It an interface like the Berkeley sockets
layer.  The semantics it wants right now are usually mapped to
TCP/IP when used on an IP network.  Though I suspect SCTP might
be a better fit.  

But right now nothing in the IP stack is a particularly good fit.

Right now there is a very strong feeling among most of the people
using and developing on clusters that by and large what they are doing
is not of interest to the general kernel community, and so has no
chance of going in.   So you see hack piled on top of hack piled on
top of hack.

Mostly I think the that is less true, at least if they can stand the
process of severe code review and cleaning up their code.  If we can
put in code to scale the kernel to 64 processors.  NIC drivers for
fast interconnects and a few similar tweaks can't hurt either.  

But of course to get through the peer review process people need
to understand what they are doing.

> > There is one place in low latency communications that I can think
> > of where TCP/IP is not the proper solution.  For low latency
> > communication the checksum is at the wrong end of the packet.
> 
> That's one of the few things ATM's AAL5 got right. But in the end,
> I think it doesn't really matter. At 1 Gbps, an MTU-sized packet
> flies by within 13 us. At 10 Gbps, it's only 1.3 us. At that point,
> you may well treat it as an atomic unit.

So store and forward of packets in a 3 layer switch hierarchy, at 1.3 us
per copy. 1.3us to the NIC + 1.3us to the first switch chip + 1.3us to the
second switch chip + 1.3us to the top level switch chip + 1.3us to a middle layer
switch chip + 1.3us to the receiving NIC + 1.3us the receiver.

1.3us * 7 = 9.1us to deliver a packet to the other side.  That is
still quite painful.  Right now I can get better latencies over any of
the cluster interconnects.  I think 5 us is the current low end, with
the high end being about 1 us.

Quite often in MPI when a message is sent the program cannot continue
until the reply is received.  Possibly this is a fundamental problem
with the application programming model, encouraging applications to
be latency sensitive.  But it is a well established API and
programming paradigm so it has to be lived with.

All of this is pretty much the reverse of the TOE case.  Things are
latency sensitive because real work needs to be done.  And the more
latency you have the slower that work gets done.  

A lot of the NICs which are used for MPI tend to be smart for two
reasons.  1) So they can do source routing. 2) So they can safely
export some of their interface to user space, so in the fast path
they can bypass the kernel.

Eric

Re: TOE brain dump

[Werner Almesberger]

Eric W. Biederman wrote:
> MPI is not a transport.  It an interface like the Berkeley sockets
> layer.

Hmm, but doesn't it also unify transport semantics (i.e. chop
TCP streams into messages), maybe add reliability to transports
that don't have it, and provide addressing ? Okay, perhaps you
wouldn't call this a transport in the OSI sense, but it still
seems to have considerably more functionality than just
providing an API.

> Mostly I think the that is less true, at least if they can stand the
> process of severe code review and cleaning up their code.

Hmm, people putting dozens of millions into building clusters
can't afford to have what is probably their most essential
infrastructure code reviewed and cleaned up ? Oh dear.

> But of course to get through the peer review process people need
> to understand what they are doing.

A good point :-)

> So store and forward of packets in a 3 layer switch hierarchy, at 1.3 us
> per copy.

But your switch could just do cut-through, no ? Or do they
need to recompute checksums ?

> A lot of the NICs which are used for MPI tend to be smart for two
> reasons.  1) So they can do source routing. 2) So they can safely
> export some of their interface to user space, so in the fast path
> they can bypass the kernel.

The second part could be interesting for TOE, too. Only that
the interface exported would just be the socket interface.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[Eric W. Biederman]

Werner Almesberger <werner@almesberger.net> writes:

> Eric W. Biederman wrote:
> > MPI is not a transport.  It an interface like the Berkeley sockets
> > layer.
> 
> Hmm, but doesn't it also unify transport semantics (i.e. chop
> TCP streams into messages), maybe add reliability to transports
> that don't have it, and provide addressing ? Okay, perhaps you
> wouldn't call this a transport in the OSI sense, but it still
> seems to have considerably more functionality than just
> providing an API.

Those are all features of the MPI implementation.  It is
not that MPI does not have an underlying transport.  MPI has
a lot of underlying transports.  And there is a different MPI
implementation for each transport.  Although a lot of them start
with a common base.
 
> > Mostly I think the that is less true, at least if they can stand the
> > process of severe code review and cleaning up their code.
> 
> Hmm, people putting dozens of millions into building clusters
> can't afford to have what is probably their most essential
> infrastructure code reviewed and cleaned up ? Oh dear.

Afford, they can do.  A lot of the users are researchers and
a lot of people doing the code are researchers.  So corralling
them up and getting production quality code can be a challenge,
or getting them to take small enough steps that they don't
frighten the rest of the world.

Plus ten million dollars pretty much buys you a spot in the top 10 of
the top 500 supercomputers.  The bulk of the clusters are a lot less
expensive than that.
 
> > But of course to get through the peer review process people need
> > to understand what they are doing.
> 
> A good point :-)
> 
> > So store and forward of packets in a 3 layer switch hierarchy, at 1.3 us
> > per copy.
> 
> But your switch could just do cut-through, no ? Or do they
> need to recompute checksums ?

Correct, switches can and generally do implement cut-through in that
kind of environment.  I was just showing that even at 10Gbps treating
a packet as an atomic unit has issues.  cut-through is necessary
to keep your latency down.  Do any ethernet switches do cut-through?
 
> > A lot of the NICs which are used for MPI tend to be smart for two
> > reasons.  1) So they can do source routing. 2) So they can safely
> > export some of their interface to user space, so in the fast path
> > they can bypass the kernel.
> 
> The second part could be interesting for TOE, too. Only that
> the interface exported would just be the socket interface.

Agreed.  

Eric

Re: TOE brain dump

[Werner Almesberger]

Eric W. Biederman wrote:
> Afford, they can do.  A lot of the users are researchers and
> a lot of people doing the code are researchers.  So corralling
> them up and getting production quality code can be a challenge,

Ah, the joy of herding cats :-) But I guess you just need a
sufficiently competent and sufficiently well-funded group
that goes ahead and does it. There is usually little point
in directly involving everyone who may have an opinion.

> to keep your latency down.  Do any ethernet switches do cut-through?

According to Google, many at least claim to do this.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[Jesse Pollard]

On Tuesday 05 August 2003 12:19, Eric W. Biederman wrote:
> Werner Almesberger <werner@almesberger.net> writes:
> > Eric W. Biederman wrote:
> > > The optimized for low latency cases seem to have a strong
> > > market in clusters.
> >
> > Clusters have captive, no, _desperate_ customers ;-) And it
> > seems that people are just as happy putting MPI as their
> > transport on top of all those link-layer technologies.
>
> MPI is not a transport.  It an interface like the Berkeley sockets
> layer.  The semantics it wants right now are usually mapped to
> TCP/IP when used on an IP network.  Though I suspect SCTP might
> be a better fit.
>
> But right now nothing in the IP stack is a particularly good fit.
>
> Right now there is a very strong feeling among most of the people
> using and developing on clusters that by and large what they are doing
> is not of interest to the general kernel community, and so has no
> chance of going in.   So you see hack piled on top of hack piled on
> top of hack.
>
> Mostly I think the that is less true, at least if they can stand the
> process of severe code review and cleaning up their code.  If we can
> put in code to scale the kernel to 64 processors.  NIC drivers for
> fast interconnects and a few similar tweaks can't hurt either.
>
> But of course to get through the peer review process people need
> to understand what they are doing.
>
> > > There is one place in low latency communications that I can think
> > > of where TCP/IP is not the proper solution.  For low latency
> > > communication the checksum is at the wrong end of the packet.
> >
> > That's one of the few things ATM's AAL5 got right. But in the end,
> > I think it doesn't really matter. At 1 Gbps, an MTU-sized packet
> > flies by within 13 us. At 10 Gbps, it's only 1.3 us. At that point,
> > you may well treat it as an atomic unit.
>
> So store and forward of packets in a 3 layer switch hierarchy, at 1.3 us
> per copy. 1.3us to the NIC + 1.3us to the first switch chip + 1.3us to the
> second switch chip + 1.3us to the top level switch chip + 1.3us to a middle
> layer switch chip + 1.3us to the receiving NIC + 1.3us the receiver.
>
> 1.3us * 7 = 9.1us to deliver a packet to the other side.  That is
> still quite painful.  Right now I can get better latencies over any of
> the cluster interconnects.  I think 5 us is the current low end, with
> the high end being about 1 us.

I think you are off here since the second and third layer should not recompute
checksums other than for the header (if they even did that). Most of the
switches I used (mind, not configured) were wire speed. Only header checksums
had recomputes, and I understood it was only for routing.

> Quite often in MPI when a message is sent the program cannot continue
> until the reply is received.  Possibly this is a fundamental problem
> with the application programming model, encouraging applications to
> be latency sensitive.  But it is a well established API and
> programming paradigm so it has to be lived with.
>
> All of this is pretty much the reverse of the TOE case.  Things are
> latency sensitive because real work needs to be done.  And the more
> latency you have the slower that work gets done.
>
> A lot of the NICs which are used for MPI tend to be smart for two
> reasons.  1) So they can do source routing. 2) So they can safely
> export some of their interface to user space, so in the fast path
> they can bypass the kernel.

And bypass any security checks required. A single rogue MPI application
using such an interface can/will bring the cluster down.

Now this is not as much of a problem since many clusters use a standalone
internal network, AND are single application clusters. These clusters
tend to be relatively small (32 - 64 nodes? perhaps 16-32 is better. The
clusters I've worked with have always been large 128-300 nodes, so I'm
not a good judge of "small").

This is immediately broken when you schedule two or more batch jobs on
a cluster in parallel.

It is also broken if the two jobs require different security contexts.

Re: TOE brain dump

[Andy Isaacson]

On Wed, Aug 06, 2003 at 07:46:33AM -0500, Jesse Pollard wrote:
> On Tuesday 05 August 2003 12:19, Eric W. Biederman wrote:
> > So store and forward of packets in a 3 layer switch hierarchy, at 1.3 us
> > per copy. 1.3us to the NIC + 1.3us to the first switch chip + 1.3us to the
> > second switch chip + 1.3us to the top level switch chip + 1.3us to a middle
> > layer switch chip + 1.3us to the receiving NIC + 1.3us the receiver.
> >
> > 1.3us * 7 = 9.1us to deliver a packet to the other side.  That is
> > still quite painful.  Right now I can get better latencies over any of
> > the cluster interconnects.  I think 5 us is the current low end, with
> > the high end being about 1 us.
> 
> I think you are off here since the second and third layer should not recompute
> checksums other than for the header (if they even did that). Most of the
> switches I used (mind, not configured) were wire speed. Only header checksums
> had recomputes, and I understood it was only for routing.

The switches may be "wire speed" but that doesn't help the latency any.
AFAIK all GigE switches are store-and-forward, which automatically costs
you the full 1.3us for each link hop.  (I didn't check Eric's numbers,
so I don't know that 1.3us is the right value, but it sounds right.)
Also I think you might be confused about what Eric meant by "3 layer
switch hierarchy"; he's referring to a tree topology network with
layer-one switches connecting hosts, layer-two switches connecting
layer-one switches, and layer-three switches connecting layer-two
switches.  This means that your worst-case node-to-node latency has 6
wire hops with 7 "read the entire packet into memory" operations,
depending on how you count the initiating node's generation of the
packet.

[snip]
> > Quite often in MPI when a message is sent the program cannot continue
> > until the reply is received.  Possibly this is a fundamental problem
> > with the application programming model, encouraging applications to
> > be latency sensitive.  But it is a well established API and
> > programming paradigm so it has to be lived with.

This is true, in HPC.  Some of the problem is the APIs encouraging such
behavior; another part of the problem is that sometimes, the problem has
fundamental latency dependencies that cannot be programmed around.

> > A lot of the NICs which are used for MPI tend to be smart for two
> > reasons.  1) So they can do source routing. 2) So they can safely
> > export some of their interface to user space, so in the fast path
> > they can bypass the kernel.
> 
> And bypass any security checks required. A single rogue MPI application
> using such an interface can/will bring the cluster down.

This is just false.  Kernel bypass (done properly) has no negative
effect on system stability, either on-node or on-network.  By "done
properly" I mean that the NIC has mappings programmed into it by the
kernel at app-startup time, and properly bounds-checks all remote DMA,
and has a method for verifying that incoming packets are not rogue or
corrupt.  (Of course a rogue *kernel* can probably interfere with other
*applications* on the network it's connected to, by inserting malicious
packets into the datastream, but even that is soluble with cookies or
routing checks.  However, I don't believe any systems try to defend
against rogue nodes today.)

I believe that Myrinet's hardware has the capability to meet the "kernel
bypass done properly" requirement I state above; I make no claim that
their GM implementation actually meets the requirement (although I think
it might).  It's pretty likely that QSW's Elan hardware can, too, but I
know even less about that.

-andy

Re: TOE brain dump

[Jesse Pollard]

On Wednesday 06 August 2003 11:25, Andy Isaacson wrote:
> On Wed, Aug 06, 2003 at 07:46:33AM -0500, Jesse Pollard wrote:
> > On Tuesday 05 August 2003 12:19, Eric W. Biederman wrote:
> > > So store and forward of packets in a 3 layer switch hierarchy, at 1.3
> > > us per copy. 1.3us to the NIC + 1.3us to the first switch chip + 1.3us
> > > to the second switch chip + 1.3us to the top level switch chip + 1.3us
> > > to a middle layer switch chip + 1.3us to the receiving NIC + 1.3us the
> > > receiver.
> > >
> > > 1.3us * 7 = 9.1us to deliver a packet to the other side.  That is
> > > still quite painful.  Right now I can get better latencies over any of
> > > the cluster interconnects.  I think 5 us is the current low end, with
> > > the high end being about 1 us.
> >
> > I think you are off here since the second and third layer should not
> > recompute checksums other than for the header (if they even did that).
> > Most of the switches I used (mind, not configured) were wire speed. Only
> > header checksums had recomputes, and I understood it was only for
> > routing.
>
> The switches may be "wire speed" but that doesn't help the latency any.
> AFAIK all GigE switches are store-and-forward, which automatically costs
> you the full 1.3us for each link hop.  (I didn't check Eric's numbers,
> so I don't know that 1.3us is the right value, but it sounds right.)
> Also I think you might be confused about what Eric meant by "3 layer
> switch hierarchy"; he's referring to a tree topology network with
> layer-one switches connecting hosts, layer-two switches connecting
> layer-one switches, and layer-three switches connecting layer-two
> switches.  This means that your worst-case node-to-node latency has 6
> wire hops with 7 "read the entire packet into memory" operations,
> depending on how you count the initiating node's generation of the
> packet.

If it reads the packet into memory before starting transmission, it isn't
"wire speed". It is a router.

> [snip]
>
> > > Quite often in MPI when a message is sent the program cannot continue
> > > until the reply is received.  Possibly this is a fundamental problem
> > > with the application programming model, encouraging applications to
> > > be latency sensitive.  But it is a well established API and
> > > programming paradigm so it has to be lived with.
>
> This is true, in HPC.  Some of the problem is the APIs encouraging such
> behavior; another part of the problem is that sometimes, the problem has
> fundamental latency dependencies that cannot be programmed around.
>
> > > A lot of the NICs which are used for MPI tend to be smart for two
> > > reasons.  1) So they can do source routing. 2) So they can safely
> > > export some of their interface to user space, so in the fast path
> > > they can bypass the kernel.
> >
> > And bypass any security checks required. A single rogue MPI application
> > using such an interface can/will bring the cluster down.
>
> This is just false.  Kernel bypass (done properly) has no negative
> effect on system stability, either on-node or on-network.  By "done
> properly" I mean that the NIC has mappings programmed into it by the
> kernel at app-startup time, and properly bounds-checks all remote DMA,
> and has a method for verifying that incoming packets are not rogue or
> corrupt.  (Of course a rogue *kernel* can probably interfere with other
> *applications* on the network it's connected to, by inserting malicious
> packets into the datastream, but even that is soluble with cookies or
> routing checks.  However, I don't believe any systems try to defend
> against rogue nodes today.)

Just because the packet gets transfered to a buffer correctly does not
mean that buffer is the one it should have been sent to. If it didn't
have this problem, then there would be no kernel TCP/IP interaction. Just
open the ethernet device and start writing/reading. Ooops. known security
failure.

>
> I believe that Myrinet's hardware has the capability to meet the "kernel
> bypass done properly" requirement I state above; I make no claim that
> their GM implementation actually meets the requirement (although I think
> it might).  It's pretty likely that QSW's Elan hardware can, too, but I
> know even less about that.

since the routing is done is user mode, as part of the library, it can be
used to directly affect processes NOT owned by the user. This bypasses
the kernel security checks by definition. Already known to happen with
raw myrinet, so there is a kernel layer on top of it to shield it (or
at least try to). If there is no kernel involvement, then there can be
no restrictions on what can be passed down the line to the device. Now
some of the modifications for myrinet were to use normal TCP/IP to establish
source/destination header information, then bypass any packet handshake, but
force EACH packet to include the pre-established source/destination header 
info. This is equivalent to UDP, but without any checksums, and sometimes
can bypass part of the kernel cache. Unfortunately, it also means that
sometimes incoming data is NOT destined for the user, and must be 
erased/copied before the final destination is achieved. This introduces leaks 
due to the race condition caused by the transfer to the wrong buffer.

You can't DMA directly to a users buffer, because you MUST verify the header
before the data... and you can't do that until the buffer is in memory...
So bypassing the kernel generates security failures.

This is already a problem in fibre channel devices, and in other network
devices. Anytime you bypass the kernel security you also void any 
restrictions on the network, and any hosts it is attached to.

Re: TOE brain dump

[Andy Isaacson]

On Wed, Aug 06, 2003 at 01:58:59PM -0500, Jesse Pollard wrote:
> On Wednesday 06 August 2003 11:25, Andy Isaacson wrote:
> > The switches may be "wire speed" but that doesn't help the latency any.
> > AFAIK all GigE switches are store-and-forward, which automatically costs
> > you the full 1.3us for each link hop.  (I didn't check Eric's numbers,
> > so I don't know that 1.3us is the right value, but it sounds right.)
> > Also I think you might be confused about what Eric meant by "3 layer
> > switch hierarchy"; he's referring to a tree topology network with
> > layer-one switches connecting hosts, layer-two switches connecting
> > layer-one switches, and layer-three switches connecting layer-two
> > switches.  This means that your worst-case node-to-node latency has 6
> > wire hops with 7 "read the entire packet into memory" operations,
> > depending on how you count the initiating node's generation of the
> > packet.
> 
> If it reads the packet into memory before starting transmission, it isn't
> "wire speed". It is a router.

[Please read an implied "I might be totally off base here, since I've
 never designed an Ethernet switch" disclaimer into this paragraph.]

This statement is completely false.  Ethernet switches *do* read the
packet into memory before starting transmission.  This must be so,
because an Ethernet switch does not propagate runts, jabber frames, or
frames with an incorrect ethernet crc.  If the switch starts
transmission before it's received the last bit, it is provably
impossible for it to avoid propagating crc-failing-frames; ergo,
switches must have the entire packet on hand before starting
transmission.

> > > > A lot of the NICs which are used for MPI tend to be smart for two
> > > > reasons.  1) So they can do source routing. 2) So they can safely
> > > > export some of their interface to user space, so in the fast path
> > > > they can bypass the kernel.
> > >
> > > And bypass any security checks required. A single rogue MPI application
> > > using such an interface can/will bring the cluster down.
> >
> > This is just false.  Kernel bypass (done properly) has no negative
> > effect on system stability, either on-node or on-network.  By "done
> > properly" I mean that the NIC has mappings programmed into it by the
> > kernel at app-startup time, and properly bounds-checks all remote DMA,
> > and has a method for verifying that incoming packets are not rogue or
> > corrupt.  (Of course a rogue *kernel* can probably interfere with other
> > *applications* on the network it's connected to, by inserting malicious
> > packets into the datastream, but even that is soluble with cookies or
> > routing checks.  However, I don't believe any systems try to defend
> > against rogue nodes today.)
> 
> Just because the packet gets transfered to a buffer correctly does not
> mean that buffer is the one it should have been sent to. If it didn't
> have this problem, then there would be no kernel TCP/IP interaction. Just
> open the ethernet device and start writing/reading. Ooops. known security
> failure.

You're ignoring the fact that there's a complete, programmable RISC CPU
on the Myrinet card which is running code (the MCP, Myrinet Control
Program) installed into it by the kernel.  The kernel tells the MCP to
allow access to a given app (by mapping a page of PCI IO addresses into
the user's virtual address space), and the MCP checks the user's DMA
requests for validity.  The user cannot generate arbitrary Myrinet
routing requests, cannot write to arbitrary addresses, cannot send
messages to hosts not in his allowed lists, et cetera.  We do know that
the buffer is the one it should have been sent to, because the MCP on
the sending end verified that it was an allowed destination host, and
the MCP on the receiving end verified that the destination address was
valid.  Myrinet Inc even offers a SDK allowing you to write your own
MCP, if you so desire, and various research projects have done precisely
that.

Demonstrating that dumb Ethernet cards cannot be smart does not
demonstrate that smart FooNet cards cannot be smart.  (s/FooNet/$x/ as
desired.)

> > I believe that Myrinet's hardware has the capability to meet the "kernel
> > bypass done properly" requirement I state above; I make no claim that
> > their GM implementation actually meets the requirement (although I think
> > it might).  It's pretty likely that QSW's Elan hardware can, too, but I
> > know even less about that.
> 
> since the routing is done is user mode, as part of the library, it can be
> used to directly affect processes NOT owned by the user.  This
> bypasses the kernel security checks by definition.

The routing is done on the MCP, not in a library.  (Or at least, it
could be -- I don't know offhand how GM1 and GM2 work.)  This is not an
insoluble problem.

> Already known to happen with raw myrinet, so there is a kernel layer
> on top of it to shield it (or at least try to).

Perhaps that's the case with GM1 (I don't know) but it is not a
fundamental flaw of the hardware or the network.

> If there is no kernel involvement, then there can be no restrictions
> on what can be passed down the line to the device.

The MCP provides the necessary checking.

> Now some of the modifications for myrinet were to use normal TCP/IP to
> establish source/destination header information, then bypass any
> packet handshake, but force EACH packet to include the pre-established
> source/destination header info.

I don't know what you're talking about here; perhaps this was some early
"TCP over Myrinet" thing.  Currently on a host with GM1 running, the
myri0 interface shows up as an almost-normal Ethernet interface, and
most of the relevant networking ioctls work just fine.  I can even
tcpdump it.

On a related topic, there is a Myrinet line card with a GigE port
available.  I haven't looked into the software end deeply, but
apparently you just stick a standard Myrinet route to that switch port
on the front of the Myrinet frame, append an Ethernet frame, and your
Myrinet host can send GigE packets without bother.  I don't know how
incoming ethernet packets are routed, alas -- presumably a Myrinet route
is encoded in the MAC somehow.

> This is equivalent to UDP, but without any checksums, and sometimes
> can bypass part of the kernel cache. Unfortunately, it also means that
> sometimes incoming data is NOT destined for the user, and must be 
> erased/copied before the final destination is achieved. This introduces leaks 
> due to the race condition caused by the transfer to the wrong buffer.
> 
> You can't DMA directly to a users buffer, because you MUST verify the header
> before the data... and you can't do that until the buffer is in memory...
> So bypassing the kernel generates security failures.

Again, the security problems are solved by having the MCP check the
necessary conditions.  You bring up a good point WRT error resilience,
though -- I don't know how Myrinet handles media bit errors.

You *can* DMA directly to a user's buffer, because the necessary header
information was checked on the MCP before the bits even touch the PCI
bus.

> This is already a problem in fibre channel devices, and in other network
> devices. Anytime you bypass the kernel security you also void any 
> restrictions on the network, and any hosts it is attached to.

Sufficiently advanced HBA hardware and software solve this problem.
Please pick another windmill to tilt at.  (Like the error one; I need to
find out what the answer to that is.)

-andy

RE: TOE brain dump

[David Schwartz]

> This statement is completely false.  Ethernet switches *do* read the
> packet into memory before starting transmission.

	Some do. Some don't. Some are configurable.

> This must be so,
> because an Ethernet switch does not propagate runts, jabber frames, or
> frames with an incorrect ethernet crc.

	If they use cut-through switching, they do. Some use adaptive switching,
which means they use cut-through switching but change to store and forward
if there are too many runts, jabber frames, bad CRCs, and so on.

	Obviously, you can't always do a cut-through. If the target port is busy,
cut-through is impossible. If the ports are different speeds, cut-through is
impossible. The Intel 510T switch for my home network does adaptive
switching with configurable error thresholds. In fact, it's even smarter
than that, with an intermediate mode that suppresses runts without doing a
full store and forward. See:
http://www.intel.com/support/express/switches/23188.htm

> If the switch starts
> transmission before it's received the last bit, it is provably
> impossible for it to avoid propagating crc-failing-frames; ergo,
> switches must have the entire packet on hand before starting
> transmission.

	Except not all switches always avoid propogating bad frames.

	DS

Re: TOE brain dump

[Ben Greear]

Jeff Garzik wrote:

> So, fix the other end of the pipeline too, otherwise this fast network 
> stuff is flashly but pointless.  If you want to serve up data from disk, 
> then start creating PCI cards that have both Serial ATA and ethernet 
> connectors on them :)  Cut out the middleman of the host CPU and host 

I for one would love to see something like this, and not just Serial ATA..
but maybe 8x Serial ATA and RAID :)

Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: TOE brain dump

[Alan Shih]

A DMA xfer that fills the NIC pipe with IDE source. That's not very hard...
need a lot of bufferring/FIFO though.  May require large modification to the
file serving applications?

Alan

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Ben Greear
Sent: Saturday, August 02, 2003 9:02 PM
To: Jeff Garzik
Cc: Nivedita Singhvi; Werner Almesberger; netdev@oss.sgi.com;
linux-kernel@vger.kernel.org
Subject: Re: TOE brain dump


Jeff Garzik wrote:

> So, fix the other end of the pipeline too, otherwise this fast network
> stuff is flashly but pointless.  If you want to serve up data from disk,
> then start creating PCI cards that have both Serial ATA and ethernet
> connectors on them :)  Cut out the middleman of the host CPU and host

I for one would love to see something like this, and not just Serial ATA..
but maybe 8x Serial ATA and RAID :)

Ben


--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: TOE brain dump

[Jeff Garzik]

Alan Shih wrote:
> A DMA xfer that fills the NIC pipe with IDE source. That's not very hard...
> need a lot of bufferring/FIFO though.  May require large modification to the
> file serving applications?


Nope, that's using the existing sendfile(2) facility.

	Jeff

RE: TOE brain dump

[David Lang]

do you really want the processor on the card to be tunning
apache/NFS/Samba/etc ?

putting enough linux on the card to act as a router (which would include
the netfilter stuff) is one thing. putting the userspace code that
interfaces with the outside world for file transfers is something else.

if you really want the disk connected to your network card you are just
talking a low-end linux box. forget all this stuff about it being on a
card and just use a full box (economys of scale will make this cheaper)

making a firewall that's a core system with a dozen slave systems attached
to it (the network cards) sounds like the type of clustering that Linux
has been used for for compute nodes. complicated to setup, but extremely
powerful and scalable once configured.

if you want more then a router on the card then Alan Cox is right, just
add another processor to the system, it's easier and cheaper.

David Lang


 On
Sat, 2 Aug 2003, Alan Shih wrote:

> Date: Sat, 2 Aug 2003 23:22:52 -0700
> From: Alan Shih <alan@storlinksemi.com>
> To: Ben Greear <greearb@candelatech.com>, Jeff Garzik <jgarzik@pobox.com>
> Cc: Nivedita Singhvi <niv@us.ibm.com>,
>      Werner Almesberger <werner@almesberger.net>, netdev@oss.sgi.com,
>      linux-kernel@vger.kernel.org
> Subject: RE: TOE brain dump
>
> A DMA xfer that fills the NIC pipe with IDE source. That's not very hard...
> need a lot of bufferring/FIFO though.  May require large modification to the
> file serving applications?
>
> Alan
>
> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Ben Greear
> Sent: Saturday, August 02, 2003 9:02 PM
> To: Jeff Garzik
> Cc: Nivedita Singhvi; Werner Almesberger; netdev@oss.sgi.com;
> linux-kernel@vger.kernel.org
> Subject: Re: TOE brain dump
>
>
> Jeff Garzik wrote:
>
> > So, fix the other end of the pipeline too, otherwise this fast network
> > stuff is flashly but pointless.  If you want to serve up data from disk,
> > then start creating PCI cards that have both Serial ATA and ethernet
> > connectors on them :)  Cut out the middleman of the host CPU and host
>
> I for one would love to see something like this, and not just Serial ATA..
> but maybe 8x Serial ATA and RAID :)
>
> Ben
>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: TOE brain dump

[Werner Almesberger]

David Lang wrote:
> do you really want the processor on the card to be tunning
> apache/NFS/Samba/etc ?

If it runs a Linux kernel, that's not a problem. Whether you
actually want to do this or not, becomes an entirely separate
issue.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

RE: TOE brain dump

[Alan Shih]

On an embedded system, no processor will be fast enough to compete with
direct DMA xfer.  So just provide sendfile hooks that allow the kernel to
initiate data filling from source to dest then allow TSO to take place.
Kernel still needs to take care of the TCP stack.

I don't see this as building extensive customization though.

Alan

-----Original Message-----
From: David Lang [mailto:david.lang@digitalinsight.com]
Sent: Sunday, August 03, 2003 1:26 AM
To: Alan Shih
Cc: Ben Greear; Jeff Garzik; Nivedita Singhvi; Werner Almesberger;
netdev@oss.sgi.com; linux-kernel@vger.kernel.org
Subject: RE: TOE brain dump


do you really want the processor on the card to be tunning
apache/NFS/Samba/etc ?

putting enough linux on the card to act as a router (which would include
the netfilter stuff) is one thing. putting the userspace code that
interfaces with the outside world for file transfers is something else.

if you really want the disk connected to your network card you are just
talking a low-end linux box. forget all this stuff about it being on a
card and just use a full box (economys of scale will make this cheaper)

making a firewall that's a core system with a dozen slave systems attached
to it (the network cards) sounds like the type of clustering that Linux
has been used for for compute nodes. complicated to setup, but extremely
powerful and scalable once configured.

if you want more then a router on the card then Alan Cox is right, just
add another processor to the system, it's easier and cheaper.

David Lang


 On
Sat, 2 Aug 2003, Alan Shih wrote:

> Date: Sat, 2 Aug 2003 23:22:52 -0700
> From: Alan Shih <alan@storlinksemi.com>
> To: Ben Greear <greearb@candelatech.com>, Jeff Garzik <jgarzik@pobox.com>
> Cc: Nivedita Singhvi <niv@us.ibm.com>,
>      Werner Almesberger <werner@almesberger.net>, netdev@oss.sgi.com,
>      linux-kernel@vger.kernel.org
> Subject: RE: TOE brain dump
>
> A DMA xfer that fills the NIC pipe with IDE source. That's not very
hard...
> need a lot of bufferring/FIFO though.  May require large modification to
the
> file serving applications?
>
> Alan
>
> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Ben Greear
> Sent: Saturday, August 02, 2003 9:02 PM
> To: Jeff Garzik
> Cc: Nivedita Singhvi; Werner Almesberger; netdev@oss.sgi.com;
> linux-kernel@vger.kernel.org
> Subject: Re: TOE brain dump
>
>
> Jeff Garzik wrote:
>
> > So, fix the other end of the pipeline too, otherwise this fast network
> > stuff is flashly but pointless.  If you want to serve up data from disk,
> > then start creating PCI cards that have both Serial ATA and ethernet
> > connectors on them :)  Cut out the middleman of the host CPU and host
>
> I for one would love to see something like this, and not just Serial ATA..
> but maybe 8x Serial ATA and RAID :)
>
> Ben
>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: TOE brain dump

[Alan Cox]

On Sul, 2003-08-03 at 05:01, Ben Greear wrote:
> Jeff Garzik wrote:
> 
> > So, fix the other end of the pipeline too, otherwise this fast network 
> > stuff is flashly but pointless.  If you want to serve up data from disk, 
> > then start creating PCI cards that have both Serial ATA and ethernet 
> > connectors on them :)  Cut out the middleman of the host CPU and host 
> 
> I for one would love to see something like this, and not just Serial ATA..
> but maybe 8x Serial ATA and RAID :)

There is a protocol floating around for ATA over ethernet, no TCP layer
or nasty latency eating complexities in the middle

Re: TOE brain dump

[Ingo Oeser]

Hi Jeff,

On Sat, Aug 02, 2003 at 03:08:52PM -0400, Jeff Garzik wrote:
> So, fix the other end of the pipeline too, otherwise this fast network 
> stuff is flashly but pointless.  If you want to serve up data from disk, 
> then start creating PCI cards that have both Serial ATA and ethernet 
> connectors on them :)  Cut out the middleman of the host CPU and host 
> memory bus instead of offloading portions of TCP that do not need to be 
> offloaded.

Exactly what I suggested: sys_ioroute()

"Providing generic pipelines and io routing as Linux service"
Msg-ID: <20030718134235.K639@nightmaster.csn.tu-chemnitz.de>

on linux-kernel and linux-fsdevel

Be my guest.

I know, that you mean doing it in hardware, but you cannot
accelerate sth. which the kernel doesn't do ;-)

Regards

Ingo Oeser

RE: TOE brain dump

[Alan Shih]

Hmm,

So would main processor still need a copy of the data for re-transmission?
Won't that defeat the purpose?

Alan

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Ingo Oeser
Sent: Monday, August 04, 2003 7:36 AM
To: Jeff Garzik
Cc: Nivedita Singhvi; Werner Almesberger; netdev@oss.sgi.com;
linux-kernel@vger.kernel.org
Subject: Re: TOE brain dump


Hi Jeff,

On Sat, Aug 02, 2003 at 03:08:52PM -0400, Jeff Garzik wrote:
> So, fix the other end of the pipeline too, otherwise this fast network
> stuff is flashly but pointless.  If you want to serve up data from disk,
> then start creating PCI cards that have both Serial ATA and ethernet
> connectors on them :)  Cut out the middleman of the host CPU and host
> memory bus instead of offloading portions of TCP that do not need to be
> offloaded.

Exactly what I suggested: sys_ioroute()

"Providing generic pipelines and io routing as Linux service"
Msg-ID: <20030718134235.K639@nightmaster.csn.tu-chemnitz.de>

on linux-kernel and linux-fsdevel

Be my guest.

I know, that you mean doing it in hardware, but you cannot
accelerate sth. which the kernel doesn't do ;-)

Regards

Ingo Oeser

Re: TOE brain dump

[Ingo Oeser]

On Mon, Aug 04, 2003 at 10:19:21AM -0700, Alan Shih wrote:
> So would main processor still need a copy of the data for re-transmission?
> Won't that defeat the purpose?

No, since I didn't state that a retransmission is done along the
pipe, because you cannot go back in a pipeline.

A retransmission can be done at the end of the pipe, where this
can also be done in hardware.

Regards

Ingo Oeser

Re: TOE brain dump

[Alan Cox]

On Sad, 2003-08-02 at 18:04, Werner Almesberger wrote:
>  - last but not least, keeping TOE firmware up to date with the
>    TCP/IP stack in the mainstream kernel will require - for each
>    such TOE device - a significant and continuous effort over a
>    long period of time

or even the protocol and protocol refinements..

>  - instead of putting a different stack on the TOE, a
>    general-purpose processor (probably with some enhancements,
>    and certainly with optimized data paths) is added to the NIC

Like say an opteron in the 2nd socket on the motherboard

> Benefits:
> 
>  - putting the CPU next to the NIC keeps data paths short, and
>    allows for all kinds of optimizations (e.g. a pipelined
>    memory architecture)

It moves the cost it doesnt make it vanish

If I read you right you are arguing for a second processor running
Linux.with its own independant memory bus. AMD make those already its
called AMD64. I don't know anyone thinking at that level about
partitioning one as an I/O processor.

Re: TOE brain dump

[Werner Almesberger]

Alan Cox wrote:
> It moves the cost it doesnt make it vanish

I don't think it really can. What it can do is reduce the
overhead (which usually translates to latency and burstiness)
and the sharing.

> If I read you right you are arguing for a second processor running
> Linux.with its own independant memory bus. AMD make those already its
> called AMD64. I don't know anyone thinking at that level about
> partitioning one as an I/O processor.

That's taking this idea to an extreme, yes. I'd think of
using something as big as an amd64 for this as "too
expensive", but perhaps it's cheap enough in the long run,
compared to some "optimized" design.

It would certainly have the advantage of already solving
various consistency and compatibility issues. (That is, if
your host CPUs is/are also amd64.)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: TOE brain dump

[Alan Cox]

On Sad, 2003-08-02 at 23:14, Werner Almesberger wrote:
> That's taking this idea to an extreme, yes. I'd think of
> using something as big as an amd64 for this as "too
> expensive", but perhaps it's cheap enough in the long run,
> compared to some "optimized" design.

Volume makes cheap. If you look at software v hardware raid controllers
the hardware people are permanently being killed by the low volume of
cards.