Re: TOE brain dump

Andre Hedrick
2003-08-06T07:23:11Z

Jeff,

Do be sure to check that your data payload is correct.
Everyone knows that a router/gateway/switch with a sticky bit in its
memory will recompute the net crc16 checksum insure it pass the to the nic
regardless. It is amazing how much data can be corrupted by a network
environment via all the NFS/NBD/etc wantabie storage products out there.

Just a chuckle for you to ponder.

--a

On Sun, 3 Aug 2003, Jeff Garzik wrote:

> Werner Almesberger wrote:
> > Jeff Garzik wrote:
> >
> >>jabbering at the same time. TCP is a "one size fits all" solution, but
> >>it doesn't work well for everyone.
> >
> >
> > But then, ten "optimized xxPs" that work well in two different
> > scenarios each, but not so good in the 98 others, wouldn't be
> > much fun either.
> >
> > It's been tried a number of times. Usually, real life sneaks
> > in at one point or another, leaving behind a complex mess.
> > When they've sorted out these problems, regular TCP has caught
> > up with the great optimized transport protocols. At that point,
> > they return to their niche, sometimes tail between legs and
> > muttering curses, sometimes shaking their fist and boldly
> > proclaiming how badly they'll rub TCP in the dirt in the next
> > round. Maybe they shed off some of the complexity, and trade it
> > for even more aggressive optimization, which puts them into
> > their niche even more firmly. Eventually, they fade away.
> >
> > There are cases where TCP doesn't work well, like a path of
> > badly mismatched link layers, but such paths don't treat any
> > protocol following the end-to-end principle kindly.
> >
> > Another problem of TCP is that it has grown a bit too many
> > knobs you need to turn before it works over your really fast
> > really long pipe. (In one of the OLS after dinner speeches,
> > this was quite appropriately called the "wizard gap".)
> >
> >
> >>It's obviously not over a WAN...
> >
> >
> > That's why NFS turned off UDP checksums ;-) As soon as you put
> > it on IP, it will crawl to distances you didn't imagine in your
> > wildest dreams. It always does.
>
> Really fast, really long pipes in practice don't exist for 99.9% of all
> Internet users.
>
>
> When you approach traffic levels that push you want to offload most of
> the TCP net stack, then TCP isn't the right solution for you anymore,
> all things considered.
>
>
> The Linux net stack just isn't built to be offloaded. TOE engines will
> either need to (1) fall back to Linux software for all-but-the-common
> case (otherwise netfilter, etc. break), or, (2) will need to be
> hideously complex beasts themselves. And I can't see ASIC and firmware
> designers being excited about implementing netfilter on a PCI card :)
>
> Unfortunately some vendors seem to choosing TOE option #3: TCP offload
> which introduces many limitations (connection limits, netfilter not
> supported, etc.) which Linux never had before. Vendors don't seem to
> realize TOE has real potential to damage the "good network neighbor"
> image the net stack has. The Linux net stack's behavior is known,
> documented, predictable. TOE changes all that.
>
> There is one interesting TOE solution, that I have yet to see created:
> run Linux on an embedded processor, on the NIC. This stripped-down
> Linux kernel would perform all the header parsing, checksumming, etc.
> into the NIC's local RAM. The Linux OS driver interface becomes a
> virtual interface with a large MTU, that communicates from host CPU to
> NIC across the PCI bus using jumbo-ethernet-like data frames.
> Management frames would control the ethernet interface on the other side
> of the PCI bus "tunnel".
>
>
> >>So, fix the other end of the pipeline too, otherwise this fast network
> >>stuff is flashly but pointless. If you want to serve up data from disk,
> >>then start creating PCI cards that have both Serial ATA and ethernet
> >>connectors on them :) Cut out the middleman of the host CPU and host
> >>memory bus instead of offloading portions of TCP that do not need to be
> >>offloaded.
> >
> >
> > That's a good point. A hierarchical memory structure can help
> > here. Moving one end closer to the hardware, and letting it
> > know (e.g. through sendfile) that also the other end is close
> > (or can be reached more directly that through some hopelessly
> > crowded main bus) may help too.
>
> Definitely.
>
> Jeff
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>