TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)

[Jeff Garzik <jgarzik@pobox.com>]

Herbert Xu wrote:
> On Tue, Jun 27, 2006 at 11:24:25PM -0400, Jeff Garzik wrote:
>> I don't see how that position has changed?
>>
>> http://linux-net.osdl.org/index.php/TOE
> 
> Well I must say that RDMA over TCP smells very much like TOE.  They've
> got an ARP table, a routing table, and presumably a TCP stack.

A PCI device that presents itself as a SCSI controller, but under the 
hood is really iSCSI-over-TCP smells like TOE.  Running a virtualized 
Linux guest on top of a proprietary stack [which provides networking 
services to guests] also smells like TOE.  :)

If a TOE vendors wants to do TOE in a way that is transparent to the 
kernel, more power to them.  Such non-Linux TCP stack solutions still 
suffer many of the problems listed at the web page above, but at least 
they impose no burden on kernel maintenance.

i.e. we really _do not_ want to get into the habit of co-managing arp 
tables, routing tables, filtering rules, and dozens of other such 
resources with multiple remote, independent TCP stack.  We have enough 
complexity as it is today, coordinating between the random variations of 
SMP, uniprocessor, and NUMA machines out there.  Not to mention 
competing with under-the-hood firmware actions (ASF) on NICs.

As an aside, RDMA over TCP just seems silly.  TCP was _not_ meant to do 
the things that RDMA users want.  The infiniband/RDMA programming model 
is an ultra-low-latency polling model where one or two apps are allowed 
to completely consume the machine, either busy-waiting or processing 
messages.

Unfortunately I don't have more details, so you just get a generalized 
rant :)

	Jeff