Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)

[Steve Wise <swise@opengridcomputing.com>]

On Wed, 2006-06-28 at 14:29 +1000, Herbert Xu wrote:
> On Wed, Jun 28, 2006 at 12:18:25AM -0400, Jeff Garzik wrote:
> > 
> > A PCI device that presents itself as a SCSI controller, but under the 
> > hood is really iSCSI-over-TCP smells like TOE.  Running a virtualized 
> > Linux guest on top of a proprietary stack [which provides networking 
> > services to guests] also smells like TOE.  :)
> 
> Agreed.  However, when they start adding hooks to the ARP table, the
> routing table, and PMTU management, it begs the question what more is
> there to add for TOE (well, user-space driven TOE at least)?
>  
> > Unfortunately I don't have more details, so you just get a generalized 
> > rant :)
> 
> OK, the patch under discussion here adds hooks to all the stuff in the
> previous paragraph for the purpose of RDMA over TCP (well I must say
> that the exact RDMA application/hardware has never been clearly given
> but this is what I can gather from the previous posts).

There are Ammasso and Chelsio RDMA/Ethernet drivers in the openib.org
svn iwarp branch today.  The goal is to submit them for review and
inclusion into linux.  The Ammasso driver has been through 3 review
cycles on lkml and netdev.  There are other vendors with drivers, but
they're currently not disclosing any information to me about their
status.

Applications: 

kernel: NFS-RDMA, iSER, RDP.  
user:  MPI, uDAPL (both middle ware).

The Ammasso driver is a different model.  It actually has a full
TCP/ARP/ICMP stack and doesn't require these hooks.  But the RDMA/TCP
model defined and implemented, I think, by most vendors is a model where
the HW is doing a limited TCP offload, relying on the native stack for
L2 and L3 integration (as described in the netevent patch).


> Put it another way, I think the dividing line between TOE and iSCSI or
> virtualisation is exactly the interface between them and the Linux kernel.
> If the interface is an existing one such as SCSI or standard IP then it's
> OK.  However, when it starts poking in the guts of the Linux stack I'd say
> that it has crossed the line.
> 

Don't these netevent hooks have utility for other purposes?  IE:  Should
we really shoot changes to linux _just because_ they might possibly
enable TOE? 

Steve.