Re: TOE brain dump

[Werner Almesberger <werner@almesberger.net>]

Eric W. Biederman wrote:
> MPI is not a transport.  It an interface like the Berkeley sockets
> layer.

Hmm, but doesn't it also unify transport semantics (i.e. chop
TCP streams into messages), maybe add reliability to transports
that don't have it, and provide addressing ? Okay, perhaps you
wouldn't call this a transport in the OSI sense, but it still
seems to have considerably more functionality than just
providing an API.

> Mostly I think the that is less true, at least if they can stand the
> process of severe code review and cleaning up their code.

Hmm, people putting dozens of millions into building clusters
can't afford to have what is probably their most essential
infrastructure code reviewed and cleaned up ? Oh dear.

> But of course to get through the peer review process people need
> to understand what they are doing.

A good point :-)

> So store and forward of packets in a 3 layer switch hierarchy, at 1.3 us
> per copy.

But your switch could just do cut-through, no ? Or do they
need to recompute checksums ?

> A lot of the NICs which are used for MPI tend to be smart for two
> reasons.  1) So they can do source routing. 2) So they can safely
> export some of their interface to user space, so in the fast path
> they can bypass the kernel.

The second part could be interesting for TOE, too. Only that
the interface exported would just be the socket interface.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/