RE: InfiniBand BOF @ LSM - topics of interest

Woodruff, Robert J
2002-05-15T23:58:35Z

>OR is it saner to layer TCP/IP etc. over IB... it seems to me the point
>others were making was that there is more to "network" style services
>than just passing bits from here to there...will IB have low level
>support many of the features people have come to expect?

Yes someone will layer IP over IB to support the gazillion applications
that currently run over that protocol without any code changes. This should
perform
as well as 10 gigabit Ethernet, but may cost less.

However, since InfiniBand is already a reliable transport, one can bypass
TCP,
so someone could invent a new address family for sockets,
say AF_INFINIBANDO, that is much more light weight than the existing TCP/IP
stack.
Thus with a small change to the application, a good performance increase can
be attained.
This is probably how the InfiniBand standard sockets direct protocol will be
initially
implemented for Linux. Next, one could entertain a concept similar to
Winsock Direct,
where no change is needed to the application, and the kernel has a swtich in
the
AF_INET code path that bypasses the TCP processing for packets destined for
the InfiniBand subnet. This will likely have to come later for Linux
after the performance value of bypassing TCP has
actually been demonstrated. People are currently skeptical that
a concept that has been demonstrated to have performance advantages
in Windows 2000 will achieve the same result if implemented in
Linux.

Finally, there are those that want to run on almost the bare metal. These
are
typically the high performance computing types, or some of the large data
base
vendors that want to use InfiniBand for clustering.
For them, there is an InfiniBand Access Layer that allows
direct access to the hardware (via a thin S/W layer)
from user space applications. One could
probably mmap() an ethernet NIC registers into user space, but
probably only one process.
InfiniBand was designed to specifically allow mmap()ing the hardware
into user space, even for multiple processes.

As for a high level answer to the security question,
InfiniBand has the concept of protection domains
and read and write keys with checking built into the hardware.
This provides protection from people RDMAing to the wrong place.

Kind of a high level simplistic view, see the InfiniBand
specification for details. There is also an open source
project on sourceforege that is working on code. See
the home page for details.