Re: [RFC] RDMA bridge for ROCE and Infiniband

[Leon Romanovsky <leon@kernel.org>]

On Fri, Nov 12, 2021 at 01:04:13PM +0100, Christoph Lameter wrote:
> We have a larger Infiniband deployment and want to gradually move servers
> to a new fabric that is using RDMA over Ethernet using ROCE. The IB
> Fabric is mission critical and complex due to the need to have various
> ways to prevent failover with one of them being a secondary IB fabric.
> 
> It is not possible resourcewise to simply rebuild the system using ROCE
> and then switch over.
> 
> Both Fabrics use RDMA and we need some way for nodes on each cluster to
> communicate with each other. We do not really need memory to memory
> transfers. What we use from the RDMA
> stack is basically messaging and multicast. So the really
> core services of the RDMA stack are not needed. Also UD/UDP is sufficient,
> the other protocols may not be necessary to support.
> 
> Any ideas on how to do this would be appreciated. I have not found anything
> that could help us here, so we are interested in creating a new piece of
> Open Source RDMA software that allows the briding of native IB traffic to ROCE.
> 
> 
> Basic Design
> ------------
> Lets say we have a single system that functions as a *bridge* and has one
> interface going to Infiniband and one going to Ethernet with ROCE.
> 
> In our use case we do not need any "true" RDMA in the sense of memory to
> memory transfers. We only need UDP and UD messsaging and support for
> multicast.
> 
> In order to simplify the multicast aspects, the bridge will simply
> subscribe to the multicast groups of interest when the bridge software
> starts up.
> 
> PROXYARP for regular IP / IPoIB traffic
> --------------------------------------
> It is possible to do proxyarp on both sides of a Linux system that is
> connected both to Infiniband and ROCE. And thus this can already be seen
> as a single IP subnet from the Kernel stack perspective and communication
> is not a problem for non RDMA traffic.
> 
> PROXYARP means that the MAC address of the bridge is used for all IPoIB
> addresses on the IB side of the bridge. Similar the GID of the bridge is
> used in all IPoIB Packets on the IB side of the bridge that come from the
> ROCE side.
> 
> The kernel already removes and adds IPoIB and IP headers as needed. So
> this works for regular IP traffic but not for native IB / RDMA packets.
> IP traffic is only used for non performance critical aspects of
> our application and so the performance on this level is not a concern.
> 
> Each of the host in the bridge IP subnet has 3 addresses: An IPv4
> address, a MAC address and a GID.
> 
> 
> RDMA Packets (Native IB and ROCE)
> =================================
> ROCE v2 packets are basically IB packets with another header on top so
> the simplistic idealistic version of how this is going to
> work is by stripping and adding the UDP ROCE v2 headers to the IB packet.
> 
> ROCE packets
> ------------
> UDP roce packets send to the IP addresses on the ROCE side have the MAC
> address of the bridge. So these already contain the IP address for the
> other side that can be used to lookup the GID in order to convert the
> packet and forward it to the Infiniband node.
> 
> UD packets
> ----------
> Routing capabilities are limited on the Infiniband side but one could
> construct a way to map GIDs for the hosts on the ROCE side to the LID
> of the bridge by using the ACM daemon.
> 
> There will be complications regards to RDMA_CM support and the details
> of mapping characteristics between packets but hopefully this will
> be fairly manageable.
> 
> Multicast packets
> -----------------
> 
> Multicast packets can be converted easily since there is a direct
> mapping possible between the MAC address used for a Multicast group and
> the MGID in the Infiniband fabric. Otherwise this process is similar
> to UD/UDP traffic.
> 
> 
> Implementation
> ==============
> 
> There are basically three ways to implement this:
> 
> A) As add on to the RDMA stack in the Linux Kernel
> B) As a user space process that operates on RAW sockets and receives traffic
>    filtered by the NIC or by the kernel to process. It wouild use the same
>    raw sockets to send these packets to the other side.
> C) In firmware / logic of the NIC. This is out of reach of us here
>    I think.
> 
> 
> Inbound from ROCE
> -----------------
> This can be done like in the RXE driver. Simply listening to the UDP port
> for ROCE will get us the traffic we need to operate on.
> 
> Outbound to ROCE
> ----------------
> A Ethernet RAW socket will allow us to create arbitrary datagrams as needed.
> This has been widely done before in numerous settings and code is already
> open sources that does stuff like this. So this is fairly straightforward.
> 
> Inbound from Infiniband
> -----------------------
> The challenge here is to isolate the traffic that is destined for the bridge
> itself from traffic that needs to be forwarded. One way would be to force
> the inclusion of a Global Header in each packet so that the GID can be
> matched. When the GID does not match the bridge then the traffic would be
> forwarded to the code which could then do the necessary packet conversion.
> 
> I do not know of any way that something like this has been done before.
> Potentially this means working with flow steering and dealing with firmware
> issues in the NIC.
> 
> Outbound to Infiniband
> ----------------------
> I saw in a recent changelog for the Mellanox NICs that the ability has
> been added to send raw IB datagrams. If that can be used to construct
> a packet that is coming from one of the GIDs associated with the ROCE IP
> addresses then this will work.
> 
> Otherwise we need to have some way to set the GID for outbound packets
> to make this work.
> 
> The logic needed on Infiniband is similar to that required for an
> Infiniband router.
> 
> 
> 
> 
> The biggest risk here seems to be the Infiniband side of things. Is there
> a way to create a filter for the traffic we need?
> 
> Any tips and suggestions on how to approach this problem would be appreciated.

Mellanox has Skyway product, which is IB to ETH gateway.
https://www.nvidia.com/en-us/networking/infiniband/skyway/

I imagine that it can be extended to perform IB to RoCE too,
because it uses steering to perform IB to ETH translation.

Thanks

> 
> 
> Christoph Lameter, 12. November 2021
>