RDMA Multicasting

RDMA Multicasting

[Allen Andrews]

All,

I am trying to find out if RDMA Multicasting is supported on RoCE in Linux.
If so, is it supported above the verbs interface?

Allen Andrews
Emulex

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA Multicasting

[Christoph Lameter]

On Wed, 8 Apr 2015, Allen Andrews wrote:

> I am trying to find out if RDMA Multicasting is supported on RoCE in Linux.
> If so, is it supported above the verbs interface?

RDMA multicasting? You mean sending ud(or udp) messages via the RDMA API?
That works even without RoCE.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA Multicasting

[Christoph Lameter]

On Thu, 9 Apr 2015, Caitlin Bestler wrote:

> RDMA requires specific delivery semantics. Successful completion of an
> untagged message (an RDMA Send) implies that all prior tagged
> (Write/Read) packets have been successfully placed in user memory.

That seems to be incompatible with a multicast delivery mechanism
because there is state to all endpoints involved.

> Just putting RDMA packets over an unreliable transport will not accomplish that.

I am not sure what an "RDMA packet" is. RDMA is a memory to memory
transfer action. A packet is information send on a medium. A packet that
describes an RDMA action to be taken between certain endpoints?

Infiniband is lossless and thus what "unreliable" means is also quite
foggy.

> You are probably better off using RDMA ideas over UDP/UD, and doing the
> direct memory placement from your own code, instead.

So send the memory transfer info via multicast datagram to the
endpoints and then run the transfer from the endpoint.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA Multicasting

[Christoph Lameter]

On Thu, 9 Apr 2015, Caitlin Bestler wrote:

> > Infiniband is lossless and thus what "unreliable" means is also quite
> > foggy.
>
> InfiniBand is not lossless. It does a superb job of avoiding drops caused
> by congestion. But applications should not assume that all UD messages
> have been received, but should add their own checking. That is why the
> U stands for Unreliable.

Well applications can screw up yes but the fabric *IS* lossless.

> > > You are probably better off using RDMA ideas over UDP/UD, and doing the
> > > direct memory placement from your own code, instead.
> >
> > So send the memory transfer info via multicast datagram to the
> > endpoints and then run the transfer from the endpoint.
>
> Yes, possibly in a kermel module.

Why? You can simply do this already from userspace with verbs messaging
and RDMA tranfers.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA Multicasting

[Atchley, Scott]

On Apr 10, 2015, at 11:14 AM, Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org> wrote:

> On Thu, 9 Apr 2015, Caitlin Bestler wrote:
> 
>>> Infiniband is lossless and thus what "unreliable" means is also quite
>>> foggy.
>> 
>> InfiniBand is not lossless. It does a superb job of avoiding drops caused
>> by congestion. But applications should not assume that all UD messages
>> have been received, but should add their own checking. That is why the
>> U stands for Unreliable.
> 
> Well applications can screw up yes but the fabric *IS* lossless.

On a very rare bit flip, the switches or HCAs can drop a UD packet. Lossless, but not guaranteed.

> 
>>>> You are probably better off using RDMA ideas over UDP/UD, and doing the
>>>> direct memory placement from your own code, instead.
>>> 
>>> So send the memory transfer info via multicast datagram to the
>>> endpoints and then run the transfer from the endpoint.
>> 
>> Yes, possibly in a kermel module.
> 
> Why? You can simply do this already from userspace with verbs messaging
> and RDMA tranfers.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA Multicasting

[Bob Ciotti]

On 4/10/15 8:14 AM, Christoph Lameter wrote:
> On Thu, 9 Apr 2015, Caitlin Bestler wrote:
>
>>> Infiniband is lossless and thus what "unreliable" means is also quite
>>> foggy.
>>
>> InfiniBand is not lossless. It does a superb job of avoiding drops caused
>> by congestion. But applications should not assume that all UD messages
>> have been received, but should add their own checking. That is why the
>> U stands for Unreliable.
>
> Well applications can screw up yes but the fabric *IS* lossless.

This is a very bad assumption. Fabric *is not* lossless.

Packet loss happens - for various reasons that are typically specific to the environment.
We have found several broken pieces of SW that assumed RC connections are reliable, and
do bad things when the qp hits its retry limit. So not only loosing 1 packet, but 7 in a
row. Building SW that relied on RC - well OK, but assuming UD reliable in all environments?
If HW never broke, and systems were never overloaded, room temperatures never fluctuated, IB
cables were always reliable... Its not that uncommon to see packet loss on a perfectly functioning
(large) system let alone one thats having HW issues...


>>>> You are probably better off using RDMA ideas over UDP/UD, and doing the
>>>> direct memory placement from your own code, instead.
>>>
>>> So send the memory transfer info via multicast datagram to the
>>> endpoints and then run the transfer from the endpoint.
>>
>> Yes, possibly in a kermel module.
>
> Why? You can simply do this already from userspace with verbs messaging
> and RDMA tranfers.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDMA Multicasting

[Christoph Lameter]

On Fri, 10 Apr 2015, Bob Ciotti wrote:

> > Well applications can screw up yes but the fabric *IS* lossless.
>
> This is a very bad assumption. Fabric *is not* lossless.

I guess then we need to get our money back because it was advertised to be
lossless....

> Packet loss happens - for various reasons that are typically specific to the
> environment.

Right. Even the most reliable transport will not work if someone goes
postal and gets a big sledgehammer and starts hitting the equipment with
that.

> If HW never broke, and systems were never overloaded, room temperatures never
> fluctuated, IB
> cables were always reliable... Its not that uncommon to see packet loss on a
> perfectly functioning
> (large) system let alone one thats having HW issues...

If things break then yes but otherwise the fabric is expected to behave as
advertised.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html