Seeking Guidance: Creating an IBV Multicast Group?

Seeking Guidance: Creating an IBV Multicast Group?

[Andrew Sheinberg]

Hello all,

I’m not sure if this the right place to ask, but I will give it a try.

I have a system with many initialized UD queue pairs (info for address handle creations and qp numbers exchanged out-of-band). I am only using libibverbs for establishment (purposefully not using librdmacm, to allow for more flexible environment configuration) — everything is working smoothly for unicast.  Now I would like to create a multicast group and attach some of these queue pairs (ibv_mcast_attach); however I am struggling to find any details on how to create such a group (and obtain a proper MGID and MLID). 

I found a few examples online but am left with questions:
	- There is code within perftest's "multicast_resources.c", but this seems a bit hacky and oddly verbose
	- There is code within Nvidia Docs’  "Programming Examples using IBV” showcasing joining an already created multicast group at a given IP address using rdma_cm, but It is unclear how to create the group in the first place


Questions (please correct me if these do not make sense):

1. What is the role of the OpenSM — is there a C API?
	- Are there any examples using opensm programmatically and not with CLI?
	- Does the API differ on InfiniBand vs. RoCEv2 fabric?

2. Is there any high-level documentation to describe the role of libibumad? (Looking at the man pages on a per-function basis is a bit too fine-grained for my understanding as of now).
	- I also see libibmad — what is the responsibility breakdown between these two?
	- How do they relate to OpenSM?

Any guidance is greatly appreciated.

Thanks,
Andrew Sheinberg

Re: Seeking Guidance: Creating an IBV Multicast Group?

[Zhu Yanjun]

在 2024/7/30 1:10, Andrew Sheinberg 写道:
> Hello all,
> 
> I’m not sure if this the right place to ask, but I will give it a try.
> 
> I have a system with many initialized UD queue pairs (info for address handle creations and qp numbers exchanged out-of-band). I am only using libibverbs for establishment (purposefully not using librdmacm, to allow for more flexible environment configuration) — everything is working smoothly for unicast.  Now I would like to create a multicast group and attach some of these queue pairs (ibv_mcast_attach); however I am struggling to find any details on how to create such a group (and obtain a proper MGID and MLID).
> 
> I found a few examples online but am left with questions:
> 	- There is code within perftest's "multicast_resources.c", but this seems a bit hacky and oddly verbose
> 	- There is code within Nvidia Docs’  "Programming Examples using IBV” showcasing joining an already created multicast group at a given IP address using rdma_cm, but It is unclear how to create the group in the first place
> 
> 
> Questions (please correct me if these do not make sense):
> 
> 1. What is the role of the OpenSM — is there a C API?
> 	- Are there any examples using opensm programmatically and not with CLI?
> 	- Does the API differ on InfiniBand vs. RoCEv2 fabric?

I have made tests with Infiniband vs. RoCEv2. From my perspective, I 
think, there are some differences between the 2 fabrics.

To the basic functionalities, the difference between 2 fabrics is small. 
But to the extended functionalities, the difference is big.

The difference is based on the different features.

I just made some simple tests. Perhaps some engineers who made a lot of 
tests can give more suggestions about this.

Zhu Yanjun

> 
> 2. Is there any high-level documentation to describe the role of libibumad? (Looking at the man pages on a per-function basis is a bit too fine-grained for my understanding as of now).
> 	- I also see libibmad — what is the responsibility breakdown between these two?
> 	- How do they relate to OpenSM?
> 
> Any guidance is greatly appreciated.
> 
> Thanks,
> Andrew Sheinberg

Re: Seeking Guidance: Creating an IBV Multicast Group?

[Leon Romanovsky]

On Mon, Jul 29, 2024 at 01:10:29PM -0400, Andrew Sheinberg wrote:
> Hello all,
> 
> I’m not sure if this the right place to ask, but I will give it a try.
> 
> I have a system with many initialized UD queue pairs (info for address handle creations and qp numbers exchanged out-of-band). I am only using libibverbs for establishment (purposefully not using librdmacm, to allow for more flexible environment configuration) — everything is working smoothly for unicast.  Now I would like to create a multicast group and attach some of these queue pairs (ibv_mcast_attach); however I am struggling to find any details on how to create such a group (and obtain a proper MGID and MLID). 

First call to ibv_attach_mcast() will create a new multicast group.

> 
> I found a few examples online but am left with questions:
> 	- There is code within perftest's "multicast_resources.c", but this seems a bit hacky and oddly verbose
> 	- There is code within Nvidia Docs’  "Programming Examples using IBV” showcasing joining an already created multicast group at a given IP address using rdma_cm, but It is unclear how to create the group in the first place
> 
> 
> Questions (please correct me if these do not make sense):
> 
> 1. What is the role of the OpenSM — is there a C API?

SM stands for subnet management and its goal to oversee fabric topology
and manage path routing among the nodes.

Thanks

Re: Seeking Guidance: Creating an IBV Multicast Group?

[Konstantin Taranov]

> I have a system with many initialized UD queue pairs (info for address handle
> creations and qp numbers exchanged out-of-band). I am only using libibverbs
> for establishment (purposefully not using librdmacm, to allow for more flexible
> environment configuration) — everything is working smoothly for unicast.  Now
> I would like to create a multicast group and attach some of these queue pairs
> (ibv_mcast_attach); however I am struggling to find any details on how to create
> such a group (and obtain a proper MGID and MLID).
> 

Hey,

I only worked with multicast on Infiniband, and for that I used `saquery -g` to get
available configured multicast lids and gids. I guess they were preconfigured in the SM
in the rack I used.

> 1. What is the role of the OpenSM — is there a C API?
>         - Are there any examples using opensm programmatically and not with CLI?
>         - Does the API differ on InfiniBand vs. RoCEv2 fabric?
> 

There is no SM for RoCEv2. You need a SM in IB to get lids of your adapters and configure pkeys and etc.
I am not an expert in SM but there are pages explaining how to configure it correctly.
Typically, you have one node running SM server or SM is in an IB switch.

- Konstantin