rxe mcast

rxe mcast

[Bob Pearson]

Jason,

I wrote a test program to verify that multicast is working in rxe
since there don't seem to be any pyverbs tests that actually test this.
It turns out that it doesn't work for several reasons most of which
I have fixed. But there is one that I can't figure out.

The rxe driver calls dev_mc_add() with a MAC address to add the
multicast MAC address to the device which shows up when you type
'ip maddr'. But nothing comes through from the network to the driver.
Wireshark does see the packets but they don't get to the IP or
UDP layers in the netdev stack.

If I run iperf -u and bind to a multicast address it works fine and
when you type ip maddr you get both a link: MAC address and an
inet: IP multicast address. In fact if you start an iperf server with 
the same multicast address I am testing with and just let it sit
there my test traffic also come through to the rxe driver.
So creating the IP mcast address seems critical to letting the
netdev stack receive traffic.

I tried creating a separate UDP socket bound to the mcast address
but it doesn't seem to create the required IP address.

Any ideas would be welcome.

Bob

Re: rxe mcast

[Jason Gunthorpe]

On Wed, Nov 01, 2023 at 01:07:28PM -0500, Bob Pearson wrote:
> Jason,
> 
> I wrote a test program to verify that multicast is working in rxe
> since there don't seem to be any pyverbs tests that actually test this.
> It turns out that it doesn't work for several reasons most of which
> I have fixed. But there is one that I can't figure out.
> 
> The rxe driver calls dev_mc_add() with a MAC address to add the
> multicast MAC address to the device which shows up when you type
> 'ip maddr'. But nothing comes through from the network to the driver.
> Wireshark does see the packets but they don't get to the IP or
> UDP layers in the netdev stack.

I think you also need to attach a multicast IP address to the RXE
socket ?

> So creating the IP mcast address seems critical to letting the
> netdev stack receive traffic.

Yes
 
> I tried creating a separate UDP socket bound to the mcast address
> but it doesn't seem to create the required IP address.

I don't think that is how you do it... There is a set sock opt thing
you need, IIRC. It has been a long time since I last wrote code for it

Check what iperf did and that will be the basic thing that has to be
cloned in the kernel

Jason

Re: rxe mcast

[Bob Pearson]

On 11/1/23 13:29, Jason Gunthorpe wrote:
> On Wed, Nov 01, 2023 at 01:07:28PM -0500, Bob Pearson wrote:
>> Jason,
>>
>> I wrote a test program to verify that multicast is working in rxe
>> since there don't seem to be any pyverbs tests that actually test this.
>> It turns out that it doesn't work for several reasons most of which
>> I have fixed. But there is one that I can't figure out.
>>
>> The rxe driver calls dev_mc_add() with a MAC address to add the
>> multicast MAC address to the device which shows up when you type
>> 'ip maddr'. But nothing comes through from the network to the driver.
>> Wireshark does see the packets but they don't get to the IP or
>> UDP layers in the netdev stack.
> 
> I think you also need to attach a multicast IP address to the RXE
> socket ?
> 
>> So creating the IP mcast address seems critical to letting the
>> netdev stack receive traffic.
> 
> Yes
>   
>> I tried creating a separate UDP socket bound to the mcast address
>> but it doesn't seem to create the required IP address.
> 
> I don't think that is how you do it... There is a set sock opt thing
> you need, IIRC. It has been a long time since I last wrote code for it
> 
> Check what iperf did and that will be the basic thing that has to be
> cloned in the kernel
> 
> Jason


Thanks Jason,

That found it. I traced set socket opt down to ip_mc_join/leave_group
and ipv6_sock_mc_join/drop which seems to be working. Now ip maddr
reports both the link and inet addresses and packets are getting up
to the rxe driver without running iperf on the side.

Some fixes coming soon.

Bob