Question on DSA switches, IGMP forwarding and switchdev

Question on DSA switches, IGMP forwarding and switchdev

[Daniel Mack]

Hi,

I'm working on a custom board featuring a Marvell mv88e6085 Ethernet
switch controlled by the Linux DSA driver, and I'm facing an issue with
IGMP packet flows.

Consider two Ethernet stations, each connected to the switch on a
dedicated port. A Linux bridge combines the two ports. In my setup, I
need these two stations to send and receive multicast traffic, with IGMP
snooping enabled.

When an IGMP query enters the switch, it is redirected to the CPU port
as all 'external' ports are configured for IGMP/MLP snooping by the
driver. The issue that I'm seeing is that the Linux bridge does not
forward the IGMP frames to any other port, no matter whether the bridge
is in snooping mode or not. This needs to happen however, otherwise the
stations will not see IGMP queries, and unsolicited membership reports
are not being transferred either.

I've traced these frames through the bridge code and figured forwarding
fails in should_deliver() in net/bridge/br_forward.c because
nbp_switchdev_allowed_egress() denies it due to the fact that the frame
has already been forwarded by the same parent device. This check causes
all manual software forwarding of frames between two such switch ports
to fail. Note that IGMP traffic is the only class of communication that
is affected by this as it is not handled in hardware.

So my question now is how to fix that. Would the DSA driver need to mark
the ports as independent somehow?


Thanks,
Daniel

Re: Question on DSA switches, IGMP forwarding and switchdev

[Andrew Lunn]

On Fri, Jun 19, 2020 at 11:31:04PM +0200, Daniel Mack wrote:
> Hi,
> 
> I'm working on a custom board featuring a Marvell mv88e6085 Ethernet
> switch controlled by the Linux DSA driver, and I'm facing an issue with
> IGMP packet flows.
> 
> Consider two Ethernet stations, each connected to the switch on a
> dedicated port. A Linux bridge combines the two ports. In my setup, I
> need these two stations to send and receive multicast traffic, with IGMP
> snooping enabled.
> 
> When an IGMP query enters the switch, it is redirected to the CPU port
> as all 'external' ports are configured for IGMP/MLP snooping by the
> driver. The issue that I'm seeing is that the Linux bridge does not
> forward the IGMP frames to any other port, no matter whether the bridge
> is in snooping mode or not. This needs to happen however, otherwise the
> stations will not see IGMP queries, and unsolicited membership reports
> are not being transferred either.

Hi Daniel

I think all the testing i've done in this area i've had the bridge
acting as the IGMP queirer. Hence it has replied to the query, rather
than forward it out other ports.

So this could be a bug.

> I've traced these frames through the bridge code and figured forwarding
> fails in should_deliver() in net/bridge/br_forward.c because
> nbp_switchdev_allowed_egress() denies it due to the fact that the frame
> has already been forwarded by the same parent device.

To get this far, has the bridge determined it is not the elected
querier?  I guess it must of done. Otherwise it would not be
forwarding it.

> So my question now is how to fix that. Would the DSA driver need to mark
> the ports as independent somehow?

The problem here is:

https://elixir.bootlin.com/linux/v5.8-rc1/source/net/dsa/tag_edsa.c#L159

Setting offload_fwd_mark means the switch has forwarded the frame as
needed to other ports of the switch. If the frame is an IGMP query
frame, and the bridge is not the elected quierer, i guess we need to
set this false? Or we need an FDB in the switch to forward it. What
group address is being used?

    Andrew

RE: Question on DSA switches, IGMP forwarding and switchdev

[Jason Cobham]

> -----Original Message-----
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Andrew Lunn
> Sent: Friday, June 19, 2020 2:58 PM
> To: Daniel Mack
> Cc: netdev@vger.kernel.org; Ido Schimmel; Jiri Pirko; Ivan Vecera; Florian
> Fainelli
> Subject: Re: Question on DSA switches, IGMP forwarding and switchdev
> 
> On Fri, Jun 19, 2020 at 11:31:04PM +0200, Daniel Mack wrote:
> > Hi,
> >
> > I'm working on a custom board featuring a Marvell mv88e6085 Ethernet
> > switch controlled by the Linux DSA driver, and I'm facing an issue with
> > IGMP packet flows.
> >
> > Consider two Ethernet stations, each connected to the switch on a
> > dedicated port. A Linux bridge combines the two ports. In my setup, I
> > need these two stations to send and receive multicast traffic, with IGMP
> > snooping enabled.
> >
> > When an IGMP query enters the switch, it is redirected to the CPU port
> > as all 'external' ports are configured for IGMP/MLP snooping by the
> > driver. The issue that I'm seeing is that the Linux bridge does not
> > forward the IGMP frames to any other port, no matter whether the bridge
> > is in snooping mode or not. This needs to happen however, otherwise the
> > stations will not see IGMP queries, and unsolicited membership reports
> > are not being transferred either.
> 
> Hi Daniel
> 
> I think all the testing i've done in this area i've had the bridge
> acting as the IGMP queirer. Hence it has replied to the query, rather
> than forward it out other ports.
> 
> So this could be a bug.
> 
> > I've traced these frames through the bridge code and figured forwarding
> > fails in should_deliver() in net/bridge/br_forward.c because
> > nbp_switchdev_allowed_egress() denies it due to the fact that the frame
> > has already been forwarded by the same parent device.
> 
> To get this far, has the bridge determined it is not the elected
> querier?  I guess it must of done. Otherwise it would not be
> forwarding it.
> 
> > So my question now is how to fix that. Would the DSA driver need to mark
> > the ports as independent somehow?
> 
> The problem here is:
> 
> https://elixir.bootlin.com/linux/v5.8-rc1/source/net/dsa/tag_edsa.c#L159
> 
> Setting offload_fwd_mark means the switch has forwarded the frame as
> needed to other ports of the switch. If the frame is an IGMP query
> frame, and the bridge is not the elected quierer, i guess we need to
> set this false? Or we need an FDB in the switch to forward it. What
> group address is being used?
> 
>     Andrew

I've run into the same issue. To resolve it,  In my case, in the same file, I've had to send all IGMP control traffic to the CPU:
	skb->offload_fwd_mark = 1;
	switch (ih->type) {
		case IGMP_HOST_MEMBERSHIP_REPORT:
		case IGMPV2_HOST_MEMBERSHIP_REPORT:
		case IGMPV3_HOST_MEMBERSHIP_REPORT:
		case IGMP_HOST_MEMBERSHIP_QUERY:
		case IGMP_HOST_LEAVE_MESSAGE:
			skb->offload_fwd_mark = 0;
		break;
	}

I'd be interested if there is a better way.

Jason

Re: Question on DSA switches, IGMP forwarding and switchdev

[Andrew Lunn]

> I've run into the same issue. To resolve it,  In my case, in the same file, I've had to send all IGMP control traffic to the CPU:
> 	skb->offload_fwd_mark = 1;
> 	switch (ih->type) {
> 		case IGMP_HOST_MEMBERSHIP_REPORT:
> 		case IGMPV2_HOST_MEMBERSHIP_REPORT:
> 		case IGMPV3_HOST_MEMBERSHIP_REPORT:
> 		case IGMP_HOST_MEMBERSHIP_QUERY:
> 		case IGMP_HOST_LEAVE_MESSAGE:
> 			skb->offload_fwd_mark = 0;
> 		break;
> 	}
> 
> I'd be interested if there is a better way.

It might depend on the switch generation, but i think some switches
indicate the sort of packet in the DSA header. For 6390, Octet 3 of
the header, bits 3-5 contains a code.

0=BDPU
1=Frame2Reg
2=IGMP/MLD
3=Policy
4=ARP Mirror
5=Policy Mirror

We can look at these bits and not set skb->offload_fwd_mark depending
on its value.

The 6352 family has the same bits. 6161 has a few less bits, but does
have IGMP/MLD. I don't know about the 6085. Do you have the datasheet?

     Andrew

Re: Question on DSA switches, IGMP forwarding and switchdev

[Daniel Mack]

Hi Andrew,

Thanks a lot for the quick reply!

On 6/19/20 11:58 PM, Andrew Lunn wrote:
> On Fri, Jun 19, 2020 at 11:31:04PM +0200, Daniel Mack wrote:

>> When an IGMP query enters the switch, it is redirected to the CPU port
>> as all 'external' ports are configured for IGMP/MLP snooping by the
>> driver. The issue that I'm seeing is that the Linux bridge does not
>> forward the IGMP frames to any other port, no matter whether the bridge
>> is in snooping mode or not. This needs to happen however, otherwise the
>> stations will not see IGMP queries, and unsolicited membership reports
>> are not being transferred either.
> 
> I think all the testing i've done in this area i've had the bridge
> acting as the IGMP queirer. Hence it has replied to the query, rather
> than forward it out other ports.

Yes, if the bridge is itself generating the queries, this works.

> To get this far, has the bridge determined it is not the elected
> querier?  I guess it must of done. Otherwise it would not be
> forwarding it.

No, the querier is connected to one of the switch ports in a larger
topology. But the bridge must still forward such frames, otherwise IGMP
queries won't reach the senders, and membership reports won't reach the
querier.

> The problem here is:
> 
> https://elixir.bootlin.com/linux/v5.8-rc1/source/net/dsa/tag_edsa.c#L159

Ah, right!

> Setting offload_fwd_mark means the switch has forwarded the frame as
> needed to other ports of the switch. If the frame is an IGMP query
> frame, and the bridge is not the elected quierer, i guess we need to
> set this false? Or we need an FDB in the switch to forward it. What
> group address is being used?

If such a frame is ingressing, the software bridge must be able to
forward it again. So I suppose we need to set the forward flag to false
here, yes.


Thanks,
Daniel

Re: Question on DSA switches, IGMP forwarding and switchdev

[Daniel Mack]

On 6/20/20 12:36 AM, Andrew Lunn wrote:
>> I've run into the same issue. To resolve it,  In my case, in the same file, I've had to send all IGMP control traffic to the CPU:
>> 	skb->offload_fwd_mark = 1;
>> 	switch (ih->type) {
>> 		case IGMP_HOST_MEMBERSHIP_REPORT:
>> 		case IGMPV2_HOST_MEMBERSHIP_REPORT:
>> 		case IGMPV3_HOST_MEMBERSHIP_REPORT:
>> 		case IGMP_HOST_MEMBERSHIP_QUERY:
>> 		case IGMP_HOST_LEAVE_MESSAGE:
>> 			skb->offload_fwd_mark = 0;
>> 		break;
>> 	}
>>
>> I'd be interested if there is a better way.
> 
> It might depend on the switch generation, but i think some switches
> indicate the sort of packet in the DSA header. For 6390, Octet 3 of
> the header, bits 3-5 contains a code.
> 
> 0=BDPU
> 1=Frame2Reg
> 2=IGMP/MLD
> 3=Policy
> 4=ARP Mirror
> 5=Policy Mirror
> 
> We can look at these bits and not set skb->offload_fwd_mark depending
> on its value.
> 
> The 6352 family has the same bits. 6161 has a few less bits, but does
> have IGMP/MLD. I don't know about the 6085. Do you have the datasheet?

Sorry, I was mistaken in the model the board is using, it's a 6352.

So yes, we can read the code here, but I'm wondering which packet types
would then get this flag set, and which won't. Because in case of
IGMP/MLD, the packets are in fact forwarded, but the meaning of the flag
in skb is to prevent the skb from being forwarded further, which seems
wrong in all cases.

I'm thinking maybe the flag should never be set?


Thanks,
Daniel

Re: Question on DSA switches, IGMP forwarding and switchdev

[Andrew Lunn]

> So yes, we can read the code here, but I'm wondering which packet types
> would then get this flag set, and which won't. Because in case of
> IGMP/MLD, the packets are in fact forwarded, but the meaning of the flag
> in skb is to prevent the skb from being forwarded further, which seems
> wrong in all cases.
> 
> I'm thinking maybe the flag should never be set?

It is a while since i did much with multicast, so please correct me
when i'm wrong...

IGMP can use different group addresses as far as i remember.
Join/leave uses the group address of interest. But query can use
224.0.0.1 or the group address.

The bridge should learn from joins, either spontaneous or as a reply
to a query. When it sees a join, it should add a multicast FDB to the
hardware for the group, so traffic is forwarded out that port.

So for real multicast traffic, we do want the flag set, the hardware
should be doing the forwarding. If we don't set the flag, we end up
with duplication when the SW bridge also forwards the multicast
traffic.

For IGMP/MLD itself, we probably need to see what the switch does. For
IGMP using the group address, does the multicast FDB rule match and
cause the hardware to forward the IGMP? If yes, then we need the flag
set, otherwise the IGMP gets duplicated. If no, then we don't want the
flag set, and leave the SW bridge to do the forwarding, or reply
itself if it is the querier.

For IGMP using 224.0.0.1, do we ever get a multicast FDB added for
that?

It sounds like you have a better test setup than i have. Can you play
with this?

6352 uses the EDSA tagger. But the same bits exist in the DSA tag,
which the 6390 uses, due to incompatibility reasons. So it would be
nice to extend both taggers.

     Thanks
	Andrew

Re: Question on DSA switches, IGMP forwarding and switchdev

[Daniel Mack]

On 6/20/20 4:35 PM, Andrew Lunn wrote:
>> So yes, we can read the code here, but I'm wondering which packet types
>> would then get this flag set, and which won't. Because in case of
>> IGMP/MLD, the packets are in fact forwarded, but the meaning of the flag
>> in skb is to prevent the skb from being forwarded further, which seems
>> wrong in all cases.
>>
>> I'm thinking maybe the flag should never be set?
> 
> It is a while since i did much with multicast, so please correct me
> when i'm wrong...
> 
> IGMP can use different group addresses as far as i remember.
> Join/leave uses the group address of interest. But query can use
> 224.0.0.1 or the group address.
> 
> The bridge should learn from joins, either spontaneous or as a reply
> to a query. When it sees a join, it should add a multicast FDB to the
> hardware for the group, so traffic is forwarded out that port.

Yes, except it's the MDB in this case. But the bridge must also forward
the IGMP queries and reports to other ports, otherwise a cascaded
multicast router won't see the membership reports and hence won't send
frames to our switch.

> So for real multicast traffic, we do want the flag set, the hardware
> should be doing the forwarding. If we don't set the flag, we end up
> with duplication when the SW bridge also forwards the multicast
> traffic.

Yes, agreed.

> For IGMP/MLD itself, we probably need to see what the switch does. For
> IGMP using the group address, does the multicast FDB rule match and
> cause the hardware to forward the IGMP?

No, because snooping is enabled on all ports of the switch by the
driver, all IGMP frames are redirected to the CPU port and not egressed
on any other port, regardless of the entries in the ATU.

> If yes, then we need the flag
> set, otherwise the IGMP gets duplicated. If no, then we don't want the
> flag set, and leave the SW bridge to do the forwarding, 

Exactly.

> or reply
> itself if it is the querier.

If it has memberships itself.

> 6352 uses the EDSA tagger. But the same bits exist in the DSA tag,
> which the 6390 uses, due to incompatibility reasons. So it would be
> nice to extend both taggers.

But the tag format seems to be a bit different on EDSA from what you
described in your other mail. In my datasheet, the code bits are spread
across octet 2 and 3.

I'll send a patch for EDSA that works for me and solves the problem I
had, and then leave it to someone with access to the datasheets of
variants implementing the DSA tagging to replicate and test that.


Thanks for your help!
Daniel