Re: Question: DSA on Marvell 6390R switch dropping EAPOL packets with unicast MAC DA

Re: Question: DSA on Marvell 6390R switch dropping EAPOL packets with unicast MAC DA

[Vladimir Oltean]

Hi Harry,

On Thu, Mar 09, 2023 at 08:19:27PM +1000, Qingtao Cao wrote:
> In this success case, there are two major differences from the Setup One:
> 1. The MAC DA in the EAPOL packet of Request Identity, the 2nd packet in
> the above example, was the multicast address 0180 c200 0003.I guess this is
> understandable since the hostapd has no port-level authentication so it has
> to use a multicast address to reach the potential supplicant.
> 2. The DSA tag = "0108 efff", specifically, the mode = 0, or *TO_CPU*.

> So, how to make the EAPOL a success in the Setup One when the MAC DA is a
> unicast address instead of a multicast address?

My summary of your issue description is: when wpa_supplicant on your DSA
box receives packets with the MAC DA of the PAE group address (01:80:c2:00:00:03),
it can process them. When it receives packets with the MAC DA equal to
the port's unicast MAC address, it can't.

How have you determined that this is a problem with the kernel?

In the title, you claim "switch dropping packets", but there is no
evidence backing this claim in the email body. Furthermore, we do see
those packets in tcpdump. To drop a packet would mean that some drop
counter increases, either on the DSA switch interface
(ethtool -S net2p9 | grep -v ': 0'), or on the DSA master
(ethtool -S eth2cpu | grep -v ': 0'). Note that on the DSA master, DSA
also appends the port counters of the switch's CPU port.

I am also curious to see drop counters when tcpdump is active, and when
it isn't active. This is because tcpdump puts the interface in
promiscuous mode (accept packets with any MAC DA), so there may be a
problem there as well. The command "ip link set eth2cpu promisc on" may
also help to diagnose whether it is a filtering problem on the DSA master.

I have not used wpa_supplicant, but I guess it might be possible that
the application is simply written to bind() the socket() only to the PAE
group address (01:80:c2:00:00:03). Otherwise said, the kernel doesn't
drop the unicast packets, but the application is written such that it
doesn't process them. This is just speculation. Apart from inspecting
the hostapd source code (which I haven't done), "strace" could also be
used to determine this.

If no ethtool counters increment, there is no reason to suspect a
hardware drop of your packets. In that case, the kernel network stack
could still drop them somewhere. This set of commands run on the DSA box
might help identify where:

CONFIG_NET_DROP_MONITOR=y
sudo apt install libpcap-dev binutils-dev libtool
git clone https://github.com/nhorman/dropwatch.git
cd dropwatch
./autogen.sh
dropwatch -l kas

I have no reason to suspect at the moment that any change to net/dsa/tag_dsa.c
would be needed. Until proven otherwise, the difference between FORWARD
and TO_CPU DSA headers is a red herring IMO.

Hope this helps.

Re: Question: DSA on Marvell 6390R switch dropping EAPOL packets with unicast MAC DA

[Vladimir Oltean]

On Thu, Mar 09, 2023 at 03:16:01PM +0200, Vladimir Oltean wrote:
> Speaking of bridging, is the net2p9 interface under any bridge during
> your testing, or does it operate as a standalone interface? If it's
> under a bridge, could you repeat the experiment with it as standalone?

Ah, I shouldn't have asked this, because the information you've provided:

> > if (trunk)
> >     skb->offload_fwd_mark = true;
> > else if (!trap)
> >     dsa_default_offload_fwd_mark(skb);
> > 
> > will set skb->offload_fwd_mark = 1.

is enough for me to determine that yes, net2p9 *is* under a bridge.
This is because of the implementation:

static inline void dsa_default_offload_fwd_mark(struct sk_buff *skb)
{
	struct dsa_port *dp = dsa_slave_to_port(skb->dev);

	skb->offload_fwd_mark = !!(dp->bridge);
}

if it wasn't under a bridge, then skb->offload_fwd_mark would have never
been 1.

Ok, so having this said, now I have a pretty strong suspicion as to what
the issue is.

The RX handler of the software bridge layer - br_handle_frame() - steals
packets from the bridge port interface (net2p9) before those packets
ever reach the PF_PACKET socket opened by wpa_supplicant. "Steals" means
"returns RX_HANDLER_CONSUMED and not RX_HANDLER_PASS".

There is a netfilter hook through which you can tell the bridge
"hey, don't steal this kind of traffic, leave it on the bridge port!"
The command, applied to EAPOL, would look something like this:

/sbin/ebtables --table broute --append BROUTING --protocol 0x888e --jump DROP

Would you mind giving this a try (and do the rest of the debugging later)?

Re: Question: DSA on Marvell 6390R switch dropping EAPOL packets with unicast MAC DA

[Vladimir Oltean]

Hi Harry

On Fri, Mar 10, 2023 at 08:34:12AM +1000, Qingtao Cao wrote:
> Hi Vladimir,
> 
> Yes, yes, yes!

Please resolve your problems with top-posting style and plain text
emails pointed out earlier. This reply did not make it to the mailing
list either, AFAICS.

> 
> The net2p9 along with all other ports are under a bridge (eth2) on my box,
> which is setup by a script using ip and brctl command after reboot.
> 
> My box previously supports a kernel configuration without any DSA driver
> enabled, so all those marvell 6390r switches work as a "dumb" one: their
> backend is the intel i210 and switch ports just act as its PHYs, so they
> are used as a normal home network switch.
> 
> Whereas the new configuration enables kernel DSA and driver, the physical
> interface eth2 was renamed to eth2cpu and a bridge interface named eth2 was

it's slightly unconventional to name a bridge interface "eth2", but okay...

> created to enslave all switch ports. From brctl showstp command, I could
> see the net2p9 had entered all the way to the forwarding state when I
> kicked off the wpa_supplicant on it.

the modern command for that would be "bridge link", but okay

> 
> Of course, I would try the ebtables command first things first in the
> morning today!
> 
> If it works, then I would have more questions, such as:
> 1. is the bridge layer RX handler involved just because net2p9 is enslaved
> to a bridge? (although wpa_supplicant was not operating on the bridge
> interface)

yes

> 2. if yes, then running wpa_supplicant against the net2p9 before creation
> of the bridge, would this prevent the bridge RX handler from stealing the
> packet?

yes

> 3. if still yes, then perhaps I could run the wpa_supplicant before setting
> up the bridge, so that the d-link switch port could still be authorized and
> all the rest DSA switch ports could still share the same uplink via net2p9

that's a question for you to answer

doesn't the port need reauthorization from time to time? if it does, do
you plan to temporarily remove it from the bridge and add it back?

> 4. if yes again, would the bridge RX handler steal other protocols than
> EAPOL?

the source code of br_handle_frame() that it steals *all* packets except
for a select few, namely packets sent to the reserved link-local multicast
range (01-80-c2-00-00-xx).

> how would we come up with a more generalized solution to prevent
> this from happening?

You'd need to talk to the bridge maintainers about this. Currently,
stealing is avoided for frames for which the frame_handler() of their
protocol added via br_add_frame() (currently MRP and CFM) returns an
error. I suppose this mechanism could be extended/abused to add new
protocols which the bridge should always know it should never steal:
their frame_handler() identifies them and returns an error.

However the list is bound to get quite large (PTP over IP needs this too)
https://lore.kernel.org/netdev/871r3gbdxv.fsf@kurt/
and not all bridge users need all rules, but if the rules get built-in,
then all users need to suffer the performance penalty associated with
the frame_type lookups. So I guess this is why we have the netfilter
based system. I would expect some pushback from bridge maintainers, but
I guess it's worth a shot.

> 
> BTW, running wpa_supplicant on the bridge interface doesn't work, just
> because of EAPOL Start packet with that multicast address as MAC DA will be
> dropped by the bridge since STP has been enabled on it.

I didn't suggest that you run wpa_supplicant on the bridge interface.
I wasn't expecting it to work.