dsa: mv88e6xxx losing DHCPv6 solicit packets / IPv6 multicast packets?

dsa: mv88e6xxx losing DHCPv6 solicit packets / IPv6 multicast packets?

[Marek Behún]

Hi,

a customer of ours filed a ticket saying that when using upstream kernel
(5.8.0-rc6 on Debian 10) on Turris MOX (88e6190 switch) with DSA with
default configuration, the switch is losing DHCPv6 solicit packets /
IPv6 multicast packets sent to ff02::1::2 address.

> Specifically, it seems the 88E6190 hardware switches in the Peridot
> module is swallowing IPv6 multicast packets (sent to ff02::1:2 ).

> We tested this by mirroring the Mox LAN port on the switch and saw the
> DHCPv6 solicit packet arriving out of the switch but the Mox kernel
> didn't see it (using tcpdump).

Is this issue known?

Marek

Re: dsa: mv88e6xxx losing DHCPv6 solicit packets / IPv6 multicast packets?

[Chris Packham]

Hi Marek,

On 24/07/20 2:46 am, Marek Behún wrote:
> Hi,
>
> a customer of ours filed a ticket saying that when using upstream kernel
> (5.8.0-rc6 on Debian 10) on Turris MOX (88e6190 switch) with DSA with
> default configuration, the switch is losing DHCPv6 solicit packets /
> IPv6 multicast packets sent to ff02::1::2 address.
>
>> Specifically, it seems the 88E6190 hardware switches in the Peridot
>> module is swallowing IPv6 multicast packets (sent to ff02::1:2 ).
>> We tested this by mirroring the Mox LAN port on the switch and saw the
>> DHCPv6 solicit packet arriving out of the switch but the Mox kernel
>> didn't see it (using tcpdump).
> Is this issue known?

I can't speak to the Peridot specifically but other Marvell silicon I've 
dealt with does try to avoid trapping packets to the CPU. Normally you 
would set specific registers/table entries to declare interest in 
particular reserved multicast groups.

I had a quick skim of the Peridot docs and the references to reserved 
multicast I see are all about the 802.1D BPDUs.

It might be necessary to configure MLD snooping or add an FDB entry to 
get the ff02::1::2 packets to the CPU.

There is also the possibility that the CPUs Ethernet port is dropping 
the packets for similar reasons. I'd expect Linux to handle that 
correctly put perhaps with a DSA configuration it skips the multicast 
reception config.

As another thought do you know what DHCPv6 client/server is being used. 
There was a fairly recent bugfix for busybox that was needed because the 
v6 code was using the wrong MAC address.

Re: dsa: mv88e6xxx losing DHCPv6 solicit packets / IPv6 multicast packets?

[Florian Fainelli]

On 7/23/20 2:27 PM, Chris Packham wrote:
> Hi Marek,
> 
> On 24/07/20 2:46 am, Marek Behún wrote:
>> Hi,
>>
>> a customer of ours filed a ticket saying that when using upstream kernel
>> (5.8.0-rc6 on Debian 10) on Turris MOX (88e6190 switch) with DSA with
>> default configuration, the switch is losing DHCPv6 solicit packets /
>> IPv6 multicast packets sent to ff02::1::2 address.
>>
>>> Specifically, it seems the 88E6190 hardware switches in the Peridot
>>> module is swallowing IPv6 multicast packets (sent to ff02::1:2 ).
>>> We tested this by mirroring the Mox LAN port on the switch and saw the
>>> DHCPv6 solicit packet arriving out of the switch but the Mox kernel
>>> didn't see it (using tcpdump).
>> Is this issue known?
> 
> I can't speak to the Peridot specifically but other Marvell silicon I've 
> dealt with does try to avoid trapping packets to the CPU. Normally you 
> would set specific registers/table entries to declare interest in 
> particular reserved multicast groups.
> 
> I had a quick skim of the Peridot docs and the references to reserved 
> multicast I see are all about the 802.1D BPDUs.
> 
> It might be necessary to configure MLD snooping or add an FDB entry to 
> get the ff02::1::2 packets to the CPU.
> 
> There is also the possibility that the CPUs Ethernet port is dropping 
> the packets for similar reasons. I'd expect Linux to handle that 
> correctly put perhaps with a DSA configuration it skips the multicast 
> reception config.
> 
> As another thought do you know what DHCPv6 client/server is being used. 
> There was a fairly recent bugfix for busybox that was needed because the 
> v6 code was using the wrong MAC address.

Unlike standalone Ethernet NICs, DSA network devices do not currently
filter multicast or unicast using their ndo_set_rx_mode() at the switch
level, however they do program the DSA master network device's RX
filter, so this could be a source of bugs as well, possibly.

There is work in progress to address that by Vladimir [1], however by
default the switch is expected to flood unknown MC and UC towards the
CPU port.

[1]: https://patchwork.ozlabs.org/project/netdev/list/?series=178540
-- 
Florian

Re: dsa: mv88e6xxx losing DHCPv6 solicit packets / IPv6 multicast packets?

[Tj (Elloe Linux)]

> As another thought do you know what DHCPv6 client/server is being
> used.
> There was a fairly recent bugfix for busybox that was needed because
>the v6 code was using the wrong MAC address.

I'm the customer experiencing this issue. It appears unrelated to the
DHCP server software. On the Turris Mox with Debian 10 we have
isc-dhcp-server 4.4.1-2. Clients are Xubuntu 20.04 withNetworkManager
1.22.10-1ubuntu2.1 using isc-dclient 4.4.1-2.1ubuntu5.

Quoting from another email I sent to Turris:

We've now done more testing and CONFIRMED the Mox is losing DHCPv6
solicit packets.

Specifically, it seems the 88E6190 hardware switches in the Peridot
module is swallowing IPv6 multicast packets (sent to ff02::1:2 ).

We tested this by mirroring the Mox LAN port on an external switch and
saw the DHCPv6 solicit packet egress the switch but the Mox kernel
didn't see it ingress (using tcpdump).

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Tj (Elloe Linux)]

MV88E6085 switch not passing IPv6 multicast packets to CPU.

Seems to be related to interface not being in promiscuous mode.

This issue has been ongoing since at least July 2020. Latest v5.10-rc3
still suffers the issue on a Turris Mox with mv88e6085. We've not been
able to reproduce it on the Turris v4.14 stable kernel series so it
appears to be a regression.

Mox is using Debian 10 Buster.

First identified due to DHCPv6 leases not being renewed on clients being
served by isc-dhcp-server on the Mox.

Analysis showed the client IPv6 multicast solicit packets were being
received by the Mox hardware (proved via a mirror port on a managed LAN
switch) but the CPU was not receiving them (observed using tcpdump).

Further investigation has identified this also affects IPv6 neighbour
discovery for clients when not using frequent RAs from the Mox.

Currently we've found two reproducible scenarios:

1) with isc-dhcp-server configured with very short lease times (180
seconds). After mox reboot (or systemctl restart systemd-networkd)
clients successfully obtain a lease and a couple of RENEWs (requested
after 90 seconds) but then all goes silent, Mox OS no longer sees the
IPv6 multicast RENEW packets and client leases expire.

2) Immediately after reboot when DHCPv6 renewals are still possible if
on the Mox we do "tcdump -ni eth1 ip6" and immediately terminate,
tcpdump takes the interface out of promiscuous mode and IPv6 multicast
packets immediately cease to be received by the CPU. If we use 'tcpdump
--no-promiscuous-mode ..." so on termination it doesn't try to take the
interface out of promiscuous mode IPv6 multicast packets continue to be
seen by the CPU.

We've been pointed to the mv8e6xxx_dump tool and can capture data but
not sure what specifically to look for.

We've also added some pr_info() debugging into mvneta to analyse when
promiscuous mode is enabled or disabled since this seems to be strongly
related to the issue.

We believe there's a big clue in being able to reset the issue by
restarting systemd-networkd on the Mox. We've looked for but not found
any clues or indications of services on the Mox causing this but aren't
ruling this out.

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Andrew Lunn]

> 1) with isc-dhcp-server configured with very short lease times (180
> seconds). After mox reboot (or systemctl restart systemd-networkd)
> clients successfully obtain a lease and a couple of RENEWs (requested
> after 90 seconds) but then all goes silent, Mox OS no longer sees the
> IPv6 multicast RENEW packets and client leases expire.

So it takes about 3 minutes to reproduce this?

Can you do a git bisect to figure out which change broke it? It will
take you maybe 5 minutes per step, and given the wide range of
kernels, i'm guessing you need around 15 steps. So maybe two hours of
work.

	Andrew

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Tj (Elloe Linux)]

On 14/11/2020 15:56, Andrew Lunn wrote:
>> 1) with isc-dhcp-server configured with very short lease times (180
>> seconds). After mox reboot (or systemctl restart systemd-networkd)
>> clients successfully obtain a lease and a couple of RENEWs (requested
>> after 90 seconds) but then all goes silent, Mox OS no longer sees the
>> IPv6 multicast RENEW packets and client leases expire.
> 
> So it takes about 3 minutes to reproduce this?
> 
> Can you do a git bisect to figure out which change broke it? It will
> take you maybe 5 minutes per step, and given the wide range of
> kernels, i'm guessing you need around 15 steps. So maybe two hours of
> work.
> 
> 	Andrew
>

I'll check if we can - the problem might be the Turris Mox kernel is
based on a board support package drop by Marvell so I'm not clear right
now how divergent they are. Hopefully the Turris kernel devs can help on
that.

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Marek Behun]

On Sat, 14 Nov 2020 16:06:33 +0000
"Tj (Elloe Linux)" <ml.linux@elloe.vision> wrote:

> On 14/11/2020 15:56, Andrew Lunn wrote:
> >> 1) with isc-dhcp-server configured with very short lease times (180
> >> seconds). After mox reboot (or systemctl restart systemd-networkd)
> >> clients successfully obtain a lease and a couple of RENEWs (requested
> >> after 90 seconds) but then all goes silent, Mox OS no longer sees the
> >> IPv6 multicast RENEW packets and client leases expire.  
> > 
> > So it takes about 3 minutes to reproduce this?
> > 
> > Can you do a git bisect to figure out which change broke it? It will
> > take you maybe 5 minutes per step, and given the wide range of
> > kernels, i'm guessing you need around 15 steps. So maybe two hours of
> > work.
> > 
> > 	Andrew
> >  
> 
> I'll check if we can - the problem might be the Turris Mox kernel is
> based on a board support package drop by Marvell so I'm not clear right
> now how divergent they are. Hopefully the Turris kernel devs can help on
> that.
> 
No, TurrisOS kernel is just current OpenWRT 4.14 kernel with some
patches added onto it. Turris MOX is also supported in upstream kernel.
If this error can be reproduced on upstream kernel, you can try bisect
there...

Marek

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Vladimir Oltean]

On Sat, Nov 14, 2020 at 03:39:28PM +0000, Tj (Elloe Linux) wrote:
> MV88E6085 switch not passing IPv6 multicast packets to CPU.
>
> Seems to be related to interface not being in promiscuous mode.
>
> This issue has been ongoing since at least July 2020. Latest v5.10-rc3
> still suffers the issue on a Turris Mox with mv88e6085. We've not been
> able to reproduce it on the Turris v4.14 stable kernel series so it
> appears to be a regression.
>
> Mox is using Debian 10 Buster.
>
> First identified due to DHCPv6 leases not being renewed on clients being
> served by isc-dhcp-server on the Mox.
>
> Analysis showed the client IPv6 multicast solicit packets were being
> received by the Mox hardware (proved via a mirror port on a managed LAN
> switch) but the CPU was not receiving them (observed using tcpdump).
>
> Further investigation has identified this also affects IPv6 neighbour
> discovery for clients when not using frequent RAs from the Mox.
>
> Currently we've found two reproducible scenarios:
>
> 1) with isc-dhcp-server configured with very short lease times (180
> seconds). After mox reboot (or systemctl restart systemd-networkd)
> clients successfully obtain a lease and a couple of RENEWs (requested
> after 90 seconds) but then all goes silent, Mox OS no longer sees the
> IPv6 multicast RENEW packets and client leases expire.
>
> 2) Immediately after reboot when DHCPv6 renewals are still possible if
> on the Mox we do "tcdump -ni eth1 ip6" and immediately terminate,
> tcpdump takes the interface out of promiscuous mode and IPv6 multicast
> packets immediately cease to be received by the CPU. If we use 'tcpdump
> --no-promiscuous-mode ..." so on termination it doesn't try to take the
> interface out of promiscuous mode IPv6 multicast packets continue to be
> seen by the CPU.
>
> We've been pointed to the mv8e6xxx_dump tool and can capture data but
> not sure what specifically to look for.
>
> We've also added some pr_info() debugging into mvneta to analyse when
> promiscuous mode is enabled or disabled since this seems to be strongly
> related to the issue.
>
> We believe there's a big clue in being able to reset the issue by
> restarting systemd-networkd on the Mox. We've looked for but not found
> any clues or indications of services on the Mox causing this but aren't
> ruling this out.

Is there a simple step-by-step reproducer for the issue, that doesn't
require a lot of configuration? I've got a Mox with the 6190 switch
running net-next and Buildroot that I could try on.

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Tj (Elloe Linux)]

On 14/11/2020 18:49, Vladimir Oltean wrote:
> On Sat, Nov 14, 2020 at 03:39:28PM +0000, Tj (Elloe Linux) wrote:
>> MV88E6085 switch not passing IPv6 multicast packets to CPU.

> Is there a simple step-by-step reproducer for the issue, that doesn't
> require a lot of configuration? I've got a Mox with the 6190 switch
> running net-next and Buildroot that I could try on.

Our set-up is Mox A (CPU) + G (mini-PCIe) + F (4x USB 3.0) + 3 x E (8
port Marvell switch) + D (SFP cage)

Whilst working on this we've moved one of the E modules to another A CPU
in our lab so as not to mess with the gateway.

Running Debian 10, using systemd-networkd, which configures:

eth0 (WAN) static IPv4 and IPv6 - DHCP=no
eth1 (uplink to the switch ports) DHCP=no
lan1 (connected to external managed switch) Bridge=br-lan
br-lan static IPv4 and IPv6, Kind=bridge, IPForward=true

Whilst we're working on this issue only lan1 is connected to anything
external; a 48-port managed switch. No connection to SPF either.

We assign an IPv6 from our delegated /48 prefix to br-lan and have
isc-dhcp-server configured on a very short lease (180 seconds) to issue
leases.

On a LAN client we request a lease using:

dhclient -d -6 wlp4s0

Usually, if this is started just after the Mox systemd-networkd was
restarted, it'll manage to obtain and then renew a lease about 3 or 4 times.

These will show up in the Mox logs too.

At some point, with absolutely nothing showing in any Mox log in the
meantime, additional renewals will fail.

We later noticed that after this happens sometime later clients on the
network lose IPv6 connectivity to the Mox because neighbour discovery is
also failing - took us a while to spot this because the Mox occasionally
sends RAs at which point the clients can talk to the Mox again. The
symptom here was unexplained random-length 'hangs' of SSH sessions to
the Mox that would affect LAN clients only when the neighbour table
entry had expired.

I'm trying to create a very small reproducer root file-system on the lab
Mox.

Right now I've not been able to reproduce it on the lab unit even with a
clone of the gateway Mox's micro SD-card, but that seems due to it
failing to complete a regular boot - hence creating a fresh root
file-system.

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Andrew Lunn]

> At some point, with absolutely nothing showing in any Mox log in the
> meantime, additional renewals will fail.

What might be interesting is running

ip monitor

and

bridge monitor

Look for neighbours being timed out do to inactivity.

       Andrew

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Tj (Elloe Linux)]

[On 15/11/2020 16:02, Andrew Lunn wrote:

> What might be interesting is running
> 
> ip monitor
> 
> and
> 
> bridge monitor
> 
> Look for neighbours being timed out do to inactivity.

Funny you write that! This afternoon I've narrowed it down although I
still don't understand the 'why'.

Watching on the 'good' (lab) and 'bad' (gateway) Mox devices I noticed that:

# bridge -d -s mdb show

23: br-lan  br-lan  ff02::2  temp   257.05

23: br-lan  br-lan  ff05::2  temp   257.05

23: br-lan  br-lan  ff02::6a  temp   257.05
23: br-lan  br-lan  ff02::1:ff77:2b20  temp   257.05
23: br-lan  br-lan  ff02::1:ff00:ffff  temp   257.05

23: br-lan  br-lan  ff02::fb  temp   257.05

23: br-lan  br-lan  ff02::1:ff00:0  temp   257.05
23: br-lan  br-lan  ff02::1:2  temp   257.05
23: br-lan  br-lan  ff05::1:3  temp   257.05

indicates that the entries time out on 'bad' but are reset to a high
value on 'good'

# bridge monitor on 'bad' reported:

Deleted Deleted 23: br-lan  br-lan  ff02::2  temp
Deleted Deleted 23: br-lan  br-lan  ff05::2  temp
Deleted Deleted 23: br-lan  br-lan  ff02::6a  temp
Deleted Deleted 23: br-lan  br-lan  ff02::1:ff77:2b20  temp
Deleted Deleted 23: br-lan  br-lan  ff02::1:ff00:ffff  temp
Deleted Deleted 23: br-lan  br-lan  ff02::fb  temp
Deleted Deleted 23: br-lan  br-lan  ff02::1:ff00:0  temp
Deleted Deleted 23: br-lan  br-lan  ff02::1:2  temp
Deleted Deleted 23: br-lan  br-lan  ff05::1:3  temp

On the laptop I'm testing from (tcpdump always on the laptop):

Using tcpdump I *think* enp2s0 (wired link direct into lan1 on 'good')
always showed the laptop sending multicast listener report v2 packets on
a regular cadence of about 60-100 seconds as well as the DHCPv6
solicit/renews and that cadence matched when the timers on the output of
"bridge -d -s mdb show" reset to approximately 258.

But for wlp4s0 (wifi to 'bad') the DHCPv6 solicit/renew didn't seem to
be accompanied by multicast listener reports and the mdb timers expired.

I need to re-affirm that tomorrow because I've got slightly lost
attempting to compare multiple aspects on both 'good' and 'bad' and seem
to be seeing inconsistent results.

On the laptops we are using Xubuntu 20.04 amd64 with NetworkManager.
I'll try to test from a range of different devices tomorrow in case this
is only affecting staff laptops.

Many thanks for the pointers.

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Andrew Lunn]

> Using tcpdump I *think* enp2s0 (wired link direct into lan1 on 'good')
> always showed the laptop sending multicast listener report v2 packets on
> a regular cadence of about 60-100 seconds as well as the DHCPv6
> solicit/renews and that cadence matched when the timers on the output of
> "bridge -d -s mdb show" reset to approximately 258.
> 
> But for wlp4s0 (wifi to 'bad') the DHCPv6 solicit/renew didn't seem to
> be accompanied by multicast listener reports and the mdb timers expired.

So check if you have an IGMP querier in the network. If not, try
turning it on in the bridge,

ip link set br0 type bridge mcast_querier 1

   Andrew

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Tj (Elloe Linux)]

On 15/11/2020 17:27, Andrew Lunn wrote:

> So check if you have an IGMP querier in the network. If not, try
> turning it on in the bridge,
> 
> ip link set br0 type bridge mcast_querier 1

Thanks Andrew - that does indeed seem to have solved the issue.

I'm relieved this isn't a hardware or driver issue after all but annoyed
we didn't figure this out ourselves months ago!

Is there any other kernel 'nob' to alter this? I'm trying to understand
why we're seeing two different results with seemingly identical
kernel/OS versions and network configurations.

Re: dsa: mv88e6xxx not receiving IPv6 multicast packets

[Tj (Elloe Linux)]

On 15/11/2020 17:27, Andrew Lunn wrote:

> So check if you have an IGMP querier in the network. If not, try
> turning it on in the bridge,
> 
> ip link set br0 type bridge mcast_querier 
Thankfully it turns out this is totally unrelated to Linux - our TP-Link
Jetstream T1600G-PS has some unfortunate default behaviour and a bug.

Specifically, we are operating an IPv6-only network and Layer 2 MLD
snooping was enabled and set to forward unknown multicast groups and as
such the switch should be broadcasting all multicast packets.

However, buried in the TP-Link manual there's a note that says:

"Note: IGMP Snooping and MLD Snooping share the setting of Unknown
Multicast Groups, so you have to enable IGMP Snooping globally on the L2
FEATURES > Multicast > IGMP Snooping > Global Config page at the same
time."

We hadn't enabled IGMP snooping since we don't use IPv4!

Many thanks for the help resolving this and apologies for mis-reporting it.