Re: mvpp2 switch from gmac to xlg requires ifdown/ifup

["Marek Behún" <kabel@kernel.org>]

On Fri, 23 Jul 2021 09:05:38 +0100
"Russell King (Oracle)" <linux@armlinux.org.uk> wrote:

> On Fri, Jul 23, 2021 at 03:52:02AM +0200, Marek Behún wrote:
> > Hello Russell (and possibly others),
> > 
> > I discovered that with mvpp2 when switching from gmac (sgmii or
> > 2500base-x mode) to xlg (10gbase-r mode) due to phylink requesting this
> > change, the link won't come up unless I do
> >   ifconfig ethX down
> >   ifconfig ethX up
> > 
> > Can be reproduced on MacchiatoBIN:
> > 1. connect the two 10g RJ-45 ports (88X3310 PHY) with one cable
> > 2. bring the interfaces up
> > 3. the PHYs should link in 10gbase-t, links on MACs will go up in
> >    10gbase-r
> > 4. use ethtool on one of the interfaces to advertise modes only up to
> >    2500base-t
> > 5. the PHYs will unlink and then link at 2.5gbase-t, links on MACs will
> >    go up in 2500base-x
> > 6. use ethtool on the same interface as in step 4 to advertise all
> >    supported modes
> > 
> > 7. the PHYs will unlink and then link at 10gbase-t, BUT MACs won't link
> >    !!!
> > 8. execute
> >      ifconfig ethX down ; ifconfig ethX up
> >    on both interfaces. After this, the MACs will successfully link in
> >    10gbase-r with the PHYs
> > 
> > It seems that the mvpp2 driver code needs to make additional stuff when
> > chaning mode from gmac to xlg. I haven't yet been able to find out
> > what, though.
> > 
> > BTW I discovered this because I am working on adding support for
> > 5gbase-r mode to mvpp2, so that the PHY can support 5gbase-t on copper
> > side.
> > The ifdown/ifup cycle is required when switching from gmac to xlg, i.e.:
> > 	sgmii		to	5gbase-r
> > 	sgmii		to	10gbase-r
> > 	2500base-x	to	5gbase-r
> > 	2500base-x	to	10gbase-r
> > but also when switching from xlg to different xlg:
> > 	5gbase-r	to	10gbase-r
> > 	10gbase-r	to	5gbase-r
> > 
> > Did someone notice this bug? I see that Russell made some changes in
> > the phylink pcs API that touched mvpp2 (the .mac_config method got
> > split into .mac_prepare, .mac_config and .mac_finish, and also some
> > other changes). I haven't tried yet if the switch from gmac to xlg
> > worked at some time in the past. But if it did, maybe these changes
> > could be the cause?  
> 
> What are the PHY leds doing when you encounter this bug?
> 

Table summary:

			PHY0/eth0	PHY1/eth1
			green	yellow	green	yellow
after boot		ON	OFF	ON	OFF
eth0 up			ON	OFF	ON	OFF
eth1 up			blink	ON	blink	ON
eth0 adv -10g -5g	blink	OFF	blink	OFF
eth0 adv +5g *	 	OFF	OFF	OFF	OFF
eth0 down		ON	OFF	ON	OFF
eth0 up			blink**	OFF	blink** OFF
eth1 down		ON	OFF	ON	OFF
eth1 up			blink	OFF	blink	OFF

 (*  PHYs are linked now, but MACs are not)
 (** blinks only for a while after link, pings do not work,
     read my opinion below)
 (The last 5 lines basically the same happens if I set it to advertise
  10g instead of 5g, but in case of 10g the yellow LED is ON when the
  PHYs are linked.)

In words:

After boot, the green LED is ON on both PHYs.

Bringing both interfaces up changes nothing.

Plugging cable so that they link (at 10gbase-t) bring the yellow LEDs
ON, and the green LEDs OFF, but the green LEDs blinks on activity.
(For example when pinging eth0's ipv6 address via eth1.)

Disabling advertisement of 10gbase-t and 5gbase-t on eth0 makes the
PHYs link at 2.5gbase-t. Both LEDs are OFF, but green blinks on
activity.

Enabling advertisement of 5gbase-t makes the PHYs link, but the MACs
do not link with the PHYs, and there is no blinking on activity, and
pings do not work.

Taking one interface (eth0) down and up makes the PHYs link (we are
still at 5gbase-t), and the green LEDs blink for a few times because of
activity on both PHYs. But the pings do not work. I think this is
because the eth0's PHY sent some neighbour discovery packets, and the
eth1's PHY received them. But pings do not work because those packets
don't go from eth1's PHY to eth1's MAC.

Taking the second interface (eth1) down and up makes the PHYs again
link at 5gbase-t, and the green LEDs start blinking on activity. Pings
work.

Marek