net: phy: micrel KSZ9031 ifdown ifup issue

net: phy: micrel KSZ9031 ifdown ifup issue

[Paul Thomas]

Hello,

I think I'm seeing an issue with the PHY hardware or PHY driver. What
happens is sometimes (but not always) when I do 'ip link set eth0
down' followed by 'ip link set eth0 up' I don't ever see an
auto-negotiation again. LEDs don't come on, ethtool reports 'Link
detected: no'. Even physically unplugging and plugging the network
cable doesn't bring it back. I have to do a reboot to get the
networking back.

When the networking is started I don't see any issue forcing
negotiations by unplugging and plugging the cable. I get standard
messages like this all day long:
[   21.031793] 003: macb ff0b0000.ethernet eth0: link down
[   26.142835] 003: macb ff0b0000.ethernet eth0: link up (1000/Full)

One thing that makes me think this is the PHY is that we have another
Ethernet port using the DP83867 PHY and I can always do ifdown/ifup
with it.

This is using a 5.2.10 kernel on arm64 zynqmp platform with the macb driver.

Is this something anyone else has seen? I know there is some Errata
with this part, but I'm hoping there is something to fix or work
around this. Any thoughts on where to look or add debugging would
appreciated.

thanks,
Paul

Re: net: phy: micrel KSZ9031 ifdown ifup issue

[Andrew Lunn]

On Fri, Sep 13, 2019 at 10:42:38AM -0400, Paul Thomas wrote:
> Hello,
> 
> I think I'm seeing an issue with the PHY hardware or PHY driver. What
> happens is sometimes (but not always) when I do 'ip link set eth0
> down' followed by 'ip link set eth0 up' I don't ever see an
> auto-negotiation again. LEDs don't come on, ethtool reports 'Link
> detected: no'. Even physically unplugging and plugging the network
> cable doesn't bring it back. I have to do a reboot to get the
> networking back.
> 
> When the networking is started I don't see any issue forcing
> negotiations by unplugging and plugging the cable. I get standard
> messages like this all day long:
> [   21.031793] 003: macb ff0b0000.ethernet eth0: link down
> [   26.142835] 003: macb ff0b0000.ethernet eth0: link up (1000/Full)
> 
> One thing that makes me think this is the PHY is that we have another
> Ethernet port using the DP83867 PHY and I can always do ifdown/ifup
> with it.
> 
> This is using a 5.2.10 kernel on arm64 zynqmp platform with the macb driver.
> 
> Is this something anyone else has seen? I know there is some Errata
> with this part, but I'm hoping there is something to fix or work
> around this. Any thoughts on where to look or add debugging would
> appreciated.

Hi Paul

Are you using interrupts, or polling? If interrupts, try polling?
Seems unlikely, but you could be missing an interrupt.

There is a fix from Antoine Tenart which suggests asym pause can be an
issue? What pause setup are you using? But this is a known issue,
which 5.2 should have the fix for.

    Andrew

Re: net: phy: micrel KSZ9031 ifdown ifup issue

[Paul Thomas]

Hi Andrew,

I did some more investigation, and what seems to be happening is the
device get's stuck in auto-negotiation. I looked at this using
phytool:
https://github.com/wkz/phytool

When it is in the good state I see that reg 0x01 is 0x796d where bit
1.2 reports 'Link is up' and bit 1.5 reports 'Auto-negotiation process
complete'. However, once I get to the bad state (it may take several
tries of ifdown, ifup to get there) then reg 0x01 is 0x7649 reporting
'Link is down' and 'Auto-negotiation process not completed'. This can
be fixed by resetting the phy './phytool write eth0/3/0 0x9140'

So, I guess that means the driver is doing what it is supposed to?
Could we add quirk or something to reset the phy again from the driver
if auto-negotiation doesn't complete with x seconds?

> Are you using interrupts, or polling? If interrupts, try polling?
> Seems unlikely, but you could be missing an interrupt.
It must be polling, the interrupt from the PHY is run in the
schematic, but it is not used in the hw or device-tree configuration.

>
> There is a fix from Antoine Tenart which suggests asym pause can be an
> issue? What pause setup are you using? But this is a known issue,
> which 5.2 should have the fix for.
Yes, this kernel includes this asym pause workaround. Reg 4.11:10 is 0
# ./phytool read eth0/3/4
0x01e1

This is the last little Ethernet issue that we are having, it would be
nice if we could find a solution.

thanks,
Paul

Re: net: phy: micrel KSZ9031 ifdown ifup issue

[Andrew Lunn]

> When it is in the good state I see that reg 0x01 is 0x796d where bit
> 1.2 reports 'Link is up' and bit 1.5 reports 'Auto-negotiation process
> complete'. However, once I get to the bad state (it may take several
> tries of ifdown, ifup to get there) then reg 0x01 is 0x7649 reporting
> 'Link is down' and 'Auto-negotiation process not completed'. This can
> be fixed by resetting the phy './phytool write eth0/3/0 0x9140'
> 
> So, I guess that means the driver is doing what it is supposed to?
> Could we add quirk or something to reset the phy again from the driver
> if auto-negotiation doesn't complete with x seconds?

Hi Paul

Adding a timeout would make sense. But please try to hide all this
inside the PHY driver. Since it is being polled, the read_status()
should be called once per second, so you should be able to handle all
this inside that driver callback.

     Andrew

Re: net: phy: micrel KSZ9031 ifdown ifup issue

[Paul Thomas]

On Mon, Sep 16, 2019 at 11:13 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > When it is in the good state I see that reg 0x01 is 0x796d where bit
> > 1.2 reports 'Link is up' and bit 1.5 reports 'Auto-negotiation process
> > complete'. However, once I get to the bad state (it may take several
> > tries of ifdown, ifup to get there) then reg 0x01 is 0x7649 reporting
> > 'Link is down' and 'Auto-negotiation process not completed'. This can
> > be fixed by resetting the phy './phytool write eth0/3/0 0x9140'
> >
> > So, I guess that means the driver is doing what it is supposed to?
> > Could we add quirk or something to reset the phy again from the driver
> > if auto-negotiation doesn't complete with x seconds?
>
> Hi Paul
>
> Adding a timeout would make sense. But please try to hide all this
> inside the PHY driver. Since it is being polled, the read_status()
> should be called once per second, so you should be able to handle all
> this inside that driver callback.
Thanks Andrew,

It looks like there are more issues than just the auto-negotiation.
Even when the negotiated link comes back I never see any more rx
packets from the macb driver. I'll look into this more.

thanks,
Paul