IMX6 FEC connection drops occasionally with 'MDIO read timeout'

IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Tim Harvey]

Greetings,

I've had a couple of users report that they are getting occasional
link drops when using IMX6 FEC on our boards which are using a Marvell
88E1510 PHY:

[ 9348.474418] fec 2188000.ethernet eth0: MDIO read timeout
[ 9350.478804] fec 2188000.ethernet eth0: Link is Down

If they bring the interface back up it works fine.

The most recent version of Linux they have tested and seen this on is
v4.9 with CONFIG_MARVELL_PHY=y

Has anyone else run into this or know what the issue could be? Should
there be any retries at the MDIO level or above?

Regards,

Tim

Re: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Andrew Lunn]

On Fri, Apr 07, 2017 at 07:47:17AM -0700, Tim Harvey wrote:
> Greetings,
> 
> I've had a couple of users report that they are getting occasional
> link drops when using IMX6 FEC on our boards which are using a Marvell
> 88E1510 PHY:
> 
> [ 9348.474418] fec 2188000.ethernet eth0: MDIO read timeout
> [ 9350.478804] fec 2188000.ethernet eth0: Link is Down
> 
> If they bring the interface back up it works fine.
> 
> The most recent version of Linux they have tested and seen this on is
> v4.9 with CONFIG_MARVELL_PHY=y
> 
> Has anyone else run into this or know what the issue could be? Should
> there be any retries at the MDIO level or above?

Hi Tim

MDIO is assumed to be reliable. It failing is bad.

However, ETIMEOUT is interesting. The MDIO protocol has no flow
control. There is no way for the PHY to delay replying.

I would suggest you look at the power management code. What i have
seen with the FEC, when using switches attached to the FEC, is that if
the clocks are turned off, you get ETIMEOUT from MDIO.

    Andrew

Re: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Tim Harvey]

On Fri, Apr 7, 2017 at 9:09 AM, Andrew Lunn <andrew@lunn.ch> wrote:
> On Fri, Apr 07, 2017 at 07:47:17AM -0700, Tim Harvey wrote:
>> Greetings,
>>
>> I've had a couple of users report that they are getting occasional
>> link drops when using IMX6 FEC on our boards which are using a Marvell
>> 88E1510 PHY:
>>
>> [ 9348.474418] fec 2188000.ethernet eth0: MDIO read timeout
>> [ 9350.478804] fec 2188000.ethernet eth0: Link is Down
>>
>> If they bring the interface back up it works fine.
>>
>> The most recent version of Linux they have tested and seen this on is
>> v4.9 with CONFIG_MARVELL_PHY=y
>>
>> Has anyone else run into this or know what the issue could be? Should
>> there be any retries at the MDIO level or above?
>
> Hi Tim
>
> MDIO is assumed to be reliable. It failing is bad.
>
> However, ETIMEOUT is interesting. The MDIO protocol has no flow
> control. There is no way for the PHY to delay replying.
>
> I would suggest you look at the power management code. What i have
> seen with the FEC, when using switches attached to the FEC, is that if
> the clocks are turned off, you get ETIMEOUT from MDIO.
>
>     Andrew

Andrew,

Thanks for the reply. Your talking about suspend/resume power
management right? The users reporting this were not using
suspend/resume.

With regards to clock are you talking about the IPG clock? Is there
any other way that would get turned off other than fec suspend/resume?

Tim

Re: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Fabio Estevam]

Hi Tim,

On Wed, Apr 12, 2017 at 1:15 PM, Tim Harvey <tharvey@gateworks.com> wrote:
>
> Andrew,
>
> Thanks for the reply. Your talking about suspend/resume power
> management right? The users reporting this were not using
> suspend/resume.
>
> With regards to clock are you talking about the IPG clock? Is there
> any other way that would get turned off other than fec suspend/resume?

Yes, through pm_runtime.

Can you check if this quick debug change help?

--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -3606,8 +3606,6 @@ static int __maybe_unused
fec_runtime_suspend(struct device *dev)
        struct net_device *ndev = dev_get_drvdata(dev);
        struct fec_enet_private *fep = netdev_priv(ndev);

-       clk_disable_unprepare(fep->clk_ipg);
-
        return 0;
 }


If you don't see the problem with it, then it means we need to fix the
pm runtime support in this driver.

Most likely pm runtime is turning off the clocks when it should not.

Re: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Tim Harvey]

On Wed, Apr 12, 2017 at 9:26 AM, Fabio Estevam <festevam@gmail.com> wrote:
> Hi Tim,
>
> On Wed, Apr 12, 2017 at 1:15 PM, Tim Harvey <tharvey@gateworks.com> wrote:
>>
>> Andrew,
>>
>> Thanks for the reply. Your talking about suspend/resume power
>> management right? The users reporting this were not using
>> suspend/resume.
>>
>> With regards to clock are you talking about the IPG clock? Is there
>> any other way that would get turned off other than fec suspend/resume?
>
> Yes, through pm_runtime.
>
> Can you check if this quick debug change help?
>
> --- a/drivers/net/ethernet/freescale/fec_main.c
> +++ b/drivers/net/ethernet/freescale/fec_main.c
> @@ -3606,8 +3606,6 @@ static int __maybe_unused
> fec_runtime_suspend(struct device *dev)
>         struct net_device *ndev = dev_get_drvdata(dev);
>         struct fec_enet_private *fep = netdev_priv(ndev);
>
> -       clk_disable_unprepare(fep->clk_ipg);
> -
>         return 0;
>  }
>
>
> If you don't see the problem with it, then it means we need to fix the
> pm runtime support in this driver.
>
> Most likely pm runtime is turning off the clocks when it should not.

Fabio,

Ok, I understand now. We will disable the IPG clock disable and see if
that makes a difference.

Tim

Re: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Andy Duan]

On 2017年04月13日 00:54, Tim Harvey wrote:
> On Wed, Apr 12, 2017 at 9:26 AM, Fabio Estevam <festevam@gmail.com> wrote:
>> Hi Tim,
>>
>> On Wed, Apr 12, 2017 at 1:15 PM, Tim Harvey <tharvey@gateworks.com> wrote:
>>> Andrew,
>>>
>>> Thanks for the reply. Your talking about suspend/resume power
>>> management right? The users reporting this were not using
>>> suspend/resume.
>>>
>>> With regards to clock are you talking about the IPG clock? Is there
>>> any other way that would get turned off other than fec suspend/resume?
>> Yes, through pm_runtime.
>>
>> Can you check if this quick debug change help?
>>
>> --- a/drivers/net/ethernet/freescale/fec_main.c
>> +++ b/drivers/net/ethernet/freescale/fec_main.c
>> @@ -3606,8 +3606,6 @@ static int __maybe_unused
>> fec_runtime_suspend(struct device *dev)
>>          struct net_device *ndev = dev_get_drvdata(dev);
>>          struct fec_enet_private *fep = netdev_priv(ndev);
>>
>> -       clk_disable_unprepare(fep->clk_ipg);
>> -
>>          return 0;
>>   }
>>
>>
>> If you don't see the problem with it, then it means we need to fix the
>> pm runtime support in this driver.
>>
>> Most likely pm runtime is turning off the clocks when it should not.
> Fabio,
>
> Ok, I understand now. We will disable the IPG clock disable and see if
> that makes a difference.
>
> Tim
Firstly, pls try the change suggested by Andrew.
I guess system enter to wait mode, and enet irq cannot wakeup system in 
real time that causes mii irq much latency. then mii bus access timeout.
If so, can you add below change to your dts file and try it ?

iomux pinctrl:
                 pinctrl_enet_irq: enetirqgrp {
                         fsl,pins = <
MX6QDL_PAD_GPIO_6__ENET_IRQ             0x000b1
                         >;
                 };


&fec {
         ...
         pinctrl-0 = <&pinctrl_enet &pinctrl_enet_irq>;
         interrupts-extended = <&gpio1 6 0x04>, <&gpc 0 119 0x04>;
         ...
};

&i2c3 {
         ...
         status = "disabled";
};


Regards,
Andy

Re: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Andrew Lunn]

On Wed, Apr 12, 2017 at 09:15:11AM -0700, Tim Harvey wrote:
> On Fri, Apr 7, 2017 at 9:09 AM, Andrew Lunn <andrew@lunn.ch> wrote:
> > On Fri, Apr 07, 2017 at 07:47:17AM -0700, Tim Harvey wrote:
> >> Greetings,
> >>
> >> I've had a couple of users report that they are getting occasional
> >> link drops when using IMX6 FEC on our boards which are using a Marvell
> >> 88E1510 PHY:
> >>
> >> [ 9348.474418] fec 2188000.ethernet eth0: MDIO read timeout
> >> [ 9350.478804] fec 2188000.ethernet eth0: Link is Down
> >>
> >> If they bring the interface back up it works fine.
> >>
> >> The most recent version of Linux they have tested and seen this on is
> >> v4.9 with CONFIG_MARVELL_PHY=y
> >>
> >> Has anyone else run into this or know what the issue could be? Should
> >> there be any retries at the MDIO level or above?
> >
> > Hi Tim
> >
> > MDIO is assumed to be reliable. It failing is bad.
> >
> > However, ETIMEOUT is interesting. The MDIO protocol has no flow
> > control. There is no way for the PHY to delay replying.
> >
> > I would suggest you look at the power management code. What i have
> > seen with the FEC, when using switches attached to the FEC, is that if
> > the clocks are turned off, you get ETIMEOUT from MDIO.
> >
> >     Andrew
> 
> Andrew,
> 
> Thanks for the reply. Your talking about suspend/resume power
> management right? The users reporting this were not using
> suspend/resume.

Hi Tim

As Fabio just pointed out, i was meaning run time power management.

It used to be, that the FEC driver powered up the clocks when the
interface was opened. But when using a switch connected to the MDIO
bus, you can be using the MDIO bus while the interface is still
closed. And that did not work. So the runtime PM was extended to the
MDIO bus.

It could be there is a race condition in this code somewhere.

   Andrew

RE: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Andy Duan]

From: Tim Harvey <tharvey@gateworks.com> Sent: Friday, April 07, 2017 10:47 PM
>To: netdev <netdev@vger.kernel.org>
>Cc: Fabio Estevam <fabio.estevam@nxp.com>; Lucas Stach
><l.stach@pengutronix.de>; Andy Duan <fugang.duan@nxp.com>; Koen
>Vandeputte <koen.vandeputte@ncentric.com>
>Subject: IMX6 FEC connection drops occasionally with 'MDIO read timeout'
>
>Greetings,
>
>I've had a couple of users report that they are getting occasional link drops
>when using IMX6 FEC on our boards which are using a Marvell
>88E1510 PHY:
>
>[ 9348.474418] fec 2188000.ethernet eth0: MDIO read timeout [ 9350.478804]
>fec 2188000.ethernet eth0: Link is Down
>
>If they bring the interface back up it works fine.
>
>The most recent version of Linux they have tested and seen this on is
>v4.9 with CONFIG_MARVELL_PHY=y
>
>Has anyone else run into this or know what the issue could be? Should there
>be any retries at the MDIO level or above?
>
>Regards,
>
>Tim

Pls keep ipg clock always on and try it.
In general, MDIO timeout maybe caused by ipg clock off and mii interrupt lost. 


Regards,
Andy

Re: IMX6 FEC connection drops occasionally with 'MDIO read timeout'

[Tim Harvey]

On Sun, Apr 9, 2017 at 6:42 PM, Andy Duan <fugang.duan@nxp.com> wrote:
> From: Tim Harvey <tharvey@gateworks.com> Sent: Friday, April 07, 2017 10:47 PM
>>To: netdev <netdev@vger.kernel.org>
>>Cc: Fabio Estevam <fabio.estevam@nxp.com>; Lucas Stach
>><l.stach@pengutronix.de>; Andy Duan <fugang.duan@nxp.com>; Koen
>>Vandeputte <koen.vandeputte@ncentric.com>
>>Subject: IMX6 FEC connection drops occasionally with 'MDIO read timeout'
>>
>>Greetings,
>>
>>I've had a couple of users report that they are getting occasional link drops
>>when using IMX6 FEC on our boards which are using a Marvell
>>88E1510 PHY:
>>
>>[ 9348.474418] fec 2188000.ethernet eth0: MDIO read timeout [ 9350.478804]
>>fec 2188000.ethernet eth0: Link is Down
>>
>>If they bring the interface back up it works fine.
>>
>>The most recent version of Linux they have tested and seen this on is
>>v4.9 with CONFIG_MARVELL_PHY=y
>>
>>Has anyone else run into this or know what the issue could be? Should there
>>be any retries at the MDIO level or above?
>>
>>Regards,
>>
>>Tim
>
> Pls keep ipg clock always on and try it.
> In general, MDIO timeout maybe caused by ipg clock off and mii interrupt lost.
>

Andy,

Sounds like what Andrew was talking about. In this case suspend/resume
is not being used. Is there any other way IPG can get turned off that
I should be looking at?

Tim