iMX6Q FEC: transmit queue 0 timed out

iMX6Q FEC: transmit queue 0 timed out

[Holger Schurig]

Hi guys,

on my IMX6 board with RMII and a LAN8720 PYH I don't get ethernet
running under Linux. It is, however, working from barebox.

I tried
* 3.15-rc8 "bare"
* 3.15-rc8 with three FEC related patches from linux-next ("net: fec:
correct the MDIO clock source" and two PM related patches)
* 3.15-rc8 with Russell King's cubie-patchbomb, I applied only the
net-fec-patches 71...125, which a substitution for
109-net-fec-increase-transmit-ring-size.patch, because that one didn't
apply automatically)

But all seemed to never been able to transmit: a tshark running on a
different computer didn't notice any packets from the iMX6Q board, not
even ARP. However, the barebox boot loader *IS* able to get data over
the ethernet, so the hardware must be ok ...

Any idea on how I could debug this further?



barebox:/ global linux.bootargs.root="root=/dev/ram rdinit=/sbin/init"
barebox:/ ifup eth0
barebox:/ mkdir -p /mnt
barebox:/ mount -t tftp $eth0.serverip /mnt
barebox:/ bootm -v -r /mnt/image/boot/initrd.img.gz
/mnt/linux/arch/arm/boot/zImage
T
Loading ARM Linux zImage '/mnt/linux/arch/arm/boot/zImage'
OS image not yet relocated
Loading initrd GZIP compressed '/mnt/image/boot/initrd.img.gz'
initrd image not yet relocated
Passing control to ARM zImage handler
no OS load address, defaulting to 0x10708000
no initrd load address, defaulting to 0x109ca000
commandline: console=ttymxc0,115200n8 quiet root=/dev/ram rdinit=/sbin/init

Starting kernel at 0x10708000, initrd at 0x109ca000, oftree at 0x10a99000...

/ # ifconfig eth0
eth0      Link encap:Ethernet  HWaddr A2:EB:72:86:51:1F
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
/ # dmesg -c >/dev/null
/ # ifconfig eth0 192.168.233.198
/ # dmesg -c
fec 2188000.ethernet eth0: Freescale FEC PHY driver [SMSC
LAN8710/LAN8720] (mii_bus:phy_addr=2188000.ethernet:00, irq=-1)
/ # ping -c 1 192.168.233.20
PING 192.168.233.20 (192.168.233.20): 56 data bytes


--------------> At this point, nothing happens. So I press Ctrl-C.


--- 192.168.233.20 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
/ # dmesg
random: nonblocking pool is initialized
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:264 dev_watchdog+0x1c0/0x248()
NETDEV WATCHDOG: eth0 (fec): transmit queue 0 timed out
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.15.0-rc8 #2
[<c0012e0c>] (unwind_backtrace) from [<c00102d0>] (show_stack+0x10/0x14)
[<c00102d0>] (show_stack) from [<c0356c28>] (dump_stack+0x78/0x94)
[<c0356c28>] (dump_stack) from [<c001c7b4>] (warn_slowpath_common+0x60/0x84)
[<c001c7b4>] (warn_slowpath_common) from [<c001c804>]
(warn_slowpath_fmt+0x2c/0x3c)
[<c001c804>] (warn_slowpath_fmt) from [<c02e1b40>] (dev_watchdog+0x1c0/0x248)
[<c02e1b40>] (dev_watchdog) from [<c0024e70>] (call_timer_fn.isra.35+0x20/0x80)
[<c0024e70>] (call_timer_fn.isra.35) from [<c0025700>]
(run_timer_softirq+0x1d0/0x214)
[<c0025700>] (run_timer_softirq) from [<c001fe34>] (__do_softirq+0xec/0x220)
[<c001fe34>] (__do_softirq) from [<c00201a0>] (irq_exit+0x84/0xa8)
[<c00201a0>] (irq_exit) from [<c000e068>] (handle_IRQ+0x6c/0x90)
[<c000e068>] (handle_IRQ) from [<c00084e0>] (gic_handle_irq+0x3c/0x60)
[<c00084e0>] (gic_handle_irq) from [<c0010d40>] (__irq_svc+0x40/0x50)
Exception stack(0xc04c3f70 to 0xc04c3fb8)
3f60:                                     eefbb550 00000000 00014b1e 00000000
3f80: c04c2000 c04c2000 c04b9af0 ef7fcac0 1000406a 412fc09a 00000000 00000000
3fa0: 00000000 c04c3fb8 c000e1ac c000e1b0 60000013 ffffffff
[<c0010d40>] (__irq_svc) from [<c000e1b0>] (arch_cpu_idle+0x24/0x2c)
[<c000e1b0>] (arch_cpu_idle) from [<c004b2b8>] (cpu_startup_entry+0x98/0x110)
[<c004b2b8>] (cpu_startup_entry) from [<c04999ec>] (start_kernel+0x2e8/0x344)
---[ end trace a7d71cbbbcf0c938 ]---
fec 2188000.ethernet eth0: TX ring dump
Nr     SC     addr       len  SKB
  0    0x9c00 0x3e2b8000   42 eea21b40 40000000
  1 S  0x0000 0x00000000    0   (null) 40000000
  2    0x0000 0x00000000    0   (null) 40000000
  3    0x0000 0x00000000    0   (null) 40000000
126    0x0000 0x00000000    0   (null) 40000000
127  H 0x2000 0x00000000    0   (null) 40000000
fec 2188000.ethernet eth0: TX ring dump
Nr     SC     addr       len  SKB
  0    0x9c00 0x3e2b8000   42 eea21840 40000000
  1 S  0x0000 0x00000000    0   (null) 40000000
...
126    0x0000 0x00000000    0   (null) 40000000
127  H 0x2000 0x00000000    0   (null) 40000000
fec 2188000.ethernet eth0: TX ring dump
Nr     SC     addr       len  SKB
  0    0x9c00 0x3e2b8000   42 eea219c0 40000000
  1 S  0x0000 0x00000000    0   (null) 40000000
...
126    0x0000 0x00000000    0   (null) 40000000
127  H 0x2000 0x00000000    0   (null) 40000000
/ # grep ether /proc/interrupts
150:        156          0          0          0       GIC 150  2188000.ethernet
151:          0          0          0          0       GIC 151  2188000.ethernet

iMX6Q FEC: transmit queue 0 timed out

[Russell King - ARM Linux]

On Mon, Jun 02, 2014 at 02:27:41PM +0200, Holger Schurig wrote:
> But all seemed to never been able to transmit: a tshark running on a
> different computer didn't notice any packets from the iMX6Q board, not
> even ARP. However, the barebox boot loader *IS* able to get data over
> the ethernet, so the hardware must be ok ...

Are the pinmux settings correct?

Specifically, look at the transmit clocking arrangement.  From your
logs, it appears that the FEC has not even tried to process the packet
in the TX ring.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

iMX6Q FEC: transmit queue 0 timed out

[Holger Schurig]

Transmit clocking and pinmux?  Do you mean ENET_TX_EN?  Yep, that's
set. Somehow barebox can send, it's just Linux that doesn't want to do
it. Note that ENET_TX_CLK is not muxed, as it is for MII, not for
RMII.

In case I confuse what you meant, here are excerpts from my DTS:

&fec {
   pinctrl-names = "default";
   pinctrl-0 = <&pinctrl_fec>;
   phy-mode = "rmii";
   phy-reset-gpios = <&gpio2 4 GPIO_ACTIVE_LOW>;
   status = "okay";
};

...

pinctrl_fec: fecgrp {
   fsl,pins = <
      MX6QDL_PAD_ENET_TXD0__ENET_TX_DATA0 0x1b030
      MX6QDL_PAD_ENET_TXD1__ENET_TX_DATA1 0x1b030
      MX6QDL_PAD_ENET_TX_EN__ENET_TX_EN   0x1b030
      MX6QDL_PAD_ENET_RXD0__ENET_RX_DATA0 0x1b030
      MX6QDL_PAD_ENET_RXD1__ENET_RX_DATA1 0x1b030
      MX6QDL_PAD_ENET_RX_ER__ENET_RX_ER   0x1b030
      MX6QDL_PAD_ENET_MDIO__ENET_MDIO     0x0B850
      MX6QDL_PAD_ENET_MDC__ENET_MDC       0x1b030
      MX6QDL_PAD_ENET_CRS_DV__ENET_RX_EN  0x1b030
      MX6QDL_PAD_GPIO_16__ENET_REF_CLK    0x1b030
      >;
};


Or did you mean the clocks internal to the CPU?  So far I didn't
really care for them, because I thought that the clocking framework in
conjunction with the device tree takes care of that. Hmm, in the
refernce manual, chapter 23.3 they talk about mac0_txmem_clk, which
isn't in that list. If I read the manual correct, this is enabled by
enet_clk_enable, so let's look into CCGR1:

/ # devmem 0x20C406c
0xF0FCFC00
                                      ...
Hmm, the first 'C' from the right contains bits 10-11 (CCM_CCGR1.CG5)
and they are both set, so enet_clk_enable is also on.

iMX6Q FEC: transmit queue 0 timed out

[Russell King - ARM Linux]

On Mon, Jun 02, 2014 at 03:15:21PM +0200, Holger Schurig wrote:
> Transmit clocking and pinmux?  Do you mean ENET_TX_EN?  Yep, that's
> set. Somehow barebox can send, it's just Linux that doesn't want to do
> it. Note that ENET_TX_CLK is not muxed, as it is for MII, not for
> RMII.
> 
> In case I confuse what you meant, here are excerpts from my DTS:

It's probably ENET_REF_CLK that's the cause of the problem.

> pinctrl_fec: fecgrp {
>    fsl,pins = <
>       MX6QDL_PAD_ENET_TXD0__ENET_TX_DATA0 0x1b030
>       MX6QDL_PAD_ENET_TXD1__ENET_TX_DATA1 0x1b030
>       MX6QDL_PAD_ENET_TX_EN__ENET_TX_EN   0x1b030
>       MX6QDL_PAD_ENET_RXD0__ENET_RX_DATA0 0x1b030
>       MX6QDL_PAD_ENET_RXD1__ENET_RX_DATA1 0x1b030
>       MX6QDL_PAD_ENET_RX_ER__ENET_RX_ER   0x1b030
>       MX6QDL_PAD_ENET_MDIO__ENET_MDIO     0x0B850
>       MX6QDL_PAD_ENET_MDC__ENET_MDC       0x1b030
>       MX6QDL_PAD_ENET_CRS_DV__ENET_RX_EN  0x1b030
>       MX6QDL_PAD_GPIO_16__ENET_REF_CLK    0x1b030
>       >;
> };

On the Microsom, I need:

      MX6QDL_PAD_GPIO_16__ENET_REF_CLK        0xc0000000

Does that work for you?

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

iMX6Q FEC: transmit queue 0 timed out

[Holger Schurig]

> On the Microsom, I need:
>
>       MX6QDL_PAD_GPIO_16__ENET_REF_CLK        0xc0000000
>
> Does that work for you?

Still works in Barebox, but in Linux:

/ # ifconfig eth0 192.168.233.198
fec 2188000.ethernet eth0: MDIO write timeout
fec 2188000.ethernet eth0: MDIO read timeout
fec 2188000.ethernet eth0: MDIO write timeout
fec 2188000.ethernet eth0: could not attach to PHY
ifconfig: SIOCSIFFLAGS: Connection timed out
/ #

I see that some in-Linux-tree DTS have this value, others have
0x4001b0a8 (one in-barebox-tree DTS has a simple 0x1b0b0). Both SION
settings create the same error at ifup time for me. And Barebox
doesn't care for any of them, continue to do TFTP all the time.

iMX6Q FEC: transmit queue 0 timed out

[fugang.duan at freescale.com]

From: Holger Schurig <holgerschurig@gmail.com> Data: Monday, June 02, 2014 10:11 PM
> To: Russell King - ARM Linux
> Cc: linux-arm-kernel at lists.infradead.org
> Subject: Re: iMX6Q FEC: transmit queue 0 timed out
> 
> > On the Microsom, I need:
> >
> >       MX6QDL_PAD_GPIO_16__ENET_REF_CLK        0xc0000000
> >
> > Does that work for you?
> 
> Still works in Barebox, but in Linux:
> 
> / # ifconfig eth0 192.168.233.198
> fec 2188000.ethernet eth0: MDIO write timeout fec 2188000.ethernet eth0:
> MDIO read timeout fec 2188000.ethernet eth0: MDIO write timeout fec
> 2188000.ethernet eth0: could not attach to PHY
> ifconfig: SIOCSIFFLAGS: Connection timed out / #
> 
> I see that some in-Linux-tree DTS have this value, others have
> 0x4001b0a8 (one in-barebox-tree DTS has a simple 0x1b0b0). Both SION
> settings create the same error at ifup time for me. And Barebox doesn't
> care for any of them, continue to do TFTP all the time.

You can use the setting "0x4001b0a8" that SION bit set. And please make sure GPR1[21] is set ?
For imx6q/dl silicon, enet RMII mode, ANATOP/CCM supply 50Mhz clock for enet reference clock and phy like:
ANATOP enet PLL --> GPIO_16 PAD  --> output 50Mhz to phy
				|
				|
			 Loopback to enet RMII reference clock

1. GPIO_16 SION bit must be set.
2. GPR1[21] must be set.

Thanks,
Andy

iMX6Q FEC: transmit queue 0 timed out

[Holger Schurig]

Thanks for you help.

Okay, first I enabled SION, I used MX6QDL_PAD_GPIO_16__ENET_REF_CLK
0x4001b0a8. Barebox still worked, Linux not.

As soon as I set GPR1[21], Barebox can no longer boot via TFTP:

barebox:/ initrd
Kernel + Initrd via TFTP
eth0: error frame: 0x4c226028 0x00000804
T eth0: error frame: 0x4c226048 0x00000804
T eth0: error frame: 0x4c226050 0x00000810
T eth0: error frame: 0x4c226098 0x00000804
T error: First block is not block 1 (2)
could not open /mnt/image/boot/initrd.img.gz: Invalid argument
handler failed with: Invalid argument
barebox:/


Also, it looks like Linux itself sets this bit, don't know where.
Because after barebox couldn't boot via TFTP anymore, I changed
barebox back to not set it:

barebox:/ md 0x20e0004+4
020e0004: ff0d001b                                           ....
barebox:/ initrd
Kernel + Initrd via TFTP

Loading ARM Linux zImage '/mnt/linux/arch/arm/boot/zImage'
OS image not yet relocated
Loading initrd GZIP compressed '/mnt/image/boot/initrd.img.gz'
initrd image not yet relocated
Passing control to ARM zImage handler
no OS load address, defaulting to 0x10708000
no initrd load address, defaulting to 0x109ca000
commandline: console=ttymxc0,115200n8 quiet root=/dev/ram rdinit=/sbin/init

Starting kernel at 0x10708000, initrd at 0x109ca000, oftree at 0x10a9b000...

/ # devmem 0x20e0004
0xFF2D301B


Here, Barebox shows it's not set, and Linux (busybox' devmem command)
show that it's set. I did the obvious thing and cleared that bit in
Linux (via devmem 0x20e0004 32 0xFF0D301B), but that didn't make
"ifconfig eth0 192.168.233.198" work, in both cases the same error
messages:

fec 2188000.ethernet eth0: MDIO write timeout
fec 2188000.ethernet eth0: MDIO read timeout
fec 2188000.ethernet eth0: MDIO write timeout
fec 2188000.ethernet eth0: could not attach to PHY
ifconfig: SIOCSIFFLAGS: Connection timed out


I wonder if I need to turn on this anatop clock?  How?

iMX6Q FEC: transmit queue 0 timed out

[Holger Schurig]

Hi Andy,

thank you very much, I guess that this will lead me into the right
direction. It actually made me understand what SION does, the RM
wasn't too enlighting to me.

The thing seems to be that the FEC can produce the 50 MHz reference
clock ... or it can get the reference clock from outside. The same is
true for the LAN8720AI phy: can can produce the refclock (when it has
a 25 MHz quartz, which is the case in my board), or it can consume it
from outside. While in barebox (I'm not using u-boot), the PHY makes
the 50 MHz refclk by itself, because GPR1[21] is not set and it works.

The PHY can be controlled if he creates or consume the REF CLK with
the signal on it's pin nINTSEL during the low->high transition of the
reset. I guess that whoever toggles this (i must dig into it, but the
reset pin is described in the device tree) doesn't care (or can't
care) how nINTSEL is set and so in Linux mode the modes of the two
devices don't harmonize.

Next week I've again time for this issue, I'll report back.

iMX6Q FEC: transmit queue 0 timed out

[Russell King - ARM Linux]

On Wed, Jun 04, 2014 at 01:50:54PM +0200, Holger Schurig wrote:
> The PHY can be controlled if he creates or consume the REF CLK with
> the signal on it's pin nINTSEL during the low->high transition of the
> reset. I guess that whoever toggles this (i must dig into it, but the
> reset pin is described in the device tree) doesn't care (or can't
> care) how nINTSEL is set and so in Linux mode the modes of the two
> devices don't harmonize.

It sounds like you need to add appropriate pull-ups/pull-downs on the
iMX6 configure the phy.  This is why imx6qdl-microsom-ar8035.dtsi has
different settings from the normal RGMII pullups/downs - in the case
of AR8035, a number of pins including the receive pins determine reset
options.  See the comments in that file concerning "pin strapping".

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

iMX6Q FEC: transmit queue 0 timed out

[Holger Schurig]

Andy,

to revive an old thread: I now got ethernet working. Basically, I
needed to set the system so that the PHY is generating the ref-clock.
I found even code in arch/arm/mach-imx/mach-imx6q.c that controls
setting of GPR1[21] from the device tree. However, I wasn't really
able to come up with a device-tree that triggers the code
IMX6Q_GPR1_ENET_CLK_SEL_PAD. For example, Neither
Documentation/devicetree/bindings/net/fsl-fec.txt nor any other file
there mention anything on how to do this.

Currently I disable the 3rd clock. I changed the default
         clocks = <&clks 117>, <&clks 117>, <&clks 190>;
         clock-names = "ipg", "ahb", "ptp";
(from imx6qdl.dtsi) to my dts:
         clocks = <&clks 117>, <&clks 117>;
         clock-names = "ipg", "ahb";

Would I have needed to create some dummy clock, e.g. "ext-phy-clk" and
specified that as a third clock in my dts?


Russel,

the 8720A PHY is configured for int/ext clock via it's nINTSEL pin.
And our hardware designers put that pin to a LED, so I cannot
influence it from the IOMUXC function block of the CPU. Thanks for the
idea anyway, maybe someone else finds this in google :-)

iMX6Q FEC: transmit queue 0 timed out

[Shawn Guo]

On Wed, Jul 16, 2014 at 10:41:10AM +0200, Holger Schurig wrote:
> Andy,
> 
> to revive an old thread: I now got ethernet working. Basically, I
> needed to set the system so that the PHY is generating the ref-clock.
> I found even code in arch/arm/mach-imx/mach-imx6q.c that controls
> setting of GPR1[21] from the device tree. However, I wasn't really
> able to come up with a device-tree that triggers the code
> IMX6Q_GPR1_ENET_CLK_SEL_PAD. For example, Neither
> Documentation/devicetree/bindings/net/fsl-fec.txt nor any other file
> there mention anything on how to do this.
> 
> Currently I disable the 3rd clock. I changed the default
>          clocks = <&clks 117>, <&clks 117>, <&clks 190>;
>          clock-names = "ipg", "ahb", "ptp";
> (from imx6qdl.dtsi) to my dts:
>          clocks = <&clks 117>, <&clks 117>;
>          clock-names = "ipg", "ahb";
> 
> Would I have needed to create some dummy clock, e.g. "ext-phy-clk" and
> specified that as a third clock in my dts?

Since in your setup PHY is generating the reference clock, you need to
define this clock in your board dts, and overwrite the 'ptp' clock with
it, something like the following.

clocks {
	#address-cells = <1>;
	#size-cells = <0>;

	rmii_clk: clock at 0 {
		compatible = "fixed-clock";
		reg = <0>;
		#clock-cells = <0>;
		clock-frequency = <25000000>;  /* 25MHz for example */
		clock-output-names = "rmii_ref_clk";
	};
};

&fec {
        clocks = <&clks 117>, <&clks 117>, <&rmii_clk>;
};