Linux kernel hangs if using RV1108 with MSZ8863 switch with two ports connected

Linux kernel hangs if using RV1108 with MSZ8863 switch with two ports connected

[Otavio Salvador]

Hi,

I have a custom design based on Rockchip RV1108 that uses an MSZ8863
switch running kernel 4.19.

The dts part is as follows:

&gmac {
pinctrl-names = "default";
pinctrl-0 = <&rmii_pins>;
snps,reset-gpio = <&gpio1 RK_PC1 GPIO_ACTIVE_LOW>;
snps,reset-active-low;
clock_in_out = "output";
status = "okay";
};

RV1108 GMAC is connected to KSZ8863 port 3 and after kernel boots, I
can put an Ethernet cable from my router to the uplink port of
KSZ8863, which makes the RV1108 board and a Linux PC connected to the
other KSZ8863 port to both get IP addresses.

So in this usecase the setup is working fine.

However, if the RV1108 board boots with both Ethernet cables to the
KSZ8863 switch connected, then the kernel silently hangs.

Any suggestions as to what I should do in order to avoid the kernel to
hang with the two Ethernet cables connected?

The system boots fine without any Ethernet cable connected or with
only one Ethernet cable connected.
Here is the log of the system booting with no Ethernet cable connected:
http://dark-code.bulix.org/9kfff9-506410

It is only when both cables are connected that the kernel silently hangs.

Also, with the vendor 3.10 kernel such hang does not happen.

Thanks

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: Linux kernel hangs if using RV1108 with MSZ8863 switch with two ports connected

[Andrew Lunn]

On Fri, Nov 16, 2018 at 04:28:29PM -0200, Otavio Salvador wrote:
> Hi,
> 
> I have a custom design based on Rockchip RV1108 that uses an MSZ8863
> switch running kernel 4.19.
> 
> The dts part is as follows:
> 
> &gmac {
> pinctrl-names = "default";
> pinctrl-0 = <&rmii_pins>;
> snps,reset-gpio = <&gpio1 RK_PC1 GPIO_ACTIVE_LOW>;
> snps,reset-active-low;
> clock_in_out = "output";
> status = "okay";
> };
> 
> RV1108 GMAC is connected to KSZ8863 port 3 and after kernel boots, I
> can put an Ethernet cable from my router to the uplink port of
> KSZ8863, which makes the RV1108 board and a Linux PC connected to the
> other KSZ8863 port to both get IP addresses.
> 
> So in this usecase the setup is working fine.
> 
> However, if the RV1108 board boots with both Ethernet cables to the
> KSZ8863 switch connected, then the kernel silently hangs.

Hi Otavio

By silently, you mean it prints nothing at all?

I would try building the kernel with all the lock debugging turned
on. That might find something even with your working case, if there is
a potential deadlock.

If the kernel dies very early, you might need to enable "kernel
low-level debugping print and EARLY_PRINTK, in order to see anything.

	  Andrew

Re: Linux kernel hangs if using RV1108 with MSZ8863 switch with two ports connected

[Otavio Salvador]

Hi Andrew,

On Fri, Nov 16, 2018 at 9:40 PM Andrew Lunn <andrew@lunn.ch> wrote:
> On Fri, Nov 16, 2018 at 04:28:29PM -0200, Otavio Salvador wrote:
> > RV1108 GMAC is connected to KSZ8863 port 3 and after kernel boots, I
> > can put an Ethernet cable from my router to the uplink port of
> > KSZ8863, which makes the RV1108 board and a Linux PC connected to the
> > other KSZ8863 port to both get IP addresses.
> >
> > So in this usecase the setup is working fine.
> >
> > However, if the RV1108 board boots with both Ethernet cables to the
> > KSZ8863 switch connected, then the kernel silently hangs.
>
> By silently, you mean it prints nothing at all?

Yes, it prints but suddenly hangs.

> I would try building the kernel with all the lock debugging turned
> on. That might find something even with your working case, if there is
> a potential deadlock.
>
> If the kernel dies very early, you might need to enable "kernel
> low-level debugping print and EARLY_PRINTK, in order to see anything.

The kernel starts booting normally and then hangs like this when two
Ethernet cables are connected to the KSZ8863 switch:
http://dark-code.bulix.org/3xexu5-507563

This has the lock detection, inside kernel hacking, enabled.

Any hint?

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9 9981-7854          Mobile: +1 (347) 903-9750

Re: Linux kernel hangs if using RV1108 with MSZ8863 switch with two ports connected

[Andrew Lunn]

> The kernel starts booting normally and then hangs like this when two
> Ethernet cables are connected to the KSZ8863 switch:
> http://dark-code.bulix.org/3xexu5-507563
> 
> This has the lock detection, inside kernel hacking, enabled.

Maybe turn on all the hung-task debug and magic key support.  With
magic key, you might be able to get a backtrace of where it is
spinning.

Maybe also add #define DEBUG at the top of drivers/net/phy/phy.c.
Does it hang during a PHY state transition?

Maybe both PHYs are interrupting at the same time, and the interrupt
code is broken?

Maybe look at the switch driver and see if there is any code which is
executed on link up. Put some printk() in there.

If you PHYs are using interrupt mode, maybe disable that and use
polling.

Do you know if this ever worked properly before? If you know when it
did work, you can do a git bisect to narrow it down to the one patch
which broke it..

Basically, at the moment, you just need to try lots of things, to
narrow it down.

       Andrew

Re: Linux kernel hangs if using RV1108 with MSZ8863 switch with two ports connected

[Heiko Stübner]

Hi all,

Am Sonntag, 18. November 2018, 19:12:30 CET schrieb Andrew Lunn:
> > The kernel starts booting normally and then hangs like this when two
> > Ethernet cables are connected to the KSZ8863 switch:
> > http://dark-code.bulix.org/3xexu5-507563
>
> > This has the lock detection, inside kernel hacking, enabled.
> 
> Maybe turn on all the hung-task debug and magic key support.  With
> magic key, you might be able to get a backtrace of where it is
> spinning.
> 
> Maybe also add #define DEBUG at the top of drivers/net/phy/phy.c.
> Does it hang during a PHY state transition?
> 
> Maybe both PHYs are interrupting at the same time, and the interrupt
> code is broken?
> 
> Maybe look at the switch driver and see if there is any code which is
> executed on link up. Put some printk() in there.
> 
> If you PHYs are using interrupt mode, maybe disable that and use
> polling.
> 
> Do you know if this ever worked properly before? If you know when it
> did work, you can do a git bisect to narrow it down to the one patch
> which broke it..
> 
> Basically, at the moment, you just need to try lots of things, to
> narrow it down.

Your hang also seems to happen around the time the kernel disables
unused clocks and regulators.

So you might also try with these functions disabled, as it may be caused
by some clock or regulator handled wrongly there (I think it's called
clk_ignore_unused as kernel commandline but please double-check
and you'll need to check for a regulator equivalent yourself).

And as I think it might be some sort of driver-related issue, you could
also enable debugging in the driver-core [drivers/base/dd.c]
by either #define DEBUG or just redefining dev_dbg temporarily ;-)
	#define dev_dbg dev_warn
or so.

That may help finding the culprit of your hang.


Heiko