issues to bring up two VSC8531 PHYs

issues to bring up two VSC8531 PHYs

[Ron Eggler]

Hi,
I am trying to bring up a pair of VSC9531 PHYs on an embedded system. I'm using a Yocto build and have altered the device tree with the following patch:
https://github.com/MistySOM/meta-mistysom/blob/phy-enable/recipes-kernel/linux/smarc-rzg2l/0001-add-vsc8531-userspace-dts.patch
I installed mdio-tools and can see the interfaces like:
# mdio
11c20000.ethernet-ffffffff
11c30000.ethernet-ffffffff

Also, I hooked up a logic analyzer to the mdio lines and can see communications happening at boot time. Also, it appears that it's able to read the link status correctly (when a cable is plugged):
# mdio 11c20000.ethernet-ffffffff
  DEV      PHY-ID  LINK
0x00  0x00070572  up

Yet, ifconfig doesn't show the interfaces and I get:
# ifconfig eth0 up
[  140.542939] ravb 11c20000.ethernet eth0: failed to connect PHY
SIOCSIFFLAGS: No such file or directory
When I try to bring it up

# ip l displays the interfaces as:
4: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
     link/ether 9a:ab:83:16:65:36 brd ff:ff:ff:ff:ff:ff
5: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
     link/ether 2a:9d:bf:09:8d:c3 brd ff:ff:ff:ff:ff:ff

Where am I going from here? I have experimented with drilling down into bitwise analysis of the MDIO communications. I'm uncertain though if this is my best bet, does someone here have any insight and can provide me with some guidance?

Thanks a lot!
-- 
Ron

Re: issues to bring up two VSC8531 PHYs

[Andrew Lunn]

On Wed, Apr 12, 2023 at 01:11:45PM -0700, Ron Eggler wrote:
> Hi,
> I am trying to bring up a pair of VSC9531 PHYs on an embedded system. I'm using a Yocto build and have altered the device tree with the following patch:
> https://github.com/MistySOM/meta-mistysom/blob/phy-enable/recipes-kernel/linux/smarc-rzg2l/0001-add-vsc8531-userspace-dts.patch

+        phy0: ethernet-phy@7 {
+                compatible = "ethernet-phy-ieee802.3-c45";
+                reg = <0>;

There is a minor DT issue here. I assume the PHY is on address 0 of
the bus? Then ethernet-phy@7 should be ethernet-phy@0. This is pretty
much cosmetic, but when you come to submit the board for inclusion in
mainline, you need to have this correct.

> I installed mdio-tools and can see the interfaces like:
> # mdio
> 11c20000.ethernet-ffffffff
> 11c30000.ethernet-ffffffff
> 
> Also, I hooked up a logic analyzer to the mdio lines and can see communications happening at boot time. Also, it appears that it's able to read the link status correctly (when a cable is plugged):
> # mdio 11c20000.ethernet-ffffffff
>  DEV      PHY-ID  LINK
> 0x00  0x00070572  up

So that matches with:

/linux/drivers/net/phy$ grep -r 000705 *
mscc/mscc.h:#define PHY_ID_VSC8531			  0x00070570

Take a look in sys/bus/mdio_bus/devices/ Any sign of the two PHYs?
 
> Yet, ifconfig doesn't show the interfaces and I get:
> # ifconfig eth0 up
> [  140.542939] ravb 11c20000.ethernet eth0: failed to connect PHY

So this is where you need to start debugging. Why cannot it find the
PHY?

Your DT patch does not show a phy-handle. Is there one inherited from
a .dtsi files? Is phy-mode also set?

    Andrew

Re: issues to bring up two VSC8531 PHYs

[Heiner Kallweit]

On 12.04.2023 22:11, Ron Eggler wrote:
> Hi,
> I am trying to bring up a pair of VSC9531 PHYs on an embedded system. I'm using a Yocto build and have altered the device tree with the following patch:
> https://github.com/MistySOM/meta-mistysom/blob/phy-enable/recipes-kernel/linux/smarc-rzg2l/0001-add-vsc8531-userspace-dts.patch
> I installed mdio-tools and can see the interfaces like:
> # mdio
> 11c20000.ethernet-ffffffff
> 11c30000.ethernet-ffffffff
> 
> Also, I hooked up a logic analyzer to the mdio lines and can see communications happening at boot time. Also, it appears that it's able to read the link status correctly (when a cable is plugged):
> # mdio 11c20000.ethernet-ffffffff
>  DEV      PHY-ID  LINK
> 0x00  0x00070572  up
> 
AFAICS there's no PHY driver yet for this model. The generic driver may or may not work.
Best add a PHY driver.

> Yet, ifconfig doesn't show the interfaces and I get:
> # ifconfig eth0 up
> [  140.542939] ravb 11c20000.ethernet eth0: failed to connect PHY
> SIOCSIFFLAGS: No such file or directory
> When I try to bring it up
> 
> # ip l displays the interfaces as:
> 4: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
>     link/ether 9a:ab:83:16:65:36 brd ff:ff:ff:ff:ff:ff
> 5: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
>     link/ether 2a:9d:bf:09:8d:c3 brd ff:ff:ff:ff:ff:ff
> 
> Where am I going from here? I have experimented with drilling down into bitwise analysis of the MDIO communications. I'm uncertain though if this is my best bet, does someone here have any insight and can provide me with some guidance?
> 
Any specific reason why you set the compatible to ethernet-phy-ieee802.3-c45 for a c22 PHY?

> Thanks a lot!

It would be advisable to include the phylib maintainers when asking phylib-related questions.

Re: issues to bring up two VSC8531 PHYs

[Andrew Lunn]

> > Also, I hooked up a logic analyzer to the mdio lines and can see communications happening at boot time. Also, it appears that it's able to read the link status correctly (when a cable is plugged):
> > # mdio 11c20000.ethernet-ffffffff
> >  DEV      PHY-ID  LINK
> > 0x00  0x00070572  up
> > 
> AFAICS there's no PHY driver yet for this model. The generic driver may or may not work.
> Best add a PHY driver.

Hi Heiner

mscc.h:#define PHY_ID_VSC8531			  0x00070570

mscc_main.c:
        .phy_id         = PHY_ID_VSC8531,
        .name           = "Microsemi VSC8531",
        .phy_id_mask    = 0xfffffff0,
        /* PHY_GBIT_FEATURES */
 
> Any specific reason why you set the compatible to
> ethernet-phy-ieee802.3-c45 for a c22 PHY?

Ah, i missed that! The driver only uses phy_read/phy_write, not
phy_write_mmd() and phy_read_mmd().

Remove the compatible string. It is not needed for C22 PHYs.

       Andrew

Re: issues to bring up two VSC8531 PHYs

[Heiner Kallweit]

On 13.04.2023 00:20, Andrew Lunn wrote:
>>> Also, I hooked up a logic analyzer to the mdio lines and can see communications happening at boot time. Also, it appears that it's able to read the link status correctly (when a cable is plugged):
>>> # mdio 11c20000.ethernet-ffffffff
>>>  DEV      PHY-ID  LINK
>>> 0x00  0x00070572  up
>>>
>> AFAICS there's no PHY driver yet for this model. The generic driver may or may not work.
>> Best add a PHY driver.
> 
> Hi Heiner
> 
> mscc.h:#define PHY_ID_VSC8531			  0x00070570
> 
> mscc_main.c:

OK, missed that. I just looked at the vitesse driver which also covers
a number of VSCxxxx PHY's.

>         .phy_id         = PHY_ID_VSC8531,
>         .name           = "Microsemi VSC8531",
>         .phy_id_mask    = 0xfffffff0,
>         /* PHY_GBIT_FEATURES */
>  
>> Any specific reason why you set the compatible to
>> ethernet-phy-ieee802.3-c45 for a c22 PHY?
> 
> Ah, i missed that! The driver only uses phy_read/phy_write, not
> phy_write_mmd() and phy_read_mmd().
> 
> Remove the compatible string. It is not needed for C22 PHYs.
> 
>        Andrew

Re: issues to bring up two VSC8531 PHYs

[Ron Eggler]

On 2023-04-12 3:37 p.m., Heiner Kallweit wrote:
> On 13.04.2023 00:20, Andrew Lunn wrote:
>>>> Also, I hooked up a logic analyzer to the mdio lines and can see communications happening at boot time. Also, it appears that it's able to read the link status correctly (when a cable is plugged):
>>>> # mdio 11c20000.ethernet-ffffffff
>>>>   DEV      PHY-ID  LINK
>>>> 0x00  0x00070572  up
>>>>
>>> AFAICS there's no PHY driver yet for this model. The generic driver may or may not work.
>>> Best add a PHY driver.
>> Hi Heiner
>>
>> mscc.h:#define PHY_ID_VSC8531			  0x00070570
>>
>> mscc_main.c:
> OK, missed that. I just looked at the vitesse driver which also covers
> a number of VSCxxxx PHY's.
>
>>          .phy_id         = PHY_ID_VSC8531,
>>          .name           = "Microsemi VSC8531",
>>          .phy_id_mask    = 0xfffffff0,
>>          /* PHY_GBIT_FEATURES */
>>   
>>> Any specific reason why you set the compatible to
>>> ethernet-phy-ieee802.3-c45 for a c22 PHY?
Since the VSC8531 has a clause 45 register space, I assumed that it is a 
clause 45 PHY.
>> Ah, i missed that! The driver only uses phy_read/phy_write, not
>> phy_write_mmd() and phy_read_mmd().
>>
>> Remove the compatible string. It is not needed for C22 PHYs.

Anyways, I changed the patch specify "ethernet-phy-ieee802.3-c22" 
instead, it seems c22 is just a fallback if it's not specified per 
phy.txt - Documentation/devicetree/bindings/net/phy.txt - Linux source 
code (v4.14) - Bootlin 
<https://elixir.bootlin.com/linux/v4.14/source/Documentation/devicetree/bindings/net/phy.txt>


Okay, I tried it out, booted up and see my network interfaces:

# ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
         ether f6:1f:f8:73:ce:f4  txqueuelen 1000  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
         device interrupt 170

eth1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
         ether 7a:18:b6:23:bf:f6  txqueuelen 1000  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
         device interrupt 173

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536  metric 1
         inet 127.0.0.1  netmask 255.0.0.0
         loop  txqueuelen 1000  (Local Loopback)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I'm pumped!

This is a huge step forward, Thank you Heiner & Andrew!

I've set IPs manually (haven't dealt with DHCP yet) but am not able to 
transfer any data yet.
(tried pinging hosts in local network). Also IP doesn't appear to be 
visible on network. Pinging localhost or own IP works fine.

I connected it with a patch cable to a laptop and fired up tcpdump on 
the laptop.
I can see ARP requests going out (from the laptop) but VSC8531s are not 
responding (tried both ports).
What else can I do from here, is it time to probe the RGMII signals on 
the board?


Thanks!

-- 

Ron

Re: issues to bring up two VSC8531 PHYs

[Andrew Lunn]

> Anyways, I changed the patch specify "ethernet-phy-ieee802.3-c22" instead,
> it seems c22 is just a fallback if it's not specified per phy.txt -
> Documentation/devicetree/bindings/net/phy.txt - Linux source code (v4.14) -
> Bootlin <https://elixir.bootlin.com/linux/v4.14/source/Documentation/devicetree/bindings/net/phy.txt>

I would not trust 4.14 Documentation. That has been dead for a long
long time. We generally request you report networking issues against
the net-next tree, or the last -rcX kernel.

> I connected it with a patch cable to a laptop and fired up tcpdump on the
> laptop.
> I can see ARP requests going out (from the laptop) but VSC8531s are not
> responding (tried both ports).
> What else can I do from here, is it time to probe the RGMII signals on the
> board?

What phy-mode are you using. Generally rgmii-id is what you want, so
that the PHY adds the RGMII delays.

You can also try:

ethtool --phy-statistics ethX

The statistics are mostly about errors, not correctly received frames,
so there might not be anything useful.

     Andrew

Re: issues to bring up two VSC8531 PHYs

[Ron Eggler]

Andrew, Russel & list,

Please aplogize for the delay in follow up!

On 2023-04-13 1:27 p.m., Andrew Lunn wrote:
>> Anyways, I changed the patch specify "ethernet-phy-ieee802.3-c22" instead,
>> it seems c22 is just a fallback if it's not specified per phy.txt -
>> Documentation/devicetree/bindings/net/phy.txt - Linux source code (v4.14) -
>> Bootlin<https://elixir.bootlin.com/linux/v4.14/source/Documentation/devicetree/bindings/net/phy.txt>
> I would not trust 4.14 Documentation. That has been dead for a long
> long time. We generally request you report networking issues against
> the net-next tree, or the last -rcX kernel.

Yeah, got it and I should have probably looked at the docs for the 
kernel version we're using which is 5.10.83-cip1 which leads me to:

https://elixir.bootlin.com/linux/v5.10.83/source/Documentation/devicetree/bindings/net/ethernet-phy.yaml

it does list "ethernet-phy-ieee802.3-c22" as an option.

I'm still a bit puzzled though because the VSC8531 datasheet clearly 
indicates that it's got a clause 45 register space: 
https://ww1.microchip.com/downloads/en/DeviceDoc/VMDS-10514.pdf page 54.

>> I connected it with a patch cable to a laptop and fired up tcpdump on the
>> laptop.
>> I can see ARP requests going out (from the laptop) but VSC8531s are not
>> responding (tried both ports).
>> What else can I do from here, is it time to probe the RGMII signals on the
>> board?
> What phy-mode are you using. Generally rgmii-id is what you want, so
> that the PHY adds the RGMII delays.

Okay great, I've added the following patch to my build:

diff --git a/r9a07g044.dtsi.orig b/r9a07g044.dtsi
index 16ff035..727815c 100644
--- a/arch/arm64/boot/dts/renesas/r9a07g044.dtsi
+++ b/arch/arm64/boot/dts/renesas/r9a07g044.dtsi
@@ -951,7 +951,7 @@
                                     <GIC_SPI 85 IRQ_TYPE_LEVEL_HIGH>,
                                     <GIC_SPI 86 IRQ_TYPE_LEVEL_HIGH>;
                        interrupt-names = "mux", "fil", "arp_ns";
-                       phy-mode = "rgmii";
+                       phy-mode = "rgmii-id";
                        clocks = <&cpg CPG_MOD R9A07G044_ETH0_CLK_AXI>,
                                 <&cpg CPG_MOD R9A07G044_ETH0_CLK_CHI>,
                                 <&cpg CPG_CORE R9A07G044_CLK_HP>;
@@ -971,7 +971,7 @@
                                     <GIC_SPI 88 IRQ_TYPE_LEVEL_HIGH>,
                                     <GIC_SPI 89 IRQ_TYPE_LEVEL_HIGH>;
                        interrupt-names = "mux", "fil", "arp_ns";
-                       phy-mode = "rgmii";
+                       phy-mode = "rgmii-id";
                        clocks = <&cpg CPG_MOD R9A07G044_ETH1_CLK_AXI>,
                                 <&cpg CPG_MOD R9A07G044_ETH1_CLK_CHI>,
                                 <&cpg CPG_CORE R9A07G044_CLK_HP>;

> You can also try:
>
> ethtool --phy-statistics ethX

after appliaction of the above patch, ethtool tells me

# ethtool --phy-statistics eth0
PHY statistics:
      phy_receive_errors: 65535
     phy_idle_errors: 255

and this along with the frame counters in ifconfig looks like there are 
no frames going back and forth between the MPU and the PHY:


# ifconfig
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
         inet 192.168.1.123  netmask 255.255.255.0  broadcast 192.168.1.255
         ether e4:19:2f:15:1f:c1  txqueuelen 1000  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 20  collisions 0
         device interrupt 170

eth1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
         inet 192.168.1.234  netmask 255.255.0.0  broadcast 192.168.255.255
         ether f6:c8:2f:5e:01:7c  txqueuelen 1000  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 79  collisions 0
         device interrupt 173

/proc/interrupts reports a few interrupts on eth0 & eth1:

# cat /proc/interrupts
            CPU0       CPU1
  11:     117135      97562     GICv3  27 Level     arch_timer
  13:          0          0     GICv3 202 Level     10001200.timer
  20:          0          0     GICv3 209 Level     10001200.timer
  56:          0          0     GICv3 358 Level     10049c00.ssi
  60:          0          0     GICv3 362 Level     1004a000.ssi
  65:          0          0     GICv3 448 Level     1004b000.spi:rx
  66:          0          0     GICv3 449 Level     1004b000.spi:tx
  67:          0          0     GICv3 412 Level 1004b800.serial:rx err
  68:          26          0     GICv3 414 Level 1004b800.serial:rx full
  69:      3478          0     GICv3 415 Level 1004b800.serial:tx empty
  70:          0          0     GICv3 413 Level 1004b800.serial:break
  71:        230          0     GICv3 416 Level 1004b800.serial:rx ready
  73:          0          0     GICv3 458 Level     canfd.g_err
  74:          0          0     GICv3 459 Level     canfd.g_recc
  75:          0          0     GICv3 454 Level     canfd.ch0_err
  77:          0          0     GICv3 460 Level     canfd.ch0_trx
  78:          0          0     GICv3 455 Level     canfd.ch1_err
  80:          0          0     GICv3 461 Level     canfd.ch1_trx
  87:          0          0     GICv3 382 Level     riic-tend
  88:          0          0     GICv3 380 Edge      riic-rdrf
  89:          3          0     GICv3 381 Edge      riic-tdre
  90:          1          0     GICv3 384 Level     riic-stop
  92:          1          0     GICv3 383 Level     riic-nack
  95:          0          0     GICv3 390 Level     riic-tend
  96:          0          0     GICv3 388 Edge      riic-rdrf
  97:          2          0     GICv3 389 Edge      riic-tdre
  98:          1          0     GICv3 392 Level     riic-stop
100:          1          0     GICv3 391 Level     riic-nack
103:         17          0     GICv3 406 Level     riic-tend
104:          0          0     GICv3 404 Edge      riic-rdrf
105:         51          0     GICv3 405 Edge      riic-tdre
106:         17          0     GICv3 408 Level     riic-stop
108:          0          0     GICv3 407 Level     riic-nack
111:          0          0     GICv3 379 Edge      10059000.adc
113:          5          0     GICv3 476 Level     tint0
114:          1          0     GICv3 477 Level     tint1
115:          0          0     GICv3 478 Level     tint2
116:          0          0     GICv3 479 Level     tint3
117:          0          0     GICv3 480 Level     tint4
118:          0          0     GICv3 481 Level     tint5
119:          0          0     GICv3 482 Level     tint6
120:          0          0     GICv3 483 Level     tint7
121:          0          0     GICv3 484 Level     tint8
122:          0          0     GICv3 485 Level     tint9
123:          0          0     GICv3 486 Level     tint10
124:          0          0     GICv3 487 Level     tint11
125:          0          0     GICv3 488 Level     tint12
126:          0          0     GICv3 489 Level     tint13
127:          0          0     GICv3 490 Level     tint14
128:          0          0     GICv3 491 Level     tint15
129:          0          0     GICv3 492 Level     tint16
130:          0          0     GICv3 493 Level     tint17
131:          0          0     GICv3 494 Level     tint18
132:          0          0     GICv3 495 Level     tint19
133:          0          0     GICv3 496 Level     tint20
134:          0          0     GICv3 497 Level     tint21
135:          0          0     GICv3 498 Level     tint22
136:          0          0     GICv3 499 Level     tint23
137:          0          0     GICv3 500 Level     tint24
138:          0          0     GICv3 501 Level     tint25
139:          0          0     GICv3 502 Level     tint26
140:          0          0     GICv3 503 Level     tint27
141:          0          0     GICv3 504 Level     tint28
142:          0          0     GICv3 505 Level     tint29
143:          0          0     GICv3 506 Level     tint30
144:          0          0     GICv3 507 Level     tint31
145:          0          0     GICv3 173 Edge      error
146:          0          0     GICv3 157 Edge 11820000.dma-controller:0
147:          0          0     GICv3 158 Edge 11820000.dma-controller:1
148:          0          0     GICv3 159 Edge 11820000.dma-controller:2
149:          0          0     GICv3 160 Edge 11820000.dma-controller:3
150:          0          0     GICv3 161 Edge 11820000.dma-controller:4
151:          0          0     GICv3 162 Edge 11820000.dma-controller:5
152:          0          0     GICv3 163 Edge 11820000.dma-controller:6
153:          0          0     GICv3 164 Edge 11820000.dma-controller:7
154:          0          0     GICv3 165 Edge 11820000.dma-controller:8
155:          0          0     GICv3 166 Edge 11820000.dma-controller:9
156:          0          0     GICv3 167 Edge 11820000.dma-controller:10
157:          0          0     GICv3 168 Edge 11820000.dma-controller:11
158:          0          0     GICv3 169 Edge 11820000.dma-controller:12
159:          0          0     GICv3 170 Edge 11820000.dma-controller:13
160:          0          0     GICv3 171 Edge 11820000.dma-controller:14
161:          0          0     GICv3 172 Edge 11820000.dma-controller:15
162:          0          0     GICv3 186 Level     11840000.gpu
163:          0          0     GICv3 187 Level     11840000.gpu
164:          1          0     GICv3 185 Level     11840000.gpu
166:        790          0     GICv3 136 Level     11c00000.mmc
167:          0          0     GICv3 137 Level     11c00000.mmc
168:      27937          0     GICv3 138 Level     11c10000.mmc
169:          0          0     GICv3 139 Level     11c10000.mmc
170:         20          0     GICv3 116 Level     eth0
173:         79          0     GICv3 119 Level     eth1
176:          0          0     GICv3 123 Level     ohci_hcd:usb3
177:          0          0     GICv3 128 Level     ohci_hcd:usb4
178:          0          0     GICv3 124 Level     ehci_hcd:usb1
179:          0          0     GICv3 129 Level     ehci_hcd:usb2
180:          0          0     GICv3 126 Level 11c50200.usb-phy
181:          0          0     GICv3 131 Level 11c70200.usb-phy
182:          0          0     GICv3 132 Edge      11c60000.usb
186:          0          0     GICv3 302 Edge      10048400.gpt
187:          0          0     GICv3 303 Edge      10048400.gpt
194:          0          0     GICv3 310 Edge      10048400.gpt
199:          0          0     GICv3 181 Level     10870000.vsp
208:          0          0     GICv3  80 Edge      timer@12801800
209:          0          0     GICv3 199 Level     rzg2l_cru
211:          1          0  11030000.pin-controller   8 Edge 
11c20000.ethernet-ffffffff:00
212:          3          0  11030000.pin-controller   9 Edge 
11c30000.ethernet-ffffffff:00
IPI0:      1995       6562       Rescheduling interrupts
IPI1:        200         161       Function call interrupts
IPI2:         0          0       CPU stop interrupts
IPI3:         0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0       Timer broadcast interrupts
IPI5:         0          0       IRQ work interrupts
IPI6:         0          0       CPU wake-up interrupts
Err:          0


-- 


*RON EGGLER*
Firmware Engineer
(he/him/his)
www.mistywest.com
MistyWest Logo

Re: issues to bring up two VSC8531 PHYs

[Andrew Lunn]

> > You can also try:
> > 
> > ethtool --phy-statistics ethX
> 
> after appliaction of the above patch, ethtool tells me
> 
> # ethtool --phy-statistics eth0
> PHY statistics:
>      phy_receive_errors: 65535
>     phy_idle_errors: 255

So these have saturated. Often these counters don't wrap, they stop at
the maximum value.

These errors also indicate your problem is probably not between the
MAC and the PHY, but between the PHY and the RJ45 socket. Or maybe how
the PHY is clocked. It might not have a stable clock, or the wrong
clock frequency.

    Andrew

Re: issues to bring up two VSC8531 PHYs

[Ron Eggler]

On 4/21/23 09:35, Andrew Lunn wrote:
>>> You can also try:
>>>
>>> ethtool --phy-statistics ethX
>> after appliaction of the above patch, ethtool tells me
>>
>> # ethtool --phy-statistics eth0
>> PHY statistics:
>>       phy_receive_errors: 65535
>>      phy_idle_errors: 255
> So these have saturated. Often these counters don't wrap, they stop at
> the maximum value.
>
> These errors also indicate your problem is probably not between the
> MAC and the PHY, but between the PHY and the RJ45 socket. Or maybe how
> the PHY is clocked. It might not have a stable clock, or the wrong
> clock frequency.

The man page (https://www.man7.org/linux/man-pages/man8/ethtool.8.html) 
does not give any details about what phy_receive_errors or 
phy_idle_errors refer to exactly, is there any documentation about it 
that I could not find?


Ron

-- 
RON EGGLER Firmware Engineer (he/him/his) www.mistywest.com

Re: issues to bring up two VSC8531 PHYs

[Florian Fainelli]

On 4/21/2023 3:55 PM, Ron Eggler wrote:
> 
> On 4/21/23 09:35, Andrew Lunn wrote:
>>>> You can also try:
>>>>
>>>> ethtool --phy-statistics ethX
>>> after appliaction of the above patch, ethtool tells me
>>>
>>> # ethtool --phy-statistics eth0
>>> PHY statistics:
>>>       phy_receive_errors: 65535
>>>      phy_idle_errors: 255
>> So these have saturated. Often these counters don't wrap, they stop at
>> the maximum value.
>>
>> These errors also indicate your problem is probably not between the
>> MAC and the PHY, but between the PHY and the RJ45 socket. Or maybe how
>> the PHY is clocked. It might not have a stable clock, or the wrong
>> clock frequency.
> 
> The man page (https://www.man7.org/linux/man-pages/man8/ethtool.8.html) 
> does not give any details about what phy_receive_errors or 
> phy_idle_errors refer to exactly, is there any documentation about it 
> that I could not find?

The statistics are inherently PHY specific and how a driver writer 
choses to map a name to a specific PHY counter is backed within the driver.
-- 
Florian

Re: issues to bring up two VSC8531 PHYs

[Ron Eggler]

On 4/21/23 09:35, Andrew Lunn wrote:
>>> You can also try:
>>>
>>> ethtool --phy-statistics ethX
>> after appliaction of the above patch, ethtool tells me
>>
>> # ethtool --phy-statistics eth0
>> PHY statistics:
>>       phy_receive_errors: 65535
>>      phy_idle_errors: 255
> So these have saturated. Often these counters don't wrap, they stop at
> the maximum value.
>
> These errors also indicate your problem is probably not between the
> MAC and the PHY, but between the PHY and the RJ45 socket. Or maybe how
> the PHY is clocked. It might not have a stable clock, or the wrong
> clock frequency.

Oh and as for the clock, there's a 25MHz, +/- 20ppm  crystal hooked up 
to XTAL1 & XTAL2 pins which is a supported clock source (per datasheet) 
but I also see in the following:

For RGMII mode: Configure register 20E2 (to access register 20E2, 
register 31 must be set to 2).
Set bit 11 to 0 and set RX_CLK delay and TX_CLK delay accordingly 
through bit [6:4] and/or bit [2:0]
respectively.

I figured out that the register 20E2 is in the extended register space 
which is accessible when register 0x1F is set to 0x0002. I've been able 
to set and read it, I found a nice loop from David Creger that reads the 
complete register space and prints it to stdout:

# x=0; while  [ $x -le 31 ]; do printf  "Register 0x%02X = " $x ; 
phytool read eth0/0/$x  $(( x++ )) ; done
Register 0x00 = 0x1040
Register 0x01 = 0x796d
Register 0x02 = 0x0007
Register 0x03 = 0x0572
Register 0x04 = 0x0101
Register 0x05 = 0x45e1
Register 0x06 = 0x0005
Register 0x07 = 0x2801
Register 0x08 = 0000
Register 0x09 = 0x0200
Register 0x0A = 0x4000
Register 0x0B = 0000
Register 0x0C = 0000
Register 0x0D = 0000
Register 0x0E = 0000
Register 0x0F = 0x3000
Register 0x10 = 0x028e
Register 0x11 = 0x0100
Register 0x12 = 0x0800
Register 0x13 = 0000
Register 0x14 = 0000
Register 0x15 = 0000
Register 0x16 = 0000
Register 0x17 = 0000
Register 0x18 = 0000
Register 0x19 = 0000
Register 0x1A = 0000
Register 0x1B = 0x0ff0
Register 0x1C = 0000
Register 0x1D = 0000
Register 0x1E = 0x0030
Register 0x1F = 0x0002

and per the datasheet, the extended registers space maps into the above 
registers 16 - 30 like:

Address Name
16E2 Cu PMD Transmit Control
17E2 EEE Control
18E2–19E2 Reserved
20E2 RGMII Control
21E2 Wake-on-LAN MAC Address [15:0]
22E2 Wake-on-LAN MAC Address [31:16]
23E2 Wake-on-LAN MAC Address [47:32]
24E2 Secure-On Password [15:0]
25E2 Secure-On Password [31:16]
26E2 Secure-On Password [47:32]
27E2 Wake-on-LAN and MAC Interface Control
28E2 Extended Interrupt Mask
29E2 Extended Interrupt Status
30E2 Reserved

So per my calculations:

Register 0x10 = -> 16E2
Register 0x11 = -> 17E2
Register 0x12 = -> 18E2
Register 0x13 = -> 19E2
Register 0x14 = -> 20E2
Register 0x15 = -> 21E2
Register 0x16 = -> 22E2
Register 0x17 = -> 23E2
Register 0x18 = -> 24E2
Register 0x19 = -> 25E2
Register 0x1A = -> 26E2
Register 0x1B = -> 27E2
Register 0x1C = -> 28E2
Register 0x1D = -> 29E2
Register 0x1E = -> 30E2

20 (0x14) maps into 20E2 but I'm not fully certain what RX_CLK and 
TX_CLK should be set to, can anyone help me out, here?

datasheet: 
https://www.microchip.com/content/dam/mchp/documents/OTH/ProductDocuments/DataSheets/VMDS-10514.pdf

Also, should the above delay configuration not be part of the driver and 
configurable through the device tree?

https://www.kernel.org/doc/Documentation/devicetree/bindings/net/mscc-phy-vsc8531.txt

-- 
RON EGGLER Firmware Engineer (he/him/his) www.mistywest.com

Re: issues to bring up two VSC8531 PHYs

[Ron Eggler]

On 4/21/23 17:09, Florian Fainelli wrote:
>
>
> On 4/21/2023 3:55 PM, Ron Eggler wrote:
>>
>> On 4/21/23 09:35, Andrew Lunn wrote:
>>>>> You can also try:
>>>>>
>>>>> ethtool --phy-statistics ethX
>>>> after appliaction of the above patch, ethtool tells me
>>>>
>>>> # ethtool --phy-statistics eth0
>>>> PHY statistics:
>>>>       phy_receive_errors: 65535
>>>>      phy_idle_errors: 255
>>> So these have saturated. Often these counters don't wrap, they stop at
>>> the maximum value.
>>>
>>> These errors also indicate your problem is probably not between the
>>> MAC and the PHY, but between the PHY and the RJ45 socket. Or maybe how
>>> the PHY is clocked. It might not have a stable clock, or the wrong
>>> clock frequency.
>>
>> The man page 
>> (https://www.man7.org/linux/man-pages/man8/ethtool.8.html) does not 
>> give any details about what phy_receive_errors or phy_idle_errors 
>> refer to exactly, is there any documentation about it that I could 
>> not find?
>
> The statistics are inherently PHY specific and how a driver writer 
> choses to map a name to a specific PHY counter is backed within the 
> driver.
Thank you, I think I have moved past this now:
When I reboot, both RX & TX_CLK delay values are set to 0x0044 which 
equates 2.0ns delay and this actually lets me monitor traffic on the 
local network with tcpdump but still, my arp address doesn't go out and 
while my arp table gets populated, I'm not able to get any ping responses:
My interface in question:
# ifconfig eth0
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
         inet 192.168.1.123  netmask 255.255.255.0  broadcast 192.168.1.255
         ether 92:95:1c:76:8c:3e  txqueuelen 1000  (Ethernet)
         RX packets 94  bytes 22123 (21.6 KiB)
         RX errors 0  dropped 36  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
         device interrupt 170
arp table:
# arp
Address                  HWtype  HWaddress           Flags 
Mask            Iface
192.168.1.222            ether   54:04:a6:f3:19:db C                     
eth0
192.168.1.223            ether   68:ec:c5:ca:13:9f C                     
eth0
none of these hosts would reply to pings though but
# tcpdump -i eth0 ip
shows me traffic on the local network
the phy statistics now look like:
# ethtool --phy-statistics eth0
PHY statistics:
      phy_receive_errors: 0
      phy_false_carrier: 0
      phy_cu_media_link_disconnect: 0
      phy_cu_media_crc_good_count: 9667
      phy_cu_media_crc_error_count: 0
It appears like RX packets are getting dropped but interestingly the TX 
packets are showing 0 even though the ping command should send out some 
data:
# ifconfig eth0
eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500  metric 1
         inet 192.168.1.123  netmask 255.255.255.0  broadcast 192.168.1.255
         ether 92:95:1c:76:8c:3e  txqueuelen 1000  (Ethernet)
         RX packets 9885  bytes 2202753 (2.1 MiB)
         RX errors 0  dropped 3916  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
         device interrupt 170

what could be going on here and how can I trouble shoot this further?

Thank you!


-- 
RON EGGLER Firmware Engineer (he/him/his) www.mistywest.com