ocelot-pcs - qsgmii planning

ocelot-pcs - qsgmii planning

[Colin Foster]

Hi Alexandre, Vladimir, and those with interest of Ocelot chips,

I'm starting this thread to just touch base before I dive into any PCS
changes for Ocelot. I've appreciated all your guidance, and the only
time I felt I've been led astray is when Alexandre told me it should be
easy :-)

I'm at the point where I'm starting to integrate the additional 4 copper
ports of the VSC7512 reference board. They are 4 ports connected through
a QSGMII bus, to a VSC8514 phy.

The 8514 driver seems to be getting invoked, and running just fine.
Also, I was able to slightly modify (hack*)
drivers/phy/mscc/phy-ocelot-serdes.c to work with my in-development
ocelot-mfd. I believe that is what I need to configure the HSIO
registers.

(*the device_is_mfd info I was using falls apart with the
HSIO/syscon/mfd implementation here, sadly. A new probe function would
easily clean that up, but it is more for me to think about... I digress)

I'm using these device tree settings:

    port@4 {
        reg = <4>;
        label = "swp4";
        status = "okay";
        phy-handle = <&sw_phy4>;
        phy-mode = "qsgmii";
        phys = <&serdes 4 SERDES6G(0)>;
    };
    port@5 {
        reg = <5>;
        label = "swp5";
        status = "okay";
        phy-handle = <&sw_phy5>;
        phy-mode = "qsgmii";
        phys = <&serdes 5 SERDES6G(0)>;
    };
...
    serdes: serdes {
        compatible = "mscc,vsc7514-serdes";
        #phy-cells = <2>;
    };
    mdio1: mdio1 {
        compatible = "mscc,ocelot-miim",
        pinctrl-names = "default";
        pinctrl-0 = <&miim1>;
        #address-cells = <1>;
        #size-cells = <0>;

        sw_phy4: ethernet-phy@4 {
            reg = <0x4>;
        };
    };

[    3.886787] libphy: ocelot_ext MDIO bus: probed
[    5.345891] ocelot-ext-switch ocelot-ext-switch: PHY [ocelot-ext-switch-mii:00] driver [Generic PHY] (irq=POLL)
[    5.357341] ocelot-ext-switch ocelot-ext-switch: configuring for phy/internal link mode
[    5.372525] ocelot-ext-switch ocelot-ext-switch swp1 (uninitialized): PHY [ocelot-ext-switch-mii:01] driver [Generic PHY] (irq=POLL)
[    5.388865] ocelot-ext-switch ocelot-ext-switch swp2 (uninitialized): PHY [ocelot-ext-switch-mii:02] driver [Generic PHY] (irq=POLL)
[    5.405086] ocelot-ext-switch ocelot-ext-switch swp3 (uninitialized): PHY [ocelot-ext-switch-mii:03] driver [Generic PHY] (irq=POLL)
[    6.291876] ocelot-ext-switch ocelot-ext-switch swp4 (uninitialized): PHY [ocelot-miim1-mii:04] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.471891] ocelot-ext-switch ocelot-ext-switch swp5 (uninitialized): PHY [ocelot-miim1-mii:05] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.651895] ocelot-ext-switch ocelot-ext-switch swp6 (uninitialized): PHY [ocelot-miim1-mii:06] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
[    6.831879] ocelot-ext-switch ocelot-ext-switch swp7 (uninitialized): PHY [ocelot-miim1-mii:07] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)


It seems like that, along with everything in vsc7514_phylink_mac_config,
should be everything I need for operation of the four ports through the
8512. I've added OCELOT_QUIRK_SGMII_PORTS_MUST_BE_UP - but I'm not sure
that's a quirk I need. Plus the only behavior it currently adds is once
the port is up, it never comes back down.

The current behavior I'm seeing is links and rates get detected, packets
appear to be getting transmitted (ethtool stats) but they aren't, and
nothing is received on either end.

Is there something I'm missing with the device tree? Or is this the
purpose of the PCS driver I'm looking into? I'm getting a feeling that
my configuration is correct, and that I need to add SerDes support for
these ports in phylink_mac_config... I noticed that there's the "SGMII
only for now" comment, and when I look at the reference application for
the 7512 there's a comment "external phy uses QSGMII interface" that
appears to set the SGMII_MODE_ENA bit to 0.


Thank you as always for your time,

Colin Foster.

Re: ocelot-pcs - qsgmii planning

[Vladimir Oltean]

Hi Colin,

I'm sorry, but I have no direct experience with the SERDES and PCS of
the Ocelot switches, since those blocks were completely replaced from
the NXP instantiations I'm playing with.

On Thu, Jan 06, 2022 at 01:50:03AM -0800, Colin Foster wrote:
> Hi Alexandre, Vladimir, and those with interest of Ocelot chips,
> 
> I'm starting this thread to just touch base before I dive into any PCS
> changes for Ocelot. I've appreciated all your guidance, and the only
> time I felt I've been led astray is when Alexandre told me it should be
> easy :-)
> 
> I'm at the point where I'm starting to integrate the additional 4 copper
> ports of the VSC7512 reference board. They are 4 ports connected through
> a QSGMII bus, to a VSC8514 phy.
> 
> The 8514 driver seems to be getting invoked, and running just fine.
> Also, I was able to slightly modify (hack*)
> drivers/phy/mscc/phy-ocelot-serdes.c to work with my in-development
> ocelot-mfd. I believe that is what I need to configure the HSIO
> registers.

And are you also getting a reference to it and calling it, like the
ocelot driver does here?

	err = phy_set_mode_ext(serdes, PHY_MODE_ETHERNET, phy_mode);

Could we see that portion of the code in the felix driver?

> 
> (*the device_is_mfd info I was using falls apart with the
> HSIO/syscon/mfd implementation here, sadly. A new probe function would
> easily clean that up, but it is more for me to think about... I digress)
> 
> I'm using these device tree settings:
> 
>     port@4 {
>         reg = <4>;
>         label = "swp4";
>         status = "okay";
>         phy-handle = <&sw_phy4>;
>         phy-mode = "qsgmii";
>         phys = <&serdes 4 SERDES6G(0)>;
>     };
>     port@5 {
>         reg = <5>;
>         label = "swp5";
>         status = "okay";
>         phy-handle = <&sw_phy5>;
>         phy-mode = "qsgmii";
>         phys = <&serdes 5 SERDES6G(0)>;
>     };
> ...
>     serdes: serdes {
>         compatible = "mscc,vsc7514-serdes";
>         #phy-cells = <2>;
>     };
>     mdio1: mdio1 {
>         compatible = "mscc,ocelot-miim",
>         pinctrl-names = "default";
>         pinctrl-0 = <&miim1>;
>         #address-cells = <1>;
>         #size-cells = <0>;
> 
>         sw_phy4: ethernet-phy@4 {
>             reg = <0x4>;
>         };
>     };
> 
> [    3.886787] libphy: ocelot_ext MDIO bus: probed
> [    5.345891] ocelot-ext-switch ocelot-ext-switch: PHY [ocelot-ext-switch-mii:00] driver [Generic PHY] (irq=POLL)
> [    5.357341] ocelot-ext-switch ocelot-ext-switch: configuring for phy/internal link mode
> [    5.372525] ocelot-ext-switch ocelot-ext-switch swp1 (uninitialized): PHY [ocelot-ext-switch-mii:01] driver [Generic PHY] (irq=POLL)
> [    5.388865] ocelot-ext-switch ocelot-ext-switch swp2 (uninitialized): PHY [ocelot-ext-switch-mii:02] driver [Generic PHY] (irq=POLL)
> [    5.405086] ocelot-ext-switch ocelot-ext-switch swp3 (uninitialized): PHY [ocelot-ext-switch-mii:03] driver [Generic PHY] (irq=POLL)
> [    6.291876] ocelot-ext-switch ocelot-ext-switch swp4 (uninitialized): PHY [ocelot-miim1-mii:04] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> [    6.471891] ocelot-ext-switch ocelot-ext-switch swp5 (uninitialized): PHY [ocelot-miim1-mii:05] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> [    6.651895] ocelot-ext-switch ocelot-ext-switch swp6 (uninitialized): PHY [ocelot-miim1-mii:06] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> [    6.831879] ocelot-ext-switch ocelot-ext-switch swp7 (uninitialized): PHY [ocelot-miim1-mii:07] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> 
> 
> It seems like that, along with everything in vsc7514_phylink_mac_config,
> should be everything I need for operation of the four ports through the
> 8512. I've added OCELOT_QUIRK_SGMII_PORTS_MUST_BE_UP - but I'm not sure
> that's a quirk I need. Plus the only behavior it currently adds is once
> the port is up, it never comes back down.

Correct. I noticed that the code was structured to do this when I
converted it to phylink, but I don't know if it's needed either. I just
named that behavior as OCELOT_QUIRK_QSGMII_PORTS_MUST_BE_UP and made it
optional.

> The current behavior I'm seeing is links and rates get detected, packets
> appear to be getting transmitted (ethtool stats) but they aren't, and
> nothing is received on either end.

could you also check ethtool --phy-statistics swp4 on both ends?

> Is there something I'm missing with the device tree? Or is this the
> purpose of the PCS driver I'm looking into? I'm getting a feeling that
> my configuration is correct, and that I need to add SerDes support for
> these ports in phylink_mac_config... I noticed that there's the "SGMII
> only for now" comment, and when I look at the reference application for
> the 7512 there's a comment "external phy uses QSGMII interface" that
> appears to set the SGMII_MODE_ENA bit to 0.

SGMII_MODE_ENA in PCS1G_MODE_CFG refers to the selection between Cisco
SGMII/QSGMII vs IEEE 802.3 fiber modes. Different autonegotiation code
words being transmitted. It's correct to leave it the way it is. It's
curious that the reference application suggests to set it to 0 for QSGMII.
I suppose that if in-band autoneg isn't enabled in PCS1G_ANEG_CFG, it
doesn't really matter what kind of autoneg is used.

For SGMII vs QSGMII, the serdes_set_mode() function in
phy-ocelot-serdes.c should set HSIO_HW_CFG_QSGMII_ENA where appropriate.

> Thank you as always for your time,
> 
> Colin Foster.

What bootloader do you use? Do you have VSC8514 PHY initialization of
any sort in the bootloader?

Re: ocelot-pcs - qsgmii planning

[Colin Foster]

Hi Vladimir,

On Thu, Jan 06, 2022 at 12:48:37PM +0200, Vladimir Oltean wrote:
> Hi Colin,
> 
> I'm sorry, but I have no direct experience with the SERDES and PCS of
> the Ocelot switches, since those blocks were completely replaced from
> the NXP instantiations I'm playing with.

I realized that might be the case as I was writing this email. Thank you
for still helping.

> 
> On Thu, Jan 06, 2022 at 01:50:03AM -0800, Colin Foster wrote:
> > Hi Alexandre, Vladimir, and those with interest of Ocelot chips,
> > 
> > I'm starting this thread to just touch base before I dive into any PCS
> > changes for Ocelot. I've appreciated all your guidance, and the only
> > time I felt I've been led astray is when Alexandre told me it should be
> > easy :-)
> > 
> > I'm at the point where I'm starting to integrate the additional 4 copper
> > ports of the VSC7512 reference board. They are 4 ports connected through
> > a QSGMII bus, to a VSC8514 phy.
> > 
> > The 8514 driver seems to be getting invoked, and running just fine.
> > Also, I was able to slightly modify (hack*)
> > drivers/phy/mscc/phy-ocelot-serdes.c to work with my in-development
> > ocelot-mfd. I believe that is what I need to configure the HSIO
> > registers.
> 
> And are you also getting a reference to it and calling it, like the
> ocelot driver does here?
> 
> 	err = phy_set_mode_ext(serdes, PHY_MODE_ETHERNET, phy_mode);
> 
> Could we see that portion of the code in the felix driver?

Oh, no I'm not. That's probably the hint that I need. I see that happens
in VSC7514 probe before calling ocelot_port_init, unlike felix. Alright,
there's some work for me to do! Thanks :)

> 
> > 
> > (*the device_is_mfd info I was using falls apart with the
> > HSIO/syscon/mfd implementation here, sadly. A new probe function would
> > easily clean that up, but it is more for me to think about... I digress)
> > 
> > I'm using these device tree settings:
> > 
> >     port@4 {
> >         reg = <4>;
> >         label = "swp4";
> >         status = "okay";
> >         phy-handle = <&sw_phy4>;
> >         phy-mode = "qsgmii";
> >         phys = <&serdes 4 SERDES6G(0)>;
> >     };
> >     port@5 {
> >         reg = <5>;
> >         label = "swp5";
> >         status = "okay";
> >         phy-handle = <&sw_phy5>;
> >         phy-mode = "qsgmii";
> >         phys = <&serdes 5 SERDES6G(0)>;
> >     };
> > ...
> >     serdes: serdes {
> >         compatible = "mscc,vsc7514-serdes";
> >         #phy-cells = <2>;
> >     };
> >     mdio1: mdio1 {
> >         compatible = "mscc,ocelot-miim",
> >         pinctrl-names = "default";
> >         pinctrl-0 = <&miim1>;
> >         #address-cells = <1>;
> >         #size-cells = <0>;
> > 
> >         sw_phy4: ethernet-phy@4 {
> >             reg = <0x4>;
> >         };
> >     };
> > 
> > [    3.886787] libphy: ocelot_ext MDIO bus: probed
> > [    5.345891] ocelot-ext-switch ocelot-ext-switch: PHY [ocelot-ext-switch-mii:00] driver [Generic PHY] (irq=POLL)
> > [    5.357341] ocelot-ext-switch ocelot-ext-switch: configuring for phy/internal link mode
> > [    5.372525] ocelot-ext-switch ocelot-ext-switch swp1 (uninitialized): PHY [ocelot-ext-switch-mii:01] driver [Generic PHY] (irq=POLL)
> > [    5.388865] ocelot-ext-switch ocelot-ext-switch swp2 (uninitialized): PHY [ocelot-ext-switch-mii:02] driver [Generic PHY] (irq=POLL)
> > [    5.405086] ocelot-ext-switch ocelot-ext-switch swp3 (uninitialized): PHY [ocelot-ext-switch-mii:03] driver [Generic PHY] (irq=POLL)
> > [    6.291876] ocelot-ext-switch ocelot-ext-switch swp4 (uninitialized): PHY [ocelot-miim1-mii:04] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> > [    6.471891] ocelot-ext-switch ocelot-ext-switch swp5 (uninitialized): PHY [ocelot-miim1-mii:05] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> > [    6.651895] ocelot-ext-switch ocelot-ext-switch swp6 (uninitialized): PHY [ocelot-miim1-mii:06] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> > [    6.831879] ocelot-ext-switch ocelot-ext-switch swp7 (uninitialized): PHY [ocelot-miim1-mii:07] driver [Microsemi GE VSC8514 SyncE] (irq=POLL)
> > 
> > 
> > It seems like that, along with everything in vsc7514_phylink_mac_config,
> > should be everything I need for operation of the four ports through the
> > 8512. I've added OCELOT_QUIRK_SGMII_PORTS_MUST_BE_UP - but I'm not sure
> > that's a quirk I need. Plus the only behavior it currently adds is once
> > the port is up, it never comes back down.
> 
> Correct. I noticed that the code was structured to do this when I
> converted it to phylink, but I don't know if it's needed either. I just
> named that behavior as OCELOT_QUIRK_QSGMII_PORTS_MUST_BE_UP and made it
> optional.

I'll report my findings.

> 
> > The current behavior I'm seeing is links and rates get detected, packets
> > appear to be getting transmitted (ethtool stats) but they aren't, and
> > nothing is received on either end.
> 
> could you also check ethtool --phy-statistics swp4 on both ends?

phy_cu_media_link_disconnect increments as expected. No activity on
phy_cu_media_crc_good_count. I feel like it probably isn't worth digging
too much into this until I fix the phy_set_mode_ext.

> 
> > Is there something I'm missing with the device tree? Or is this the
> > purpose of the PCS driver I'm looking into? I'm getting a feeling that
> > my configuration is correct, and that I need to add SerDes support for
> > these ports in phylink_mac_config... I noticed that there's the "SGMII
> > only for now" comment, and when I look at the reference application for
> > the 7512 there's a comment "external phy uses QSGMII interface" that
> > appears to set the SGMII_MODE_ENA bit to 0.
> 
> SGMII_MODE_ENA in PCS1G_MODE_CFG refers to the selection between Cisco
> SGMII/QSGMII vs IEEE 802.3 fiber modes. Different autonegotiation code
> words being transmitted. It's correct to leave it the way it is. It's
> curious that the reference application suggests to set it to 0 for QSGMII.
> I suppose that if in-band autoneg isn't enabled in PCS1G_ANEG_CFG, it
> doesn't really matter what kind of autoneg is used.
> 
> For SGMII vs QSGMII, the serdes_set_mode() function in
> phy-ocelot-serdes.c should set HSIO_HW_CFG_QSGMII_ENA where appropriate.

I agree. I think you've answered my most important question, which is
whether there's an obvious issue in my Device Tree / understanding. A
blunder there can lead to significant loss of time making this work the
wrong way. Shuffling datasheets, reference apps, register dumps...

> 
> > Thank you as always for your time,
> > 
> > Colin Foster.
> 
> What bootloader do you use? Do you have VSC8514 PHY initialization of
> any sort in the bootloader?

U-Boot (now you know my secret next project...) that is basically the
default configuration of BeagleBone. Definitely no VSC8514 knowledge by
way of SPI > 7512 > MDIO(1) > 8514.