upstream Linux support for Ethernet combo ports via external mux

upstream Linux support for Ethernet combo ports via external mux

[Daniel Golle]

Hi,

Looking for ways to support a passive SerDes mux in vanilla Linux I
found Maxime's slides "Multi-port and Multi-PHY Ethernet interfaces"[1].

The case I want to support is probably quite common nowadays but isn't
covered there nor implemented in Linux.

 +----------------------------+
 |            SoC             |
 |    +-----------------+     |
 |    |       MAC       |     |
 |    +----+-------+----+     |
 |         |  PCS  |   +------+
 |         +---=---+   | GPIO |
 +-------------=-------+---=--+
              |            |
           +---=---+       |
           |  Mux  <-------+
           +-=---=-+
             |   |
            /     \
     +-----=-+   +-=-----+
     |  PHY  |   |  SFP  |
     +-------+   +-------+

So other than it was when SoCs didn't have built-in PCSs, now the SFP is
not connected to the PHY, but there is an additional mux IC controlled
by the SoC to connect the serialized MII either to the PHY (in case no
SFP is inserted) or to the SFP (in case a module is inserted).

MediaTek came up with a vendor-specific solution[2] for that which works
well -- but obviously it would be much nicer to have generic, vendor-
agnostic support for such setups in phylink, ideally based on the
existing gpio-mux driver.

So I imagine something like a generic phylink-mux, controlled by hooking
to the module_insert and module_remove remove SFP ops (assuming the
moddef0 signal is connected...).

Before I get my hands dirty, please join my line of thought for one
moment, so we can agree on a sketch:

Does everyone agree that phylink would be the right place to do this?

DT bindings could look like this (option A):
 ...
    mux: mux-controller {
        compatible = "gpio-mux";
        #mux-control-cells = <0>;

        mux-gpios = <&pio 7 GPIO_ACTIVE_HIGH>;
    };

    mux0: mii-mux {
        compatible = "mii-mux";
        mux-controls = <&mux>;
        #address-cells = <1>;
        #size-cells = <0>;

        channel@0 {
            reg = <0>;
            sfp = <&sfp0>;
            managed = "in-band-status";
            module-presence-controls-mux;
        };

        channel@1 {
            reg = <1>;
            phy-handle = <&phy0>;
            phy-connection-type = "sgmii";
        };
    };
  };

  soc {
      ethernet@12340000 {
          mii-mux = <&mux0>;
      };
  };
    

or like this (option B):
 ...
    mux: mux-controller {
        compatible = "gpio-mux";
        #mux-control-cells = <0>;

        mux-gpios = <&pio 7 GPIO_ACTIVE_HIGH>;
    };
  };

  soc {
      ethernet@12340000 {
          sfp = <&sfp0>;
          phy = <&phy0>;
          phy-connection-type = "sgmii";
          mux-controls = <&mux>;
      };
  };


Obviously option A is more expressive, but also more complex, and I'm
not 100% sure if all that complexity is really needed in practise.
However, for "better safe than sorry" reasons I'd opt for option A,
unless anyone comes up with a better idea.

Let me know what you think.


Cheers


Daniel


[1]: https://netdevconf.org/0x17/docs/netdev-0x17-paper2-talk-slides/multi-port-multi-phy-interfaces.pdf
[2]: https://git01.mediatek.com/plugins/gitiles/openwrt/feeds/mtk-openwrt-feeds/+/refs/heads/master/autobuild/unified/global/24.10/files/target/linux/mediatek/patches-6.6/999-2708-net-ethernet-mtk_eth_soc-support-ethernet-passive-mu.patch

Re: upstream Linux support for Ethernet combo ports via external mux

[Andrew Lunn]

On Mon, Feb 10, 2025 at 11:11:07PM +0000, Daniel Golle wrote:
> Hi,
> 
> Looking for ways to support a passive SerDes mux in vanilla Linux I
> found Maxime's slides "Multi-port and Multi-PHY Ethernet interfaces"[1].

Maxime is still working on this. There was a patchset posted
recently...

> The case I want to support is probably quite common nowadays but isn't
> covered there nor implemented in Linux.
> 
>  +----------------------------+
>  |            SoC             |
>  |    +-----------------+     |
>  |    |       MAC       |     |
>  |    +----+-------+----+     |
>  |         |  PCS  |   +------+
>  |         +---=---+   | GPIO |
>  +-------------=-------+---=--+
>               |            |
>            +---=---+       |
>            |  Mux  <-------+
>            +-=---=-+
>              |   |
>             /     \
>      +-----=-+   +-=-----+
>      |  PHY  |   |  SFP  |
>      +-------+   +-------+
> 
> So other than it was when SoCs didn't have built-in PCSs, now the SFP is
> not connected to the PHY, but there is an additional mux IC controlled
> by the SoC to connect the serialized MII either to the PHY (in case no
> SFP is inserted) or to the SFP (in case a module is inserted).
> 
> MediaTek came up with a vendor-specific solution[2] for that which works
> well -- but obviously it would be much nicer to have generic, vendor-
> agnostic support for such setups in phylink, ideally based on the
> existing gpio-mux driver.

I don't actually understand how it can work. For the PHY the PCS
probably needs to be running SGMII. For the SFP it probably wants
1000BaseX. It looks like phylink has no idea the mux has been flipped,
so it needs to reprogram the PCS. You cannot just create two phylink
instances and flip flop between them. You need phylink actively
involved so it can correctly manage the PCS.

	Andrew

Re: upstream Linux support for Ethernet combo ports via external mux

[Maxime Chevallier]

Hi Daniel,

On Mon, 10 Feb 2025 23:11:07 +0000
Daniel Golle <daniel@makrotopia.org> wrote:

> Hi,
> 
> Looking for ways to support a passive SerDes mux in vanilla Linux I
> found Maxime's slides "Multi-port and Multi-PHY Ethernet interfaces"[1].

I'm actively working on getting this exact case supported. The MUX I
have is an RGMII MUX, but gpio-driven just as in your case :) I'll make
sure to CC: you in the next iterations

> The case I want to support is probably quite common nowadays but isn't
> covered there nor implemented in Linux.
> 
>  +----------------------------+
>  |            SoC             |
>  |    +-----------------+     |
>  |    |       MAC       |     |
>  |    +----+-------+----+     |
>  |         |  PCS  |   +------+
>  |         +---=---+   | GPIO |
>  +-------------=-------+---=--+
>               |            |
>            +---=---+       |
>            |  Mux  <-------+
>            +-=---=-+
>              |   |
>             /     \
>      +-----=-+   +-=-----+
>      |  PHY  |   |  SFP  |
>      +-------+   +-------+

The work in question is here :

https://lore.kernel.org/netdev/20250210095542.721bf967@fedora-1.home/

Preliminary work was merged here :

https://lore.kernel.org/netdev/20240821151009.1681151-1-maxime.chevallier@bootlin.com/

To summarize the whole process :

I started with a use-case similar as yours, but instead of PHY - SFP
combo I have a PHY-PHY combo. That's fairly analoguous to what you have
(say you plug a copper SFP on the cage, you now have 2 parallel PHYs
behind the MAC)

So the first step was the intrduction of phy_link_topology to allow
tracking PHYs attached to a MAC, even if they aren't directly on the
active datapath (the PHY isolated by the MUX). That was clearly a
pre-requisite for any kind of muxing.

From all the feedback (among other, from Oleksji Rempel, Russell King,
Marek Behùn), it appears that this setup you have is not that unusual,
and also ressembles other setups such as :

        +-- SFP
MAC - PHY
        +-- RJ45

If we look at the big picture from the end-user's point of view,
there's no difference between a MAC connected to a dual-port PHY and a
MAC connected to a PHY + SFP through a MUX, it's just a NIC with 2
ports.

So where I'm at today is that I'm trying to get a "generic"
representation of front-facing ports accepted, that will be the kernel
objects used for configuration.

If user wants to enable the SFP port for example, if the SFP is behind
a MUX, the MUX will be cntrolled to set the correct output, and if the
SFP is behind a PHY, then the PHY driver will be in charge of the
mux'ing.

> So other than it was when SoCs didn't have built-in PCSs, now the SFP is
> not connected to the PHY, but there is an additional mux IC controlled
> by the SoC to connect the serialized MII either to the PHY (in case no
> SFP is inserted) or to the SFP (in case a module is inserted).
> 
> MediaTek came up with a vendor-specific solution[2] for that which works
> well -- but obviously it would be much nicer to have generic, vendor-
> agnostic support for such setups in phylink, ideally based on the
> existing gpio-mux driver.
> 
> So I imagine something like a generic phylink-mux, controlled by hooking
> to the module_insert and module_remove remove SFP ops (assuming the
> moddef0 signal is connected...).
> 
> Before I get my hands dirty, please join my line of thought for one
> moment, so we can agree on a sketch:
> 
> Does everyone agree that phylink would be the right place to do this?

As Andrew says, we can't really let phylink alone handle that. The
approach I have taken is to model the MUX as a dedicated driver, that
would have "parent" ops allowing to notify whatever is upstream of the
mux to know about state change :

> DT bindings could look like this (option A):
>  ...
>     mux: mux-controller {
>         compatible = "gpio-mux";
>         #mux-control-cells = <0>;
> 
>         mux-gpios = <&pio 7 GPIO_ACTIVE_HIGH>;
>     };
> 
>     mux0: mii-mux {
>         compatible = "mii-mux";
>         mux-controls = <&mux>;
>         #address-cells = <1>;
>         #size-cells = <0>;
> 
>         channel@0 {
>             reg = <0>;
>             sfp = <&sfp0>;
>             managed = "in-band-status";
>             module-presence-controls-mux;
>         };
> 
>         channel@1 {
>             reg = <1>;
>             phy-handle = <&phy0>;
>             phy-connection-type = "sgmii";
>         };
>     };
>   };
> 
>   soc {
>       ethernet@12340000 {
>           mii-mux = <&mux0>;
>       };
>   };

I have some code written for that, and I have a very similar binding in
the end :

		eth1: ucc@2200 {
			far-id = <0 1 3 4 5 6 7>;
			device_type = "network";
			compatible = "ucc_geth", "cs,mia-far";
			cell-index = <3>;
			reg = <0x2200 0x200>;
			interrupts = <34>;
			interrupt-parent = <&qeic>;
			local-mac-address = [ 00 00 00 00 00 00 ];
			rx-clock-name = "clk12";
			tx-clock-name = "clk12";
			phy-mux-handle = <&phymux>;
			pio-handle = <&ucc3pio>;
			phy-connection-type = "rmii";
		};

		phymux: phymux@0 {
			compatible = "gpio-phy-mux";
			status = "okay";
			select-gpios = <&qe_pio_c 21 1>;

			phy-node@0 {
				phy-handle = <&phy2>;
				phy-mode = "rmii";
			};

			phy-node@1 {
				active-low;
				phy-handle = <&phy3>;
				phy-mode = "rmii";
			};
		};

The mux node can of course include SFP.

> 
> or like this (option B):
>  ...
>     mux: mux-controller {
>         compatible = "gpio-mux";
>         #mux-control-cells = <0>;
> 
>         mux-gpios = <&pio 7 GPIO_ACTIVE_HIGH>;
>     };
>   };
> 
>   soc {
>       ethernet@12340000 {
>           sfp = <&sfp0>;
>           phy = <&phy0>;
>           phy-connection-type = "sgmii";
>           mux-controls = <&mux>;
>       };
>   };
> 
> 
> Obviously option A is more expressive, but also more complex, and I'm
> not 100% sure if all that complexity is really needed in practise.
> However, for "better safe than sorry" reasons I'd opt for option A,
> unless anyone comes up with a better idea.

Maybe we can sync our work, I'm all in to get more hands on this. The
current roadmap I have in mind is :

 - Get the port representation stable and accepted
 - Get muxing working for the "simple" case of PHY-driven muxes, which
includes a new netlink API, and muxing operations for front-facing ports
 - Get PHY muxes going, with a PHY mux framework and mux drivers
(gpio-driven, MMIO-register driven, completely passive muxes based on
PHY powerdown...)

I've made some attemps to get the phy mux support first, but we've
discussed that extensively with Andrew and Russell on that thread
here and the path forward would be to get that problem first from the
perspective of the ports :

https://lore.kernel.org/netdev/20241004161601.2932901-1-maxime.chevallier@bootlin.com/

Thanks,

Maxime