[RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[Nicolò Veronese]

Hi,

I and some folks in CC are working to properly port all the
 functions of a Zyxel ex5601-t0 to OpenWrt.

The manufacturer decided to use a single SerDes connected
 to both an SPF cage and an RJ45 phy. A simple GPIO is
 used to control a 2 Channel 2:1 MUX to switch the two SGMII pairs
 between the RJ45 and the SFP.

  ┌─────┐  ┌──────┐   ┌─────────┐
  │     │  │      │   │         │
  │     │  │      ├───┤ SFP     │
  │     │  │      │   └─────────┘
  │     │  │      │
  │ MAC ├──┤ MUX  │   ┌─────────┐
  │     │  │      │   │         │
  │     │  │      │   │ RJ45    │
  │     │  │      ├───┤ 2.5G PHY│
  │     │  │      │   │         │
  └─────┘  └───▲──┘   └─────────┘
               │
  MUX-GPIO ────┘

Other vendors may implement this differently (e.g. Keenetic
 KN-1011 has a similar setup, although the PHY is doing all the work),
 but this seems a common enough approach to produce cheap CPEs
 with multiple interface types for fiber internet.

In this particular case, Zyxel implemented a user-land script[1] that is
 continuously polling GPIO 57 (Moddef0) of the SFP cage.

Once an SFP module is detected the process continues as follows:
- An MDIO command disables the RJ45 PHY
- A GPIO write to GPIO 10 that switches the MUX to the SFP interface
- The MAC is then re-configured by writing directly to the SoC registers
 using a mix of lookup tables and heuristics/try-catch.

This allows Zyxel to configure the MAC with the supposedly correct SFP
 interface speed, bypassing any well-established interface speed
 auto-detection and negotiation logic.

Zyxel also configures the GMAC at boot with a fixed speed of 2500M
 forcing the link status to be always up irrespective of the real
 physical interface status.

On SFP disconnect the process is simply applied in reverse.

We are looking for guidance on how to design changes that could achieve
 the following goals and could be accepted upstream in the future:

 - SFP and RJ45 speed auto-sensing and auto-negotiation working
 - Automatic SFP/RJ45 switching
 - Failsafe logic in case both media are connected.
 - Reduce overall potential power consumption and rate-adaption by not
   having the GMAC always switched to 2500M mode without reason.
 - (optional) default configurable logic for both failsafe and idle status

References:
[1]: https://github.com/pameruoso/zyxel-ex5601t0/blob/V5.70(ACDZ.0)C0/target/linux/mediatek/ex5601t0/base-files/bin/sfp_wan.sh

Re: [RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[Russell King (Oracle)]

On Tue, Aug 29, 2023 at 05:12:48PM +0200, Nicolò Veronese wrote:
> Hi,
> 
> I and some folks in CC are working to properly port all the
>  functions of a Zyxel ex5601-t0 to OpenWrt.
> 
> The manufacturer decided to use a single SerDes connected
>  to both an SPF cage and an RJ45 phy. A simple GPIO is
>  used to control a 2 Channel 2:1 MUX to switch the two SGMII pairs
>  between the RJ45 and the SFP.
> 
>   ┌─────┐  ┌──────┐   ┌─────────┐
>   │     │  │      │   │         │
>   │     │  │      ├───┤ SFP     │
>   │     │  │      │   └─────────┘
>   │     │  │      │
>   │ MAC ├──┤ MUX  │   ┌─────────┐
>   │     │  │      │   │         │
>   │     │  │      │   │ RJ45    │
>   │     │  │      ├───┤ 2.5G PHY│
>   │     │  │      │   │         │
>   └─────┘  └───▲──┘   └─────────┘
>                │
>   MUX-GPIO ────┘

This is do-able in software, but is far from a good idea.

Yes, it would be possible to "disconnect" the RJ45 PHY from the netdev,
and switch to the SFP and back again. It would be relatively easy for
phylink to do that. What phylink would need to do is to keep track of
the SFP PHY and netdev-native PHY independently, and multiplex between
the two. It would also have to manage the netdev->phydev pointer.
Any changes to this must be done under the rtnl lock.

So technically it's possible. However, there is no notification to
userspace when such a change may occur. There's also the issue that
userspace may be in the process of issuing ethtool commands that are
affecting one of the PHYs. While holding the rtnl lock will block
those calls, a change between the PHY and e.g. a PHY on the SFP
would cause the ethtool command to target a different PHY from what
was the original target.

To solve that sanely, every PHY-based ethtool probably needs a way
to specify which PHY the command is intended for, but then there's
the question of how userspace users react to that - because it's
likely more than just modifying the ethtool utility, ethtool
commands are probably used from many programs.

IMHO, it needs a bit of thought beyond "what can we do to support a
mux".

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[Daniel Golle]

On Tue, Aug 29, 2023 at 04:38:42PM +0100, Russell King (Oracle) wrote:
> On Tue, Aug 29, 2023 at 05:12:48PM +0200, Nicolò Veronese wrote:
> > Hi,
> > 
> > I and some folks in CC are working to properly port all the
> >  functions of a Zyxel ex5601-t0 to OpenWrt.
> > 
> > The manufacturer decided to use a single SerDes connected
> >  to both an SPF cage and an RJ45 phy. A simple GPIO is
> >  used to control a 2 Channel 2:1 MUX to switch the two SGMII pairs
> >  between the RJ45 and the SFP.
> > 
> >   ┌─────┐  ┌──────┐   ┌─────────┐
> >   │     │  │      │   │         │
> >   │     │  │      ├───┤ SFP     │
> >   │     │  │      │   └─────────┘
> >   │     │  │      │
> >   │ MAC ├──┤ MUX  │   ┌─────────┐
> >   │     │  │      │   │         │
> >   │     │  │      │   │ RJ45    │
> >   │     │  │      ├───┤ 2.5G PHY│
> >   │     │  │      │   │         │
> >   └─────┘  └───▲──┘   └─────────┘
> >                │
> >   MUX-GPIO ────┘

Note that most recent MediaTek SoCs have a similar setup built-into the
SoC itself: One MAC can either be internally connected to a built-in
2.5G TP PHY or used with an external PHY or SFP via SerDes ie.
USXGMII/10GBase-R/5GBase-R/2500Base-X/1000Base-X/SGMII **which doesn't
share any pins with the TP PHY**. Hence board manufactorers are likely
to build devices exposing that MAC in both ways, as SFP cage and 2.5G
PHY, because the 2.5G PHY basically comes "for free".

In this case switching the MAC between the two is done using a mux bit
in ETHSYS syscon. However, one should also take care of powering on
and off the 2.5G PHY and maintain the trapdoor in the MDIO bus
allowing the access either built-in PHY or an external PHY at the same
address, so it's a bit more complicated than just a single bit.

> 
> This is do-able in software, but is far from a good idea.
> 
> Yes, it would be possible to "disconnect" the RJ45 PHY from the netdev,
> and switch to the SFP and back again. It would be relatively easy for
> phylink to do that. What phylink would need to do is to keep track of
> the SFP PHY and netdev-native PHY independently, and multiplex between
> the two. It would also have to manage the netdev->phydev pointer.
> Any changes to this must be done under the rtnl lock.
> 
> So technically it's possible. However, there is no notification to
> userspace when such a change may occur. There's also the issue that
> userspace may be in the process of issuing ethtool commands that are
> affecting one of the PHYs. While holding the rtnl lock will block
> those calls, a change between the PHY and e.g. a PHY on the SFP
> would cause the ethtool command to target a different PHY from what
> was the original target.

I can see that the lack of such notification is already an issue even
without the change described above. I've actually struggled with that
just a few days ago:
Some of the SFP+ modules I use for testing expose the built-in
marvell10g PHY via I2C using the RollBall protocol. The protocol
apparently requires a long waiting time between power-on and being
able to access the PHY (something like 25 seconds). So any ethtool
command issued after boot and before the 25 seconds have passed will
have no effect as of today, because the PHY is only being attached
after that. And this is not only a problem when hot-plugging the
module, obviously, but also when having it plugged in already
during boot.

So having userspace notification "some about the PHY has changed"
would already be nice to have, because that'd would an easy way
OpenWrt's netifd could know that it has to re-read supported
capabilities and re-apply ethtool link settings.

> 
> To solve that sanely, every PHY-based ethtool probably needs a way
> to specify which PHY the command is intended for, but then there's
> the question of how userspace users react to that - because it's
> likely more than just modifying the ethtool utility, ethtool
> commands are probably used from many programs.

Maybe just a cookie to make sure "sessions" of ethtool calls are
not broken by a changed from PHY to SFP or vice versa?
Ie. GLINKSETTINGS will should give you a cookie to be used with
SLINKSETTINGS and so one. The cookie may not identify the PHY, but
rather just the previous call to GLINKSETTINGS.

And this could even be optional from perspective of userspace.

> 
> IMHO, it needs a bit of thought beyond "what can we do to support a
> mux".
> 

Another thing which came to my mind is the existing port field in
many ethtool ops which could be either PORT_TP or PORT_FIBRE to
destinguish the TP PHY from the SFP at least for xLINKSETTINGS.

Re: [RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[Russell King (Oracle)]

On Tue, Aug 29, 2023 at 06:37:07PM +0100, Daniel Golle wrote:
> Another thing which came to my mind is the existing port field in
> many ethtool ops which could be either PORT_TP or PORT_FIBRE to
> destinguish the TP PHY from the SFP at least for xLINKSETTINGS.

However, PORT_TP also gets used for RJ45 SFPs, so that doesn't
distinguish between a built-in PHY and a SFP PHY.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[Jakub Kicinski]

On Tue, 29 Aug 2023 16:38:42 +0100 Russell King (Oracle) wrote:
> So technically it's possible. However, there is no notification to
> userspace when such a change may occur. There's also the issue that
> userspace may be in the process of issuing ethtool commands that are
> affecting one of the PHYs. While holding the rtnl lock will block
> those calls, a change between the PHY and e.g. a PHY on the SFP
> would cause the ethtool command to target a different PHY from what
> was the original target.
> 
> To solve that sanely, every PHY-based ethtool probably needs a way
> to specify which PHY the command is intended for, but then there's
> the question of how userspace users react to that - because it's
> likely more than just modifying the ethtool utility, ethtool
> commands are probably used from many programs.

Would it simplify anything if we only did the selection from ndo_open?
We can send a notification to user space that the SFP got plugged in,
but its up to user space to down / up the interface to use it?

Re: [RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[Andrew Lunn]

> To solve that sanely, every PHY-based ethtool probably needs a way
> to specify which PHY the command is intended for, but then there's
> the question of how userspace users react to that - because it's
> likely more than just modifying the ethtool utility, ethtool
> commands are probably used from many programs.

This idea of extending ethtool with a PHY ID has discussed last
year. It helps solve some of the problems discussed here. You can then
enumerate all the PHYs connected to a MAC, and operate on each PHY
independently.

https://lore.kernel.org/netdev/20221017105100.0cb33490@pc-8.home/

	Andrew

Re: [RFC] RJ45 to SFP auto-sensing and switching in mux-ed single-mac devices (XOR RJ/SFP)

[Maxime Chevallier]

Hello everyone,

On Mon, 4 Sep 2023 00:51:05 +0200
Andrew Lunn <andrew@lunn.ch> wrote:

> > To solve that sanely, every PHY-based ethtool probably needs a way
> > to specify which PHY the command is intended for, but then there's
> > the question of how userspace users react to that - because it's
> > likely more than just modifying the ethtool utility, ethtool
> > commands are probably used from many programs.  
> 
> This idea of extending ethtool with a PHY ID has discussed last
> year. It helps solve some of the problems discussed here. You can then
> enumerate all the PHYs connected to a MAC, and operate on each PHY
> independently.
> 
> https://lore.kernel.org/netdev/20221017105100.0cb33490@pc-8.home/

Indeed and I'm actively working on this right now, I have an RFC series
that'll be sent during the week, I'll make sure to CC everyone.

As stated this isn't an easy problem, my course of action is the
following to address this : 

 - First allow addressing individual PHYs attached to the same netdev,
without taking the mux into consideration for now. There are already
cases where several PHYs are attached to one MAC, which is when we have
a PHY between the MAC and SFP port, and a PHY in the SFP module.

As Russell said, there are some ethtool operations today that target
PHYs (cable testing, plca, but more importantly maybe timestamping).
With PHY addressing, we could imagine using the SFP's PHY for
timestamping.

The RFC will be strictly about that, adding the ability to list PHYs
(including th ones in SFP modules), get information on them, but the
main part really is about that id, that we can use in subsequent
commands. I'm also adding a netlink notification upon PHY
hotplugging/removal.

For the actual muxing my current idea is to better model the PHY port,
and allow userspace to pick which port to use (or auto-switch). The
reasonning is that there are a lot of topologies that lead to this
situation : 

 - Your case, with a real mux, switchign between 1 PHY and 1 SFP port :

                 /-- PHY -- RJ45 (8P8C)
   MAC --- mux --|
                 \-- SFP ( -- PHY ) -- RJ45/Fiber

 - PHYs that have an internal mux (the 88x3310 for example, or some
ports of the 88e6390X switch) :

                /-- RJ45
   MAC -- PHY --|
                \-- SFP -- RJ45/Fiber


 - Finally we have products in the wild using a pure-software mux :

        /-- PHY -- RJ45
  MAC --|
        \-- PHY -- RJ45

(muxing is done by putting one of the 2 PHYs in isolate mode).

I think for userspace, it would be better to directly configure which
front-facing port they want to see being used, and the current
representation of PORT_TP/PORT_FIBRE/etc. doesn't give enough details
about that. So I would add the phy_port enumeration (using the PHY id
previously introduced, each PHY would report the ports they have), then
muxing capability. I think we would have a pretty good overview
of the real topology at that point.

This is challenging, and will probably take quite some iterations to
get it right, as there are lots of things to consider, but hopefully 
as this subject is gaining traction (there are already a few people
interested in supporting such a thing) we can make it work.

Thanks,

Maxime


> 	Andrew
>