Is it currently possible to connect SFP to dsa port?

Is it currently possible to connect SFP to dsa port?

[Marek Behun]

Hello,

I have a question regarding DSA/SFP code.

We are working on a modular router where it is possible to connect
multiple switch cards and at most one SFP cage card to a router.

Does the sfp/dsa driver support connecting SFP port to a DSA port? For
example the Marvell 6390 switch has two SGMII ports and 8 1Gbps PHYs.
One can then connect those 8 1Gbps PHYs to RJ-45 connectors, the first
SGMII port to the CPU and the second SGMII port to SFP.

I think this is currently not possible, but am not sure.

Thanks.

Marek Behun

Re: Is it currently possible to connect SFP to dsa port?

[Florian Fainelli]

On 08/07/2018 12:00 PM, Marek Behun wrote:
> Hello,
> 
> I have a question regarding DSA/SFP code.
> 
> We are working on a modular router where it is possible to connect
> multiple switch cards and at most one SFP cage card to a router.
> 
> Does the sfp/dsa driver support connecting SFP port to a DSA port? For
> example the Marvell 6390 switch has two SGMII ports and 8 1Gbps PHYs.
> One can then connect those 8 1Gbps PHYs to RJ-45 connectors, the first
> SGMII port to the CPU and the second SGMII port to SFP.
> 
> I think this is currently not possible, but am not sure.

DSA recently gained support for PHYLINK which would allow you to do
that, please make sure your tree has this commit at least:

aab9c4067d2389d0adfc9c53806437df7b0fe3d5 ("net: dsa: Plug in PHYLINK
support").
-- 
Florian

Re: Is it currently possible to connect SFP to dsa port?

[Andrew Lunn]

On Tue, Aug 07, 2018 at 09:00:23PM +0200, Marek Behun wrote:
> Hello,
> 
> I have a question regarding DSA/SFP code.
> 
> We are working on a modular router where it is possible to connect
> multiple switch cards and at most one SFP cage card to a router.
> 
> Does the sfp/dsa driver support connecting SFP port to a DSA port? For
> example the Marvell 6390 switch has two SGMII ports and 8 1Gbps PHYs.
> One can then connect those 8 1Gbps PHYs to RJ-45 connectors, the first
> SGMII port to the CPU and the second SGMII port to SFP.

Hi Marek

I'm currently working on exactly this!

My primary aim is to allow SFP on ports 9 and 10. ports 2-7 can also
be connected to SFPs, but that is a bit harder, so it is going to take
me a little longer.

I hope i can post patches in the next few days.

  Andrew

Re: Is it currently possible to connect SFP to dsa port?

[Andrew Lunn]

On Tue, Aug 07, 2018 at 12:43:11PM -0700, Florian Fainelli wrote:
> 
> 
> On 08/07/2018 12:00 PM, Marek Behun wrote:
> > Hello,
> > 
> > I have a question regarding DSA/SFP code.
> > 
> > We are working on a modular router where it is possible to connect
> > multiple switch cards and at most one SFP cage card to a router.
> > 
> > Does the sfp/dsa driver support connecting SFP port to a DSA port? For
> > example the Marvell 6390 switch has two SGMII ports and 8 1Gbps PHYs.
> > One can then connect those 8 1Gbps PHYs to RJ-45 connectors, the first
> > SGMII port to the CPU and the second SGMII port to SFP.
> > 
> > I think this is currently not possible, but am not sure.
> 
> DSA recently gained support for PHYLINK which would allow you to do
> that, please make sure your tree has this commit at least:
> 
> aab9c4067d2389d0adfc9c53806437df7b0fe3d5 ("net: dsa: Plug in PHYLINK
> support").

Hi Florian

That gives enough to not break existing setups using the mv88e6xxx
driver. But to really use PHYLINK, more code is needed. I've taken
some of Russell Kings work, and merged that in, which gives us proper
support for validate, config_mac, etc. But Russell's code is missing
calls to phylink_mac_change, and handling SERDES interfaces, which are
needed for SFPs. I've got some code which is near, but it broke D in
DSA on my board :-(

    Andrew

Re: Is it currently possible to connect SFP to dsa port?

[Marek Behún]

On Tue, 7 Aug 2018 21:58:20 +0200
Andrew Lunn <andrew@lunn.ch> wrote:

> Hi Marek
> 
> I'm currently working on exactly this!
> 
> My primary aim is to allow SFP on ports 9 and 10. ports 2-7 can also
> be connected to SFPs, but that is a bit harder, so it is going to take
> me a little longer.
> 
> I hope i can post patches in the next few days.
> 
>   Andrew

Hi, this is cool :-) On our board it will be possible to connect
SFP extender board to port 10, and I was afraid it might have been me
to write the support for this :).

Let me know when you have the first version ready, I will try it. Our
router can connect SFP board directly to the CPU board, or there may up
to three mv88e6190 switch boards between SFP board and cpu board.

Do you already have a stable dts binding for how to bind sfp to dsa
port?

Btw: some SFP modules can operate in 2500BASE-X mode. Currently the SFP
driver does not support this, and there even isn't code in the
mainline kernel for mvneta to switch to 2500BASEX. On Armada 3720 this
has to be done by configuring comphy, which is currently done in
u-boot. As a side project I am working on supporting Armada 3720 comphy
in the kernel, so that it will be possible to set 2500BASE-X mode.

The mv88e6190/6390 can configure ports 9 and 10 to either
1000BASE-X/SGMII od 2500BASE-X, but this has to be done by setting a
pin and reseting the switch. Our board is able to configure this pin,
and the plan is to do this in u-boot when SFP board is detected to be
connected to the switch board. In this case the port is configured to
1000BASE-X.

I am wondering how complicated would it be to support changing these
modes for port 9 and 10 in mv88exxxx driver. There could be GPIO port
definitions in devicetree for P9_MODE and P10_MODE (and RESET), and
then the driver would be able to change these modes.

Since the switch has to be reseted to change the mode I think all the
dsa lan ports would have to be unused when doing this, correct?

And still how more complicated would it be if for example the RESET
gpio is shared between multiple switches, while the P9_MODE and
P10_MODE gpios are unique for each switch, like this:

    RESET-----+--------+--------+
              |        |        |
  +-----+  +--+--+  +--+--+  +--+--+  +-----+
  | cpu |==| sw1 |==| sw2 |==| sw3 |==| sfp |
  +-----+  +-+-+-+  +-+-+-+  +-+-+-+  +-----+
             | |      | |      | |
 P9_MODE[1]--+ |      | |      | |
P10_MODE[1]----+      | |      | |     == means SGMII/2500BASE-X
 P9_MODE[2]-----------+ |      | |            connection
P10_MODE[2]-------------+      | |
 P9_MODE[3]--------------------+ |
P10_MODE[3]----------------------+

Marek

Re: Is it currently possible to connect SFP to dsa port?

[Russell King - ARM Linux]

On Wed, Aug 08, 2018 at 03:00:13PM +0200, Marek Behún wrote:
> Btw: some SFP modules can operate in 2500BASE-X mode. Currently the SFP
> driver does not support this, and there even isn't code in the
> mainline kernel for mvneta to switch to 2500BASEX. On Armada 3720 this
> has to be done by configuring comphy, which is currently done in
> u-boot. As a side project I am working on supporting Armada 3720 comphy
> in the kernel, so that it will be possible to set 2500BASE-X mode.

With mvneta, it works the same way as other Marvell stuff - the comphy
needs to be told to up-clock the 1000base-X to 2500base-X.  That's all
that 2500base-X is - an up-clocked 1000base-X.

With Armada 388, it's just a matter of poking one register to configure
the comphy to 2500base-X.

As mvneta has no support for comphys, and I don't have the full details
for the comphy stuff, I've not bothered implementing anything for mvneta
other than some bare bones support in my repository to support the mode,
and resorting to poking the comphy register with devmem2 as required.

However, I can assure you that SFP definitely does support 2500base-X.
As 2500base-X is rather unofficial (there's no 802.3 standard for it)
there's no bits in SFP module eeprom that identify a module as
supporting this mode.

Consequently, the only way I'm aware of is to detect a Fibrechannel
SFP which supports the range of bitrates that would allow this mode
to work.  Currently, the code does this:

        /* For fibre channel SFP, derive possible BaseX modes */
        if (id->base.fc_speed_100 ||
            id->base.fc_speed_200 ||
            id->base.fc_speed_400) {
                if (id->base.br_nominal >= 31)
                        phylink_set(modes, 2500baseX_Full);
                if (id->base.br_nominal >= 12)
                        phylink_set(modes, 1000baseX_Full);
        }

So, the SFP core supports it.  What's expected from network drivers is
that they say they support 2500base-X, and then things get rather icky.
There is no way to autonegotiate between 2500base-X and 1000base-X.

What results is ultimately up to the driver implementation, but how I
have it locally via a local "phylink: add helper for configuring
2500BaseX modes" commit is:

- report 2500baseX_Full and 1000baseX_Full as supported to userspace.
- if both 2500baseX_Full and 1000baseX_Full are attempted to be set
   as advertised modes, clear 1000baseX_Full.
- select PHY_INTERFACE_MODE_2500BASEX or PHY_INTERFACE_MODE_1000BASEX
   depending whether the resulting advertising mask wants 2500baseX_Full
   or 1000baseX_Full.  If manual speed configuration, select between
   the PHY interface modes according to the requested speed.

With the exception of this commit (which is probably going to provoke
discussion about exactly what semantics we want) the SFP and phylink
code is all in place to support 2500base-X - I've proved it out using
a Macchiatobin (with the Marvell mvpp2x driver) and a Clearfog.

Without this patch, we detect in sfp_parse_support() that the module
is capable of 2500base-X, and, provided the network interface supports
2500base-X, sfp_select_interface() will select the 2500base-X PHY
interface mode, otherwise it'll select 1000base-X mode.

> The mv88e6190/6390 can configure ports 9 and 10 to either
> 1000BASE-X/SGMII od 2500BASE-X, but this has to be done by setting a
> pin and reseting the switch. Our board is able to configure this pin,
> and the plan is to do this in u-boot when SFP board is detected to be
> connected to the switch board. In this case the port is configured to
> 1000BASE-X.

It's really not that simple, unfortunately.  If you are talking about
pluggable SFPs (btw, SFP itself means the hot-pluggable modules, not
the soldered down modules, although we support both), SFP modules can
require *either* 1000base-X or SGMII.  There are some copper modules
with a PHY on board that want 1000base-X, but the vast majority of
copper modules want SGMII.  Conversely, gigabit fiber modules want
1000base-X.

Even more annoying, there is nothing that definitively tells you which
mode you should be using - and if you use the wrong mode, it may or
may not work, and you may or may not end up with a mismatched config
at either end of the link.

So, merely detecting a SFP modules presence doesn't tell you which
mode you should be in - and the annoying part is, if you hotplug the
module, you might need to change the port mode.

-- 
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps up
According to speedtest.net: 13Mbps down 490kbps up

Re: Is it currently possible to connect SFP to dsa port?

[Andrew Lunn]

> Let me know when you have the first version ready, I will try it. Our
> router can connect SFP board directly to the CPU board, or there may up
> to three mv88e6190 switch boards between SFP board and cpu board.

So you are using D in DSA? The board i'm using has two SFF modules,
both on the second switch in the chain. But D in DSA makes no
difference to SFPs.

> Do you already have a stable dts binding for how to bind sfp to dsa
> port?

Nothing special is needed. It just follows the standard SFP binding
for a MAC port. The DT changes for the board i'm using will be part of
the patchset i send.

> The mv88e6190/6390 can configure ports 9 and 10 to either
> 1000BASE-X/SGMII od 2500BASE-X, but this has to be done by setting a
> pin and reseting the switch.

I've been using 6390X, not 6390. The cmode nibble is writable for
ports 9 and 10. So you can change between 1000BASE-X, SGMII and
2500BASE-X by writing to the cmode. I assume 6390 is the same. As
Russell pointed out, you need this dynamic behaviour, since SFP
modules are hot pluggable, and you need to configure the mode
depending on what the module is. Also, what the peer is can also make
a difference. The board i have, the MAC/SFF combination can do
2500Base-X, by my peer cannot. Phylink defaults to 2500Base-X, so i
need to use ethtool to force it to 1000Base-X in order to get a link
with the peer.

     Andrew