Quirks for exotic SFP module

Quirks for exotic SFP module

[Lorenz Brun]

Hi netdev members,

I have some SFP modules which contain a G.fast modem (Metanoia MT5321). 
In my case I have ones without built-in flash, which means that they 
come up in bootloader mode. Their EEPROM is emulated by the onboard 
CPU/DSP and is pretty much completely incorrect, the claimed checksum 
is 0x00. Luckily there seems to be valid vendor and part number 
information to quirk off of.

I've implemented a detection mechanism analogous to the Cotsworks one, 
which catches my modules. Since the bootloader is in ROM, we sadly 
cannot overwrite the bad data, so I just made it skip the CRC check if 
this is an affected device and the expected CRC is zero.

There is also the issue of the module advertising 1000BASE-T:

 Identifier : 0x03 (SFP)
 Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
 Connector : 0x22 (RJ45)
 Transceiver codes : 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00
 Transceiver type : Ethernet: 1000BASE-T
 Encoding : 0x01 (8B/10B)
 BR, Nominal : 0MBd
 Rate identifier : 0x00 (unspecified)
 Length (SMF,km) : 0km
 Length (SMF) : 0m
 Length (50um) : 0m
 Length (62.5um) : 0m
 Length (Copper) : 0m
 Length (OM3) : 0m
 Laser wavelength : 0nm
 Vendor name : METANOIA
 Vendor OUI : 00:00:00
 Vendor PN : MT5321
 Vendor rev : 0001
 Option values : 0x08 0x00
 Option : Retimer or CDR implemented
 BR margin, max : 0%
 BR margin, min : 0%
 Vendor SN :
 Date code : ________

But the module internally has an AR8033 1000BASE-X to RGMII converter 
which is then connected to the modem SoC, so as far as I am aware this 
is incorrect and could cause Linux to do things like autonegotiation 
which definitely does not work here. So this probably needs to be 
quirked away. Anything else I have missed which needs to be changed?

Also, should the quirks and the workaround for the missing checksum be 
separate?

Thanks for your help!
Regards,
Lorenz

Re: Quirks for exotic SFP module

[Andrew Lunn]

On Fri, May 05, 2023 at 07:39:12PM +0200, Lorenz Brun wrote:
> Hi netdev members,

For SFP matters, please Cc: SFP maintainer, and the PHY
maintainers. See the MAINTAINERS file.

> I have some SFP modules which contain a G.fast modem (Metanoia MT5321). In
> my case I have ones without built-in flash, which means that they come up in
> bootloader mode. Their EEPROM is emulated by the onboard CPU/DSP and is
> pretty much completely incorrect, the claimed checksum is 0x00. Luckily
> there seems to be valid vendor and part number information to quirk off of.

It is amazing how many SFP manufactures cannot get the simple things
like CRC right. 

> I've implemented a detection mechanism analogous to the Cotsworks one, which
> catches my modules. Since the bootloader is in ROM, we sadly cannot
> overwrite the bad data, so I just made it skip the CRC check if this is an
> affected device and the expected CRC is zero.

Sounds sensible. Probably pointless, because SFP manufactures don't
seem to care about quality, but please do print an warning that the
bad checksum is being ignored.

> There is also the issue of the module advertising 1000BASE-T:

Probably something for Russell, but what should be advertised?
1000Base-X? 

> But the module internally has an AR8033 1000BASE-X to RGMII converter which
> is then connected to the modem SoC, so as far as I am aware this is
> incorrect and could cause Linux to do things like autonegotiation which
> definitely does not work here.

Is there anything useful to be gained by talking to the PHY? Since it
appears to be just a media converter, i guess the PHY having link is
not useful. Does the LOS GPIO tell you about the G.Fast modem status?

    Andrew

Re: Quirks for exotic SFP module

[Lorenz Brun]

Am Fr, 5. Mai 2023 um 20:53:43 +02:00:00 schrieb Andrew Lunn 
<andrew@lunn.ch>:
> On Fri, May 05, 2023 at 07:39:12PM +0200, Lorenz Brun wrote:
>>  Hi netdev members,
> 
> For SFP matters, please Cc: SFP maintainer, and the PHY
> maintainers. See the MAINTAINERS file.
Thanks, I'll check that next time.

> 
>>  I have some SFP modules which contain a G.fast modem (Metanoia 
>> MT5321). In
>>  my case I have ones without built-in flash, which means that they 
>> come up in
>>  bootloader mode. Their EEPROM is emulated by the onboard CPU/DSP 
>> and is
>>  pretty much completely incorrect, the claimed checksum is 0x00. 
>> Luckily
>>  there seems to be valid vendor and part number information to quirk 
>> off of.
> 
> It is amazing how many SFP manufactures cannot get the simple things
> like CRC right.
> 
>>  I've implemented a detection mechanism analogous to the Cotsworks 
>> one, which
>>  catches my modules. Since the bootloader is in ROM, we sadly cannot
>>  overwrite the bad data, so I just made it skip the CRC check if 
>> this is an
>>  affected device and the expected CRC is zero.
> 
> Sounds sensible. Probably pointless, because SFP manufactures don't
> seem to care about quality, but please do print an warning that the
> bad checksum is being ignored.
I'm doing that in my WIP patch already.

> 
>>  There is also the issue of the module advertising 1000BASE-T:
> 
> Probably something for Russell, but what should be advertised?
> 1000Base-X?
I'd have gone for 1000Base-X as well, but I'll let others weigh in. 
1000BASE-T is definitely wrong IMO as there is no twisted pair Ethernet 
layer anywhere (there is twisted pair, but it's G.fast). AFAIK 
1000Base-X should stop Linux from messing with anything as it does not 
have any non-EEPROM-based management, right?

> 
>>  But the module internally has an AR8033 1000BASE-X to RGMII 
>> converter which
>>  is then connected to the modem SoC, so as far as I am aware this is
>>  incorrect and could cause Linux to do things like autonegotiation 
>> which
>>  definitely does not work here.
> 
> Is there anything useful to be gained by talking to the PHY? Since it
> appears to be just a media converter, i guess the PHY having link is
> not useful. Does the LOS GPIO tell you about the G.Fast modem status?
AFAIK you cannot talk to the PHY as there isn't really an Ethernet PHY. 
The RGMII doesn't go to a PHY, it goes to the host interface which is a 
MAC of the modem SoC. Think of the modem SoC more of a computer then a 
plain modem.
While the SFP in its entirety behaves similar to a media converter the 
G.fast link is L2-agnostic and, even though it does in most cases carry 
Ethernet frames, any Ethernet managment interfaces (autoneg, ...) can't 
and shouldn't be extended to it.

I actually haven't checked the LOS GPIO. This thing runs ~1MiB of 
firmware and two different proprietary management protocols which I've 
reverse-engineered over which you can get tons of data about the 
current modem and link status. You need those to boot the SoC anyways. 
The TX disable GPIO puts the modem SoC into reset state and is used in 
case you use a host-based watchdog for the module.

Regards,
Lorenz

Re: Quirks for exotic SFP module

[Andrew Lunn]

> > 
> > >  But the module internally has an AR8033 1000BASE-X to RGMII
> > > converter which
> > >  is then connected to the modem SoC, so as far as I am aware this is
> > >  incorrect and could cause Linux to do things like autonegotiation
> > > which
> > >  definitely does not work here.
> > 
> > Is there anything useful to be gained by talking to the PHY? Since it
> > appears to be just a media converter, i guess the PHY having link is
> > not useful. Does the LOS GPIO tell you about the G.Fast modem status?

> AFAIK you cannot talk to the PHY as there isn't really an Ethernet PHY.

So i2c-detect does not find anything other than at address 0x50?

Often the PHY can be access via an MDIO bus over I2C at some other
address on the bus. The linux SFP code might be trying, even
succeeding, in instantiating such a bus and finding the PHY. And then
a PHY driver will be loaded to drive the PHY. This is how Copper SFP
modules work. However, most Copper SFP use a Marvell PHY, not
Atheros. And RollBall SFP use a different MDIO over i2c protocol.

> I actually haven't checked the LOS GPIO. This thing runs ~1MiB of firmware
> and two different proprietary management protocols which I've
> reverse-engineered over which you can get tons of data about the current
> modem and link status. You need those to boot the SoC anyways. The TX
> disable GPIO puts the modem SoC into reset state and is used in case you use
> a host-based watchdog for the module.

So i guess you are not passing the GPIO for TX disable in your DT
blob. And maybe not LOS. If you do, it must be doing something
sensible, because phylink does not allow the carrier to go up if LOS
is active. Although the EEPROM can indicate LOS is not
implemented. But that assumes the EEPROM contents are sane.

Russell King will be interested in a binary dump from ethtool -m.

   Andrew

Re: Quirks for exotic SFP module

[Lorenz Brun]

Am Sa, 6. Mai 2023 um 02:03:32 +02:00:00 schrieb Andrew Lunn 
<andrew@lunn.ch>:
>>  >
>>  > >  But the module internally has an AR8033 1000BASE-X to RGMII
>>  > > converter which
>>  > >  is then connected to the modem SoC, so as far as I am aware 
>> this is
>>  > >  incorrect and could cause Linux to do things like 
>> autonegotiation
>>  > > which
>>  > >  definitely does not work here.
>>  >
>>  > Is there anything useful to be gained by talking to the PHY? 
>> Since it
>>  > appears to be just a media converter, i guess the PHY having link 
>> is
>>  > not useful. Does the LOS GPIO tell you about the G.Fast modem 
>> status?
> 
>>  AFAIK you cannot talk to the PHY as there isn't really an Ethernet 
>> PHY.
> 
> So i2c-detect does not find anything other than at address 0x50?
> 
> Often the PHY can be access via an MDIO bus over I2C at some other
> address on the bus. The linux SFP code might be trying, even
> succeeding, in instantiating such a bus and finding the PHY. And then
> a PHY driver will be loaded to drive the PHY. This is how Copper SFP
> modules work. However, most Copper SFP use a Marvell PHY, not
> Atheros. And RollBall SFP use a different MDIO over i2c protocol.

I tested and I got a bunch of addresses showing up on i2c master 
connected to the module. 1b, 30, 31, 34, 35, 36, 50 and 53. But I'm 
still not sure why we'd want to talk MDIO with this module. AFAIK MDIO 
is an Ethernet thing, the module is talking G.fast to the outside which 
is a completely different protocol from a completely different family 
of protocols. It has its own management protocol which runs over 
Ethernet.

> 
>>  I actually haven't checked the LOS GPIO. This thing runs ~1MiB of 
>> firmware
>>  and two different proprietary management protocols which I've
>>  reverse-engineered over which you can get tons of data about the 
>> current
>>  modem and link status. You need those to boot the SoC anyways. The 
>> TX
>>  disable GPIO puts the modem SoC into reset state and is used in 
>> case you use
>>  a host-based watchdog for the module.
> 
> So i guess you are not passing the GPIO for TX disable in your DT
> blob. And maybe not LOS. If you do, it must be doing something
> sensible, because phylink does not allow the carrier to go up if LOS
> is active. Although the EEPROM can indicate LOS is not
> implemented. But that assumes the EEPROM contents are sane.

TX disable works, but in a quite drastic way by just putting the entire 
SoC into reset.
LOS is wired up, but I am currently not able to test its behavior.

> Russell King will be interested in a binary dump from ethtool -m.
The output of that command is already included in the top post.

OT but my messages to Russell King cannot be delivered
mx0.armlinux.org.uk: <lorenz@brun.one> is locally blocked. If this is 
incorrect, please contact the postmaster.

I haven't knowingly sent any messages to him before so I have no idea 
why I'd be blocked. My sender IP isn't on any public blacklist MultiRBL 
knows about. My DKIM/DMARC setup is also working.

Regards,
Lorenz
> 

Re: Quirks for exotic SFP module

[Andrew Lunn]

> I tested and I got a bunch of addresses showing up on i2c master connected
> to the module. 1b, 30, 31, 34, 35, 36, 50 and 53. But I'm still not sure why
> we'd want to talk MDIO with this module. AFAIK MDIO is an Ethernet thing,
> the module is talking G.fast to the outside which is a completely different
> protocol from a completely different family of protocols. It has its own
> management protocol which runs over Ethernet.

One reason you might want to talk to the PHY is to correct is
configuration. 1000Base-X includes inband signalling. There are some
Copper SFP which have the inband signalling disabled. And that can
make the host unhappy, it fails to link up. It varies from host to
host. Some work, some don't.

> OT but my messages to Russell King cannot be delivered
> mx0.armlinux.org.uk: <lorenz@brun.one> is locally blocked. If this is
> incorrect, please contact the postmaster.
> 
> I haven't knowingly sent any messages to him before so I have no idea why
> I'd be blocked. My sender IP isn't on any public blacklist MultiRBL knows
> about. My DKIM/DMARC setup is also working.

Russell, what do your mail logs say?

	 Andrew

Re: Quirks for exotic SFP module

[Lorenz Brun]

Am Sa, 6. Mai 2023 um 03:05:10 +02:00:00 schrieb Andrew Lunn 
<andrew@lunn.ch>:
>>  I tested and I got a bunch of addresses showing up on i2c master 
>> connected
>>  to the module. 1b, 30, 31, 34, 35, 36, 50 and 53. But I'm still not 
>> sure why
>>  we'd want to talk MDIO with this module. AFAIK MDIO is an Ethernet 
>> thing,
>>  the module is talking G.fast to the outside which is a completely 
>> different
>>  protocol from a completely different family of protocols. It has 
>> its own
>>  management protocol which runs over Ethernet.
> 
> One reason you might want to talk to the PHY is to correct is
> configuration. 1000Base-X includes inband signalling. There are some
> Copper SFP which have the inband signalling disabled. And that can
> make the host unhappy, it fails to link up. It varies from host to
> host. Some work, some don't.

Oh, so you're talking about signalling on the AR8033 <-> Linux Host 
part of the link. I actually wasn't aware that 1000Base-X did in-band 
signalling, TIL. Since the I2C bus is connected to the modem SoC it 
would have to forward any MDIO to the AR8033 transceiver, right? This 
would also be a bit weird as the AR8033 is connected "backwards", i.e. 
with RGMII facing towards the Modem SoC and 1000Base-X towards the 
Linux host.

Regards,
Lorenz

Re: Quirks for exotic SFP module

[Russell King (Oracle)]

On Fri, May 05, 2023 at 08:53:43PM +0200, Andrew Lunn wrote:
> On Fri, May 05, 2023 at 07:39:12PM +0200, Lorenz Brun wrote:
> > Hi netdev members,
> 
> For SFP matters, please Cc: SFP maintainer, and the PHY
> maintainers. See the MAINTAINERS file.

... and it doesn't help that $author is using a *.one vanity domain
(all the new TLDs that were announced a number of years back quickly
became brand new sources of nothing but spam, so I blocked them when
the first spams came through on the assumption that no one would be
silly enough to originate email from any of them.)

> > I have some SFP modules which contain a G.fast modem (Metanoia MT5321). In
> > my case I have ones without built-in flash, which means that they come up in
> > bootloader mode. Their EEPROM is emulated by the onboard CPU/DSP and is
> > pretty much completely incorrect, the claimed checksum is 0x00. Luckily
> > there seems to be valid vendor and part number information to quirk off of.
> 
> It is amazing how many SFP manufactures cannot get the simple things
> like CRC right. 
> 
> > I've implemented a detection mechanism analogous to the Cotsworks one, which
> > catches my modules. Since the bootloader is in ROM, we sadly cannot
> > overwrite the bad data, so I just made it skip the CRC check if this is an
> > affected device and the expected CRC is zero.
> 
> Sounds sensible. Probably pointless, because SFP manufactures don't
> seem to care about quality, but please do print an warning that the
> bad checksum is being ignored.
> 
> > There is also the issue of the module advertising 1000BASE-T:
> 
> Probably something for Russell, but what should be advertised?
> 1000Base-X? 
> 
> > But the module internally has an AR8033 1000BASE-X to RGMII converter which
> > is then connected to the modem SoC, so as far as I am aware this is
> > incorrect and could cause Linux to do things like autonegotiation which
> > definitely does not work here.
> 
> Is there anything useful to be gained by talking to the PHY? Since it
> appears to be just a media converter, i guess the PHY having link is
> not useful. Does the LOS GPIO tell you about the G.Fast modem status?

AR803x PHYs don't support I2C directly unlike the MV88E1111, and tend
not to be accessible. Only though additional hardware to convert I2C to
MDIO would it be accessible.

If it's programmed to use 1000base-X on the host side, then that's what
it will be using.

I'm not sure what the original author is talking about when referring to
autonegotiation - autonegotiation for _which_ part of the link?

There's 1000base-X autonegotiation, and then there's RGMII in-band
signalling that is sometimes called (imho incorrectly) autonegotiation.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: Quirks for exotic SFP module

[Russell King (Oracle)]

On Sat, May 06, 2023 at 02:03:32AM +0200, Andrew Lunn wrote:
> > > 
> > > >  But the module internally has an AR8033 1000BASE-X to RGMII
> > > > converter which
> > > >  is then connected to the modem SoC, so as far as I am aware this is
> > > >  incorrect and could cause Linux to do things like autonegotiation
> > > > which
> > > >  definitely does not work here.
> > > 
> > > Is there anything useful to be gained by talking to the PHY? Since it
> > > appears to be just a media converter, i guess the PHY having link is
> > > not useful. Does the LOS GPIO tell you about the G.Fast modem status?
> 
> > AFAIK you cannot talk to the PHY as there isn't really an Ethernet PHY.
> 
> So i2c-detect does not find anything other than at address 0x50?
> 
> Often the PHY can be access via an MDIO bus over I2C at some other
> address on the bus. The linux SFP code might be trying, even
> succeeding, in instantiating such a bus and finding the PHY. And then
> a PHY driver will be loaded to drive the PHY. This is how Copper SFP
> modules work. However, most Copper SFP use a Marvell PHY, not
> Atheros. And RollBall SFP use a different MDIO over i2c protocol.

Given that the PHY is in 1000base-X to RGMII mode, this is not a usual
setup, and its probably something the PHYLIB driver won't expect. So
we probably don't want to be talking to the PHY, and we probably just
want to talk 1000base-X to the module.

> > I actually haven't checked the LOS GPIO. This thing runs ~1MiB of firmware
> > and two different proprietary management protocols which I've
> > reverse-engineered over which you can get tons of data about the current
> > modem and link status. You need those to boot the SoC anyways. The TX
> > disable GPIO puts the modem SoC into reset state and is used in case you use
> > a host-based watchdog for the module.
> 
> So i guess you are not passing the GPIO for TX disable in your DT
> blob. And maybe not LOS. If you do, it must be doing something
> sensible, because phylink does not allow the carrier to go up if LOS
> is active. Although the EEPROM can indicate LOS is not
> implemented. But that assumes the EEPROM contents are sane.
> 
> Russell King will be interested in a binary dump from ethtool -m.

Definitely.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: Quirks for exotic SFP module

[Russell King (Oracle)]

On Sat, May 06, 2023 at 03:05:10AM +0200, Andrew Lunn wrote:
> > OT but my messages to Russell King cannot be delivered
> > mx0.armlinux.org.uk: <lorenz@brun.one> is locally blocked. If this is
> > incorrect, please contact the postmaster.
> > 
> > I haven't knowingly sent any messages to him before so I have no idea why
> > I'd be blocked. My sender IP isn't on any public blacklist MultiRBL knows
> > about. My DKIM/DMARC setup is also working.
> 
> Russell, what do your mail logs say?

Explained in my first reply. *.one was blocked because it was a source
of spam when first "announced".

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: Quirks for exotic SFP module

[Andrew Lunn]

> Oh, so you're talking about signalling on the AR8033 <-> Linux Host part of
> the link. I actually wasn't aware that 1000Base-X did in-band signalling,
> TIL. Since the I2C bus is connected to the modem SoC it would have to
> forward any MDIO to the AR8033 transceiver, right? This would also be a bit
> weird as the AR8033 is connected "backwards", i.e. with RGMII facing towards
> the Modem SoC and 1000Base-X towards the Linux host.

I2C allows for there to be multiple devices on the bus. So a PHY which
supports I2C could be placed on the bus along side the SoC
implementing the modem.

However, Russell indicated that the Atheros PHY does not have native
I2C. So it is unlikely to be on the bus. This is probably why the
Marvell PHY is used a lot, not many PHYs do have I2C.

	Andrew

Re: Quirks for exotic SFP module

[Russell King (Oracle)]

On Sat, May 06, 2023 at 03:35:42PM +0200, Andrew Lunn wrote:
> > Oh, so you're talking about signalling on the AR8033 <-> Linux Host part of
> > the link. I actually wasn't aware that 1000Base-X did in-band signalling,
> > TIL. Since the I2C bus is connected to the modem SoC it would have to
> > forward any MDIO to the AR8033 transceiver, right? This would also be a bit
> > weird as the AR8033 is connected "backwards", i.e. with RGMII facing towards
> > the Modem SoC and 1000Base-X towards the Linux host.

(To Lorenz)

It's not backwards at all. For a fibre link, 1000baseX is carried over
the fibre, and it looks like this:

Host MAC <==> Host PCS <==1000baseX==> Remote PCS <==> Remote MAC

The host has no access to the remote PCS.

In your case:

Host MAC <==> Host PCS <==1000baseX==> AR8033 <==RGMII==> SoC

How is this any different? I would say the AR8033 is up to the SoC to
manage itself. The fact that the SoC does something with the packets
to them stuff them out to the rest of the world is neither here nor
there. In the 1000base-X over fibre example above, the SoC could be
something designed for routing applications inside a network switch/
router.

Please don't get hung up on "there is a PHY on the module, I want
access to it!" As you're not talking twisted-pair ethernet, you
don't, there is nothing we need to control there.

The fact the module wants 1000base-X on its host interface is just
what it wants - and that it specifies that it offers 1000base-T
compliance is just the manufacturer being idiotic (as seems to be
the case with a lot of SFPs.)

Just add a quirk removing the 1000base-T capability, setting
1000base-X in the supported mask, and also clear the
PHY_INTERFACE_MODE_SGMII and set PHY_INTERFACE_MODE_1000BASEX in
the interfaces mask.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: Quirks for exotic SFP module

[Lorenz Brun]

Am Sa, 6. Mai 2023 um 14:54:55 +01:00:00 schrieb Russell King (Oracle) 
<linux@armlinux.org.uk>:
> It's not backwards at all. For a fibre link, 1000baseX is carried over
> the fibre, and it looks like this:
> 
> Host MAC <==> Host PCS <==1000baseX==> Remote PCS <==> Remote MAC
> 
> The host has no access to the remote PCS.
> 
> In your case:
> 
> Host MAC <==> Host PCS <==1000baseX==> AR8033 <==RGMII==> SoC
> 
> How is this any different? I would say the AR8033 is up to the SoC to
> manage itself. The fact that the SoC does something with the packets
> to them stuff them out to the rest of the world is neither here nor
> there. In the 1000base-X over fibre example above, the SoC could be
> something designed for routing applications inside a network switch/
> router.
> 
> Please don't get hung up on "there is a PHY on the module, I want
> access to it!" As you're not talking twisted-pair ethernet, you
> don't, there is nothing we need to control there.

That was my initial thought as well, Andrew told me to look for some 
interface to talk with a PHY.

> 
> The fact the module wants 1000base-X on its host interface is just
> what it wants - and that it specifies that it offers 1000base-T
> compliance is just the manufacturer being idiotic (as seems to be
> the case with a lot of SFPs.)
> 
> Just add a quirk removing the 1000base-T capability, setting
> 1000base-X in the supported mask, and also clear the
> PHY_INTERFACE_MODE_SGMII and set PHY_INTERFACE_MODE_1000BASEX in
> the interfaces mask.

Sounds good, I'll do that. Thanks for your help!

Regards,
Lorenz