Linux network PHY initial configuration for ports not 'up'

Linux network PHY initial configuration for ports not 'up'

[Tim Harvey]

Greetings,

What is the policy for configuration of network PHY's for ports that
are not brought 'up'?

I work with boards with several PHY's that have invalid link
configuration which does not get fixed until the port is brought up.
One could argue that this is fine because the port isn't up but in the
case of LED misconfiguration people wonder why the LED's are not
configured properly until the port is brought up (or they wonder why
LEDs are ilumnated at all for a port that isn't up). Another example
would be a PHY with EEE errata where EEE should be disabled but this
doesn't happen utnil the port is brought up yet while the port is
'down' a link with EEE is still established at the PHY level with a
link partner. One could also point out that power is being used to
link PHY's that should not even be linked.

In other words, should a MAC driver somehow trigger a PHY to get
initialized (as in fixups and allowing a physical link) even if the
MAC port is not up? If so, how is this done currently?

Best Regards,

Tim

Re: Linux network PHY initial configuration for ports not 'up'

[Florian Fainelli]

On 10/7/24 09:48, Tim Harvey wrote:
> Greetings,
> 
> What is the policy for configuration of network PHY's for ports that
> are not brought 'up'?
> 
> I work with boards with several PHY's that have invalid link
> configuration which does not get fixed until the port is brought up.
> One could argue that this is fine because the port isn't up but in the
> case of LED misconfiguration people wonder why the LED's are not
> configured properly until the port is brought up (or they wonder why
> LEDs are ilumnated at all for a port that isn't up). Another example
> would be a PHY with EEE errata where EEE should be disabled but this
> doesn't happen utnil the port is brought up yet while the port is
> 'down' a link with EEE is still established at the PHY level with a
> link partner. One could also point out that power is being used to
> link PHY's that should not even be linked.
> 
> In other words, should a MAC driver somehow trigger a PHY to get
> initialized (as in fixups and allowing a physical link) even if the
> MAC port is not up? If so, how is this done currently?

There are drivers that have historically brought up Ethernet PHYs in the 
MAC's probe routine. This is fine in premise, and you get a bit of speed 
up because by the time the network interface is opened by user-space you 
have usually finished auto-negotiation. This does mean that usually the 
PHY is already in the UP state.

The caveat with that approach is that it does not conserve power, and it 
assumes that the network device will end-up being used shortly 
thereafter, which is not a given.

For LEDs, I would argue that if you care about having some sensible 
feedback, the place where this belongs is the boot loader, because you 
can address any kernel short comings there: lack of a kernel driver for 
said PHY/MAC, network never being brought up, etc.

For errata like EEE, it seems fine to address that at link up time.
-- 
Florian

Re: Linux network PHY initial configuration for ports not 'up'

[Tim Harvey]

On Mon, Oct 7, 2024 at 10:28 AM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
> On 10/7/24 09:48, Tim Harvey wrote:
> > Greetings,
> >
> > What is the policy for configuration of network PHY's for ports that
> > are not brought 'up'?
> >
> > I work with boards with several PHY's that have invalid link
> > configuration which does not get fixed until the port is brought up.
> > One could argue that this is fine because the port isn't up but in the
> > case of LED misconfiguration people wonder why the LED's are not
> > configured properly until the port is brought up (or they wonder why
> > LEDs are ilumnated at all for a port that isn't up). Another example
> > would be a PHY with EEE errata where EEE should be disabled but this
> > doesn't happen utnil the port is brought up yet while the port is
> > 'down' a link with EEE is still established at the PHY level with a
> > link partner. One could also point out that power is being used to
> > link PHY's that should not even be linked.
> >
> > In other words, should a MAC driver somehow trigger a PHY to get
> > initialized (as in fixups and allowing a physical link) even if the
> > MAC port is not up? If so, how is this done currently?
>
> There are drivers that have historically brought up Ethernet PHYs in the
> MAC's probe routine. This is fine in premise, and you get a bit of speed
> up because by the time the network interface is opened by user-space you
> have usually finished auto-negotiation. This does mean that usually the
> PHY is already in the UP state.

Hi Florian,

Can you point me to an example of a driver that does 'not' do this? I
can not find an example where the PHY isn't UP regardless of the MAC
state (maybe I'm biased due to the boards I've been working with most
in the last couple of years) but then again its not because the MAC
driver brought the PHY up, its because it doesn't take it down and it
was up on power-up.

Some examples that I just looked at where if your OS does not bring up
the MAC the PHY is still UP
- imx8m FEC with DP83867 PHY
- KSZ9897S (ksz9447) switch/phy

>
> The caveat with that approach is that it does not conserve power, and it
> assumes that the network device will end-up being used shortly
> thereafter, which is not a given.

agreed... it seems wrong from a power perspective to have those PHY's
up. I recall not to many years ago when a Gbit PHY link cost 1W... and
I think we are currently way worse than that for a 10Gbps PHY link.

Then again think of the case where you have a switch with ports
unconfigured yet connected to a partner and all the LED's are lit up
(giving the impression visually that the ports are up).

>
> For LEDs, I would argue that if you care about having some sensible
> feedback, the place where this belongs is the boot loader, because you
> can address any kernel short comings there: lack of a kernel driver for
> said PHY/MAC, network never being brought up, etc.

I agree that boot firmware can and perhaps should do this but often
the PHY config that is done in the boot loader gets undone in the
Linux PHY driver if the reset pin is exposed to the Linux or in some
cases by soft reset done in the Linux PHY driver, or in other cases
blatant re-configuration of LED's in the Linux PHY driver without
using DT properties (intel-xway.c does this).

>
> For errata like EEE, it seems fine to address that at link up time.

one would think that makes sense as well but the case I just ran into
was where a KSZ9897S switch had a network cable to a link partner and
the link partner would 'flap' with its link giong up and down due to
EEE errata until the KSZ9897S port was brought up which disabled EEE.
In this specific case EEE could have been disabled in U-Boot but that
would also require some changes as U-Boot does the same thing as Linux
currently - it only configures PHY's that are active.

Best Regards,

Tim

Re: Linux network PHY initial configuration for ports not 'up'

[Florian Fainelli]

On 10/7/24 11:18, Tim Harvey wrote:
> On Mon, Oct 7, 2024 at 10:28 AM Florian Fainelli <f.fainelli@gmail.com> wrote:
>>
>> On 10/7/24 09:48, Tim Harvey wrote:
>>> Greetings,
>>>
>>> What is the policy for configuration of network PHY's for ports that
>>> are not brought 'up'?
>>>
>>> I work with boards with several PHY's that have invalid link
>>> configuration which does not get fixed until the port is brought up.
>>> One could argue that this is fine because the port isn't up but in the
>>> case of LED misconfiguration people wonder why the LED's are not
>>> configured properly until the port is brought up (or they wonder why
>>> LEDs are ilumnated at all for a port that isn't up). Another example
>>> would be a PHY with EEE errata where EEE should be disabled but this
>>> doesn't happen utnil the port is brought up yet while the port is
>>> 'down' a link with EEE is still established at the PHY level with a
>>> link partner. One could also point out that power is being used to
>>> link PHY's that should not even be linked.
>>>
>>> In other words, should a MAC driver somehow trigger a PHY to get
>>> initialized (as in fixups and allowing a physical link) even if the
>>> MAC port is not up? If so, how is this done currently?
>>
>> There are drivers that have historically brought up Ethernet PHYs in the
>> MAC's probe routine. This is fine in premise, and you get a bit of speed
>> up because by the time the network interface is opened by user-space you
>> have usually finished auto-negotiation. This does mean that usually the
>> PHY is already in the UP state.
> 
> Hi Florian,
> 
> Can you point me to an example of a driver that does 'not' do this? I
> can not find an example where the PHY isn't UP regardless of the MAC
> state (maybe I'm biased due to the boards I've been working with most
> in the last couple of years) but then again its not because the MAC
> driver brought the PHY up, its because it doesn't take it down and it
> was up on power-up.

Essentially any Ethernet MAC driver that calls phy_connect() in their 
.probe() routine would be doing this. bgmac.c is one such example, most, 
if not all of the time it deals with fixed-link PHYs because it is the 
Ethernet controller used with integrated switches, though occasionally 
there might an external PHY connected to it. There are certainly more 
examples.

> 
> Some examples that I just looked at where if your OS does not bring up
> the MAC the PHY is still UP
> - imx8m FEC with DP83867 PHY
> - KSZ9897S (ksz9447) switch/phy
> 
>>
>> The caveat with that approach is that it does not conserve power, and it
>> assumes that the network device will end-up being used shortly
>> thereafter, which is not a given.
> 
> agreed... it seems wrong from a power perspective to have those PHY's
> up. I recall not to many years ago when a Gbit PHY link cost 1W... and
> I think we are currently way worse than that for a 10Gbps PHY link.

Quite likely, yes.

> 
> Then again think of the case where you have a switch with ports
> unconfigured yet connected to a partner and all the LED's are lit up
> (giving the impression visually that the ports are up).
> 
>>
>> For LEDs, I would argue that if you care about having some sensible
>> feedback, the place where this belongs is the boot loader, because you
>> can address any kernel short comings there: lack of a kernel driver for
>> said PHY/MAC, network never being brought up, etc.
> 
> I agree that boot firmware can and perhaps should do this but often
> the PHY config that is done in the boot loader gets undone in the
> Linux PHY driver if the reset pin is exposed to the Linux or in some
> cases by soft reset done in the Linux PHY driver, or in other cases
> blatant re-configuration of LED's in the Linux PHY driver without
> using DT properties (intel-xway.c does this).

Unfortunately if you care about consistency or independence between the 
boot stages, you have to duplicate things a tiny bit, for the reasons I 
mentioned that while you might bring-up networking in u-boot, you may 
not in Linux, or vice versa.

It's all wonderful if you can come to an agreement as to what belongs to 
the boot loader configuration and what belongs to the OS configuration, 
but in practice there is quite a bit of overlap due to each one being 
somewhat independent. I don't think there is a hard and fast set of 
rules, because all of this is inherently PHY specific, but there should 
be some general consistency that applies, starting with LEDs. Best if 
you can just HW strap though...

> 
>>
>> For errata like EEE, it seems fine to address that at link up time.
> 
> one would think that makes sense as well but the case I just ran into
> was where a KSZ9897S switch had a network cable to a link partner and
> the link partner would 'flap' with its link giong up and down due to
> EEE errata until the KSZ9897S port was brought up which disabled EEE.
> In this specific case EEE could have been disabled in U-Boot but that
> would also require some changes as U-Boot does the same thing as Linux
> currently - it only configures PHY's that are active.

OK, but unfortunately I don't see how you can avoid not making those 
changes in u-boot.
-- 
Florian

Re: Linux network PHY initial configuration for ports not 'up'

[Tim Harvey]

On Mon, Oct 7, 2024 at 11:35 AM Florian Fainelli <f.fainelli@gmail.com> wrote:
>
> On 10/7/24 11:18, Tim Harvey wrote:
> > On Mon, Oct 7, 2024 at 10:28 AM Florian Fainelli <f.fainelli@gmail.com> wrote:
> >>
> >> On 10/7/24 09:48, Tim Harvey wrote:
> >>> Greetings,
> >>>
> >>> What is the policy for configuration of network PHY's for ports that
> >>> are not brought 'up'?
> >>>
> >>> I work with boards with several PHY's that have invalid link
> >>> configuration which does not get fixed until the port is brought up.
> >>> One could argue that this is fine because the port isn't up but in the
> >>> case of LED misconfiguration people wonder why the LED's are not
> >>> configured properly until the port is brought up (or they wonder why
> >>> LEDs are ilumnated at all for a port that isn't up). Another example
> >>> would be a PHY with EEE errata where EEE should be disabled but this
> >>> doesn't happen utnil the port is brought up yet while the port is
> >>> 'down' a link with EEE is still established at the PHY level with a
> >>> link partner. One could also point out that power is being used to
> >>> link PHY's that should not even be linked.
> >>>
> >>> In other words, should a MAC driver somehow trigger a PHY to get
> >>> initialized (as in fixups and allowing a physical link) even if the
> >>> MAC port is not up? If so, how is this done currently?
> >>
> >> There are drivers that have historically brought up Ethernet PHYs in the
> >> MAC's probe routine. This is fine in premise, and you get a bit of speed
> >> up because by the time the network interface is opened by user-space you
> >> have usually finished auto-negotiation. This does mean that usually the
> >> PHY is already in the UP state.
> >
> > Hi Florian,
> >
> > Can you point me to an example of a driver that does 'not' do this? I
> > can not find an example where the PHY isn't UP regardless of the MAC
> > state (maybe I'm biased due to the boards I've been working with most
> > in the last couple of years) but then again its not because the MAC
> > driver brought the PHY up, its because it doesn't take it down and it
> > was up on power-up.
>
> Essentially any Ethernet MAC driver that calls phy_connect() in their
> .probe() routine would be doing this. bgmac.c is one such example, most,
> if not all of the time it deals with fixed-link PHYs because it is the
> Ethernet controller used with integrated switches, though occasionally
> there might an external PHY connected to it. There are certainly more
> examples.
>
> >
> > Some examples that I just looked at where if your OS does not bring up
> > the MAC the PHY is still UP
> > - imx8m FEC with DP83867 PHY
> > - KSZ9897S (ksz9447) switch/phy
> >
> >>
> >> The caveat with that approach is that it does not conserve power, and it
> >> assumes that the network device will end-up being used shortly
> >> thereafter, which is not a given.
> >
> > agreed... it seems wrong from a power perspective to have those PHY's
> > up. I recall not to many years ago when a Gbit PHY link cost 1W... and
> > I think we are currently way worse than that for a 10Gbps PHY link.
>
> Quite likely, yes.
>
> >
> > Then again think of the case where you have a switch with ports
> > unconfigured yet connected to a partner and all the LED's are lit up
> > (giving the impression visually that the ports are up).
> >
> >>
> >> For LEDs, I would argue that if you care about having some sensible
> >> feedback, the place where this belongs is the boot loader, because you
> >> can address any kernel short comings there: lack of a kernel driver for
> >> said PHY/MAC, network never being brought up, etc.
> >
> > I agree that boot firmware can and perhaps should do this but often
> > the PHY config that is done in the boot loader gets undone in the
> > Linux PHY driver if the reset pin is exposed to the Linux or in some
> > cases by soft reset done in the Linux PHY driver, or in other cases
> > blatant re-configuration of LED's in the Linux PHY driver without
> > using DT properties (intel-xway.c does this).
>
> Unfortunately if you care about consistency or independence between the
> boot stages, you have to duplicate things a tiny bit, for the reasons I
> mentioned that while you might bring-up networking in u-boot, you may
> not in Linux, or vice versa.
>
> It's all wonderful if you can come to an agreement as to what belongs to
> the boot loader configuration and what belongs to the OS configuration,
> but in practice there is quite a bit of overlap due to each one being
> somewhat independent. I don't think there is a hard and fast set of
> rules, because all of this is inherently PHY specific, but there should
> be some general consistency that applies, starting with LEDs. Best if
> you can just HW strap though...
>
> >
> >>
> >> For errata like EEE, it seems fine to address that at link up time.
> >
> > one would think that makes sense as well but the case I just ran into
> > was where a KSZ9897S switch had a network cable to a link partner and
> > the link partner would 'flap' with its link giong up and down due to
> > EEE errata until the KSZ9897S port was brought up which disabled EEE.
> > In this specific case EEE could have been disabled in U-Boot but that
> > would also require some changes as U-Boot does the same thing as Linux
> > currently - it only configures PHY's that are active.
>
> OK, but unfortunately I don't see how you can avoid not making those
> changes in u-boot.

So you are thinking the right place to address this is in boot
firmware and the network switch init there should go and disable all
PHY's by default then? I'm good with that approach and can code that
up.

But then based on the point that Linux and boot loader should be doing
the same things and be independent from each other I would think the
kernel should do this as well.

Best Regards,

Tim

Re: Linux network PHY initial configuration for ports not 'up'

[Andrew Lunn]

> So you are thinking the right place to address this is in boot
> firmware and the network switch init there should go and disable all
> PHY's by default then? I'm good with that approach and can code that
> up.

I've seen some use cases where the link should be up as fast as
possible. If you force the link down until it is configured up, you
can be adding a little over 1 second delay for autoneg to
complete. I've even see cases there u-boot kicks off autoneg, before
jumping into Linux so that by the time Linux userspace is running and
admin up the interface, autoneg has completed and the link is usable.

There are diverse requirements here and any changes somebody wants to
make is likely to cause a regression for somebody.

	Andrew