PHY vs. MAC ethtool Wake-on-LAN selection

PHY vs. MAC ethtool Wake-on-LAN selection

[Florian Fainelli]

Hi all,

I would like to request your feedback on implementing a solution so that 
we can properly deal with choosing PHY or MAC Wake-on-LAN configuration.

The typical use case that I am after is the following: an Android TV box 
which is asleep as often as possible and needs to wake-up when a local 
network device wants to "cast" to the box. This happens when your phone 
does a mDNS query towards a multicast IPv4 address and searches for a 
particular service.

Now, consider the existing system's capabilities:

- system supports both "standby" (as written in /sys/power/state) with 
all the blocks powered on and "mem" with only a subset of the SoC 
powered on (a small always on island and supply)

- Ethernet MAC (bcmgenet) is capable of doing Wake-on-LAN using Magic 
Packets (g) with password (s) or network filters (f) and is powered on 
in the "standby" (as written in /sys/power/state) suspend state, and 
completely powered off (by hardware) in the "mem" state

- Ethernet PHY (broadcom.c, no code there to support WoL yet) is capable 
of doing Wake-on-LAN using Magic Packets (g) with password (s) or a 
48-bit MAC destination address (f) match allowing us to match on say, 
Broadcom and Multicast. That PHY is on during both the "standby" and 
"mem" suspend states

The network filtering capability of the Ethernet MAC in the "standby" 
state is superior to that of the Ethernet PHY and would allow for finer 
filtering of the network packets, so it should be preferred when the 
standby state is "standby" so we can limit spurious wake-ups on 
multicast traffic and specifically that not matching the desired 
service. There may also be capability to match on all of "gsf" instead 
of just "g" or "s" so it is preferred.

When the standby state is "mem" however we would want to use the PHY 
because that is the only one in the always-on island that can be active, 
even if it has coarse filtering capabilities.

Since bd8c9ba3b1e2037af5af4e48aea1087212838179 ("PM / suspend: Export 
pm_suspend_target_state") drivers do have the ability to determine which 
suspend state we are about to enter at ->suspend() time, so with the 
knowledge about the system as to which of the MAC or the PHY will remain 
on (using appropriate Device Tree properties for instance: always-on) 
and service Wake-on-LAN, a driver could make an appropriate decision as 
to whether it wants to program the MAC or the PHY with the Wake-on-LAN 
configuration.

The programming of the Wake-on-LAN is typically done at ->suspend() time 
and not necessarily at the time the user is requesting it, and at the 
time the user configures Wake-on-LAN we do not know yet the target 
suspend state.

This is a problem that could be punted to user-space in that it controls 
which suspend mode the system will enter. We could therefore assume that 
user space should know which Wake-on-LAN configuration to apply, even if 
that could mean "double" programming of both the MAC and PHY, knowing 
that the MAC will be off so the PHY will take over. The problem I see 
with that is that approach:

- you must always toggle between Wake-on-LAN programming depending upon 
the system standby mode which may not always be practical

- the behavior can vary wildly between platforms depending upon 
capabilities of the drivers and their bugs^w implementation

- we are still missing the ability to install a specific Wake-on-LAN 
configuration towards the desired MAC or PHY entity

The few drivers that call phy_ethtool_{set,get}_wol() except lan743x do 
not actually support Wake-on-LAN at the MAC level, so that is an easy 
decision to make for them because it is the only way they can support 
Wake-on-LAN.

What I envision we could do is add a ETHTOOL_A_WOL_DEVICE u8 field and 
have it take the values: 0 (default), 1 (MAC), 2 (PHY), 3 (both) and you 
would do the following on the command line:

ethtool -s eth0 wol g # default/existing mode, leave it to the driver
ethtool -s eth0 wol g target mac # target the MAC only
ethtool -s eth0 wol g target phy # target the PHY only
ethtool -s eth0 wol g target mac+phy # target both MAC and PHY

Is that over engineering or do you see other platforms possibly needing 
that distinction? Heiner, how about r8169, are there similar constraints 
with respect to which part of the controller is on/off during S2, S3 or S5?

Thanks!
-- 
Florian

Re: PHY vs. MAC ethtool Wake-on-LAN selection

[Andrew Lunn]

> - Ethernet MAC (bcmgenet) is capable of doing Wake-on-LAN using Magic
> Packets (g) with password (s) or network filters (f) and is powered on in
> the "standby" (as written in /sys/power/state) suspend state, and completely
> powered off (by hardware) in the "mem" state
> 
> - Ethernet PHY (broadcom.c, no code there to support WoL yet) is capable of
> doing Wake-on-LAN using Magic Packets (g) with password (s) or a 48-bit MAC
> destination address (f) match allowing us to match on say, Broadcom and
> Multicast. That PHY is on during both the "standby" and "mem" suspend states

Marvell systems are similar. The mvneta hardware has support for WOL,
and has quite a capable filter. But there is no driver support. WOL is
simply forwarded to the PHY.

> What I envision we could do is add a ETHTOOL_A_WOL_DEVICE u8 field and have
> it take the values: 0 (default), 1 (MAC), 2 (PHY), 3 (both) and you would do
> the following on the command line:
> 
> ethtool -s eth0 wol g # default/existing mode, leave it to the driver
> ethtool -s eth0 wol g target mac # target the MAC only
> ethtool -s eth0 wol g target phy # target the PHY only
> ethtool -s eth0 wol g target mac+phy # target both MAC and PHY

This API seems like a start, but is it going to be limiting? It does
not appear you can say:

ethtool -s eth0 wol g target phy wol f target mac

So make use of magic packet in the PHY and filtering in the MAC.
ETHTOOL_A_WOL_DEVICE u8 appears to apply to all WoL options, not one
u8 per option.

And does mac+phy mean both will generate an interrupt? I'm assuming
the default of 0 means do whatever undefined behaviour we have now. Do
we need another value, 4 (auto) and the MAC driver will first try to
offload to the PHY, and if that fails, it does it at the MAC, with the
potential for some options to be in the MAC and some in the PHY?

> Is that over engineering or do you see other platforms possibly needing that
> distinction?

The over engineering question is clearly valid. Do we actually need to
support all possible options? I've made little use of WoL, so i don't
think i can answer that question.

Also, when it comes to implementation, i would suggest trying to pull
the interpretation of target and decision for MAC or PHY out into
helpers. The complexity is quite high, and if we want uniform
implementation, we want one implementation of it.

	Andrew

Re: PHY vs. MAC ethtool Wake-on-LAN selection

[Russell King (Oracle)]

On Sun, Jun 27, 2021 at 07:06:45PM +0200, Andrew Lunn wrote:
> > - Ethernet MAC (bcmgenet) is capable of doing Wake-on-LAN using Magic
> > Packets (g) with password (s) or network filters (f) and is powered on in
> > the "standby" (as written in /sys/power/state) suspend state, and completely
> > powered off (by hardware) in the "mem" state
> > 
> > - Ethernet PHY (broadcom.c, no code there to support WoL yet) is capable of
> > doing Wake-on-LAN using Magic Packets (g) with password (s) or a 48-bit MAC
> > destination address (f) match allowing us to match on say, Broadcom and
> > Multicast. That PHY is on during both the "standby" and "mem" suspend states
> 
> Marvell systems are similar. The mvneta hardware has support for WOL,
> and has quite a capable filter. But there is no driver support. WOL is
> simply forwarded to the PHY.
> 
> > What I envision we could do is add a ETHTOOL_A_WOL_DEVICE u8 field and have
> > it take the values: 0 (default), 1 (MAC), 2 (PHY), 3 (both) and you would do
> > the following on the command line:
> > 
> > ethtool -s eth0 wol g # default/existing mode, leave it to the driver
> > ethtool -s eth0 wol g target mac # target the MAC only
> > ethtool -s eth0 wol g target phy # target the PHY only
> > ethtool -s eth0 wol g target mac+phy # target both MAC and PHY
> 
> This API seems like a start, but is it going to be limiting? It does
> not appear you can say:
> 
> ethtool -s eth0 wol g target phy wol f target mac
> 
> So make use of magic packet in the PHY and filtering in the MAC.
> ETHTOOL_A_WOL_DEVICE u8 appears to apply to all WoL options, not one
> u8 per option.
> 
> And does mac+phy mean both will generate an interrupt? I'm assuming
> the default of 0 means do whatever undefined behaviour we have now. Do
> we need another value, 4 (auto) and the MAC driver will first try to
> offload to the PHY, and if that fails, it does it at the MAC, with the
> potential for some options to be in the MAC and some in the PHY?

Another question concerns the capabilities of the MAC and PHY in each
low power mode. Consider that userspace wishes to program the system
to wakeup when a certain packet is received. How does it know whether
it needs to program that into the MAC or the PHY or both?

Should that level of detail be available to userspace, or kept within
the driver?

For example, if userspace requests destination MAC address wakeup, then
shouldn't the driver be making the decision about which of the MAC or
PHY gets programmed to cause the wakeup depending on which mode the
system will be switching to and whether the appropriate blocks can be
left powered?

Another question would be - if the PHY can only do magic packet and
remains powered, and the MAC can only do destination MAC but is powered
down in the "mem" state, what do we advertise to the user. If the user
selects destination MAC and then requests the system enter "mem" state,
then what? Should we try to do the best we can?

Should we at the very least be advertising which WOL modes are
supported in each power state?

Sorry for adding to the questions...

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: PHY vs. MAC ethtool Wake-on-LAN selection

[Florian Fainelli]

On 6/27/2021 12:09 PM, Russell King (Oracle) wrote:
> On Sun, Jun 27, 2021 at 07:06:45PM +0200, Andrew Lunn wrote:
>>> - Ethernet MAC (bcmgenet) is capable of doing Wake-on-LAN using Magic
>>> Packets (g) with password (s) or network filters (f) and is powered on in
>>> the "standby" (as written in /sys/power/state) suspend state, and completely
>>> powered off (by hardware) in the "mem" state
>>>
>>> - Ethernet PHY (broadcom.c, no code there to support WoL yet) is capable of
>>> doing Wake-on-LAN using Magic Packets (g) with password (s) or a 48-bit MAC
>>> destination address (f) match allowing us to match on say, Broadcom and
>>> Multicast. That PHY is on during both the "standby" and "mem" suspend states
>>
>> Marvell systems are similar. The mvneta hardware has support for WOL,
>> and has quite a capable filter. But there is no driver support. WOL is
>> simply forwarded to the PHY.
>>
>>> What I envision we could do is add a ETHTOOL_A_WOL_DEVICE u8 field and have
>>> it take the values: 0 (default), 1 (MAC), 2 (PHY), 3 (both) and you would do
>>> the following on the command line:
>>>
>>> ethtool -s eth0 wol g # default/existing mode, leave it to the driver
>>> ethtool -s eth0 wol g target mac # target the MAC only
>>> ethtool -s eth0 wol g target phy # target the PHY only
>>> ethtool -s eth0 wol g target mac+phy # target both MAC and PHY
>>
>> This API seems like a start, but is it going to be limiting? It does
>> not appear you can say:
>>
>> ethtool -s eth0 wol g target phy wol f target mac
>>
>> So make use of magic packet in the PHY and filtering in the MAC.
>> ETHTOOL_A_WOL_DEVICE u8 appears to apply to all WoL options, not one
>> u8 per option.
>>
>> And does mac+phy mean both will generate an interrupt? I'm assuming
>> the default of 0 means do whatever undefined behaviour we have now. Do
>> we need another value, 4 (auto) and the MAC driver will first try to
>> offload to the PHY, and if that fails, it does it at the MAC, with the
>> potential for some options to be in the MAC and some in the PHY?
> 
> Another question concerns the capabilities of the MAC and PHY in each
> low power mode. Consider that userspace wishes to program the system
> to wakeup when a certain packet is received. How does it know whether
> it needs to program that into the MAC or the PHY or both?

There is no way right now to know other than just having user-space be 
customized to the desired platform which is something that works 
reasonably well for Android, not so much for other distros.

> 
> Should that level of detail be available to userspace, or kept within
> the driver?
> 
> For example, if userspace requests destination MAC address wakeup, then
> shouldn't the driver be making the decision about which of the MAC or
> PHY gets programmed to cause the wakeup depending on which mode the
> system will be switching to and whether the appropriate blocks can be
> left powered?
> 
> Another question would be - if the PHY can only do magic packet and
> remains powered, and the MAC can only do destination MAC but is powered
> down in the "mem" state, what do we advertise to the user. If the user
> selects destination MAC and then requests the system enter "mem" state,
> then what? Should we try to do the best we can?

This is the part where it may be reasonable to lean on to user-space to 
program either the MAC or the PHY in a way that makes sense to support a 
Wake-on-LAN scheme, whether that means that ethtool should also report 
which modes are supported depending on the target system suspend such 
that user-space has information to make an appropriate decision may just 
be the next step.

> 
> Should we at the very least be advertising which WOL modes are
> supported in each power state?

It would make sense to do that, I do wonder if the reporting may be more 
complicated in case there are device-specific power domains that we need 
to be aware of, instead of just a report per PM_SUSPEND_* mode defined 
in include/linux/suspend.h.
-- 
Florian

Re: PHY vs. MAC ethtool Wake-on-LAN selection

[Heiner Kallweit]

On 27.06.2021 05:53, Florian Fainelli wrote:
> Hi all,
> 
> I would like to request your feedback on implementing a solution so that we can properly deal with choosing PHY or MAC Wake-on-LAN configuration.
> 
> The typical use case that I am after is the following: an Android TV box which is asleep as often as possible and needs to wake-up when a local network device wants to "cast" to the box. This happens when your phone does a mDNS query towards a multicast IPv4 address and searches for a particular service.
> 
> Now, consider the existing system's capabilities:
> 
> - system supports both "standby" (as written in /sys/power/state) with all the blocks powered on and "mem" with only a subset of the SoC powered on (a small always on island and supply)
> 
> - Ethernet MAC (bcmgenet) is capable of doing Wake-on-LAN using Magic Packets (g) with password (s) or network filters (f) and is powered on in the "standby" (as written in /sys/power/state) suspend state, and completely powered off (by hardware) in the "mem" state
> 
> - Ethernet PHY (broadcom.c, no code there to support WoL yet) is capable of doing Wake-on-LAN using Magic Packets (g) with password (s) or a 48-bit MAC destination address (f) match allowing us to match on say, Broadcom and Multicast. That PHY is on during both the "standby" and "mem" suspend states
> 
> The network filtering capability of the Ethernet MAC in the "standby" state is superior to that of the Ethernet PHY and would allow for finer filtering of the network packets, so it should be preferred when the standby state is "standby" so we can limit spurious wake-ups on multicast traffic and specifically that not matching the desired service. There may also be capability to match on all of "gsf" instead of just "g" or "s" so it is preferred.
> 
> When the standby state is "mem" however we would want to use the PHY because that is the only one in the always-on island that can be active, even if it has coarse filtering capabilities.
> 
> Since bd8c9ba3b1e2037af5af4e48aea1087212838179 ("PM / suspend: Export pm_suspend_target_state") drivers do have the ability to determine which suspend state we are about to enter at ->suspend() time, so with the knowledge about the system as to which of the MAC or the PHY will remain on (using appropriate Device Tree properties for instance: always-on) and service Wake-on-LAN, a driver could make an appropriate decision as to whether it wants to program the MAC or the PHY with the Wake-on-LAN configuration.
> 
> The programming of the Wake-on-LAN is typically done at ->suspend() time and not necessarily at the time the user is requesting it, and at the time the user configures Wake-on-LAN we do not know yet the target suspend state.
> 
> This is a problem that could be punted to user-space in that it controls which suspend mode the system will enter. We could therefore assume that user space should know which Wake-on-LAN configuration to apply, even if that could mean "double" programming of both the MAC and PHY, knowing that the MAC will be off so the PHY will take over. The problem I see with that is that approach:
> 
> - you must always toggle between Wake-on-LAN programming depending upon the system standby mode which may not always be practical
> 
> - the behavior can vary wildly between platforms depending upon capabilities of the drivers and their bugs^w implementation
> 
> - we are still missing the ability to install a specific Wake-on-LAN configuration towards the desired MAC or PHY entity
> 
> The few drivers that call phy_ethtool_{set,get}_wol() except lan743x do not actually support Wake-on-LAN at the MAC level, so that is an easy decision to make for them because it is the only way they can support Wake-on-LAN.
> 
> What I envision we could do is add a ETHTOOL_A_WOL_DEVICE u8 field and have it take the values: 0 (default), 1 (MAC), 2 (PHY), 3 (both) and you would do the following on the command line:
> 
> ethtool -s eth0 wol g # default/existing mode, leave it to the driver
> ethtool -s eth0 wol g target mac # target the MAC only
> ethtool -s eth0 wol g target phy # target the PHY only
> ethtool -s eth0 wol g target mac+phy # target both MAC and PHY
> 
> Is that over engineering or do you see other platforms possibly needing that distinction? Heiner, how about r8169, are there similar constraints with respect to which part of the controller is on/off during S2, S3 or S5?
> 
Hi Florian,

your question regarding r8169 is a little bit hard to answer because there's no public chip documentation.
What I can say is how the driver behaves. If WOL is enabled then the MAC is kept active (receiver enabled,
no PLL power-down). This indicates that for certain WOL types MAC support is needed, I didn't test yet
whether e.g. WAKE_PHY works with the MAC disabled.

> Thanks!

Heiner