Changing devlink port flavor dynamically for DSA

Changing devlink port flavor dynamically for DSA

[Florian Fainelli]

Hi all,

On a BCM7278 system, we have two ports of the switch: 5 and 8, that
connect to separate Ethernet MACs that the host/CPU can control. In
premise they are both interchangeable because the switch supports
configuring the management port to be either 5 or 8 and the Ethernet
MACs are two identical instances.

The Ethernet MACs are scheduled differently across the memory controller
(they have different bandwidth and priority allocations) so it is
desirable to select an Ethernet MAC capable of sustaining bandwidth and
latency for host networking. Our current (in the downstream kernel) use
case is to expose port 5 solely as a control end-point to the user and
leave it to the user how they wish to use the Ethernet MAC behind port
5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
disabled. Port 5 of that switch does not make use of Broadcom tags in
that case, since ARL-based forwarding works just fine.

The current Device Tree representation that we have for that system
makes it possible for either port to be elected as the CPU port from a
DSA perspective as they both have an "ethernet" phandle property that
points to the appropriate Ethernet MAC node, because of that the DSA
framework treats them as CPU ports.

My current line of thinking is to permit a port to be configured as
either "cpu" or "user" flavor and do that through devlink. This can
create some challenges but hopefully this also paves the way for finally
supporting "multi-CPU port" configurations. I am thinking something like
this would be how I would like it to be configured:

# First configure port 8 as the new CPU port
devlink port set pci/0000:01:00.0/8 type cpu
# Now unmap port 5 from being a CPU port
devlink port set pci/0000:01:00.0/1 type eth

and this would do a simple "swap" of all user ports being now associated
with port 8, and no longer with port 5, thus permitting port 5 from
becoming a standard user port. Or maybe, we need to do this as an atomic
operation in order to avoid a switch being configured with no CPU port
anymore, so something like this instead:

devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8

The latter could also be used to define groups of ports within a switch
that has multiple CPU ports, e.g.:

# Ports 1 through 4 "bound" to CPU port 5:

for i in $(seq 0 3)
do
	devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
done

# Ports 7 bound to CPU port 8:

devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8

Let me know what you think!

Thanks
-- 
Florian

Re: Changing devlink port flavor dynamically for DSA

[Vivien Didelot]

On Sun, 5 Apr 2020 13:42:29 -0700, Florian Fainelli <f.fainelli@gmail.com> wrote:
> Hi all,
> 
> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
> connect to separate Ethernet MACs that the host/CPU can control. In
> premise they are both interchangeable because the switch supports
> configuring the management port to be either 5 or 8 and the Ethernet
> MACs are two identical instances.
> 
> The Ethernet MACs are scheduled differently across the memory controller
> (they have different bandwidth and priority allocations) so it is
> desirable to select an Ethernet MAC capable of sustaining bandwidth and
> latency for host networking. Our current (in the downstream kernel) use
> case is to expose port 5 solely as a control end-point to the user and
> leave it to the user how they wish to use the Ethernet MAC behind port
> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
> disabled. Port 5 of that switch does not make use of Broadcom tags in
> that case, since ARL-based forwarding works just fine.
> 
> The current Device Tree representation that we have for that system
> makes it possible for either port to be elected as the CPU port from a
> DSA perspective as they both have an "ethernet" phandle property that
> points to the appropriate Ethernet MAC node, because of that the DSA
> framework treats them as CPU ports.
> 
> My current line of thinking is to permit a port to be configured as
> either "cpu" or "user" flavor and do that through devlink. This can
> create some challenges but hopefully this also paves the way for finally
> supporting "multi-CPU port" configurations. I am thinking something like
> this would be how I would like it to be configured:
> 
> # First configure port 8 as the new CPU port
> devlink port set pci/0000:01:00.0/8 type cpu
> # Now unmap port 5 from being a CPU port
> devlink port set pci/0000:01:00.0/1 type eth
> 
> and this would do a simple "swap" of all user ports being now associated
> with port 8, and no longer with port 5, thus permitting port 5 from
> becoming a standard user port. Or maybe, we need to do this as an atomic
> operation in order to avoid a switch being configured with no CPU port
> anymore, so something like this instead:
> 
> devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
> 
> The latter could also be used to define groups of ports within a switch
> that has multiple CPU ports, e.g.:
> 
> # Ports 1 through 4 "bound" to CPU port 5:
> 
> for i in $(seq 0 3)
> do
> 	devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
> done
> 
> # Ports 7 bound to CPU port 8:
> 
> devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8
> 
> Let me know what you think!

From the DSA perspective, this would just be a devlink callback translated
to a ds->ops callback implemented by the driver to validate the different
types supported by a target port as well as switching between them at runtime.

I like this idea actually.


Thanks,

Vivien

Re: Changing devlink port flavor dynamically for DSA

[Jiri Pirko]

Sun, Apr 05, 2020 at 10:42:29PM CEST, f.fainelli@gmail.com wrote:
>Hi all,
>
>On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>connect to separate Ethernet MACs that the host/CPU can control. In
>premise they are both interchangeable because the switch supports
>configuring the management port to be either 5 or 8 and the Ethernet
>MACs are two identical instances.
>
>The Ethernet MACs are scheduled differently across the memory controller
>(they have different bandwidth and priority allocations) so it is
>desirable to select an Ethernet MAC capable of sustaining bandwidth and
>latency for host networking. Our current (in the downstream kernel) use
>case is to expose port 5 solely as a control end-point to the user and
>leave it to the user how they wish to use the Ethernet MAC behind port
>5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>disabled. Port 5 of that switch does not make use of Broadcom tags in
>that case, since ARL-based forwarding works just fine.
>
>The current Device Tree representation that we have for that system
>makes it possible for either port to be elected as the CPU port from a
>DSA perspective as they both have an "ethernet" phandle property that
>points to the appropriate Ethernet MAC node, because of that the DSA
>framework treats them as CPU ports.
>
>My current line of thinking is to permit a port to be configured as
>either "cpu" or "user" flavor and do that through devlink. This can
>create some challenges but hopefully this also paves the way for finally
>supporting "multi-CPU port" configurations. I am thinking something like
>this would be how I would like it to be configured:
>
># First configure port 8 as the new CPU port
>devlink port set pci/0000:01:00.0/8 type cpu
># Now unmap port 5 from being a CPU port
>devlink port set pci/0000:01:00.0/1 type eth

You are mixing "type" and "flavour".

Flavours: cpu/physical. I guess that is what you wanted to set, correct?

I'm not sure, it would make sense. The CPU port is still CPU port, it is
just not used. You can never make is really "physical", am I correct? 


btw, we already implement port "type" setting. To "eth" and "ib". This
is how you can change the type of fabric for mlx4 driver.


>
>and this would do a simple "swap" of all user ports being now associated
>with port 8, and no longer with port 5, thus permitting port 5 from
>becoming a standard user port. Or maybe, we need to do this as an atomic
>operation in order to avoid a switch being configured with no CPU port
>anymore, so something like this instead:
>
>devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
>
>The latter could also be used to define groups of ports within a switch
>that has multiple CPU ports, e.g.:
>
># Ports 1 through 4 "bound" to CPU port 5:
>
>for i in $(seq 0 3)
>do
>	devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
>done
>
># Ports 7 bound to CPU port 8:
>
>devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8

It is basically a mapping of physical port to CPU port, isn't it?

How about something like?
devlink port set pci/0000:01:00.0/1 cpu_master pci/0000:01:00.0/5
devlink port set pci/0000:01:00.0/2 cpu_master pci/0000:01:00.0/5
devlink port set pci/0000:01:00.0/7 cpu_master pci/0000:01:00.0/8

If CPU port would have 0 mapped ports, it would mean it is disabled.
What do you think?


>
>Let me know what you think!
>
>Thanks
>-- 
>Florian

Re: Changing devlink port flavor dynamically for DSA

[Florian Fainelli]

On 4/6/2020 11:04 AM, Jiri Pirko wrote:
> Sun, Apr 05, 2020 at 10:42:29PM CEST, f.fainelli@gmail.com wrote:
>> Hi all,
>>
>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>> connect to separate Ethernet MACs that the host/CPU can control. In
>> premise they are both interchangeable because the switch supports
>> configuring the management port to be either 5 or 8 and the Ethernet
>> MACs are two identical instances.
>>
>> The Ethernet MACs are scheduled differently across the memory controller
>> (they have different bandwidth and priority allocations) so it is
>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>> latency for host networking. Our current (in the downstream kernel) use
>> case is to expose port 5 solely as a control end-point to the user and
>> leave it to the user how they wish to use the Ethernet MAC behind port
>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>> that case, since ARL-based forwarding works just fine.
>>
>> The current Device Tree representation that we have for that system
>> makes it possible for either port to be elected as the CPU port from a
>> DSA perspective as they both have an "ethernet" phandle property that
>> points to the appropriate Ethernet MAC node, because of that the DSA
>> framework treats them as CPU ports.
>>
>> My current line of thinking is to permit a port to be configured as
>> either "cpu" or "user" flavor and do that through devlink. This can
>> create some challenges but hopefully this also paves the way for finally
>> supporting "multi-CPU port" configurations. I am thinking something like
>> this would be how I would like it to be configured:
>>
>> # First configure port 8 as the new CPU port
>> devlink port set pci/0000:01:00.0/8 type cpu
>> # Now unmap port 5 from being a CPU port
>> devlink port set pci/0000:01:00.0/1 type eth
> 
> You are mixing "type" and "flavour".
> 
> Flavours: cpu/physical. I guess that is what you wanted to set, correct?

Correct, flavor is really what we want to change here.

> 
> I'm not sure, it would make sense. The CPU port is still CPU port, it is
> just not used. You can never make is really "physical", am I correct? 

True, although with DSA as you may know if we have a DSA_PORT_TYPE_CPU
(or DSA_PORT_TYPE_DSA), then we do not create a corresponding net_device
instance because that would duplicate the Ethernet MAC net_device. This
is largely the reason for suggesting doing this via devlink (so that we
do not rely on a net_device handle). So by changing from a
DSA_PORT_TYPE_CPU flavor to DSA_PORT_TYPE_USER, this means you would now
see a corresponding net_device instance. Conversely when you migrate
from DSA_PORT_TYPE_USER to DSA_PORT_TYPE_CPU, the corresponding
net_device would be removed.

Or maybe we finally bite the bullet and create net_device representors
for all port types...

> 
> 
> btw, we already implement port "type" setting. To "eth" and "ib". This
> is how you can change the type of fabric for mlx4 driver.
> 
> 
>>
>> and this would do a simple "swap" of all user ports being now associated
>> with port 8, and no longer with port 5, thus permitting port 5 from
>> becoming a standard user port. Or maybe, we need to do this as an atomic
>> operation in order to avoid a switch being configured with no CPU port
>> anymore, so something like this instead:
>>
>> devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
>>
>> The latter could also be used to define groups of ports within a switch
>> that has multiple CPU ports, e.g.:
>>
>> # Ports 1 through 4 "bound" to CPU port 5:
>>
>> for i in $(seq 0 3)
>> do
>> 	devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
>> done
>>
>> # Ports 7 bound to CPU port 8:
>>
>> devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8
> 
> It is basically a mapping of physical port to CPU port, isn't it?
> 
> How about something like?
> devlink port set pci/0000:01:00.0/1 cpu_master pci/0000:01:00.0/5
> devlink port set pci/0000:01:00.0/2 cpu_master pci/0000:01:00.0/5
> devlink port set pci/0000:01:00.0/7 cpu_master pci/0000:01:00.0/8
> 
> If CPU port would have 0 mapped ports, it would mean it is disabled.
> What do you think?

Yes, this makes sense.
-- 
Florian

Re: Changing devlink port flavor dynamically for DSA

[Jiri Pirko]

Mon, Apr 06, 2020 at 08:11:18PM CEST, f.fainelli@gmail.com wrote:
>
>
>On 4/6/2020 11:04 AM, Jiri Pirko wrote:
>> Sun, Apr 05, 2020 at 10:42:29PM CEST, f.fainelli@gmail.com wrote:
>>> Hi all,
>>>
>>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>>> connect to separate Ethernet MACs that the host/CPU can control. In
>>> premise they are both interchangeable because the switch supports
>>> configuring the management port to be either 5 or 8 and the Ethernet
>>> MACs are two identical instances.
>>>
>>> The Ethernet MACs are scheduled differently across the memory controller
>>> (they have different bandwidth and priority allocations) so it is
>>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>>> latency for host networking. Our current (in the downstream kernel) use
>>> case is to expose port 5 solely as a control end-point to the user and
>>> leave it to the user how they wish to use the Ethernet MAC behind port
>>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>>> that case, since ARL-based forwarding works just fine.
>>>
>>> The current Device Tree representation that we have for that system
>>> makes it possible for either port to be elected as the CPU port from a
>>> DSA perspective as they both have an "ethernet" phandle property that
>>> points to the appropriate Ethernet MAC node, because of that the DSA
>>> framework treats them as CPU ports.
>>>
>>> My current line of thinking is to permit a port to be configured as
>>> either "cpu" or "user" flavor and do that through devlink. This can
>>> create some challenges but hopefully this also paves the way for finally
>>> supporting "multi-CPU port" configurations. I am thinking something like
>>> this would be how I would like it to be configured:
>>>
>>> # First configure port 8 as the new CPU port
>>> devlink port set pci/0000:01:00.0/8 type cpu
>>> # Now unmap port 5 from being a CPU port
>>> devlink port set pci/0000:01:00.0/1 type eth
>> 
>> You are mixing "type" and "flavour".
>> 
>> Flavours: cpu/physical. I guess that is what you wanted to set, correct?
>
>Correct, flavor is really what we want to change here.
>
>> 
>> I'm not sure, it would make sense. The CPU port is still CPU port, it is
>> just not used. You can never make is really "physical", am I correct? 
>
>True, although with DSA as you may know if we have a DSA_PORT_TYPE_CPU
>(or DSA_PORT_TYPE_DSA), then we do not create a corresponding net_device
>instance because that would duplicate the Ethernet MAC net_device. This
>is largely the reason for suggesting doing this via devlink (so that we
>do not rely on a net_device handle). So by changing from a
>DSA_PORT_TYPE_CPU flavor to DSA_PORT_TYPE_USER, this means you would now
>see a corresponding net_device instance. Conversely when you migrate
>from DSA_PORT_TYPE_USER to DSA_PORT_TYPE_CPU, the corresponding
>net_device would be removed.

Wait, why would you ever want to have a netdevice for CPU port (changed
to "user")? What am I missing? Is there a usecase, some multi-person
port?


>
>Or maybe we finally bite the bullet and create net_device representors
>for all port types...
>
>> 
>> 
>> btw, we already implement port "type" setting. To "eth" and "ib". This
>> is how you can change the type of fabric for mlx4 driver.
>> 
>> 
>>>
>>> and this would do a simple "swap" of all user ports being now associated
>>> with port 8, and no longer with port 5, thus permitting port 5 from
>>> becoming a standard user port. Or maybe, we need to do this as an atomic
>>> operation in order to avoid a switch being configured with no CPU port
>>> anymore, so something like this instead:
>>>
>>> devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
>>>
>>> The latter could also be used to define groups of ports within a switch
>>> that has multiple CPU ports, e.g.:
>>>
>>> # Ports 1 through 4 "bound" to CPU port 5:
>>>
>>> for i in $(seq 0 3)
>>> do
>>> 	devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
>>> done
>>>
>>> # Ports 7 bound to CPU port 8:
>>>
>>> devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8
>> 
>> It is basically a mapping of physical port to CPU port, isn't it?
>> 
>> How about something like?
>> devlink port set pci/0000:01:00.0/1 cpu_master pci/0000:01:00.0/5
>> devlink port set pci/0000:01:00.0/2 cpu_master pci/0000:01:00.0/5
>> devlink port set pci/0000:01:00.0/7 cpu_master pci/0000:01:00.0/8
>> 
>> If CPU port would have 0 mapped ports, it would mean it is disabled.
>> What do you think?
>
>Yes, this makes sense.
>-- 
>Florian

Re: Changing devlink port flavor dynamically for DSA

[Florian Fainelli]

On 4/6/2020 11:20 AM, Jiri Pirko wrote:
> Mon, Apr 06, 2020 at 08:11:18PM CEST, f.fainelli@gmail.com wrote:
>>
>>
>> On 4/6/2020 11:04 AM, Jiri Pirko wrote:
>>> Sun, Apr 05, 2020 at 10:42:29PM CEST, f.fainelli@gmail.com wrote:
>>>> Hi all,
>>>>
>>>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>>>> connect to separate Ethernet MACs that the host/CPU can control. In
>>>> premise they are both interchangeable because the switch supports
>>>> configuring the management port to be either 5 or 8 and the Ethernet
>>>> MACs are two identical instances.
>>>>
>>>> The Ethernet MACs are scheduled differently across the memory controller
>>>> (they have different bandwidth and priority allocations) so it is
>>>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>>>> latency for host networking. Our current (in the downstream kernel) use
>>>> case is to expose port 5 solely as a control end-point to the user and
>>>> leave it to the user how they wish to use the Ethernet MAC behind port
>>>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>>>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>>>> that case, since ARL-based forwarding works just fine.
>>>>
>>>> The current Device Tree representation that we have for that system
>>>> makes it possible for either port to be elected as the CPU port from a
>>>> DSA perspective as they both have an "ethernet" phandle property that
>>>> points to the appropriate Ethernet MAC node, because of that the DSA
>>>> framework treats them as CPU ports.
>>>>
>>>> My current line of thinking is to permit a port to be configured as
>>>> either "cpu" or "user" flavor and do that through devlink. This can
>>>> create some challenges but hopefully this also paves the way for finally
>>>> supporting "multi-CPU port" configurations. I am thinking something like
>>>> this would be how I would like it to be configured:
>>>>
>>>> # First configure port 8 as the new CPU port
>>>> devlink port set pci/0000:01:00.0/8 type cpu
>>>> # Now unmap port 5 from being a CPU port
>>>> devlink port set pci/0000:01:00.0/1 type eth
>>>
>>> You are mixing "type" and "flavour".
>>>
>>> Flavours: cpu/physical. I guess that is what you wanted to set, correct?
>>
>> Correct, flavor is really what we want to change here.
>>
>>>
>>> I'm not sure, it would make sense. The CPU port is still CPU port, it is
>>> just not used. You can never make is really "physical", am I correct? 
>>
>> True, although with DSA as you may know if we have a DSA_PORT_TYPE_CPU
>> (or DSA_PORT_TYPE_DSA), then we do not create a corresponding net_device
>> instance because that would duplicate the Ethernet MAC net_device. This
>> is largely the reason for suggesting doing this via devlink (so that we
>> do not rely on a net_device handle). So by changing from a
>> DSA_PORT_TYPE_CPU flavor to DSA_PORT_TYPE_USER, this means you would now
>> see a corresponding net_device instance. Conversely when you migrate
>>from DSA_PORT_TYPE_USER to DSA_PORT_TYPE_CPU, the corresponding
>> net_device would be removed.
> 
> Wait, why would you ever want to have a netdevice for CPU port (changed
> to "user")? What am I missing? Is there a usecase, some multi-person
> port?
> 

I believe I explained the use case in my first email. The way the Device
Tree description is currently laid out makes it that Port 5 gets picked
up as the CPU port for the system. This is because the DSA layer stops
whenever it encounters the first "CPU" port, which it determines by
having an "ethernet" phandle (reference) to an Ethernet MAC controller node.

The Ethernet MAC controller behind Port 5 does not have the necessary
bandwidth allocation at the memory controller level to sustain Gigabit
traffic with 64B packets. Instead we want to use Port 8 and the Ethernet
MAC connected to it which has been budgeted to support that bandwidth.

The reason why we want to have a representor (DSA_PORT_TYPE_USER) for
port 5 of the switch is also because we need to set-up Compact Field
Processor (CFP) rules which need to identify specific packets ingressing
a specific port number and to be redirected towards another port (7) for
audio/video streaming processing. The interface to configure CFP is
currently ethool::rxnfc and that requires a net_device handle,
cls_flower would be the same AFAICT.
-- 
Florian

Re: Changing devlink port flavor dynamically for DSA

[Jiri Pirko]

Mon, Apr 06, 2020 at 08:41:25PM CEST, f.fainelli@gmail.com wrote:
>
>
>On 4/6/2020 11:20 AM, Jiri Pirko wrote:
>> Mon, Apr 06, 2020 at 08:11:18PM CEST, f.fainelli@gmail.com wrote:
>>>
>>>
>>> On 4/6/2020 11:04 AM, Jiri Pirko wrote:
>>>> Sun, Apr 05, 2020 at 10:42:29PM CEST, f.fainelli@gmail.com wrote:
>>>>> Hi all,
>>>>>
>>>>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>>>>> connect to separate Ethernet MACs that the host/CPU can control. In
>>>>> premise they are both interchangeable because the switch supports
>>>>> configuring the management port to be either 5 or 8 and the Ethernet
>>>>> MACs are two identical instances.
>>>>>
>>>>> The Ethernet MACs are scheduled differently across the memory controller
>>>>> (they have different bandwidth and priority allocations) so it is
>>>>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>>>>> latency for host networking. Our current (in the downstream kernel) use
>>>>> case is to expose port 5 solely as a control end-point to the user and
>>>>> leave it to the user how they wish to use the Ethernet MAC behind port
>>>>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>>>>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>>>>> that case, since ARL-based forwarding works just fine.
>>>>>
>>>>> The current Device Tree representation that we have for that system
>>>>> makes it possible for either port to be elected as the CPU port from a
>>>>> DSA perspective as they both have an "ethernet" phandle property that
>>>>> points to the appropriate Ethernet MAC node, because of that the DSA
>>>>> framework treats them as CPU ports.
>>>>>
>>>>> My current line of thinking is to permit a port to be configured as
>>>>> either "cpu" or "user" flavor and do that through devlink. This can
>>>>> create some challenges but hopefully this also paves the way for finally
>>>>> supporting "multi-CPU port" configurations. I am thinking something like
>>>>> this would be how I would like it to be configured:
>>>>>
>>>>> # First configure port 8 as the new CPU port
>>>>> devlink port set pci/0000:01:00.0/8 type cpu
>>>>> # Now unmap port 5 from being a CPU port
>>>>> devlink port set pci/0000:01:00.0/1 type eth
>>>>
>>>> You are mixing "type" and "flavour".
>>>>
>>>> Flavours: cpu/physical. I guess that is what you wanted to set, correct?
>>>
>>> Correct, flavor is really what we want to change here.
>>>
>>>>
>>>> I'm not sure, it would make sense. The CPU port is still CPU port, it is
>>>> just not used. You can never make is really "physical", am I correct? 
>>>
>>> True, although with DSA as you may know if we have a DSA_PORT_TYPE_CPU
>>> (or DSA_PORT_TYPE_DSA), then we do not create a corresponding net_device
>>> instance because that would duplicate the Ethernet MAC net_device. This
>>> is largely the reason for suggesting doing this via devlink (so that we
>>> do not rely on a net_device handle). So by changing from a
>>> DSA_PORT_TYPE_CPU flavor to DSA_PORT_TYPE_USER, this means you would now
>>> see a corresponding net_device instance. Conversely when you migrate
>>>from DSA_PORT_TYPE_USER to DSA_PORT_TYPE_CPU, the corresponding
>>> net_device would be removed.
>> 
>> Wait, why would you ever want to have a netdevice for CPU port (changed
>> to "user")? What am I missing? Is there a usecase, some multi-person
>> port?
>> 
>
>I believe I explained the use case in my first email. The way the Device
>Tree description is currently laid out makes it that Port 5 gets picked
>up as the CPU port for the system. This is because the DSA layer stops
>whenever it encounters the first "CPU" port, which it determines by
>having an "ethernet" phandle (reference) to an Ethernet MAC controller node.
>
>The Ethernet MAC controller behind Port 5 does not have the necessary
>bandwidth allocation at the memory controller level to sustain Gigabit
>traffic with 64B packets. Instead we want to use Port 8 and the Ethernet
>MAC connected to it which has been budgeted to support that bandwidth.
>
>The reason why we want to have a representor (DSA_PORT_TYPE_USER) for
>port 5 of the switch is also because we need to set-up Compact Field
>Processor (CFP) rules which need to identify specific packets ingressing
>a specific port number and to be redirected towards another port (7) for
>audio/video streaming processing. The interface to configure CFP is
>currently ethool::rxnfc and that requires a net_device handle,
>cls_flower would be the same AFAICT.

But if you don't use port 5 as CPU port, you don't use if for anything.
Okay, you say that you need it for ethtool setup. But can't you use
other netdevs of another port (physical port) for that?

My point is, port 5 would not ever become physical port, equivalent to
port 1 for example. Will still be CPU port. Only used/unused.

Re: Changing devlink port flavor dynamically for DSA

[Florian Fainelli]

On 4/6/2020 12:00 PM, Jiri Pirko wrote:
> Mon, Apr 06, 2020 at 08:41:25PM CEST, f.fainelli@gmail.com wrote:
>>
>>
>> On 4/6/2020 11:20 AM, Jiri Pirko wrote:
>>> Mon, Apr 06, 2020 at 08:11:18PM CEST, f.fainelli@gmail.com wrote:
>>>>
>>>>
>>>> On 4/6/2020 11:04 AM, Jiri Pirko wrote:
>>>>> Sun, Apr 05, 2020 at 10:42:29PM CEST, f.fainelli@gmail.com wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>>>>>> connect to separate Ethernet MACs that the host/CPU can control. In
>>>>>> premise they are both interchangeable because the switch supports
>>>>>> configuring the management port to be either 5 or 8 and the Ethernet
>>>>>> MACs are two identical instances.
>>>>>>
>>>>>> The Ethernet MACs are scheduled differently across the memory controller
>>>>>> (they have different bandwidth and priority allocations) so it is
>>>>>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>>>>>> latency for host networking. Our current (in the downstream kernel) use
>>>>>> case is to expose port 5 solely as a control end-point to the user and
>>>>>> leave it to the user how they wish to use the Ethernet MAC behind port
>>>>>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>>>>>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>>>>>> that case, since ARL-based forwarding works just fine.
>>>>>>
>>>>>> The current Device Tree representation that we have for that system
>>>>>> makes it possible for either port to be elected as the CPU port from a
>>>>>> DSA perspective as they both have an "ethernet" phandle property that
>>>>>> points to the appropriate Ethernet MAC node, because of that the DSA
>>>>>> framework treats them as CPU ports.
>>>>>>
>>>>>> My current line of thinking is to permit a port to be configured as
>>>>>> either "cpu" or "user" flavor and do that through devlink. This can
>>>>>> create some challenges but hopefully this also paves the way for finally
>>>>>> supporting "multi-CPU port" configurations. I am thinking something like
>>>>>> this would be how I would like it to be configured:
>>>>>>
>>>>>> # First configure port 8 as the new CPU port
>>>>>> devlink port set pci/0000:01:00.0/8 type cpu
>>>>>> # Now unmap port 5 from being a CPU port
>>>>>> devlink port set pci/0000:01:00.0/1 type eth
>>>>>
>>>>> You are mixing "type" and "flavour".
>>>>>
>>>>> Flavours: cpu/physical. I guess that is what you wanted to set, correct?
>>>>
>>>> Correct, flavor is really what we want to change here.
>>>>
>>>>>
>>>>> I'm not sure, it would make sense. The CPU port is still CPU port, it is
>>>>> just not used. You can never make is really "physical", am I correct? 
>>>>
>>>> True, although with DSA as you may know if we have a DSA_PORT_TYPE_CPU
>>>> (or DSA_PORT_TYPE_DSA), then we do not create a corresponding net_device
>>>> instance because that would duplicate the Ethernet MAC net_device. This
>>>> is largely the reason for suggesting doing this via devlink (so that we
>>>> do not rely on a net_device handle). So by changing from a
>>>> DSA_PORT_TYPE_CPU flavor to DSA_PORT_TYPE_USER, this means you would now
>>>> see a corresponding net_device instance. Conversely when you migrate
>>> >from DSA_PORT_TYPE_USER to DSA_PORT_TYPE_CPU, the corresponding
>>>> net_device would be removed.
>>>
>>> Wait, why would you ever want to have a netdevice for CPU port (changed
>>> to "user")? What am I missing? Is there a usecase, some multi-person
>>> port?
>>>
>>
>> I believe I explained the use case in my first email. The way the Device
>> Tree description is currently laid out makes it that Port 5 gets picked
>> up as the CPU port for the system. This is because the DSA layer stops
>> whenever it encounters the first "CPU" port, which it determines by
>> having an "ethernet" phandle (reference) to an Ethernet MAC controller node.
>>
>> The Ethernet MAC controller behind Port 5 does not have the necessary
>> bandwidth allocation at the memory controller level to sustain Gigabit
>> traffic with 64B packets. Instead we want to use Port 8 and the Ethernet
>> MAC connected to it which has been budgeted to support that bandwidth.
>>
>> The reason why we want to have a representor (DSA_PORT_TYPE_USER) for
>> port 5 of the switch is also because we need to set-up Compact Field
>> Processor (CFP) rules which need to identify specific packets ingressing
>> a specific port number and to be redirected towards another port (7) for
>> audio/video streaming processing. The interface to configure CFP is
>> currently ethool::rxnfc and that requires a net_device handle,
>> cls_flower would be the same AFAICT.
> 
> But if you don't use port 5 as CPU port, you don't use if for anything.
> Okay, you say that you need it for ethtool setup. But can't you use
> other netdevs of another port (physical port) for that?

Not really, the net_device gives you the physical port number, and there
is no way for the user to specify something different. It would also be
utterly confusing that you used whatever net_device exists to configure
an ingress matching rule to another physical port...

> 
> My point is, port 5 would not ever become physical port, equivalent to
> port 1 for example. Will still be CPU port. Only used/unused.

This is a valid point and question. If we are to support dynamically
changing DSA port flavor, there would be an expectation that when the
CPU port becomes DSA_PORT_TYPE_USER, we would be creating a net_device
representor for it at that time (since that is what would happen if we
did let DSA create devices in the first place). Conversely, if being an
USER port and being changed to a CPU port we would remove the net_device.
-- 
Florian

Re: Changing devlink port flavor dynamically for DSA

[Jiri Pirko]

Mon, Apr 06, 2020 at 09:52:03PM CEST, f.fainelli@gmail.com wrote:
>
>
>On 4/6/2020 12:00 PM, Jiri Pirko wrote:
>> Mon, Apr 06, 2020 at 08:41:25PM CEST, f.fainelli@gmail.com wrote:
>>>
>>>
>>> On 4/6/2020 11:20 AM, Jiri Pirko wrote:
>>>> Mon, Apr 06, 2020 at 08:11:18PM CEST, f.fainelli@gmail.com wrote:
>>>>>
>>>>>
>>>>> On 4/6/2020 11:04 AM, Jiri Pirko wrote:
>>>>>> Sun, Apr 05, 2020 at 10:42:29PM CEST, f.fainelli@gmail.com wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>>>>>>> connect to separate Ethernet MACs that the host/CPU can control. In
>>>>>>> premise they are both interchangeable because the switch supports
>>>>>>> configuring the management port to be either 5 or 8 and the Ethernet
>>>>>>> MACs are two identical instances.
>>>>>>>
>>>>>>> The Ethernet MACs are scheduled differently across the memory controller
>>>>>>> (they have different bandwidth and priority allocations) so it is
>>>>>>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>>>>>>> latency for host networking. Our current (in the downstream kernel) use
>>>>>>> case is to expose port 5 solely as a control end-point to the user and
>>>>>>> leave it to the user how they wish to use the Ethernet MAC behind port
>>>>>>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>>>>>>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>>>>>>> that case, since ARL-based forwarding works just fine.
>>>>>>>
>>>>>>> The current Device Tree representation that we have for that system
>>>>>>> makes it possible for either port to be elected as the CPU port from a
>>>>>>> DSA perspective as they both have an "ethernet" phandle property that
>>>>>>> points to the appropriate Ethernet MAC node, because of that the DSA
>>>>>>> framework treats them as CPU ports.
>>>>>>>
>>>>>>> My current line of thinking is to permit a port to be configured as
>>>>>>> either "cpu" or "user" flavor and do that through devlink. This can
>>>>>>> create some challenges but hopefully this also paves the way for finally
>>>>>>> supporting "multi-CPU port" configurations. I am thinking something like
>>>>>>> this would be how I would like it to be configured:
>>>>>>>
>>>>>>> # First configure port 8 as the new CPU port
>>>>>>> devlink port set pci/0000:01:00.0/8 type cpu
>>>>>>> # Now unmap port 5 from being a CPU port
>>>>>>> devlink port set pci/0000:01:00.0/1 type eth
>>>>>>
>>>>>> You are mixing "type" and "flavour".
>>>>>>
>>>>>> Flavours: cpu/physical. I guess that is what you wanted to set, correct?
>>>>>
>>>>> Correct, flavor is really what we want to change here.
>>>>>
>>>>>>
>>>>>> I'm not sure, it would make sense. The CPU port is still CPU port, it is
>>>>>> just not used. You can never make is really "physical", am I correct? 
>>>>>
>>>>> True, although with DSA as you may know if we have a DSA_PORT_TYPE_CPU
>>>>> (or DSA_PORT_TYPE_DSA), then we do not create a corresponding net_device
>>>>> instance because that would duplicate the Ethernet MAC net_device. This
>>>>> is largely the reason for suggesting doing this via devlink (so that we
>>>>> do not rely on a net_device handle). So by changing from a
>>>>> DSA_PORT_TYPE_CPU flavor to DSA_PORT_TYPE_USER, this means you would now
>>>>> see a corresponding net_device instance. Conversely when you migrate
>>>> >from DSA_PORT_TYPE_USER to DSA_PORT_TYPE_CPU, the corresponding
>>>>> net_device would be removed.
>>>>
>>>> Wait, why would you ever want to have a netdevice for CPU port (changed
>>>> to "user")? What am I missing? Is there a usecase, some multi-person
>>>> port?
>>>>
>>>
>>> I believe I explained the use case in my first email. The way the Device
>>> Tree description is currently laid out makes it that Port 5 gets picked
>>> up as the CPU port for the system. This is because the DSA layer stops
>>> whenever it encounters the first "CPU" port, which it determines by
>>> having an "ethernet" phandle (reference) to an Ethernet MAC controller node.
>>>
>>> The Ethernet MAC controller behind Port 5 does not have the necessary
>>> bandwidth allocation at the memory controller level to sustain Gigabit
>>> traffic with 64B packets. Instead we want to use Port 8 and the Ethernet
>>> MAC connected to it which has been budgeted to support that bandwidth.
>>>
>>> The reason why we want to have a representor (DSA_PORT_TYPE_USER) for
>>> port 5 of the switch is also because we need to set-up Compact Field
>>> Processor (CFP) rules which need to identify specific packets ingressing
>>> a specific port number and to be redirected towards another port (7) for
>>> audio/video streaming processing. The interface to configure CFP is
>>> currently ethool::rxnfc and that requires a net_device handle,
>>> cls_flower would be the same AFAICT.
>> 
>> But if you don't use port 5 as CPU port, you don't use if for anything.
>> Okay, you say that you need it for ethtool setup. But can't you use
>> other netdevs of another port (physical port) for that?
>
>Not really, the net_device gives you the physical port number, and there
>is no way for the user to specify something different. It would also be
>utterly confusing that you used whatever net_device exists to configure
>an ingress matching rule to another physical port...
>
>> 
>> My point is, port 5 would not ever become physical port, equivalent to
>> port 1 for example. Will still be CPU port. Only used/unused.
>
>This is a valid point and question. If we are to support dynamically
>changing DSA port flavor, there would be an expectation that when the
>CPU port becomes DSA_PORT_TYPE_USER, we would be creating a net_device
>representor for it at that time (since that is what would happen if we
>did let DSA create devices in the first place). Conversely, if being an
>USER port and being changed to a CPU port we would remove the net_device.

Ah, I think I understand. Port 5 will just become as any other port, it
is just CPU facing. But can be used in the same way as the physical
port, right? Just the traffic would not go to the wire but to CPU.

Re: Changing devlink port flavor dynamically for DSA

[Vladimir Oltean]

Hi Florian,

On Sun, 5 Apr 2020 at 23:42, Florian Fainelli <f.fainelli@gmail.com> wrote:
>
> Hi all,
>
> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
> connect to separate Ethernet MACs that the host/CPU can control. In
> premise they are both interchangeable because the switch supports
> configuring the management port to be either 5 or 8 and the Ethernet
> MACs are two identical instances.
>
> The Ethernet MACs are scheduled differently across the memory controller
> (they have different bandwidth and priority allocations) so it is
> desirable to select an Ethernet MAC capable of sustaining bandwidth and
> latency for host networking. Our current (in the downstream kernel) use
> case is to expose port 5 solely as a control end-point to the user and
> leave it to the user how they wish to use the Ethernet MAC behind port
> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
> disabled. Port 5 of that switch does not make use of Broadcom tags in
> that case, since ARL-based forwarding works just fine.
>
> The current Device Tree representation that we have for that system
> makes it possible for either port to be elected as the CPU port from a
> DSA perspective as they both have an "ethernet" phandle property that
> points to the appropriate Ethernet MAC node, because of that the DSA
> framework treats them as CPU ports.
>
> My current line of thinking is to permit a port to be configured as
> either "cpu" or "user" flavor and do that through devlink. This can
> create some challenges but hopefully this also paves the way for finally
> supporting "multi-CPU port" configurations. I am thinking something like
> this would be how I would like it to be configured:
>
> # First configure port 8 as the new CPU port
> devlink port set pci/0000:01:00.0/8 type cpu
> # Now unmap port 5 from being a CPU port
> devlink port set pci/0000:01:00.0/1 type eth
>
> and this would do a simple "swap" of all user ports being now associated
> with port 8, and no longer with port 5, thus permitting port 5 from
> becoming a standard user port. Or maybe, we need to do this as an atomic
> operation in order to avoid a switch being configured with no CPU port
> anymore, so something like this instead:
>
> devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
>
> The latter could also be used to define groups of ports within a switch
> that has multiple CPU ports, e.g.:
>
> # Ports 1 through 4 "bound" to CPU port 5:
>
> for i in $(seq 0 3)
> do
>         devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
> done
>
> # Ports 7 bound to CPU port 8:
>
> devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8
>
> Let me know what you think!
>
> Thanks
> --
> Florian

What is missing from your argumentation is what would the new devlink
mechanism of changing the CPU port bring for your particular use case.
I mean you can already remove the "ethernet" device tree property from
port 5 and end up exactly with the configuration that you want, no?

Regards,
-Vladimir

Re: Changing devlink port flavor dynamically for DSA

[Florian Fainelli]

On 4/8/2020 12:51 PM, Vladimir Oltean wrote:
> Hi Florian,
> 
> On Sun, 5 Apr 2020 at 23:42, Florian Fainelli <f.fainelli@gmail.com> wrote:
>>
>> Hi all,
>>
>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>> connect to separate Ethernet MACs that the host/CPU can control. In
>> premise they are both interchangeable because the switch supports
>> configuring the management port to be either 5 or 8 and the Ethernet
>> MACs are two identical instances.
>>
>> The Ethernet MACs are scheduled differently across the memory controller
>> (they have different bandwidth and priority allocations) so it is
>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>> latency for host networking. Our current (in the downstream kernel) use
>> case is to expose port 5 solely as a control end-point to the user and
>> leave it to the user how they wish to use the Ethernet MAC behind port
>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>> that case, since ARL-based forwarding works just fine.
>>
>> The current Device Tree representation that we have for that system
>> makes it possible for either port to be elected as the CPU port from a
>> DSA perspective as they both have an "ethernet" phandle property that
>> points to the appropriate Ethernet MAC node, because of that the DSA
>> framework treats them as CPU ports.
>>
>> My current line of thinking is to permit a port to be configured as
>> either "cpu" or "user" flavor and do that through devlink. This can
>> create some challenges but hopefully this also paves the way for finally
>> supporting "multi-CPU port" configurations. I am thinking something like
>> this would be how I would like it to be configured:
>>
>> # First configure port 8 as the new CPU port
>> devlink port set pci/0000:01:00.0/8 type cpu
>> # Now unmap port 5 from being a CPU port
>> devlink port set pci/0000:01:00.0/1 type eth
>>
>> and this would do a simple "swap" of all user ports being now associated
>> with port 8, and no longer with port 5, thus permitting port 5 from
>> becoming a standard user port. Or maybe, we need to do this as an atomic
>> operation in order to avoid a switch being configured with no CPU port
>> anymore, so something like this instead:
>>
>> devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
>>
>> The latter could also be used to define groups of ports within a switch
>> that has multiple CPU ports, e.g.:
>>
>> # Ports 1 through 4 "bound" to CPU port 5:
>>
>> for i in $(seq 0 3)
>> do
>>         devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
>> done
>>
>> # Ports 7 bound to CPU port 8:
>>
>> devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8
>>
>> Let me know what you think!
>>
>> Thanks
>> --
>> Florian
> 
> What is missing from your argumentation is what would the new devlink
> mechanism of changing the CPU port bring for your particular use case.
> I mean you can already remove the "ethernet" device tree property from
> port 5 and end up exactly with the configuration that you want, no?

That's what I do in our downstream tree for now, should I submit this
upstream? I doubt it would be accepted.
-- 
Florian

Re: Changing devlink port flavor dynamically for DSA

[Vladimir Oltean]

On Wed, 8 Apr 2020 at 23:05, Florian Fainelli <f.fainelli@gmail.com> wrote:
>
>
>
> On 4/8/2020 12:51 PM, Vladimir Oltean wrote:
> > Hi Florian,
> >
> > On Sun, 5 Apr 2020 at 23:42, Florian Fainelli <f.fainelli@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
> >> connect to separate Ethernet MACs that the host/CPU can control. In
> >> premise they are both interchangeable because the switch supports
> >> configuring the management port to be either 5 or 8 and the Ethernet
> >> MACs are two identical instances.
> >>
> >> The Ethernet MACs are scheduled differently across the memory controller
> >> (they have different bandwidth and priority allocations) so it is
> >> desirable to select an Ethernet MAC capable of sustaining bandwidth and
> >> latency for host networking. Our current (in the downstream kernel) use
> >> case is to expose port 5 solely as a control end-point to the user and
> >> leave it to the user how they wish to use the Ethernet MAC behind port
> >> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
> >> disabled. Port 5 of that switch does not make use of Broadcom tags in
> >> that case, since ARL-based forwarding works just fine.
> >>
> >> The current Device Tree representation that we have for that system
> >> makes it possible for either port to be elected as the CPU port from a
> >> DSA perspective as they both have an "ethernet" phandle property that
> >> points to the appropriate Ethernet MAC node, because of that the DSA
> >> framework treats them as CPU ports.
> >>
> >> My current line of thinking is to permit a port to be configured as
> >> either "cpu" or "user" flavor and do that through devlink. This can
> >> create some challenges but hopefully this also paves the way for finally
> >> supporting "multi-CPU port" configurations. I am thinking something like
> >> this would be how I would like it to be configured:
> >>
> >> # First configure port 8 as the new CPU port
> >> devlink port set pci/0000:01:00.0/8 type cpu
> >> # Now unmap port 5 from being a CPU port
> >> devlink port set pci/0000:01:00.0/1 type eth
> >>
> >> and this would do a simple "swap" of all user ports being now associated
> >> with port 8, and no longer with port 5, thus permitting port 5 from
> >> becoming a standard user port. Or maybe, we need to do this as an atomic
> >> operation in order to avoid a switch being configured with no CPU port
> >> anymore, so something like this instead:
> >>
> >> devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
> >>
> >> The latter could also be used to define groups of ports within a switch
> >> that has multiple CPU ports, e.g.:
> >>
> >> # Ports 1 through 4 "bound" to CPU port 5:
> >>
> >> for i in $(seq 0 3)
> >> do
> >>         devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
> >> done
> >>
> >> # Ports 7 bound to CPU port 8:
> >>
> >> devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8
> >>
> >> Let me know what you think!
> >>
> >> Thanks
> >> --
> >> Florian
> >
> > What is missing from your argumentation is what would the new devlink
> > mechanism of changing the CPU port bring for your particular use case.
> > I mean you can already remove the "ethernet" device tree property from
> > port 5 and end up exactly with the configuration that you want, no?
>
> That's what I do in our downstream tree for now, should I submit this
> upstream? I doubt it would be accepted.
> --
> Florian

This is exactly what we do for the NXP LS1028A (ocelot/felix driver),
where we enable just one of the 2 CPU ports by default (and the other
one, just as a simple user port in the very few situations that
require it):
https://github.com/torvalds/linux/blob/master/arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi#L943
Although to be fair, for LS1028A we can't even dream of enabling DSA
tagging on both CPU ports at the same time, since there is a hardware
limitation in place that only a single port may carry DSA tags at any
given moment in time.

-Vladimir

Re: Changing devlink port flavor dynamically for DSA

[Florian Fainelli]

On 4/8/2020 1:10 PM, Vladimir Oltean wrote:
> On Wed, 8 Apr 2020 at 23:05, Florian Fainelli <f.fainelli@gmail.com> wrote:
>>
>>
>>
>> On 4/8/2020 12:51 PM, Vladimir Oltean wrote:
>>> Hi Florian,
>>>
>>> On Sun, 5 Apr 2020 at 23:42, Florian Fainelli <f.fainelli@gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> On a BCM7278 system, we have two ports of the switch: 5 and 8, that
>>>> connect to separate Ethernet MACs that the host/CPU can control. In
>>>> premise they are both interchangeable because the switch supports
>>>> configuring the management port to be either 5 or 8 and the Ethernet
>>>> MACs are two identical instances.
>>>>
>>>> The Ethernet MACs are scheduled differently across the memory controller
>>>> (they have different bandwidth and priority allocations) so it is
>>>> desirable to select an Ethernet MAC capable of sustaining bandwidth and
>>>> latency for host networking. Our current (in the downstream kernel) use
>>>> case is to expose port 5 solely as a control end-point to the user and
>>>> leave it to the user how they wish to use the Ethernet MAC behind port
>>>> 5. Some customers use it to bridge Wi-Fi traffic, some simply keep it
>>>> disabled. Port 5 of that switch does not make use of Broadcom tags in
>>>> that case, since ARL-based forwarding works just fine.
>>>>
>>>> The current Device Tree representation that we have for that system
>>>> makes it possible for either port to be elected as the CPU port from a
>>>> DSA perspective as they both have an "ethernet" phandle property that
>>>> points to the appropriate Ethernet MAC node, because of that the DSA
>>>> framework treats them as CPU ports.
>>>>
>>>> My current line of thinking is to permit a port to be configured as
>>>> either "cpu" or "user" flavor and do that through devlink. This can
>>>> create some challenges but hopefully this also paves the way for finally
>>>> supporting "multi-CPU port" configurations. I am thinking something like
>>>> this would be how I would like it to be configured:
>>>>
>>>> # First configure port 8 as the new CPU port
>>>> devlink port set pci/0000:01:00.0/8 type cpu
>>>> # Now unmap port 5 from being a CPU port
>>>> devlink port set pci/0000:01:00.0/1 type eth
>>>>
>>>> and this would do a simple "swap" of all user ports being now associated
>>>> with port 8, and no longer with port 5, thus permitting port 5 from
>>>> becoming a standard user port. Or maybe, we need to do this as an atomic
>>>> operation in order to avoid a switch being configured with no CPU port
>>>> anymore, so something like this instead:
>>>>
>>>> devlink port set pci/0000:01:00.0/5 type eth mgmt pci/0000:01:00.0/8
>>>>
>>>> The latter could also be used to define groups of ports within a switch
>>>> that has multiple CPU ports, e.g.:
>>>>
>>>> # Ports 1 through 4 "bound" to CPU port 5:
>>>>
>>>> for i in $(seq 0 3)
>>>> do
>>>>         devlink port set pci/0000:01:00.0/$i type eth mgmt pci/0000:01:00.0/5
>>>> done
>>>>
>>>> # Ports 7 bound to CPU port 8:
>>>>
>>>> devlink port set pci/0000:01:00.0/1 type eth mgmt pci/0000:01:00.0/8
>>>>
>>>> Let me know what you think!
>>>>
>>>> Thanks
>>>> --
>>>> Florian
>>>
>>> What is missing from your argumentation is what would the new devlink
>>> mechanism of changing the CPU port bring for your particular use case.
>>> I mean you can already remove the "ethernet" device tree property from
>>> port 5 and end up exactly with the configuration that you want, no?
>>
>> That's what I do in our downstream tree for now, should I submit this
>> upstream? I doubt it would be accepted.
>> --
>> Florian
> 
> This is exactly what we do for the NXP LS1028A (ocelot/felix driver),
> where we enable just one of the 2 CPU ports by default (and the other
> one, just as a simple user port in the very few situations that
> require it):
> https://github.com/torvalds/linux/blob/master/arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi#L943
> Although to be fair, for LS1028A we can't even dream of enabling DSA
> tagging on both CPU ports at the same time, since there is a hardware
> limitation in place that only a single port may carry DSA tags at any
> given moment in time.

The firmware is provided by the boot loader in my case, and while it can
be changed, dealing with incompatibilities is a support burden, so
instead, I have two lines of code in the sf2 driver to delete the DT
property and that gets me going.

I still think there is value in being able to assign groups of user
ports to a specific CPU/management port (maybe even DSA, say if we
supported trunking/bonding at some point).
-- 
Florian