RFC: dsa: add support for multiple CPU ports

RFC: dsa: add support for multiple CPU ports

[Mathieu Olivari]

Hi all,

I’m writing a DSA driver for some QCA network switches already supported in
OpenWrt using swconfig. They connect to the CPU using MDIO for configuration,
and xMII ports for data. The main difference with what is supported comes from
the fact that most of these switches actually have multiple xMII connections to
the same CPU. Something like this:
(extending the picture from http://lwn.net/Articles/302333/)

	+-----------+       +-----------+
	|           | RGMII |           |
	|       eth0+-------+           +------ 1000baseT MDI ("WAN")
	|        wan|       |  7-port   +------ 1000baseT MDI ("LAN1")
	|   CPU     |       |  ethernet +------ 1000baseT MDI ("LAN2")
	|           | RGMII |  switch   +------ 1000baseT MDI ("LAN3")
	|       eth1+-------+  w/5 PHYs +------ 1000baseT MDI ("LAN4")
	|        lan|       |           |
	+-----------+       +-----------+
	          |   MDIO     |
	          \------------/

In a typical configuration, we configure the switch to isolate WAN & LAN from
each other.

The current DSA implementation is based on the assumption that there can be only
one xMII connection between the switch and the CPU (this switch port is known as
the CPU port), which obviously doesn't work anymore in that case. So I'm trying
to figure out the best way to extend DSA to support these hardware.

Here is my proposal. I was hoping to get feedback before i get too much down the
implementation path:
*add an optional phandle (cpu_dev) to the existing port device node, which would
 reference the ethernet device node this port is connected to
*save the corresponding net_device in the "ports" field of "struct dsa_switch"
*instrument the network layer to be able to look-up which switch port is
 connected to a particular net_device.

The third point would allow to perform operations currently limited to switch
ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB
statistics in addition to the eth0 statistics.

Thoughts?

Thanks,
Mathieu

Re: RFC: dsa: add support for multiple CPU ports

[Andrew Lunn]

On Tue, Mar 10, 2015 at 12:01:29PM -0700, Mathieu Olivari wrote:
> Hi all,
> 
> I???m writing a DSA driver for some QCA network switches already supported in
> OpenWrt using swconfig. They connect to the CPU using MDIO for configuration,
> and xMII ports for data. The main difference with what is supported comes from
> the fact that most of these switches actually have multiple xMII connections to
> the same CPU. Something like this:
> (extending the picture from http://lwn.net/Articles/302333/)
> 
> 	+-----------+       +-----------+
> 	|           | RGMII |           |
> 	|       eth0+-------+           +------ 1000baseT MDI ("WAN")
> 	|        wan|       |  7-port   +------ 1000baseT MDI ("LAN1")
> 	|   CPU     |       |  ethernet +------ 1000baseT MDI ("LAN2")
> 	|           | RGMII |  switch   +------ 1000baseT MDI ("LAN3")
> 	|       eth1+-------+  w/5 PHYs +------ 1000baseT MDI ("LAN4")
> 	|        lan|       |           |
> 	+-----------+       +-----------+
> 	          |   MDIO     |
> 	          \------------/
> 
> In a typical configuration, we configure the switch to isolate WAN & LAN from
> each other.

Hi Mathieu

By default, all DSA ports are isolated from each other. If you want to
join them together you need to setup a bridge and add the ports to the
bridge. There are patches being worked on to push this bridge state
down into the hardware, so the hardware will then bridge across these
ports, rather than having to do it in software. So long as you don't
add WAN to the bridge, it will be kept isolated.

I had a different solution in mind for multiple CPU ports. I've no
idea if it actually works though, i've not had time to investigate.
It would actually put the host CPU ports into a switch trunk, and use
team/bond driver on the host. You then get one logical 2Gbp link to
the switch and run DSA over that.

There have also been some patches to create trunks, but they were for
normal ports, not CPU ports. They should however be a good starting
point for what the switch driver needs to do to create a trunk towards
the CPU.

I think this scheme might also work without having to change the DSA
binding. There is nothing in the binding documentation that there can
only be one CPU port. So if two or more are found, the DSA framework
can do the trunking setup.

    Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Andrew Lunn]

> The third point would allow to perform operations currently limited to switch
> ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB
> statistics in addition to the eth0 statistics.

Hi Mathieu

This already works with DSA. I have a WRT1900AC which has a 7 port
switch. With DSA i have:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
    link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff

internet would be your WAN port in your diagram. I just named it the
same as the label on the case.

I can get MIB statistics in the normal way for the DSA ports, e.g:

root@wrt1900ac:~# ethtool -S lan1
NIC statistics:
     tx_packets: 153009
     tx_bytes: 13307317
     rx_packets: 161136
     rx_bytes: 21840683
...

	Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Mathieu Olivari]

On Tue, Mar 10, 2015 at 08:31:51PM +0100, Andrew Lunn wrote:
> > The third point would allow to perform operations currently limited to switch
> > ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB
> > statistics in addition to the eth0 statistics.
> 
> Hi Mathieu
> 
> This already works with DSA. I have a WRT1900AC which has a 7 port
> switch. With DSA i have:
> 
> # ip link show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
>     link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
> 4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> 5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> 6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> 7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> 8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> 
> internet would be your WAN port in your diagram. I just named it the
> same as the label on the case.
> 
> I can get MIB statistics in the normal way for the DSA ports, e.g:
> 
> root@wrt1900ac:~# ethtool -S lan1
> NIC statistics:
>      tx_packets: 153009
>      tx_bytes: 13307317
>      rx_packets: 161136
>      rx_bytes: 21840683
> ...
> 
> 	Andrew
> 
> 

I was thinking about the MIB counters, maintained by the switch, on the CPU
port(s). 

On this picture:

+-----------+       +--------------------+
|           | RGMII |                    |
|       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
|        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
|   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
|           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
|       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
|        lan|       |                    |
+-----------+       +--------------------+
          |     MDIO   |
          \------------/

We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU
ports don't have a net_device, there isn't a way to access MIB stats of
P0 & P6.

If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts,
we could use it to lookup which switch port is connected to this net_dev and
show the corresponding counters when running "ethtool -S ethN".

Re: RFC: dsa: add support for multiple CPU ports

[Andrew Lunn]

> I was thinking about the MIB counters, maintained by the switch, on the CPU
> port(s). 
> 
> On this picture:
> 
> +-----------+       +--------------------+
> |           | RGMII |                    |
> |       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
> |        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
> |   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
> |           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
> |       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
> |        lan|       |                    |
> +-----------+       +--------------------+
>           |     MDIO   |
>           \------------/
> 
> We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU
> ports don't have a net_device, there isn't a way to access MIB stats of
> P0 & P6.

Ah, sorry, i misunderstood you. As you say, the current DSA does not
give you access to these counters. Apart from these counters, are
there any other needs to have access to these ports? The core idea of
hardware acceleration is that it should look like linux as normal.  P0
and P6 are an internal detail, and you cannot actually use them. You
cannot send packets, they are something special. So unless there is a
good reason, we should not expose them.

> If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts,
> we could use it to lookup which switch port is connected to this net_dev and
> show the corresponding counters when running "ethtool -S ethN".

But that is not what the user expects. They expect the statistic
counters for ethN, not the switch port.

	 Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Mathieu Olivari]

On Tue, Mar 10, 2015 at 08:21:01PM +0100, Andrew Lunn wrote:
> On Tue, Mar 10, 2015 at 12:01:29PM -0700, Mathieu Olivari wrote:
> > Hi all,
> > 
> > I???m writing a DSA driver for some QCA network switches already supported in
> > OpenWrt using swconfig. They connect to the CPU using MDIO for configuration,
> > and xMII ports for data. The main difference with what is supported comes from
> > the fact that most of these switches actually have multiple xMII connections to
> > the same CPU. Something like this:
> > (extending the picture from http://lwn.net/Articles/302333/)
> > 
> > 	+-----------+       +-----------+
> > 	|           | RGMII |           |
> > 	|       eth0+-------+           +------ 1000baseT MDI ("WAN")
> > 	|        wan|       |  7-port   +------ 1000baseT MDI ("LAN1")
> > 	|   CPU     |       |  ethernet +------ 1000baseT MDI ("LAN2")
> > 	|           | RGMII |  switch   +------ 1000baseT MDI ("LAN3")
> > 	|       eth1+-------+  w/5 PHYs +------ 1000baseT MDI ("LAN4")
> > 	|        lan|       |           |
> > 	+-----------+       +-----------+
> > 	          |   MDIO     |
> > 	          \------------/
> > 
> > In a typical configuration, we configure the switch to isolate WAN & LAN from
> > each other.
> 
> Hi Mathieu
> 
> By default, all DSA ports are isolated from each other. If you want to
> join them together you need to setup a bridge and add the ports to the
> bridge. There are patches being worked on to push this bridge state
> down into the hardware, so the hardware will then bridge across these
> ports, rather than having to do it in software. So long as you don't
> add WAN to the bridge, it will be kept isolated.
> 
> I had a different solution in mind for multiple CPU ports. I've no
> idea if it actually works though, i've not had time to investigate.
> It would actually put the host CPU ports into a switch trunk, and use
> team/bond driver on the host. You then get one logical 2Gbp link to
> the switch and run DSA over that.
> 

I could see it working on the Tx path - as the destination port is specified
in the header -, but on the Rx path, how would the switch figure out which
CPU port it should send the packet to?

These switches doesn't have a concept of bonding, so this decision is generally
based on the internal ARL table, and is automatically learnt by looking at the
src MAC@ of the incoming packets.
When using bonding, the switch would see both eth0 & eth1 MAC@ on both of
its CPU ports. The destination CPU port would be unexpected at best; I could
see some switches being able to support this, but most of them would not.

Thoughts?

> There have also been some patches to create trunks, but they were for
> normal ports, not CPU ports. They should however be a good starting
> point for what the switch driver needs to do to create a trunk towards
> the CPU.
> 
> I think this scheme might also work without having to change the DSA
> binding. There is nothing in the binding documentation that there can
> only be one CPU port. So if two or more are found, the DSA framework
> can do the trunking setup.
> 

At the very least, we would need to treat "dsa,ethernet" as an array,
and specify the list of ethernet device node that connects to the switch.
I still think putting this information in the port section makes sense,
as it represents the board layout more accurately than having a global
node at a dsa level.

>     Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Andrew Lunn]

> > I had a different solution in mind for multiple CPU ports. I've no
> > idea if it actually works though, i've not had time to investigate.
> > It would actually put the host CPU ports into a switch trunk, and use
> > team/bond driver on the host. You then get one logical 2Gbp link to
> > the switch and run DSA over that.
> > 
> 
> I could see it working on the Tx path - as the destination port is specified
> in the header -, but on the Rx path, how would the switch figure out which
> CPU port it should send the packet to?
>
> These switches doesn't have a concept of bonding, so this decision is generally
> based on the internal ARL table, and is automatically learnt by looking at the
> src MAC@ of the incoming packets.

All Marvell switches, and according to Florian SF2 and B53 support
trunk groups. I assume the switch does load balancing between ports in
a trunk. On the CPU you can then use the header to determine which
port the packet came from.

> When using bonding, the switch would see both eth0 & eth1 MAC@ on both of
> its CPU ports. The destination CPU port would be unexpected at best; I could
> see some switches being able to support this, but most of them would not.

At the moment, this is purely an idea. I do have two boards where i
can test it out with Marvell devices, once i get some spare time.
Florian might be able to comment about SF2 and B53, if we thinks
trunking and broadcom tags can be combined.

> At the very least, we would need to treat "dsa,ethernet" as an array,
> and specify the list of ethernet device node that connects to the switch.

True.

> I still think putting this information in the port section makes sense,
> as it represents the board layout more accurately than having a global
> node at a dsa level.

Yes, it does make sense to have it in port. But to keep backwards
compatibility, we need at keep dsa,ethernet, with at least one
interface.

	Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Mathieu Olivari]

On Tue, Mar 10, 2015 at 10:13:15PM +0100, Andrew Lunn wrote:
> > I was thinking about the MIB counters, maintained by the switch, on the CPU
> > port(s). 
> > 
> > On this picture:
> > 
> > +-----------+       +--------------------+
> > |           | RGMII |                    |
> > |       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
> > |        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
> > |   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
> > |           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
> > |       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
> > |        lan|       |                    |
> > +-----------+       +--------------------+
> >           |     MDIO   |
> >           \------------/
> > 
> > We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU
> > ports don't have a net_device, there isn't a way to access MIB stats of
> > P0 & P6.
> 
> Ah, sorry, i misunderstood you. As you say, the current DSA does not
> give you access to these counters. Apart from these counters, are
> there any other needs to have access to these ports? The core idea of
> hardware acceleration is that it should look like linux as normal.  P0
> and P6 are an internal detail, and you cannot actually use them. You
> cannot send packets, they are something special. So unless there is a
> good reason, we should not expose them.
> 

That is true, but the switch does have information to share about this
port (stats). Having a standard way to retrieve this information would
be nice.

> > If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts,
> > we could use it to lookup which switch port is connected to this net_dev and
> > show the corresponding counters when running "ethtool -S ethN".
> 
> But that is not what the user expects. They expect the statistic
> counters for ethN, not the switch port.

The user would probably want to access both. It should not replace ethN
counters obviously, but these are real switch port (just without any PHY),
so the user would want to have the same level of information as it has on
the other logical switch ports.

I was wondering whether it made sense to show counters for both ethN and
P0 when running "ethtool -S ethN". Or maybe add a new argument to
ethtool. Else, there is still debugfs but it means the same counters
will be accessible through ethtool (for PHY ports) and debugfs (for MAC
ports), which is not very consistent.

> 
> 	 Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Florian Fainelli]

2015-03-10 13:42 GMT-07:00 Mathieu Olivari <mathieu@codeaurora.org>:
> On Tue, Mar 10, 2015 at 08:31:51PM +0100, Andrew Lunn wrote:
>> > The third point would allow to perform operations currently limited to switch
>> > ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB
>> > statistics in addition to the eth0 statistics.
>>
>> Hi Mathieu
>>
>> This already works with DSA. I have a WRT1900AC which has a 7 port
>> switch. With DSA i have:
>>
>> # ip link show
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
>>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
>> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
>>     link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
>> 4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
>>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
>> 5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
>>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
>> 6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
>>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
>> 7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
>>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
>> 8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
>>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
>>
>> internet would be your WAN port in your diagram. I just named it the
>> same as the label on the case.
>>
>> I can get MIB statistics in the normal way for the DSA ports, e.g:
>>
>> root@wrt1900ac:~# ethtool -S lan1
>> NIC statistics:
>>      tx_packets: 153009
>>      tx_bytes: 13307317
>>      rx_packets: 161136
>>      rx_bytes: 21840683
>> ...
>>
>>       Andrew
>>
>>
>
> I was thinking about the MIB counters, maintained by the switch, on the CPU
> port(s).
>
> On this picture:
>
> +-----------+       +--------------------+
> |           | RGMII |                    |
> |       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
> |        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
> |   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
> |           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
> |       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
> |        lan|       |                    |
> +-----------+       +--------------------+
>           |     MDIO   |
>           \------------/
>
> We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU
> ports don't have a net_device, there isn't a way to access MIB stats of
> P0 & P6.

That's right, we actually discussed that earlier during the swconfig
discussions, and one of the reasons why this is not exposed is because
it can be confusing for users, and the difference between your conduit
interfaces: eth0 or eth1, and their switch-facing interfaces: p0 or
p6, is that p0 and p6 are only useful for counters, but not for much
else, in particular not data passing.

>
> If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts,
> we could use it to lookup which switch port is connected to this net_dev and
> show the corresponding counters when running "ethtool -S ethN".

Once we have that, interfaces p0 and p6 are not going to be
sending/receiving traffic, their conduit interfaces: eth0 and eth1,
will, and those could also maintain statistics in their own MIB (when
using unmodified Ethernet MAC drivers). I agree this is useful in xMII
cases to debug interface problems between a switch and a CPU Ethernet
MAC for instance.
-- 
Florian

Re: RFC: dsa: add support for multiple CPU ports

[Mathieu Olivari]

On Tue, Mar 10, 2015 at 05:01:49PM -0700, Florian Fainelli wrote:
> 2015-03-10 13:42 GMT-07:00 Mathieu Olivari <mathieu@codeaurora.org>:
> > On Tue, Mar 10, 2015 at 08:31:51PM +0100, Andrew Lunn wrote:
> >> > The third point would allow to perform operations currently limited to switch
> >> > ports. Typically, "ethtool -S eth0" could return the corresponding port's MIB
> >> > statistics in addition to the eth0 statistics.
> >>
> >> Hi Mathieu
> >>
> >> This already works with DSA. I have a WRT1900AC which has a 7 port
> >> switch. With DSA i have:
> >>
> >> # ip link show
> >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
> >>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> >> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 532
> >>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> >> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 532
> >>     link/ether 52:33:3a:a7:c1:8a brd ff:ff:ff:ff:ff:ff
> >> 4: lan4@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
> >>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> >> 5: lan3@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
> >>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> >> 6: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
> >>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> >> 7: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
> >>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> >> 8: internet@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
> >>     link/ether 94:10:3e:80:bc:f3 brd ff:ff:ff:ff:ff:ff
> >>
> >> internet would be your WAN port in your diagram. I just named it the
> >> same as the label on the case.
> >>
> >> I can get MIB statistics in the normal way for the DSA ports, e.g:
> >>
> >> root@wrt1900ac:~# ethtool -S lan1
> >> NIC statistics:
> >>      tx_packets: 153009
> >>      tx_bytes: 13307317
> >>      rx_packets: 161136
> >>      rx_bytes: 21840683
> >> ...
> >>
> >>       Andrew
> >>
> >>
> >
> > I was thinking about the MIB counters, maintained by the switch, on the CPU
> > port(s).
> >
> > On this picture:
> >
> > +-----------+       +--------------------+
> > |           | RGMII |                    |
> > |       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
> > |        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
> > |   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
> > |           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
> > |       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
> > |        lan|       |                    |
> > +-----------+       +--------------------+
> >           |     MDIO   |
> >           \------------/
> >
> > We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU
> > ports don't have a net_device, there isn't a way to access MIB stats of
> > P0 & P6.
> 
> That's right, we actually discussed that earlier during the swconfig
> discussions, and one of the reasons why this is not exposed is because
> it can be confusing for users, and the difference between your conduit
> interfaces: eth0 or eth1, and their switch-facing interfaces: p0 or
> p6, is that p0 and p6 are only useful for counters, but not for much
> else, in particular not data passing.
> 

When there is only one CPU port, the connection between CPU port and the
other switch ports is implicit obviously. But if there is more, and if
the switch doesn't support bonding, the user would probably expect to
configure it & define how P0/P6 connects to the other ports.

We already have existing interface for eth0/eth1. Maybe we should
consider allowing them to be bridge. User configuration would look like
this:

brctl addbr br-lan
brctl addif br-lan eth1
brctl addif br-lan lan1
brctl addif br-lan lan2
brctl addif br-lan lan3
brctl addif br-lan lan4

> >
> > If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts,
> > we could use it to lookup which switch port is connected to this net_dev and
> > show the corresponding counters when running "ethtool -S ethN".
> 
> Once we have that, interfaces p0 and p6 are not going to be
> sending/receiving traffic, their conduit interfaces: eth0 and eth1,
> will, and those could also maintain statistics in their own MIB (when
> using unmodified Ethernet MAC drivers). I agree this is useful in xMII
> cases to debug interface problems between a switch and a CPU Ethernet
> MAC for instance.
> -- 
> Florian

Re: RFC: dsa: add support for multiple CPU ports

[David Miller]

From: Mathieu Olivari <mathieu@codeaurora.org>
Date: Tue, 10 Mar 2015 15:53:26 -0700

> I was wondering whether it made sense to show counters for both ethN and
> P0 when running "ethtool -S ethN". Or maybe add a new argument to
> ethtool. Else, there is still debugfs but it means the same counters
> will be accessible through ethtool (for PHY ports) and debugfs (for MAC
> ports), which is not very consistent.

debugfs is only for things that are throw away and upon which userspace
cannot depend.

Re: RFC: dsa: add support for multiple CPU ports

[Jiri Pirko]

Tue, Mar 10, 2015 at 10:13:15PM CET, andrew@lunn.ch wrote:
>> I was thinking about the MIB counters, maintained by the switch, on the CPU
>> port(s). 
>> 
>> On this picture:
>> 
>> +-----------+       +--------------------+
>> |           | RGMII |                    |
>> |       eth0+-------+ P0              P1 +------ 1000baseT MDI ("WAN")
>> |        wan|       |      7-port     P2 +------ 1000baseT MDI ("LAN1")
>> |   CPU     |       |      ethernet   P3 +------ 1000baseT MDI ("LAN2")
>> |           | RGMII |      switch     P4 +------ 1000baseT MDI ("LAN3")
>> |       eth1+-------+ P6   w/5 PHYs   P5 +------ 1000baseT MDI ("LAN4")
>> |        lan|       |                    |
>> +-----------+       +--------------------+
>>           |     MDIO   |
>>           \------------/
>> 
>> We can see MIB stats of P1/P2/P3/P4/P5 by using ethtool -S. But as CPU
>> ports don't have a net_device, there isn't a way to access MIB stats of
>> P0 & P6.
>
>Ah, sorry, i misunderstood you. As you say, the current DSA does not
>give you access to these counters. Apart from these counters, are
>there any other needs to have access to these ports? The core idea of
>hardware acceleration is that it should look like linux as normal.  P0
>and P6 are an internal detail, and you cannot actually use them. You
>cannot send packets, they are something special. So unless there is a
>good reason, we should not expose them.

I completely agree. Let's keep this internal ports scop within the
driver/dsa code.


>
>> If we have the eth0 <--> P0 and eth1 <--> P6 relationship information in dts,
>> we could use it to lookup which switch port is connected to this net_dev and
>> show the corresponding counters when running "ethtool -S ethN".
>
>But that is not what the user expects. They expect the statistic
>counters for ethN, not the switch port.
>
>	 Andrew
>--
>To unsubscribe from this list: send the line "unsubscribe netdev" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RFC: dsa: add support for multiple CPU ports

[Andrew Lunn]

> We already have existing interface for eth0/eth1. Maybe we should
> consider allowing them to be bridge. User configuration would look like
> this:
> 
> brctl addbr br-lan
> brctl addif br-lan eth1
> brctl addif br-lan lan1
> brctl addif br-lan lan2
> brctl addif br-lan lan3
> brctl addif br-lan lan4

Think about this from the perspective of somebody how does not know
there is a switch.  When i look at this, to me it means packets from
lan1, lan2, lan3, lan4 and eth1 are bridged together. So we are going
to get packets sent out eth1 without a tag on it, and the switch is
very likely to drop it, since the port is in a mode which expects a
tag. You are not using the brctl command with its normal meaning,
which is bad.

Configuring this at the bridge layer is also wrong. What conduit a DSA
slave interfaces uses is a DSA layer issue.

Before we can come up with a nice solution, i think we first need to
understand what the switches are capable off. If trunking does work,
it is a relatively nice system for the user to configure, in that
there is nothing for the user to configure! Assuming the switch can do
reasonable load balancing, we also get near optimal usage of the two
conduits.

I don't know when i will have time to play with this, and if somebody
comes along with a better idea, i'm very open to adopting that idea
over mine.

      Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Mathieu Olivari]

On Wed, Mar 11, 2015 at 02:30:57PM +0100, Andrew Lunn wrote:
> > We already have existing interface for eth0/eth1. Maybe we should
> > consider allowing them to be bridge. User configuration would look like
> > this:
> > 
> > brctl addbr br-lan
> > brctl addif br-lan eth1
> > brctl addif br-lan lan1
> > brctl addif br-lan lan2
> > brctl addif br-lan lan3
> > brctl addif br-lan lan4
> 
> Think about this from the perspective of somebody how does not know
> there is a switch.  When i look at this, to me it means packets from
> lan1, lan2, lan3, lan4 and eth1 are bridged together. So we are going
> to get packets sent out eth1 without a tag on it, and the switch is
> very likely to drop it, since the port is in a mode which expects a
> tag. You are not using the brctl command with its normal meaning,
> which is bad.
> 
> Configuring this at the bridge layer is also wrong. What conduit a DSA
> slave interfaces uses is a DSA layer issue.
> 
> Before we can come up with a nice solution, i think we first need to
> understand what the switches are capable off. If trunking does work,
> it is a relatively nice system for the user to configure, in that
> there is nothing for the user to configure! Assuming the switch can do
> reasonable load balancing, we also get near optimal usage of the two
> conduits.
> 

Agreed. The switches I'm talking about (QCA8xxx/AR8xxx) don't support
a mode which would allow them to use all the CPU ports as one logical
bonding interface. I'm not sure how many of the switches in OpenWrt do.

I know some of these switches don't even support tagging. So I'd be
surprised if they're all capable of bonding.

The base usage scenario for these switches is using port based vlan.
The user knows very well there is a switch (OpenWrt even has a webpage
for it), and he provides a static configuration by telling it how he
want the ports to be connected to each other.
In the previous example, it would be something like creating 2 logical
switches: P0/P1 and P2/P3/P4/P5/P6. Or it could be something like
(lan1 - P0/P1/P2) and (lan2 - P3/P4) and (lan3 - P5/P6). Or anything else.
That's the mode that OpenWrt is using in pretty much all its target. It is
also the way they get configured in most home routers in general.

I understand the value of using bonding. But we should also have a way to
configure more simple hardware - which is typically using port based vlan.

> I don't know when i will have time to play with this, and if somebody
> comes along with a better idea, i'm very open to adopting that idea
> over mine.

I think DSA should probably support both. I think it should allow
bonding for switches who support it, but it should also allow for port
based vlan mapped by the user.

Another way to support these could be to make the cpu port "optional".
These switches would have no cpu port, all the ports would have a
net_device (including the xMII ones). They would get bridged, and
packets that are sent to interface connected to cpu would be dropped by
the switch xmit function.
I didn't try that, but it should work.

> 
>       Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Andrew Lunn]

> Agreed. The switches I'm talking about (QCA8xxx/AR8xxx) don't support
> a mode which would allow them to use all the CPU ports as one logical
> bonding interface.

I had a 60 second look at https://www.codeaurora.org/cgit/quic/qsdk/oss/lklm/qca-ssdk/

It looks like dess, isis, and isisc have some support for trunks, and
the rest do not.

> I know some of these switches don't even support tagging. So I'd be
> surprised if they're all capable of bonding.

So we are going to end up with multiple solutions depending on what
the hardware is capable of.

> I think DSA should probably support both. I think it should allow
> bonding for switches who support it, but it should also allow for port
> based vlan mapped by the user.

So the question is, how do you make the existing concepts in Linux
support this, so configuration is how you expect it to be, without
special cases, oddities, etc.

	Andrew

Re: RFC: dsa: add support for multiple CPU ports

[Mathieu Olivari]

On Thu, Mar 12, 2015 at 01:19:26AM +0100, Andrew Lunn wrote:
> > Agreed. The switches I'm talking about (QCA8xxx/AR8xxx) don't support
> > a mode which would allow them to use all the CPU ports as one logical
> > bonding interface.
> 
> I had a 60 second look at https://www.codeaurora.org/cgit/quic/qsdk/oss/lklm/qca-ssdk/
> 
> It looks like dess, isis, and isisc have some support for trunks, and
> the rest do not.
> 

Yes, they do. But not on the CPU port, unfortunately.

> > I know some of these switches don't even support tagging. So I'd be
> > surprised if they're all capable of bonding.
> 
> So we are going to end up with multiple solutions depending on what
> the hardware is capable of.
> 
> > I think DSA should probably support both. I think it should allow
> > bonding for switches who support it, but it should also allow for port
> > based vlan mapped by the user.
> 
> So the question is, how do you make the existing concepts in Linux
> support this, so configuration is how you expect it to be, without
> special cases, oddities, etc.
> 

I'll start by ignoring the second CPU port for now and see how far I go.

> 	Andrew