DSA vs. SWTICHDEV ?

DSA vs. SWTICHDEV ?

[Joakim Tjernlund]

I am trying to wrap my head around these two "devices" and have a hard time telling them apart.
We are looking att adding a faily large switch(over PCIe) to our board and from what I can tell
switchdev is the new way to do it but DSA is still there. Is it possible to just list
how they differ?

What can switchdev do that DSA cannot?

What can DSA do that switchdev cannot?


Can one enable switchdev and dsa for the same switch device?

 Jocke 

Re: DSA vs. SWTICHDEV ?

[Andrew Lunn]

On Wed, Nov 30, 2016 at 08:50:34AM +0000, Joakim Tjernlund wrote:
> I am trying to wrap my head around these two "devices" and have a hard time telling them apart.
> We are looking att adding a faily large switch(over PCIe) to our board and from what I can tell
> switchdev is the new way to do it but DSA is still there. Is it possible to just list
> how they differ?

Hi Joakim

If the interface you use to send frames from the host to the switch is
PCIe, you probably want to use switchdev directly.

DSA devices all use a host Ethernet interface to send frames to the
switch. DSA sits under switchdev, and effectively provides a lot of
the common stuff needed for implementing switch drivers of this
sort. It creates the slave interfaces, links the MAC to the PHY, has
one uniform device tree binding which all DSA switches have, deals
with encapsulation/decapsulating frames sent over the master device,
etc.

	    Andrew

Re: DSA vs. SWTICHDEV ?

[Joakim Tjernlund]

On Wed, 2016-11-30 at 14:52 +0100, Andrew Lunn wrote:
> On Wed, Nov 30, 2016 at 08:50:34AM +0000, Joakim Tjernlund wrote:
> > I am trying to wrap my head around these two "devices" and have a hard time telling them apart.
> > We are looking att adding a faily large switch(over PCIe) to our board and from what I can tell
> > switchdev is the new way to do it but DSA is still there. Is it possible to just list
> > how they differ?
> 
> Hi Joakim

Hi Andrew, thanks for answering

> 
> If the interface you use to send frames from the host to the switch is
> PCIe, you probably want to use switchdev directly.

OK, we will have a few ethernet I/F's connected too but I these should be used
as normal interfaces just interfacing a switch.

> 
> DSA devices all use a host Ethernet interface to send frames to the
> switch. DSA sits under switchdev, and effectively provides a lot of
> the common stuff needed for implementing switch drivers of this
> sort. It creates the slave interfaces, links the MAC to the PHY, has
> one uniform device tree binding which all DSA switches have, deals
> with encapsulation/decapsulating frames sent over the master device,
> etc.

And switchdev can do all this over PCIe instead? Can you have a switch tree in switchdev too?

We probably need to push a custom tag for final step (after the switch) and I think that
should be doable with a custom VLAN driver over one/several of switchdev's virtual
interfaces? 

 Joakim

Re: DSA vs. SWTICHDEV ?

[Andrew Lunn]

On Wed, Nov 30, 2016 at 02:30:43PM +0000, Joakim Tjernlund wrote:
> On Wed, 2016-11-30 at 14:52 +0100, Andrew Lunn wrote:
> > On Wed, Nov 30, 2016 at 08:50:34AM +0000, Joakim Tjernlund wrote:
> > > I am trying to wrap my head around these two "devices" and have a hard time telling them apart.
> > > We are looking att adding a faily large switch(over PCIe) to our board and from what I can tell
> > > switchdev is the new way to do it but DSA is still there. Is it possible to just list
> > > how they differ?
> > 
> > Hi Joakim
> 
> Hi Andrew, thanks for answering
> 
> > 
> > If the interface you use to send frames from the host to the switch is
> > PCIe, you probably want to use switchdev directly.
> 
> OK, we will have a few ethernet I/F's connected too but I these should be used
> as normal interfaces just interfacing a switch.

That does not make much sense.

Maybe time to backtrack a bit. The Linux concept for switch/router
chips is that they are just hardware accelerators for what Linux can
already do in software. Each port of the switch is just a normal Linux
interface.  ip link show will list each port. ip addr add can be used
to add an IP address to the interface.  You want to switch frames
between two ports: Create a linux bridge and put the interfaces into
it. Via switchdev you get a call into the hardware to accelerate
this. If the hardware cannot accelerate it, it is done in software as
normal.  Want to combine two ports into a trunk: Add a team interface
and make the port interfaces slaves of the team interface. Via
switchdev, you ask the hardware to accelerate this. If it cannot, it
is done in software.

So back your connecting a few host interfaces to the switch. This is
logically putting a cable between two interfaces on the same host. You
are making a loopback. Why do that? Sure it is possible, but it is an
odd architecture.

> And switchdev can do all this over PCIe instead? Can you have a
> switch tree in switchdev too?

Mellonex says so, but i don't think they have actually implemented it.

	 Andrew

Re: DSA vs. SWTICHDEV ?

[Joakim Tjernlund]

On Wed, 2016-11-30 at 16:25 +0100, Andrew Lunn wrote:
> On Wed, Nov 30, 2016 at 02:30:43PM +0000, Joakim Tjernlund wrote:
> > On Wed, 2016-11-30 at 14:52 +0100, Andrew Lunn wrote:
> > > On Wed, Nov 30, 2016 at 08:50:34AM +0000, Joakim Tjernlund wrote:
> > > > I am trying to wrap my head around these two "devices" and have a hard time telling them apart.
> > > > We are looking att adding a faily large switch(over PCIe) to our board and from what I can tell
> > > > switchdev is the new way to do it but DSA is still there. Is it possible to just list
> > > > how they differ?
> > > 
> > > Hi Joakim
> > 
> > Hi Andrew, thanks for answering
> > 
> > > 
> > > If the interface you use to send frames from the host to the switch is
> > > PCIe, you probably want to use switchdev directly.
> > 
> > OK, we will have a few ethernet I/F's connected too but I these should be used
> > as normal interfaces just interfacing a switch.
> 
> That does not make much sense.
> 
> Maybe time to backtrack a bit. The Linux concept for switch/router
> chips is that they are just hardware accelerators for what Linux can
> already do in software. Each port of the switch is just a normal Linux
> interface.  ip link show will list each port. ip addr add can be used
> to add an IP address to the interface.  You want to switch frames
> between two ports: Create a linux bridge and put the interfaces into
> it. Via switchdev you get a call into the hardware to accelerate
> this. If the hardware cannot accelerate it, it is done in software as
> normal.  Want to combine two ports into a trunk: Add a team interface
> and make the port interfaces slaves of the team interface. Via
> switchdev, you ask the hardware to accelerate this. If it cannot, it
> is done in software.
> 
> So back your connecting a few host interfaces to the switch. This is
> logically putting a cable between two interfaces on the same host. You
> are making a loopback. Why do that? Sure it is possible, but it is an
> odd architecture.

This is an embedded system with several boards in a subrack.
Each board has eth I/F connected to a switch to communicate with each other.
One of the board will also house the actual switch device and manage the switch.
Then the normal app just communicates over the physical eth I/F like any other board
in the system. There is a "manage switch app" which brings the switch up and partition
phys VLANs etc. (each phys I/F would be a a separate domain so no loop)

I guess I could skip the phys I/F and have the switch app create a virtual eth0 I/F over PCIe
instead to save eth MACS but the above is safer should there be some problems/limitations in
swicthdev plus switchdev does not exist in u-boot so it would be a lot of effort to
get a working eth I/F inside u-boot.
I can still can still create a bridge I/F etc. should I need to.
Does the above make sense to you ?
 
> 
> > And switchdev can do all this over PCIe instead? Can you have a
> > switch tree in switchdev too?
> 
> Mellonex says so, but i don't think they have actually implemented it.

Not impl. any of DSAs features? What can you do with a Mellonex switch then?

 Jocke

Re: DSA vs. SWTICHDEV ?

[Andrew Lunn]

> This is an embedded system with several boards in a subrack.
> Each board has eth I/F connected to a switch to communicate with each other.
> One of the board will also house the actual switch device and manage the switch.
> Then the normal app just communicates over the physical eth I/F like any other board
> in the system. There is a "manage switch app" which brings the switch up and partition
> phys VLANs etc. (each phys I/F would be a a separate domain so no loop)

So you are planning on throwing away the "manage switch app", and just
use standard linux networking commands? That is what switchdev is all
about really, throwing away the vendor SDK for the switch, making a
switch just a bunch on interfaces on the host which you manage as
normal interfaces.

> I guess I could skip the phys I/F and have the switch app create a virtual eth0 I/F over PCIe

No need to create this interface. It will exist if you go the
switchdev route.

> > > And switchdev can do all this over PCIe instead? Can you have a
> > > switch tree in switchdev too?
> > 
> > Mellonex says so, but i don't think they have actually implemented it.
> 
> Not impl. any of DSAs features? What can you do with a Mellonex switch then?

They don't have a tree of switches, as far as i know. Just a single
switch. But DSA does support a tree of switches, that is what the D in
DSA means, distributed. And there are a couple of boards which have 2
to 4 switches in a tree.

I think this is partially down to market segments. Mellonex market is
top of rack switches. High port count, very high bandwidth. DSA is
more wireless access points, set top boxes, generally up to 7 ports of
1Gbps and a few custom embedded products which need more ports, so
build a tree of switches.

      Andrew

Re: DSA vs. SWTICHDEV ?

[Joakim Tjernlund]

On Wed, 2016-11-30 at 17:55 +0100, Andrew Lunn wrote:
> > This is an embedded system with several boards in a subrack.
> > Each board has eth I/F connected to a switch to communicate with each other.
> > One of the board will also house the actual switch device and manage the switch.
> > Then the normal app just communicates over the physical eth I/F like any other board
> > in the system. There is a "manage switch app" which brings the switch up and partition
> > phys VLANs etc. (each phys I/F would be a a separate domain so no loop)
> 
> So you are planning on throwing away the "manage switch app", and just
> use standard linux networking commands? That is what switchdev is all
> about really, throwing away the vendor SDK for the switch, making a
> switch just a bunch on interfaces on the host which you manage as
> normal interfaces.

Something like that. I need to run routing protocols on the switch I/Fs and egress
pkgs on selected switch I/Fs bypassing ARP, just like DSA does with its vendor
tags.

> 
> > I guess I could skip the phys I/F and have the switch app create a virtual eth0 I/F over PCIe
> 
> No need to create this interface. It will exist if you go the
> switchdev route.
> 
> > > > And switchdev can do all this over PCIe instead? Can you have a
> > > > switch tree in switchdev too?
> > > 
> > > Mellonex says so, but i don't think they have actually implemented it.
> > 
> > Not impl. any of DSAs features? What can you do with a Mellonex switch then?
> 
> They don't have a tree of switches, as far as i know. Just a single
> switch. But DSA does support a tree of switches, that is what the D in
> DSA means, distributed. And there are a couple of boards which have 2
> to 4 switches in a tree.

We might have a tree as well so now I really wonder: Given we write a
proper switchdev driver, can it support switchtrees without touching
switchdev infra structure? If not I guess we will attach a physical
eth I/F to the switch and use both DSA and switchdev to support both trees
and HW offload. 

> 
> I think this is partially down to market segments. Mellonex market is
> top of rack switches. High port count, very high bandwidth. DSA is
> more wireless access points, set top boxes, generally up to 7 ports of
> 1Gbps and a few custom embedded products which need more ports, so
> build a tree of switches.

We have on an existing board with a BCM ROBO switch with lots of ports(>24),
managed over SPI. Looking at BCM DSA tag code it looks like it only supports
some 8 ports or so. I still have to find out if this is a limitation in BCM tagging
protocol or if just not impl. in DSA yet.

Re: DSA vs. SWTICHDEV ?

[Andrew Lunn]

> Something like that. I need to run routing protocols on the switch I/Fs and egress
> pkgs on selected switch I/Fs bypassing ARP, just like DSA does with its vendor
> tags.

Does the switch have an equivalent tagging protocol? If you are
building a tree of switches you need something like this for frames
going from the host via intermediate switches and out a specific port
on a remote switch.

> We might have a tree as well so now I really wonder: Given we write a
> proper switchdev driver, can it support switchtrees without touching
> switchdev infra structure?

Jiri Pirko <jiri@resnulli.us> is probably the best person to ask about
this. DSA hides the knowledge that there is multiple switches. To
switchdev, a tree of switches looks like one switch. This is not
because of switchdev, it is just the existing DSA code worked when
switchdev came along.

 If not I guess we will attach a physical
> eth I/F to the switch and use both DSA and switchdev to support both trees
> and HW offload. 

This only works if the switch has the necessary tagging protocol to
pass through multiple switches.

> We have on an existing board with a BCM ROBO switch with lots of ports(>24),
> managed over SPI. Looking at BCM DSA tag code it looks like it only supports
> some 8 ports or so. I still have to find out if this is a limitation in BCM tagging
> protocol or if just not impl. in DSA yet.

Hi Florian, care to comment?

As far as i understand, the tag used for SF2 and B53 does not support
a tree of switches. But the big ROBO switches might have a different
tagging protocol.

  Andrew

Re: DSA vs. SWTICHDEV ?

[Jiri Pirko]

Wed, Nov 30, 2016 at 07:09:27PM CET, andrew@lunn.ch wrote:
>> Something like that. I need to run routing protocols on the switch I/Fs and egress
>> pkgs on selected switch I/Fs bypassing ARP, just like DSA does with its vendor
>> tags.
>
>Does the switch have an equivalent tagging protocol? If you are
>building a tree of switches you need something like this for frames
>going from the host via intermediate switches and out a specific port
>on a remote switch.
>
>> We might have a tree as well so now I really wonder: Given we write a
>> proper switchdev driver, can it support switchtrees without touching
>> switchdev infra structure?
>
>Jiri Pirko <jiri@resnulli.us> is probably the best person to ask about
>this. DSA hides the knowledge that there is multiple switches. To
>switchdev, a tree of switches looks like one switch. This is not
>because of switchdev, it is just the existing DSA code worked when
>switchdev came along.

Looks like the hw is DSA-ish. If I'm not mistaken about that, should be
handled as a part of DSA.


>
> If not I guess we will attach a physical
>> eth I/F to the switch and use both DSA and switchdev to support both trees
>> and HW offload. 
>
>This only works if the switch has the necessary tagging protocol to
>pass through multiple switches.
>
>> We have on an existing board with a BCM ROBO switch with lots of ports(>24),
>> managed over SPI. Looking at BCM DSA tag code it looks like it only supports
>> some 8 ports or so. I still have to find out if this is a limitation in BCM tagging
>> protocol or if just not impl. in DSA yet.
>
>Hi Florian, care to comment?
>
>As far as i understand, the tag used for SF2 and B53 does not support
>a tree of switches. But the big ROBO switches might have a different
>tagging protocol.
>
>  Andrew

Re: DSA vs. SWTICHDEV ?

[Florian Fainelli]

On 11/30/2016 09:44 AM, Joakim Tjernlund wrote:
> On Wed, 2016-11-30 at 17:55 +0100, Andrew Lunn wrote:
>>> This is an embedded system with several boards in a subrack.
>>> Each board has eth I/F connected to a switch to communicate with each other.
>>> One of the board will also house the actual switch device and manage the switch.
>>> Then the normal app just communicates over the physical eth I/F like any other board
>>> in the system. There is a "manage switch app" which brings the switch up and partition
>>> phys VLANs etc. (each phys I/F would be a a separate domain so no loop)
>>
>> So you are planning on throwing away the "manage switch app", and just
>> use standard linux networking commands? That is what switchdev is all
>> about really, throwing away the vendor SDK for the switch, making a
>> switch just a bunch on interfaces on the host which you manage as
>> normal interfaces.
> 
> Something like that. I need to run routing protocols on the switch I/Fs and egress
> pkgs on selected switch I/Fs bypassing ARP, just like DSA does with its vendor
> tags.
> 
>>
>>> I guess I could skip the phys I/F and have the switch app create a virtual eth0 I/F over PCIe
>>
>> No need to create this interface. It will exist if you go the
>> switchdev route.
>>
>>>>> And switchdev can do all this over PCIe instead? Can you have a
>>>>> switch tree in switchdev too?
>>>>
>>>> Mellonex says so, but i don't think they have actually implemented it.
>>>
>>> Not impl. any of DSAs features? What can you do with a Mellonex switch then?
>>
>> They don't have a tree of switches, as far as i know. Just a single
>> switch. But DSA does support a tree of switches, that is what the D in
>> DSA means, distributed. And there are a couple of boards which have 2
>> to 4 switches in a tree.
> 
> We might have a tree as well so now I really wonder: Given we write a
> proper switchdev driver, can it support switchtrees without touching
> switchdev infra structure? If not I guess we will attach a physical
> eth I/F to the switch and use both DSA and switchdev to support both trees
> and HW offload. 

switchdev in itself really is the glue layer between the networking
stack and how to push specific objects down to the Ethernet switch
driver, and that Ethernet switch driver. Switchdev does not enforce a
specific network device driver model object, and just provides standard
hooks for your network devices to register with switchdev in order to
push/receive offloads. DSA on the other hand, utilizes switchdev to get
notifications about offloads from the networking stack, but also exposes
a clearly and well defined Ethernet switch device driver model, as
Andrew described, it creates per-port network devices, binds the ports
to their PHYs (built-in, or external), and also takes care of
encapsulating/decapsulating the switch specific tagging protocol.

We should probably put that in some crystal clear sentence somewhere in
Documentation/networking/ but switchdev and DSA are complementary and
not competitors, they just do not tackle the problems from the same angle.

> 
>>
>> I think this is partially down to market segments. Mellonex market is
>> top of rack switches. High port count, very high bandwidth. DSA is
>> more wireless access points, set top boxes, generally up to 7 ports of
>> 1Gbps and a few custom embedded products which need more ports, so
>> build a tree of switches.
> 
> We have on an existing board with a BCM ROBO switch with lots of ports(>24),
> managed over SPI. Looking at BCM DSA tag code it looks like it only supports
> some 8 ports or so. I still have to find out if this is a limitation in BCM tagging
> protocol or if just not impl. in DSA yet.

Oh cool, can you share the model by chance? I suspect the tagging format
of that switch is going to be different than what net/dsa/tag_brcm.c, so
feel free to add something NET_DSA_TAG_BRCM8B (for 8 bytes) or something
like that.

Note that DSA currently hardcodes the maximum number of ports to 14
(DSA_MAX_PORTS), but this should obviously be something dynamically
determined based on probing the switch device.

Can you also evaluate if using drivers/net/dsa/b53/ would work for you?
My hope would be that they preserved the register compatibility here,
but since this has a large number of ports, it may have completely
offset most registers.

BTW, there is #linuxswitch on Freenode if you want to chat!
-- 
Florian

Re: DSA vs. SWTICHDEV ?

[Joakim Tjernlund]

On Wed, 2016-11-30 at 10:10 -0800, Florian Fainelli wrote:
> On 11/30/2016 09:44 AM, Joakim Tjernlund wrote:
> > On Wed, 2016-11-30 at 17:55 +0100, Andrew Lunn wrote:
> > > > This is an embedded system with several boards in a subrack.
> > > > Each board has eth I/F connected to a switch to communicate with each other.
> > > > One of the board will also house the actual switch device and manage the switch.
> > > > Then the normal app just communicates over the physical eth I/F like any other board
> > > > in the system. There is a "manage switch app" which brings the switch up and partition
> > > > phys VLANs etc. (each phys I/F would be a a separate domain so no loop)
> > > 
> > > So you are planning on throwing away the "manage switch app", and just
> > > use standard linux networking commands? That is what switchdev is all
> > > about really, throwing away the vendor SDK for the switch, making a
> > > switch just a bunch on interfaces on the host which you manage as
> > > normal interfaces.
> > 
> > Something like that. I need to run routing protocols on the switch I/Fs and egress
> > pkgs on selected switch I/Fs bypassing ARP, just like DSA does with its vendor
> > tags.
> > 
> > > 
> > > > I guess I could skip the phys I/F and have the switch app create a virtual eth0 I/F over PCIe
> > > 
> > > No need to create this interface. It will exist if you go the
> > > switchdev route.
> > > 
> > > > > > And switchdev can do all this over PCIe instead? Can you have a
> > > > > > switch tree in switchdev too?
> > > > > 
> > > > > Mellonex says so, but i don't think they have actually implemented it.
> > > > 
> > > > Not impl. any of DSAs features? What can you do with a Mellonex switch then?
> > > 
> > > They don't have a tree of switches, as far as i know. Just a single
> > > switch. But DSA does support a tree of switches, that is what the D in
> > > DSA means, distributed. And there are a couple of boards which have 2
> > > to 4 switches in a tree.
> > 
> > We might have a tree as well so now I really wonder: Given we write a
> > proper switchdev driver, can it support switchtrees without touching
> > switchdev infra structure? If not I guess we will attach a physical
> > eth I/F to the switch and use both DSA and switchdev to support both trees
> > and HW offload. 
> 
> switchdev in itself really is the glue layer between the networking
> stack and how to push specific objects down to the Ethernet switch
> driver, and that Ethernet switch driver. Switchdev does not enforce a
> specific network device driver model object, and just provides standard
> hooks for your network devices to register with switchdev in order to
> push/receive offloads. DSA on the other hand, utilizes switchdev to get
> notifications about offloads from the networking stack, but also exposes
> a clearly and well defined Ethernet switch device driver model, as
> Andrew described, it creates per-port network devices, binds the ports
> to their PHYs (built-in, or external), and also takes care of
> encapsulating/decapsulating the switch specific tagging protocol.

Lets see, switchdev is mainly for offloading L2/L3 into HW and does NOT create
virtual I/F(one for each phys sw port) so if my only goal is to offload I don't
need DSA? (How do one create routes if no virtual I/Fs I wonder ..) 

DSA then does create virt. I/Fs and manages switch trees, to actually tx pkg it needs
a phys I/F using vendor specific tags(for PCIe connected switches I can envision using
the PCIe connection instead).

Who is the master when using both DSA and switchdev w.r.t initil conf/bring up of switch?
configuring VLANs?

> 
> We should probably put that in some crystal clear sentence somewhere in
> Documentation/networking/ but switchdev and DSA are complementary and
> not competitors, they just do not tackle the problems from the same angle.

Yes :)

> 
> > 
> > > 
> > > I think this is partially down to market segments. Mellonex market is
> > > top of rack switches. High port count, very high bandwidth. DSA is
> > > more wireless access points, set top boxes, generally up to 7 ports of
> > > 1Gbps and a few custom embedded products which need more ports, so
> > > build a tree of switches.
> > 
> > We have on an existing board with a BCM ROBO switch with lots of ports(>24),
> > managed over SPI. Looking at BCM DSA tag code it looks like it only supports
> > some 8 ports or so. I still have to find out if this is a limitation in BCM tagging
> > protocol or if just not impl. in DSA yet.
> 
> Oh cool, can you share the model by chance? I suspect the tagging format

I think it is an BCM5322

> of that switch is going to be different than what net/dsa/tag_brcm.c, so
> feel free to add something NET_DSA_TAG_BRCM8B (for 8 bytes) or something
> like that.

I will have to look at that, don't have an docs handy.

> 
> Note that DSA currently hardcodes the maximum number of ports to 14
> (DSA_MAX_PORTS), but this should obviously be something dynamically
> determined based on probing the switch device.
> 
> Can you also evaluate if using drivers/net/dsa/b53/ would work for you?
> My hope would be that they preserved the register compatibility here,
> but since this has a large number of ports, it may have completely
> offset most registers.

I will have a look, I sure want to try out DSA/switchdev on existing boards if I can :)

> 
> BTW, there is #linuxswitch on Freenode if you want to chat!

Re: DSA vs. SWTICHDEV ?

[Florian Fainelli]

On 11/30/2016 10:44 AM, Joakim Tjernlund wrote:
> On Wed, 2016-11-30 at 10:10 -0800, Florian Fainelli wrote:
>> On 11/30/2016 09:44 AM, Joakim Tjernlund wrote:
>>> On Wed, 2016-11-30 at 17:55 +0100, Andrew Lunn wrote:
>>>>> This is an embedded system with several boards in a subrack.
>>>>> Each board has eth I/F connected to a switch to communicate with each other.
>>>>> One of the board will also house the actual switch device and manage the switch.
>>>>> Then the normal app just communicates over the physical eth I/F like any other board
>>>>> in the system. There is a "manage switch app" which brings the switch up and partition
>>>>> phys VLANs etc. (each phys I/F would be a a separate domain so no loop)
>>>>
>>>> So you are planning on throwing away the "manage switch app", and just
>>>> use standard linux networking commands? That is what switchdev is all
>>>> about really, throwing away the vendor SDK for the switch, making a
>>>> switch just a bunch on interfaces on the host which you manage as
>>>> normal interfaces.
>>>
>>> Something like that. I need to run routing protocols on the switch I/Fs and egress
>>> pkgs on selected switch I/Fs bypassing ARP, just like DSA does with its vendor
>>> tags.
>>>
>>>>
>>>>> I guess I could skip the phys I/F and have the switch app create a virtual eth0 I/F over PCIe
>>>>
>>>> No need to create this interface. It will exist if you go the
>>>> switchdev route.
>>>>
>>>>>>> And switchdev can do all this over PCIe instead? Can you have a
>>>>>>> switch tree in switchdev too?
>>>>>>
>>>>>> Mellonex says so, but i don't think they have actually implemented it.
>>>>>
>>>>> Not impl. any of DSAs features? What can you do with a Mellonex switch then?
>>>>
>>>> They don't have a tree of switches, as far as i know. Just a single
>>>> switch. But DSA does support a tree of switches, that is what the D in
>>>> DSA means, distributed. And there are a couple of boards which have 2
>>>> to 4 switches in a tree.
>>>
>>> We might have a tree as well so now I really wonder: Given we write a
>>> proper switchdev driver, can it support switchtrees without touching
>>> switchdev infra structure? If not I guess we will attach a physical
>>> eth I/F to the switch and use both DSA and switchdev to support both trees
>>> and HW offload. 
>>
>> switchdev in itself really is the glue layer between the networking
>> stack and how to push specific objects down to the Ethernet switch
>> driver, and that Ethernet switch driver. Switchdev does not enforce a
>> specific network device driver model object, and just provides standard
>> hooks for your network devices to register with switchdev in order to
>> push/receive offloads. DSA on the other hand, utilizes switchdev to get
>> notifications about offloads from the networking stack, but also exposes
>> a clearly and well defined Ethernet switch device driver model, as
>> Andrew described, it creates per-port network devices, binds the ports
>> to their PHYs (built-in, or external), and also takes care of
>> encapsulating/decapsulating the switch specific tagging protocol.
> 
> Lets see, switchdev is mainly for offloading L2/L3 into HW and does NOT create
> virtual I/F(one for each phys sw port) so if my only goal is to offload I don't
> need DSA? (How do one create routes if no virtual I/Fs I wonder ..) 

switchdev does not create network devices per-se, but a driver utilizing
switchdev should do that instead. How packet transmission/reception
works on these network devices is outside of the scope of switchdev.

> 
> DSA then does create virt. I/Fs and manages switch trees, to actually tx pkg it needs
> a phys I/F using vendor specific tags(for PCIe connected switches I can envision using
> the PCIe connection instead).

DSA can operate without a tag (DSA_TAG_PROTO_NONE) but the general idea
is that the management entity, in order to properly identify which
per-port network device is sending traffic to/from, needs some kind of
tag. Its most rudimentary form can be a per-port VLAN ID, but ideally a
proper switch tag (EDSA, DSA, Broadcom tag) is better suited for that.

The "master" network device in DSA is just a normal Ethernet interface
which receives these tags, and transmits then to the managemenet/CPU
port of the switch it is attached to.

> 
> Who is the master when using both DSA and switchdev w.r.t initil conf/bring up of switch?
> configuring VLANs?

You start with all ports being all segregated and just able to talk to
the CPU/management port. If you are in a switch tree (distributed), then
the uplinks between the switches should make such traffic pass through
all the way to the management port. In order to configure VLANs, you
need to create a bridge device, and then use "bridge vlan ..." to
configure the VLAN member ship of ports.

> 
>>
>> We should probably put that in some crystal clear sentence somewhere in
>> Documentation/networking/ but switchdev and DSA are complementary and
>> not competitors, they just do not tackle the problems from the same angle.
> 
> Yes :)
> 
>>
>>>
>>>>
>>>> I think this is partially down to market segments. Mellonex market is
>>>> top of rack switches. High port count, very high bandwidth. DSA is
>>>> more wireless access points, set top boxes, generally up to 7 ports of
>>>> 1Gbps and a few custom embedded products which need more ports, so
>>>> build a tree of switches.
>>>
>>> We have on an existing board with a BCM ROBO switch with lots of ports(>24),
>>> managed over SPI. Looking at BCM DSA tag code it looks like it only supports
>>> some 8 ports or so. I still have to find out if this is a limitation in BCM tagging
>>> protocol or if just not impl. in DSA yet.
>>
>> Oh cool, can you share the model by chance? I suspect the tagging format
> 
> I think it is an BCM5322

OK, so this one is actually capable of supporting something that is a
real distributed architecture design, cool! The format of the tag is way
different than what net/dsa/tag_brcm.c supports today, and it also
requires a lot more help from the networking stack with respect to
unicast/broadcast/multicast since you need to adjust the opcode of the
tag accordingly. Nothing that cannot be supported though.

> 
>> of that switch is going to be different than what net/dsa/tag_brcm.c, so
>> feel free to add something NET_DSA_TAG_BRCM8B (for 8 bytes) or something
>> like that.
> 
> I will have to look at that, don't have an docs handy.

The device looks largely register compatible with the definitions in
drivers/net/dsa/b53/b53_regs.h with a few notable exceptions:

- port control overrides B53_PORT_CTRL are not at offset 0 + (port
number), but A0 + port number
- link status summary in the status page is slightly offset down

But for the most part, in particular the ARL and VLAN pages, seems like
b53 would not require too much conditionals to get this switch to work.

> 
>>
>> Note that DSA currently hardcodes the maximum number of ports to 14
>> (DSA_MAX_PORTS), but this should obviously be something dynamically
>> determined based on probing the switch device.
>>
>> Can you also evaluate if using drivers/net/dsa/b53/ would work for you?
>> My hope would be that they preserved the register compatibility here,
>> but since this has a large number of ports, it may have completely
>> offset most registers.
> 
> I will have a look, I sure want to try out DSA/switchdev on existing boards if I can :)

Good!
-- 
Florian

Re: DSA vs. SWTICHDEV ?

[Murali Karicheri]

On 11/30/2016 03:50 AM, Joakim Tjernlund wrote:
> I am trying to wrap my head around these two "devices" and have a hard time telling them apart.
> We are looking att adding a faily large switch(over PCIe) to our board and from what I can tell
> switchdev is the new way to do it but DSA is still there. Is it possible to just list
> how they differ?
> 
> What can switchdev do that DSA cannot?
> 
> What can DSA do that switchdev cannot?
> 
> 
> Can one enable switchdev and dsa for the same switch device?
> 
>  Jocke 
> 

DSA/Switchdev experts,

Nice to see this discussion as I am trying to evaluat what model
works best for our hardware. From my evaluation so far, DSA can be
used even though there is no tag protocol used between the Host and
Switch. In our hardware, the Host and Switch are part of the SoC.
The Host interface is a shared memory with queues implemented at
hardware. The Phy is attached to the mii ports externally on the board.
Also this hardware is programmable through firmware. More details 
can be seen at http://processors.wiki.ti.com/index.php/PRU-ICSS
PRU can run a firmware to configure the hardware in one of the following:-

1. EMAC mode where it appears as two Ethernet ports
2. Switch mode where it implements a simple Ethernet switch. Currently
   it doesn't have address learning capability, but in future it
   can.
3. Switch with HSR/PRP offload where it provides HSR/PRP protocol
   support and cut through switch.

So a device need to function in one of the modes. A a regular Ethernet
driver that provides two network devices, one per port, and switchdev
for each physical port (in switch mode) will look ideal in this case.
This will allow attaching the associated interfaces to a bridge (where
a L2 offload is possible in the future). This also helps to attach the
interfaces to an HSR device at the top layer like bridge to support
HSR/PRP protocol with offload possible to the PRU Switch.

Using a DSA for this appears to be adding more complexity to the driver
model and may not be ideal. What do you think? 

-- 
Murali Karicheri
Linux Kernel, Keystone

Re: DSA vs. SWTICHDEV ?

[Andrew Lunn]

Hi Murali 

> 2. Switch mode where it implements a simple Ethernet switch. Currently
>    it doesn't have address learning capability, but in future it
>    can.

If it does not have address learning capabilities, does it act like a
plain old hub? What comes in one port goes out all others?

Or can you do the learning in software on the host and program tables,
which the hardware then uses?

> 3. Switch with HSR/PRP offload where it provides HSR/PRP protocol
>    support and cut through switch.
> 
> So a device need to function in one of the modes. A a regular Ethernet
> driver that provides two network devices, one per port, and switchdev
> for each physical port (in switch mode) will look ideal in this case.

Yes, this seems the right model to use.

     Andrew

Re: DSA vs. SWTICHDEV ?

[Murali Karicheri]

Hi Andrew,
On 12/01/2016 12:31 PM, Andrew Lunn wrote:
> Hi Murali 
> 
>> 2. Switch mode where it implements a simple Ethernet switch. Currently
>>    it doesn't have address learning capability, but in future it
>>    can.
> 
> If it does not have address learning capabilities, does it act like a
> plain old hub? What comes in one port goes out all others?

Thanks for the response!

Yes. It is a plain hub. it replicates frame to both ports. So need to
run a bridge layer for address learning in software.

> 
> Or can you do the learning in software on the host and program tables,
> which the hardware then uses?
> 

I think not. I see we have a non Linux implementation that does address
learning in software using a hash table and look up MAC for each packet
to see which port it needs to be sent to.

Murali

>> 3. Switch with HSR/PRP offload where it provides HSR/PRP protocol
>>    support and cut through switch.
>>
>> So a device need to function in one of the modes. A a regular Ethernet
>> driver that provides two network devices, one per port, and switchdev
>> for each physical port (in switch mode) will look ideal in this case.
> 
> Yes, this seems the right model to use.
> 
>      Andrew
> 


-- 
Murali Karicheri
Linux Kernel, Keystone

Re: DSA vs. SWTICHDEV ?

[Andrew Lunn]

On Thu, Dec 01, 2016 at 04:38:47PM -0500, Murali Karicheri wrote:
> Hi Andrew,
> On 12/01/2016 12:31 PM, Andrew Lunn wrote:
> > Hi Murali 
> > 
> >> 2. Switch mode where it implements a simple Ethernet switch. Currently
> >>    it doesn't have address learning capability, but in future it
> >>    can.
> > 
> > If it does not have address learning capabilities, does it act like a
> > plain old hub? What comes in one port goes out all others?
> 
> Thanks for the response!
> 
> Yes. It is a plain hub. it replicates frame to both ports. So need to
> run a bridge layer for address learning in software.

Hi Murali

It would be good if you start thinking about all the different
directions. It is not just host to port A and host to port B. What
about port A to Port B? Can it do that in hardware? 

> I think not. I see we have a non Linux implementation that does address
> learning in software using a hash table and look up MAC for each packet
> to see which port it needs to be sent to.

I think i need to read more about the switch. I'm starting to wonder
if it has enough intelligence to be usable. Switchdev is about pushing
configuration down into the switch. It does not however sound like
there is that much which is configurable in this switch.

      Andrew