Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz]

I have a system with the following configuration:

   * eth0 is a regular Ethernet MAC, connected to a hardware switch.
   * port[1-4] are the in-kernel representation of the switch.
   * team0 is a regular team-interface which bonds port1 and port2.

      +-------+
      | team0 |
      +---+---+
          |
     +----+----+
     |         |
 +---+---+ +---+---+ +-------+ +-------+
 | port1 | | port2 | | port3 | | port4 | 
 +---+---+ +---+---+ +---+---+ +---+---+
     |         |         |         |
     +---------+----+----+---------+
                    |
                +---+---+
                | eth0  |
                +-------+

Both the switch and the team driver attaches rx-handlers to their
lower layers. The problem is that team expects LACP frames that
are intercepted by team0, to also be delivered to the port interface
by __netif_receive_skb_core (in the final iteration of the registered
protocols). However in this case, when two rx-handlers are cascaded,
the skb will be delivered to eth0, since that is the original device
(orig_dev).

There are a few ways this can be solved as far as i can see:

   1. Introduce a new rx-handler return code that specifies that
   skb->dev has been altered, like RX_HANDLER_ANOTHER, and that we
   also want to change orig_dev to this new device.

   2. Change the switch driver to return RX_HANDLER_CONSUMED and then
   queue the skb all over again, though I am afraid that this will eat
   some cycles.

   3. Keep track of all traversed interfaces and change the final
   protocol iteration to deliver the skb to all intermediate devices.

What would be your suggestion?

-- 
Thanks
 - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Jiri Pirko]

Fri, Feb 21, 2014 at 11:01:58AM CET, tobias@waldekranz.com wrote:
>I have a system with the following configuration:
>
>   * eth0 is a regular Ethernet MAC, connected to a hardware switch.
>   * port[1-4] are the in-kernel representation of the switch.
>   * team0 is a regular team-interface which bonds port1 and port2.
>
>      +-------+
>      | team0 |
>      +---+---+
>          |
>     +----+----+
>     |         |
> +---+---+ +---+---+ +-------+ +-------+
> | port1 | | port2 | | port3 | | port4 | 
> +---+---+ +---+---+ +---+---+ +---+---+
>     |         |         |         |
>     +---------+----+----+---------+
>                    |
>                +---+---+
>                | eth0  |
>                +-------+
>
>Both the switch and the team driver attaches rx-handlers to their
>lower layers. The problem is that team expects LACP frames that
>are intercepted by team0, to also be delivered to the port interface
>by __netif_receive_skb_core (in the final iteration of the registered
>protocols). However in this case, when two rx-handlers are cascaded,
>the skb will be delivered to eth0, since that is the original device
>(orig_dev).
>
>There are a few ways this can be solved as far as i can see:
>
>   1. Introduce a new rx-handler return code that specifies that
>   skb->dev has been altered, like RX_HANDLER_ANOTHER, and that we
>   also want to change orig_dev to this new device.
>
>   2. Change the switch driver to return RX_HANDLER_CONSUMED and then
>   queue the skb all over again, though I am afraid that this will eat
>   some cycles.

What "switch driver" are you reffering to?



>
>   3. Keep track of all traversed interfaces and change the final
>   protocol iteration to deliver the skb to all intermediate devices.
>
>What would be your suggestion?
>
>-- 
>Thanks
> - wkz
>
>--
>To unsubscribe from this list: send the line "unsubscribe netdev" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz]

> What "switch driver" are you reffering to?
> 

The driver that is in charge of multiplexing packets from the switch
chip out to the port interfaces. It registers an rx-handler on eth0
and forwards packets to port[1-4] by reading a custom tag (similar to
802.1Q) inserted by the switch.

-- 
Thanks
 - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Jiri Pirko]

Fri, Feb 21, 2014 at 12:12:57PM CET, tobias@waldekranz.com wrote:
>> What "switch driver" are you reffering to?
>> 
>
>The driver that is in charge of multiplexing packets from the switch
>chip out to the port interfaces. It registers an rx-handler on eth0
>and forwards packets to port[1-4] by reading a custom tag (similar to
>802.1Q) inserted by the switch.

Allright, but what's the name? I would like to find it and see the code.

>
>-- 
>Thanks
> - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz]

On Fri, Feb 21, 2014 at 12:37:53PM +0100, Jiri Pirko wrote:
> Fri, Feb 21, 2014 at 12:12:57PM CET, tobias@waldekranz.com wrote:
> >> What "switch driver" are you reffering to?
> >> 
> >
> >The driver that is in charge of multiplexing packets from the switch
> >chip out to the port interfaces. It registers an rx-handler on eth0
> >and forwards packets to port[1-4] by reading a custom tag (similar to
> >802.1Q) inserted by the switch.
> 
> Allright, but what's the name? I would like to find it and see the code.

There's not really a name :), but I have made the source available on github:

https://github.com/wkz/switchcore

The relevant parts for this discussion should be in sc_main.c.

> 
> >
> >-- 
> >Thanks
> > - wkz

-- 
Thanks
 - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Jiri Pirko]

Fri, Feb 21, 2014 at 01:02:44PM CET, tobias@waldekranz.com wrote:
>On Fri, Feb 21, 2014 at 12:37:53PM +0100, Jiri Pirko wrote:
>> Fri, Feb 21, 2014 at 12:12:57PM CET, tobias@waldekranz.com wrote:
>> >> What "switch driver" are you reffering to?
>> >> 
>> >
>> >The driver that is in charge of multiplexing packets from the switch
>> >chip out to the port interfaces. It registers an rx-handler on eth0
>> >and forwards packets to port[1-4] by reading a custom tag (similar to
>> >802.1Q) inserted by the switch.
>> 
>> Allright, but what's the name? I would like to find it and see the code.
>
>There's not really a name :), but I have made the source available on github:
>
>https://github.com/wkz/switchcore

Are you aware of existing DSA infrastructure?

>
>The relevant parts for this discussion should be in sc_main.c.
>
>> 
>> >
>> >-- 
>> >Thanks
>> > - wkz
>
>-- 
>Thanks
> - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz]

On Fri, Feb 21, 2014 at 02:05:53PM +0100, Jiri Pirko wrote:
> Fri, Feb 21, 2014 at 01:02:44PM CET, tobias@waldekranz.com wrote:
> >On Fri, Feb 21, 2014 at 12:37:53PM +0100, Jiri Pirko wrote:
> >> Fri, Feb 21, 2014 at 12:12:57PM CET, tobias@waldekranz.com wrote:
> >> >> What "switch driver" are you reffering to?
> >> >> 
> >> >
> >> >The driver that is in charge of multiplexing packets from the switch
> >> >chip out to the port interfaces. It registers an rx-handler on eth0
> >> >and forwards packets to port[1-4] by reading a custom tag (similar to
> >> >802.1Q) inserted by the switch.
> >> 
> >> Allright, but what's the name? I would like to find it and see the code.
> >
> >There's not really a name :), but I have made the source available on github:
> >
> >https://github.com/wkz/switchcore
> 
> Are you aware of existing DSA infrastructure?
> 

Yes I am, we have our reasons for using our own driver. Most of it
having to do with dynamic configuration, multiple hardware channel
support and a variety of exotic hardware setups.

Do you have any input on the original question?

> >
> >The relevant parts for this discussion should be in sc_main.c.
> >
> >> 
> >> >
> >> >-- 
> >> >Thanks
> >> > - wkz
> >
> >-- 
> >Thanks
> > - wkz

-- 
Thanks
 - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Jiri Pirko]

Fri, Feb 21, 2014 at 03:31:14PM CET, tobias@waldekranz.com wrote:
>On Fri, Feb 21, 2014 at 02:05:53PM +0100, Jiri Pirko wrote:
>> Fri, Feb 21, 2014 at 01:02:44PM CET, tobias@waldekranz.com wrote:
>> >On Fri, Feb 21, 2014 at 12:37:53PM +0100, Jiri Pirko wrote:
>> >> Fri, Feb 21, 2014 at 12:12:57PM CET, tobias@waldekranz.com wrote:
>> >> >> What "switch driver" are you reffering to?
>> >> >> 
>> >> >
>> >> >The driver that is in charge of multiplexing packets from the switch
>> >> >chip out to the port interfaces. It registers an rx-handler on eth0
>> >> >and forwards packets to port[1-4] by reading a custom tag (similar to
>> >> >802.1Q) inserted by the switch.
>> >> 
>> >> Allright, but what's the name? I would like to find it and see the code.
>> >
>> >There's not really a name :), but I have made the source available on github:
>> >
>> >https://github.com/wkz/switchcore
>> 
>> Are you aware of existing DSA infrastructure?
>> 
>
>Yes I am, we have our reasons for using our own driver. Most of it
>having to do with dynamic configuration, multiple hardware channel
>support and a variety of exotic hardware setups.

Would not it be better to make DSA better in order to achieve your goal?

>
>Do you have any input on the original question?

I'm not really sure how to handle this. I fail to clearly understand the
topology you are describing...

>
>> >
>> >The relevant parts for this discussion should be in sc_main.c.
>> >
>> >> 
>> >> >
>> >> >-- 
>> >> >Thanks
>> >> > - wkz
>> >
>> >-- 
>> >Thanks
>> > - wkz
>
>-- 
>Thanks
> - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz]

On Fri, Feb 21, 2014 at 03:48:27PM +0100, Jiri Pirko wrote:
> Fri, Feb 21, 2014 at 03:31:14PM CET, tobias@waldekranz.com wrote:
> >On Fri, Feb 21, 2014 at 02:05:53PM +0100, Jiri Pirko wrote:
> >> Fri, Feb 21, 2014 at 01:02:44PM CET, tobias@waldekranz.com wrote:
> >> >On Fri, Feb 21, 2014 at 12:37:53PM +0100, Jiri Pirko wrote:
> >> >> Fri, Feb 21, 2014 at 12:12:57PM CET, tobias@waldekranz.com wrote:
> >> >> >> What "switch driver" are you reffering to?
> >> >> >> 
> >> >> >
> >> >> >The driver that is in charge of multiplexing packets from the switch
> >> >> >chip out to the port interfaces. It registers an rx-handler on eth0
> >> >> >and forwards packets to port[1-4] by reading a custom tag (similar to
> >> >> >802.1Q) inserted by the switch.
> >> >> 
> >> >> Allright, but what's the name? I would like to find it and see the code.
> >> >
> >> >There's not really a name :), but I have made the source available on github:
> >> >
> >> >https://github.com/wkz/switchcore
> >> 
> >> Are you aware of existing DSA infrastructure?
> >> 
> >
> >Yes I am, we have our reasons for using our own driver. Most of it
> >having to do with dynamic configuration, multiple hardware channel
> >support and a variety of exotic hardware setups.
> 
> Would not it be better to make DSA better in order to achieve your goal?
> 

Sure, that is something we are considering. But before we do that we
need to make sure that our model works, end-to-end, for all important
use-cases.

> >
> >Do you have any input on the original question?
> 
> I'm not really sure how to handle this. I fail to clearly understand the
> topology you are describing...
> 
> >
> >> >
> >> >The relevant parts for this discussion should be in sc_main.c.
> >> >
> >> >> 
> >> >> >
> >> >> >-- 
> >> >> >Thanks
> >> >> > - wkz
> >> >
> >> >-- 
> >> >Thanks
> >> > - wkz
> >
> >-- 
> >Thanks
> > - wkz

-- 
Thanks
 - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Flavio Leitner]

On Fri, Feb 21, 2014 at 11:01:58AM +0100, Tobias Waldekranz wrote:
> I have a system with the following configuration:
> 
>    * eth0 is a regular Ethernet MAC, connected to a hardware switch.
>    * port[1-4] are the in-kernel representation of the switch.
>    * team0 is a regular team-interface which bonds port1 and port2.
> 
>       +-------+
>       | team0 |
>       +---+---+
>           |
>      +----+----+
>      |         |
>  +---+---+ +---+---+ +-------+ +-------+
>  | port1 | | port2 | | port3 | | port4 | 
>  +---+---+ +---+---+ +---+---+ +---+---+
>      |         |         |         |
>      +---------+----+----+---------+
>                     |
>                 +---+---+
>                 | eth0  |
>                 +-------+
> 
> Both the switch and the team driver attaches rx-handlers to their
> lower layers. The problem is that team expects LACP frames that
> are intercepted by team0, to also be delivered to the port interface
> by __netif_receive_skb_core (in the final iteration of the registered
> protocols). However in this case, when two rx-handlers are cascaded,
> the skb will be delivered to eth0, since that is the original device
> (orig_dev).

Indeed. It is not possible to update orig_dev, so it is stuck
on eth0 regardless of all the stacked interfaces.

> There are a few ways this can be solved as far as i can see:
> 
>    1. Introduce a new rx-handler return code that specifies that
>    skb->dev has been altered, like RX_HANDLER_ANOTHER, and that we
>    also want to change orig_dev to this new device.

But then you need to know which rx-handler will run next, because
the orig_dev might still be implicit wanted.
 
>    2. Change the switch driver to return RX_HANDLER_CONSUMED and then
>    queue the skb all over again, though I am afraid that this will eat
>    some cycles.

Before queue again, you need to update skb->dev, right? This seems
similar to (1).

>    3. Keep track of all traversed interfaces and change the final
>    protocol iteration to deliver the skb to all intermediate devices.

That's what I thought too.

fbl

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[David Miller]

From: Jiri Pirko <jiri@resnulli.us>
Date: Fri, 21 Feb 2014 15:48:27 +0100

> Would not it be better to make DSA better in order to achieve your goal?

+1

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz]

On Fri, Feb 21, 2014 at 02:10:35PM -0300, Flavio Leitner wrote:
> On Fri, Feb 21, 2014 at 11:01:58AM +0100, Tobias Waldekranz wrote:
> > I have a system with the following configuration:
> > 
> >    * eth0 is a regular Ethernet MAC, connected to a hardware switch.
> >    * port[1-4] are the in-kernel representation of the switch.
> >    * team0 is a regular team-interface which bonds port1 and port2.
> > 
> >       +-------+
> >       | team0 |
> >       +---+---+
> >           |
> >      +----+----+
> >      |         |
> >  +---+---+ +---+---+ +-------+ +-------+
> >  | port1 | | port2 | | port3 | | port4 | 
> >  +---+---+ +---+---+ +---+---+ +---+---+
> >      |         |         |         |
> >      +---------+----+----+---------+
> >                     |
> >                 +---+---+
> >                 | eth0  |
> >                 +-------+
> > 
> > Both the switch and the team driver attaches rx-handlers to their
> > lower layers. The problem is that team expects LACP frames that
> > are intercepted by team0, to also be delivered to the port interface
> > by __netif_receive_skb_core (in the final iteration of the registered
> > protocols). However in this case, when two rx-handlers are cascaded,
> > the skb will be delivered to eth0, since that is the original device
> > (orig_dev).
> 
> Indeed. It is not possible to update orig_dev, so it is stuck
> on eth0 regardless of all the stacked interfaces.
> 
> > There are a few ways this can be solved as far as i can see:
> > 
> >    1. Introduce a new rx-handler return code that specifies that
> >    skb->dev has been altered, like RX_HANDLER_ANOTHER, and that we
> >    also want to change orig_dev to this new device.
> 
> But then you need to know which rx-handler will run next, because
> the orig_dev might still be implicit wanted.

The next rx-handler is resolved by looking at skb->dev->rx_handler, so
that shouldn't be a problem. Or am I missing something?

>  
> >    2. Change the switch driver to return RX_HANDLER_CONSUMED and then
> >    queue the skb all over again, though I am afraid that this will eat
> >    some cycles.
> 
> Before queue again, you need to update skb->dev, right? This seems
> similar to (1).

Exactly. It is very similar. The difference is that I would have to
call netif_rx and go through a lot of code again, instead of taking
the fast path via the another_round-loop in __netif_receive_skb_core.

> 
> >    3. Keep track of all traversed interfaces and change the final
> >    protocol iteration to deliver the skb to all intermediate devices.
> 
> That's what I thought too.

Ok, so if I provide such a patch, you think it would have a good
chance of getting accepted?

What are the performance implications of deliver_skb-ing the packet to
each intermediate layer? Considering that most of those packets will
probably not be of interest to anyone.

> 
> fbl

-- 
Thanks
 - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz]

On Fri, Feb 21, 2014 at 12:23:59PM -0500, David Miller wrote:
> From: Jiri Pirko <jiri@resnulli.us>
> Date: Fri, 21 Feb 2014 15:48:27 +0100
> 
> > Would not it be better to make DSA better in order to achieve your goal?
> 
> +1

Ok, message received. I will try to merge the changes back into the
mainline DSA infrastructure. I honestly didn't think it would be of
much interest to anyone.

The original question is really separate from that though. In this
case it is a DSA driver at the bottom, but the question applies to any
case where there are multiple rx-handlers in a cascade
configuration. Wouldn't you agree?

-- 
Thanks
 - wkz

Re: Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Flavio Leitner]

On Mon, Feb 24, 2014 at 08:22:23AM +0100, Tobias Waldekranz wrote:
> On Fri, Feb 21, 2014 at 02:10:35PM -0300, Flavio Leitner wrote:
> > On Fri, Feb 21, 2014 at 11:01:58AM +0100, Tobias Waldekranz wrote:
> > > I have a system with the following configuration:
> > > 
> > >    * eth0 is a regular Ethernet MAC, connected to a hardware switch.
> > >    * port[1-4] are the in-kernel representation of the switch.
> > >    * team0 is a regular team-interface which bonds port1 and port2.
> > > 
> > >       +-------+
> > >       | team0 |
> > >       +---+---+
> > >           |
> > >      +----+----+
> > >      |         |
> > >  +---+---+ +---+---+ +-------+ +-------+
> > >  | port1 | | port2 | | port3 | | port4 | 
> > >  +---+---+ +---+---+ +---+---+ +---+---+
> > >      |         |         |         |
> > >      +---------+----+----+---------+
> > >                     |
> > >                 +---+---+
> > >                 | eth0  |
> > >                 +-------+
> > > 
> > > Both the switch and the team driver attaches rx-handlers to their
> > > lower layers. The problem is that team expects LACP frames that
> > > are intercepted by team0, to also be delivered to the port interface
> > > by __netif_receive_skb_core (in the final iteration of the registered
> > > protocols). However in this case, when two rx-handlers are cascaded,
> > > the skb will be delivered to eth0, since that is the original device
> > > (orig_dev).
> > 
> > Indeed. It is not possible to update orig_dev, so it is stuck
> > on eth0 regardless of all the stacked interfaces.
> > 
> > > There are a few ways this can be solved as far as i can see:
> > > 
> > >    1. Introduce a new rx-handler return code that specifies that
> > >    skb->dev has been altered, like RX_HANDLER_ANOTHER, and that we
> > >    also want to change orig_dev to this new device.
> > 
> > But then you need to know which rx-handler will run next, because
> > the orig_dev might still be implicit wanted.
> 
> The next rx-handler is resolved by looking at skb->dev->rx_handler, so
> that shouldn't be a problem. Or am I missing something?

Yes, but all rx_handlers assume that orig_dev is intact and will be
delivered at the end. That will change with this solution and I don't
know if it can break any rx-handler.


> > >    2. Change the switch driver to return RX_HANDLER_CONSUMED and then
> > >    queue the skb all over again, though I am afraid that this will eat
> > >    some cycles.
> > 
> > Before queue again, you need to update skb->dev, right? This seems
> > similar to (1).
> 
> Exactly. It is very similar. The difference is that I would have to
> call netif_rx and go through a lot of code again, instead of taking
> the fast path via the another_round-loop in __netif_receive_skb_core.

Right.

> > >    3. Keep track of all traversed interfaces and change the final
> > >    protocol iteration to deliver the skb to all intermediate devices.
> > 
> > That's what I thought too.
> 
> Ok, so if I provide such a patch, you think it would have a good
> chance of getting accepted?

Well, the final word comes from David and certainly you will need
to justify why fixing DSA isn't enough.

> What are the performance implications of deliver_skb-ing the packet to
> each intermediate layer? Considering that most of those packets will
> probably not be of interest to anyone.

Good question.

fbl