Deliver skbs to intermediate interfaces, when using cascaded rx-handlers

[Tobias Waldekranz <tobias@waldekranz.com>]

I have a system with the following configuration:

   * eth0 is a regular Ethernet MAC, connected to a hardware switch.
   * port[1-4] are the in-kernel representation of the switch.
   * team0 is a regular team-interface which bonds port1 and port2.

      +-------+
      | team0 |
      +---+---+
          |
     +----+----+
     |         |
 +---+---+ +---+---+ +-------+ +-------+
 | port1 | | port2 | | port3 | | port4 | 
 +---+---+ +---+---+ +---+---+ +---+---+
     |         |         |         |
     +---------+----+----+---------+
                    |
                +---+---+
                | eth0  |
                +-------+

Both the switch and the team driver attaches rx-handlers to their
lower layers. The problem is that team expects LACP frames that
are intercepted by team0, to also be delivered to the port interface
by __netif_receive_skb_core (in the final iteration of the registered
protocols). However in this case, when two rx-handlers are cascaded,
the skb will be delivered to eth0, since that is the original device
(orig_dev).

There are a few ways this can be solved as far as i can see:

   1. Introduce a new rx-handler return code that specifies that
   skb->dev has been altered, like RX_HANDLER_ANOTHER, and that we
   also want to change orig_dev to this new device.

   2. Change the switch driver to return RX_HANDLER_CONSUMED and then
   queue the skb all over again, though I am afraid that this will eat
   some cycles.

   3. Keep track of all traversed interfaces and change the final
   protocol iteration to deliver the skb to all intermediate devices.

What would be your suggestion?

-- 
Thanks
 - wkz