Re: [PATCH 0/6] offload Linux LAG devices to the TC datapath

[Jakub Kicinski <jakub.kicinski@netronome.com>]

On Wed, 27 Jun 2018 23:07:29 +0300, Or Gerlitz wrote:
> On Wed, Jun 27, 2018 at 1:31 AM, Jakub Kicinski
> <jakub.kicinski@netronome.com> wrote:
> > On Tue, 26 Jun 2018 17:57:08 +0300, Or Gerlitz wrote:  
> 
> >> 2. re the egress side of things. Some NIC HWs can't just use LAG
> >> as the egress port destination of an ACL (tc rule) and the HW rule
> >> needs to be duplicated to both HW ports. So... in that case, you
> >> see the HW driver doing the duplication (:() or we can somehow
> >> make it happen from user-space?  
> 
> > It's the TC core that does the duplication.  Drivers which don't need
> > the duplication (e.g. mlxsw) will not register a new callback for each
> > port on which shared block is bound.  They will keep one list of rules,
> > and a list of ports that those rules apply to.  
> 
> [snip]
> 
> > Drivers which need duplication (multiplication) (all NICs?) have to
> > register a new callback for each port bound to a shared block.  And TC
> > will call those drivers as many times as they have callbacks registered
> > == as many times as they have ports bound to the block.  Each time
> > callback is invoked the driver will figure out the ingress port based
> > on the cb_priv and use <ingress, cookie> as the key in its rule table
> > (or have a separate rule table per ingress port).  
> 
> [snip snip]
> 
> > I may be wrong, but I think you split the rules tables per port for mlx5  
> 
> correct,  currently I have a rule table per physical port.
> 
> > So again you just register a callback every time shared block is bound,
> > and then TC core will send add/remove rule commands down to the driver,
> > relaying existing rules as well if needed.  
> 
> Let's see, the NIC uplink rep port devices were bounded (say) by ovs to
> a shared-block because they are the lower devices (hate the slavish jargon)
> of a bond device.
> 
> Next, the TC stack will invoke the callback over these ports, when ingress
> rule is added on the bond.
> 
> But we are talking on ingress rule set on a non-uplink rep (VF rep) port,
> where bonding is the egress of the rule. I guess the callback which you probably
> refer to (you hinted there below) is the egdev one, correct? you are suggesting
> that bonding will do egdev registration... I am a bit confused.

Ah, you really meant egress.  We don't have this problem, but yes, I
think you could register an egdev callback for each lower.  You won't
get the nice rule replay from egdev as of today, though :(

> > Does that clarify things or were you asking more about the active
> > passive thing John mentioned or some way to save rule space?  
> 
> no (didn't refer to active-passive) and no (didn't look to save rule space)
> yes for active-active in a HW that needs duplicated rules (NICs).
> 
> >> 3. for the case of overlay networks, e.g OVS based vxlan tunnel, the
> >> ingress (decap) rule is set on the vxlan device. Jakub, you mentioned
> >> a possible kernel patch to the HW (nfp, mlx5) drivers to have them bind
> >> to the tunnel device for ingress rules. If we have agreed way to identify
> >> uplink representors, can we do that from ovs too?  
> >
> > I'm not sure, there can be multiple tunnel devices.  Plus we really
> > want to know the tunnel type, e.g. vxlan vs geneve, so simple shared
> > block propagation will probably not cut it.  If that's what you're
> > referring to.  
> 
> isn't knowing the tunnel type already missing today? I saw you
> started patches the tunnel key set action for Geneve, does upstream
> + the patches you sent works or more is missing to get geneve encap
> through the TC stack?

Yes, knowing tunnel type missing today, but hopefully it won't be once
we get to redesign of egdev :)  Today we only support decap on standard
ports :/  Encap is fine, though.  FWIW Geneve already works on the nfp,
the work from Simon & Pieter we posted is adding support for the
options.