From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jakub Kicinski Subject: Re: [PATCH 0/6] offload Linux LAG devices to the TC datapath Date: Wed, 27 Jun 2018 16:08:11 -0700 Message-ID: <20180627160811.57250c26@cakuba.netronome.com> References: <1529588161-15934-1-git-send-email-john.hurley@netronome.com> <8f406548-8f90-b658-fcd1-342d702b3445@mellanox.com> <20180626153140.6a06eb97@cakuba.netronome.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Or Gerlitz , John Hurley , Jiri Pirko , Linux Netdev List , ASAP_Direct_Dev , Simon Horman , Andy Gospodarek To: Or Gerlitz Return-path: Received: from mail-qk0-f194.google.com ([209.85.220.194]:37476 "EHLO mail-qk0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751854AbeF0XIQ (ORCPT ); Wed, 27 Jun 2018 19:08:16 -0400 Received: by mail-qk0-f194.google.com with SMTP id t79-v6so2000210qke.4 for ; Wed, 27 Jun 2018 16:08:15 -0700 (PDT) In-Reply-To: Sender: netdev-owner@vger.kernel.org List-ID: On Wed, 27 Jun 2018 23:07:29 +0300, Or Gerlitz wrote: > On Wed, Jun 27, 2018 at 1:31 AM, Jakub Kicinski > wrote: > > On Tue, 26 Jun 2018 17:57:08 +0300, Or Gerlitz wrote: > > >> 2. re the egress side of things. Some NIC HWs can't just use LAG > >> as the egress port destination of an ACL (tc rule) and the HW rule > >> needs to be duplicated to both HW ports. So... in that case, you > >> see the HW driver doing the duplication (:() or we can somehow > >> make it happen from user-space? > > > It's the TC core that does the duplication. Drivers which don't need > > the duplication (e.g. mlxsw) will not register a new callback for each > > port on which shared block is bound. They will keep one list of rules, > > and a list of ports that those rules apply to. > > [snip] > > > Drivers which need duplication (multiplication) (all NICs?) have to > > register a new callback for each port bound to a shared block. And TC > > will call those drivers as many times as they have callbacks registered > > == as many times as they have ports bound to the block. Each time > > callback is invoked the driver will figure out the ingress port based > > on the cb_priv and use as the key in its rule table > > (or have a separate rule table per ingress port). > > [snip snip] > > > I may be wrong, but I think you split the rules tables per port for mlx5 > > correct, currently I have a rule table per physical port. > > > So again you just register a callback every time shared block is bound, > > and then TC core will send add/remove rule commands down to the driver, > > relaying existing rules as well if needed. > > Let's see, the NIC uplink rep port devices were bounded (say) by ovs to > a shared-block because they are the lower devices (hate the slavish jargon) > of a bond device. > > Next, the TC stack will invoke the callback over these ports, when ingress > rule is added on the bond. > > But we are talking on ingress rule set on a non-uplink rep (VF rep) port, > where bonding is the egress of the rule. I guess the callback which you probably > refer to (you hinted there below) is the egdev one, correct? you are suggesting > that bonding will do egdev registration... I am a bit confused. Ah, you really meant egress. We don't have this problem, but yes, I think you could register an egdev callback for each lower. You won't get the nice rule replay from egdev as of today, though :( > > Does that clarify things or were you asking more about the active > > passive thing John mentioned or some way to save rule space? > > no (didn't refer to active-passive) and no (didn't look to save rule space) > yes for active-active in a HW that needs duplicated rules (NICs). > > >> 3. for the case of overlay networks, e.g OVS based vxlan tunnel, the > >> ingress (decap) rule is set on the vxlan device. Jakub, you mentioned > >> a possible kernel patch to the HW (nfp, mlx5) drivers to have them bind > >> to the tunnel device for ingress rules. If we have agreed way to identify > >> uplink representors, can we do that from ovs too? > > > > I'm not sure, there can be multiple tunnel devices. Plus we really > > want to know the tunnel type, e.g. vxlan vs geneve, so simple shared > > block propagation will probably not cut it. If that's what you're > > referring to. > > isn't knowing the tunnel type already missing today? I saw you > started patches the tunnel key set action for Geneve, does upstream > + the patches you sent works or more is missing to get geneve encap > through the TC stack? Yes, knowing tunnel type missing today, but hopefully it won't be once we get to redesign of egdev :) Today we only support decap on standard ports :/ Encap is fine, though. FWIW Geneve already works on the nfp, the work from Simon & Pieter we posted is adding support for the options.