From mboxrd@z Thu Jan 1 00:00:00 1970 From: Pablo Neira Ayuso Subject: Re: [PATCH net-next,v3 00/12] add flow_rule infrastructure Date: Mon, 26 Nov 2018 20:33:36 +0100 Message-ID: <20181126193336.6vpumuc2ekt6oudy@salvia> References: <20181121025132.14305-1-pablo@netfilter.org> <20181122162220.GB8353@localhost.localdomain> <20181122210832.GD14375@localhost.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: netdev@vger.kernel.org, davem@davemloft.net, thomas.lendacky@amd.com, f.fainelli@gmail.com, ariel.elior@cavium.com, michael.chan@broadcom.com, santosh@chelsio.com, madalin.bucur@nxp.com, yisen.zhuang@huawei.com, salil.mehta@huawei.com, jeffrey.t.kirsher@intel.com, tariqt@mellanox.com, saeedm@mellanox.com, jiri@mellanox.com, idosch@mellanox.com, jakub.kicinski@netronome.com, peppe.cavallaro@st.com, grygorii.strashko@ti.com, andrew@lunn.ch, vivien.didelot@savoirfairelinux.com, alexandre.torgue@st.com, joabreu@synopsys.com, linux-net-drivers@solarflare.com, ganeshgr@chelsio.com, ogerlitz@mellanox.com, Manish.Chopra@cavium.com To: Marcelo Ricardo Leitner Return-path: Received: from mail.us.es ([193.147.175.20]:48006 "EHLO mail.us.es" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726140AbeK0G2t (ORCPT ); Tue, 27 Nov 2018 01:28:49 -0500 Received: from antivirus1-rhel7.int (unknown [192.168.2.11]) by mail.us.es (Postfix) with ESMTP id 6E858DA4CC for ; Mon, 26 Nov 2018 20:33:39 +0100 (CET) Received: from antivirus1-rhel7.int (localhost [127.0.0.1]) by antivirus1-rhel7.int (Postfix) with ESMTP id 59321DA7F1 for ; Mon, 26 Nov 2018 20:33:39 +0100 (CET) Content-Disposition: inline In-Reply-To: <20181122210832.GD14375@localhost.localdomain> Sender: netdev-owner@vger.kernel.org List-ID: Hi Marcelo, On Thu, Nov 22, 2018 at 07:08:32PM -0200, Marcelo Ricardo Leitner wrote: > On Thu, Nov 22, 2018 at 02:22:20PM -0200, Marcelo Ricardo Leitner wrote: > > On Wed, Nov 21, 2018 at 03:51:20AM +0100, Pablo Neira Ayuso wrote: > > > Hi, > > > > > > This patchset is the third iteration [1] [2] [3] to introduce a kernel > > > intermediate (IR) to express ACL hardware offloads. > > > > On v2 cover letter you had: > > > > """ > > However, cost of this layer is very small, adding 1 million rules via > > tc -batch, perf shows: > > > > 0.06% tc [kernel.vmlinux] [k] tc_setup_flow_action > > """ > > > > The above doesn't include time spent on children calls and I'm worried > > about the new allocation done by flow_rule_alloc(), as it can impact > > rule insertion rate. I'll run some tests here and report back. > > I'm seeing +60ms on 1.75s (~3.4%) to add 40k flower rules on ingress > with skip_hw and tc in batch mode, with flows like: > > filter add dev p6p2 parent ffff: protocol ip prio 1 flower skip_hw > src_mac ec:13:db:00:00:00 dst_mac ec:14:c2:00:00:00 src_ip > 56.0.0.0 dst_ip 55.0.0.0 action drop > > Only 20ms out of those 60ms were consumed within fl_change() calls > (considering children calls), though. > > Do you see something similar? I used current net-next (d59da3fbfe3f) > and with this patchset applied. I see lots of send() and recv() in tc -batch via strace, using this example rule, repeating it N times: filter add dev eth0 parent ffff: protocol ip pref 1 flower dst_mac f4:52:14:10:df:92 action mirred egress redirect dev eth1 This is taking ~8 seconds for 40k rules from my old laptop [*], this is already not too fast (without my patchset). I remember we discussed about adding support for real batching for tc - probably we can probably do this transparently by assuming that if the skbuff length mismatches nlmsghdr->len field, then we enter the batch mode from the kernel. This would require to update iproute2 to use libmnl batching routines, or code that follows similar approach otherwise. [*] 0.5 seconds in nft (similar ruleset), this is using netlink batching.