* Re: [PATCH 1/2] mpls: add handlers
From: Roopa Prabhu @ 2017-08-13 3:29 UTC (permalink / raw)
To: Amine Kherbouche; +Cc: David Lamparter, netdev@vger.kernel.org
In-Reply-To: <c6fc221b-c0fa-1dbc-d1e2-a746b722dcd7@6wind.com>
On Sat, Aug 12, 2017 at 6:35 AM, Amine Kherbouche
<amine.kherbouche@6wind.com> wrote:
>
>
> On 11/08/2017 16:37, Roopa Prabhu wrote:
>>
>> On Fri, Aug 11, 2017 at 5:34 AM, David Lamparter <equinox@diac24.net>
>> wrote:
>>>
>>> On Thu, Aug 10, 2017 at 10:28:36PM +0200, Amine Kherbouche wrote:
>>>>
>>>> Mpls handler allows creation/deletion of mpls routes without using
>>>> rtnetlink. When an incoming mpls packet matches this route, the saved
>>>> function handler is called.
>>>
>>> Since I originally authored this patch, I have come to believe that it
>>> might be unneccessarily complicated. It is unlikely that a lot of
>>> different "handlers" will exist here; the only things I can think of
>>> are VPLS support and BIER-MPLS multicast replication. I'm not saying
>>> it's a bad idea, but, well, this was in the README that I gave to 6WIND
>>> with this code:
>>>
>>> ...
>>
>> yes, I would also prefer just exporting the functions and calling
>> them directly instead of adding a
>> handler layer. We can move to that later if it becomes necessary.
>
> I understand that the handler layer is an overhead (as said by David's
> note), and I agree with the solution for exporting the mpls functions that
> allows route creation/deletion, but how about forwarding the right mpls
> packet to the right vpls device with the device ptr? I don't see
> another way.
hmm...ok, so you are adding a mpls route to get into vpls_rcv and you
want this route to carry the vpls_rcv information. Ideally if you knew
the route is pointing to a vpls device kind, you can directly call
vpls_rcv.
But, am not sure if a route is necessary here either.
It just seems like the vpls device information is duplicated in the
mpls route per vpls dev. Wondering if we can skip the route part and
always do a lookup on vpls-id/label in mpls_rcv to send it to a
vpls_rcv if there is a match. This will be the l2 handler for mpls.
^ permalink raw reply
* Re: [PATCH net-next] liquidio: update debug console logging mechanism
From: Leon Romanovsky @ 2017-08-13 5:55 UTC (permalink / raw)
To: Felix Manlunas
Cc: davem, netdev, raghu.vatsavayi, derek.chickles, satananda.burla,
ricardo.farrington
In-Reply-To: <20170812014314.GA1012@felix-thinkpad.cavium.com>
[-- Attachment #1: Type: text/plain, Size: 945 bytes --]
On Fri, Aug 11, 2017 at 06:43:14PM -0700, Felix Manlunas wrote:
> From: Rick Farrington <ricardo.farrington@cavium.com>
>
> - remove logging dependency upon global func octeon_console_debug_enabled()
> - abstract debug console logging using console structure (via function ptr)
> to allow for more flexible logging
>
> Signed-off-by: Rick Farrington <ricardo.farrington@cavium.com>
> Signed-off-by: Raghu Vatsavayi <raghu.vatsavayi@cavium.com>
> Signed-off-by: Felix Manlunas <felix.manlunas@cavium.com>
> ---
> drivers/net/ethernet/cavium/liquidio/lio_main.c | 44 +++++++++++++++++-
> .../net/ethernet/cavium/liquidio/octeon_console.c | 54 ++++++++++++++--------
> .../net/ethernet/cavium/liquidio/octeon_device.h | 17 +++++--
> 3 files changed, 90 insertions(+), 25 deletions(-)
>
I'm probably missing something important, but why do you need your custom console
implementation if kernel is full of such built-in options?
Thanks
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: Driver profiles RFC
From: Jiri Pirko @ 2017-08-13 6:32 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Arkadi Sharshevsky, netdev, David Miller, ivecera, roopa,
Florian Fainelli, Vivien Didelot, john.fastabend, Andrew Lunn,
mlxsw
In-Reply-To: <20170811145738.6f92e451@cakuba.netronome.com>
Fri, Aug 11, 2017 at 11:57:38PM CEST, kubakici@wp.pl wrote:
>On Tue, 8 Aug 2017 16:15:41 +0300, Arkadi Sharshevsky wrote:
>> Driver <--> Devlink API
>> =======================
>> Each driver will register his resources with default values at init in
>> a similar way to DPIPE table registration. In case those resources already
>> exist the default values are discarded. The user will be able to dump and
>> update the resources. In order for the changes to take place the user will
>> need to re-initiate the driver by a specific devlink knob.
>
>What seems missing from the examples is the ability to dump the
>different states - the "pending" configuration and the currently
>applied one.
Agreed, I miss this too.
>
>> The above described procedure will require extra reload of the driver.
>> This can be improved as a future optimization.
>
>I'm a bit lost, this says driver reload is required...
driver will provide "commit" callback. What he does is up to him. In
case of mlxsw, we have to reinstantiate the driver, because FW reset is
required.
>
>> UAPI
>> ====
>> The user will be able to update the resources on a per resource basis:
>>
>> $devlink dpipe resource set pci/0000:03:00.0 Mem_Linear 2M
>>
>> For some resources the size is fixed, for example the size of the internal
>> memory cannot be changed. It is provided merely in order to reflect the
>> nested structure of the resource and to imply the user that Mem = Linear +
>> Hash, thus a set operation on it will fail.
>>
>> The user can dump the current resource configuration:
>>
>> #devlink dpipe resource dump tree pci/0000:03:00.0 Mem
>>
>> The user can specify 'tree' in order to show all the nested resources under
>> the specified one. In case no 'resource name' is specified the TOP hierarchy
>> will be dumped.
>>
>> After successful resource update the drivers hould be re-instantiated in
>> order for the changes to take place:
>>
>> $devlink reload pci/0000:03:00.0
>
>... but this shows a devlink reload tigger, so no driver reload? Were
>you describing two possible solutions? One with persistent kernel
>database of configs (persistent across driver reloads) and one with no
>persistence and the driver is managing reinit internally when triggered
>via devlink?
This is misunderstanding. There is no driver reload (modprobe -r &&
modprobe)
>
>Another thing that comes to mind is - in case HW/FW reinit takes long
>would it make sense to incorporate some form of pre-population of those
>defaults somehow? If user knows exactly the config they want upon
>boot, it would seem cleaner if the reconfig did not have to happen and
>devices started out in the right mode.
We were discussing it. There are couple of ways to achieve that, none of
that is very nice. So we decided to leave this for later, when/if it is
needed.
^ permalink raw reply
* [PATCH net] openvswitch: fix skb_panic due to the incorrect actions attrlen
From: Liping Zhang @ 2017-08-13 7:04 UTC (permalink / raw)
To: pshelar, davem; +Cc: netdev, Liping Zhang, Neil McKee
From: Liping Zhang <zlpnobody@gmail.com>
For sw_flow_actions, the actions_len only represents the kernel part's
size, and when we dump the actions to the userspace, we will do the
convertions, so it's true size may become bigger than the actions_len.
But unfortunately, for OVS_PACKET_ATTR_ACTIONS, we use the actions_len
to alloc the skbuff, so the user_skb's size may become insufficient and
oops will happen like this:
skbuff: skb_over_panic: text:ffffffff8148fabf len:1749 put:157 head:
ffff881300f39000 data:ffff881300f39000 tail:0x6d5 end:0x6c0 dev:<NULL>
------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:129!
[...]
Call Trace:
<IRQ>
[<ffffffff8148be82>] skb_put+0x43/0x44
[<ffffffff8148fabf>] skb_zerocopy+0x6c/0x1f4
[<ffffffffa0290d36>] queue_userspace_packet+0x3a3/0x448 [openvswitch]
[<ffffffffa0292023>] ovs_dp_upcall+0x30/0x5c [openvswitch]
[<ffffffffa028d435>] output_userspace+0x132/0x158 [openvswitch]
[<ffffffffa01e6890>] ? ip6_rcv_finish+0x74/0x77 [ipv6]
[<ffffffffa028e277>] do_execute_actions+0xcc1/0xdc8 [openvswitch]
[<ffffffffa028e3f2>] ovs_execute_actions+0x74/0x106 [openvswitch]
[<ffffffffa0292130>] ovs_dp_process_packet+0xe1/0xfd [openvswitch]
[<ffffffffa0292b77>] ? key_extract+0x63c/0x8d5 [openvswitch]
[<ffffffffa029848b>] ovs_vport_receive+0xa1/0xc3 [openvswitch]
[...]
Also we can find that the actions_len is much little than the orig_len:
crash> struct sw_flow_actions 0xffff8812f539d000
struct sw_flow_actions {
rcu = {
next = 0xffff8812f5398800,
func = 0xffffe3b00035db32
},
orig_len = 1384,
actions_len = 592,
actions = 0xffff8812f539d01c
}
So as a quick fix, use the orig_len instead of the actions_len to alloc
the user_skb.
Last, this oops happened on our system running a relative old kernel, but
the same risk still exists on the mainline, since we use the wrong
actions_len from the beginning.
Fixes: ccea74457bbd ("openvswitch: include datapath actions with sampled-packet upcall to userspace")
Cc: Neil McKee <neil.mckee@inmon.com>
Signed-off-by: Liping Zhang <zlpnobody@gmail.com>
---
net/openvswitch/actions.c | 39 +++++++++++++++++++++++++--------------
net/openvswitch/datapath.c | 2 +-
net/openvswitch/datapath.h | 1 +
3 files changed, 27 insertions(+), 15 deletions(-)
diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c
index e4610676299b..799a22dfb89e 100644
--- a/net/openvswitch/actions.c
+++ b/net/openvswitch/actions.c
@@ -48,6 +48,7 @@ struct deferred_action {
struct sk_buff *skb;
const struct nlattr *actions;
int actions_len;
+ int actions_attrlen;
/* Store pkt_key clone when creating deferred action. */
struct sw_flow_key pkt_key;
@@ -135,7 +136,8 @@ static struct deferred_action *action_fifo_put(struct action_fifo *fifo)
static struct deferred_action *add_deferred_actions(struct sk_buff *skb,
const struct sw_flow_key *key,
const struct nlattr *actions,
- const int actions_len)
+ const int actions_len,
+ const int actions_attrlen)
{
struct action_fifo *fifo;
struct deferred_action *da;
@@ -146,6 +148,7 @@ static struct deferred_action *add_deferred_actions(struct sk_buff *skb,
da->skb = skb;
da->actions = actions;
da->actions_len = actions_len;
+ da->actions_attrlen = actions_attrlen;
da->pkt_key = *key;
}
@@ -166,6 +169,7 @@ static int clone_execute(struct datapath *dp, struct sk_buff *skb,
struct sw_flow_key *key,
u32 recirc_id,
const struct nlattr *actions, int len,
+ int actions_attrlen,
bool last, bool clone_flow_key);
static void update_ethertype(struct sk_buff *skb, struct ethhdr *hdr,
@@ -880,7 +884,7 @@ static void do_output(struct datapath *dp, struct sk_buff *skb, int out_port,
static int output_userspace(struct datapath *dp, struct sk_buff *skb,
struct sw_flow_key *key, const struct nlattr *attr,
const struct nlattr *actions, int actions_len,
- uint32_t cutlen)
+ int actions_attrlen, uint32_t cutlen)
{
struct dp_upcall_info upcall;
const struct nlattr *a;
@@ -921,6 +925,7 @@ static int output_userspace(struct datapath *dp, struct sk_buff *skb,
/* Include actions. */
upcall.actions = actions;
upcall.actions_len = actions_len;
+ upcall.actions_attrlen = actions_attrlen;
break;
}
@@ -936,7 +941,7 @@ static int output_userspace(struct datapath *dp, struct sk_buff *skb,
*/
static int sample(struct datapath *dp, struct sk_buff *skb,
struct sw_flow_key *key, const struct nlattr *attr,
- bool last)
+ int actions_attrlen, bool last)
{
struct nlattr *actions;
struct nlattr *sample_arg;
@@ -957,8 +962,8 @@ static int sample(struct datapath *dp, struct sk_buff *skb,
}
clone_flow_key = !arg->exec;
- return clone_execute(dp, skb, key, 0, actions, rem, last,
- clone_flow_key);
+ return clone_execute(dp, skb, key, 0, actions, rem, actions_attrlen,
+ last, clone_flow_key);
}
static void execute_hash(struct sk_buff *skb, struct sw_flow_key *key,
@@ -1083,13 +1088,14 @@ static int execute_recirc(struct datapath *dp, struct sk_buff *skb,
BUG_ON(!is_flow_key_valid(key));
recirc_id = nla_get_u32(a);
- return clone_execute(dp, skb, key, recirc_id, NULL, 0, last, true);
+ return clone_execute(dp, skb, key, recirc_id, NULL, 0, 0, last, true);
}
/* Execute a list of actions against 'skb'. */
static int do_execute_actions(struct datapath *dp, struct sk_buff *skb,
struct sw_flow_key *key,
- const struct nlattr *attr, int len)
+ const struct nlattr *attr, int len,
+ int actions_attrlen)
{
const struct nlattr *a;
int rem;
@@ -1130,8 +1136,8 @@ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb,
}
case OVS_ACTION_ATTR_USERSPACE:
- output_userspace(dp, skb, key, a, attr,
- len, OVS_CB(skb)->cutlen);
+ output_userspace(dp, skb, key, a, attr, len,
+ actions_attrlen, OVS_CB(skb)->cutlen);
OVS_CB(skb)->cutlen = 0;
break;
@@ -1181,7 +1187,7 @@ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb,
case OVS_ACTION_ATTR_SAMPLE: {
bool last = nla_is_last(a, rem);
- err = sample(dp, skb, key, a, last);
+ err = sample(dp, skb, key, a, actions_attrlen, last);
if (last)
return err;
@@ -1231,6 +1237,7 @@ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb,
static int clone_execute(struct datapath *dp, struct sk_buff *skb,
struct sw_flow_key *key, u32 recirc_id,
const struct nlattr *actions, int len,
+ int actions_attrlen,
bool last, bool clone_flow_key)
{
struct deferred_action *da;
@@ -1258,7 +1265,8 @@ static int clone_execute(struct datapath *dp, struct sk_buff *skb,
__this_cpu_inc(exec_actions_level);
err = do_execute_actions(dp, skb, clone,
- actions, len);
+ actions, len,
+ actions_attrlen);
if (clone_flow_key)
__this_cpu_dec(exec_actions_level);
@@ -1270,7 +1278,7 @@ static int clone_execute(struct datapath *dp, struct sk_buff *skb,
}
/* Out of 'flow_keys' space. Defer actions */
- da = add_deferred_actions(skb, key, actions, len);
+ da = add_deferred_actions(skb, key, actions, len, actions_attrlen);
if (da) {
if (!actions) { /* Recirc action */
key = &da->pkt_key;
@@ -1309,10 +1317,12 @@ static void process_deferred_actions(struct datapath *dp)
struct sk_buff *skb = da->skb;
struct sw_flow_key *key = &da->pkt_key;
const struct nlattr *actions = da->actions;
+ int actions_attrlen = da->actions_attrlen;
int actions_len = da->actions_len;
if (actions)
- do_execute_actions(dp, skb, key, actions, actions_len);
+ do_execute_actions(dp, skb, key, actions, actions_len,
+ actions_attrlen);
else
ovs_dp_process_packet(skb, key);
} while (!action_fifo_is_empty(fifo));
@@ -1338,7 +1348,8 @@ int ovs_execute_actions(struct datapath *dp, struct sk_buff *skb,
}
err = do_execute_actions(dp, skb, key,
- acts->actions, acts->actions_len);
+ acts->actions, acts->actions_len,
+ acts->orig_len);
if (level == 1)
process_deferred_actions(dp);
diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c
index 45fe8c8a884d..66162e64e8b5 100644
--- a/net/openvswitch/datapath.c
+++ b/net/openvswitch/datapath.c
@@ -398,7 +398,7 @@ static size_t upcall_msg_size(const struct dp_upcall_info *upcall_info,
/* OVS_PACKET_ATTR_ACTIONS */
if (upcall_info->actions_len)
- size += nla_total_size(upcall_info->actions_len);
+ size += nla_total_size(upcall_info->actions_attrlen);
/* OVS_PACKET_ATTR_MRU */
if (upcall_info->mru)
diff --git a/net/openvswitch/datapath.h b/net/openvswitch/datapath.h
index 5d8dcd88815f..d7dfba5893b4 100644
--- a/net/openvswitch/datapath.h
+++ b/net/openvswitch/datapath.h
@@ -124,6 +124,7 @@ struct dp_upcall_info {
const struct nlattr *userdata;
const struct nlattr *actions;
int actions_len;
+ int actions_attrlen;
u32 portid;
u8 cmd;
u16 mru;
--
2.13.0
^ permalink raw reply related
* Re: [PATCH v5 09/11] ARM: dts: rk3228-evb: Enable the integrated PHY for gmac
From: kbuild test robot @ 2017-08-13 11:03 UTC (permalink / raw)
To: David Wu
Cc: kbuild-all-JC7UmRfGjtg, davem-fT/PcQaiUtIeIZ0/mPfg9Q,
heiko-4mtYJXux2i+zQB+pC5nmwQ, andrew-g2DYL2Zd6BY,
f.fainelli-Re5JQEeQqe8AvxtiuMwx3w, robh+dt-DgEjT+Ai2ygdnm+yROfE0A,
mark.rutland-5wv7dgnIgG8, catalin.marinas-5wv7dgnIgG8,
will.deacon-5wv7dgnIgG8, olof-nZhT3qVonbNeoWH0uzbU5w,
linux-I+IVW8TIWO2tmTQ+vhA3Yw, arnd-r2nGTMty4D4,
peppe.cavallaro-qxv4g6HH51o, alexandre.torgue-qxv4g6HH51o,
huangtao-TNX95d0MmH7DzftRWevZcw, hwg-TNX95d0MmH7DzftRWevZcw,
netdev-u79uwXL29TY76Z2rM5mHXA,
linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
devicetree-u79uwXL29TY76Z2rM5mHXA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, David Wu
In-Reply-To: <1502373668-2767-1-git-send-email-david.wu-TNX95d0MmH7DzftRWevZcw@public.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 961 bytes --]
Hi David,
[auto build test ERROR on net/master]
[also build test ERROR on v4.13-rc4 next-20170811]
[cannot apply to net-next/master]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/David-Wu/Add-the-integrated-PHY-support/20170813-163538
config: arm-omap2plus_defconfig (attached as .config)
compiler: arm-linux-gnueabi-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm
All errors (new ones prefixed by >>):
>> ERROR: Input tree has errors, aborting (use -f to force output)
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 30001 bytes --]
^ permalink raw reply
* [PATCH 0/4] constify net platform_device_id
From: Arvind Yadav @ 2017-08-13 11:11 UTC (permalink / raw)
To: wg, mkl, madalin.bucur, davem, sergei.shtylyov, grygorii.strashko
Cc: linux-kernel, netdev, linux-can
platform_device_id are not supposed to change at runtime. All functions
working with platform_device_id provided by <linux/platform_device.h>
work with const platform_device_id. So mark the non-const structs as const.
Arvind Yadav (4):
[PATCH 1/4] can: constify platform_device_id
[PATCH 2/4] net: dpaa_eth: constify platform_device_id
[PATCH 3/4] net: sh_eth: constify platform_device_id
[PATCH 4/4] net: ti: cpsw:: constify platform_device_id
drivers/net/can/c_can/c_can_platform.c | 2 +-
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 2 +-
drivers/net/ethernet/renesas/sh_eth.c | 2 +-
drivers/net/ethernet/ti/cpsw.c | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
--
2.7.4
^ permalink raw reply
* [PATCH 1/4] can: constify platform_device_id
From: Arvind Yadav @ 2017-08-13 11:11 UTC (permalink / raw)
To: wg, mkl, madalin.bucur, davem, sergei.shtylyov, grygorii.strashko
Cc: linux-kernel, netdev, linux-can
In-Reply-To: <1502622705-11169-1-git-send-email-arvind.yadav.cs@gmail.com>
platform_device_id are not supposed to change at runtime. All functions
working with platform_device_id provided by <linux/platform_device.h>
work with const platform_device_id. So mark the non-const structs as
const.
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
---
drivers/net/can/c_can/c_can_platform.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/can/c_can/c_can_platform.c b/drivers/net/can/c_can/c_can_platform.c
index e36d105..46a746e 100644
--- a/drivers/net/can/c_can/c_can_platform.c
+++ b/drivers/net/can/c_can/c_can_platform.c
@@ -220,7 +220,7 @@ static const struct c_can_driver_data am3352_dcan_drvdata = {
.raminit_bits = am3352_raminit_bits,
};
-static struct platform_device_id c_can_id_table[] = {
+static const struct platform_device_id c_can_id_table[] = {
{
.name = KBUILD_MODNAME,
.driver_data = (kernel_ulong_t)&c_can_drvdata,
--
2.7.4
^ permalink raw reply related
* [PATCH 2/4] net: dpaa_eth: constify platform_device_id
From: Arvind Yadav @ 2017-08-13 11:12 UTC (permalink / raw)
To: wg, mkl, madalin.bucur, davem, sergei.shtylyov, grygorii.strashko
Cc: linux-kernel, netdev
platform_device_id are not supposed to change at runtime. All functions
working with platform_device_id provided by <linux/platform_device.h>
work with const platform_device_id. So mark the non-const structs as
const.
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
---
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c
index 757b873..4521b33 100644
--- a/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c
+++ b/drivers/net/ethernet/freescale/dpaa/dpaa_eth.c
@@ -2829,7 +2829,7 @@ static int dpaa_remove(struct platform_device *pdev)
return err;
}
-static struct platform_device_id dpaa_devtype[] = {
+static const struct platform_device_id dpaa_devtype[] = {
{
.name = "dpaa-ethernet",
.driver_data = 0,
--
2.7.4
^ permalink raw reply related
* [PATCH 3/4] net: sh_eth: constify platform_device_id
From: Arvind Yadav @ 2017-08-13 11:12 UTC (permalink / raw)
To: wg, mkl, madalin.bucur, davem, sergei.shtylyov, grygorii.strashko
Cc: linux-kernel, netdev, linux-renesas-soc
platform_device_id are not supposed to change at runtime. All functions
working with platform_device_id provided by <linux/platform_device.h>
work with const platform_device_id. So mark the non-const structs as
const.
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
---
drivers/net/ethernet/renesas/sh_eth.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c
index d2dc0a8..d2e88a3 100644
--- a/drivers/net/ethernet/renesas/sh_eth.c
+++ b/drivers/net/ethernet/renesas/sh_eth.c
@@ -3402,7 +3402,7 @@ static const struct dev_pm_ops sh_eth_dev_pm_ops = {
#define SH_ETH_PM_OPS NULL
#endif
-static struct platform_device_id sh_eth_id_table[] = {
+static const struct platform_device_id sh_eth_id_table[] = {
{ "sh7619-ether", (kernel_ulong_t)&sh7619_data },
{ "sh771x-ether", (kernel_ulong_t)&sh771x_data },
{ "sh7724-ether", (kernel_ulong_t)&sh7724_data },
--
2.7.4
^ permalink raw reply related
* [PATCH 4/4] net: ti: cpsw:: constify platform_device_id
From: Arvind Yadav @ 2017-08-13 11:13 UTC (permalink / raw)
To: wg, mkl, madalin.bucur, davem, sergei.shtylyov, grygorii.strashko
Cc: linux-kernel, netdev, linux-omap
platform_device_id are not supposed to change at runtime. All functions
working with platform_device_id provided by <linux/platform_device.h>
work with const platform_device_id. So mark the non-const structs as
const.
Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com>
---
drivers/net/ethernet/ti/cpsw.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index 1850e34..2866964 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -2827,7 +2827,7 @@ static int cpsw_probe_dual_emac(struct cpsw_priv *priv)
#define CPSW_QUIRK_IRQ BIT(0)
-static struct platform_device_id cpsw_devtype[] = {
+static const struct platform_device_id cpsw_devtype[] = {
{
/* keep it for existing comaptibles */
.name = "cpsw",
--
2.7.4
^ permalink raw reply related
* Re: [PATCH net] Revert "vhost: cache used event for better performance"
From: Koichiro Den @ 2017-08-13 14:11 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang; +Cc: netdev, linux-kernel, kvm, virtualization
In-Reply-To: <20170809073422-mutt-send-email-mst@kernel.org>
Thanks for your comments, Michael and Jason. And I'm sorry about late response.
To be honest, I am on a summer vacation until next Tuesday.
I noticed that what I wrote was not sufficient. Regardless of caching mechanism
existence, the "event" could legitimately be at any point out of the latest
interval, which vhost_notify checks it against, meaning that if it's out of the
interval we cannot distinguish whether or not it lags behind or has a lead. And
it seems to conform to the spec. Thanks for leading me to the spec. The corner
case I point out here is:
(0) event idx feature turned on + TX napi turned off
-> (1) guest-side TX traffic bursting occurs and delayed callback set
-> (2) some interruption triggers skb_xmit_done
-> (3) guest-side modest traffic makes the interval proceed to unbounded extent
without updating "event" since NO_INTERRUPT continues to be set on its shadow
flag.
IMHO, if you plan to make TX napi the only choice, doing this sort of
optimisation beforehand seems likely to be in vain.
So, if the none-TX napi case continues to coexist, what I would like to suggest
is not just the sort of my last email, but like making maximum staleness of
"event" less than or equal to vq.num, and afterward caching suggestion.
Otherwise, I guess I should not repost my last email since it would make matters
too complicated even though it will soon be removed when TX-napi becomes the
only choice.
Thanks!
On Wed, 2017-08-09 at 07:37 +0300, Michael S. Tsirkin wrote:
> On Wed, Aug 09, 2017 at 10:38:10AM +0800, Jason Wang wrote:
> > I think don't think current code can work well if vq.num is grater than
> > 2^15. Since all cached idx is u16. This looks like a bug which needs to be
> > fixed.
>
> That's a limitation of virtio 1.0.
>
> > > * else if the interval of vq.num is [2^15, 2^16):
> > > the logic in the original patch (809ecb9bca6a9) suffices
> > > * else (= less than 2^15) (optional):
> > > checking only (vring_need_event(vq->last_used_event, new + vq->num, new)
> > > would suffice.
> > >
> > > Am I missing something, or is this irrelevant?
>
> Could you pls repost the suggestion copying virtio-dev mailing list
> (subscriber only, sorry about that, but host/guest ABI discussions
> need to copy that list)?
>
> > Looks not, I think this may work. Let me do some test.
> >
> > Thanks
>
> I think that at this point it's prudent to add a feature bit
> as the virtio spec does not require to never move the event index back.
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* RE: [patch v1 1/2] Allow Mellanox network vendor to be configured if only I2C bus is configured
From: Ohad Oz @ 2017-08-13 15:26 UTC (permalink / raw)
To: Leon Romanovsky
Cc: davem@davemloft.net, netdev@vger.kernel.org, jiri@resnulli.us,
Saeed Mahameed, Vadim Pasternak, system-sw-low-level
In-Reply-To: <20170812143715.GN24282@mtr-leonro.local>
> -----Original Message-----
> From: Leon Romanovsky [mailto:leon@kernel.org]
> Sent: Saturday, August 12, 2017 5:37 PM
> To: Ohad Oz <ohado@mellanox.com>
> Cc: davem@davemloft.net; netdev@vger.kernel.org; jiri@resnulli.us; Saeed
> Mahameed <saeedm@mellanox.com>; Vadim Pasternak
> <vadimp@mellanox.com>; system-sw-low-level <system-sw-low-
> level@mellanox.com>
> Subject: Re: [patch v1 1/2] Allow Mellanox network vendor to be configured
> if only I2C bus is configured
>
> On Thu, Aug 10, 2017 at 05:11:51PM +0000, Ohad Oz wrote:
> > Patch allows Mellanox devices on system with no PCI, but with I2C only.
> >
>
> Did you test mlx5 device on such system? Did it work for you?
Yes, I did. With PCI config set to disable mlx5 drivers are not built.
Only the following:
/build/drivers/net/Ethernet/mellanox/mlxsw/mlxsw_core.ko
/build/drivers/net/Ethernet/mellanox /mlxsw/mlxsw_i2c.ko
/build/drivers/net/Ethernet/mellanox /mlxsw/mlxsw_minimal.ko
While with both options on all drivers are built inc mlx5.
> What is the changelog between v0 and v1 of these patches?
No changelog, I've started with v1.
>
> > Signed-off-by: Ohad Oz <ohado@mellanox.com>
> > ---
> > drivers/net/ethernet/mellanox/Kconfig | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/ethernet/mellanox/Kconfig
> b/drivers/net/ethernet/mellanox/Kconfig
> > index 84a2007..0949741 100644
> > --- a/drivers/net/ethernet/mellanox/Kconfig
> > +++ b/drivers/net/ethernet/mellanox/Kconfig
> > @@ -5,7 +5,7 @@
> > config NET_VENDOR_MELLANOX
> > bool "Mellanox devices"
> > default y
> > - depends on PCI
> > + depends on PCI || I2C
> > ---help---
> > If you have a network (Ethernet) card belonging to this class, say Y.
> >
> > --
> > 2.8.0
> >
^ permalink raw reply
* Re: [iproute PATCH 06/51] iplink_vrf: Complain if main table is not found
From: David Ahern @ 2017-08-13 15:58 UTC (permalink / raw)
To: Phil Sutter, Stephen Hemminger; +Cc: netdev
In-Reply-To: <20170812120510.28750-7-phil@nwl.cc>
On 8/12/17 6:04 AM, Phil Sutter wrote:
> Signed-off-by: Phil Sutter <phil@nwl.cc>
> ---
> ip/iplink_vrf.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/ip/iplink_vrf.c b/ip/iplink_vrf.c
> index 917630e853375..809eda5de8f6e 100644
> --- a/ip/iplink_vrf.c
> +++ b/ip/iplink_vrf.c
> @@ -131,7 +131,10 @@ __u32 ipvrf_get_table(const char *name)
> &answer.n, sizeof(answer)) < 0) {
> /* special case "default" vrf to be the main table */
> if (errno == ENODEV && !strcmp(name, "default"))
> - rtnl_rttable_a2n(&tb_id, "main");
> + if (rtnl_rttable_a2n(&tb_id, "main"))
> + fprintf(stderr,
> + "BUG: RTTable \"main\" not found.\n");
> +
>
> return tb_id;
> }
>
Acked-by: David Ahern <dsahern@gmail.com>
^ permalink raw reply
* Re: [iproute PATCH 14/51] ipvrf: Don't try to close an invalid fd
From: David Ahern @ 2017-08-13 15:59 UTC (permalink / raw)
To: Phil Sutter, Stephen Hemminger; +Cc: netdev
In-Reply-To: <20170812120510.28750-15-phil@nwl.cc>
On 8/12/17 6:04 AM, Phil Sutter wrote:
> Signed-off-by: Phil Sutter <phil@nwl.cc>
> ---
> ip/ipvrf.c | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/ip/ipvrf.c b/ip/ipvrf.c
> index 0094cf8557cd7..92e2db98ca7d7 100644
> --- a/ip/ipvrf.c
> +++ b/ip/ipvrf.c
> @@ -268,7 +268,7 @@ static int vrf_configure_cgroup(const char *path, int ifindex)
> fprintf(stderr,
> "Failed to open cgroup path: '%s'\n",
> strerror(errno));
> - goto out;
> + return rc;
> }
>
> /*
> @@ -290,13 +290,14 @@ static int vrf_configure_cgroup(const char *path, int ifindex)
> if (bpf_prog_attach_fd(prog_fd, cg_fd, BPF_CGROUP_INET_SOCK_CREATE)) {
> fprintf(stderr, "Failed to attach prog to cgroup: '%s'\n",
> strerror(errno));
> - goto out;
> + goto out2;
> }
>
> rc = 0;
> +out2:
> + close(prog_fd);
> out:
> close(cg_fd);
> - close(prog_fd);
>
> return rc;
> }
>
Acked-by: David Ahern <dsahern@gmail.com>
^ permalink raw reply
* Re: [iproute PATCH 15/51] ipvrf: Fix error path of vrf_switch()
From: David Ahern @ 2017-08-13 16:00 UTC (permalink / raw)
To: Phil Sutter, Stephen Hemminger; +Cc: netdev
In-Reply-To: <20170812120510.28750-16-phil@nwl.cc>
On 8/12/17 6:04 AM, Phil Sutter wrote:
> Apart from trying to close(-1), this also leaked memory.
>
> Signed-off-by: Phil Sutter <phil@nwl.cc>
> ---
> ip/ipvrf.c | 9 +++++----
> 1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/ip/ipvrf.c b/ip/ipvrf.c
> index 92e2db98ca7d7..75cc026d072b8 100644
> --- a/ip/ipvrf.c
> +++ b/ip/ipvrf.c
> @@ -373,12 +373,12 @@ static int vrf_switch(const char *name)
>
> /* -1 on length to add '/' to the end */
> if (ipvrf_get_netns(netns, sizeof(netns) - 1) < 0)
> - return -1;
> + goto out;
>
> if (vrf_path(vpath, sizeof(vpath)) < 0) {
> fprintf(stderr, "Failed to get base cgroup path: %s\n",
> strerror(errno));
> - return -1;
> + goto out;
> }
>
> /* if path already ends in netns then don't add it again */
> @@ -429,13 +429,14 @@ static int vrf_switch(const char *name)
> snprintf(pid, sizeof(pid), "%d", getpid());
> if (write(fd, pid, strlen(pid)) < 0) {
> fprintf(stderr, "Failed to join cgroup\n");
> - goto out;
> + goto out2;
> }
>
> rc = 0;
> +out2:
> + close(fd);
> out:
> free(mnt);
> - close(fd);
>
> return rc;
> }
>
Acked-by: David Ahern <dsahern@gmail.com>
^ permalink raw reply
* Re: [PATCH net] Revert "vhost: cache used event for better performance"
From: Koichiro Den @ 2017-08-13 16:12 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang; +Cc: netdev, linux-kernel, kvm, virtualization
In-Reply-To: <1502633460.3547.1.camel@klaipeden.com>
Sorry I mistakenly focused on NET case, please pass it over. I will do any
relevant suggestion in patch-based way. Thanks.
On Sun, 2017-08-13 at 23:11 +0900, Koichiro Den wrote:
> Thanks for your comments, Michael and Jason. And I'm sorry about late
> response.
> To be honest, I am on a summer vacation until next Tuesday.
>
> I noticed that what I wrote was not sufficient. Regardless of caching
> mechanism
> existence, the "event" could legitimately be at any point out of the latest
> interval, which vhost_notify checks it against, meaning that if it's out of
> the
> interval we cannot distinguish whether or not it lags behind or has a
> lead. And
> it seems to conform to the spec. Thanks for leading me to the spec. The corner
> case I point out here is:
> (0) event idx feature turned on + TX napi turned off
> -> (1) guest-side TX traffic bursting occurs and delayed callback set
> -> (2) some interruption triggers skb_xmit_done
> -> (3) guest-side modest traffic makes the interval proceed to unbounded
> extent
> without updating "event" since NO_INTERRUPT continues to be set on its shadow
> flag.
>
> IMHO, if you plan to make TX napi the only choice, doing this sort of
> optimisation beforehand seems likely to be in vain.
>
> So, if the none-TX napi case continues to coexist, what I would like to
> suggest
> is not just the sort of my last email, but like making maximum staleness of
> "event" less than or equal to vq.num, and afterward caching suggestion.
> Otherwise, I guess I should not repost my last email since it would make
> matters
> too complicated even though it will soon be removed when TX-napi becomes the
> only choice.
>
> Thanks!
>
>
> On Wed, 2017-08-09 at 07:37 +0300, Michael S. Tsirkin wrote:
> > On Wed, Aug 09, 2017 at 10:38:10AM +0800, Jason Wang wrote:
> > > I think don't think current code can work well if vq.num is grater than
> > > 2^15. Since all cached idx is u16. This looks like a bug which needs to be
> > > fixed.
> >
> > That's a limitation of virtio 1.0.
> >
> > > > * else if the interval of vq.num is [2^15, 2^16):
> > > > the logic in the original patch (809ecb9bca6a9) suffices
> > > > * else (= less than 2^15) (optional):
> > > > checking only (vring_need_event(vq->last_used_event, new + vq->num, new)
> > > > would suffice.
> > > >
> > > > Am I missing something, or is this irrelevant?
> >
> > Could you pls repost the suggestion copying virtio-dev mailing list
> > (subscriber only, sorry about that, but host/guest ABI discussions
> > need to copy that list)?
> >
> > > Looks not, I think this may work. Let me do some test.
> > >
> > > Thanks
> >
> > I think that at this point it's prudent to add a feature bit
> > as the virtio spec does not require to never move the event index back.
> >
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: unregister_netdevice: waiting for eth0 to become free. Usage count = 1
From: David Ahern @ 2017-08-13 16:24 UTC (permalink / raw)
To: Wei Wang, Ido Schimmel
Cc: Cong Wang, John Stultz, Martin KaFai Lau, lkml,
Network Development, Linux USB List, David S. Miller,
Felipe Balbi
In-Reply-To: <CAEA6p_Cngs9Nx9OZ15oRMu_43oZTYHjzcw2vzgemfEyh_g0nNA@mail.gmail.com>
On 8/12/17 1:42 PM, Wei Wang wrote:
> Hi Ido,
>
>>> - if ((rt->dst.dev == dev || !dev) &&
>>> + if ((rt->dst.dev == dev || !dev ||
>>> + rt->rt6i_idev->dev == dev) &&
>>
>> Can you please explain why this line is needed? While host routes aren't
>> removed from the FIB by rt6_ifdown() (when dst.dev goes down), they are
>> removed later on in addrconf_ifdown().
>>
>
> Yes.. Agree. But one difference is that if the route is removed from
> addrconf_ifdown(), dst_dev_put() won't be called to release the
> devices before doing dst_release(). It is OK if dst_release() sees the
> refcnt on dst already drops to 0 and directly destroys the dst. But I
> think it will cause problem if at the time, the dst is still held by
> some other users because then the refcnt on the device going down will
> not get released.
> That's why I think we should remove the dst with either dst->dev ==
> going down dev or rt6->rt6i_idev->dev == going down dev from the fib6
> tree always because there, we always call dst_dev_put() to release the
> device.
>
>> With your patch, if I check the return value of ip6_del_rt() in
>> __ipv6_ifa_notify() I see that -ENONET is returned. Because the host
>> route was already removed by rt6_ifdown(). When the line in question is
>> removed from the patch I don't get the error anymore.
>>
>
> Right. That is expected as the route is already removed from the tree.
>
>> Is it possible that in John's case the host route was correctly removed
>> from the FIB and that the unreleased reference was due to a wrong check
>> in ip6_dst_ifdown() (which you patched correctly AFAICT)?
>>
>
> Yes. possible. But as I explained earlier, I still think we should
> also remove routes with rt6->rt6i_idev->dev == going down dev from the
> tree.
Looking at my patch to move host routes from loopback to device with the
address, I have this:
@@ -2789,7 +2808,8 @@ static int fib6_ifdown(struct rt6_info *rt, void *arg)
const struct arg_dev_net *adn = arg;
const struct net_device *dev = adn->dev;
- if ((rt->dst.dev == dev || !dev) &&
+ if ((rt->dst.dev == dev || !dev ||
+ (netdev_unregistering(dev) && rt->rt6i_idev->dev == dev)) &&
rt != adn->net->ipv6.ip6_null_entry &&
(rt->rt6i_nsiblings == 0 ||
(dev && netdev_unregistering(dev)) ||
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Paweł Staszewski @ 2017-08-13 16:58 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Linux Kernel Network Developers
In-Reply-To: <af5b3482-8f26-52b5-4a00-3a5a42b7449f@itcare.pl>
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
(ixgbe in kernel driver kernel 4.13.0-rc4-next-20170811)
ethtool settings for both tests:
ethtool -K $ifc gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple off
ethtool -L $ifc combined 16
ethtool -C $ifc rx-usecs 2
ethtool -G $ifc rx 4096 tx 1024
16 CORES / 16 RSS QUEUES
Tx traffic on vlan:
RX Interface:
enp216s0f0
TX Interface
vlan1000 added to enp216s0f1 interface (with vlan 1000 ip address assigned)
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;6939008;416325120;6938696;402411192
1;16;64;6941952;416444160;6941745;402558918
2;16;64;6960576;417584640;6960707;403698718
3;16;64;6940736;416486400;6941820;402503876
4;16;64;6927680;415741440;6927420;401853870
5;16;64;6929792;415687680;6929917;401839196
6;16;64;6950400;416989440;6950661;403026166
7;16;64;6953664;417216000;6953454;403260544
8;16;64;6948480;416851200;6948800;403023266
9;16;64;6924160;415422720;6924092;401542468
100% load on all 16 Cores.
vs
RX interface from traffic generator:
enp216s0f0
TX interface to the sink:
enp216s0f1
No vlan used
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;10280176;793608540;10298496;596796568
1;16;64;10046928;600978780;10046022;582527002
2;16;64;10032956;601827420;10026097;581515656
3;16;64;10051503;602252460;10067880;582420804
4;16;64;10016204;602725140;10017358;582644800
5;16;64;10035575;602437620;10059504;582067294
6;16;64;10041667;603069780;10057865;582477412
7;16;64;10044448;600027420;10046526;581022018
8;16;64;10022436;601121100;10025946;581904314
9;16;64;10036231;602514960;10058724;582180684
So we have 10Mpps forwarded
- have problems with pktgen on my traffic generator to push more than
10M but this low budget hardware so.. :)
And there are still free cpu cycles so probabbly can forward at line 10G
rate 14Mpps
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.00 0.00 0.00 0.00 0.00 20.91
0.00 0.00 0.00 79.09
Average: 0 0.00 0.00 0.00 0.00 0.00 0.09 0.00
0.00 0.00 99.91
Average: 1 0.03 0.00 0.03 0.00 0.00 0.00 0.00
0.00 0.00 99.94
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 5 0.00 0.00 0.18 0.00 0.00 0.00 0.00
0.00 0.00 99.82
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 10 0.00 0.00 0.03 0.24 0.00 0.00 0.00
0.00 0.00 99.74
Average: 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 14 0.00 0.00 0.00 0.00 0.00 92.38
0.00 0.00 0.00 7.62
Average: 15 0.00 0.00 0.00 0.00 0.00 85.88
0.00 0.00 0.00 14.12
Average: 16 0.00 0.00 0.00 0.00 0.00 64.91
0.00 0.00 0.00 35.09
Average: 17 0.00 0.00 0.00 0.00 0.00 66.76
0.00 0.00 0.00 33.24
Average: 18 0.00 0.00 0.00 0.00 0.00 65.57
0.00 0.00 0.00 34.43
Average: 19 0.00 0.00 0.00 0.00 0.00 66.38
0.00 0.00 0.00 33.62
Average: 20 0.00 0.00 0.00 0.00 0.00 72.97
0.00 0.00 0.00 27.03
Average: 21 0.00 0.00 0.00 0.00 0.00 70.80
0.00 0.00 0.00 29.20
Average: 22 0.00 0.00 0.00 0.00 0.00 66.44
0.00 0.00 0.00 33.56
Average: 23 0.00 0.00 0.00 0.00 0.00 66.12
0.00 0.00 0.00 33.88
Average: 24 0.00 0.00 0.00 0.00 0.00 68.35
0.00 0.00 0.00 31.65
Average: 25 0.00 0.00 0.00 0.00 0.00 71.79
0.00 0.00 0.00 28.21
Average: 26 0.00 0.00 0.00 0.00 0.00 70.24
0.00 0.00 0.00 29.76
Average: 27 0.00 0.00 0.00 0.00 0.00 73.24
0.00 0.00 0.00 26.76
Average: 28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 39 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 42 0.00 0.00 0.00 0.00 0.00 84.27
0.00 0.00 0.00 15.73
Average: 43 0.00 0.00 0.00 0.00 0.00 84.50
0.00 0.00 0.00 15.50
Average: 44 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 45 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 46 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 47 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 48 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 49 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 51 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 52 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 54 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: CPU intr/s
Average: all 3559661.68
Average: 0 628.53
Average: 1 537.62
Average: 2 525.00
Average: 3 558.29
Average: 4 546.79
Average: 5 522.85
Average: 6 508.06
Average: 7 568.88
Average: 8 529.56
Average: 9 535.29
Average: 10 530.09
Average: 11 539.53
Average: 12 520.82
Average: 13 531.32
Average: 14 73315.68
Average: 15 115983.15
Average: 16 254446.09
Average: 17 253067.79
Average: 18 254446.35
Average: 19 252457.29
Average: 20 213928.18
Average: 21 232770.32
Average: 22 263906.85
Average: 23 260065.09
Average: 24 243609.74
Average: 25 218122.53
Average: 26 237405.38
Average: 27 217582.76
Average: 28 548.29
Average: 29 569.12
Average: 30 540.74
Average: 31 517.50
Average: 32 521.59
Average: 33 544.85
Average: 34 520.91
Average: 35 553.29
Average: 36 545.32
Average: 37 518.44
Average: 38 557.26
Average: 39 541.71
Average: 40 515.21
Average: 41 520.82
Average: 42 137722.38
Average: 43 135737.59
Average: 44 524.97
Average: 45 538.24
Average: 46 580.38
Average: 47 567.62
Average: 48 555.53
Average: 49 561.50
Average: 50 537.65
Average: 51 565.09
Average: 52 536.12
Average: 53 570.44
Average: 54 535.38
Average: 55 567.88
Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
Average: 0 0.00 250.03 0.18 69.09 0.00
0.00 6.59 250.03 0.00 52.62
Average: 1 0.00 228.94 0.00 0.00 0.00
0.00 10.82 249.79 0.00 48.06
Average: 2 0.00 244.53 0.00 0.00 0.00
0.00 0.00 249.79 0.00 30.68
Average: 3 0.00 249.53 0.00 0.00 0.00
0.00 0.00 249.91 0.00 58.85
Average: 4 0.00 243.71 0.00 0.00 0.00
0.00 0.00 249.94 0.00 53.15
Average: 5 0.00 247.06 0.00 0.00 0.00
0.00 0.00 249.44 0.00 26.35
Average: 6 0.00 249.38 0.00 0.00 0.00
0.00 0.00 249.91 0.00 8.76
Average: 7 0.00 226.03 0.00 0.00 0.00
0.00 0.00 249.88 0.00 92.97
Average: 8 0.00 244.29 0.00 0.00 0.00
0.00 0.00 249.79 0.00 35.47
Average: 9 0.00 247.71 0.00 0.00 0.00
0.00 0.00 249.94 0.00 37.65
Average: 10 0.00 241.85 0.00 0.00 0.00
0.00 0.00 249.65 0.00 38.59
Average: 11 0.00 249.00 0.00 0.00 0.00
0.00 0.00 249.94 0.00 40.59
Average: 12 0.00 248.85 0.00 0.00 0.00
0.00 0.00 249.88 0.00 22.09
Average: 13 0.00 249.12 0.00 0.00 0.00
0.00 0.00 249.88 0.00 32.32
Average: 14 0.00 212.97 147.44 72760.47 0.00
0.00 0.00 189.35 0.00 5.44
Average: 15 0.00 233.94 139.21 115367.26 0.00
0.00 0.00 225.03 0.00 17.71
Average: 16 0.00 245.41 158.94 253784.06 0.00
0.00 0.00 244.65 0.00 13.03
Average: 17 0.00 245.74 164.09 252402.41 0.00
0.00 0.00 244.85 0.00 10.71
Average: 18 0.00 245.56 161.32 253778.00 0.00
0.00 0.00 244.97 0.00 16.50
Average: 19 0.00 245.38 161.35 251789.41 0.00
0.00 0.00 244.53 0.00 16.62
Average: 20 0.00 244.97 201.59 213226.74 0.00
0.00 0.00 244.24 0.00 10.65
Average: 21 0.00 245.59 174.82 232072.85 0.00
0.00 0.00 244.82 0.00 32.24
Average: 22 0.00 245.44 157.47 263244.59 0.00
0.00 0.00 244.74 0.00 14.62
Average: 23 0.00 245.53 165.09 259398.41 0.00
0.00 0.00 244.94 0.00 11.12
Average: 24 0.00 245.38 179.47 242922.85 0.00
0.00 0.00 244.85 0.00 17.18
Average: 25 0.00 245.47 195.15 217419.79 0.00
0.00 0.00 244.50 0.00 17.62
Average: 26 0.00 245.41 182.18 236714.38 0.00
0.00 0.00 244.62 0.00 18.79
Average: 27 0.00 244.94 196.65 216886.68 0.00
0.00 0.00 244.44 0.00 10.06
Average: 28 0.00 247.35 0.00 0.00 0.00
0.00 0.00 249.91 0.00 51.03
Average: 29 0.00 228.94 0.00 0.00 0.00
0.00 0.00 249.79 0.00 90.38
Average: 30 0.00 244.50 0.00 0.00 0.00
0.00 0.00 249.82 0.00 46.41
Average: 31 0.00 249.53 0.00 0.00 0.00
0.00 0.00 249.97 0.00 18.00
Average: 32 0.00 243.82 0.00 0.00 0.00
0.00 0.00 249.97 0.00 27.79
Average: 33 0.00 247.03 0.00 0.00 0.00
0.00 0.00 249.47 0.00 48.35
Average: 34 0.00 249.38 0.00 0.00 0.00
0.00 0.00 249.94 0.00 21.59
Average: 35 0.00 226.00 0.00 0.00 0.00
0.00 0.00 249.88 0.00 77.41
Average: 36 0.00 244.29 0.00 0.00 0.00
0.00 0.00 249.85 0.00 51.18
Average: 37 0.00 247.71 0.00 0.00 0.00
0.00 0.00 249.97 0.00 20.76
Average: 38 0.00 241.85 0.00 0.00 0.00
0.00 0.00 249.68 0.00 65.74
Average: 39 0.00 249.00 0.00 0.00 0.00
0.00 0.00 249.94 0.00 42.76
Average: 40 0.00 248.85 0.00 0.00 0.00
0.00 0.00 249.85 0.00 16.50
Average: 41 0.00 249.12 0.00 0.00 0.00
0.00 0.00 249.94 0.00 21.76
Average: 42 0.00 240.26 112.85 137160.06 0.00
0.00 0.00 203.44 0.00 5.76
Average: 43 0.00 237.91 122.91 135139.38 0.00
0.00 0.00 231.79 0.00 5.59
Average: 44 0.00 249.15 0.00 0.00 0.00
0.00 0.00 248.26 0.00 27.56
Average: 45 0.00 249.26 0.00 0.00 0.00
0.00 0.00 248.62 0.00 40.35
Average: 46 0.00 249.26 0.00 0.00 0.00
0.00 0.00 248.47 0.00 82.65
Average: 47 0.00 249.26 0.00 0.00 0.00
0.00 0.00 248.50 0.00 69.85
Average: 48 0.00 249.09 0.00 0.00 0.00
0.00 0.00 248.38 0.00 58.06
Average: 49 0.00 249.32 0.00 0.00 0.00
0.00 0.00 248.59 0.00 63.59
Average: 50 0.00 249.35 0.00 0.00 0.00
0.00 0.00 248.32 0.00 39.97
Average: 51 0.00 249.24 0.00 0.00 0.00
0.00 0.00 248.29 0.00 67.56
Average: 52 0.00 249.18 0.00 0.00 0.00
0.00 0.00 247.91 0.00 39.03
Average: 53 0.00 249.18 0.00 0.00 0.00
0.00 0.00 248.41 0.00 72.85
Average: 54 0.00 249.18 0.00 0.00 0.00
0.00 0.00 248.06 0.00 38.15
Average: 55 0.00 249.41 0.18 0.00 0.00
0.00 0.00 248.71 0.00 69.59
W dniu 2017-08-12 o 19:27, Paweł Staszewski pisze:
> Hi and thanks for reply
>
>
>
> W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
>> On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> Hi
>>>
>>> I made some tests for performance comparison.
>> Thanks for doing this. Feel free to Cc me, if you do more of these
>> tests (so I don't miss them on the mailing list).
>>
>> I don't understand stand if you are reporting a potential problem?
>>
>> It would be good if you can provide a short summary section (of the
>> issue) in the _start_ of the email, and then provide all this nice data
>> afterwards, to back your case.
>>
>> My understanding is, you report:
>>
>> 1. VLANs on ixgbe show a 30-40% slowdown
>> 2. System stopped scaling after 7+ CPUs
> This is not only problem/bug report - but some kind of comparision
> plus some toughts about possible problems :)
> And can help somebody when searching the net for possible expectations :)
> Also - dono better list where are the smartest people that know what
> is going in kernel with networking :)
>
> Next time i will place summary on top - sorry :)
>
>>
>>> Tested HW (FORWARDING HOST):
>>>
>>> Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>> Interesting, I've not heard about a Intel CPU called "Gold" before now,
>> but it does exist:
>> https://ark.intel.com/products/123541/Intel-Xeon-Gold-6132-Processor-19_25M-Cache-2_60-GHz
>>
>>
>>> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
>>> (rev 01)
>> This is one of my all time favorite NICs!
> Yes this is a good NIC - will have connectx-4 2x100G by monday so will
> also do some tests
>
>>> Test diagram:
>>>
>>>
>>> TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
>>> (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
>>>
>>> Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
>>> 172.16.0.1 to 172.16.0.255
>>>
>>> TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
>> What kind of traffic flow? E.g. distribution, many/few source IPs...
>
> Traffic generator is pktgen so udp flows - better paste parameters
> from pktgen:
> UDP_MIN=9
> UDP_MAX=19
>
> pg_set $dev "dst_min 172.16.0.1"
> pg_set $dev "dst_max 172.16.0.100"
>
> # Setup random UDP port src range
> #pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "udp_src_min $UDP_MIN"
> pg_set $dev "udp_src_max $UDP_MAX"
>
>
>>
>>> Settings used for FORWARDING HOST (changed param. was only number of
>>> RSS
>>> combined queues + set affinity assignment for them to fit with first
>>> numa node where 2x10G port card is installed)
>>>
>>> ixgbe driver used from kernel (in-kernel build - not a module)
>>>
>> Nice with a script showing you setup, thanks. I would be good if it had
>> comments, telling why you think this is a needed setup adjustment.
>>
>>> #!/bin/sh
>>> ifc='enp216s0f0 enp216s0f1'
>>> for i in $ifc
>>> do
>>> ip link set up dev $i
>>> ethtool -A $i autoneg off rx off tx off
>> Good:
>> Turning off Ethernet flow control, to avoid receiver being the
>> bottleneck via pause-frames.
> Yes - enabled flow controll is really bad :)
>>> ethtool -G $i rx 4096 tx 1024
>> You adjust the RX and TX ring queue sizes, this have effects that you
>> don't realize. Especially for the ixgbe driver, which have a page
>> recycle trick tied to the RX ring queue size.
> rx ring 4096 and tx ring 1024
> - this is because have best performance then with average packet size
> from 64 to 1500 bytes
>
> Can be a little better performance for smaller frames like 64 - with
> rx ring set to 1024
> below 1 core/1 RSS queue with rx ring set to 1024
>
> 0;1;64;1530112;91772160;1529919;88724208
> 1;1;64;1531584;91872000;1531520;88813196
> 2;1;64;1531392;91895040;1531262;88831930
> 3;1;64;1530880;91875840;1531201;88783558
> 4;1;64;1530688;91829760;1530688;88768826
> 5;1;64;1530432;91810560;1530624;88764940
> 6;1;64;1530880;91868160;1530878;88787328
> 7;1;64;1530496;91845120;1530560;88765114
> 8;1;64;1530496;91837440;1530687;88772538
> 9;1;64;1530176;91795200;1530496;88735360
>
> so from 1.47Mpps to 1.53Mpps
>
> But with bigger packets > 200 performance is better when rx is set to
> 4096
>
>
>>
>>> ip link set $i txqueuelen 1000
>> Setting tx queue len to the default 1000 seems redundant.
> Yes cause i'm changing this parameter also to see if any impact on
> performance we have
>>
>>> ethtool -C $i rx-usecs 10
>> Adjusting this also have effects you might not realize. This actually
>> also affect the page recycle scheme of ixgbe. And can sometimes be
>> used to solve stalling on DMA TX completions, which could be you issue
>> here.
> same here - rx-usecs - setting to 10 was kind of compromise to have
> good performance with big ans small packet sizes
>
> Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to
> 256 (1Core/1RSS queue):
> 0;1;64;1506304;90424320;1506626;87402868
> 1;1;64;1505536;90343680;1504830;87321088
> 2;1;64;1506880;90416640;1507522;87388120
> 3;1;64;1511040;90700800;1511682;87684864
> 4;1;64;1511040;90681600;1511102;87662476
> 5;1;64;1511488;90712320;1511614;87673728
> 6;1;64;1511296;90700800;1511038;87669900
> 7;1;64;1513344;90773760;1513280;87751680
> 8;1;64;1513536;90850560;1513470;87807360
> 9;1;64;1512128;90696960;1512000;87696000
>
> And rx-usecs set to 1
> 0;1;64;1533632;92037120;1533504;88954368
> 1;1;64;1533632;92006400;1533570;88943348
> 2;1;64;1533504;91994880;1533504;88931980
> 3;1;64;1532864;91979520;1532674;88902516
> 4;1;64;1533952;92044800;1534080;88961792
> 5;1;64;1533888;92048640;1534270;88969100
> 6;1;64;1533952;92037120;1534082;88969216
> 7;1;64;1533952;92021760;1534208;88969332
> 8;1;64;1533056;91983360;1532930;88883724
> 9;1;64;1533760;92021760;1533886;88946828
>
> rx-useck set to 2
> 0;1;64;1522432;91334400;1522304;88301056
> 1;1;64;1521920;91330560;1522496;88286208
> 2;1;64;1522496;91322880;1522432;88304768
> 3;1;64;1523456;91422720;1523649;88382762
> 4;1;64;1527680;91676160;1527424;88601728
> 5;1;64;1527104;91626240;1526912;88572032
> 6;1;64;1527424;91641600;1527424;88590592
> 7;1;64;1526336;91572480;1526912;88523776
> 8;1;64;1527040;91637760;1526912;88579456
> 9;1;64;1527040;91595520;1526784;88553472
>
> rx-usecs set to 3
> 0;1;64;1526272;91549440;1526592;88527488
> 1;1;64;1526528;91560960;1526272;88516352
> 2;1;64;1525952;91580160;1525888;88527488
> 3;1;64;1525504;91511040;1524864;88456960
> 4;1;64;1526272;91568640;1526208;88494080
> 5;1;64;1525568;91545600;1525312;88494080
> 6;1;64;1526144;91584000;1526080;88512640
> 7;1;64;1525376;91530240;1525376;88482944
> 8;1;64;1526784;91607040;1526592;88549760
> 9;1;64;1526208;91560960;1526528;88512640
>
>
>>
>>> ethtool -L $i combined 16
>>> ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
>>> tx-nocache-copy on ntuple on
>> Here are many setting above.
> Yes mostly NIC defaults besides the ntuple that is on (for testing
> some nfc drop filters - and trying to test also tc-offload )
>
>> GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
>> this result in approx 10% slowdown.
> Ok lets give a try :)
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on
> ntuple on
> rx-usecs 10
> 1 CPU / 1 RSS QUEUE
>
> 0;1;64;1609344;96537600;1609279;93327104
> 1;1;64;1608320;96514560;1608256;93293812
> 2;1;64;1608000;96487680;1608125;93267770
> 3;1;64;1608320;96522240;1608576;93297524
> 4;1;64;1605888;96387840;1606211;93148986
> 5;1;64;1601472;96072960;1601600;92870644
> 6;1;64;1602624;96180480;1602243;92959674
> 7;1;64;1601728;96107520;1602113;92907764
> 8;1;64;1602176;96122880;1602176;92933806
> 9;1;64;1603904;96253440;1603777;93045208
>
> A little better performance 1.6Mpps
> But wondering if disabling tso will have no performance impact for tcp
> traffic ...
> Will try to get some pktgen like pktgen-dpdk that can generate also
> tcp traffic - to compare this.
>
>
>>
>> AFAIK "tx-nocache-copy on" was also determined to be a bad option.
> I set this to on cause i have better performance (a little 10kpps for
> this test)
> below same test as above with tx-nocache-copy off
>
> 0;1;64;1591552;95496960;1591230;92313654
> 1;1;64;1596224;95738880;1595842;92555066
> 2;1;64;1595456;95700480;1595201;92521774
> 3;1;64;1595456;95723520;1595072;92528966
> 4;1;64;1595136;95692800;1595457;92503040
> 5;1;64;1594624;95631360;1594496;92473402
> 6;1;64;1596224;95761920;1595778;92551180
> 7;1;64;1595200;95700480;1595331;92521542
> 8;1;64;1595584;95692800;1595457;92521426
> 9;1;64;1594624;95662080;1594048;92469574
>
>
>
>
>>
>> The "ntuple on" AFAIK disables the flow-director in the NIC. I though
>> this would actually help VLAN traffic, but I guess not.
> yes I enabled this cause was thinking that can help with traffic on vlans
>
> below same test with ntuple off
> so all settings for ixgbe:
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off
> ntuple off
> rx-usecs 10
> rx-flow-hash udp4 sdfn
>
> 0;1;64;1611840;96691200;1611905;93460794
> 1;1;64;1610688;96645120;1610818;93427328
> 2;1;64;1610752;96668160;1610497;93442176
> 3;1;64;1610624;96664320;1610817;93427212
> 4;1;64;1610752;96652800;1610623;93412480
> 5;1;64;1610048;96614400;1610112;93404940
> 6;1;64;1611264;96641280;1611390;93427212
> 7;1;64;1611008;96691200;1610942;93468160
> 8;1;64;1610048;96652800;1609984;93408652
> 9;1;64;1611136;96641280;1610690;93434636
>
> Performance is a little better
> and now with tx-nocache-copy on
>
> 0;1;64;1597248;95834880;1597311;92644096
> 1;1;64;1597888;95865600;1597824;92677446
> 2;1;64;1597952;95834880;1597822;92644038
> 3;1;64;1597568;95877120;1597375;92685044
> 4;1;64;1597184;95827200;1597314;92629190
> 5;1;64;1597696;95842560;1597565;92625652
> 6;1;64;1597312;95834880;1597376;92644038
> 7;1;64;1597568;95873280;1597634;92647924
> 8;1;64;1598400;95919360;1598849;92699602
> 9;1;64;1597824;95873280;1598208;92684928
>
>
> That is weird - so enabling tx-nocache-copy with disabled ntuple have
> bad performance impact - but with enabled ntuple there is no
> performance impact
>
>
>
>>
>>
>>> ethtool -N $i rx-flow-hash udp4 sdfn
>> Why do you change the NICs flow-hash?
> whan used 16 cores / 16 rss queues - there was better load
> distribution over all cores when sdfn rx-flow-hash enabled
>
>>
>>> done
>>>
>>> ip link set up dev enp216s0f0
>>> ip link set up dev enp216s0f1
>>>
>>> ip a a 10.0.0.1/30 dev enp216s0f0
>>>
>>> ip link add link enp216s0f1 name vlan1000 type vlan id 1000
>>> ip link set up dev vlan1000
>>> ip a a 10.0.0.5/30 dev vlan1000
>>>
>>>
>>> ip route add 172.16.0.0/12 via 10.0.0.6
>>>
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
>>> #cat /sys/devices/system/node/node1/cpulist
>>> #14-27,42-55
>>> #cat /sys/devices/system/node/node0/cpulist
>>> #0-13,28-41
>> Is this a NUMA system?
> This is 2x CPU 6132 - so have two separate pcie access to the nic -
> need to check what cpu is assigned to pcie where network card is
> connected to have network card on local cpu where all irq's are binded
>
>>
>>> #################################################
>>>
>>>
>>> Looks like forwarding performance when using vlans on ixgbe is less
>>> that
>>> without vlans for about 30-40% (wondering if this is some vlan
>>> offloading problem and ixgbe)
>> I would see this as a problem/bug that enabling VLANs cost this much.
> Yes - was thinking that with tx/rx vlan offloading there will be not
> much performance impact when vlans used.
>
>>> settings below:
>>>
>>> ethtool -k enp216s0f0
>>> Features for enp216s0f0:
>>> Cannot get device udp-fragmentation-offload settings: Operation not
>>> supported
>>> rx-checksumming: on
>>> tx-checksumming: on
>>> tx-checksum-ipv4: off [fixed]
>>> tx-checksum-ip-generic: on
>>> tx-checksum-ipv6: off [fixed]
>>> tx-checksum-fcoe-crc: off [fixed]
>>> tx-checksum-sctp: on
>>> scatter-gather: on
>>> tx-scatter-gather: on
>>> tx-scatter-gather-fraglist: off [fixed]
>>> tcp-segmentation-offload: on
>>> tx-tcp-segmentation: on
>>> tx-tcp-ecn-segmentation: off [fixed]
>>> tx-tcp-mangleid-segmentation: on
>>> tx-tcp6-segmentation: on
>>> udp-fragmentation-offload: off
>>> generic-segmentation-offload: off
>>> generic-receive-offload: on
>>> large-receive-offload: off
>>> rx-vlan-offload: on
>>> tx-vlan-offload: on
>>> ntuple-filters: on
>>> receive-hashing: on
>>> highdma: on [fixed]
>>> rx-vlan-filter: on
>>> vlan-challenged: off [fixed]
>>> tx-lockless: off [fixed]
>>> netns-local: off [fixed]
>>> tx-gso-robust: off [fixed]
>>> tx-fcoe-segmentation: off [fixed]
>>> tx-gre-segmentation: on
>>> tx-gre-csum-segmentation: on
>>> tx-ipxip4-segmentation: on
>>> tx-ipxip6-segmentation: on
>>> tx-udp_tnl-segmentation: on
>>> tx-udp_tnl-csum-segmentation: on
>>> tx-gso-partial: on
>>> tx-sctp-segmentation: off [fixed]
>>> tx-esp-segmentation: off [fixed]
>>> fcoe-mtu: off [fixed]
>>> tx-nocache-copy: on
>>> loopback: off [fixed]
>>> rx-fcs: off [fixed]
>>> rx-all: off
>>> tx-vlan-stag-hw-insert: off [fixed]
>>> rx-vlan-stag-hw-parse: off [fixed]
>>> rx-vlan-stag-filter: off [fixed]
>>> l2-fwd-offload: off
>>> hw-tc-offload: off
>>> esp-hw-offload: off [fixed]
>>> esp-tx-csum-hw-offload: off [fixed]
>>> rx-udp_tunnel-port-offload: on
>>>
>>>
>>> Another thing is that forwarding performance does not scale with number
>>> of cores when 7+ cores are reached
>> I've seen problems with using Hyper-Threading CPUs. Could it be that
>> above 7 CPUs you are starting to use sibling-cores ?
> Turbostats can help here:
> Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI
> C1 C2 C1% C2% CPU%c1 CPU%c6 CoreTmp PkgTmp
> PkgWatt RAMWatt PKG_% RAM_%
> - - - 72 2.27 3188 2600 194844 0
> 64 69282 0.07 97.83 18.38 79.36 -4 54 123.49 16.08
> 0.00 0.00
> 0 0 0 8 0.74 1028 2600 1513 0
> 32 1462 1.50 97.99 10.92 88.34 47 51 58.34 5.34
> 0.00 0.00
> 0 0 28 7 0.67 1015 2600 1255 0
> 12 1249 0.96 98.61 10.99
> 0 1 1 7 0.68 1019 2600 1260 0
> 0 1260 0.00 99.54 8.44 90.88 49
> 0 1 29 9 0.71 1208 2600 1252 0
> 0 1253 0.00 99.48 8.41
> 0 2 2 7 0.67 1019 2600 1261 0
> 0 1260 0.00 99.54 8.44 90.89 48
> 0 2 30 7 0.67 1017 2600 1255 0
> 0 1255 0.00 99.55 8.44
> 0 3 3 7 0.68 1019 2600 1260 0
> 0 1259 0.00 99.53 8.46 90.86 -4
> 0 3 31 7 0.67 1017 2600 1256 0
> 0 1256 0.00 99.55 8.46
> 0 4 4 7 0.67 1027 2600 1260 0
> 0 1260 0.00 99.54 8.43 90.90 -4
> 0 4 32 7 0.66 1018 2600 1255 0
> 0 1255 0.00 99.55 8.44
> 0 5 5 7 0.68 1020 2600 1260 0
> 0 1257 0.00 99.54 8.44 90.89 50
> 0 5 33 7 0.68 1019 2600 1255 0
> 0 1255 0.00 99.55 8.43
> 0 6 6 7 0.70 1019 2600 1260 0
> 0 1259 0.00 99.53 8.43 90.87 -4
> 0 6 34 7 0.70 1019 2600 1255 0
> 0 1255 0.00 99.54 8.43
> 0 8 7 7 0.68 1019 2600 1262 0
> 0 1261 0.00 99.52 8.42 90.90 50
> 0 8 35 7 0.67 1019 2600 1255 0
> 0 1255 0.00 99.55 8.43
> 0 9 8 7 0.68 1019 2600 1260 0
> 0 1257 0.00 99.54 8.40 90.92 49
> 0 9 36 7 0.66 1017 2600 1255 0
> 0 1255 0.00 99.55 8.41
> 0 10 9 7 0.66 1018 2600 1257 0
> 0 1257 0.00 99.54 8.40 90.94 -4
> 0 10 37 7 0.66 1018 2600 1255 0
> 0 1255 0.00 99.55 8.41
> 0 11 10 7 0.66 1019 2600 1257 0
> 0 1259 0.00 99.54 8.56 90.77 -4
> 0 11 38 7 0.66 1018 2600 1255 0
> 3 1252 0.19 99.36 8.57
> 0 12 11 7 0.67 1019 2600 1260 0
> 0 1260 0.00 99.54 8.44 90.88 -4
> 0 12 39 7 0.67 1019 2600 1255 0
> 0 1256 0.00 99.55 8.44
> 0 13 12 7 0.68 1019 2600 1257 0
> 4 1254 0.32 99.22 8.67 90.65 -4
> 0 13 40 7 0.69 1019 2600 1256 0
> 4 1253 0.24 99.31 8.66
> 0 14 13 7 0.71 1020 2600 1260 0
> 0 1259 0.00 99.53 8.41 90.88 -4
> 0 14 41 7 0.72 1020 2600 1255 0
> 0 1255 0.00 99.54 8.40
> 1 0 14 3564 99.19 3594 2600 125472 0
> 0 0 0.00 0.00 0.81 0.00 54 54 65.15 10.74
> 0.00 0.00
> 1 0 42 3 0.07 3701 2600 1255 0
> 0 1255 0.00 99.95 99.93
> 1 1 15 11 0.32 3301 2600 1257 0
> 0 1257 0.00 99.81 26.37 73.31 42
> 1 1 43 10 0.31 3301 2600 1255 0
> 0 1255 0.00 99.82 26.38
> 1 2 16 10 0.31 3301 2600 1257 0
> 0 1257 0.00 99.81 26.37 73.32 39
> 1 2 44 10 0.32 3301 2600 1255 0
> 0 1255 0.00 99.82 26.36
> 1 3 17 10 0.32 3301 2600 1257 0
> 0 1257 0.00 99.81 26.40 73.28 39
> 1 3 45 11 0.32 3301 2600 1255 0
> 0 1255 0.00 99.81 26.40
> 1 4 18 10 0.32 3301 2600 1257 0
> 0 1257 0.00 99.82 26.40 73.28 40
> 1 4 46 11 0.32 3301 2600 1255 0
> 0 1255 0.00 99.82 26.40
> 1 5 19 11 0.33 3301 2600 1257 0
> 0 1257 0.00 99.81 26.40 73.27 39
> 1 5 47 11 0.33 3300 2600 1255 0
> 0 1255 0.00 99.82 26.40
> 1 6 20 12 0.35 3301 2600 1257 0
> 0 1257 0.00 99.81 26.38 73.27 42
> 1 6 48 12 0.36 3301 2600 1255 0
> 0 1255 0.00 99.81 26.37
> 1 8 21 11 0.33 3301 2600 1257 0
> 0 1257 0.00 99.82 26.37 73.29 42
> 1 8 49 11 0.33 3301 2600 1255 0
> 0 1255 0.00 99.82 26.38
> 1 9 22 10 0.32 3300 2600 1257 0
> 0 1257 0.00 99.82 26.35 73.34 41
> 1 9 50 10 0.30 3301 2600 1255 0
> 0 1255 0.00 99.82 26.36
> 1 10 23 10 0.31 3301 2600 1257 0
> 0 1257 0.00 99.82 26.37 73.33 41
> 1 10 51 10 0.31 3301 2600 1255 0
> 0 1255 0.00 99.82 26.36
> 1 11 24 10 0.32 3301 2600 1257 0
> 0 1257 0.00 99.81 26.62 73.06 41
> 1 11 52 10 0.32 3301 2600 1255 0
> 4 1251 0.32 99.50 26.62
> 1 12 25 11 0.33 3301 2600 1257 0
> 0 1257 0.00 99.81 26.39 73.28 41
> 1 12 53 11 0.33 3301 2600 1258 0
> 0 1254 0.00 99.82 26.38
> 1 13 26 12 0.36 3317 2600 1259 0
> 0 1258 0.00 99.79 26.41 73.23 39
> 1 13 54 11 0.34 3301 2600 1255 0
> 0 1254 0.00 99.82 26.42
> 1 14 27 12 0.36 3301 2600 1257 0
> 5 1251 0.24 99.58 26.54 73.10 41
> 1 14 55 12 0.36 3300 2600 1255 0
> 0 1254 0.00 99.82 26.54
>
>
> So it looks like in all tests i'm using core+sibling
> But side effect of this is that :
> 33 * 100.0 = 3300.0 MHz max turbo 28 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 24 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 20 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 14 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 12 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 8 active cores
> 35 * 100.0 = 3500.0 MHz max turbo 4 active cores
> 37 * 100.0 = 3700.0 MHz max turbo 2 active cores
>
> So more cores = less MHz per core/sibling
>
>>
>>> perf top:
>>>
>>> PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> 16.32% [kernel] [k] skb_dst_force
>>> 16.30% [kernel] [k] dst_release
>>> 15.11% [kernel] [k] rt_cache_valid
>>> 12.62% [kernel] [k] ipv4_mtu
>> It seems a little strange that these 4 functions are on the top
> Yes dono why there is ipv4_mtu called and taking soo much cycles
>
>>
>>> 5.60% [kernel] [k] do_raw_spin_lock
>> Why is calling/taking this lock? (Use perf call-graph recording).
> can be hard to paste it here:)
> attached file
>
>>
>>> 3.03% [kernel] [k] fib_table_lookup
>>> 2.70% [kernel] [k] ip_finish_output2
>>> 2.10% [kernel] [k] dev_gro_receive
>>> 1.89% [kernel] [k] eth_type_trans
>>> 1.81% [kernel] [k] ixgbe_poll
>>> 1.15% [kernel] [k] ixgbe_xmit_frame_ring
>>> 1.06% [kernel] [k] __build_skb
>>> 1.04% [kernel] [k] __dev_queue_xmit
>>> 0.97% [kernel] [k] ip_rcv
>>> 0.78% [kernel] [k] netif_skb_features
>>> 0.74% [kernel] [k] ipt_do_table
>> Unloading netfilter modules, will give more performance, but it
>> semifake to do so.
> Compiled in kernel - only in filter mode - with ipv4+ipv6 - no other
> modules conntrack or other .
>>> 0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
>>> 0.64% [kernel] [k] ip_forward
>>> 0.59% [kernel] [k] __netif_receive_skb_core
>>> 0.55% [kernel] [k] dev_hard_start_xmit
>>> 0.53% [kernel] [k] ip_route_input_rcu
>>> 0.53% [kernel] [k] ip_rcv_finish
>>> 0.51% [kernel] [k] page_frag_free
>>> 0.50% [kernel] [k] kmem_cache_alloc
>>> 0.50% [kernel] [k] udp_v4_early_demux
>>> 0.44% [kernel] [k] skb_release_data
>>> 0.42% [kernel] [k] inet_gro_receive
>>> 0.40% [kernel] [k] sch_direct_xmit
>>> 0.39% [kernel] [k] __local_bh_enable_ip
>>> 0.33% [kernel] [k] netdev_pick_tx
>>> 0.33% [kernel] [k] validate_xmit_skb
>>> 0.28% [kernel] [k] fib_validate_source
>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>> 0.25% [kernel] [k] eth_header
>>> 0.23% [kernel] [k] get_dma_ops
>>> 0.22% [kernel] [k] skb_network_protocol
>>> 0.21% [kernel] [k] ip_output
>>> 0.21% [kernel] [k] vlan_dev_hard_start_xmit
>>> 0.20% [kernel] [k] ixgbe_alloc_rx_buffers
>>> 0.18% [kernel] [k] nf_hook_slow
>>> 0.18% [kernel] [k] apic_timer_interrupt
>>> 0.18% [kernel] [k] virt_to_head_page
>>> 0.18% [kernel] [k] build_skb
>>> 0.16% [kernel] [k] swiotlb_map_page
>>> 0.16% [kernel] [k] ip_finish_output
>>> 0.16% [kernel] [k] udp4_gro_receive
>>>
>>>
>>> RESULTS:
>>>
>>> CSV format - delimeter ";"
>>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;1;64;1470912;88247040;1470720;85305530
>>> 1;1;64;1470912;88285440;1470977;85335110
>>> 2;1;64;1470464;88247040;1470402;85290508
>>> 3;1;64;1471424;88262400;1471230;85353728
>>> 4;1;64;1468736;88166400;1468672;85201652
>>> 5;1;64;1470016;88181760;1469949;85234944
>>> 6;1;64;1470720;88247040;1470466;85290624
>>> 7;1;64;1471232;88277760;1471167;85346246
>>> 8;1;64;1469184;88170240;1469249;85216326
>>> 9;1;64;1470592;88227840;1470847;85294394
>> Single core 1.47Mpps seems a little low, I would expect 2Mpps.
>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;2;64;2413120;144802560;2413245;139975924
>>> 1;2;64;2415296;144913920;2415356;140098188
>>> 2;2;64;2416768;144898560;2416573;140105670
>>> 3;2;64;2418176;145056000;2418110;140261806
>>> 4;2;64;2416512;144990720;2416509;140172950
>>> 5;2;64;2415168;144860160;2414466;140064780
>>> 6;2;64;2416960;144983040;2416833;140190930
>>> 7;2;64;2413632;144768000;2413568;140001734
>>> 8;2;64;2415296;144898560;2414589;140087168
>>> 9;2;64;2416576;144963840;2416892;140190930
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;3;64;3419008;205155840;3418882;198239244
>>> 1;3;64;3428032;205585920;3427971;198744234
>>> 2;3;64;3425472;205536000;3425344;198677260
>>> 3;3;64;3425088;205470720;3425156;198603136
>>> 4;3;64;3427648;205693440;3426883;198773888
>>> 5;3;64;3426880;205670400;3427392;198796044
>>> 6;3;64;3429120;205678080;3430140;198848186
>>> 7;3;64;3422976;205355520;3423490;198458136
>>> 8;3;64;3423168;205336320;3423486;198495372
>>> 9;3;64;3424384;205493760;3425538;198617868
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;4;64;4406464;264364800;4405244;255560296
>>> 1;4;64;4404672;264349440;4405122;255541504
>>> 2;4;64;4402368;264049920;4403326;255188864
>>> 3;4;64;4401344;264076800;4400702;255207134
>>> 4;4;64;4385536;263074560;4386620;254312716
>>> 5;4;64;4386560;263189760;4385404;254379532
>>> 6;4;64;4398784;263857920;4399031;255025288
>>> 7;4;64;4407232;264445440;4407998;255637900
>>> 8;4;64;4413184;264698880;4413758;255875816
>>> 9;4;64;4411328;264526080;4411906;255712372
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;5;64;5094464;305871360;5094464;295657262
>>> 1;5;64;5090816;305514240;5091201;295274810
>>> 2;5;64;5088384;305387520;5089792;295175108
>>> 3;5;64;5079296;304869120;5079484;294680368
>>> 4;5;64;5092992;305544960;5094207;295349166
>>> 5;5;64;5092416;305502720;5093372;295334260
>>> 6;5;64;5080896;304896000;5081090;294677004
>>> 7;5;64;5085376;305114880;5086401;294933058
>>> 8;5;64;5092544;305575680;5092036;295356938
>>> 9;5;64;5093056;305652480;5093832;295449506
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;6;64;5705088;342351360;5705784;330965110
>>> 1;6;64;5710272;342743040;5707591;331373952
>>> 2;6;64;5703424;342182400;5701826;330776552
>>> 3;6;64;5708736;342604800;5707963;331147462
>>> 4;6;64;5710144;342654720;5712067;331202910
>>> 5;6;64;5712064;342777600;5711361;331292288
>>> 6;6;64;5710144;342585600;5708607;331144272
>>> 7;6;64;5699840;342021120;5697853;330609222
>>> 8;6;64;5701184;342124800;5702909;330653592
>>> 9;6;64;5711360;342735360;5713283;331247686
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;7;64;6244416;374603520;6243591;362180072
>>> 1;7;64;6230912;374016000;6231490;361534126
>>> 2;7;64;6244800;374776320;6244866;362224326
>>> 3;7;64;6238720;374376960;6238261;361838510
>>> 4;7;64;6218816;373079040;6220413;360683962
>>> 5;7;64;6224320;373566720;6225086;361017404
>>> 6;7;64;6224000;373570560;6221370;360936088
>>> 7;7;64;6210048;372741120;6210627;360212654
>>> 8;7;64;6231616;374035200;6231537;361445502
>>> 9;7;64;6227840;373724160;6228802;361162752
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;8;64;6251840;375144960;6251849;362609678
>>> 1;8;64;6250816;375014400;6250881;362547038
>>> 2;8;64;6257728;375432960;6257160;362911104
>>> 3;8;64;6255552;375325440;6255622;362822074
>>> 4;8;64;6243776;374576640;6243270;362120622
>>> 5;8;64;6237184;374296320;6237690;361790080
>>> 6;8;64;6240960;374415360;6240714;361927366
>>> 7;8;64;6222784;373317120;6223746;360854424
>>> 8;8;64;6225920;373593600;6227014;361154980
>>> 9;8;64;6238528;374304000;6237701;361845238
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;14;64;6486144;389184000;6486135;376236488
>>> 1;14;64;6454912;387390720;6454222;374466734
>>> 2;14;64;6441152;386480640;6440431;373572780
>>> 3;14;64;6450240;386972160;6450870;374070014
>>> 4;14;64;6465600;387997440;6467221;375089654
>>> 5;14;64;6448384;386860800;6448000;373980230
>>> 6;14;64;6452352;387095040;6452148;374168904
>>> 7;14;64;6441984;386507520;6443203;373665058
>>> 8;14;64;6456704;387340800;6455744;374429092
>>> 9;14;64;6464640;387901440;6465218;374949004
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;16;64;6939008;416325120;6938696;402411192
>>> 1;16;64;6941952;416444160;6941745;402558918
>>> 2;16;64;6960576;417584640;6960707;403698718
>>> 3;16;64;6940736;416486400;6941820;402503876
>>> 4;16;64;6927680;415741440;6927420;401853870
>>> 5;16;64;6929792;415687680;6929917;401839196
>>> 6;16;64;6950400;416989440;6950661;403026166
>>> 7;16;64;6953664;417216000;6953454;403260544
>>> 8;16;64;6948480;416851200;6948800;403023266
>>> 9;16;64;6924160;415422720;6924092;401542468
>> I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
>> issue/bug. You could be stalling on DMA TX completion being too slow,
>> but you already increased the interval and increased the TX ring queue
>> size. You could play with those setting and see if it changes this?
>>
>> Could you try my napi_monitor tool in:
>> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
>>
>> Also provide the output from:
>> mpstat -P ALL -u -I SCPU -I SUM 2
> with 16 cores / 16 RSS queues
> Average: CPU %usr %nice %sys %iowait %irq %soft
> %steal %guest %gnice %idle
> Average: all 0.00 0.00 0.01 0.00 0.00 28.57
> 0.00 0.00 0.00 71.42
> Average: 0 0.00 0.00 0.04 0.00 0.00 0.08
> 0.00 0.00 0.00 99.88
> Average: 1 0.00 0.00 0.12 0.00 0.00 0.00
> 0.00 0.00 0.00 99.88
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 7 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 11 0.08 0.00 0.04 0.00 0.00 0.00
> 0.00 0.00 0.00 99.88
> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 14 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 15 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 16 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 17 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 18 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 19 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 20 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 21 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 22 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 23 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 24 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 25 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 26 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 27 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 28 0.00 0.00 0.04 0.00 0.00 0.00
> 0.00 0.00 0.00 99.96
> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 33 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 36 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 37 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 39 0.04 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 99.96
> Average: 40 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 41 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 42 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 43 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 44 0.00 0.00 0.04 0.17 0.00 0.00
> 0.00 0.00 0.00 99.79
> Average: 45 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 46 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 47 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 48 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 49 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 50 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 51 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 52 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 53 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 54 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 55 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
>
> Average: CPU intr/s
> Average: all 123596.08
> Average: 0 646.38
> Average: 1 500.54
> Average: 2 511.67
> Average: 3 534.25
> Average: 4 542.21
> Average: 5 531.54
> Average: 6 554.58
> Average: 7 535.88
> Average: 8 544.58
> Average: 9 536.42
> Average: 10 575.46
> Average: 11 601.12
> Average: 12 502.08
> Average: 13 575.46
> Average: 14 5917.92
> Average: 15 5949.58
> Average: 16 7021.29
> Average: 17 7299.71
> Average: 18 7391.67
> Average: 19 7354.25
> Average: 20 7543.42
> Average: 21 7354.25
> Average: 22 7322.33
> Average: 23 7368.71
> Average: 24 7429.00
> Average: 25 7406.46
> Average: 26 7400.67
> Average: 27 7447.21
> Average: 28 517.00
> Average: 29 549.54
> Average: 30 529.33
> Average: 31 533.83
> Average: 32 541.25
> Average: 33 541.17
> Average: 34 532.50
> Average: 35 545.17
> Average: 36 528.96
> Average: 37 509.92
> Average: 38 520.12
> Average: 39 523.29
> Average: 40 530.75
> Average: 41 542.33
> Average: 42 5921.71
> Average: 43 5949.42
> Average: 44 503.04
> Average: 45 542.75
> Average: 46 582.50
> Average: 47 581.71
> Average: 48 495.29
> Average: 49 524.38
> Average: 50 527.92
> Average: 51 528.12
> Average: 52 456.38
> Average: 53 477.00
> Average: 54 440.92
> Average: 55 568.83
>
> Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
> IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
> Average: 0 0.00 250.00 0.17 87.00
> 0.00 0.00 45.46 250.00 0.00 13.75
> Average: 1 0.00 233.42 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 17.21
> Average: 2 0.00 249.04 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 12.67
> Average: 3 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 34.42
> Average: 4 0.00 248.67 0.17 0.00
> 0.00 0.00 0.00 249.96 0.00 43.42
> Average: 5 0.00 249.46 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 32.17
> Average: 6 0.00 249.79 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 54.92
> Average: 7 0.00 240.12 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 45.79
> Average: 8 0.00 247.42 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 47.25
> Average: 9 0.00 249.29 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 37.17
> Average: 10 0.00 248.75 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 76.79
> Average: 11 0.00 249.29 0.00 0.00
> 0.00 0.00 42.79 249.83 0.00 59.21
> Average: 12 0.00 249.83 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 2.29
> Average: 13 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 75.62
> Average: 14 0.00 148.21 0.17 5758.04
> 0.00 0.00 0.00 8.42 0.00 3.08
> Average: 15 0.00 148.42 0.46 5789.25
> 0.00 0.00 0.00 8.33 0.00 3.12
> Average: 16 0.00 142.62 0.79 6866.46
> 0.00 0.00 0.00 8.29 0.00 3.12
> Average: 17 0.00 143.17 0.42 7145.00
> 0.00 0.00 0.00 8.08 0.00 3.04
> Average: 18 0.00 153.62 0.42 7226.42
> 0.00 0.00 0.00 8.04 0.00 3.17
> Average: 19 0.00 150.46 0.46 7192.21
> 0.00 0.00 0.00 8.04 0.00 3.08
> Average: 20 0.00 145.21 0.17 7386.50
> 0.00 0.00 0.00 8.29 0.00 3.25
> Average: 21 0.00 150.96 0.46 7191.37
> 0.00 0.00 0.00 8.25 0.00 3.21
> Average: 22 0.00 146.67 0.54 7163.96
> 0.00 0.00 0.00 8.04 0.00 3.12
> Average: 23 0.00 151.38 0.42 7205.75
> 0.00 0.00 0.00 8.00 0.00 3.17
> Average: 24 0.00 153.33 0.17 7264.12
> 0.00 0.00 0.00 8.08 0.00 3.29
> Average: 25 0.00 153.21 0.17 7241.83
> 0.00 0.00 0.00 7.96 0.00 3.29
> Average: 26 0.00 153.96 0.17 7234.88
> 0.00 0.00 0.00 8.38 0.00 3.29
> Average: 27 0.00 151.71 0.79 7283.25
> 0.00 0.00 0.00 8.04 0.00 3.42
> Average: 28 0.00 245.71 0.00 0.00
> 0.00 0.00 0.00 249.50 0.00 21.79
> Average: 29 0.00 233.21 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 66.46
> Average: 30 0.00 248.92 0.00 0.00
> 0.00 0.00 0.00 250.00 0.00 30.42
> Average: 31 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 33.96
> Average: 32 0.00 248.67 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 42.62
> Average: 33 0.00 249.46 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 41.79
> Average: 34 0.00 249.79 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 32.83
> Average: 35 0.00 240.12 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 55.08
> Average: 36 0.00 247.42 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 31.58
> Average: 37 0.00 249.29 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 10.71
> Average: 38 0.00 248.75 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 21.50
> Average: 39 0.00 249.50 0.00 0.00
> 0.00 0.00 0.00 249.83 0.00 23.96
> Average: 40 0.00 249.83 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 30.96
> Average: 41 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 42.50
> Average: 42 0.00 148.38 0.71 5761.00
> 0.00 0.00 0.00 8.25 0.00 3.38
> Average: 43 0.00 147.21 0.50 5790.33
> 0.00 0.00 0.00 8.00 0.00 3.38
> Average: 44 0.00 248.96 0.00 0.00
> 0.00 0.00 0.00 248.13 0.00 5.96
> Average: 45 0.00 249.04 0.00 0.00
> 0.00 0.00 0.00 248.88 0.00 44.83
> Average: 46 0.00 248.96 0.00 0.00
> 0.00 0.00 0.00 248.58 0.00 84.96
> Average: 47 0.00 249.00 0.00 0.00
> 0.00 0.00 0.00 248.75 0.00 83.96
> Average: 48 0.00 249.12 0.00 0.00
> 0.00 0.00 0.00 132.83 0.00 113.33
> Average: 49 0.00 249.12 0.00 0.00
> 0.00 0.00 0.00 248.62 0.00 26.62
> Average: 50 0.00 248.92 0.00 0.00
> 0.00 0.00 0.00 248.58 0.00 30.42
> Average: 51 0.00 249.08 0.00 0.00
> 0.00 0.00 0.00 248.42 0.00 30.63
> Average: 52 0.00 249.21 0.00 0.00
> 0.00 0.00 0.00 131.96 0.00 75.21
> Average: 53 0.00 249.08 0.00 0.00
> 0.00 0.00 0.00 136.12 0.00 91.79
> Average: 54 0.00 249.00 0.00 0.00
> 0.00 0.00 0.00 136.79 0.00 55.12
> Average: 55 0.00 249.04 0.00 0.00
> 0.00 0.00 0.00 248.71 0.00 71.08
>
>
>
^ permalink raw reply
* Re: Regression: Bug 196547 - Since 4.12 - bonding module not working with wireless drivers
From: Andreas Born @ 2017-08-13 17:42 UTC (permalink / raw)
To: James Feeney
Cc: Kalle Valo, Arend van Spriel, Mahesh Bandewar, Andy Gospodarek,
David Miller, netdev-u79uwXL29TY76Z2rM5mHXA,
linux-wireless-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <deb37db2-a55b-4010-475a-e756e67d3bc9-fvV4AYHggTTR7s880joybQ@public.gmane.org>
2017-08-12 21:30 GMT+02:00 James Feeney <james-fvV4AYHggTTR7s880joybQ@public.gmane.org>:
>
>
> Andreas patch failed to address the continuous, *10-times per second* warning
> which will "spam" the log file, sometimes the console, whenever the test fails:
> if (bond_update_speed_duplex(slave) && bond_needs_speed_duplex(bond)) {...}
> which then has:
> netdev_warn(bond->dev, "failed to get link speed/duplex for%s\n",
> slave->dev->name);
>
> That is the sort of irresponsible code that "works fine" as long as there are no
> errors, and it never actually runs!
>
> I'm guessing that the simple fix is to use "net_warn_ratelimited()" instead of
> "netdev_warn()", where net/core/utils.c says:
>
> /*
> * All net warning printk()s should be guarded by this function.
> */
> int net_ratelimit(void)
> {
> return __ratelimit(&net_ratelimit_state);
> }
>
> though Andreas has also suggested "pr_warn_ratelimited()", which instead uses
> "__ratelimit(&_rs)".
>
> Here's a link to the rate-limiting patch, after Andreas' patch is already
> applied, since you say that David has already applied the first patch:
>
> https://bugzilla.kernel.org/attachment.cgi?id=257903
>
The other day I also sent a patch for part 2 of your bug report on
this list. Better safe than sorry by doing one thing per patch. This
second patch received feedback and its v2 is currently awaiting
review. Sorry, for informing you only now. I should have cc'd you in
the first place. You can lookup the state of that patch on patchwork.
[1] It's basically the same as previous versions just changing the
original code even less.
.
I sort of expect that this second patch won't be queued for stable.
But let's see... Essentially it's a regression too and additionally
would've been fixed by a plain revert.
On a side note I would recommend some of my own reading to you about
patch submission in general [1] and on netdev specifically [2].
Putting your suggested changes in the right form makes everyone's life
easier especially your own saving you lots of back and forth by mail.
Seeing the amount of mail on this list during the last days was reason
enough to comprehend the necessity of those standards.
And, just wondering, who's going to eventually close that bugreport?
https://bugzilla.kernel.org/show_bug.cgi?id=196547
Cheers
Andreas
[1] https://patchwork.ozlabs.org/patch/800770/
[2] https://www.kernel.org/doc/html/latest/process/submitting-patches.html
[3] https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/tree/Documentation/networking/netdev-FAQ.txt
^ permalink raw reply
* Re: unregister_netdevice: waiting for eth0 to become free. Usage count = 1
From: Wei Wang @ 2017-08-13 20:56 UTC (permalink / raw)
To: David Ahern
Cc: Ido Schimmel, Cong Wang, John Stultz, Martin KaFai Lau, lkml,
Network Development, Linux USB List, David S. Miller,
Felipe Balbi
In-Reply-To: <61d95fe2-edfd-9fc4-5e21-5b96e4e03c9f@gmail.com>
> Looking at my patch to move host routes from loopback to device with the
> address, I have this:
>
> @@ -2789,7 +2808,8 @@ static int fib6_ifdown(struct rt6_info *rt, void *arg)
> const struct arg_dev_net *adn = arg;
> const struct net_device *dev = adn->dev;
>
> - if ((rt->dst.dev == dev || !dev) &&
> + if ((rt->dst.dev == dev || !dev ||
> + (netdev_unregistering(dev) && rt->rt6i_idev->dev == dev)) &&
> rt != adn->net->ipv6.ip6_null_entry &&
> (rt->rt6i_nsiblings == 0 ||
> (dev && netdev_unregistering(dev)) ||
As you explained earlier, after your patch, all entries in the fib6
tree will have rt->dst.dev be the same as rt->rt6i_idev->dev except
those ones created by p6_rt_cache_alloc() and ip6_rt_pcpu_alloc().
Then the above newly added check is mainly to catch those cached dst
entries (created by ip6_rt_cached_alloc()). right?
And it is required because __ipv6_ifa_notify() -> ip6_del_rt() won't
take care of those cached dst entries.
Then I think I should wait for your patches to get merged before
submitting my patch?
Thanks.
Wei
On Sun, Aug 13, 2017 at 9:24 AM, David Ahern <dsahern@gmail.com> wrote:
> On 8/12/17 1:42 PM, Wei Wang wrote:
>> Hi Ido,
>>
>>>> - if ((rt->dst.dev == dev || !dev) &&
>>>> + if ((rt->dst.dev == dev || !dev ||
>>>> + rt->rt6i_idev->dev == dev) &&
>>>
>>> Can you please explain why this line is needed? While host routes aren't
>>> removed from the FIB by rt6_ifdown() (when dst.dev goes down), they are
>>> removed later on in addrconf_ifdown().
>>>
>>
>> Yes.. Agree. But one difference is that if the route is removed from
>> addrconf_ifdown(), dst_dev_put() won't be called to release the
>> devices before doing dst_release(). It is OK if dst_release() sees the
>> refcnt on dst already drops to 0 and directly destroys the dst. But I
>> think it will cause problem if at the time, the dst is still held by
>> some other users because then the refcnt on the device going down will
>> not get released.
>> That's why I think we should remove the dst with either dst->dev ==
>> going down dev or rt6->rt6i_idev->dev == going down dev from the fib6
>> tree always because there, we always call dst_dev_put() to release the
>> device.
>>
>>> With your patch, if I check the return value of ip6_del_rt() in
>>> __ipv6_ifa_notify() I see that -ENONET is returned. Because the host
>>> route was already removed by rt6_ifdown(). When the line in question is
>>> removed from the patch I don't get the error anymore.
>>>
>>
>> Right. That is expected as the route is already removed from the tree.
>>
>>> Is it possible that in John's case the host route was correctly removed
>>> from the FIB and that the unreleased reference was due to a wrong check
>>> in ip6_dst_ifdown() (which you patched correctly AFAICT)?
>>>
>>
>> Yes. possible. But as I explained earlier, I still think we should
>> also remove routes with rt6->rt6i_idev->dev == going down dev from the
>> tree.
>
> Looking at my patch to move host routes from loopback to device with the
> address, I have this:
>
> @@ -2789,7 +2808,8 @@ static int fib6_ifdown(struct rt6_info *rt, void *arg)
> const struct arg_dev_net *adn = arg;
> const struct net_device *dev = adn->dev;
>
> - if ((rt->dst.dev == dev || !dev) &&
> + if ((rt->dst.dev == dev || !dev ||
> + (netdev_unregistering(dev) && rt->rt6i_idev->dev == dev)) &&
> rt != adn->net->ipv6.ip6_null_entry &&
> (rt->rt6i_nsiblings == 0 ||
> (dev && netdev_unregistering(dev)) ||
>
>
^ permalink raw reply
* RE: [PATCH net-next v2] openvswitch: enable NSH support
From: Jan Scheurich @ 2017-08-13 21:13 UTC (permalink / raw)
To: Jiri Benc; +Cc: Yang, Yi Y, netdev@vger.kernel.org, dev@openvswitch.org
In-Reply-To: <20170811122256.2cec123e@griffin>
> From: Jiri Benc [mailto:jbenc@redhat.com]
> Sent: Friday, 11 August, 2017 12:23
>
> On Fri, 11 Aug 2017 10:09:36 +0000, Jan Scheurich wrote:
> > Unless someone can explain to me why the datapath should understand the
> > internal structure/format of metadata in push_nsh, I would strongly
> > vote to keep the metadata as variable length octet sequence in the
> > non-structured OVS_ACTION_ATTR_PUSH_NSH
>
> Could be but it still needs to be in a different attribute and not in
> the ovs_action_push_nsh structure.
>
> Separate attributes for MD1/MD2 has the advantage of easier validation:
> with a separate MD1 type attribute, the size check is easier. With an
> unstructured MD attribute, we'd need to look into the
> OVS_ACTION_ATTR_NSH_BASE_HEADER attribute for mdtype and then validate
> the unstructured MD attribute size manually. Not a big deal, though.
> I don't have strong opinion here.
>
> But I do have strong opinion that MD needs to go into a separate
> attribute, whether there are separate attributes for MD1/2 or not.
Jiri, I am not too familiar with conventions on the OVS netlink interface regarding the handling of variable length fields. What is the benefit of structuring the push_nsh action into
OVS_ACTION_ATTR_PUSH_NSH
+-- OVS_ACTION_ATTR_NSH_BASE_HEADER
+-- OVS_ACTION_ATTR_NSH_MD
instead of grouping the base header fields and the variable length MD into a single monolithic attribute OVS_ACTION_ATTR_PUSH_NSH? Is the main concern a potential future need to add additional parameters to the push_nsh datapath action? Are there examples for such structured actions other than OVS_ACTION_ATTR_CT where the need is obvious because it is very polymorphic?
BR, Jan
BTW: The name OVS_ACTION_ATTR_NSH_BASE_HEADER is misleading because in the NSH draft the term base header is used for the first 32-bit word, whereas here it includes also the 32-bit Service Path header.
^ permalink raw reply
* [PATCH net] ipv4: route: fix inet_rtm_getroute induced crash
From: Florian Westphal @ 2017-08-13 22:52 UTC (permalink / raw)
To: netdev; +Cc: Florian Westphal, Roopa Prabhu, David Ahern
"ip route get $daddr iif eth0 from $saddr" causes:
BUG: KASAN: use-after-free in ip_route_input_rcu+0x1535/0x1b50
Call Trace:
ip_route_input_rcu+0x1535/0x1b50
ip_route_input_noref+0xf9/0x190
tcp_v4_early_demux+0x1a4/0x2b0
ip_rcv+0xbcb/0xc05
__netif_receive_skb+0x9c/0xd0
netif_receive_skb_internal+0x5a8/0x890
Problem is that inet_rtm_getroute calls either ip_route_input_rcu (if an
iif was provided) or ip_route_output_key_hash_rcu.
But ip_route_input_rcu, unlike ip_route_output_key_hash_rcu, already
associates the dst_entry with the skb. This clears the SKB_DST_NOREF
bit (i.e. skb_dst_drop will release/free the entry while it should not).
Thus only set the dst if we called ip_route_output_key_hash_rcu().
I tested this patch by running:
while true;do ip r get 10.0.1.2;done > /dev/null &
while true;do ip r get 10.0.1.2 iif eth0 from 10.0.1.1;done > /dev/null &
... and saw no crash or memory leak.
Cc: Roopa Prabhu <roopa@cumulusnetworks.com>
Cc: David Ahern <dsahern@gmail.com>
Fixes: ba52d61e0ff ("ipv4: route: restore skb_dst_set in inet_rtm_getroute")
Signed-off-by: Florian Westphal <fw@strlen.de>
---
net/ipv4/route.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 0383e66f59bc..7effa62beed3 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -2750,12 +2750,13 @@ static int inet_rtm_getroute(struct sk_buff *in_skb, struct nlmsghdr *nlh,
err = 0;
if (IS_ERR(rt))
err = PTR_ERR(rt);
+ else
+ skb_dst_set(skb, &rt->dst);
}
if (err)
goto errout_free;
- skb_dst_set(skb, &rt->dst);
if (rtm->rtm_flags & RTM_F_NOTIFY)
rt->rt_flags |= RTCF_NOTIFY;
--
2.13.0
^ permalink raw reply related
* Re: unregister_netdevice: waiting for eth0 to become free. Usage count = 1
From: David Ahern @ 2017-08-13 23:08 UTC (permalink / raw)
To: Wei Wang
Cc: Ido Schimmel, Cong Wang, John Stultz, Martin KaFai Lau, lkml,
Network Development, Linux USB List, David S. Miller,
Felipe Balbi
In-Reply-To: <CAEA6p_BtBdPviFVRC4UoJchbz7eH9fM=KDihYA3zUQebmNBpdA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
On 8/13/17 2:56 PM, Wei Wang wrote:
>> Looking at my patch to move host routes from loopback to device with the
>> address, I have this:
>>
>> @@ -2789,7 +2808,8 @@ static int fib6_ifdown(struct rt6_info *rt, void *arg)
>> const struct arg_dev_net *adn = arg;
>> const struct net_device *dev = adn->dev;
>>
>> - if ((rt->dst.dev == dev || !dev) &&
>> + if ((rt->dst.dev == dev || !dev ||
>> + (netdev_unregistering(dev) && rt->rt6i_idev->dev == dev)) &&
>> rt != adn->net->ipv6.ip6_null_entry &&
>> (rt->rt6i_nsiblings == 0 ||
>> (dev && netdev_unregistering(dev)) ||
>
> As you explained earlier, after your patch, all entries in the fib6
> tree will have rt->dst.dev be the same as rt->rt6i_idev->dev except
> those ones created by p6_rt_cache_alloc() and ip6_rt_pcpu_alloc().
> Then the above newly added check is mainly to catch those cached dst
> entries (created by ip6_rt_cached_alloc()). right?
> And it is required because __ipv6_ifa_notify() -> ip6_del_rt() won't
> take care of those cached dst entries.
>
> Then I think I should wait for your patches to get merged before
> submitting my patch?
no. your patch will need to go back to 4.12; my changes will not be
appropriate for that.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Alexander Duyck @ 2017-08-14 0:07 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Jesper Dangaard Brouer, Linux Kernel Network Developers
In-Reply-To: <af5b3482-8f26-52b5-4a00-3a5a42b7449f@itcare.pl>
On Sat, Aug 12, 2017 at 10:27 AM, Paweł Staszewski
<pstaszewski@itcare.pl> wrote:
> Hi and thanks for reply
>
>
>
> W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
>>
>> On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> Hi
>>>
>>> I made some tests for performance comparison.
>>
>> Thanks for doing this. Feel free to Cc me, if you do more of these
>> tests (so I don't miss them on the mailing list).
>>
>> I don't understand stand if you are reporting a potential problem?
>>
>> It would be good if you can provide a short summary section (of the
>> issue) in the _start_ of the email, and then provide all this nice data
>> afterwards, to back your case.
>>
>> My understanding is, you report:
>>
>> 1. VLANs on ixgbe show a 30-40% slowdown
>> 2. System stopped scaling after 7+ CPUs
So I had read through most of this before I realized what it was you
were reporting. As far as the behavior there are a few things going
on. I have some additional comments below but they are mostly based on
what I had read up to that point.
As far as possible issues for item 1. The VLAN adds 4 bytes of data of
the payload, when it is stripped it can result in a packet that is 56
bytes. These missing 8 bytes can cause issues as it forces the CPU to
do a read/modify/write every time the device writes to the 64B cache
line instead of just doing it as a single write. This can be very
expensive and hurt performance. In addition it adds 4 bytes on the
wire, so if you are sending the same 64B packets over the VLAN
interface it is bumping them up to 68B to make room for the VLAN tag.
I am suspecting you are encountering one of these type of issues. You
might try tweaking the packet sizes in increments of 4 to see if there
is a sweet spot that you might be falling out of or into.
Item 2 is a known issue with the NICs supported by ixgbe, at least for
anything 82599 and later. The issue here is that there isn't really an
Rx descriptor cache so to try and optimize performance the hardware
will try to write back as many descriptors it has ready for the ring
requesting writeback. The problem is as you add more rings it means
the writes get smaller as they are triggering more often. So what you
end up seeing is that for each additional ring you add the performance
starts dropping as soon as the rings are no longer being fully
saturated. You can tell this has happened when the CPUs in use
suddenly all stop reporting 100% softirq use. So for example to
perform at line rate with 64B packets you would need something like
XDP and to keep the ring count small, like maybe 2 rings. Any more
than that and the performance will start to drop as you hit PCIe
bottlenecks.
> This is not only problem/bug report - but some kind of comparision plus
> some toughts about possible problems :)
> And can help somebody when searching the net for possible expectations :)
> Also - dono better list where are the smartest people that know what is
> going in kernel with networking :)
>
> Next time i will place summary on top - sorry :)
>
>>
>>> Tested HW (FORWARDING HOST):
>>>
>>> Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>
>> Interesting, I've not heard about a Intel CPU called "Gold" before now,
>> but it does exist:
>>
>> https://ark.intel.com/products/123541/Intel-Xeon-Gold-6132-Processor-19_25M-Cache-2_60-GHz
>>
>>
>>> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>>
>> This is one of my all time favorite NICs!
>
> Yes this is a good NIC - will have connectx-4 2x100G by monday so will also
> do some tests
>
>>
>>>
>>> Test diagram:
>>>
>>>
>>> TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
>>> (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
>>>
>>> Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
>>> 172.16.0.1 to 172.16.0.255
>>>
>>> TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
>>
>> What kind of traffic flow? E.g. distribution, many/few source IPs...
>
>
> Traffic generator is pktgen so udp flows - better paste parameters from
> pktgen:
> UDP_MIN=9
> UDP_MAX=19
>
> pg_set $dev "dst_min 172.16.0.1"
> pg_set $dev "dst_max 172.16.0.100"
>
> # Setup random UDP port src range
> #pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "udp_src_min $UDP_MIN"
> pg_set $dev "udp_src_max $UDP_MAX"
>
>
>>
>>
>>>
>>> Settings used for FORWARDING HOST (changed param. was only number of RSS
>>> combined queues + set affinity assignment for them to fit with first
>>> numa node where 2x10G port card is installed)
>>>
>>> ixgbe driver used from kernel (in-kernel build - not a module)
>>>
>> Nice with a script showing you setup, thanks. I would be good if it had
>> comments, telling why you think this is a needed setup adjustment.
>>
>>> #!/bin/sh
>>> ifc='enp216s0f0 enp216s0f1'
>>> for i in $ifc
>>> do
>>> ip link set up dev $i
>>> ethtool -A $i autoneg off rx off tx off
>>
>> Good:
>> Turning off Ethernet flow control, to avoid receiver being the
>> bottleneck via pause-frames.
>
> Yes - enabled flow controll is really bad :)
>>>
>>> ethtool -G $i rx 4096 tx 1024
>>
>> You adjust the RX and TX ring queue sizes, this have effects that you
>> don't realize. Especially for the ixgbe driver, which have a page
>> recycle trick tied to the RX ring queue size.
>
> rx ring 4096 and tx ring 1024
> - this is because have best performance then with average packet size from
> 64 to 1500 bytes
The problem is this has huge negative effects on the CPU caches.
Generally less is more. When I perform tests I will usually drop the
ring size for Tx to 128 and Rx to 256. That reduces the descriptor
caches per ring to 1 page each for the Tx and Rx. With an increased
interrupt rate you should be able to service this optimally without
too much issue.
Also for these type of tests the Tx ring never really gets over 64
packets anyway since a single Tx ring is always populated by a single
Rx ring so as long as there isn't any flow control in play the Tx
queue should always be empty when the Rx clean-up begins and it will
only be populated with up to NAPI poll weight worth of packets.
> Can be a little better performance for smaller frames like 64 - with rx ring
> set to 1024
> below 1 core/1 RSS queue with rx ring set to 1024
>
> 0;1;64;1530112;91772160;1529919;88724208
> 1;1;64;1531584;91872000;1531520;88813196
> 2;1;64;1531392;91895040;1531262;88831930
> 3;1;64;1530880;91875840;1531201;88783558
> 4;1;64;1530688;91829760;1530688;88768826
> 5;1;64;1530432;91810560;1530624;88764940
> 6;1;64;1530880;91868160;1530878;88787328
> 7;1;64;1530496;91845120;1530560;88765114
> 8;1;64;1530496;91837440;1530687;88772538
> 9;1;64;1530176;91795200;1530496;88735360
>
> so from 1.47Mpps to 1.53Mpps
>
> But with bigger packets > 200 performance is better when rx is set to 4096
This is likely due to the interrupt moderation on the adapter. Instead
of adjusting the ring size up you might try pushing the time between
interrupts down. I have generally found around 25 usecs is best. You
can change the rx-usecs value via ethtool -C to get the rate you want.
You should find that it will perform better that way since you put
less stress on the CPU caches.
>
>>
>>> ip link set $i txqueuelen 1000
>>
>> Setting tx queue len to the default 1000 seems redundant.
>
> Yes cause i'm changing this parameter also to see if any impact on
> performance we have
>>
>>
>>> ethtool -C $i rx-usecs 10
>>
>> Adjusting this also have effects you might not realize. This actually
>> also affect the page recycle scheme of ixgbe. And can sometimes be
>> used to solve stalling on DMA TX completions, which could be you issue
>> here.
>
> same here - rx-usecs - setting to 10 was kind of compromise to have good
> performance with big ans small packet sizes
>From my personal experience I can say that 10 is probably too
aggressive. The logic for trying to find an ideal interrupt rate for
these kind of tests is actually pretty simple. What you want to do is
have the updates coming fast enough that you never hit the point of
descriptor starvation, but at the same time you don't want them coming
too quickly otherwise you limit how many descriptors can be coalesced
into a single PCI DMA write since the descriptors have to be flushed
when an interrupt is triggered.
> Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256
> (1Core/1RSS queue):
> 0;1;64;1506304;90424320;1506626;87402868
> 1;1;64;1505536;90343680;1504830;87321088
> 2;1;64;1506880;90416640;1507522;87388120
> 3;1;64;1511040;90700800;1511682;87684864
> 4;1;64;1511040;90681600;1511102;87662476
> 5;1;64;1511488;90712320;1511614;87673728
> 6;1;64;1511296;90700800;1511038;87669900
> 7;1;64;1513344;90773760;1513280;87751680
> 8;1;64;1513536;90850560;1513470;87807360
> 9;1;64;1512128;90696960;1512000;87696000
>
> And rx-usecs set to 1
> 0;1;64;1533632;92037120;1533504;88954368
> 1;1;64;1533632;92006400;1533570;88943348
> 2;1;64;1533504;91994880;1533504;88931980
> 3;1;64;1532864;91979520;1532674;88902516
> 4;1;64;1533952;92044800;1534080;88961792
> 5;1;64;1533888;92048640;1534270;88969100
> 6;1;64;1533952;92037120;1534082;88969216
> 7;1;64;1533952;92021760;1534208;88969332
> 8;1;64;1533056;91983360;1532930;88883724
> 9;1;64;1533760;92021760;1533886;88946828
>
> rx-useck set to 2
> 0;1;64;1522432;91334400;1522304;88301056
> 1;1;64;1521920;91330560;1522496;88286208
> 2;1;64;1522496;91322880;1522432;88304768
> 3;1;64;1523456;91422720;1523649;88382762
> 4;1;64;1527680;91676160;1527424;88601728
> 5;1;64;1527104;91626240;1526912;88572032
> 6;1;64;1527424;91641600;1527424;88590592
> 7;1;64;1526336;91572480;1526912;88523776
> 8;1;64;1527040;91637760;1526912;88579456
> 9;1;64;1527040;91595520;1526784;88553472
>
> rx-usecs set to 3
> 0;1;64;1526272;91549440;1526592;88527488
> 1;1;64;1526528;91560960;1526272;88516352
> 2;1;64;1525952;91580160;1525888;88527488
> 3;1;64;1525504;91511040;1524864;88456960
> 4;1;64;1526272;91568640;1526208;88494080
> 5;1;64;1525568;91545600;1525312;88494080
> 6;1;64;1526144;91584000;1526080;88512640
> 7;1;64;1525376;91530240;1525376;88482944
> 8;1;64;1526784;91607040;1526592;88549760
> 9;1;64;1526208;91560960;1526528;88512640
>
>
>>
>>> ethtool -L $i combined 16
>>> ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
>>> tx-nocache-copy on ntuple on
>>
>> Here are many setting above.
>
> Yes mostly NIC defaults besides the ntuple that is on (for testing some nfc
> drop filters - and trying to test also tc-offload )
>
>> GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
>> this result in approx 10% slowdown.
>
> Ok lets give a try :)
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple
> on
> rx-usecs 10
> 1 CPU / 1 RSS QUEUE
>
> 0;1;64;1609344;96537600;1609279;93327104
> 1;1;64;1608320;96514560;1608256;93293812
> 2;1;64;1608000;96487680;1608125;93267770
> 3;1;64;1608320;96522240;1608576;93297524
> 4;1;64;1605888;96387840;1606211;93148986
> 5;1;64;1601472;96072960;1601600;92870644
> 6;1;64;1602624;96180480;1602243;92959674
> 7;1;64;1601728;96107520;1602113;92907764
> 8;1;64;1602176;96122880;1602176;92933806
> 9;1;64;1603904;96253440;1603777;93045208
>
> A little better performance 1.6Mpps
> But wondering if disabling tso will have no performance impact for tcp
> traffic ...
If you were passing TCP traffic through the router GRO/TSO would
impact things, but for UDP it just adds overhead.
> Will try to get some pktgen like pktgen-dpdk that can generate also tcp
> traffic - to compare this.
>
>
>>
>> AFAIK "tx-nocache-copy on" was also determined to be a bad option.
>
> I set this to on cause i have better performance (a little 10kpps for this
> test)
> below same test as above with tx-nocache-copy off
>
> 0;1;64;1591552;95496960;1591230;92313654
> 1;1;64;1596224;95738880;1595842;92555066
> 2;1;64;1595456;95700480;1595201;92521774
> 3;1;64;1595456;95723520;1595072;92528966
> 4;1;64;1595136;95692800;1595457;92503040
> 5;1;64;1594624;95631360;1594496;92473402
> 6;1;64;1596224;95761920;1595778;92551180
> 7;1;64;1595200;95700480;1595331;92521542
> 8;1;64;1595584;95692800;1595457;92521426
> 9;1;64;1594624;95662080;1594048;92469574
If I recall it should have no actual impact one way or the other. The
tx-nocache-copy option should only impact socket traffic, not routing
since if I recall correctly it only impacts copies from userspace.
>> The "ntuple on" AFAIK disables the flow-director in the NIC. I though
>> this would actually help VLAN traffic, but I guess not.
>
> yes I enabled this cause was thinking that can help with traffic on vlans
>
> below same test with ntuple off
> so all settings for ixgbe:
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple
> off
> rx-usecs 10
> rx-flow-hash udp4 sdfn
>
> 0;1;64;1611840;96691200;1611905;93460794
> 1;1;64;1610688;96645120;1610818;93427328
> 2;1;64;1610752;96668160;1610497;93442176
> 3;1;64;1610624;96664320;1610817;93427212
> 4;1;64;1610752;96652800;1610623;93412480
> 5;1;64;1610048;96614400;1610112;93404940
> 6;1;64;1611264;96641280;1611390;93427212
> 7;1;64;1611008;96691200;1610942;93468160
> 8;1;64;1610048;96652800;1609984;93408652
> 9;1;64;1611136;96641280;1610690;93434636
>
> Performance is a little better
> and now with tx-nocache-copy on
>
> 0;1;64;1597248;95834880;1597311;92644096
> 1;1;64;1597888;95865600;1597824;92677446
> 2;1;64;1597952;95834880;1597822;92644038
> 3;1;64;1597568;95877120;1597375;92685044
> 4;1;64;1597184;95827200;1597314;92629190
> 5;1;64;1597696;95842560;1597565;92625652
> 6;1;64;1597312;95834880;1597376;92644038
> 7;1;64;1597568;95873280;1597634;92647924
> 8;1;64;1598400;95919360;1598849;92699602
> 9;1;64;1597824;95873280;1598208;92684928
>
>
> That is weird - so enabling tx-nocache-copy with disabled ntuple have bad
> performance impact - but with enabled ntuple there is no performance impact
I would leave the ntuple feature enabled if you are routing simply
because that disables the ixgbe feature ATR which can have a negative
impact on routing tests (causes reordering).
>>
>>
>>> ethtool -N $i rx-flow-hash udp4 sdfn
>>
>> Why do you change the NICs flow-hash?
>
> whan used 16 cores / 16 rss queues - there was better load distribution over
> all cores when sdfn rx-flow-hash enabled
That is to be expected. The default hash will only has on IPv4
addresses. Enabling the use of UDP ports would allow for more entropy.
If you want similar performance without resorting to hashing on ports
you would have to change the source/destination IP addresses.
>>
>>> done
>>>
>>> ip link set up dev enp216s0f0
>>> ip link set up dev enp216s0f1
>>>
>>> ip a a 10.0.0.1/30 dev enp216s0f0
>>>
>>> ip link add link enp216s0f1 name vlan1000 type vlan id 1000
>>> ip link set up dev vlan1000
>>> ip a a 10.0.0.5/30 dev vlan1000
>>>
>>>
>>> ip route add 172.16.0.0/12 via 10.0.0.6
>>>
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
>>> #cat /sys/devices/system/node/node1/cpulist
>>> #14-27,42-55
>>> #cat /sys/devices/system/node/node0/cpulist
>>> #0-13,28-41
>>
>> Is this a NUMA system?
>
> This is 2x CPU 6132 - so have two separate pcie access to the nic - need to
> check what cpu is assigned to pcie where network card is connected to have
> network card on local cpu where all irq's are binded
>
>>
>>
>>>
>>> #################################################
>>>
>>>
>>> Looks like forwarding performance when using vlans on ixgbe is less that
>>> without vlans for about 30-40% (wondering if this is some vlan
>>> offloading problem and ixgbe)
>>
>> I would see this as a problem/bug that enabling VLANs cost this much.
>
> Yes - was thinking that with tx/rx vlan offloading there will be not much
> performance impact when vlans used.
What is the rate difference? Also did you account for the header size
when noticing that there is a difference in rates? I just want to make
sure we aren't seen an issue where you are expecting a rate of
14.88Mpps when VLAN tags drop the rate due to header overhead down to
something like 14.2Mpps if I recall correctly.
>
>>
>>>
>>> settings below:
>>>
>>> ethtool -k enp216s0f0
>>> Features for enp216s0f0:
>>> Cannot get device udp-fragmentation-offload settings: Operation not
>>> supported
>>> rx-checksumming: on
>>> tx-checksumming: on
>>> tx-checksum-ipv4: off [fixed]
>>> tx-checksum-ip-generic: on
>>> tx-checksum-ipv6: off [fixed]
>>> tx-checksum-fcoe-crc: off [fixed]
>>> tx-checksum-sctp: on
>>> scatter-gather: on
>>> tx-scatter-gather: on
>>> tx-scatter-gather-fraglist: off [fixed]
>>> tcp-segmentation-offload: on
>>> tx-tcp-segmentation: on
>>> tx-tcp-ecn-segmentation: off [fixed]
>>> tx-tcp-mangleid-segmentation: on
>>> tx-tcp6-segmentation: on
>>> udp-fragmentation-offload: off
>>> generic-segmentation-offload: off
>>> generic-receive-offload: on
>>> large-receive-offload: off
>>> rx-vlan-offload: on
>>> tx-vlan-offload: on
>>> ntuple-filters: on
>>> receive-hashing: on
>>> highdma: on [fixed]
>>> rx-vlan-filter: on
>>> vlan-challenged: off [fixed]
>>> tx-lockless: off [fixed]
>>> netns-local: off [fixed]
>>> tx-gso-robust: off [fixed]
>>> tx-fcoe-segmentation: off [fixed]
>>> tx-gre-segmentation: on
>>> tx-gre-csum-segmentation: on
>>> tx-ipxip4-segmentation: on
>>> tx-ipxip6-segmentation: on
>>> tx-udp_tnl-segmentation: on
>>> tx-udp_tnl-csum-segmentation: on
>>> tx-gso-partial: on
>>> tx-sctp-segmentation: off [fixed]
>>> tx-esp-segmentation: off [fixed]
>>> fcoe-mtu: off [fixed]
>>> tx-nocache-copy: on
>>> loopback: off [fixed]
>>> rx-fcs: off [fixed]
>>> rx-all: off
>>> tx-vlan-stag-hw-insert: off [fixed]
>>> rx-vlan-stag-hw-parse: off [fixed]
>>> rx-vlan-stag-filter: off [fixed]
>>> l2-fwd-offload: off
>>> hw-tc-offload: off
>>> esp-hw-offload: off [fixed]
>>> esp-tx-csum-hw-offload: off [fixed]
>>> rx-udp_tunnel-port-offload: on
>>>
>>>
>>> Another thing is that forwarding performance does not scale with number
>>> of cores when 7+ cores are reached
>>
>> I've seen problems with using Hyper-Threading CPUs. Could it be that
>> above 7 CPUs you are starting to use sibling-cores ?
>>
I would suspect that it may be more than likely the case. One thing
you might look at doing is CPU pinning the interrupts for the NIC in a
1:1 fashion so that the queues are all bound to separate cores without
them sharing between Hyper-threads.
> Turbostats can help here:
> Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI C1
> C2 C1% C2% CPU%c1 CPU%c6 CoreTmp PkgTmp PkgWatt RAMWatt
> PKG_% RAM_%
> - - - 72 2.27 3188 2600 194844 0 64
> 69282 0.07 97.83 18.38 79.36 -4 54 123.49 16.08 0.00
> 0.00
> 0 0 0 8 0.74 1028 2600 1513 0 32
> 1462 1.50 97.99 10.92 88.34 47 51 58.34 5.34 0.00
> 0.00
> 0 0 28 7 0.67 1015 2600 1255 0 12
> 1249 0.96 98.61 10.99
> 0 1 1 7 0.68 1019 2600 1260 0 0
> 1260 0.00 99.54 8.44 90.88 49
> 0 1 29 9 0.71 1208 2600 1252 0 0
> 1253 0.00 99.48 8.41
> 0 2 2 7 0.67 1019 2600 1261 0 0
> 1260 0.00 99.54 8.44 90.89 48
> 0 2 30 7 0.67 1017 2600 1255 0 0
> 1255 0.00 99.55 8.44
> 0 3 3 7 0.68 1019 2600 1260 0 0
> 1259 0.00 99.53 8.46 90.86 -4
> 0 3 31 7 0.67 1017 2600 1256 0 0
> 1256 0.00 99.55 8.46
> 0 4 4 7 0.67 1027 2600 1260 0 0
> 1260 0.00 99.54 8.43 90.90 -4
> 0 4 32 7 0.66 1018 2600 1255 0 0
> 1255 0.00 99.55 8.44
> 0 5 5 7 0.68 1020 2600 1260 0 0
> 1257 0.00 99.54 8.44 90.89 50
> 0 5 33 7 0.68 1019 2600 1255 0 0
> 1255 0.00 99.55 8.43
> 0 6 6 7 0.70 1019 2600 1260 0 0
> 1259 0.00 99.53 8.43 90.87 -4
> 0 6 34 7 0.70 1019 2600 1255 0 0
> 1255 0.00 99.54 8.43
> 0 8 7 7 0.68 1019 2600 1262 0 0
> 1261 0.00 99.52 8.42 90.90 50
> 0 8 35 7 0.67 1019 2600 1255 0 0
> 1255 0.00 99.55 8.43
> 0 9 8 7 0.68 1019 2600 1260 0 0
> 1257 0.00 99.54 8.40 90.92 49
> 0 9 36 7 0.66 1017 2600 1255 0 0
> 1255 0.00 99.55 8.41
> 0 10 9 7 0.66 1018 2600 1257 0 0
> 1257 0.00 99.54 8.40 90.94 -4
> 0 10 37 7 0.66 1018 2600 1255 0 0
> 1255 0.00 99.55 8.41
> 0 11 10 7 0.66 1019 2600 1257 0 0
> 1259 0.00 99.54 8.56 90.77 -4
> 0 11 38 7 0.66 1018 2600 1255 0 3
> 1252 0.19 99.36 8.57
> 0 12 11 7 0.67 1019 2600 1260 0 0
> 1260 0.00 99.54 8.44 90.88 -4
> 0 12 39 7 0.67 1019 2600 1255 0 0
> 1256 0.00 99.55 8.44
> 0 13 12 7 0.68 1019 2600 1257 0 4
> 1254 0.32 99.22 8.67 90.65 -4
> 0 13 40 7 0.69 1019 2600 1256 0 4
> 1253 0.24 99.31 8.66
> 0 14 13 7 0.71 1020 2600 1260 0 0
> 1259 0.00 99.53 8.41 90.88 -4
> 0 14 41 7 0.72 1020 2600 1255 0 0
> 1255 0.00 99.54 8.40
> 1 0 14 3564 99.19 3594 2600 125472 0 0
> 0 0.00 0.00 0.81 0.00 54 54 65.15 10.74 0.00
> 0.00
> 1 0 42 3 0.07 3701 2600 1255 0 0
> 1255 0.00 99.95 99.93
> 1 1 15 11 0.32 3301 2600 1257 0 0
> 1257 0.00 99.81 26.37 73.31 42
> 1 1 43 10 0.31 3301 2600 1255 0 0
> 1255 0.00 99.82 26.38
> 1 2 16 10 0.31 3301 2600 1257 0 0
> 1257 0.00 99.81 26.37 73.32 39
> 1 2 44 10 0.32 3301 2600 1255 0 0
> 1255 0.00 99.82 26.36
> 1 3 17 10 0.32 3301 2600 1257 0 0
> 1257 0.00 99.81 26.40 73.28 39
> 1 3 45 11 0.32 3301 2600 1255 0 0
> 1255 0.00 99.81 26.40
> 1 4 18 10 0.32 3301 2600 1257 0 0
> 1257 0.00 99.82 26.40 73.28 40
> 1 4 46 11 0.32 3301 2600 1255 0 0
> 1255 0.00 99.82 26.40
> 1 5 19 11 0.33 3301 2600 1257 0 0
> 1257 0.00 99.81 26.40 73.27 39
> 1 5 47 11 0.33 3300 2600 1255 0 0
> 1255 0.00 99.82 26.40
> 1 6 20 12 0.35 3301 2600 1257 0 0
> 1257 0.00 99.81 26.38 73.27 42
> 1 6 48 12 0.36 3301 2600 1255 0 0
> 1255 0.00 99.81 26.37
> 1 8 21 11 0.33 3301 2600 1257 0 0
> 1257 0.00 99.82 26.37 73.29 42
> 1 8 49 11 0.33 3301 2600 1255 0 0
> 1255 0.00 99.82 26.38
> 1 9 22 10 0.32 3300 2600 1257 0 0
> 1257 0.00 99.82 26.35 73.34 41
> 1 9 50 10 0.30 3301 2600 1255 0 0
> 1255 0.00 99.82 26.36
> 1 10 23 10 0.31 3301 2600 1257 0 0
> 1257 0.00 99.82 26.37 73.33 41
> 1 10 51 10 0.31 3301 2600 1255 0 0
> 1255 0.00 99.82 26.36
> 1 11 24 10 0.32 3301 2600 1257 0 0
> 1257 0.00 99.81 26.62 73.06 41
> 1 11 52 10 0.32 3301 2600 1255 0 4
> 1251 0.32 99.50 26.62
> 1 12 25 11 0.33 3301 2600 1257 0 0
> 1257 0.00 99.81 26.39 73.28 41
> 1 12 53 11 0.33 3301 2600 1258 0 0
> 1254 0.00 99.82 26.38
> 1 13 26 12 0.36 3317 2600 1259 0 0
> 1258 0.00 99.79 26.41 73.23 39
> 1 13 54 11 0.34 3301 2600 1255 0 0
> 1254 0.00 99.82 26.42
> 1 14 27 12 0.36 3301 2600 1257 0 5
> 1251 0.24 99.58 26.54 73.10 41
> 1 14 55 12 0.36 3300 2600 1255 0 0
> 1254 0.00 99.82 26.54
>
>
> So it looks like in all tests i'm using core+sibling
> But side effect of this is that :
> 33 * 100.0 = 3300.0 MHz max turbo 28 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 24 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 20 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 14 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 12 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 8 active cores
> 35 * 100.0 = 3500.0 MHz max turbo 4 active cores
> 37 * 100.0 = 3700.0 MHz max turbo 2 active cores
>
> So more cores = less MHz per core/sibling
Yes that is always a trade off. Also the ixgbe is limited in terms of
PCIe bus bandwidth. The more queues you add the worse the descriptor
overhead will be. Generally I have found that about 6 queues is ideal.
As you start getting to more than 8 the performance for 64B packets
will start to drop off as each additional queue will hurt the
descriptor cache performance as it starts writing back fewer and fewer
descriptors per write which will increase the PCIe bus overhead for
the writes.
>>
>>> perf top:
>>>
>>> PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>>
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> 16.32% [kernel] [k] skb_dst_force
>>> 16.30% [kernel] [k] dst_release
>>> 15.11% [kernel] [k] rt_cache_valid
>>> 12.62% [kernel] [k] ipv4_mtu
>>
>> It seems a little strange that these 4 functions are on the top
>
> Yes dono why there is ipv4_mtu called and taking soo much cycles
>
>>
>>> 5.60% [kernel] [k] do_raw_spin_lock
>>
>> Why is calling/taking this lock? (Use perf call-graph recording).
>
> can be hard to paste it here:)
> attached file
>
>>
>>> 3.03% [kernel] [k] fib_table_lookup
>>> 2.70% [kernel] [k] ip_finish_output2
>>> 2.10% [kernel] [k] dev_gro_receive
>>> 1.89% [kernel] [k] eth_type_trans
>>> 1.81% [kernel] [k] ixgbe_poll
>>> 1.15% [kernel] [k] ixgbe_xmit_frame_ring
>>> 1.06% [kernel] [k] __build_skb
>>> 1.04% [kernel] [k] __dev_queue_xmit
>>> 0.97% [kernel] [k] ip_rcv
>>> 0.78% [kernel] [k] netif_skb_features
>>> 0.74% [kernel] [k] ipt_do_table
>>
>> Unloading netfilter modules, will give more performance, but it
>> semifake to do so.
>
> Compiled in kernel - only in filter mode - with ipv4+ipv6 - no other modules
> conntrack or other .
>
>>> 0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
>>> 0.64% [kernel] [k] ip_forward
>>> 0.59% [kernel] [k] __netif_receive_skb_core
>>> 0.55% [kernel] [k] dev_hard_start_xmit
>>> 0.53% [kernel] [k] ip_route_input_rcu
>>> 0.53% [kernel] [k] ip_rcv_finish
>>> 0.51% [kernel] [k] page_frag_free
>>> 0.50% [kernel] [k] kmem_cache_alloc
>>> 0.50% [kernel] [k] udp_v4_early_demux
>>> 0.44% [kernel] [k] skb_release_data
>>> 0.42% [kernel] [k] inet_gro_receive
>>> 0.40% [kernel] [k] sch_direct_xmit
>>> 0.39% [kernel] [k] __local_bh_enable_ip
>>> 0.33% [kernel] [k] netdev_pick_tx
>>> 0.33% [kernel] [k] validate_xmit_skb
>>> 0.28% [kernel] [k] fib_validate_source
>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>> 0.25% [kernel] [k] eth_header
>>> 0.23% [kernel] [k] get_dma_ops
>>> 0.22% [kernel] [k] skb_network_protocol
>>> 0.21% [kernel] [k] ip_output
>>> 0.21% [kernel] [k] vlan_dev_hard_start_xmit
>>> 0.20% [kernel] [k] ixgbe_alloc_rx_buffers
>>> 0.18% [kernel] [k] nf_hook_slow
>>> 0.18% [kernel] [k] apic_timer_interrupt
>>> 0.18% [kernel] [k] virt_to_head_page
>>> 0.18% [kernel] [k] build_skb
>>> 0.16% [kernel] [k] swiotlb_map_page
>>> 0.16% [kernel] [k] ip_finish_output
>>> 0.16% [kernel] [k] udp4_gro_receive
>>>
>>>
>>> RESULTS:
>>>
>>> CSV format - delimeter ";"
>>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;1;64;1470912;88247040;1470720;85305530
>>> 1;1;64;1470912;88285440;1470977;85335110
>>> 2;1;64;1470464;88247040;1470402;85290508
>>> 3;1;64;1471424;88262400;1471230;85353728
>>> 4;1;64;1468736;88166400;1468672;85201652
>>> 5;1;64;1470016;88181760;1469949;85234944
>>> 6;1;64;1470720;88247040;1470466;85290624
>>> 7;1;64;1471232;88277760;1471167;85346246
>>> 8;1;64;1469184;88170240;1469249;85216326
>>> 9;1;64;1470592;88227840;1470847;85294394
>>
>> Single core 1.47Mpps seems a little low, I would expect 2Mpps.
>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;2;64;2413120;144802560;2413245;139975924
>>> 1;2;64;2415296;144913920;2415356;140098188
>>> 2;2;64;2416768;144898560;2416573;140105670
>>> 3;2;64;2418176;145056000;2418110;140261806
>>> 4;2;64;2416512;144990720;2416509;140172950
>>> 5;2;64;2415168;144860160;2414466;140064780
>>> 6;2;64;2416960;144983040;2416833;140190930
>>> 7;2;64;2413632;144768000;2413568;140001734
>>> 8;2;64;2415296;144898560;2414589;140087168
>>> 9;2;64;2416576;144963840;2416892;140190930
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;3;64;3419008;205155840;3418882;198239244
>>> 1;3;64;3428032;205585920;3427971;198744234
>>> 2;3;64;3425472;205536000;3425344;198677260
>>> 3;3;64;3425088;205470720;3425156;198603136
>>> 4;3;64;3427648;205693440;3426883;198773888
>>> 5;3;64;3426880;205670400;3427392;198796044
>>> 6;3;64;3429120;205678080;3430140;198848186
>>> 7;3;64;3422976;205355520;3423490;198458136
>>> 8;3;64;3423168;205336320;3423486;198495372
>>> 9;3;64;3424384;205493760;3425538;198617868
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;4;64;4406464;264364800;4405244;255560296
>>> 1;4;64;4404672;264349440;4405122;255541504
>>> 2;4;64;4402368;264049920;4403326;255188864
>>> 3;4;64;4401344;264076800;4400702;255207134
>>> 4;4;64;4385536;263074560;4386620;254312716
>>> 5;4;64;4386560;263189760;4385404;254379532
>>> 6;4;64;4398784;263857920;4399031;255025288
>>> 7;4;64;4407232;264445440;4407998;255637900
>>> 8;4;64;4413184;264698880;4413758;255875816
>>> 9;4;64;4411328;264526080;4411906;255712372
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;5;64;5094464;305871360;5094464;295657262
>>> 1;5;64;5090816;305514240;5091201;295274810
>>> 2;5;64;5088384;305387520;5089792;295175108
>>> 3;5;64;5079296;304869120;5079484;294680368
>>> 4;5;64;5092992;305544960;5094207;295349166
>>> 5;5;64;5092416;305502720;5093372;295334260
>>> 6;5;64;5080896;304896000;5081090;294677004
>>> 7;5;64;5085376;305114880;5086401;294933058
>>> 8;5;64;5092544;305575680;5092036;295356938
>>> 9;5;64;5093056;305652480;5093832;295449506
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;6;64;5705088;342351360;5705784;330965110
>>> 1;6;64;5710272;342743040;5707591;331373952
>>> 2;6;64;5703424;342182400;5701826;330776552
>>> 3;6;64;5708736;342604800;5707963;331147462
>>> 4;6;64;5710144;342654720;5712067;331202910
>>> 5;6;64;5712064;342777600;5711361;331292288
>>> 6;6;64;5710144;342585600;5708607;331144272
>>> 7;6;64;5699840;342021120;5697853;330609222
>>> 8;6;64;5701184;342124800;5702909;330653592
>>> 9;6;64;5711360;342735360;5713283;331247686
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;7;64;6244416;374603520;6243591;362180072
>>> 1;7;64;6230912;374016000;6231490;361534126
>>> 2;7;64;6244800;374776320;6244866;362224326
>>> 3;7;64;6238720;374376960;6238261;361838510
>>> 4;7;64;6218816;373079040;6220413;360683962
>>> 5;7;64;6224320;373566720;6225086;361017404
>>> 6;7;64;6224000;373570560;6221370;360936088
>>> 7;7;64;6210048;372741120;6210627;360212654
>>> 8;7;64;6231616;374035200;6231537;361445502
>>> 9;7;64;6227840;373724160;6228802;361162752
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;8;64;6251840;375144960;6251849;362609678
>>> 1;8;64;6250816;375014400;6250881;362547038
>>> 2;8;64;6257728;375432960;6257160;362911104
>>> 3;8;64;6255552;375325440;6255622;362822074
>>> 4;8;64;6243776;374576640;6243270;362120622
>>> 5;8;64;6237184;374296320;6237690;361790080
>>> 6;8;64;6240960;374415360;6240714;361927366
>>> 7;8;64;6222784;373317120;6223746;360854424
>>> 8;8;64;6225920;373593600;6227014;361154980
>>> 9;8;64;6238528;374304000;6237701;361845238
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;14;64;6486144;389184000;6486135;376236488
>>> 1;14;64;6454912;387390720;6454222;374466734
>>> 2;14;64;6441152;386480640;6440431;373572780
>>> 3;14;64;6450240;386972160;6450870;374070014
>>> 4;14;64;6465600;387997440;6467221;375089654
>>> 5;14;64;6448384;386860800;6448000;373980230
>>> 6;14;64;6452352;387095040;6452148;374168904
>>> 7;14;64;6441984;386507520;6443203;373665058
>>> 8;14;64;6456704;387340800;6455744;374429092
>>> 9;14;64;6464640;387901440;6465218;374949004
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;16;64;6939008;416325120;6938696;402411192
>>> 1;16;64;6941952;416444160;6941745;402558918
>>> 2;16;64;6960576;417584640;6960707;403698718
>>> 3;16;64;6940736;416486400;6941820;402503876
>>> 4;16;64;6927680;415741440;6927420;401853870
>>> 5;16;64;6929792;415687680;6929917;401839196
>>> 6;16;64;6950400;416989440;6950661;403026166
>>> 7;16;64;6953664;417216000;6953454;403260544
>>> 8;16;64;6948480;416851200;6948800;403023266
>>> 9;16;64;6924160;415422720;6924092;401542468
>>
>> I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
>> issue/bug. You could be stalling on DMA TX completion being too slow,
>> but you already increased the interval and increased the TX ring queue
>> size. You could play with those setting and see if it changes this?
>>
>> Could you try my napi_monitor tool in:
>>
>> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
>>
>> Also provide the output from:
>> mpstat -P ALL -u -I SCPU -I SUM 2
>
> with 16 cores / 16 RSS queues
> Average: CPU %usr %nice %sys %iowait %irq %soft %steal
> %guest %gnice %idle
> Average: all 0.00 0.00 0.01 0.00 0.00 28.57 0.00
> 0.00 0.00 71.42
> Average: 0 0.00 0.00 0.04 0.00 0.00 0.08 0.00
> 0.00 0.00 99.88
> Average: 1 0.00 0.00 0.12 0.00 0.00 0.00 0.00
> 0.00 0.00 99.88
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 11 0.08 0.00 0.04 0.00 0.00 0.00 0.00
> 0.00 0.00 99.88
> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 14 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 15 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 16 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 17 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 18 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 19 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 20 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 21 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 22 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 23 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 24 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 25 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 26 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 27 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 28 0.00 0.00 0.04 0.00 0.00 0.00 0.00
> 0.00 0.00 99.96
> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 39 0.04 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 99.96
> Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 42 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 43 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 44 0.00 0.00 0.04 0.17 0.00 0.00 0.00
> 0.00 0.00 99.79
> Average: 45 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 46 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 47 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 48 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 49 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 51 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 52 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 54 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
>
> Average: CPU intr/s
> Average: all 123596.08
> Average: 0 646.38
> Average: 1 500.54
> Average: 2 511.67
> Average: 3 534.25
> Average: 4 542.21
> Average: 5 531.54
> Average: 6 554.58
> Average: 7 535.88
> Average: 8 544.58
> Average: 9 536.42
> Average: 10 575.46
> Average: 11 601.12
> Average: 12 502.08
> Average: 13 575.46
> Average: 14 5917.92
> Average: 15 5949.58
> Average: 16 7021.29
> Average: 17 7299.71
> Average: 18 7391.67
> Average: 19 7354.25
> Average: 20 7543.42
> Average: 21 7354.25
> Average: 22 7322.33
> Average: 23 7368.71
> Average: 24 7429.00
> Average: 25 7406.46
> Average: 26 7400.67
> Average: 27 7447.21
> Average: 28 517.00
> Average: 29 549.54
> Average: 30 529.33
> Average: 31 533.83
> Average: 32 541.25
> Average: 33 541.17
> Average: 34 532.50
> Average: 35 545.17
> Average: 36 528.96
> Average: 37 509.92
> Average: 38 520.12
> Average: 39 523.29
> Average: 40 530.75
> Average: 41 542.33
> Average: 42 5921.71
> Average: 43 5949.42
> Average: 44 503.04
> Average: 45 542.75
> Average: 46 582.50
> Average: 47 581.71
> Average: 48 495.29
> Average: 49 524.38
> Average: 50 527.92
> Average: 51 528.12
> Average: 52 456.38
> Average: 53 477.00
> Average: 54 440.92
> Average: 55 568.83
>
> Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
> IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
> Average: 0 0.00 250.00 0.17 87.00 0.00 0.00
> 45.46 250.00 0.00 13.75
> Average: 1 0.00 233.42 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 17.21
> Average: 2 0.00 249.04 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 12.67
> Average: 3 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 34.42
> Average: 4 0.00 248.67 0.17 0.00 0.00 0.00
> 0.00 249.96 0.00 43.42
> Average: 5 0.00 249.46 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 32.17
> Average: 6 0.00 249.79 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 54.92
> Average: 7 0.00 240.12 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 45.79
> Average: 8 0.00 247.42 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 47.25
> Average: 9 0.00 249.29 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 37.17
> Average: 10 0.00 248.75 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 76.79
> Average: 11 0.00 249.29 0.00 0.00 0.00 0.00
> 42.79 249.83 0.00 59.21
> Average: 12 0.00 249.83 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 2.29
> Average: 13 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 75.62
> Average: 14 0.00 148.21 0.17 5758.04 0.00 0.00
> 0.00 8.42 0.00 3.08
> Average: 15 0.00 148.42 0.46 5789.25 0.00 0.00
> 0.00 8.33 0.00 3.12
> Average: 16 0.00 142.62 0.79 6866.46 0.00 0.00
> 0.00 8.29 0.00 3.12
> Average: 17 0.00 143.17 0.42 7145.00 0.00 0.00
> 0.00 8.08 0.00 3.04
> Average: 18 0.00 153.62 0.42 7226.42 0.00 0.00
> 0.00 8.04 0.00 3.17
> Average: 19 0.00 150.46 0.46 7192.21 0.00 0.00
> 0.00 8.04 0.00 3.08
> Average: 20 0.00 145.21 0.17 7386.50 0.00 0.00
> 0.00 8.29 0.00 3.25
> Average: 21 0.00 150.96 0.46 7191.37 0.00 0.00
> 0.00 8.25 0.00 3.21
> Average: 22 0.00 146.67 0.54 7163.96 0.00 0.00
> 0.00 8.04 0.00 3.12
> Average: 23 0.00 151.38 0.42 7205.75 0.00 0.00
> 0.00 8.00 0.00 3.17
> Average: 24 0.00 153.33 0.17 7264.12 0.00 0.00
> 0.00 8.08 0.00 3.29
> Average: 25 0.00 153.21 0.17 7241.83 0.00 0.00
> 0.00 7.96 0.00 3.29
> Average: 26 0.00 153.96 0.17 7234.88 0.00 0.00
> 0.00 8.38 0.00 3.29
> Average: 27 0.00 151.71 0.79 7283.25 0.00 0.00
> 0.00 8.04 0.00 3.42
> Average: 28 0.00 245.71 0.00 0.00 0.00 0.00
> 0.00 249.50 0.00 21.79
> Average: 29 0.00 233.21 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 66.46
> Average: 30 0.00 248.92 0.00 0.00 0.00 0.00
> 0.00 250.00 0.00 30.42
> Average: 31 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 33.96
> Average: 32 0.00 248.67 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 42.62
> Average: 33 0.00 249.46 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 41.79
> Average: 34 0.00 249.79 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 32.83
> Average: 35 0.00 240.12 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 55.08
> Average: 36 0.00 247.42 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 31.58
> Average: 37 0.00 249.29 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 10.71
> Average: 38 0.00 248.75 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 21.50
> Average: 39 0.00 249.50 0.00 0.00 0.00 0.00
> 0.00 249.83 0.00 23.96
> Average: 40 0.00 249.83 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 30.96
> Average: 41 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 42.50
> Average: 42 0.00 148.38 0.71 5761.00 0.00 0.00
> 0.00 8.25 0.00 3.38
> Average: 43 0.00 147.21 0.50 5790.33 0.00 0.00
> 0.00 8.00 0.00 3.38
> Average: 44 0.00 248.96 0.00 0.00 0.00 0.00
> 0.00 248.13 0.00 5.96
> Average: 45 0.00 249.04 0.00 0.00 0.00 0.00
> 0.00 248.88 0.00 44.83
> Average: 46 0.00 248.96 0.00 0.00 0.00 0.00
> 0.00 248.58 0.00 84.96
> Average: 47 0.00 249.00 0.00 0.00 0.00 0.00
> 0.00 248.75 0.00 83.96
> Average: 48 0.00 249.12 0.00 0.00 0.00 0.00
> 0.00 132.83 0.00 113.33
> Average: 49 0.00 249.12 0.00 0.00 0.00 0.00
> 0.00 248.62 0.00 26.62
> Average: 50 0.00 248.92 0.00 0.00 0.00 0.00
> 0.00 248.58 0.00 30.42
> Average: 51 0.00 249.08 0.00 0.00 0.00 0.00
> 0.00 248.42 0.00 30.63
> Average: 52 0.00 249.21 0.00 0.00 0.00 0.00
> 0.00 131.96 0.00 75.21
> Average: 53 0.00 249.08 0.00 0.00 0.00 0.00
> 0.00 136.12 0.00 91.79
> Average: 54 0.00 249.00 0.00 0.00 0.00 0.00
> 0.00 136.79 0.00 55.12
> Average: 55 0.00 249.04 0.00 0.00 0.00 0.00
> 0.00 248.71 0.00 71.08
>
>
>>
>
>
^ permalink raw reply
* Re: [patch v1 1/2] Allow Mellanox network vendor to be configured if only I2C bus is configured
From: David Miller @ 2017-08-14 0:25 UTC (permalink / raw)
To: ohado; +Cc: leon, netdev, jiri, saeedm, vadimp, system-sw-low-level
In-Reply-To: <VI1PR05MB1262B06790B2738CE1C0F889C88F0@VI1PR05MB1262.eurprd05.prod.outlook.com>
From: Ohad Oz <ohado@mellanox.com>
Date: Sun, 13 Aug 2017 15:26:56 +0000
>
>
>> -----Original Message-----
>> From: Leon Romanovsky [mailto:leon@kernel.org]
>> Sent: Saturday, August 12, 2017 5:37 PM
>> To: Ohad Oz <ohado@mellanox.com>
>> Cc: davem@davemloft.net; netdev@vger.kernel.org; jiri@resnulli.us; Saeed
>> Mahameed <saeedm@mellanox.com>; Vadim Pasternak
>> <vadimp@mellanox.com>; system-sw-low-level <system-sw-low-
>> level@mellanox.com>
>> Subject: Re: [patch v1 1/2] Allow Mellanox network vendor to be configured
>> if only I2C bus is configured
>>
>> On Thu, Aug 10, 2017 at 05:11:51PM +0000, Ohad Oz wrote:
>> > Patch allows Mellanox devices on system with no PCI, but with I2C only.
>> >
>>
>> Did you test mlx5 device on such system? Did it work for you?
>
> Yes, I did. With PCI config set to disable mlx5 drivers are not built.
> Only the following:
> /build/drivers/net/Ethernet/mellanox/mlxsw/mlxsw_core.ko
> /build/drivers/net/Ethernet/mellanox /mlxsw/mlxsw_i2c.ko
> /build/drivers/net/Ethernet/mellanox /mlxsw/mlxsw_minimal.ko
>
> While with both options on all drivers are built inc mlx5.
I'm not so sure he's interested if things build or not.
He's asking you if you actually used a Mellanox device with this
driver with PCI disabled and only I2C available.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox