* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: David Ahern @ 2018-11-10 14:56 UTC (permalink / raw)
To: Paweł Staszewski, Jesper Dangaard Brouer; +Cc: netdev, Yoel Caspersen
In-Reply-To: <602b9904-b374-c9a7-abf6-2444d02d1bc0@itcare.pl>
On 11/10/18 6:18 AM, Paweł Staszewski wrote:
>
> ./xdp_fwd enp175s0f0 enp175s0f1
> libbpf: failed to create map (name: 'stats_map'): Operation not permitted
> libbpf: failed to load object './xdp_fwd_kern.o'
Forgot I had increased locked memory:
ulimit -l unlimited
./xdp_fwd enp175s0f0 enp175s0f1
^ permalink raw reply
* Re: [PATCH mlx5-next 08/10] IB/mlx5: Call PAGE_FAULT_RESUME command asynchronously
From: Leon Romanovsky @ 2018-11-10 15:47 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, RDMA mailing list, Artemy Kovalyov, Majd Dibbiny,
Moni Shoua, Saeed Mahameed, linux-netdev
In-Reply-To: <20181109165953.GC22987@ziepe.ca>
[-- Attachment #1: Type: text/plain, Size: 3602 bytes --]
On Fri, Nov 09, 2018 at 09:59:53AM -0700, Jason Gunthorpe wrote:
> On Fri, Nov 09, 2018 at 06:26:22PM +0200, Leon Romanovsky wrote:
> > On Thu, Nov 08, 2018 at 07:49:03PM +0000, Jason Gunthorpe wrote:
> > > On Thu, Nov 08, 2018 at 09:10:15PM +0200, Leon Romanovsky wrote:
> > > > From: Moni Shoua <monis@mellanox.com>
> > > >
> > > > Telling the HCA that page fault handling is done and QP can resume
> > > > its flow is done in the context of the page fault handler. This blocks
> > > > the handling of the next work in queue without a need.
> > > > Call the PAGE_FAULT_RESUME command in an asynchronous manner and free
> > > > the workqueue to pick the next work item for handling. All tasks that
> > > > were executed after PAGE_FAULT_RESUME need to be done now
> > > > in the callback of the asynchronous command mechanism.
> > > >
> > > > Signed-off-by: Moni Shoua <monis@mellanox.com>
> > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
> > > > drivers/infiniband/hw/mlx5/odp.c | 110 +++++++++++++++++++++++++------
> > > > include/linux/mlx5/driver.h | 3 +
> > > > 2 files changed, 94 insertions(+), 19 deletions(-)
> > > >
> > > > diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
> > > > index abce55b8b9ba..0c4f469cdd5b 100644
> > > > +++ b/drivers/infiniband/hw/mlx5/odp.c
> > > > @@ -298,20 +298,78 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)
> > > > return;
> > > > }
> > > >
> > > > +struct pfault_resume_cb_ctx {
> > > > + struct mlx5_ib_dev *dev;
> > > > + struct mlx5_core_rsc_common *res;
> > > > + struct mlx5_pagefault *pfault;
> > > > +};
> > > > +
> > > > +static void page_fault_resume_callback(int status, void *context)
> > > > +{
> > > > + struct pfault_resume_cb_ctx *ctx = context;
> > > > + struct mlx5_pagefault *pfault = ctx->pfault;
> > > > +
> > > > + if (status)
> > > > + mlx5_ib_err(ctx->dev, "Resolve the page fault failed with status %d\n",
> > > > + status);
> > > > +
> > > > + if (ctx->res)
> > > > + mlx5_core_res_put(ctx->res);
> > > > + kfree(pfault);
> > > > + kfree(ctx);
> > > > +}
> > > > +
> > > > static void mlx5_ib_page_fault_resume(struct mlx5_ib_dev *dev,
> > > > + struct mlx5_core_rsc_common *res,
> > > > struct mlx5_pagefault *pfault,
> > > > - int error)
> > > > + int error,
> > > > + bool async)
> > > > {
> > > > + int ret = 0;
> > > > + u32 *out = pfault->out_pf_resume;
> > > > + u32 *in = pfault->in_pf_resume;
> > > > + u32 token = pfault->token;
> > > > int wq_num = pfault->event_subtype == MLX5_PFAULT_SUBTYPE_WQE ?
> > > > - pfault->wqe.wq_num : pfault->token;
> > > > - int ret = mlx5_core_page_fault_resume(dev->mdev,
> > > > - pfault->token,
> > > > - wq_num,
> > > > - pfault->type,
> > > > - error);
> > > > - if (ret)
> > > > - mlx5_ib_err(dev, "Failed to resolve the page fault on WQ 0x%x\n",
> > > > - wq_num);
> > > > + pfault->wqe.wq_num : pfault->token;
> > > > + u8 type = pfault->type;
> > > > + struct pfault_resume_cb_ctx *ctx = NULL;
> > > > +
> > > > + if (async)
> > > > + ctx = kmalloc(sizeof(*ctx), GFP_KERNEL);
> > >
> > > Why not allocate this ctx ast part of the mlx5_pagefault and avoid
> > > this allocation failure strategy?
> >
> > It is another way to implement it, both of them are correct.
>
> .. I think it is alot better to move this allocation, it gets rid of
> this ugly duplicated code
>
> > Can I assume that we can progress with patches except patch #2?
>
> Lets drop this one too..
Sure, thanks
>
> Jason
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]
^ permalink raw reply
* Re: [RFC PATCH 01/12] dt-bindings: soc: qcom: add IPA bindings
From: Rob Herring @ 2018-11-11 1:40 UTC (permalink / raw)
To: Alex Elder
Cc: Mark Rutland, David Miller, Arnd Bergmann, Bjorn Andersson,
ilias.apalodimas, netdev, devicetree, linux-arm-msm,
open list:ARM/QUALCOMM SUPPORT,
moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE,
linux-kernel@vger.kernel.org, syadagir, mjavid
In-Reply-To: <a4c9f0f8-d7c7-2756-cfdf-b67ce9f1ca3b@linaro.org>
On Fri, Nov 9, 2018 at 4:38 PM Alex Elder <elder@linaro.org> wrote:
>
> On 11/7/18 8:59 AM, Rob Herring wrote:
> > On Tue, Nov 6, 2018 at 6:33 PM Alex Elder <elder@linaro.org> wrote:
> >>
> >> Add the binding definitions for the "qcom,ipa" and "qcom,rmnet-ipa"
> >> device tree nodes.
> >>
> >> Signed-off-by: Alex Elder <elder@linaro.org>
> >> ---
> >> .../devicetree/bindings/soc/qcom/qcom,ipa.txt | 136 ++++++++++++++++++
> >> .../bindings/soc/qcom/qcom,rmnet-ipa.txt | 15 ++
> >> 2 files changed, 151 insertions(+)
> >> create mode 100644 Documentation/devicetree/bindings/soc/qcom/qcom,ipa.txt
> >> create mode 100644 Documentation/devicetree/bindings/soc/qcom/qcom,rmnet-ipa.txt
> >>
> >> diff --git a/Documentation/devicetree/bindings/soc/qcom/qcom,ipa.txt b/Documentation/devicetree/bindings/soc/qcom/qcom,ipa.txt
> >> new file mode 100644
> >> index 000000000000..d4d3d37df029
> >> --- /dev/null
> >> +++ b/Documentation/devicetree/bindings/soc/qcom/qcom,ipa.txt
> >> @@ -0,0 +1,136 @@
> >> +Qualcomm IPA (IP Accelerator) Driver
> >
> > Bindings are for h/w not drivers.
>
> OK. I'll drop " Driver".
>
> >> +
> >> +This binding describes the Qualcomm IPA. The IPA is capable of offloading
> >> +certain network processing tasks (e.g. filtering, routing, and NAT) from
> >> +the main processor. The IPA currently serves only as a network interface,
> >> +providing access to an LTE network available via a modem.
> >> +
> >> +The IPA sits between multiple independent "execution environments,"
> >> +including the AP subsystem (APSS) and the modem. The IPA presents
> >> +a Generic Software Interface (GSI) to each execution environment.
> >> +The GSI is an integral part of the IPA, but it is logically isolated
> >> +and has a distinct interrupt and a separately-defined address space.
> >> +
> >> + ---------- ------------- ---------
> >> + | | |G| |G| | |
> >> + | APSS |===|S| IPA |S|===| Modem |
> >> + | | |I| |I| | |
> >> + ---------- ------------- ---------
> >> +
> >> +See also:
> >> + bindings/interrupt-controller/interrupts.txt
> >> + bindings/interconnect/interconnect.txt
> >> + bindings/soc/qcom/qcom,smp2p.txt
> >> + bindings/reserved-memory/reserved-memory.txt
> >> + bindings/clock/clock-bindings.txt
> >> +
> >> +All properties defined below are required.
> >> +
> >> +- compatible:
> >> + Must be one of the following compatible strings:
> >> + "qcom,ipa-sdm845-modem_init"
> >> + "qcom,ipa-sdm845-tz_init"
> >
> > Normal order is <vendor>,<soc>-<ipblock>."
>
> I'll use "qcom,sdm845-ipa-modem-init" and "qcom,sdm845-ipa-tz-init".
> (Or just "qcom,sdm845-ipa", depending on the outcome of the discussion
> below.)
>
> > Don't use '_'.
>
> OK.
>
> > What's the difference between these 2? It can't be detected somehow?
>
> There is some early initialization, including loading some firmware,
> that must be done by trusted code. That can be done by either Trust
> Zone or the modem. If it's done by the modem, there is an additional
> step required during initialization so the modem can tell the AP
> that it has done its part, and the AP can finish IPA initialization.
>
> There is no way of detecting (e.g. by probing hardware) which is
> in effect so we use DT. I discussed this with Bjorn, who said that
> this was a situation seen elsewhere and that using compatible strings
> was the way he suggested to address it.
Okay. However, if this is common for QCom blocks maybe we should
reconsider. I think compatible makes sense if the programming model
changes.
> > This might be better expressed as a property. Then if Trustzone
> > initializes things, it can just add a property.
>
> A Boolean property to distinguish them would be fine as well, but
> I would like to address this "common" problem consistently.
>
> Bjorn, would you please weigh in?
>
> >> +
> >> +-reg:
> >> + Resources specyfing the physical address spaces of the IPA and GSI.
> >
> > typo
> >
> >> +
> >> +-reg-names:
> >> + The names of the address space ranges defined by the "reg" property.
> >> + Must be "ipa" and "gsi".
> >> +
> >> +- interrupts-extended:
> >
> > Use 'interrupts' here and describe what they are and the order. What
> > they are connected to (and the need for interrupts-extended) is
> > outside the scope of this binding.
>
> I used interrupts-extended because there were two interrupt parents
> (a "normal" interrupt controller and the interrupt controller implemented
> for SMP2P input). A paragraph here:
> bindings/interrupt-controller/interrupts.txt
> recommends "interrupts-extended" in that case.
>
> I have no objection to using just "interrupts" but can you tell me what
> I misunderstood? It seems like I need to do "interrupts-extended".
Yes, in the dts you should use interrupts-extended. However, for
documentation purposes that aspect is not important. So we just use
interrupts most everywhere.
Rob
^ permalink raw reply
* KASAN: slab-out-of-bounds Write in __xfrm_policy_unlink
From: syzbot @ 2018-11-11 1:54 UTC (permalink / raw)
To: davem, herbert, linux-kernel, netdev, steffen.klassert,
syzkaller-bugs
Hello,
syzbot found the following crash on:
HEAD commit: 29e12207174a sfc: use the new __netdev_tx_sent_queue BQL o..
git tree: net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=133c2c5d400000
kernel config: https://syzkaller.appspot.com/x/.config?x=8f559fee2fc3375a
dashboard link: https://syzkaller.appspot.com/bug?extid=1a5442803e5354a25766
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+1a5442803e5354a25766@syzkaller.appspotmail.com
==================================================================
BUG: KASAN: slab-out-of-bounds in __write_once_size
include/linux/compiler.h:209 [inline]
BUG: KASAN: slab-out-of-bounds in __hlist_del include/linux/list.h:702
[inline]
BUG: KASAN: slab-out-of-bounds in hlist_del_rcu include/linux/rculist.h:455
[inline]
BUG: KASAN: slab-out-of-bounds in __xfrm_policy_unlink+0x75f/0x810
net/xfrm/xfrm_policy.c:1241
Write of size 8 at addr ffff8801b979db48 by task blkid/15614
CPU: 1 PID: 15614 Comm: blkid Not tainted 4.20.0-rc1+ #287
kernel msg: ebtables bug: please report to author: Nentries wrong
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x244/0x39d lib/dump_stack.c:113
print_address_description.cold.7+0x9/0x1ff mm/kasan/report.c:256
kasan_report_error mm/kasan/report.c:354 [inline]
kasan_report.cold.8+0x242/0x309 mm/kasan/report.c:412
__asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:438
__write_once_size include/linux/compiler.h:209 [inline]
__hlist_del include/linux/list.h:702 [inline]
hlist_del_rcu include/linux/rculist.h:455 [inline]
__xfrm_policy_unlink+0x75f/0x810 net/xfrm/xfrm_policy.c:1241
xfrm_policy_delete+0x49/0x90 net/xfrm/xfrm_policy.c:1266
xfrm_policy_timer+0x46f/0x660 net/xfrm/xfrm_policy.c:254
call_timer_fn+0x272/0x920 kernel/time/timer.c:1326
expire_timers kernel/time/timer.c:1363 [inline]
__run_timers+0x7e5/0xc70 kernel/time/timer.c:1682
run_timer_softirq+0x52/0xb0 kernel/time/timer.c:1695
__do_softirq+0x308/0xb7e kernel/softirq.c:292
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x17f/0x1c0 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1061
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:804
</IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:761
[inline]
RIP: 0010:count_memcg_events include/linux/memcontrol.h:742 [inline]
RIP: 0010:count_memcg_event_mm include/linux/memcontrol.h:763 [inline]
RIP: 0010:handle_mm_fault+0x9a8/0xc70 mm/memory.c:3906
Code: df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 4e 02 00 00 48 83 3d 78 5b
9e 07 00 0f 84 70 01 00 00 e8 7d ea cb ff 4c 89 ff 57 9d <0f> 1f 44 00 00
e9 22 fa ff ff e8 69 ea cb ff 49 8d bd 20 04 00 00
RSP: 0000:ffff8801bbcffcc8 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: ffff8801bc1ae540 RBX: ffff8801c6f5dca0 RCX: 1ffff10037835dbc
RDX: 0000000000000000 RSI: ffffffff81b3a043 RDI: 0000000000000293
RBP: ffff8801bbcffd68 R08: ffff8801bc1aede0 R09: 0000000000000007
R10: 0000000000000000 R11: ffff8801bc1ae540 R12: 1ffff1003779ff9c
R13: 0000000000000200 R14: 0000000000000054 R15: 0000000000000293
do_user_addr_fault arch/x86/mm/fault.c:1423 [inline]
__do_page_fault+0x5e8/0xe60 arch/x86/mm/fault.c:1489
do_page_fault+0xf2/0x7e0 arch/x86/mm/fault.c:1520
page_fault+0x1e/0x30 arch/x86/entry/entry_64.S:1139
RIP: 0033:0x7f05c5e50d61
Code: 00 c3 f7 05 9d c4 30 00 04 00 00 00 74 07 48 8d 05 84 95 0b 00 c3 0f
1f 00 48 31 c0 89 f9 83 e1 3f 66 0f ef c0 83 f9 30 77 19 <f3> 0f 6f 0f 66
0f 74 c1 66 0f d7 d0 85 d2 75 7a 48 89 f8 48 83 e0
RSP: 002b:00007ffc5d3e2558 EFLAGS: 00010287
RAX: 0000000000000000 RBX: 00007f05c638487c RCX: 0000000000000020
RDX: 0000000000000005 RSI: 00007ffc5d3e2d48 RDI: 00007f05c6179a20
RBP: 00007ffc5d3e2f2b R08: 0000000000000008 R09: 0101010101010101
R10: 0000000000000000 R11: 00007f05c5e07060 R12: 0000000000403738
R13: 0000000000000001 R14: 0000000000000000 R15: 00007f05c6179a20
Allocated by task 7136:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
kasan_kmalloc+0xc7/0xe0 mm/kasan/kasan.c:553
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490
kmem_cache_alloc+0x12e/0x730 mm/slab.c:3554
mempool_alloc_slab+0x44/0x60 mm/mempool.c:505
mempool_alloc+0x193/0x4a0 mm/mempool.c:385
bvec_alloc+0x12a/0x2d0 block/bio.c:218
bio_alloc_bioset+0x47a/0x700 block/bio.c:509
bio_alloc include/linux/bio.h:393 [inline]
io_submit_init_bio fs/ext4/page-io.c:374 [inline]
io_submit_add_bh fs/ext4/page-io.c:399 [inline]
ext4_bio_write_page+0x1304/0x1bd1 fs/ext4/page-io.c:506
mpage_submit_page+0x15e/0x270 fs/ext4/inode.c:2237
mpage_process_page_bufs+0x50a/0x600 fs/ext4/inode.c:2348
mpage_prepare_extent_to_map+0xea5/0x19c0 fs/ext4/inode.c:2710
ext4_writepages+0x140c/0x41a0 fs/ext4/inode.c:2837
do_writepages+0x9a/0x1a0 mm/page-writeback.c:2328
__writeback_single_inode+0x20a/0x1660 fs/fs-writeback.c:1316
writeback_sb_inodes+0x71f/0x1210 fs/fs-writeback.c:1580
__writeback_inodes_wb+0x1b9/0x340 fs/fs-writeback.c:1649
wb_writeback+0xa73/0xfc0 fs/fs-writeback.c:1758
wb_check_start_all fs/fs-writeback.c:1882 [inline]
wb_do_writeback fs/fs-writeback.c:1908 [inline]
wb_workfn+0xee9/0x1790 fs/fs-writeback.c:1942
process_one_work+0xc90/0x1c40 kernel/workqueue.c:2153
worker_thread+0x17f/0x1390 kernel/workqueue.c:2296
kthread+0x35a/0x440 kernel/kthread.c:246
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352
Freed by task 5729:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
__kasan_slab_free+0x102/0x150 mm/kasan/kasan.c:521
kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
__cache_free mm/slab.c:3498 [inline]
kmem_cache_free+0x83/0x290 mm/slab.c:3760
mempool_free_slab+0x1d/0x30 mm/mempool.c:512
mempool_free+0xed/0x370 mm/mempool.c:494
bvec_free+0xa8/0xd0 block/bio.c:173
bio_free+0xc1/0x150 block/bio.c:259
bio_put+0x187/0x200 block/bio.c:561
ext4_end_bio+0x197/0x6e0 fs/ext4/page-io.c:343
bio_endio+0x5d2/0xb80 block/bio.c:1773
req_bio_endio block/blk-core.c:278 [inline]
blk_update_request+0x585/0xd20 block/blk-core.c:3076
scsi_end_request+0xde/0x860 drivers/scsi/scsi_lib.c:673
scsi_io_completion+0x2ce/0x1ca0 drivers/scsi/scsi_lib.c:1093
scsi_finish_command+0x579/0x970 drivers/scsi/scsi.c:248
scsi_softirq_done+0x465/0x520 drivers/scsi/scsi_lib.c:1737
blk_done_softirq+0x4c2/0x760 block/blk-softirq.c:37
__do_softirq+0x308/0xb7e kernel/softirq.c:292
The buggy address belongs to the object at ffff8801b979ddc0
which belongs to the cache biovec-max of size 8192
The buggy address is located 632 bytes to the left of
8192-byte region [ffff8801b979ddc0, ffff8801b979fdc0)
The buggy address belongs to the page:
page:ffffea0006e5e700 count:1 mapcount:0 mapping:ffff8801d79b6980 index:0x0
compound_mapcount: 0
flags: 0x2fffc0000010200(slab|head)
raw: 02fffc0000010200 ffffea000730ee08 ffffea000652d508 ffff8801d79b6980
raw: 0000000000000000 ffff8801b979ddc0 0000000100000001 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff8801b979da00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801b979da80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ffff8801b979db00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801b979db80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801b979dc00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.
^ permalink raw reply
* Re: KASAN: slab-out-of-bounds Write in __xfrm_policy_unlink
From: Dmitry Vyukov @ 2018-11-11 1:55 UTC (permalink / raw)
To: syzbot
Cc: David Miller, Herbert Xu, LKML, netdev, Steffen Klassert,
syzkaller-bugs
In-Reply-To: <000000000000e5a3e1057a59db17@google.com>
On Sat, Nov 10, 2018 at 5:54 PM, syzbot
<syzbot+1a5442803e5354a25766@syzkaller.appspotmail.com> wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: 29e12207174a sfc: use the new __netdev_tx_sent_queue BQL o..
> git tree: net-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=133c2c5d400000
> kernel config: https://syzkaller.appspot.com/x/.config?x=8f559fee2fc3375a
> dashboard link: https://syzkaller.appspot.com/bug?extid=1a5442803e5354a25766
> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>
> Unfortunately, I don't have any reproducer for this crash yet.
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+1a5442803e5354a25766@syzkaller.appspotmail.com
May be related to "KASAN: slab-out-of-bounds Write in __xfrm_policy_unlink"
https://syzkaller.appspot.com/bug?extid=aa77660edbf949365033
> ==================================================================
> BUG: KASAN: slab-out-of-bounds in __write_once_size
> include/linux/compiler.h:209 [inline]
> BUG: KASAN: slab-out-of-bounds in __hlist_del include/linux/list.h:702
> [inline]
> BUG: KASAN: slab-out-of-bounds in hlist_del_rcu include/linux/rculist.h:455
> [inline]
> BUG: KASAN: slab-out-of-bounds in __xfrm_policy_unlink+0x75f/0x810
> net/xfrm/xfrm_policy.c:1241
> Write of size 8 at addr ffff8801b979db48 by task blkid/15614
>
> CPU: 1 PID: 15614 Comm: blkid Not tainted 4.20.0-rc1+ #287
> kernel msg: ebtables bug: please report to author: Nentries wrong
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
> <IRQ>
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x244/0x39d lib/dump_stack.c:113
> print_address_description.cold.7+0x9/0x1ff mm/kasan/report.c:256
> kasan_report_error mm/kasan/report.c:354 [inline]
> kasan_report.cold.8+0x242/0x309 mm/kasan/report.c:412
> __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:438
> __write_once_size include/linux/compiler.h:209 [inline]
> __hlist_del include/linux/list.h:702 [inline]
> hlist_del_rcu include/linux/rculist.h:455 [inline]
> __xfrm_policy_unlink+0x75f/0x810 net/xfrm/xfrm_policy.c:1241
> xfrm_policy_delete+0x49/0x90 net/xfrm/xfrm_policy.c:1266
> xfrm_policy_timer+0x46f/0x660 net/xfrm/xfrm_policy.c:254
> call_timer_fn+0x272/0x920 kernel/time/timer.c:1326
> expire_timers kernel/time/timer.c:1363 [inline]
> __run_timers+0x7e5/0xc70 kernel/time/timer.c:1682
> run_timer_softirq+0x52/0xb0 kernel/time/timer.c:1695
> __do_softirq+0x308/0xb7e kernel/softirq.c:292
> invoke_softirq kernel/softirq.c:373 [inline]
> irq_exit+0x17f/0x1c0 kernel/softirq.c:413
> exiting_irq arch/x86/include/asm/apic.h:536 [inline]
> smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1061
> apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:804
> </IRQ>
> RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:761
> [inline]
> RIP: 0010:count_memcg_events include/linux/memcontrol.h:742 [inline]
> RIP: 0010:count_memcg_event_mm include/linux/memcontrol.h:763 [inline]
> RIP: 0010:handle_mm_fault+0x9a8/0xc70 mm/memory.c:3906
> Code: df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 4e 02 00 00 48 83 3d 78 5b
> 9e 07 00 0f 84 70 01 00 00 e8 7d ea cb ff 4c 89 ff 57 9d <0f> 1f 44 00 00 e9
> 22 fa ff ff e8 69 ea cb ff 49 8d bd 20 04 00 00
> RSP: 0000:ffff8801bbcffcc8 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
> RAX: ffff8801bc1ae540 RBX: ffff8801c6f5dca0 RCX: 1ffff10037835dbc
> RDX: 0000000000000000 RSI: ffffffff81b3a043 RDI: 0000000000000293
> RBP: ffff8801bbcffd68 R08: ffff8801bc1aede0 R09: 0000000000000007
> R10: 0000000000000000 R11: ffff8801bc1ae540 R12: 1ffff1003779ff9c
> R13: 0000000000000200 R14: 0000000000000054 R15: 0000000000000293
> do_user_addr_fault arch/x86/mm/fault.c:1423 [inline]
> __do_page_fault+0x5e8/0xe60 arch/x86/mm/fault.c:1489
> do_page_fault+0xf2/0x7e0 arch/x86/mm/fault.c:1520
> page_fault+0x1e/0x30 arch/x86/entry/entry_64.S:1139
> RIP: 0033:0x7f05c5e50d61
> Code: 00 c3 f7 05 9d c4 30 00 04 00 00 00 74 07 48 8d 05 84 95 0b 00 c3 0f
> 1f 00 48 31 c0 89 f9 83 e1 3f 66 0f ef c0 83 f9 30 77 19 <f3> 0f 6f 0f 66 0f
> 74 c1 66 0f d7 d0 85 d2 75 7a 48 89 f8 48 83 e0
> RSP: 002b:00007ffc5d3e2558 EFLAGS: 00010287
> RAX: 0000000000000000 RBX: 00007f05c638487c RCX: 0000000000000020
> RDX: 0000000000000005 RSI: 00007ffc5d3e2d48 RDI: 00007f05c6179a20
> RBP: 00007ffc5d3e2f2b R08: 0000000000000008 R09: 0101010101010101
> R10: 0000000000000000 R11: 00007f05c5e07060 R12: 0000000000403738
> R13: 0000000000000001 R14: 0000000000000000 R15: 00007f05c6179a20
>
> Allocated by task 7136:
> save_stack+0x43/0xd0 mm/kasan/kasan.c:448
> set_track mm/kasan/kasan.c:460 [inline]
> kasan_kmalloc+0xc7/0xe0 mm/kasan/kasan.c:553
> kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490
> kmem_cache_alloc+0x12e/0x730 mm/slab.c:3554
> mempool_alloc_slab+0x44/0x60 mm/mempool.c:505
> mempool_alloc+0x193/0x4a0 mm/mempool.c:385
> bvec_alloc+0x12a/0x2d0 block/bio.c:218
> bio_alloc_bioset+0x47a/0x700 block/bio.c:509
> bio_alloc include/linux/bio.h:393 [inline]
> io_submit_init_bio fs/ext4/page-io.c:374 [inline]
> io_submit_add_bh fs/ext4/page-io.c:399 [inline]
> ext4_bio_write_page+0x1304/0x1bd1 fs/ext4/page-io.c:506
> mpage_submit_page+0x15e/0x270 fs/ext4/inode.c:2237
> mpage_process_page_bufs+0x50a/0x600 fs/ext4/inode.c:2348
> mpage_prepare_extent_to_map+0xea5/0x19c0 fs/ext4/inode.c:2710
> ext4_writepages+0x140c/0x41a0 fs/ext4/inode.c:2837
> do_writepages+0x9a/0x1a0 mm/page-writeback.c:2328
> __writeback_single_inode+0x20a/0x1660 fs/fs-writeback.c:1316
> writeback_sb_inodes+0x71f/0x1210 fs/fs-writeback.c:1580
> __writeback_inodes_wb+0x1b9/0x340 fs/fs-writeback.c:1649
> wb_writeback+0xa73/0xfc0 fs/fs-writeback.c:1758
> wb_check_start_all fs/fs-writeback.c:1882 [inline]
> wb_do_writeback fs/fs-writeback.c:1908 [inline]
> wb_workfn+0xee9/0x1790 fs/fs-writeback.c:1942
> process_one_work+0xc90/0x1c40 kernel/workqueue.c:2153
> worker_thread+0x17f/0x1390 kernel/workqueue.c:2296
> kthread+0x35a/0x440 kernel/kthread.c:246
> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352
>
> Freed by task 5729:
> save_stack+0x43/0xd0 mm/kasan/kasan.c:448
> set_track mm/kasan/kasan.c:460 [inline]
> __kasan_slab_free+0x102/0x150 mm/kasan/kasan.c:521
> kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
> __cache_free mm/slab.c:3498 [inline]
> kmem_cache_free+0x83/0x290 mm/slab.c:3760
> mempool_free_slab+0x1d/0x30 mm/mempool.c:512
> mempool_free+0xed/0x370 mm/mempool.c:494
> bvec_free+0xa8/0xd0 block/bio.c:173
> bio_free+0xc1/0x150 block/bio.c:259
> bio_put+0x187/0x200 block/bio.c:561
> ext4_end_bio+0x197/0x6e0 fs/ext4/page-io.c:343
> bio_endio+0x5d2/0xb80 block/bio.c:1773
> req_bio_endio block/blk-core.c:278 [inline]
> blk_update_request+0x585/0xd20 block/blk-core.c:3076
> scsi_end_request+0xde/0x860 drivers/scsi/scsi_lib.c:673
> scsi_io_completion+0x2ce/0x1ca0 drivers/scsi/scsi_lib.c:1093
> scsi_finish_command+0x579/0x970 drivers/scsi/scsi.c:248
> scsi_softirq_done+0x465/0x520 drivers/scsi/scsi_lib.c:1737
> blk_done_softirq+0x4c2/0x760 block/blk-softirq.c:37
> __do_softirq+0x308/0xb7e kernel/softirq.c:292
>
> The buggy address belongs to the object at ffff8801b979ddc0
> which belongs to the cache biovec-max of size 8192
> The buggy address is located 632 bytes to the left of
> 8192-byte region [ffff8801b979ddc0, ffff8801b979fdc0)
> The buggy address belongs to the page:
> page:ffffea0006e5e700 count:1 mapcount:0 mapping:ffff8801d79b6980 index:0x0
> compound_mapcount: 0
> flags: 0x2fffc0000010200(slab|head)
> raw: 02fffc0000010200 ffffea000730ee08 ffffea000652d508 ffff8801d79b6980
> raw: 0000000000000000 ffff8801b979ddc0 0000000100000001 0000000000000000
> page dumped because: kasan: bad access detected
>
> Memory state around the buggy address:
> ffff8801b979da00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ffff8801b979da80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>>
>> ffff8801b979db00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>
> ^
> ffff8801b979db80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ffff8801b979dc00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ==================================================================
>
>
> ---
> This bug is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this bug report. See:
> https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
> syzbot.
>
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000e5a3e1057a59db17%40google.com.
> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply
* Re: KASAN: slab-out-of-bounds Write in __xfrm_policy_unlink
From: Dmitry Vyukov @ 2018-11-11 1:57 UTC (permalink / raw)
To: syzbot
Cc: David Miller, Herbert Xu, LKML, netdev, Steffen Klassert,
syzkaller-bugs
In-Reply-To: <CACT4Y+YO6i-rEAydWQPadtGw7w_ZDT58Mt6XYybnBdUQMcxh-Q@mail.gmail.com>
On Sat, Nov 10, 2018 at 5:55 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Sat, Nov 10, 2018 at 5:54 PM, syzbot
> <syzbot+1a5442803e5354a25766@syzkaller.appspotmail.com> wrote:
>> Hello,
>>
>> syzbot found the following crash on:
>>
>> HEAD commit: 29e12207174a sfc: use the new __netdev_tx_sent_queue BQL o..
>> git tree: net-next
>> console output: https://syzkaller.appspot.com/x/log.txt?x=133c2c5d400000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=8f559fee2fc3375a
>> dashboard link: https://syzkaller.appspot.com/bug?extid=1a5442803e5354a25766
>> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>>
>> Unfortunately, I don't have any reproducer for this crash yet.
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: syzbot+1a5442803e5354a25766@syzkaller.appspotmail.com
>
> May be related to "KASAN: slab-out-of-bounds Write in __xfrm_policy_unlink"
> https://syzkaller.appspot.com/bug?extid=aa77660edbf949365033
Obviously related to itself :)
I meant "KASAN: use-after-free Write in __xfrm_policy_unlink"
https://syzkaller.appspot.com/bug?id=ebeba334a8a886e3d5dc25641e201e894d4d9657
>> ==================================================================
>> BUG: KASAN: slab-out-of-bounds in __write_once_size
>> include/linux/compiler.h:209 [inline]
>> BUG: KASAN: slab-out-of-bounds in __hlist_del include/linux/list.h:702
>> [inline]
>> BUG: KASAN: slab-out-of-bounds in hlist_del_rcu include/linux/rculist.h:455
>> [inline]
>> BUG: KASAN: slab-out-of-bounds in __xfrm_policy_unlink+0x75f/0x810
>> net/xfrm/xfrm_policy.c:1241
>> Write of size 8 at addr ffff8801b979db48 by task blkid/15614
>>
>> CPU: 1 PID: 15614 Comm: blkid Not tainted 4.20.0-rc1+ #287
>> kernel msg: ebtables bug: please report to author: Nentries wrong
>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> Google 01/01/2011
>> Call Trace:
>> <IRQ>
>> __dump_stack lib/dump_stack.c:77 [inline]
>> dump_stack+0x244/0x39d lib/dump_stack.c:113
>> print_address_description.cold.7+0x9/0x1ff mm/kasan/report.c:256
>> kasan_report_error mm/kasan/report.c:354 [inline]
>> kasan_report.cold.8+0x242/0x309 mm/kasan/report.c:412
>> __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:438
>> __write_once_size include/linux/compiler.h:209 [inline]
>> __hlist_del include/linux/list.h:702 [inline]
>> hlist_del_rcu include/linux/rculist.h:455 [inline]
>> __xfrm_policy_unlink+0x75f/0x810 net/xfrm/xfrm_policy.c:1241
>> xfrm_policy_delete+0x49/0x90 net/xfrm/xfrm_policy.c:1266
>> xfrm_policy_timer+0x46f/0x660 net/xfrm/xfrm_policy.c:254
>> call_timer_fn+0x272/0x920 kernel/time/timer.c:1326
>> expire_timers kernel/time/timer.c:1363 [inline]
>> __run_timers+0x7e5/0xc70 kernel/time/timer.c:1682
>> run_timer_softirq+0x52/0xb0 kernel/time/timer.c:1695
>> __do_softirq+0x308/0xb7e kernel/softirq.c:292
>> invoke_softirq kernel/softirq.c:373 [inline]
>> irq_exit+0x17f/0x1c0 kernel/softirq.c:413
>> exiting_irq arch/x86/include/asm/apic.h:536 [inline]
>> smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1061
>> apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:804
>> </IRQ>
>> RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:761
>> [inline]
>> RIP: 0010:count_memcg_events include/linux/memcontrol.h:742 [inline]
>> RIP: 0010:count_memcg_event_mm include/linux/memcontrol.h:763 [inline]
>> RIP: 0010:handle_mm_fault+0x9a8/0xc70 mm/memory.c:3906
>> Code: df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 4e 02 00 00 48 83 3d 78 5b
>> 9e 07 00 0f 84 70 01 00 00 e8 7d ea cb ff 4c 89 ff 57 9d <0f> 1f 44 00 00 e9
>> 22 fa ff ff e8 69 ea cb ff 49 8d bd 20 04 00 00
>> RSP: 0000:ffff8801bbcffcc8 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
>> RAX: ffff8801bc1ae540 RBX: ffff8801c6f5dca0 RCX: 1ffff10037835dbc
>> RDX: 0000000000000000 RSI: ffffffff81b3a043 RDI: 0000000000000293
>> RBP: ffff8801bbcffd68 R08: ffff8801bc1aede0 R09: 0000000000000007
>> R10: 0000000000000000 R11: ffff8801bc1ae540 R12: 1ffff1003779ff9c
>> R13: 0000000000000200 R14: 0000000000000054 R15: 0000000000000293
>> do_user_addr_fault arch/x86/mm/fault.c:1423 [inline]
>> __do_page_fault+0x5e8/0xe60 arch/x86/mm/fault.c:1489
>> do_page_fault+0xf2/0x7e0 arch/x86/mm/fault.c:1520
>> page_fault+0x1e/0x30 arch/x86/entry/entry_64.S:1139
>> RIP: 0033:0x7f05c5e50d61
>> Code: 00 c3 f7 05 9d c4 30 00 04 00 00 00 74 07 48 8d 05 84 95 0b 00 c3 0f
>> 1f 00 48 31 c0 89 f9 83 e1 3f 66 0f ef c0 83 f9 30 77 19 <f3> 0f 6f 0f 66 0f
>> 74 c1 66 0f d7 d0 85 d2 75 7a 48 89 f8 48 83 e0
>> RSP: 002b:00007ffc5d3e2558 EFLAGS: 00010287
>> RAX: 0000000000000000 RBX: 00007f05c638487c RCX: 0000000000000020
>> RDX: 0000000000000005 RSI: 00007ffc5d3e2d48 RDI: 00007f05c6179a20
>> RBP: 00007ffc5d3e2f2b R08: 0000000000000008 R09: 0101010101010101
>> R10: 0000000000000000 R11: 00007f05c5e07060 R12: 0000000000403738
>> R13: 0000000000000001 R14: 0000000000000000 R15: 00007f05c6179a20
>>
>> Allocated by task 7136:
>> save_stack+0x43/0xd0 mm/kasan/kasan.c:448
>> set_track mm/kasan/kasan.c:460 [inline]
>> kasan_kmalloc+0xc7/0xe0 mm/kasan/kasan.c:553
>> kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490
>> kmem_cache_alloc+0x12e/0x730 mm/slab.c:3554
>> mempool_alloc_slab+0x44/0x60 mm/mempool.c:505
>> mempool_alloc+0x193/0x4a0 mm/mempool.c:385
>> bvec_alloc+0x12a/0x2d0 block/bio.c:218
>> bio_alloc_bioset+0x47a/0x700 block/bio.c:509
>> bio_alloc include/linux/bio.h:393 [inline]
>> io_submit_init_bio fs/ext4/page-io.c:374 [inline]
>> io_submit_add_bh fs/ext4/page-io.c:399 [inline]
>> ext4_bio_write_page+0x1304/0x1bd1 fs/ext4/page-io.c:506
>> mpage_submit_page+0x15e/0x270 fs/ext4/inode.c:2237
>> mpage_process_page_bufs+0x50a/0x600 fs/ext4/inode.c:2348
>> mpage_prepare_extent_to_map+0xea5/0x19c0 fs/ext4/inode.c:2710
>> ext4_writepages+0x140c/0x41a0 fs/ext4/inode.c:2837
>> do_writepages+0x9a/0x1a0 mm/page-writeback.c:2328
>> __writeback_single_inode+0x20a/0x1660 fs/fs-writeback.c:1316
>> writeback_sb_inodes+0x71f/0x1210 fs/fs-writeback.c:1580
>> __writeback_inodes_wb+0x1b9/0x340 fs/fs-writeback.c:1649
>> wb_writeback+0xa73/0xfc0 fs/fs-writeback.c:1758
>> wb_check_start_all fs/fs-writeback.c:1882 [inline]
>> wb_do_writeback fs/fs-writeback.c:1908 [inline]
>> wb_workfn+0xee9/0x1790 fs/fs-writeback.c:1942
>> process_one_work+0xc90/0x1c40 kernel/workqueue.c:2153
>> worker_thread+0x17f/0x1390 kernel/workqueue.c:2296
>> kthread+0x35a/0x440 kernel/kthread.c:246
>> ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352
>>
>> Freed by task 5729:
>> save_stack+0x43/0xd0 mm/kasan/kasan.c:448
>> set_track mm/kasan/kasan.c:460 [inline]
>> __kasan_slab_free+0x102/0x150 mm/kasan/kasan.c:521
>> kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
>> __cache_free mm/slab.c:3498 [inline]
>> kmem_cache_free+0x83/0x290 mm/slab.c:3760
>> mempool_free_slab+0x1d/0x30 mm/mempool.c:512
>> mempool_free+0xed/0x370 mm/mempool.c:494
>> bvec_free+0xa8/0xd0 block/bio.c:173
>> bio_free+0xc1/0x150 block/bio.c:259
>> bio_put+0x187/0x200 block/bio.c:561
>> ext4_end_bio+0x197/0x6e0 fs/ext4/page-io.c:343
>> bio_endio+0x5d2/0xb80 block/bio.c:1773
>> req_bio_endio block/blk-core.c:278 [inline]
>> blk_update_request+0x585/0xd20 block/blk-core.c:3076
>> scsi_end_request+0xde/0x860 drivers/scsi/scsi_lib.c:673
>> scsi_io_completion+0x2ce/0x1ca0 drivers/scsi/scsi_lib.c:1093
>> scsi_finish_command+0x579/0x970 drivers/scsi/scsi.c:248
>> scsi_softirq_done+0x465/0x520 drivers/scsi/scsi_lib.c:1737
>> blk_done_softirq+0x4c2/0x760 block/blk-softirq.c:37
>> __do_softirq+0x308/0xb7e kernel/softirq.c:292
>>
>> The buggy address belongs to the object at ffff8801b979ddc0
>> which belongs to the cache biovec-max of size 8192
>> The buggy address is located 632 bytes to the left of
>> 8192-byte region [ffff8801b979ddc0, ffff8801b979fdc0)
>> The buggy address belongs to the page:
>> page:ffffea0006e5e700 count:1 mapcount:0 mapping:ffff8801d79b6980 index:0x0
>> compound_mapcount: 0
>> flags: 0x2fffc0000010200(slab|head)
>> raw: 02fffc0000010200 ffffea000730ee08 ffffea000652d508 ffff8801d79b6980
>> raw: 0000000000000000 ffff8801b979ddc0 0000000100000001 0000000000000000
>> page dumped because: kasan: bad access detected
>>
>> Memory state around the buggy address:
>> ffff8801b979da00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>> ffff8801b979da80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>>>
>>> ffff8801b979db00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>>
>> ^
>> ffff8801b979db80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>> ffff8801b979dc00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>> ==================================================================
>>
>>
>> ---
>> This bug is generated by a bot. It may contain errors.
>> See https://goo.gl/tpsmEJ for more information about syzbot.
>> syzbot engineers can be reached at syzkaller@googlegroups.com.
>>
>> syzbot will keep track of this bug report. See:
>> https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
>> syzbot.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "syzkaller-bugs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to syzkaller-bugs+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/syzkaller-bugs/000000000000e5a3e1057a59db17%40google.com.
>> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply
* Re: [PATCH] add an initial version of snmp_counter.rst
From: Randy Dunlap @ 2018-11-10 16:56 UTC (permalink / raw)
To: yupeng, netdev, xiyou.wangcong
In-Reply-To: <20181109181309.5541-1-yupeng0921@gmail.com>
On 11/9/18 10:13 AM, yupeng wrote:
> The snmp_counter.rst run a set of simple experiments, explains the
> meaning of snmp counters depend on the experiments' results. This is
> an initial version, only covers a small part of the snmp counters.
>
> Signed-off-by: yupeng <yupeng0921@gmail.com>
> ---
> Documentation/networking/index.rst | 1 +
> Documentation/networking/snmp_counter.rst | 963 ++++++++++++++++++++++
> 2 files changed, 964 insertions(+)
> create mode 100644 Documentation/networking/snmp_counter.rst
Hi,
> diff --git a/Documentation/networking/snmp_counter.rst b/Documentation/networking/snmp_counter.rst
> new file mode 100644
> index 000000000000..2939c5acf675
> --- /dev/null
> +++ b/Documentation/networking/snmp_counter.rst
> @@ -0,0 +1,963 @@
> +====================
> +snmp counter tutorial
> +====================
First, I would change (almost) all occurrences of "snmp" to "SNMP", unless
it refers to a variable or struct or function etc. (something in C language)
or to the name of this file (document).
> +
> +This document explains the meaning of snmp counters. For understanding
> +their meanings better, this document doesn't explain the counters one
> +by one, but creates a set of experiments, and explains the counters
> +depend on the experiments' results. The experiments are on one or two
depending on
> +virtual machines. Except for the test commands we use in the experiments,
> +the virtual machines have no other network traffic. We use the 'nstat'
> +command to get the values of snmp counters, before every test, we run
counters;
> +'nstat -n' to update the history, so the 'nstat' output would only
> +show the changes of the snmp counters. For more information about
> +nstat, please refer:
> +
> +http://man7.org/linux/man-pages/man8/nstat.8.html
> +
> +icmp ping
> +========
s/icmp/ICMP/
> +
> +Run the ping command against the public dns server 8.8.8.8::
> +
> + nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
> + PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
> + 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
> +
> + --- 8.8.8.8 ping statistics ---
> + 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> + rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
> +
> +The nstayt result::
nstat
> +
> + nstatuser@nstat-a:~$ nstat
> + #kernel
> + IpInReceives 1 0.0
> + IpInDelivers 1 0.0
> + IpOutRequests 1 0.0
> + IcmpInMsgs 1 0.0
> + IcmpInEchoReps 1 0.0
> + IcmpOutMsgs 1 0.0
> + IcmpOutEchos 1 0.0
> + IcmpMsgInType0 1 0.0
> + IcmpMsgOutType8 1 0.0
> + IpExtInOctets 84 0.0
> + IpExtOutOctets 84 0.0
> + IpExtInNoECTPkts 1 0.0
> +
> +The nstat output could be divided into two part: one with the 'Ext'
> +keyword, another without the 'Ext' keyword. If the counter name
> +doesn't have 'Ext', it is defined by one of snmp rfc, if it has 'Ext',
by one of the SNMP RFCs; it is has
> +it is a kernel extent counter. Below we explain them one by one.
> +
> +The rfc defined counters
> +----------------------
s/rfc/RFC/
> +
> +* IpInReceives
> +The total number of input datagrams received from interfaces,
> +including those received in error.
> +
> +https://tools.ietf.org/html/rfc1213#page-26
> +
> +* IpInDelivers
> +The total number of input datagrams successfully delivered to IP
> +user-protocols (including ICMP).
> +
> +https://tools.ietf.org/html/rfc1213#page-28
> +
> +* IpOutRequests
> +The total number of IP datagrams which local IP user-protocols
> +(including ICMP) supplied to IP in requests for transmission. Note
> +that this counter does not include any datagrams counted in
> +ipForwDatagrams.
> +
> +https://tools.ietf.org/html/rfc1213#page-28
> +
> +* IcmpInMsgs
> +The total number of ICMP messages which the entity received. Note
> +that this counter includes all those counted by icmpInErrors.
> +
> +https://tools.ietf.org/html/rfc1213#page-41
> +
> +* IcmpInEchoReps
> +The number of ICMP Echo Reply messages received.
> +
> +https://tools.ietf.org/html/rfc1213#page-42
> +
> +* IcmpOutMsgs
> +The total number of ICMP messages which this entity attempted to send.
> +Note that this counter includes all those counted by icmpOutErrors.
> +
> +https://tools.ietf.org/html/rfc1213#page-43
> +
> +* IcmpOutEchos
> +The number of ICMP Echo (request) messages sent.
> +
> +https://tools.ietf.org/html/rfc1213#page-45
> +
> +IcmpMsgInType0 and IcmpMsgOutType8 are not defined by any snmp related
SNMP-related
> +RFCs, but their meaning are quite straightforward, they count the
meanings straightforward: they count the
> +packet number of specific icmp packet types. We could find the icmp
s/could/can/
s/icmp/ICMP/ 2 times
> +types here:
> +
> +https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
> +
> +Type 8 is echo, type 0 is echo reply.
> +
> +Until now, we can easily explain these items of the nstat: We sent an
> +icmp echo request, so IpOutRequests, IcmpOutMsgs, IcmpOutEchos and
ICMP
> +IcmpMsgOutType8 were increased 1. We got icmp echo reply from 8.8.8.8,
ICMP
> +so IpInReceives, IcmpInMsgs, IcmpInEchoReps, IcmpMsgInType0 were
> +increased 1. The icmp echo reply was passed to icmp layer via ip
s/icmp/ICMP/ 2 times
s/ip/IP/
> +layer, so IpInDelivers was increased 1.
> +
> +Please note, these metrics don't aware LRO/GRO, e.g., IpOutRequests
aren't aware of
> +might count 1 packet, but hardware splits it to 2, and sends them
> +separately.
> +
> +IpExtInOctets and IpExtOutOctets
> +------------------------------
> +They are linux kernel extensions, no rfc definitions. Please note,
s/linux/Linux/
s/no rfc/not RFC/
> +rfc1213 indeed defines ifInOctets and ifOutOctets, but they
> +are different things. The ifInOctets and ifOutOctets are packets
s/packets/packet/
> +size which includes the mac layer. But IpExtInOctets and IpExtOutOctets
s/size/sizes/
s/mac/MAC/
> +are only ip layer sizes.
s/ip/IP/
> +
> +In our example, an ICMP echo request has four parts:
> +* 14 bytes mac header
MAC
> +* 20 bytes ip header
IP
> +* 16 bytes icmp header
ICMP
> +* 48 bytes data (default value of the ping command)
> +
> +So IpExtInOctets value is 20+16+48=84. The IpExtOutOctets is similar.
value is similar.
> +
> +IpExtInNoECTPkts
> +---------------
> +We could find IpExtInNoECTPkts in the nstat output, but kernel provide
We can find provides
> +four similar counters, we explain them together, they are:
counters. We explain them togther; they are:
> +* IpExtInNoECTPkts
> +* IpExtInECT1Pkts
> +* IpExtInECT0Pkts
> +* IpExtInCEPkts
> +
> +They indicate four kinds of ECN IP packets, they are defined here:
packets;
> +
> +https://tools.ietf.org/html/rfc3168#page-6
> +
> +These 4 counters calculate how many packets received per ECN
> +status. They count the real frame number regardless the LRO/GRO. So
numbers regardless of LRO/GRO. So
> +for the same packet, you might find that IpInReceives count 1, but
that the IpInReceives count is 1, but
> +IpExtInNoECTPkts counts 2 or more.
the IpExtInNoECTPkts count is 2 or more.
> +
> +additional explain
> +-----------------
additional explanation
----------------------
(note: the "---" line must be the same length as the "heading" line above it.
> +The ip layer counters are recorded by the ip layer code in the kernel. I mean, if you send a packet to a lower layer directly, Linux
s/ip/IP/ 2 times
Also split that line so that it is < 80 columns wide.
Then:
This means that
if you send a packet to a lower layer directly, Linux
> +kernel won't record it. For example, tcpreplay will open an
> +AF_PACKET socket, and send the packet to layer 2, although it could send
layer 2, and although it could send
> +an IP packet, you can't find it from the nstat output. Here is an
> +example:
> +
> +We capture the ping packet by tcpdump::
> +
> + nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/ping.pcap dst 8.8.8.8
> +
> +Then run ping command::
> +
> + nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
> +
> +Terminate tcpdump by Ctrl-C, and run 'nstat -n' to update the nstat
> +history. Then run tcpreplay::
> +
> + nstatuser@nstat-a:~$ sudo tcpreplay --intf1=ens3 /tmp/ping.pcap
> + Actual: 1 packets (98 bytes) sent in 0.000278 seconds
> + Rated: 352517.9 Bps, 2.82 Mbps, 3597.12 pps
> + Flows: 1 flows, 3597.12 fps, 1 flow packets, 0 non-flow
> + Statistics for network device: ens3
> + Successful packets: 1
> + Failed packets: 0
> + Truncated packets: 0
> + Retried packets (ENOBUFS): 0
> + Retried packets (EAGAIN): 0
> +
> +Check the nstat output::
> +
> + nstatuser@nstat-a:~$ nstat
> + #kernel
> + IpInReceives 1 0.0
> + IpInDelivers 1 0.0
> + IcmpInMsgs 1 0.0
> + IcmpInEchoReps 1 0.0
> + IcmpMsgInType0 1 0.0
> + IpExtInOctets 84 0.0
> + IpExtInNoECTPkts 1 0.0
> +
> +We can see, nstat only show the received packet, because the IP layer
We can see that nstat only shows the received packet, because the IP layer
> +of kernel only know the reply of 8.8.8.8, it doesn't know what
of the kernel only knows about the reply of 8.8.8.8; it doesn't know what
> +tcpreplay sent.
> +
> +At the same time, when you use AF_INET socket, even you use the
even if you use the
> +SOCK_RAW option, the IP layer will still try to verify whether the
> +packet is an ICMP packet, if it is, kernel will still count it to its
packet is an ICMP packet. If it is, the kernel will
> +counters and you can find it in the output of nstat.
^^^ interesting.
> +
> +tcp 3 way handshake
> +==================
TCP 3-way handshake
===================
> +
> +On server side, we run::
On the server side,
> +
> + nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
> + Listening on [0.0.0.0] (family 0, port 9000)
> +
> +On client side, we run::
On the client side,
> +
> + nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
> + Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
> +
> +The server listened on tcp 9000 port, the client connected to it, they
> +completed the 3-way handshake.
> +
> +On server side, we can find below nstat output::
On the server side, we can see the following nstat output::
> +
> + nstatuser@nstat-b:~$ nstat | grep -i tcp
> + TcpPassiveOpens 1 0.0
> + TcpInSegs 2 0.0
> + TcpOutSegs 1 0.0
> + TcpExtTCPPureAcks 1 0.0
> +
> +On client side, we can find below nstat output::
On the client side, we can see the following nstat output::
> +
> + nstatuser@nstat-a:~$ nstat | grep -i tcp
> + TcpActiveOpens 1 0.0
> + TcpInSegs 1 0.0
> + TcpOutSegs 2 0.0
> +
> +Except for TcpExtTCPPureAcks, all other counters are defined by rfc1213
by rfc1213.
and that's just part of a file name; I would really prefer to see:
by RFC 1213.
> +
> +* TcpActiveOpens
> +The number of times TCP connections have made a direct transition to
> +the SYN-SENT state from the CLOSED state.
> +
> +https://tools.ietf.org/html/rfc1213#page-47
> +
> +* TcpPassiveOpens
> +The number of times TCP connections have made a direct transition to
> +the SYN-RCVD state from the LISTEN state.
> +
> +https://tools.ietf.org/html/rfc1213#page-47
> +
> +* TcpInSegs
> +The total number of segments received, including those received in
> +error. This count includes segments received on currently established
> +connections.
> +
> +https://tools.ietf.org/html/rfc1213#page-48
> +
> +* TcpOutSegs
> +The total number of segments sent, including those on current
> +connections but excluding those containing only retransmitted octets.
> +
> +https://tools.ietf.org/html/rfc1213#page-48
> +
> +
> +The TcpExtTCPPureAcks is an extension in linux kernel. When kernel
in the Linux kernel. When the kernel
> +receives a TCP packet which set ACK flag and with no data, either
sets the ACK flag and with no data, either
> +TcpExtTCPPureAcks or TcpExtTCPHPAcks will increase 1. We will discuss
> +it in a later section.
s/it/this/
> +
> +Now we can easily explain the nstat outputs on the server side and client
> +side.
> +
> +When the server received the first syn, it replied a syn+ack, and came into
it replied with a syn+ack,
> +SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
> +syn, sent syn+ack, received ack, so server sent 1 packet, received 2
> +packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ack
> +of the 3-way handshake is a pure ack without data, so
> +TcpExtTCPPureAcks increased 1.
> +
> +When the client sent syn, the client came into the SYN-SENT state, so
> +TcpActiveOpens increased 1, client sent syn, received syn+ack, sent
> +ack, so client sent 2 packets, received 1 packet, TcpInSegs increased
> +1, TcpOutSegs increased 2.
> +
> +Note: about TcpInSegs and TcpOutSegs, rfc1213 doesn't define the
> +behaviors when gso/gro/tso are enabled on the NIC (network interface
GSO/GRO/TSO
> +card). On current linux implementation, TcpOutSegs awares gso/tso, but
Linux is aware of GSO/TSO, but
> +TcpInSegs doesn't aware gro. So TcpOutSegs will count the actual
isn't aware of GRO.
> +packet number even only 1 packet is sent via tcp layer. If multiple
even if only 1 packet is sent via the TCP later.
> +packets arrived at a NIC, and they are merged into 1 packet, TcpInSegs
> +will only count 1.
> +
> +tcp disconnect
> +=============
TCP disconnect
==============
> +
> +Continue our previous example, on the server side, we have run::
Continuing
> +
> + nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
> + Listening on [0.0.0.0] (family 0, port 9000)
> +
> +On client side, we have run::
> +
> + nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
> + Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
> +
> +Now we type Ctrl-C on the client side, stop the tcp connection between the
stopping the TCP
> +two nc command. Then we check the nstat output.
commands.
> +
> +On server side::
> +
> + nstatuser@nstat-b:~$ nstat | grep -i tcp
> + TcpInSegs 2 0.0
> + TcpOutSegs 1 0.0
> + TcpExtTCPPureAcks 1 0.0
> + TcpExtTCPOrigDataSent 1 0.0
> +
> +On client side::
> +
> + nstatuser@nstat-b:~$ nstat | grep -i tcp
> + TcpInSegs 2 0.0
> + TcpOutSegs 1 0.0
> + TcpExtTCPPureAcks 1 0.0
> + TcpExtTCPOrigDataSent 1 0.0
> +
> +Wait for more than 1 minute, run nstat on client again::
> +
> + nstatuser@nstat-a:~$ nstat | grep -i tcp
> + TcpExtTW 1 0.0
> +
> +Most of the counters are explained in the previous section except
> +two: TcpExtTCPOrigDataSent and TcpExtTW. Both of them are linux kernel
s/linux/Linux/
> +extensions.
> +
> +TcpExtTW means a tcp connection is closed normally via
s/tcp/TCP/
> +time wait stage, not via tcp reuse process.
s/tcp/TCP/
> +
> +About TcpExtTCPOrigDataSent, Below kernel patch has a good explanation:
the below kernel patch has
> +
> +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd
> +
> +I pasted it here::
> +
> + TCPOrigDataSent: number of outgoing packets with original data
> + (excluding retransmission but including data-in-SYN). This counter is
> + different from TcpOutSegs because TcpOutSegs also tracks pure
> + ACKs. TCPOrigDataSent is more useful to track the TCP retransmission rate.
> +
> +the effect of gso and gro
> +=======================
the effect of GSO and GRO
=========================
> +
> +The Generic Segmentation Offload (GSO) and Generic Receive Offload
Offload (GRO)
> +would affect the metrics of the packet in/out on both ip and tcp
on both IP and TCP
> +layer. Here is an iperf example. Before the test, run below command to
layers. run the command below to
> +make sure both gso and gro are enabled on the NIC::
GSO and GRO
> +
> + $ sudo ethtool -k ens3 | egrep '(generic-segmentation-offload|generic-receive-offload)'
> + generic-segmentation-offload: on
> + generic-receive-offload: on
> +
> +On server side, run::
On the server side,
> +
> + iperf3 -s -p 9000
> +
> +On client side, run::
On the client side,
> +
> + iperf3 -c 192.168.122.251 -p 9000 -t 5 -P 10
> +
> +The server listened on tcp port 9000, the client connected to the server,
TCP
> +created 10 threads parallel, run 5 seconds. After the pierf3 stopped, we
After iperf3 stopped, we
> +run nstat on both the server and client.
> +
> +On server side::
On the server side::
> +
> + nstatuser@nstat-b:~$ nstat
> + #kernel
> + IpInReceives 36346 0.0
> + IpInDelivers 36346 0.0
> + IpOutRequests 33836 0.0
> + TcpPassiveOpens 11 0.0
> + TcpEstabResets 2 0.0
> + TcpInSegs 36346 0.0
> + TcpOutSegs 33836 0.0
> + TcpOutRsts 20 0.0
> + TcpExtDelayedACKs 26 0.0
> + TcpExtTCPHPHits 32120 0.0
> + TcpExtTCPPureAcks 16 0.0
> + TcpExtTCPHPAcks 5 0.0
> + TcpExtTCPAbortOnData 5 0.0
> + TcpExtTCPAbortOnClose 2 0.0
> + TcpExtTCPRcvCoalesce 7306 0.0
> + TcpExtTCPOFOQueue 1354 0.0
> + TcpExtTCPOrigDataSent 15 0.0
> + IpExtInOctets 311732432 0.0
> + IpExtOutOctets 1785119 0.0
> + IpExtInNoECTPkts 214032 0.0
> +
> +Client side::
On the client side::
> +
> + nstatuser@nstat-a:~$ nstat
> + #kernel
> + IpInReceives 33836 0.0
> + IpInDelivers 33836 0.0
> + IpOutRequests 43786 0.0
> + TcpActiveOpens 11 0.0
> + TcpEstabResets 10 0.0
> + TcpInSegs 33836 0.0
> + TcpOutSegs 214072 0.0
> + TcpRetransSegs 3876 0.0
> + TcpExtDelayedACKs 7 0.0
> + TcpExtTCPHPHits 5 0.0
> + TcpExtTCPPureAcks 2719 0.0
> + TcpExtTCPHPAcks 31071 0.0
> + TcpExtTCPSackRecovery 607 0.0
> + TcpExtTCPSACKReorder 61 0.0
> + TcpExtTCPLostRetransmit 90 0.0
> + TcpExtTCPFastRetrans 3806 0.0
> + TcpExtTCPSlowStartRetrans 62 0.0
> + TcpExtTCPLossProbes 38 0.0
> + TcpExtTCPSackRecoveryFail 8 0.0
> + TcpExtTCPSackShifted 203 0.0
> + TcpExtTCPSackMerged 778 0.0
> + TcpExtTCPSackShiftFallback 700 0.0
> + TcpExtTCPSpuriousRtxHostQueues 4 0.0
> + TcpExtTCPAutoCorking 14 0.0
> + TcpExtTCPOrigDataSent 214038 0.0
> + TcpExtTCPHystartTrainDetect 8 0.0
> + TcpExtTCPHystartTrainCwnd 172 0.0
> + IpExtInOctets 1785227 0.0
> + IpExtOutOctets 317789680 0.0
> + IpExtInNoECTPkts 33836 0.0
> +
> +The TcpOutSegs and IpOutRequests on the server are 33836, exactly the
> +same as IpExtInNoECTPkts, IpInReceives, IpInDelivers and TcpInSegs on
> +the client side. During iperf3 test, the server only reply very short
replies with very short
> +packets, so gso and gro has no effect on the server's reply.
so GSO and GRO have no effect
> +
> +On the client side, TcpOutSegs is 214072, IpOutRequests is 43786, the
> +tcp layer packet out is larger than ip layer packet out, because
TCP layer packet size out is larger than the IP layer packet size out, because
> +TcpOutSegs count the packet number after gso, but IpOutRequests
counts GSO,
> +doesn't. On the server side, IpExtInNoECTPkts is 214032, this number
is 214032; this number
> +is smaller a little than the TcpOutSegs on the client side (214072), it
; it
> +might cause by the packet loss. The IpInReceives, IpInDelivers and
might be caused by packet loss.
> +TcpInSegs are obviously smaller than the TcpOutSegs on the client side,
> +because these counters calculate the packet after gro.
packet counts after GRO.
> +
> +tcp counters in established state
> +================================
TCP counters in established state
=================================
> +
> +Run nc on server::
> +
> + nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
> + Listening on [0.0.0.0] (family 0, port 9000)
> +
> +Run nc on client:
> +
> + nstatuser@nstat-a:~$ nc -v nstat-b 9000
> + Connection to nstat-b 9000 port [tcp/*] succeeded!
> +
> +Input a string in the nc client ('hello' in our example):
> +
> + nstatuser@nstat-a:~$ nc -v nstat-b 9000
> + Connection to nstat-b 9000 port [tcp/*] succeeded!
> + hello
> +
> +The client side nstat output:
> +
> + nstatuser@nstat-a:~$ nstat
> + #kernel
> + IpInReceives 1 0.0
> + IpInDelivers 1 0.0
> + IpOutRequests 1 0.0
> + TcpInSegs 1 0.0
> + TcpOutSegs 1 0.0
> + TcpExtTCPPureAcks 1 0.0
> + TcpExtTCPOrigDataSent 1 0.0
> + IpExtInOctets 52 0.0
> + IpExtOutOctets 58 0.0
> + IpExtInNoECTPkts 1 0.0
> +
> +The server side nstat output:
> +
> + nstatuser@nstat-b:~$ nstat
> + #kernel
> + IpInReceives 1 0.0
> + IpInDelivers 1 0.0
> + IpOutRequests 1 0.0
> + TcpInSegs 1 0.0
> + TcpOutSegs 1 0.0
> + IpExtInOctets 58 0.0
> + IpExtOutOctets 52 0.0
> + IpExtInNoECTPkts 1 0.0
> +
> +Input a string in nc client side again ('world' in our exmaple):
> +
> + nstatuser@nstat-a:~$ nc -v nstat-b 9000
> + Connection to nstat-b 9000 port [tcp/*] succeeded!
> + hello
> + world
> +
> +Client side nstat output:
> +
> + nstatuser@nstat-a:~$ nstat
> + #kernel
> + IpInReceives 1 0.0
> + IpInDelivers 1 0.0
> + IpOutRequests 1 0.0
> + TcpInSegs 1 0.0
> + TcpOutSegs 1 0.0
> + TcpExtTCPHPAcks 1 0.0
> + TcpExtTCPOrigDataSent 1 0.0
> + IpExtInOctets 52 0.0
> + IpExtOutOctets 58 0.0
> + IpExtInNoECTPkts 1 0.0
> +
> +
> +Server side nstat output:
> +
> + nstatuser@nstat-b:~$ nstat
> + #kernel
> + IpInReceives 1 0.0
> + IpInDelivers 1 0.0
> + IpOutRequests 1 0.0
> + TcpInSegs 1 0.0
> + TcpOutSegs 1 0.0
> + TcpExtTCPHPHits 1 0.0
> + IpExtInOctets 58 0.0
> + IpExtOutOctets 52 0.0
> + IpExtInNoECTPkts 1 0.0
> +
> +Compare the first client side output and the second client side
Comparing
> +output, we could find one difference: the first one had a
> +'TcpExtTCPPureAcks', but the second one had a
> +'TcpExtTCPHPAcks'. The first server side output and the second server
> +side output had a difference too: the second server side output had a
> +TcpExtTCPHPHits, but the first server side output didn't have it. The
Next line is too long -- split it into < 80 columns.
> +network traffic patterns were exactly the same: the client sent a packet to the server, the server replied an ack. But kernel handled them in different
But the kernel handled them in different
> +ways. When kernel receives a tpc packet in the established status,
When the kernel receives a TCP packet
> +kernel has two paths to handle the packet, one is fast path, another
the kernel has two paths to handle the packet: one is the fast path and the other
> +is slow path. The comment in kernel code provides a good explanation of
is the slow path.
> +them, I paste them below:
> +
> + It is split into a fast path and a slow path. The fast path is
> + disabled when:
> + - A zero window was announced from us - zero window probing
> + is only handled properly on the slow path.
> + - Out of order segments arrived.
> + - Urgent data is expected.
> + - There is no buffer space left
> + - Unexpected TCP flags/window values/header lengths are received
> + (detected by checking the TCP header against pred_flags)
> + - Data is sent in both directions. The fast path only supports pure senders
> + or pure receivers (this means either the sequence number or the ack
> + value must stay constant)
> + - Unexpected TCP option.
> +
> +Kernel will try to use fast path unless any of the above conditions
The kernel will try to use the fast path
> +are satisfied. If the packets are out of order, kernel will handle
is satisfied. order, the kernel will handle
> +them in slow path, which means the performance might be not very
in the slow path,
> +good. Kernel would also come into slow path if the "Delayed ack" is
The kernel into the slow path
> +used, because when using "Delayed ack", the data is sent in both
> +directions. When the tcp window scale option is not used, kernel will
TCP not used, the kernel will
> +try to enable fast path immediately when the connection comes into the established
> +state, but if the tcp window scale option is used, kernel will disable
TCP used, the kernel will
> +the fast path at first, and try to enable it after kerenl receives
after the kernel receives
> +packets. We could use the 'ss' command to verify whether the window
> +scale option is used. e.g. run below command on either server or
E.g., run the command below on either the server or the
> +client:
> +
> + nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
> + Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
> + tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
> + ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
> +
> +The 'wscale:7,7' means both server and client set the window scale
> +option to 7. Now we could explain the nstat output in our test:
> +
> +In the first nstat output of client side, the client sent a packet, server
of the client side, the server
> +reply an ack, when kernel handled this ack, the fast path was not
replied with an ack; when the kernel handled this ack,
> +enabled, so the ack was counted into 'TcpExtTCPPureAcks'.
> +In the second nstat output of client side, the client sent a packet again,
the client side,
> +and received another ack from the server, this time, the fast path is
from the server; this time,
> +enabled, and the ack was qualified for fast path, so it was handled by
> +the fast path, so this ack was counted into TcpExtTCPHPAcks.
> +In the first nstat output of server side, the fast path was not enabled,
of the server side,
> +so there was no 'TcpExtTCPHPHits'.
> +In the second nstat output of server side, the fast path was enabled,
of the server side,
> +and the packet received from client qualified for fast path, so it
from the client qualified for the fast path,
> +was counted into 'TcpExtTCPHPHits'.
> +
> +tcp abort
> +========
TCP abort
=========
> +
> +Some counters indicate the reaons why tcp layer want to send a rst,
why the TCP layer may want to send a RST.
What is "rst"? or "RST" in RFCs?
> +they are:
The are:
> +* TcpExtTCPAbortOnData
> +* TcpExtTCPAbortOnClose
> +* TcpExtTCPAbortOnMemory
> +* TcpExtTCPAbortOnTimeout
> +* TcpExtTCPAbortOnLinger
> +* TcpExtTCPAbortFailed
> +
> +TcpExtTCPAbortOnData
> +-------------------
--------------------
(same length as text above it)
> +
> +It means tcp layer has data in flight, but need to close the
means the TCP layer needs
> +connection. So tcp layer sends a rst to the other side, indicate the
the TCP layer indicating
> +connection is not closed very graceful. An easy way to increase this
is not closed gracefully.
> +counter is using the SO_LINGER option. Please refer to the SO_LINGER
> +section of the socket man page:
> +
> +http://man7.org/linux/man-pages/man7/socket.7.html).
> +
> +By default, when an application closes a connection, the close function
> +will return immediately and kernel will try to send the in-flight data
and the kernel
> +async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
> +to a positive number, the close function won't return immediately, but
> +wait for the in-flight data are acked by the other side, the max wait
data to be acked by the other side;
> +time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
> +when the application closes a connection, kernel will send an rst
the kernel will send an RST
> +immediately, and increase the TcpExtTCPAbortOnData counter.
> +
> +We run nc on the server side::
> +
> + nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
> + Listening on [0.0.0.0] (family 0, port 9000)
> +
> +Run below python code on the client side::
Run the below
> +
> + import socket
> + import struct
> +
> + server = 'nstat-b' # server address
> + port = 9000
> +
> + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> + s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 0))
> + s.connect((server, port))
> + s.close()
> +
> +On client side, we could see TcpExtTCPAbortOnData increased::
On the client side,
> +
> + nstatuser@nstat-a:~$ nstat | grep -i abort
> + TcpExtTCPAbortOnData 1 0.0
> +
> +If we capture packet by tcpdump, we could see the client send rst
packets RST
> +instead of fin.
FIN.
> +
> +
> +TcpExtTCPAbortOnClose
> +--------------------
---------------------
> +
> +This counter means the tcp layer has unread data when an application
TCP
> +want to close a connection.
wants
> +
> +On the server side, we run below python script:
run the below
> +
> + import socket
> + import time
> +
> + port = 9000
> +
> + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> + s.bind(('0.0.0.0', port))
> + s.listen(1)
> + sock, addr = s.accept()
> + while True:
> + time.sleep(9999999)
> +
> +This python script listen on 9000 port, but doesn't read anything from
listens on port 9000,
> +the connection.
> +
> +On the client side, we send the string "hello" by nc:
> +
> + nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
> +
> +Then, we come back to the server side, the server has received the "hello"
side;
> +packet, and tcp layer has acked this packet, but the application didn't
and the TCP layer
> +read it yet. We type Ctrl-C to terminate the server script. Then we
> +could find TcpExtTCPAbortOnClose increased 1 on the server side:
> +
> + nstatuser@nstat-b:~$ nstat | grep -i abort
> + TcpExtTCPAbortOnClose 1 0.0
> +
> +If we run tcpdump on the server side, we could find the server sent a
> +rst after we type Ctrl-C.
RST
> +
> +TcpExtTCPAbortOnMemory
> +--------------------
----------------------
> +
> +When an application closes a tcp connection, kernel still need to track
TCP the kernel still needs
> +the connection, let it complete the tcp disconnect process. E.g. an
TCP
> +app calls the close method of a socket, kernel sends fin to the other
the kernel sends FIN
> +side of the connection, then the app has no relationship with the
> +socket any more, but kernel need to keep the socket, this socket
but the kernel needs
> +becomes an orphan socket, kernel waits for the reply of the other side,
the kernel
> +and would come to the TIME_WAIT state finally. When kernel has no
the kernel does not have
> +enough memory to keep the orphan socket, kernel would send an rst to
the kernel sends an RST to
> +the other side, and delete the socket, in such situation, kernel will
deletes the socket. In such situation, the kernel will
> +increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
> +TcpExtTCPAbortOnMemory:
> +
> +* the memory used by tcp protocol is higher than the third value of
by the TCP
> +the tcp_mem. Please refer the tcp_mem section in the tcp man page:
> +
> +http://man7.org/linux/man-pages/man7/tcp.7.html
> +
> +* the orphan socket count is higher than net.ipv4.tcp_max_orphans
> +
> +Below is an example which let the orphan socket count be higher than
lets
> +net.ipv4.tcp_max_orphans.
> +
> +Change tcp_max_orphans to a smaller value on client::
on the client::
> +
> + sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
> +
> +Client code (create 64 connection to server)::
64 connections to the server)::
> +
> + nstatuser@nstat-a:~$ cat client_orphan.py
> + import socket
> + import time
> +
> + server = 'nstat-b' # server address
> + port = 9000
> +
> + count = 64
> +
> + connection_list = []
> +
> + for i in range(64):
> + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> + s.connect((server, port))
> + connection_list.append(s)
> + print("connection_count: %d" % len(connection_list))
> +
> + while True:
> + time.sleep(99999)
> +
> +Server code (accept 64 connection from client)::
64 connections from the client)::
> +
> + nstatuser@nstat-b:~$ cat server_orphan.py
> + import socket
> + import time
> +
> + port = 9000
> + count = 64
> +
> + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> + s.bind(('0.0.0.0', port))
> + s.listen(count)
> + connection_list = []
> + while True:
> + sock, addr = s.accept()
> + connection_list.append((sock, addr))
> + print("connection_count: %d" % len(connection_list))
> +
> +Run the python scripts on server and client.
> +
> +On server::
> +
> + python3 server_orphan.py
> +
> +On client::
> +
> + python3 client_orphan.py
> +
> +Run iptables on server::
> +
> + sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
> +
> +Type Ctrl-C on client, stop client_orphan.py.
stopping
> +
> +Check TcpExtTCPAbortOnMemory on client::
on the client::
> +
> + nstatuser@nstat-a:~$ nstat | grep -i abort
> + TcpExtTCPAbortOnMemory 54 0.0
> +
> +Check orphane socket count on client::
orphaned socket count on the client::
> +
> + nstatuser@nstat-a:~$ ss -s
> + Total: 131 (kernel 0)
> + TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
> +
> + Transport Total IP IPv6
> + * 0 - -
> + RAW 1 0 1
> + UDP 1 1 0
> + TCP 14 13 1
> + INET 16 14 2
> + FRAG 0 0 0
> +
> +The explanation of the test: after run server_orphan.py and
running
> +client_orphan.py, we set up 64 connections between server and
> +client. Run the iptables command, the server will drop all packets from
command. The server
> +the client, type Ctrl-C on client_orphan.py, the system of the client
client. Type
> +would try to close these connections, and before they are closed
> +gracefully, these connections became orphan sockets. As the iptables
> +of the server blocked packets from the client, the server won't receive fin
FIN
> +from the client, so all connection on clients would be stuck on FIN_WAIT_1
> +stage, so they will keep as orphan sockets until timeout. We have echo
will be kept as We have echo-ed
> +10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
> +only keep 10 orphan sockets, for all other orphan sockets, the client
sockets;
> +system sent rst for them and delete them. We have 64 connections, so
RST and deleted them.
> +the 'ss -s' command shows the system has 10 orphan sockets, and the
> +value of TcpExtTCPAbortOnMemory was 54.
> +
> +An additional explanation about orphan socket count: You could find the
> +exactly orphan socket count by the 'ss -s' command, but when kernel
exact but when the kernel
> +decide whither increases TcpExtTCPAbortOnMemory and sends rst, kernel
decides whether to increase RST, the kernel
> +doesn't always check the exactly orphan socket count. For increasing
exact
> +performance, kernel checks an approximate count firstly, if the
the kernel first; if the
> +approximate count is more than tcp_max_orphans, kernel checks the
the kernel checks the
> +exact count again. So if the approximate count is less than
> +tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
but the exact count
> +would find TcpExtTCPAbortOnMemory is not increased at all. If
> +tcp_max_orphans is large enough, it won't occur, but if you decrease
> +tcp_max_orphans to a small value like our test, you might find this
> +issue. So in our test, the client set up 64 connections although the
although [drop "the"]
> +tcp_max_orphans is 10. If the client only set up 11 connections, we
> +can't find the change of TcpExtTCPAbortOnMemory.
> +
> +TcpExtTCPAbortOnTimeout
> +----------------------
-----------------------
> +This counter will increase when any of the tcp timers expire. In this
TCP
> +situation, kernel won't send rst, just give up the connection.
the kernel RST; it just gives up the connection.
> +Continue the previous test, we wait for several minutes, because the
Continuing
> +iptables on the server blocked the traffic, the server wouldn't receive
> +fin, and all the client's orphan sockets would timeout on the
FIN,
> +FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
So if we wait for a few minutes, we can find
> +10 timeout on the client::
10 timeouts
> +
> + nstatuser@nstat-a:~$ nstat | grep -i abort
> + TcpExtTCPAbortOnTimeout 10 0.0
> +
> +TcpExtTCPAbortOnLinger
> +---------------------
----------------------
> +When a tcp connection comes into FIN_WAIT_2 state, instead of waiting
TCP
> +for the fin packet from the other side, kernel could send a rst and
FIN the kernel an RST
> +delete the socket immediately. This is not the default behavior of
> +linux kernel tcp stack, but after configuring socket option, you could
the Linux kernel TCP stack,
> +let kernel follow this behavior. Below is an example.
find the kernel following this behavior.
> +
> +The server side code::
> +
> + nstatuser@nstat-b:~$ cat server_linger.py
> + import socket
> + import time
> +
> + port = 9000
> +
> + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> + s.bind(('0.0.0.0', port))
> + s.listen(1)
> + sock, addr = s.accept()
> + while True:
> + time.sleep(9999999)
> +
> +The client side code::
> +
> + nstatuser@nstat-a:~$ cat client_linger.py
> + import socket
> + import struct
> +
> + server = 'nstat-b' # server address
> + port = 9000
> +
> + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> + s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
> + s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
> + s.connect((server, port))
> + s.close()
> +
> +Run server_linger.py on server::
> +
> + nstatuser@nstat-b:~$ python3 server_linger.py
> +
> +Run client_linger.py on client::
> +
> + nstatuser@nstat-a:~$ python3 client_linger.py
> +
> +After run client_linger.py, check the output of nstat::
running
> +
> + nstatuser@nstat-a:~$ nstat | grep -i abort
> + TcpExtTCPAbortOnLinger 1 0.0
> +
> +TcpExtTCPAbortFailed
> +-------------------
--------------------
> +The kernel tcp layer will send rst if the RFC 2525 2.17 section is satisfied:
TCP RST
> +
> +https://tools.ietf.org/html/rfc2525#page-50
> +
> +If an internal error occurs during this process, TcpExtTCPAbortFailed
> +will be increased.
> +
> +TcpExtListenOverflows and TcpExtListenDrops
> +========================================
===========================================
> +When kernel receive a syn from a client, and if the tcp accept queue
the kernel receives a SYN TCP
> +is full, kernel will drop the syn and add 1 to TcpExtListenOverflows.
the kernel SYN
> +At the same time kernel will also add 1 to TcpExtListenDrops. When
the kernel
> +a tcp socket is in LISTEN state, and kernel need to drop a packet,
TCP the kernel needs
> +kernel would always add 1 to TcpExtListenDrops. So increase
the kernel
> +TcpExtListenOverflows would let TcpExtListenDrops increasing at the
increase at
> +same time, but TcpExtListenDrops would also increase without
> +TcpExtListenOverflows increasing, e.g. a memory allocation fail would
failure would
> +also let TcpExtListenDrops increase.
> +
> +Note: The above explain bases on kernel 4.15 or above version, on an
explanation is based on kernel version 4.15 or above; on an
> +old kernel, the tcp stack has different behavior when tcp accept queue
older TCP when the TCP
> +is full. On the old kernel, tcp stack won't drop the syn, it would
older the TCP stack SYN; it would
> +complete the 3-way handshake, but as the accept queue is full, tcp
the TCP
> +stack will keep the socket in the tcp half-open queue. As it is in the
TCP
> +half open queue, tcp stack will send syn+ack on an exponential backoff
the TCP SYN+ACK
> +timer, after client replies ack, tcp stack checks whether the accept
timer. After the client replies with ACK, the TCP stack
> +queue is still full, if it is not full, move the socket to accept
full. If it socket to the accept
> +queue, if it is full, keeps the socket in the half-open queue, at next
queue. If it is full, keep the socket queue. The next
> +time client replies ack, this socket will get another chance to move
time the client replies with ACK,
> +to the accept queue.
> +
> +Here is an example:
> +
> +On server, run the nc command, listen on port 9000::
> +
> + nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
> + Listening on [0.0.0.0] (family 0, port 9000)
> +
> +On client, run 3 nc commands in different terminals::
> +
> + nstatuser@nstat-a:~$ nc -v nstat-b 9000
> + Connection to nstat-b 9000 port [tcp/*] succeeded!
> +
> +The nc command only accepts 1 connection, and the accept queue length
> +is 1. On current linux implementation, set queue length to n means the
the current Linux
> +actual queue length is n+1. Now we create 3 connections, 1 is accepted
> +by nc, 2 in accepted queue, so the accept queue is full.
> +
> +Before running the 4th nc, we clean the nstat history on the server:
> +
> + nstatuser@nstat-b:~$ nstat -n
> +
> +Run the 4th nc on the client:
> +
> + nstatuser@nstat-a:~$ nc -v nstat-b 9000
> +
> +If the nc server is running on kernel 4.15 or higher version, you
> +won't see the "Connection to ... succeeded!" string, because kernel
the kernel
> +will drop the syn if the accept queue is full. If the nc client is running
SYN
> +on an old kernel, you could see that the connection is succeeded,
older that the connection succeeded,
> +because kernel would complete the 3-way handshake and keep the socket
the kernel
> +on the half-open queue.
> +
> +Our test is on kernel 4.15, run nstat on the server:
4.15. Run nstat on the server:
> +
> + nstatuser@nstat-b:~$ nstat
> + #kernel
> + IpInReceives 4 0.0
> + IpInDelivers 4 0.0
> + TcpInSegs 4 0.0
> + TcpExtListenOverflows 4 0.0
> + TcpExtListenDrops 4 0.0
> + IpExtInOctets 240 0.0
> + IpExtInNoECTPkts 4 0.0
> +
> +We can see both TcpExtListenOverflows and TcpExtListenDrops are 4. If
> +the time between the 4th nc and the nstat is longer, the value of
> +TcpExtListenOverflows and TcpExtListenDrops will be larger, because
> +the syn of the 4th nc is dropped, it keeps retrying.
SYN
That last sentence needs to be split up some to be more clear.
Use one or more periods ('.') and not so many commas.
--
~Randy
^ permalink raw reply
* Re: [PATCH net v2] net: sched: cls_flower: validate nested enc_opts_policy to avoid warning
From: David Miller @ 2018-11-10 17:56 UTC (permalink / raw)
To: jakub.kicinski; +Cc: oss-drivers, netdev
In-Reply-To: <20181110050626.6659-1-jakub.kicinski@netronome.com>
From: Jakub Kicinski <jakub.kicinski@netronome.com>
Date: Fri, 9 Nov 2018 21:06:26 -0800
> TCA_FLOWER_KEY_ENC_OPTS and TCA_FLOWER_KEY_ENC_OPTS_MASK can only
> currently contain further nested attributes, which are parsed by
> hand, so the policy is never actually used resulting in a W=1
> build warning:
>
> net/sched/cls_flower.c:492:1: warning: ‘enc_opts_policy’ defined but not used [-Wunused-const-variable=]
> enc_opts_policy[TCA_FLOWER_KEY_ENC_OPTS_MAX + 1] = {
>
> Add the validation anyway to avoid potential bugs when other
> attributes are added and to make the attribute structure slightly
> more clear. Validation will also set extact to point to bad
> attribute on error.
>
> Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
> Acked-by: Simon Horman <simon.horman@netronome.com>
Applied and queued up for -stable.
^ permalink raw reply
* [PATCH net-next] OVS: remove VLAN_TAG_PRESENT - fixup
From: Michał Mirosław @ 2018-11-10 18:55 UTC (permalink / raw)
To: netdev; +Cc: Pravin B Shelar, dev
It turns out I missed one VLAN_TAG_PRESENT in OVS code while rebasing.
This fixes it.
Fixes: 9df46aefafa6 ("OVS: remove use of VLAN_TAG_PRESENT")
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
net/openvswitch/flow.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/net/openvswitch/flow.c b/net/openvswitch/flow.c
index fa393815991e..57e07768c9d1 100644
--- a/net/openvswitch/flow.c
+++ b/net/openvswitch/flow.c
@@ -597,7 +597,7 @@ static int key_extract(struct sk_buff *skb, struct sw_flow_key *key)
* skb_vlan_pop(), which will later shift the ethertype into
* skb->protocol.
*/
- if (key->eth.cvlan.tci & htons(VLAN_TAG_PRESENT))
+ if (key->eth.cvlan.tci & htons(VLAN_CFI_MASK))
skb->protocol = key->eth.cvlan.tpid;
else
skb->protocol = key->eth.type;
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 1/6] net/skbuff: add macros for VLAN_PRESENT bit
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: David S. Miller, Alexei Starovoitov, Benjamin Herrenschmidt,
Daniel Borkmann, James Hogan, linux-mips, linuxppc-dev,
Michael Ellerman, Paul Burton, Paul Mackerras, Ralf Baechle,
sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Wrap VLAN_PRESENT bit using macro like PKT_TYPE_* and CLONED_*,
as used by BPF code.
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
include/linux/skbuff.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 7dcfb5591dc3..99f38779332c 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -816,6 +816,12 @@ struct sk_buff {
__u32 priority;
int skb_iif;
__u32 hash;
+#define PKT_VLAN_PRESENT_BIT 4 // CFI (12-th bit) in TCI
+#ifdef __BIG_ENDIAN
+#define PKT_VLAN_PRESENT_OFFSET() offsetof(struct sk_buff, vlan_tci)
+#else
+#define PKT_VLAN_PRESENT_OFFSET() (offsetof(struct sk_buff, vlan_tci) + 1)
+#endif
__be16 vlan_proto;
__u16 vlan_tci;
#if defined(CONFIG_NET_RX_BUSY_POLL) || defined(CONFIG_XPS)
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 0/6] Remove VLAN.CFI overload
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Alexei Starovoitov, Benjamin Herrenschmidt, Daniel Borkmann,
David S. Miller, James Hogan, linux-mips, linuxppc-dev,
Michael Ellerman, Paul Burton, Paul Mackerras, Ralf Baechle,
sparclinux
Fix BPF code/JITs to allow for separate VLAN_PRESENT flag
storage and finally move the flag to separate storage in skbuff.
This is final step to make CLAN.CFI transparent to core Linux
networking stack.
An #ifdef is introduced temporarily to mark fragments masking
VLAN_TAG_PRESENT. This is removed altogether in the final patch.
---
Michał Mirosław (6):
net/skbuff: add macros for VLAN_PRESENT bit
net/bpf: split VLAN_PRESENT bit handling from VLAN_TCI
net/bpf_jit: MIPS: split VLAN_PRESENT bit handling from VLAN_TCI
net/bpf_jit: PPC: split VLAN_PRESENT bit handling from VLAN_TCI
net/bpf_jit: SPARC: split VLAN_PRESENT bit handling from VLAN_TCI
net: remove VLAN_TAG_PRESENT
arch/mips/net/bpf_jit.c | 18 ++++++++---------
arch/powerpc/net/bpf_jit_comp.c | 15 +++++++-------
arch/sparc/net/bpf_jit_comp_32.c | 13 ++++++------
include/linux/if_vlan.h | 11 ++++++-----
include/linux/skbuff.h | 10 +++++++++-
lib/test_bpf.c | 14 +++++++------
net/core/filter.c | 34 ++++++++++++++------------------
7 files changed, 60 insertions(+), 55 deletions(-)
--
2.19.1
^ permalink raw reply
* [PATCH net-next 2/6] net/bpf: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Alexei Starovoitov, Daniel Borkmann, Benjamin Herrenschmidt,
David S. Miller, James Hogan, linux-mips, linuxppc-dev,
Michael Ellerman, Paul Burton, Paul Mackerras, Ralf Baechle,
sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
net/core/filter.c | 40 +++++++++++++++++++++-------------------
1 file changed, 21 insertions(+), 19 deletions(-)
diff --git a/net/core/filter.c b/net/core/filter.c
index e521c5ebc7d1..c151b906df53 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -296,22 +296,21 @@ static u32 convert_skb_access(int skb_field, int dst_reg, int src_reg,
break;
case SKF_AD_VLAN_TAG:
- case SKF_AD_VLAN_TAG_PRESENT:
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, vlan_tci) != 2);
- BUILD_BUG_ON(VLAN_TAG_PRESENT != 0x1000);
/* dst_reg = *(u16 *) (src_reg + offsetof(vlan_tci)) */
*insn++ = BPF_LDX_MEM(BPF_H, dst_reg, src_reg,
offsetof(struct sk_buff, vlan_tci));
- if (skb_field == SKF_AD_VLAN_TAG) {
- *insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg,
- ~VLAN_TAG_PRESENT);
- } else {
- /* dst_reg >>= 12 */
- *insn++ = BPF_ALU32_IMM(BPF_RSH, dst_reg, 12);
- /* dst_reg &= 1 */
+#ifdef VLAN_TAG_PRESENT
+ *insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg, ~VLAN_TAG_PRESENT);
+#endif
+ break;
+ case SKF_AD_VLAN_TAG_PRESENT:
+ *insn++ = BPF_LDX_MEM(BPF_B, dst_reg, src_reg, PKT_VLAN_PRESENT_OFFSET());
+ if (PKT_VLAN_PRESENT_BIT)
+ *insn++ = BPF_ALU32_IMM(BPF_RSH, dst_reg, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
*insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg, 1);
- }
break;
}
@@ -6140,19 +6139,22 @@ static u32 bpf_convert_ctx_access(enum bpf_access_type type,
break;
case offsetof(struct __sk_buff, vlan_present):
+ *target_size = 1;
+ *insn++ = BPF_LDX_MEM(BPF_B, si->dst_reg, si->src_reg,
+ PKT_VLAN_PRESENT_OFFSET());
+ if (PKT_VLAN_PRESENT_BIT)
+ *insn++ = BPF_ALU32_IMM(BPF_RSH, si->dst_reg, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
+ *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, 1);
+ break;
+
case offsetof(struct __sk_buff, vlan_tci):
- BUILD_BUG_ON(VLAN_TAG_PRESENT != 0x1000);
-
*insn++ = BPF_LDX_MEM(BPF_H, si->dst_reg, si->src_reg,
bpf_target_off(struct sk_buff, vlan_tci, 2,
target_size));
- if (si->off == offsetof(struct __sk_buff, vlan_tci)) {
- *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg,
- ~VLAN_TAG_PRESENT);
- } else {
- *insn++ = BPF_ALU32_IMM(BPF_RSH, si->dst_reg, 12);
- *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, 1);
- }
+#ifdef VLAN_TAG_PRESENT
+ *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, ~VLAN_TAG_PRESENT);
+#endif
break;
case offsetof(struct __sk_buff, cb[0]) ...
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 4/6] net/bpf_jit: PPC: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
linuxppc-dev, Alexei Starovoitov, Daniel Borkmann,
David S. Miller, James Hogan, linux-mips, Paul Burton,
Ralf Baechle, sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/powerpc/net/bpf_jit_comp.c | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index d5bfe24bb3b5..dc4a2f54e829 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -379,18 +379,20 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
hash));
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
- case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, vlan_tci) != 2);
- BUILD_BUG_ON(VLAN_TAG_PRESENT != 0x1000);
PPC_LHZ_OFFS(r_A, r_skb, offsetof(struct sk_buff,
vlan_tci));
- if (code == (BPF_ANC | SKF_AD_VLAN_TAG)) {
- PPC_ANDI(r_A, r_A, ~VLAN_TAG_PRESENT);
- } else {
- PPC_ANDI(r_A, r_A, VLAN_TAG_PRESENT);
- PPC_SRWI(r_A, r_A, 12);
- }
+#ifdef VLAN_TAG_PRESENT
+ PPC_ANDI(r_A, r_A, ~VLAN_TAG_PRESENT);
+#endif
+ break;
+ case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
+ PPC_LBZ_OFFS(r_A, r_skb, PKT_VLAN_PRESENT_OFFSET());
+ if (PKT_VLAN_PRESENT_BIT)
+ PPC_SRWI(r_A, r_A, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
+ PPC_ANDI(r_A, r_A, 1);
break;
case BPF_ANC | SKF_AD_QUEUE:
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff,
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 3/6] net/bpf_jit: MIPS: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Ralf Baechle, Paul Burton, James Hogan, linux-mips,
Alexei Starovoitov, Benjamin Herrenschmidt, Daniel Borkmann,
David S. Miller, linuxppc-dev, Michael Ellerman, Paul Mackerras,
sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/mips/net/bpf_jit.c | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/arch/mips/net/bpf_jit.c b/arch/mips/net/bpf_jit.c
index 4d8cb9bb8365..de4c6372ad9a 100644
--- a/arch/mips/net/bpf_jit.c
+++ b/arch/mips/net/bpf_jit.c
@@ -1159,19 +1159,22 @@ static int build_body(struct jit_ctx *ctx)
emit_load(r_A, r_skb, off, ctx);
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
- case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
ctx->flags |= SEEN_SKB | SEEN_A;
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff,
vlan_tci) != 2);
off = offsetof(struct sk_buff, vlan_tci);
- emit_half_load_unsigned(r_s0, r_skb, off, ctx);
- if (code == (BPF_ANC | SKF_AD_VLAN_TAG)) {
- emit_andi(r_A, r_s0, (u16)~VLAN_TAG_PRESENT, ctx);
- } else {
- emit_andi(r_A, r_s0, VLAN_TAG_PRESENT, ctx);
- /* return 1 if present */
- emit_sltu(r_A, r_zero, r_A, ctx);
- }
+ emit_half_load_unsigned(r_A, r_skb, off, ctx);
+#ifdef VLAN_TAG_PRESENT
+ emit_andi(r_A, r_A, (u16)~VLAN_TAG_PRESENT, ctx);
+#endif
+ break;
+ case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
+ ctx->flags |= SEEN_SKB | SEEN_A;
+ emit_load_byte(r_A, r_skb, PKT_VLAN_PRESENT_OFFSET(), ctx);
+ if (PKT_VLAN_PRESENT_BIT)
+ emit_srl(r_A, r_A, PKT_VLAN_PRESENT_BIT, ctx);
+ if (PKT_VLAN_PRESENT_BIT < 7)
+ emit_andi(r_A, r_A, 1, ctx);
break;
case BPF_ANC | SKF_AD_PKTTYPE:
ctx->flags |= SEEN_SKB;
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 5/6] net/bpf_jit: SPARC: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: David S. Miller, sparclinux, Alexei Starovoitov,
Benjamin Herrenschmidt, Daniel Borkmann, James Hogan, linux-mips,
linuxppc-dev, Michael Ellerman, Paul Burton, Paul Mackerras,
Ralf Baechle
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/sparc/net/bpf_jit_comp_32.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/arch/sparc/net/bpf_jit_comp_32.c b/arch/sparc/net/bpf_jit_comp_32.c
index a5ff88643d5c..48f3c04dd179 100644
--- a/arch/sparc/net/bpf_jit_comp_32.c
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -552,15 +552,18 @@ void bpf_jit_compile(struct bpf_prog *fp)
emit_skb_load32(hash, r_A);
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
- case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
emit_skb_load16(vlan_tci, r_A);
- if (code != (BPF_ANC | SKF_AD_VLAN_TAG)) {
- emit_alu_K(SRL, 12);
+#ifdef VLAN_TAG_PRESENT
+ emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
+ emit_and(r_A, r_TMP, r_A);
+#endif
+ break;
+ case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
+ __emit_skb_load8(__pkt_vlan_present_offset, r_A);
+ if (PKT_VLAN_PRESENT_BIT)
+ emit_alu_K(SRL, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
emit_andi(r_A, 1, r_A);
- } else {
- emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
- emit_and(r_A, r_TMP, r_A);
- }
break;
case BPF_LD | BPF_W | BPF_LEN:
emit_skb_load32(len, r_A);
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 6/6] net: remove VLAN_TAG_PRESENT
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Alexei Starovoitov, Benjamin Herrenschmidt, Daniel Borkmann,
David S. Miller, James Hogan, linux-mips, linuxppc-dev,
Michael Ellerman, Paul Burton, Paul Mackerras, Ralf Baechle,
sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Replace VLAN_TAG_PRESENT with single bit flag and free up
VLAN.CFI overload. Now VLAN.CFI is visible in networking stack
and can be passed around intact.
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/mips/net/bpf_jit.c | 3 ---
arch/powerpc/net/bpf_jit_comp.c | 3 ---
arch/sparc/net/bpf_jit_comp_32.c | 4 ----
include/linux/if_vlan.h | 11 ++++++-----
include/linux/skbuff.h | 16 +++++++++-------
lib/test_bpf.c | 14 ++++++++------
net/core/filter.c | 6 ------
7 files changed, 23 insertions(+), 34 deletions(-)
diff --git a/arch/mips/net/bpf_jit.c b/arch/mips/net/bpf_jit.c
index de4c6372ad9a..3a0e34f4e615 100644
--- a/arch/mips/net/bpf_jit.c
+++ b/arch/mips/net/bpf_jit.c
@@ -1164,9 +1164,6 @@ static int build_body(struct jit_ctx *ctx)
vlan_tci) != 2);
off = offsetof(struct sk_buff, vlan_tci);
emit_half_load_unsigned(r_A, r_skb, off, ctx);
-#ifdef VLAN_TAG_PRESENT
- emit_andi(r_A, r_A, (u16)~VLAN_TAG_PRESENT, ctx);
-#endif
break;
case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
ctx->flags |= SEEN_SKB | SEEN_A;
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index dc4a2f54e829..91d223cf512b 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -383,9 +383,6 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
PPC_LHZ_OFFS(r_A, r_skb, offsetof(struct sk_buff,
vlan_tci));
-#ifdef VLAN_TAG_PRESENT
- PPC_ANDI(r_A, r_A, ~VLAN_TAG_PRESENT);
-#endif
break;
case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
PPC_LBZ_OFFS(r_A, r_skb, PKT_VLAN_PRESENT_OFFSET());
diff --git a/arch/sparc/net/bpf_jit_comp_32.c b/arch/sparc/net/bpf_jit_comp_32.c
index 48f3c04dd179..84cc8f7f83e9 100644
--- a/arch/sparc/net/bpf_jit_comp_32.c
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -553,10 +553,6 @@ void bpf_jit_compile(struct bpf_prog *fp)
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
emit_skb_load16(vlan_tci, r_A);
-#ifdef VLAN_TAG_PRESENT
- emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
- emit_and(r_A, r_TMP, r_A);
-#endif
break;
case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
__emit_skb_load8(__pkt_vlan_present_offset, r_A);
diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h
index 1be5230921b5..7a541eadf78e 100644
--- a/include/linux/if_vlan.h
+++ b/include/linux/if_vlan.h
@@ -66,7 +66,6 @@ static inline struct vlan_ethhdr *vlan_eth_hdr(const struct sk_buff *skb)
#define VLAN_PRIO_MASK 0xe000 /* Priority Code Point */
#define VLAN_PRIO_SHIFT 13
#define VLAN_CFI_MASK 0x1000 /* Canonical Format Indicator */
-#define VLAN_TAG_PRESENT VLAN_CFI_MASK
#define VLAN_VID_MASK 0x0fff /* VLAN Identifier */
#define VLAN_N_VID 4096
@@ -78,8 +77,8 @@ static inline bool is_vlan_dev(const struct net_device *dev)
return dev->priv_flags & IFF_802_1Q_VLAN;
}
-#define skb_vlan_tag_present(__skb) ((__skb)->vlan_tci & VLAN_TAG_PRESENT)
-#define skb_vlan_tag_get(__skb) ((__skb)->vlan_tci & ~VLAN_TAG_PRESENT)
+#define skb_vlan_tag_present(__skb) ((__skb)->vlan_present)
+#define skb_vlan_tag_get(__skb) ((__skb)->vlan_tci)
#define skb_vlan_tag_get_id(__skb) ((__skb)->vlan_tci & VLAN_VID_MASK)
#define skb_vlan_tag_get_prio(__skb) (((__skb)->vlan_tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT)
@@ -480,7 +479,7 @@ static inline struct sk_buff *vlan_insert_tag_set_proto(struct sk_buff *skb,
*/
static inline void __vlan_hwaccel_clear_tag(struct sk_buff *skb)
{
- skb->vlan_tci = 0;
+ skb->vlan_present = 0;
}
/**
@@ -492,6 +491,7 @@ static inline void __vlan_hwaccel_clear_tag(struct sk_buff *skb)
*/
static inline void __vlan_hwaccel_copy_tag(struct sk_buff *dst, const struct sk_buff *src)
{
+ dst->vlan_present = src->vlan_present;
dst->vlan_proto = src->vlan_proto;
dst->vlan_tci = src->vlan_tci;
}
@@ -526,7 +526,8 @@ static inline void __vlan_hwaccel_put_tag(struct sk_buff *skb,
__be16 vlan_proto, u16 vlan_tci)
{
skb->vlan_proto = vlan_proto;
- skb->vlan_tci = VLAN_TAG_PRESENT | vlan_tci;
+ skb->vlan_tci = vlan_tci;
+ skb->vlan_present = 1;
}
/**
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 99f38779332c..b9aa0d1b21cf 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -777,6 +777,14 @@ struct sk_buff {
__u8 encap_hdr_csum:1;
__u8 csum_valid:1;
+#ifdef __BIG_ENDIAN_BITFIELD
+#define PKT_VLAN_PRESENT_BIT 7
+#else
+#define PKT_VLAN_PRESENT_BIT 0
+#endif
+#define PKT_VLAN_PRESENT_OFFSET() offsetof(struct sk_buff, __pkt_vlan_present_offset)
+ __u8 __pkt_vlan_present_offset[0];
+ __u8 vlan_present:1;
__u8 csum_complete_sw:1;
__u8 csum_level:2;
__u8 csum_not_inet:1;
@@ -784,8 +792,8 @@ struct sk_buff {
#ifdef CONFIG_IPV6_NDISC_NODETYPE
__u8 ndisc_nodetype:2;
#endif
+
__u8 ipvs_property:1;
-
__u8 inner_protocol_type:1;
__u8 remcsum_offload:1;
#ifdef CONFIG_NET_SWITCHDEV
@@ -816,12 +824,6 @@ struct sk_buff {
__u32 priority;
int skb_iif;
__u32 hash;
-#define PKT_VLAN_PRESENT_BIT 4 // CFI (12-th bit) in TCI
-#ifdef __BIG_ENDIAN
-#define PKT_VLAN_PRESENT_OFFSET() offsetof(struct sk_buff, vlan_tci)
-#else
-#define PKT_VLAN_PRESENT_OFFSET() (offsetof(struct sk_buff, vlan_tci) + 1)
-#endif
__be16 vlan_proto;
__u16 vlan_tci;
#if defined(CONFIG_NET_RX_BUSY_POLL) || defined(CONFIG_XPS)
diff --git a/lib/test_bpf.c b/lib/test_bpf.c
index aa22bcaec1dc..f3e570722a7e 100644
--- a/lib/test_bpf.c
+++ b/lib/test_bpf.c
@@ -39,6 +39,7 @@
#define SKB_HASH 0x1234aaab
#define SKB_QUEUE_MAP 123
#define SKB_VLAN_TCI 0xffff
+#define SKB_VLAN_PRESENT 1
#define SKB_DEV_IFINDEX 577
#define SKB_DEV_TYPE 588
@@ -725,8 +726,8 @@ static struct bpf_test tests[] = {
CLASSIC,
{ },
{
- { 1, SKB_VLAN_TCI & ~VLAN_TAG_PRESENT },
- { 10, SKB_VLAN_TCI & ~VLAN_TAG_PRESENT }
+ { 1, SKB_VLAN_TCI },
+ { 10, SKB_VLAN_TCI }
},
},
{
@@ -739,8 +740,8 @@ static struct bpf_test tests[] = {
CLASSIC,
{ },
{
- { 1, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) },
- { 10, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) }
+ { 1, SKB_VLAN_PRESENT },
+ { 10, SKB_VLAN_PRESENT }
},
},
{
@@ -5289,8 +5290,8 @@ static struct bpf_test tests[] = {
#endif
{ },
{
- { 1, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) },
- { 10, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) }
+ { 1, SKB_VLAN_PRESENT },
+ { 10, SKB_VLAN_PRESENT }
},
.fill_helper = bpf_fill_maxinsns6,
.expected_errcode = -ENOTSUPP,
@@ -6493,6 +6494,7 @@ static struct sk_buff *populate_skb(char *buf, int size)
skb->hash = SKB_HASH;
skb->queue_mapping = SKB_QUEUE_MAP;
skb->vlan_tci = SKB_VLAN_TCI;
+ skb->vlan_present = SKB_VLAN_PRESENT;
skb->vlan_proto = htons(ETH_P_IP);
dev_net_set(&dev, &init_net);
skb->dev = &dev;
diff --git a/net/core/filter.c b/net/core/filter.c
index c151b906df53..10acbc00ff6c 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -301,9 +301,6 @@ static u32 convert_skb_access(int skb_field, int dst_reg, int src_reg,
/* dst_reg = *(u16 *) (src_reg + offsetof(vlan_tci)) */
*insn++ = BPF_LDX_MEM(BPF_H, dst_reg, src_reg,
offsetof(struct sk_buff, vlan_tci));
-#ifdef VLAN_TAG_PRESENT
- *insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg, ~VLAN_TAG_PRESENT);
-#endif
break;
case SKF_AD_VLAN_TAG_PRESENT:
*insn++ = BPF_LDX_MEM(BPF_B, dst_reg, src_reg, PKT_VLAN_PRESENT_OFFSET());
@@ -6152,9 +6149,6 @@ static u32 bpf_convert_ctx_access(enum bpf_access_type type,
*insn++ = BPF_LDX_MEM(BPF_H, si->dst_reg, si->src_reg,
bpf_target_off(struct sk_buff, vlan_tci, 2,
target_size));
-#ifdef VLAN_TAG_PRESENT
- *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, ~VLAN_TAG_PRESENT);
-#endif
break;
case offsetof(struct __sk_buff, cb[0]) ...
--
2.19.1
^ permalink raw reply related
* Re: [PATCH bpf-next 0/3] bpf: Allow narrow loads with offset > 0
From: Alexei Starovoitov @ 2018-11-10 19:13 UTC (permalink / raw)
To: Andrey Ignatov; +Cc: netdev, ast, daniel, yhs, kernel-team
In-Reply-To: <cover.1541553332.git.rdna@fb.com>
On Tue, Nov 06, 2018 at 05:23:25PM -0800, Andrey Ignatov wrote:
> This patch set adds support for narrow loads with offset > 0 to BPF
> verifier.
>
> Patch 1 provides more details and is the main patch in the set.
> Patches 2 and 3 add new test cases to test_verifier and test_sock_addr
> selftests.
Applied, Thanks
^ permalink raw reply
* [net-next 1/1] tipc: improve broadcast retransmission algorithm
From: Jon Maloy @ 2018-11-10 19:23 UTC (permalink / raw)
To: netdev, davem
Cc: gordan.mihaljevic, tung.q.nguyen, hoang.h.le, jon.maloy, maloy,
xinl, ying.xue, tipc-discussion, LUU Duc Canh
From: LUU Duc Canh <canh.d.luu@dektech.com.au>
Currently, the broadcast retransmission algorithm is using the
'prev_retr' field in struct tipc_link to time stamp the latest broadcast
retransmission occasion. This helps to restrict retransmission of
individual broadcast packets to max once per 10 milliseconds, even
though all other criteria for retransmission are met.
We now move this time stamp to the control block of each individual
packet, and remove other limiting criteria. This simplifies the
retransmission algorithm, and eliminates any risk of logical errors
in selecting which packets can be retransmitted.
Acked-by: Ying Xue <ying.xue@windriver.com>
Signed-off-by: LUU Duc Canh <canh.d.luu@dektech.com.au>
Signed-off-by: Jon Maloy <jon.maloy@ericsson.com>
---
net/tipc/link.c | 59 +++++++++++++--------------------------------------------
net/tipc/msg.h | 1 +
2 files changed, 14 insertions(+), 46 deletions(-)
diff --git a/net/tipc/link.c b/net/tipc/link.c
index 201c3b5..aefb5b4 100644
--- a/net/tipc/link.c
+++ b/net/tipc/link.c
@@ -105,7 +105,7 @@ struct tipc_stats {
* @transmitq: queue for sent, non-acked messages
* @backlogq: queue for messages waiting to be sent
* @snt_nxt: next sequence number to use for outbound messages
- * @last_retransmitted: sequence number of most recently retransmitted message
+ * @prev_from: sequence number of most previous retransmission request
* @stale_cnt: counter for number of identical retransmit attempts
* @stale_limit: time when repeated identical retransmits must force link reset
* @ackers: # of peers that needs to ack each packet before it can be released
@@ -163,7 +163,7 @@ struct tipc_link {
u16 limit;
} backlog[5];
u16 snd_nxt;
- u16 last_retransm;
+ u16 prev_from;
u16 window;
u16 stale_cnt;
unsigned long stale_limit;
@@ -186,9 +186,6 @@ struct tipc_link {
u16 acked;
struct tipc_link *bc_rcvlink;
struct tipc_link *bc_sndlink;
- unsigned long prev_retr;
- u16 prev_from;
- u16 prev_to;
u8 nack_state;
bool bc_peer_is_up;
@@ -210,7 +207,7 @@ enum {
BC_NACK_SND_SUPPRESS,
};
-#define TIPC_BC_RETR_LIMIT 10 /* [ms] */
+#define TIPC_BC_RETR_LIM msecs_to_jiffies(10) /* [ms] */
/*
* Interval between NACKs when packets arrive out of order
@@ -1036,10 +1033,12 @@ static int tipc_link_retrans(struct tipc_link *l, struct tipc_link *r,
if (!skb)
return 0;
+ if (less(to, from))
+ return 0;
/* Detect repeated retransmit failures on same packet */
- if (r->last_retransm != buf_seqno(skb)) {
- r->last_retransm = buf_seqno(skb);
+ if (r->prev_from != from) {
+ r->prev_from = from;
r->stale_limit = jiffies + msecs_to_jiffies(r->tolerance);
r->stale_cnt = 0;
} else if (++r->stale_cnt > 99 && time_after(jiffies, r->stale_limit)) {
@@ -1055,6 +1054,11 @@ static int tipc_link_retrans(struct tipc_link *l, struct tipc_link *r,
continue;
if (more(msg_seqno(hdr), to))
break;
+ if (link_is_bc_sndlink(l)) {
+ if (time_before(jiffies, TIPC_SKB_CB(skb)->nxt_retr))
+ continue;
+ TIPC_SKB_CB(skb)->nxt_retr = jiffies + TIPC_BC_RETR_LIM;
+ }
_skb = __pskb_copy(skb, MIN_H_SIZE, GFP_ATOMIC);
if (!_skb)
return 0;
@@ -1734,42 +1738,6 @@ void tipc_link_bc_init_rcv(struct tipc_link *l, struct tipc_msg *hdr)
l->rcv_nxt = peers_snd_nxt;
}
-/* link_bc_retr eval()- check if the indicated range can be retransmitted now
- * - Adjust permitted range if there is overlap with previous retransmission
- */
-static bool link_bc_retr_eval(struct tipc_link *l, u16 *from, u16 *to)
-{
- unsigned long elapsed = jiffies_to_msecs(jiffies - l->prev_retr);
-
- if (less(*to, *from))
- return false;
-
- /* New retransmission request */
- if ((elapsed > TIPC_BC_RETR_LIMIT) ||
- less(*to, l->prev_from) || more(*from, l->prev_to)) {
- l->prev_from = *from;
- l->prev_to = *to;
- l->prev_retr = jiffies;
- return true;
- }
-
- /* Inside range of previous retransmit */
- if (!less(*from, l->prev_from) && !more(*to, l->prev_to))
- return false;
-
- /* Fully or partially outside previous range => exclude overlap */
- if (less(*from, l->prev_from)) {
- *to = l->prev_from - 1;
- l->prev_from = *from;
- }
- if (more(*to, l->prev_to)) {
- *from = l->prev_to + 1;
- l->prev_to = *to;
- }
- l->prev_retr = jiffies;
- return true;
-}
-
/* tipc_link_bc_sync_rcv - update rcv link according to peer's send state
*/
int tipc_link_bc_sync_rcv(struct tipc_link *l, struct tipc_msg *hdr,
@@ -1800,8 +1768,7 @@ int tipc_link_bc_sync_rcv(struct tipc_link *l, struct tipc_msg *hdr,
if (more(peers_snd_nxt, l->rcv_nxt + l->window))
return rc;
- if (link_bc_retr_eval(snd_l, &from, &to))
- rc = tipc_link_retrans(snd_l, l, from, to, xmitq);
+ rc = tipc_link_retrans(snd_l, l, from, to, xmitq);
l->snd_nxt = peers_snd_nxt;
if (link_bc_rcv_gap(l))
diff --git a/net/tipc/msg.h b/net/tipc/msg.h
index a2879e6..a092495 100644
--- a/net/tipc/msg.h
+++ b/net/tipc/msg.h
@@ -105,6 +105,7 @@ struct tipc_skb_cb {
u32 bytes_read;
u32 orig_member;
struct sk_buff *tail;
+ unsigned long nxt_retr;
bool validated;
u16 chain_imp;
u16 ackers;
--
2.1.4
^ permalink raw reply related
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Jesper Dangaard Brouer @ 2018-11-10 19:34 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev@vger.kernel.org, brouer
In-Reply-To: <e6ece370-bf62-5d36-0417-779d2345fc8d@itcare.pl>
On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> > CPU load is lower than for connectx4 - but it looks like bandwidth
> > limit is the same :)
> > But also after reaching 60Gbit/60Gbit
> >
> > bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> > input: /proc/net/dev type: rate
> > - iface Rx Tx Total
> > ==========================================================================
> >
> > enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
> > enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
> > --------------------------------------------------------------------------
> >
> > total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>
> Today reached 65/65Gbit/s
>
> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
This is weird!
How do you see / measure these drops?
> So checked other stats.
> softnet_stats shows average 1k squeezed per sec:
Is below output the raw counters? not per sec?
It would be valuable to see the per sec stats instead...
I use this tool:
https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
> cpu total dropped squeezed collision rps flow_limit
> 0 18554 0 1 0 0 0
> 1 16728 0 1 0 0 0
> 2 18033 0 1 0 0 0
> 3 17757 0 1 0 0 0
> 4 18861 0 0 0 0 0
> 5 0 0 1 0 0 0
> 6 2 0 1 0 0 0
> 7 0 0 1 0 0 0
> 8 0 0 0 0 0 0
> 9 0 0 1 0 0 0
> 10 0 0 0 0 0 0
> 11 0 0 1 0 0 0
> 12 50 0 1 0 0 0
> 13 257 0 0 0 0 0
> 14 3629115363 0 3353259 0 0 0
> 15 255167835 0 3138271 0 0 0
> 16 4240101961 0 3036130 0 0 0
> 17 599810018 0 3072169 0 0 0
> 18 432796524 0 3034191 0 0 0
> 19 41803906 0 3037405 0 0 0
> 20 900382666 0 3112294 0 0 0
> 21 620926085 0 3086009 0 0 0
> 22 41861198 0 3023142 0 0 0
> 23 4090425574 0 2990412 0 0 0
> 24 4264870218 0 3010272 0 0 0
> 25 141401811 0 3027153 0 0 0
> 26 104155188 0 3051251 0 0 0
> 27 4261258691 0 3039765 0 0 0
> 28 4 0 1 0 0 0
> 29 4 0 0 0 0 0
> 30 0 0 1 0 0 0
> 31 0 0 0 0 0 0
> 32 3 0 1 0 0 0
> 33 1 0 1 0 0 0
> 34 0 0 1 0 0 0
> 35 0 0 0 0 0 0
> 36 0 0 1 0 0 0
> 37 0 0 1 0 0 0
> 38 0 0 1 0 0 0
> 39 0 0 1 0 0 0
> 40 0 0 0 0 0 0
> 41 0 0 1 0 0 0
> 42 299758202 0 3139693 0 0 0
> 43 4254727979 0 3103577 0 0 0
> 44 1959555543 0 2554885 0 0 0
> 45 1675702723 0 2513481 0 0 0
> 46 1908435503 0 2519698 0 0 0
> 47 1877799710 0 2537768 0 0 0
> 48 2384274076 0 2584673 0 0 0
> 49 2598104878 0 2593616 0 0 0
> 50 1897566829 0 2530857 0 0 0
> 51 1712741629 0 2489089 0 0 0
> 52 1704033648 0 2495892 0 0 0
> 53 1636781820 0 2499783 0 0 0
> 54 1861997734 0 2541060 0 0 0
> 55 2113521616 0 2555673 0 0 0
>
>
> So i rised netdev backlog and budged to rly high values
> 524288 for netdev_budget and same for backlog
Does it affect the squeezed counters?
Notice, this (crazy) huge netdev_budget limit will also be limited
by /proc/sys/net/core/netdev_budget_usecs.
> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
Hmmm, this could indicated not enough NAPI bulking is occurring.
I have a BPF tool, that can give you some insight into NAPI bulking and
softirq idle/kthread starting. Called 'napi_monitor', could you try to
run this, so can try to understand this? You find the tool here:
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
> But after this changes i have less packets drops.
>
>
> Below perf top from max traffic reached:
> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ------------------------------------------------------------------------------------------
>
> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
> 8.44% [kernel] [k] mlx5e_sq_xmit
> 6.69% [kernel] [k] build_skb
> 5.21% [kernel] [k] fib_table_lookup
> 3.54% [kernel] [k] memcpy_erms
> 3.20% [kernel] [k] mlx5e_poll_rx_cq
> 2.25% [kernel] [k] vlan_do_receive
> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
> 1.95% [kernel] [k] __dev_queue_xmit
> 1.83% [kernel] [k] dev_gro_receive
> 1.79% [kernel] [k] tcp_gro_receive
> 1.73% [kernel] [k] ip_finish_output2
> 1.63% [kernel] [k] mlx5e_poll_tx_cq
> 1.49% [kernel] [k] ipt_do_table
> 1.38% [kernel] [k] inet_gro_receive
> 1.31% [kernel] [k] __netif_receive_skb_core
> 1.30% [kernel] [k] _raw_spin_lock
> 1.28% [kernel] [k] mlx5_eq_int
> 1.24% [kernel] [k] irq_entries_start
> 1.19% [kernel] [k] __build_skb
> 1.15% [kernel] [k] swiotlb_map_page
> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
> 0.94% [kernel] [k] pfifo_fast_dequeue
> 0.92% [kernel] [k] ip_route_input_rcu
> 0.86% [kernel] [k] kmem_cache_alloc
> 0.80% [kernel] [k] mlx5e_xmit
> 0.79% [kernel] [k] dev_hard_start_xmit
> 0.78% [kernel] [k] _raw_spin_lock_irqsave
> 0.74% [kernel] [k] ip_forward
> 0.72% [kernel] [k] tasklet_action_common.isra.21
> 0.68% [kernel] [k] pfifo_fast_enqueue
> 0.67% [kernel] [k] netif_skb_features
> 0.66% [kernel] [k] skb_segment
> 0.60% [kernel] [k] skb_gro_receive
> 0.56% [kernel] [k] validate_xmit_skb.isra.142
> 0.53% [kernel] [k] skb_release_data
> 0.51% [kernel] [k] mlx5e_page_release
> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
> 0.51% [kernel] [k] __qdisc_run
> 0.50% [kernel] [k] tcp4_gro_receive
> 0.49% [kernel] [k] page_frag_free
> 0.46% [kernel] [k] kmem_cache_free_bulk
> 0.43% [kernel] [k] kmem_cache_free
> 0.42% [kernel] [k] try_to_wake_up
> 0.39% [kernel] [k] _raw_spin_lock_irq
> 0.39% [kernel] [k] find_busiest_group
> 0.37% [kernel] [k] __memcpy
>
>
>
> Remember those tests are now on two separate connectx5 connected to
> two separate pcie x16 gen 3.0
That is strange... I still suspect some HW NIC issue, can you provide
ethtool stats info via tool:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
$ ethtool_stats.pl --dev enp175s0 --dev enp216s0
The tool remove zero-stats counters and report per sec stats. It makes
it easier to spot that is relevant for the given workload.
Can you give output put from:
$ ethtool --show-priv-flag DEVICE
I want you to experiment with:
ethtool --set-priv-flags DEVICE rx_striding_rq off
I think you already have played with 'rx_cqe_compress', right.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* [BUG] xfrm: unable to handle kernel NULL pointer dereference
From: Jean-Philippe Menil @ 2018-11-10 19:34 UTC (permalink / raw)
To: steffen.klassert; +Cc: herbert, davem, netdev, kuznet, yoshfuji
[-- Attachment #1: Type: text/plain, Size: 3005 bytes --]
Hi guys,
we're seeing unexpected crashes from kernel 4.15 to 4.18.17, using IPsec
VTI interfaces, on several vpn hosts, since upgrade from 4.4.
Attached, the offended oops against 4.18.
Output of decodedecode:
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00
00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe
<f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
All code
========
0: 8b 44 24 70 mov 0x70(%rsp),%eax
4: 0f c8 bswap %eax
6: 89 87 b4 00 00 00 mov %eax,0xb4(%rdi)
c: 48 8b 86 20 05 00 00 mov 0x520(%rsi),%rax
13: 8b 80 f8 14 00 00 mov 0x14f8(%rax),%eax
19: 85 c0 test %eax,%eax
1b: 75 05 jne 0x22
1d: 48 85 d2 test %rdx,%rdx
20: 74 0e je 0x30
22: 48 8b 43 58 mov 0x58(%rbx),%rax
26: 48 83 e0 fe and $0xfffffffffffffffe,%rax
2a:* f6 40 38 04 testb $0x4,0x38(%rax) <--
trapping instruction
2e: 74 7d je 0xad
30: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
37: 49 8b 44 24 20 mov 0x20(%r12),%rax
3c: 48 rex.W
3d: 39 .byte 0x39
3e: 86 20 xchg %ah,(%rax)
Code starting with the faulting instruction
===========================================
0: f6 40 38 04 testb $0x4,0x38(%rax)
4: 74 7d je 0x83
6: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
d: 49 8b 44 24 20 mov 0x20(%r12),%rax
12: 48 rex.W
13: 39 .byte 0x39
14: 86 20 xchg %ah,(%rax)
if my understanding is correct, we fail here:
/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
0x0000000000000b19 <+185>: testb $0x4,0x38(%rax)
0x0000000000000b1d <+189>: je 0xb9c <vti_rcv_cb+316>
(gdb) list *0x0000000000000b19
0xb19 is in vti_rcv_cb
(/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:1169).
1164 int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
1165
1166 if (sk && sk->sk_policy[XFRM_POLICY_IN])
1167 return __xfrm_policy_check(sk, ndir, skb, family);
1168
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
1170 (skb_dst(skb)->flags & DST_NOPOLICY) ||
1171 __xfrm_policy_check(sk, ndir, skb, family);
1172 }
1173
I really have hard time to understand why skb seem to be freed twice.
I'm not able to repeat the bug in lab, but it happened regulary in prod,
seem to depend of the workload.
Any help will be appreciated.
Let me know if you need further informations.
Regards,
Jean-Philippe
[-- Attachment #2: oops.txt --]
[-- Type: text/plain, Size: 14750 bytes --]
[ 31.154360] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
[ 31.162233] PGD 0 P4D 0
[ 31.164786] Oops: 0000 [#1] SMP PTI
[ 31.168291] CPU: 5 PID: 42 Comm: ksoftirqd/5 Not tainted 4.18.0-11-generic #12~18.04.1-Ubuntu
[ 31.176854] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016
[ 31.184980] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.189962] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.208916] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.214160] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.221328] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.228485] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.235643] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.242803] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.249964] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 31.258077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.263848] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.271004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.278163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.285320] Call Trace:
[ 31.287789] xfrm4_rcv_cb+0x4a/0x70
[ 31.291297] xfrm_input+0x58f/0x8f0
[ 31.294807] vti_input+0xaa/0x110 [ip_vti]
[ 31.298926] vti_rcv+0x33/0x3c [ip_vti]
[ 31.302783] xfrm4_esp_rcv+0x39/0x50
[ 31.306375] ip_local_deliver_finish+0x62/0x200
[ 31.310923] ip_local_deliver+0xdf/0xf0
[ 31.314775] ? ip_rcv_finish+0x420/0x420
[ 31.318718] ip_rcv_finish+0x126/0x420
[ 31.322486] ip_rcv+0x28f/0x360
[ 31.325655] ? inet_del_offload+0x40/0x40
[ 31.329686] __netif_receive_skb_core+0x48c/0xb70
[ 31.334413] ? kmem_cache_alloc+0xb4/0x1d0
[ 31.338532] ? __build_skb+0x2b/0xf0
[ 31.342128] __netif_receive_skb+0x18/0x60
[ 31.346244] ? __netif_receive_skb+0x18/0x60
[ 31.350536] netif_receive_skb_internal+0x45/0xe0
[ 31.355263] napi_gro_receive+0xc5/0xf0
[ 31.359141] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 31.364476] ? skb_release_all+0x24/0x30
[ 31.368430] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 31.373432] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 31.378333] ? __switch_to_asm+0x34/0x70
[ 31.382270] ? __switch_to_asm+0x40/0x70
[ 31.386214] ? __switch_to_asm+0x34/0x70
[ 31.391056] ? __switch_to_asm+0x40/0x70
[ 31.395905] ? __switch_to_asm+0x34/0x70
[ 31.400743] net_rx_action+0x140/0x3a0
[ 31.405379] ? __switch_to+0xad/0x500
[ 31.409887] __do_softirq+0xe4/0x2bb
[ 31.414448] run_ksoftirqd+0x2b/0x40
[ 31.418862] smpboot_thread_fn+0xfc/0x170
[ 31.423700] kthread+0x121/0x140
[ 31.427701] ? sort_range+0x30/0x30
[ 31.432040] ? kthread_create_worker_on_cpu+0x70/0x70
[ 31.437816] ret_from_fork+0x35/0x40
[ 31.442219] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev mei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel
[ 31.519488] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 31.578877] CR2: 0000000000000038
[ 31.583249] ---[ end trace c4bada38847a0075 ]---
[ 31.737166] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.737167] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.737209] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.737212] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.737213] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.737216] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.737219] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.737220] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.737222] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 31.737224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.737225] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.737227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.737228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.737230] Kernel panic - not syncing: Fatal exception in interrupt
[ 31.737264] Kernel Offset: 0x3c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 36.558279] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[ 36.566769] ------------[ cut here ]------------
[ 36.572306] sched: Unexpected reschedule of offline CPU#3!
[ 36.578714] WARNING: CPU: 5 PID: 42 at /build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/arch/x86/kernel/smp.c:128 native_smp_send_resched
ule+0x3a/0x40
[ 36.594262] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack
_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev m
ei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel
[ 36.673307] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compres
s raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x
86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 36.733827] CPU: 5 PID: 42 Comm: ksoftirqd/5 Tainted: G D 4.18.0-11-generic #12~18.04.1-Ubuntu
[ 36.745027] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016
[ 36.754392] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 36.761143] Code: c6 62 01 73 17 48 8b 05 24 d2 17 01 be fd 00 00 00 48 8b 40 30 e8 96 96 ba 00 5d c3 89 fe 48 c7 c7 90 2a cd 85 e8 26 4a 0
3 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 20 65 48
[ 36.782546] RSP: 0018:ffff9a35bfd43b68 EFLAGS: 00010082
[ 36.789019] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000006
[ 36.797379] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9a35bfd564b0
[ 36.805721] RBP: ffff9a35bfd43b68 R08: 00000000000004b7 R09: 0000000000cdcdcd
[ 36.814046] R10: 0000000000000324 R11: 00000000ffffffff R12: ffff9a35bfce2c40
[ 36.822350] R13: ffff9a3598e35c00 R14: 0000000000000008 R15: 0000000000000003
[ 36.830637] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 36.839869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.846742] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 36.855009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 36.863275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 36.871510] Call Trace:
[ 36.875048] <IRQ>
[ 36.878136] resched_curr+0x5d/0xc0
[ 36.882689] check_preempt_wakeup+0x130/0x240
[ 36.888116] check_preempt_curr+0x2d/0x90
[ 36.893184] ttwu_do_wakeup+0x1e/0x140
[ 36.897984] ttwu_do_activate+0x77/0x80
[ 36.902857] try_to_wake_up+0x1d6/0x450
[ 36.907726] ? __netif_receive_skb_core+0x48c/0xb70
[ 36.913625] default_wake_function+0x12/0x20
[ 36.918914] __wake_up_common+0x73/0x130
[ 36.923851] __wake_up_locked+0x16/0x20
[ 36.928706] ep_poll_callback+0xcb/0x2b0
[ 36.933634] __wake_up_common+0x73/0x130
[ 36.938549] __wake_up_common_lock+0x80/0xc0
[ 36.943787] ? tick_sched_do_timer+0x60/0x60
[ 36.949011] __wake_up+0x13/0x20
[ 36.953190] wake_up_klogd_work_func+0x40/0x60
[ 36.958560] irq_work_run_list+0x52/0x80
[ 36.963388] irq_work_tick+0x3b/0x50
[ 36.967841] update_process_times+0x42/0x60
[ 36.972883] tick_sched_handle+0x25/0x70
[ 36.977644] tick_sched_timer+0x3c/0x80
[ 36.982285] __hrtimer_run_queues+0x10f/0x280
[ 36.987429] hrtimer_interrupt+0xe7/0x240
[ 36.992202] ? rcu_irq_exit+0x1d/0x20
[ 36.996602] smp_apic_timer_interrupt+0x6f/0x130
[ 37.001929] apic_timer_interrupt+0xf/0x20
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 0
0 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 00 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.073287] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000000
[ 37.080999] ? panic+0x1f7/0x244
[ 37.084812] oops_end+0xce/0xe0
[ 37.088531] no_context+0x17c/0x400
[ 37.092585] ? skcipher_walk_first+0x4c/0x110
[ 37.097500] __bad_area_nosemaphore+0x115/0x1d0
[ 37.102591] ? cbc_decrypt+0xb1/0xe0 [aesni_intel]
[ 37.107926] bad_area_nosemaphore+0x14/0x20
[ 37.112654] __do_page_fault+0xd4/0x4d0
[ 37.117030] do_page_fault+0x2d/0xf0
[ 37.121143] ? skb_copy_bits+0x61/0x260
[ 37.125511] page_fault+0x1e/0x30
[ 37.129362] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 37.154892] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 37.160711] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 37.168451] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 37.176196] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 37.183950] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 37.191716] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 37.199481] ? esp_input_done2+0x6b/0x320 [esp4]
[ 37.204740] xfrm4_rcv_cb+0x4a/0x70
[ 37.208873] xfrm_input+0x58f/0x8f0
[ 37.212996] vti_input+0xaa/0x110 [ip_vti]
[ 37.217735] vti_rcv+0x33/0x3c [ip_vti]
[ 37.222209] xfrm4_esp_rcv+0x39/0x50
[ 37.226424] ip_local_deliver_finish+0x62/0x200
[ 37.231599] ip_local_deliver+0xdf/0xf0
[ 37.236072] ? ip_rcv_finish+0x420/0x420
[ 37.240635] ip_rcv_finish+0x126/0x420
[ 37.245025] ip_rcv+0x28f/0x360
[ 37.248801] ? inet_del_offload+0x40/0x40
[ 37.253457] __netif_receive_skb_core+0x48c/0xb70
[ 37.258807] ? kmem_cache_alloc+0xb4/0x1d0
[ 37.263541] ? __build_skb+0x2b/0xf0
[ 37.267749] __netif_receive_skb+0x18/0x60
[ 37.272483] ? __netif_receive_skb+0x18/0x60
[ 37.277391] netif_receive_skb_internal+0x45/0xe0
[ 37.282732] napi_gro_receive+0xc5/0xf0
[ 37.287232] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 37.293170] ? skb_release_all+0x24/0x30
[ 37.297708] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 37.303281] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 37.308752] ? __switch_to_asm+0x34/0x70
[ 37.313266] ? __switch_to_asm+0x40/0x70
[ 37.317768] ? __switch_to_asm+0x34/0x70
[ 37.322259] ? __switch_to_asm+0x40/0x70
[ 37.326742] ? __switch_to_asm+0x34/0x70
[ 37.331214] net_rx_action+0x140/0x3a0
[ 37.335519] ? __switch_to+0xad/0x500
[ 37.339728] __do_softirq+0xe4/0x2bb
[ 37.343852] run_ksoftirqd+0x2b/0x40
[ 37.347971] smpboot_thread_fn+0xfc/0x170
[ 37.352522] kthread+0x121/0x140
[ 37.356282] ? sort_range+0x30/0x30
[ 37.360303] ? kthread_create_worker_on_cpu+0x70/0x70
[ 37.365895] ret_from_fork+0x35/0x40
[ 37.370010] ---[ end trace c4bada38847a0076 ]---
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 19:49 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <20181110203409.482f39ec@redhat.com>
W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>> limit is the same :)
>>> But also after reaching 60Gbit/60Gbit
>>>
>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>> input: /proc/net/dev type: rate
>>> - iface Rx Tx Total
>>> ==========================================================================
>>>
>>> enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
>>> enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
>>> --------------------------------------------------------------------------
>>>
>>> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>> Today reached 65/65Gbit/s
>>
>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
> This is weird!
>
> How do you see / measure these drops?
Simple icmp test like ping -i 0.1
And im testing by icmp management ip address on vlan that is attacked to
one NIC (the side that is more stressed with RX)
And another icmp test is forward thru this router - host behind it
Both measurements shows same loss ratio from 0.1 to 0.5% after reaching
~45Gbit/s RX side - depends how much RX side is pushed drops vary
between 0.1 to 0.5 - even 0.6%:)
>
>
>> So checked other stats.
>> softnet_stats shows average 1k squeezed per sec:
> Is below output the raw counters? not per sec?
>
> It would be valuable to see the per sec stats instead...
> I use this tool:
> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
>
>> cpu total dropped squeezed collision rps flow_limit
>> 0 18554 0 1 0 0 0
>> 1 16728 0 1 0 0 0
>> 2 18033 0 1 0 0 0
>> 3 17757 0 1 0 0 0
>> 4 18861 0 0 0 0 0
>> 5 0 0 1 0 0 0
>> 6 2 0 1 0 0 0
>> 7 0 0 1 0 0 0
>> 8 0 0 0 0 0 0
>> 9 0 0 1 0 0 0
>> 10 0 0 0 0 0 0
>> 11 0 0 1 0 0 0
>> 12 50 0 1 0 0 0
>> 13 257 0 0 0 0 0
>> 14 3629115363 0 3353259 0 0 0
>> 15 255167835 0 3138271 0 0 0
>> 16 4240101961 0 3036130 0 0 0
>> 17 599810018 0 3072169 0 0 0
>> 18 432796524 0 3034191 0 0 0
>> 19 41803906 0 3037405 0 0 0
>> 20 900382666 0 3112294 0 0 0
>> 21 620926085 0 3086009 0 0 0
>> 22 41861198 0 3023142 0 0 0
>> 23 4090425574 0 2990412 0 0 0
>> 24 4264870218 0 3010272 0 0 0
>> 25 141401811 0 3027153 0 0 0
>> 26 104155188 0 3051251 0 0 0
>> 27 4261258691 0 3039765 0 0 0
>> 28 4 0 1 0 0 0
>> 29 4 0 0 0 0 0
>> 30 0 0 1 0 0 0
>> 31 0 0 0 0 0 0
>> 32 3 0 1 0 0 0
>> 33 1 0 1 0 0 0
>> 34 0 0 1 0 0 0
>> 35 0 0 0 0 0 0
>> 36 0 0 1 0 0 0
>> 37 0 0 1 0 0 0
>> 38 0 0 1 0 0 0
>> 39 0 0 1 0 0 0
>> 40 0 0 0 0 0 0
>> 41 0 0 1 0 0 0
>> 42 299758202 0 3139693 0 0 0
>> 43 4254727979 0 3103577 0 0 0
>> 44 1959555543 0 2554885 0 0 0
>> 45 1675702723 0 2513481 0 0 0
>> 46 1908435503 0 2519698 0 0 0
>> 47 1877799710 0 2537768 0 0 0
>> 48 2384274076 0 2584673 0 0 0
>> 49 2598104878 0 2593616 0 0 0
>> 50 1897566829 0 2530857 0 0 0
>> 51 1712741629 0 2489089 0 0 0
>> 52 1704033648 0 2495892 0 0 0
>> 53 1636781820 0 2499783 0 0 0
>> 54 1861997734 0 2541060 0 0 0
>> 55 2113521616 0 2555673 0 0 0
>>
>>
>> So i rised netdev backlog and budged to rly high values
>> 524288 for netdev_budget and same for backlog
> Does it affect the squeezed counters?
a little - but not much
After change budget from 65536 to to 524k - number of squeezed counters
for all cpus changed from 1.5k per second to 0.9-1k per second - but
increasing it more like above 524k change nothing - same 0.9 to 1k/s
squeezed
>
> Notice, this (crazy) huge netdev_budget limit will also be limited
> by /proc/sys/net/core/netdev_budget_usecs.
Yes changed that also to 1000 / 2000 / 3000 / 4000 not much difference
on squeezed - even cant see the difference
>
>> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
> Hmmm, this could indicated not enough NAPI bulking is occurring.
>
> I have a BPF tool, that can give you some insight into NAPI bulking and
> softirq idle/kthread starting. Called 'napi_monitor', could you try to
> run this, so can try to understand this? You find the tool here:
>
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
yes will try it
>
>> But after this changes i have less packets drops.
>>
>>
>> Below perf top from max traffic reached:
>> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> ------------------------------------------------------------------------------------------
>>
>> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
>> 8.44% [kernel] [k] mlx5e_sq_xmit
>> 6.69% [kernel] [k] build_skb
>> 5.21% [kernel] [k] fib_table_lookup
>> 3.54% [kernel] [k] memcpy_erms
>> 3.20% [kernel] [k] mlx5e_poll_rx_cq
>> 2.25% [kernel] [k] vlan_do_receive
>> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
>> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
>> 1.95% [kernel] [k] __dev_queue_xmit
>> 1.83% [kernel] [k] dev_gro_receive
>> 1.79% [kernel] [k] tcp_gro_receive
>> 1.73% [kernel] [k] ip_finish_output2
>> 1.63% [kernel] [k] mlx5e_poll_tx_cq
>> 1.49% [kernel] [k] ipt_do_table
>> 1.38% [kernel] [k] inet_gro_receive
>> 1.31% [kernel] [k] __netif_receive_skb_core
>> 1.30% [kernel] [k] _raw_spin_lock
>> 1.28% [kernel] [k] mlx5_eq_int
>> 1.24% [kernel] [k] irq_entries_start
>> 1.19% [kernel] [k] __build_skb
>> 1.15% [kernel] [k] swiotlb_map_page
>> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.94% [kernel] [k] pfifo_fast_dequeue
>> 0.92% [kernel] [k] ip_route_input_rcu
>> 0.86% [kernel] [k] kmem_cache_alloc
>> 0.80% [kernel] [k] mlx5e_xmit
>> 0.79% [kernel] [k] dev_hard_start_xmit
>> 0.78% [kernel] [k] _raw_spin_lock_irqsave
>> 0.74% [kernel] [k] ip_forward
>> 0.72% [kernel] [k] tasklet_action_common.isra.21
>> 0.68% [kernel] [k] pfifo_fast_enqueue
>> 0.67% [kernel] [k] netif_skb_features
>> 0.66% [kernel] [k] skb_segment
>> 0.60% [kernel] [k] skb_gro_receive
>> 0.56% [kernel] [k] validate_xmit_skb.isra.142
>> 0.53% [kernel] [k] skb_release_data
>> 0.51% [kernel] [k] mlx5e_page_release
>> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>> 0.51% [kernel] [k] __qdisc_run
>> 0.50% [kernel] [k] tcp4_gro_receive
>> 0.49% [kernel] [k] page_frag_free
>> 0.46% [kernel] [k] kmem_cache_free_bulk
>> 0.43% [kernel] [k] kmem_cache_free
>> 0.42% [kernel] [k] try_to_wake_up
>> 0.39% [kernel] [k] _raw_spin_lock_irq
>> 0.39% [kernel] [k] find_busiest_group
>> 0.37% [kernel] [k] __memcpy
>>
>>
>>
>> Remember those tests are now on two separate connectx5 connected to
>> two separate pcie x16 gen 3.0
>
> That is strange... I still suspect some HW NIC issue, can you provide
> ethtool stats info via tool:
>
> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>
> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>
> The tool remove zero-stats counters and report per sec stats. It makes
> it easier to spot that is relevant for the given workload.
yes mlnx have just too many counters that are always 0 for my case :)
Will try this also
>
> Can you give output put from:
> $ ethtool --show-priv-flag DEVICE
>
> I want you to experiment with:
ethtool --show-priv-flags enp175s0
Private flags for enp175s0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : off
rx_striding_rq : on
rx_no_csum_complete: off
>
> ethtool --set-priv-flags DEVICE rx_striding_rq off
ok i will first check on test server if this will reset my interface and
will not produce kernel panic :)
>
> I think you already have played with 'rx_cqe_compress', right.
yes - and compress increasing number of irq's but doing not much for
bandwidth same limit 60-64Gbit/s total RX+TX on one 100G port
And what is weird - that limit is in overall symetric - cause if for
example 100G port is receiving 42G traffic and transmitting 20G traffic
- when i flood rx side with pktgen or other for example icmp traffic
1/2/3/4/5G - then receiving side increase with 1/2/3/4/5Gbit of traffic
but transmitting is going down for same lvl's
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 19:56 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <7037c58d-d77d-bdd5-6c91-19cea3cbe539@itcare.pl>
W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
>
>
> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>>> limit is the same :)
>>>> But also after reaching 60Gbit/60Gbit
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> ==========================================================================
>>>>
>>>>
>>>> enp175s0: 45.09 Gb/s 15.09 Gb/s
>>>> 60.18 Gb/s
>>>> enp216s0: 15.14 Gb/s 45.19 Gb/s
>>>> 60.33 Gb/s
>>>> --------------------------------------------------------------------------
>>>>
>>>>
>>>> total: 60.45 Gb/s 60.48 Gb/s 120.93
>>>> Gb/s
>>> Today reached 65/65Gbit/s
>>>
>>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
>> This is weird!
>>
>> How do you see / measure these drops?
> Simple icmp test like ping -i 0.1
> And im testing by icmp management ip address on vlan that is attacked
> to one NIC (the side that is more stressed with RX)
> And another icmp test is forward thru this router - host behind it
>
> Both measurements shows same loss ratio from 0.1 to 0.5% after
> reaching ~45Gbit/s RX side - depends how much RX side is pushed drops
> vary between 0.1 to 0.5 - even 0.6%:)
>
>
>>
>>
>>> So checked other stats.
>>> softnet_stats shows average 1k squeezed per sec:
>> Is below output the raw counters? not per sec?
>>
>> It would be valuable to see the per sec stats instead...
>> I use this tool:
>> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
CPU total/sec dropped/sec squeezed/sec
collision/sec rx_rps/sec flow_limit/sec
CPU:00 0 0 0 0
0 0
CPU:01 0 0 0 0
0 0
CPU:02 0 0 0 0
0 0
CPU:03 0 0 0 0
0 0
CPU:04 0 0 0 0
0 0
CPU:05 0 0 0 0
0 0
CPU:06 0 0 0 0
0 0
CPU:07 0 0 0 0
0 0
CPU:08 0 0 0 0
0 0
CPU:09 0 0 0 0
0 0
CPU:10 0 0 0 0
0 0
CPU:11 0 0 0 0
0 0
CPU:12 0 0 0 0
0 0
CPU:13 0 0 0 0
0 0
CPU:14 485538 0 43 0
0 0
CPU:15 474794 0 51 0
0 0
CPU:16 449322 0 41 0
0 0
CPU:17 476420 0 46 0
0 0
CPU:18 440436 0 38 0
0 0
CPU:19 501499 0 49 0
0 0
CPU:20 459468 0 49 0
0 0
CPU:21 438928 0 47 0
0 0
CPU:22 468983 0 40 0
0 0
CPU:23 446253 0 47 0
0 0
CPU:24 451909 0 46 0
0 0
CPU:25 479373 0 55 0
0 0
CPU:26 467848 0 49 0
0 0
CPU:27 453153 0 51 0
0 0
CPU:28 0 0 0 0
0 0
CPU:29 0 0 0 0
0 0
CPU:30 0 0 0 0
0 0
CPU:31 0 0 0 0
0 0
CPU:32 0 0 0 0
0 0
CPU:33 0 0 0 0
0 0
CPU:34 0 0 0 0
0 0
CPU:35 0 0 0 0
0 0
CPU:36 0 0 0 0
0 0
CPU:37 0 0 0 0
0 0
CPU:38 0 0 0 0
0 0
CPU:39 0 0 0 0
0 0
CPU:40 0 0 0 0
0 0
CPU:41 0 0 0 0
0 0
CPU:42 466853 0 43 0
0 0
CPU:43 453059 0 54 0
0 0
CPU:44 363219 0 34 0
0 0
CPU:45 353632 0 38 0
0 0
CPU:46 371618 0 40 0
0 0
CPU:47 350518 0 46 0
0 0
CPU:48 397544 0 40 0
0 0
CPU:49 364873 0 38 0
0 0
CPU:50 383630 0 38 0
0 0
CPU:51 358771 0 39 0
0 0
CPU:52 372547 0 38 0
0 0
CPU:53 372882 0 36 0
0 0
CPU:54 366244 0 43 0
0 0
CPU:55 365886 0 39 0
0 0
Summed: 11835201 0 1217 0
0 0
>>
>>> cpu total dropped squeezed collision rps flow_limit
>>> 0 18554 0 1 0 0 0
>>> 1 16728 0 1 0 0 0
>>> 2 18033 0 1 0 0 0
>>> 3 17757 0 1 0 0 0
>>> 4 18861 0 0 0 0 0
>>> 5 0 0 1 0 0 0
>>> 6 2 0 1 0 0 0
>>> 7 0 0 1 0 0 0
>>> 8 0 0 0 0 0 0
>>> 9 0 0 1 0 0 0
>>> 10 0 0 0 0 0 0
>>> 11 0 0 1 0 0 0
>>> 12 50 0 1 0 0 0
>>> 13 257 0 0 0 0 0
>>> 14 3629115363 0 3353259 0 0 0
>>> 15 255167835 0 3138271 0 0 0
>>> 16 4240101961 0 3036130 0 0 0
>>> 17 599810018 0 3072169 0 0 0
>>> 18 432796524 0 3034191 0 0 0
>>> 19 41803906 0 3037405 0 0 0
>>> 20 900382666 0 3112294 0 0 0
>>> 21 620926085 0 3086009 0 0 0
>>> 22 41861198 0 3023142 0 0 0
>>> 23 4090425574 0 2990412 0 0 0
>>> 24 4264870218 0 3010272 0 0 0
>>> 25 141401811 0 3027153 0 0 0
>>> 26 104155188 0 3051251 0 0 0
>>> 27 4261258691 0 3039765 0 0 0
>>> 28 4 0 1 0 0 0
>>> 29 4 0 0 0 0 0
>>> 30 0 0 1 0 0 0
>>> 31 0 0 0 0 0 0
>>> 32 3 0 1 0 0 0
>>> 33 1 0 1 0 0 0
>>> 34 0 0 1 0 0 0
>>> 35 0 0 0 0 0 0
>>> 36 0 0 1 0 0 0
>>> 37 0 0 1 0 0 0
>>> 38 0 0 1 0 0 0
>>> 39 0 0 1 0 0 0
>>> 40 0 0 0 0 0 0
>>> 41 0 0 1 0 0 0
>>> 42 299758202 0 3139693 0 0 0
>>> 43 4254727979 0 3103577 0 0 0
>>> 44 1959555543 0 2554885 0 0 0
>>> 45 1675702723 0 2513481 0 0 0
>>> 46 1908435503 0 2519698 0 0 0
>>> 47 1877799710 0 2537768 0 0 0
>>> 48 2384274076 0 2584673 0 0 0
>>> 49 2598104878 0 2593616 0 0 0
>>> 50 1897566829 0 2530857 0 0 0
>>> 51 1712741629 0 2489089 0 0 0
>>> 52 1704033648 0 2495892 0 0 0
>>> 53 1636781820 0 2499783 0 0 0
>>> 54 1861997734 0 2541060 0 0 0
>>> 55 2113521616 0 2555673 0 0 0
>>>
>>>
>>> So i rised netdev backlog and budged to rly high values
>>> 524288 for netdev_budget and same for backlog
>> Does it affect the squeezed counters?
> a little - but not much
> After change budget from 65536 to to 524k - number of squeezed
> counters for all cpus changed from 1.5k per second to 0.9-1k per
> second - but increasing it more like above 524k change nothing - same
> 0.9 to 1k/s squeezed
>>
>> Notice, this (crazy) huge netdev_budget limit will also be limited
>> by /proc/sys/net/core/netdev_budget_usecs.
> Yes changed that also to 1000 / 2000 / 3000 / 4000 not much
> difference on squeezed - even cant see the difference
>
>>
>>> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
>> Hmmm, this could indicated not enough NAPI bulking is occurring.
>>
>> I have a BPF tool, that can give you some insight into NAPI bulking and
>> softirq idle/kthread starting. Called 'napi_monitor', could you try to
>> run this, so can try to understand this? You find the tool here:
>>
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
> yes will try it
>
>>> But after this changes i have less packets drops.
>>>
>>>
>>> Below perf top from max traffic reached:
>>> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>> ------------------------------------------------------------------------------------------
>>>
>>>
>>> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
>>> 8.44% [kernel] [k] mlx5e_sq_xmit
>>> 6.69% [kernel] [k] build_skb
>>> 5.21% [kernel] [k] fib_table_lookup
>>> 3.54% [kernel] [k] memcpy_erms
>>> 3.20% [kernel] [k] mlx5e_poll_rx_cq
>>> 2.25% [kernel] [k] vlan_do_receive
>>> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
>>> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
>>> 1.95% [kernel] [k] __dev_queue_xmit
>>> 1.83% [kernel] [k] dev_gro_receive
>>> 1.79% [kernel] [k] tcp_gro_receive
>>> 1.73% [kernel] [k] ip_finish_output2
>>> 1.63% [kernel] [k] mlx5e_poll_tx_cq
>>> 1.49% [kernel] [k] ipt_do_table
>>> 1.38% [kernel] [k] inet_gro_receive
>>> 1.31% [kernel] [k] __netif_receive_skb_core
>>> 1.30% [kernel] [k] _raw_spin_lock
>>> 1.28% [kernel] [k] mlx5_eq_int
>>> 1.24% [kernel] [k] irq_entries_start
>>> 1.19% [kernel] [k] __build_skb
>>> 1.15% [kernel] [k] swiotlb_map_page
>>> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
>>> 0.94% [kernel] [k] pfifo_fast_dequeue
>>> 0.92% [kernel] [k] ip_route_input_rcu
>>> 0.86% [kernel] [k] kmem_cache_alloc
>>> 0.80% [kernel] [k] mlx5e_xmit
>>> 0.79% [kernel] [k] dev_hard_start_xmit
>>> 0.78% [kernel] [k] _raw_spin_lock_irqsave
>>> 0.74% [kernel] [k] ip_forward
>>> 0.72% [kernel] [k] tasklet_action_common.isra.21
>>> 0.68% [kernel] [k] pfifo_fast_enqueue
>>> 0.67% [kernel] [k] netif_skb_features
>>> 0.66% [kernel] [k] skb_segment
>>> 0.60% [kernel] [k] skb_gro_receive
>>> 0.56% [kernel] [k] validate_xmit_skb.isra.142
>>> 0.53% [kernel] [k] skb_release_data
>>> 0.51% [kernel] [k] mlx5e_page_release
>>> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>>> 0.51% [kernel] [k] __qdisc_run
>>> 0.50% [kernel] [k] tcp4_gro_receive
>>> 0.49% [kernel] [k] page_frag_free
>>> 0.46% [kernel] [k] kmem_cache_free_bulk
>>> 0.43% [kernel] [k] kmem_cache_free
>>> 0.42% [kernel] [k] try_to_wake_up
>>> 0.39% [kernel] [k] _raw_spin_lock_irq
>>> 0.39% [kernel] [k] find_busiest_group
>>> 0.37% [kernel] [k] __memcpy
>>>
>>>
>>>
>>> Remember those tests are now on two separate connectx5 connected to
>>> two separate pcie x16 gen 3.0
>> That is strange... I still suspect some HW NIC issue, can you provide
>> ethtool stats info via tool:
>>
>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>
>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>>
>> The tool remove zero-stats counters and report per sec stats. It makes
>> it easier to spot that is relevant for the given workload.
> yes mlnx have just too many counters that are always 0 for my case :)
> Will try this also
>
But still alot of non 0 counters
Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
Ethtool(enp175s0) stat: 10265 ( 10,265) <= ch0_events /sec
Ethtool(enp175s0) stat: 11072 ( 11,072) <= ch0_poll /sec
Ethtool(enp175s0) stat: 9003 ( 9,003) <= ch10_arm /sec
Ethtool(enp175s0) stat: 10476 ( 10,476) <= ch10_events /sec
Ethtool(enp175s0) stat: 11284 ( 11,284) <= ch10_poll /sec
Ethtool(enp175s0) stat: 11211 ( 11,211) <= ch11_arm /sec
Ethtool(enp175s0) stat: 12645 ( 12,645) <= ch11_events /sec
Ethtool(enp175s0) stat: 13450 ( 13,450) <= ch11_poll /sec
Ethtool(enp175s0) stat: 9012 ( 9,012) <= ch12_arm /sec
Ethtool(enp175s0) stat: 10366 ( 10,366) <= ch12_events /sec
Ethtool(enp175s0) stat: 11074 ( 11,074) <= ch12_poll /sec
Ethtool(enp175s0) stat: 8810 ( 8,810) <= ch13_arm /sec
Ethtool(enp175s0) stat: 10177 ( 10,177) <= ch13_events /sec
Ethtool(enp175s0) stat: 10886 ( 10,886) <= ch13_poll /sec
Ethtool(enp175s0) stat: 9794 ( 9,794) <= ch14_arm /sec
Ethtool(enp175s0) stat: 11159 ( 11,159) <= ch14_events /sec
Ethtool(enp175s0) stat: 11932 ( 11,932) <= ch14_poll /sec
Ethtool(enp175s0) stat: 8703 ( 8,703) <= ch15_arm /sec
Ethtool(enp175s0) stat: 10052 ( 10,052) <= ch15_events /sec
Ethtool(enp175s0) stat: 10774 ( 10,774) <= ch15_poll /sec
Ethtool(enp175s0) stat: 6429 ( 6,429) <= ch16_arm /sec
Ethtool(enp175s0) stat: 7591 ( 7,591) <= ch16_events /sec
Ethtool(enp175s0) stat: 8223 ( 8,223) <= ch16_poll /sec
Ethtool(enp175s0) stat: 8981 ( 8,981) <= ch17_arm /sec
Ethtool(enp175s0) stat: 10229 ( 10,229) <= ch17_events /sec
Ethtool(enp175s0) stat: 10887 ( 10,887) <= ch17_poll /sec
Ethtool(enp175s0) stat: 6786 ( 6,786) <= ch18_arm /sec
Ethtool(enp175s0) stat: 7887 ( 7,887) <= ch18_events /sec
Ethtool(enp175s0) stat: 8484 ( 8,484) <= ch18_poll /sec
Ethtool(enp175s0) stat: 6080 ( 6,080) <= ch19_arm /sec
Ethtool(enp175s0) stat: 7377 ( 7,377) <= ch19_events /sec
Ethtool(enp175s0) stat: 8124 ( 8,124) <= ch19_poll /sec
Ethtool(enp175s0) stat: 7715 ( 7,715) <= ch1_arm /sec
Ethtool(enp175s0) stat: 9109 ( 9,109) <= ch1_events /sec
Ethtool(enp175s0) stat: 9923 ( 9,923) <= ch1_poll /sec
Ethtool(enp175s0) stat: 7303 ( 7,303) <= ch20_arm /sec
Ethtool(enp175s0) stat: 8514 ( 8,514) <= ch20_events /sec
Ethtool(enp175s0) stat: 9169 ( 9,169) <= ch20_poll /sec
Ethtool(enp175s0) stat: 8972 ( 8,972) <= ch21_arm /sec
Ethtool(enp175s0) stat: 10060 ( 10,060) <= ch21_events /sec
Ethtool(enp175s0) stat: 10647 ( 10,647) <= ch21_poll /sec
Ethtool(enp175s0) stat: 7729 ( 7,729) <= ch22_arm /sec
Ethtool(enp175s0) stat: 8932 ( 8,932) <= ch22_events /sec
Ethtool(enp175s0) stat: 9585 ( 9,585) <= ch22_poll /sec
Ethtool(enp175s0) stat: 8125 ( 8,125) <= ch23_arm /sec
Ethtool(enp175s0) stat: 9218 ( 9,218) <= ch23_events /sec
Ethtool(enp175s0) stat: 9805 ( 9,805) <= ch23_poll /sec
Ethtool(enp175s0) stat: 7212 ( 7,212) <= ch24_arm /sec
Ethtool(enp175s0) stat: 8369 ( 8,369) <= ch24_events /sec
Ethtool(enp175s0) stat: 8993 ( 8,993) <= ch24_poll /sec
Ethtool(enp175s0) stat: 6328 ( 6,328) <= ch25_arm /sec
Ethtool(enp175s0) stat: 7567 ( 7,567) <= ch25_events /sec
Ethtool(enp175s0) stat: 8274 ( 8,274) <= ch25_poll /sec
Ethtool(enp175s0) stat: 6210 ( 6,210) <= ch26_arm /sec
Ethtool(enp175s0) stat: 7409 ( 7,409) <= ch26_events /sec
Ethtool(enp175s0) stat: 8062 ( 8,062) <= ch26_poll /sec
Ethtool(enp175s0) stat: 7035 ( 7,035) <= ch27_arm /sec
Ethtool(enp175s0) stat: 8203 ( 8,203) <= ch27_events /sec
Ethtool(enp175s0) stat: 8840 ( 8,840) <= ch27_poll /sec
Ethtool(enp175s0) stat: 11278 ( 11,278) <= ch2_arm /sec
Ethtool(enp175s0) stat: 12632 ( 12,632) <= ch2_events /sec
Ethtool(enp175s0) stat: 13348 ( 13,348) <= ch2_poll /sec
Ethtool(enp175s0) stat: 10612 ( 10,612) <= ch3_arm /sec
Ethtool(enp175s0) stat: 11900 ( 11,900) <= ch3_events /sec
Ethtool(enp175s0) stat: 12567 ( 12,567) <= ch3_poll /sec
Ethtool(enp175s0) stat: 8936 ( 8,936) <= ch4_arm /sec
Ethtool(enp175s0) stat: 10248 ( 10,248) <= ch4_events /sec
Ethtool(enp175s0) stat: 10962 ( 10,962) <= ch4_poll /sec
Ethtool(enp175s0) stat: 11631 ( 11,631) <= ch5_arm /sec
Ethtool(enp175s0) stat: 12953 ( 12,953) <= ch5_events /sec
Ethtool(enp175s0) stat: 13629 ( 13,629) <= ch5_poll /sec
Ethtool(enp175s0) stat: 9877 ( 9,877) <= ch6_arm /sec
Ethtool(enp175s0) stat: 11114 ( 11,114) <= ch6_events /sec
Ethtool(enp175s0) stat: 11800 ( 11,800) <= ch6_poll /sec
Ethtool(enp175s0) stat: 8228 ( 8,228) <= ch7_arm /sec
Ethtool(enp175s0) stat: 9577 ( 9,577) <= ch7_events /sec
Ethtool(enp175s0) stat: 10320 ( 10,320) <= ch7_poll /sec
Ethtool(enp175s0) stat: 11808 ( 11,808) <= ch8_arm /sec
Ethtool(enp175s0) stat: 13135 ( 13,135) <= ch8_events /sec
Ethtool(enp175s0) stat: 13828 ( 13,828) <= ch8_poll /sec
Ethtool(enp175s0) stat: 10566 ( 10,566) <= ch9_arm /sec
Ethtool(enp175s0) stat: 11904 ( 11,904) <= ch9_events /sec
Ethtool(enp175s0) stat: 12634 ( 12,634) <= ch9_poll /sec
Ethtool(enp175s0) stat: 243256 ( 243,256) <= ch_arm /sec
Ethtool(enp175s0) stat: 279057 ( 279,057) <= ch_events /sec
Ethtool(enp175s0) stat: 298563 ( 298,563) <= ch_poll /sec
Ethtool(enp175s0) stat: 186677525 ( 186,677,525) <= rx0_bytes /sec
Ethtool(enp175s0) stat: 72870 ( 72,870) <=
rx0_cache_reuse /sec
Ethtool(enp175s0) stat: 145627 ( 145,627) <=
rx0_csum_complete /sec
Ethtool(enp175s0) stat: 88 ( 88) <= rx0_csum_none /sec
Ethtool(enp175s0) stat: 145715 ( 145,715) <= rx0_packets /sec
Ethtool(enp175s0) stat: 145715 ( 145,715) <=
rx0_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 198552827 ( 198,552,827) <= rx10_bytes /sec
Ethtool(enp175s0) stat: 75553 ( 75,553) <=
rx10_cache_reuse /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <=
rx10_csum_complete /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <= rx10_packets /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <=
rx10_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 200924148 ( 200,924,148) <= rx11_bytes /sec
Ethtool(enp175s0) stat: 76589 ( 76,589) <=
rx11_cache_reuse /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <=
rx11_csum_complete /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <= rx11_packets /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <=
rx11_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 186259790 ( 186,259,790) <= rx12_bytes /sec
Ethtool(enp175s0) stat: 70675 ( 70,675) <=
rx12_cache_reuse /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <=
rx12_csum_complete /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <= rx12_packets /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <=
rx12_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 189627451 ( 189,627,451) <= rx13_bytes /sec
Ethtool(enp175s0) stat: 72626 ( 72,626) <=
rx13_cache_reuse /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <=
rx13_csum_complete /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <= rx13_packets /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <=
rx13_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 199246096 ( 199,246,096) <= rx14_bytes /sec
Ethtool(enp175s0) stat: 77992 ( 77,992) <=
rx14_cache_reuse /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <=
rx14_csum_complete /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <= rx14_packets /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <=
rx14_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 189698176 ( 189,698,176) <= rx15_bytes /sec
Ethtool(enp175s0) stat: 72382 ( 72,382) <=
rx15_cache_reuse /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <=
rx15_csum_complete /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <= rx15_packets /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <=
rx15_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 143896232 ( 143,896,232) <= rx16_bytes /sec
Ethtool(enp175s0) stat: 55369 ( 55,369) <=
rx16_cache_reuse /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <=
rx16_csum_complete /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <= rx16_packets /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <=
rx16_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 171449483 ( 171,449,483) <= rx17_bytes /sec
Ethtool(enp175s0) stat: 65308 ( 65,308) <=
rx17_cache_reuse /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <=
rx17_csum_complete /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <= rx17_packets /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <=
rx17_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 141033264 ( 141,033,264) <= rx18_bytes /sec
Ethtool(enp175s0) stat: 54515 ( 54,515) <=
rx18_cache_reuse /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <=
rx18_csum_complete /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <= rx18_packets /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <=
rx18_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 163097410 ( 163,097,410) <= rx19_bytes /sec
Ethtool(enp175s0) stat: 61894 ( 61,894) <=
rx19_cache_reuse /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <=
rx19_csum_complete /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <= rx19_packets /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <=
rx19_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 181707418 ( 181,707,418) <= rx1_bytes /sec
Ethtool(enp175s0) stat: 71223 ( 71,223) <=
rx1_cache_reuse /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <=
rx1_csum_complete /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <= rx1_packets /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <=
rx1_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 161626368 ( 161,626,368) <= rx20_bytes /sec
Ethtool(enp175s0) stat: 61345 ( 61,345) <=
rx20_cache_reuse /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <=
rx20_csum_complete /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <= rx20_packets /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <=
rx20_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 138593554 ( 138,593,554) <= rx21_bytes /sec
Ethtool(enp175s0) stat: 53478 ( 53,478) <=
rx21_cache_reuse /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <=
rx21_csum_complete /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <= rx21_packets /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <=
rx21_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 149217722 ( 149,217,722) <= rx22_bytes /sec
Ethtool(enp175s0) stat: 58174 ( 58,174) <=
rx22_cache_reuse /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <=
rx22_csum_complete /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <= rx22_packets /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <=
rx22_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 147968086 ( 147,968,086) <= rx23_bytes /sec
Ethtool(enp175s0) stat: 55979 ( 55,979) <=
rx23_cache_reuse /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <=
rx23_csum_complete /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <= rx23_packets /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <=
rx23_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 145955524 ( 145,955,524) <= rx24_bytes /sec
Ethtool(enp175s0) stat: 55491 ( 55,491) <=
rx24_cache_reuse /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <=
rx24_csum_complete /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <= rx24_packets /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <=
rx24_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 155552699 ( 155,552,699) <= rx25_bytes /sec
Ethtool(enp175s0) stat: 59028 ( 59,028) <=
rx25_cache_reuse /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <=
rx25_csum_complete /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <= rx25_packets /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <=
rx25_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 144880442 ( 144,880,442) <= rx26_bytes /sec
Ethtool(enp175s0) stat: 56223 ( 56,223) <=
rx26_cache_reuse /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <=
rx26_csum_complete /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <= rx26_packets /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <=
rx26_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 154545288 ( 154,545,288) <= rx27_bytes /sec
Ethtool(enp175s0) stat: 58784 ( 58,784) <=
rx27_cache_reuse /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <=
rx27_csum_complete /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <= rx27_packets /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <=
rx27_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 182425129 ( 182,425,129) <= rx2_bytes /sec
Ethtool(enp175s0) stat: 71406 ( 71,406) <=
rx2_cache_reuse /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <=
rx2_csum_complete /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <= rx2_packets /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <=
rx2_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 188368405 ( 188,368,405) <= rx3_bytes /sec
Ethtool(enp175s0) stat: 72138 ( 72,138) <=
rx3_cache_reuse /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <=
rx3_csum_complete /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <= rx3_packets /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <=
rx3_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 186009984 ( 186,009,984) <= rx4_bytes /sec
Ethtool(enp175s0) stat: 70004 ( 70,004) <=
rx4_cache_reuse /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <=
rx4_csum_complete /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <= rx4_packets /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <=
rx4_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 198040550 ( 198,040,550) <= rx5_bytes /sec
Ethtool(enp175s0) stat: 75492 ( 75,492) <=
rx5_cache_reuse /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <=
rx5_csum_complete /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <= rx5_packets /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <=
rx5_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 182607101 ( 182,607,101) <= rx6_bytes /sec
Ethtool(enp175s0) stat: 69699 ( 69,699) <=
rx6_cache_reuse /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <=
rx6_csum_complete /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <= rx6_packets /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <=
rx6_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 174999243 ( 174,999,243) <= rx7_bytes /sec
Ethtool(enp175s0) stat: 66650 ( 66,650) <=
rx7_cache_reuse /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <=
rx7_csum_complete /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <= rx7_packets /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <=
rx7_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 204109286 ( 204,109,286) <= rx8_bytes /sec
Ethtool(enp175s0) stat: 76711 ( 76,711) <=
rx8_cache_reuse /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <=
rx8_csum_complete /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <= rx8_packets /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <=
rx8_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 183703752 ( 183,703,752) <= rx9_bytes /sec
Ethtool(enp175s0) stat: 71101 ( 71,101) <=
rx9_cache_reuse /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <=
rx9_csum_complete /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <= rx9_packets /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <=
rx9_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 2024072 ( 2,024,072) <=
rx_1024_to_1518_bytes_phy /sec
Ethtool(enp175s0) stat: 106582 ( 106,582) <=
rx_128_to_255_bytes_phy /sec
Ethtool(enp175s0) stat: 1296735 ( 1,296,735) <=
rx_1519_to_2047_bytes_phy /sec
Ethtool(enp175s0) stat: 59460 ( 59,460) <=
rx_256_to_511_bytes_phy /sec
Ethtool(enp175s0) stat: 57326 ( 57,326) <=
rx_512_to_1023_bytes_phy /sec
Ethtool(enp175s0) stat: 7159 ( 7,159) <=
rx_64_bytes_phy /sec
Ethtool(enp175s0) stat: 310730 ( 310,730) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp175s0) stat: 232 ( 232) <=
rx_broadcast_phy /sec
Ethtool(enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
Ethtool(enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
Ethtool(enp175s0) stat: 1858636 ( 1,858,636) <= rx_cache_reuse
/sec
Ethtool(enp175s0) stat: 3717060 ( 3,717,060) <=
rx_csum_complete /sec
Ethtool(enp175s0) stat: 88 ( 88) <= rx_csum_none /sec
Ethtool(enp175s0) stat: 139602 ( 139,602) <=
rx_discards_phy /sec
Ethtool(enp175s0) stat: 354 ( 354) <=
rx_multicast_phy /sec
Ethtool(enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
Ethtool(enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy
/sec
Ethtool(enp175s0) stat: 5063355121 ( 5,063,355,121) <= rx_prio0_bytes
/sec
Ethtool(enp175s0) stat: 3718759 ( 3,718,759) <=
rx_prio0_packets /sec
Ethtool(enp175s0) stat: 7193190 ( 7,193,190) <= rx_prio1_bytes
/sec
Ethtool(enp175s0) stat: 5031 ( 5,031) <=
rx_prio1_packets /sec
Ethtool(enp175s0) stat: 557 ( 557) <= rx_prio2_bytes
/sec
Ethtool(enp175s0) stat: 5 ( 5) <=
rx_prio2_packets /sec
Ethtool(enp175s0) stat: 61 ( 61) <= rx_prio3_bytes
/sec
Ethtool(enp175s0) stat: 1 ( 1) <=
rx_prio3_packets /sec
Ethtool(enp175s0) stat: 21010 ( 21,010) <= rx_prio4_bytes
/sec
Ethtool(enp175s0) stat: 39 ( 39) <=
rx_prio4_packets /sec
Ethtool(enp175s0) stat: 187 ( 187) <= rx_prio5_bytes
/sec
Ethtool(enp175s0) stat: 2 ( 2) <=
rx_prio5_packets /sec
Ethtool(enp175s0) stat: 1711 ( 1,711) <= rx_prio6_bytes
/sec
Ethtool(enp175s0) stat: 15 ( 15) <=
rx_prio6_packets /sec
Ethtool(enp175s0) stat: 19498 ( 19,498) <= rx_prio7_bytes
/sec
Ethtool(enp175s0) stat: 273 ( 273) <=
rx_prio7_packets /sec
Ethtool(enp175s0) stat: 3717148 ( 3,717,148) <=
rx_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 5737 ( 5,737) <=
rx_steer_missed_packets /sec
Ethtool(enp175s0) stat: 14573 ( 14,573) <=
rx_vport_broadcast_bytes /sec
Ethtool(enp175s0) stat: 232 ( 232) <=
rx_vport_broadcast_packets /sec
Ethtool(enp175s0) stat: 25491 ( 25,491) <=
rx_vport_multicast_bytes /sec
Ethtool(enp175s0) stat: 354 ( 354) <=
rx_vport_multicast_packets /sec
Ethtool(enp175s0) stat: 4872354516 ( 4,872,354,516) <=
rx_vport_unicast_bytes /sec
Ethtool(enp175s0) stat: 3721920 ( 3,721,920) <=
rx_vport_unicast_packets /sec
Ethtool(enp175s0) stat: 158883 ( 158,883) <=
tx0_added_vlan_packets /sec
Ethtool(enp175s0) stat: 93790423 ( 93,790,423) <= tx0_bytes /sec
Ethtool(enp175s0) stat: 158854 ( 158,854) <= tx0_cqes /sec
Ethtool(enp175s0) stat: 146499 ( 146,499) <= tx0_csum_none /sec
Ethtool(enp175s0) stat: 12384 ( 12,384) <=
tx0_csum_partial /sec
Ethtool(enp175s0) stat: 2144 ( 2,144) <= tx0_nop /sec
Ethtool(enp175s0) stat: 188173 ( 188,173) <= tx0_packets /sec
Ethtool(enp175s0) stat: 53068613 ( 53,068,613) <= tx0_tso_bytes /sec
Ethtool(enp175s0) stat: 8839 ( 8,839) <=
tx0_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx0_xmit_more /sec
Ethtool(enp175s0) stat: 165538 ( 165,538) <=
tx10_added_vlan_packets /sec
Ethtool(enp175s0) stat: 102395057 ( 102,395,057) <= tx10_bytes /sec
Ethtool(enp175s0) stat: 165303 ( 165,303) <= tx10_cqes /sec
Ethtool(enp175s0) stat: 151089 ( 151,089) <= tx10_csum_none
/sec
Ethtool(enp175s0) stat: 14448 ( 14,448) <=
tx10_csum_partial /sec
Ethtool(enp175s0) stat: 2391 ( 2,391) <= tx10_nop /sec
Ethtool(enp175s0) stat: 198798 ( 198,798) <= tx10_packets /sec
Ethtool(enp175s0) stat: 58951449 ( 58,951,449) <= tx10_tso_bytes
/sec
Ethtool(enp175s0) stat: 8987 ( 8,987) <=
tx10_tso_packets /sec
Ethtool(enp175s0) stat: 234 ( 234) <= tx10_xmit_more
/sec
Ethtool(enp175s0) stat: 166402 ( 166,402) <=
tx11_added_vlan_packets /sec
Ethtool(enp175s0) stat: 98591304 ( 98,591,304) <= tx11_bytes /sec
Ethtool(enp175s0) stat: 166384 ( 166,384) <= tx11_cqes /sec
Ethtool(enp175s0) stat: 152456 ( 152,456) <= tx11_csum_none
/sec
Ethtool(enp175s0) stat: 13946 ( 13,946) <=
tx11_csum_partial /sec
Ethtool(enp175s0) stat: 2386 ( 2,386) <= tx11_nop /sec
Ethtool(enp175s0) stat: 201615 ( 201,615) <= tx11_packets /sec
Ethtool(enp175s0) stat: 63844660 ( 63,844,660) <= tx11_tso_bytes
/sec
Ethtool(enp175s0) stat: 10515 ( 10,515) <=
tx11_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx11_xmit_more
/sec
Ethtool(enp175s0) stat: 156312 ( 156,312) <=
tx12_added_vlan_packets /sec
Ethtool(enp175s0) stat: 97068537 ( 97,068,537) <= tx12_bytes /sec
Ethtool(enp175s0) stat: 156302 ( 156,302) <= tx12_cqes /sec
Ethtool(enp175s0) stat: 142468 ( 142,468) <= tx12_csum_none
/sec
Ethtool(enp175s0) stat: 13844 ( 13,844) <=
tx12_csum_partial /sec
Ethtool(enp175s0) stat: 2278 ( 2,278) <= tx12_nop /sec
Ethtool(enp175s0) stat: 187535 ( 187,535) <= tx12_packets /sec
Ethtool(enp175s0) stat: 58368798 ( 58,368,798) <= tx12_tso_bytes
/sec
Ethtool(enp175s0) stat: 10398 ( 10,398) <=
tx12_tso_packets /sec
Ethtool(enp175s0) stat: 10 ( 10) <= tx12_xmit_more
/sec
Ethtool(enp175s0) stat: 161768 ( 161,768) <=
tx13_added_vlan_packets /sec
Ethtool(enp175s0) stat: 120232518 ( 120,232,518) <= tx13_bytes /sec
Ethtool(enp175s0) stat: 161584 ( 161,584) <= tx13_cqes /sec
Ethtool(enp175s0) stat: 144388 ( 144,388) <= tx13_csum_none
/sec
Ethtool(enp175s0) stat: 17380 ( 17,380) <=
tx13_csum_partial /sec
Ethtool(enp175s0) stat: 2425 ( 2,425) <= tx13_nop /sec
Ethtool(enp175s0) stat: 202823 ( 202,823) <= tx13_packets /sec
Ethtool(enp175s0) stat: 72804507 ( 72,804,507) <= tx13_tso_bytes
/sec
Ethtool(enp175s0) stat: 10865 ( 10,865) <=
tx13_tso_packets /sec
Ethtool(enp175s0) stat: 185 ( 185) <= tx13_xmit_more
/sec
Ethtool(enp175s0) stat: 165762 ( 165,762) <=
tx14_added_vlan_packets /sec
Ethtool(enp175s0) stat: 99271622 ( 99,271,622) <= tx14_bytes /sec
Ethtool(enp175s0) stat: 165688 ( 165,688) <= tx14_cqes /sec
Ethtool(enp175s0) stat: 153195 ( 153,195) <= tx14_csum_none
/sec
Ethtool(enp175s0) stat: 12566 ( 12,566) <=
tx14_csum_partial /sec
Ethtool(enp175s0) stat: 2195 ( 2,195) <= tx14_nop /sec
Ethtool(enp175s0) stat: 195504 ( 195,504) <= tx14_packets /sec
Ethtool(enp175s0) stat: 53717277 ( 53,717,277) <= tx14_tso_bytes
/sec
Ethtool(enp175s0) stat: 8743 ( 8,743) <=
tx14_tso_packets /sec
Ethtool(enp175s0) stat: 32 ( 32) <= tx14_xmit_more
/sec
Ethtool(enp175s0) stat: 162803 ( 162,803) <=
tx15_added_vlan_packets /sec
Ethtool(enp175s0) stat: 105591893 ( 105,591,893) <= tx15_bytes /sec
Ethtool(enp175s0) stat: 162673 ( 162,673) <= tx15_cqes /sec
Ethtool(enp175s0) stat: 147080 ( 147,080) <= tx15_csum_none
/sec
Ethtool(enp175s0) stat: 15723 ( 15,723) <=
tx15_csum_partial /sec
Ethtool(enp175s0) stat: 2355 ( 2,355) <= tx15_nop /sec
Ethtool(enp175s0) stat: 198282 ( 198,282) <= tx15_packets /sec
Ethtool(enp175s0) stat: 64278573 ( 64,278,573) <= tx15_tso_bytes
/sec
Ethtool(enp175s0) stat: 10704 ( 10,704) <=
tx15_tso_packets /sec
Ethtool(enp175s0) stat: 183 ( 183) <= tx15_xmit_more
/sec
Ethtool(enp175s0) stat: 125282 ( 125,282) <=
tx16_added_vlan_packets /sec
Ethtool(enp175s0) stat: 81835815 ( 81,835,815) <= tx16_bytes /sec
Ethtool(enp175s0) stat: 125264 ( 125,264) <= tx16_cqes /sec
Ethtool(enp175s0) stat: 113284 ( 113,284) <= tx16_csum_none
/sec
Ethtool(enp175s0) stat: 11998 ( 11,998) <=
tx16_csum_partial /sec
Ethtool(enp175s0) stat: 1773 ( 1,773) <= tx16_nop /sec
Ethtool(enp175s0) stat: 150812 ( 150,812) <= tx16_packets /sec
Ethtool(enp175s0) stat: 46027767 ( 46,027,767) <= tx16_tso_bytes
/sec
Ethtool(enp175s0) stat: 7361 ( 7,361) <=
tx16_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx16_xmit_more
/sec
Ethtool(enp175s0) stat: 131865 ( 131,865) <=
tx17_added_vlan_packets /sec
Ethtool(enp175s0) stat: 89213555 ( 89,213,555) <= tx17_bytes /sec
Ethtool(enp175s0) stat: 131409 ( 131,409) <= tx17_cqes /sec
Ethtool(enp175s0) stat: 117199 ( 117,199) <= tx17_csum_none
/sec
Ethtool(enp175s0) stat: 14665 ( 14,665) <=
tx17_csum_partial /sec
Ethtool(enp175s0) stat: 1933 ( 1,933) <= tx17_nop /sec
Ethtool(enp175s0) stat: 161576 ( 161,576) <= tx17_packets /sec
Ethtool(enp175s0) stat: 53682297 ( 53,682,297) <= tx17_tso_bytes
/sec
Ethtool(enp175s0) stat: 8825 ( 8,825) <=
tx17_tso_packets /sec
Ethtool(enp175s0) stat: 459 ( 459) <= tx17_xmit_more
/sec
Ethtool(enp175s0) stat: 122492 ( 122,492) <=
tx18_added_vlan_packets /sec
Ethtool(enp175s0) stat: 87373476 ( 87,373,476) <= tx18_bytes /sec
Ethtool(enp175s0) stat: 122138 ( 122,138) <= tx18_cqes /sec
Ethtool(enp175s0) stat: 109037 ( 109,037) <= tx18_csum_none
/sec
Ethtool(enp175s0) stat: 13455 ( 13,455) <=
tx18_csum_partial /sec
Ethtool(enp175s0) stat: 1933 ( 1,933) <= tx18_nop /sec
Ethtool(enp175s0) stat: 152821 ( 152,821) <= tx18_packets /sec
Ethtool(enp175s0) stat: 54163102 ( 54,163,102) <= tx18_tso_bytes
/sec
Ethtool(enp175s0) stat: 8674 ( 8,674) <=
tx18_tso_packets /sec
Ethtool(enp175s0) stat: 354 ( 354) <= tx18_xmit_more
/sec
Ethtool(enp175s0) stat: 119294 ( 119,294) <=
tx19_added_vlan_packets /sec
Ethtool(enp175s0) stat: 67719609 ( 67,719,609) <= tx19_bytes /sec
Ethtool(enp175s0) stat: 119262 ( 119,262) <= tx19_cqes /sec
Ethtool(enp175s0) stat: 108591 ( 108,591) <= tx19_csum_none
/sec
Ethtool(enp175s0) stat: 10703 ( 10,703) <=
tx19_csum_partial /sec
Ethtool(enp175s0) stat: 1551 ( 1,551) <= tx19_nop /sec
Ethtool(enp175s0) stat: 141344 ( 141,344) <= tx19_packets /sec
Ethtool(enp175s0) stat: 39539778 ( 39,539,778) <= tx19_tso_bytes
/sec
Ethtool(enp175s0) stat: 6310 ( 6,310) <=
tx19_tso_packets /sec
Ethtool(enp175s0) stat: 31 ( 31) <= tx19_xmit_more
/sec
Ethtool(enp175s0) stat: 157094 ( 157,094) <=
tx1_added_vlan_packets /sec
Ethtool(enp175s0) stat: 93505806 ( 93,505,806) <= tx1_bytes /sec
Ethtool(enp175s0) stat: 156935 ( 156,935) <= tx1_cqes /sec
Ethtool(enp175s0) stat: 144805 ( 144,805) <= tx1_csum_none /sec
Ethtool(enp175s0) stat: 12289 ( 12,289) <=
tx1_csum_partial /sec
Ethtool(enp175s0) stat: 2201 ( 2,201) <= tx1_nop /sec
Ethtool(enp175s0) stat: 184561 ( 184,561) <= tx1_packets /sec
Ethtool(enp175s0) stat: 50513729 ( 50,513,729) <= tx1_tso_bytes /sec
Ethtool(enp175s0) stat: 8699 ( 8,699) <=
tx1_tso_packets /sec
Ethtool(enp175s0) stat: 159 ( 159) <= tx1_xmit_more /sec
Ethtool(enp175s0) stat: 134411 ( 134,411) <=
tx20_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88898658 ( 88,898,658) <= tx20_bytes /sec
Ethtool(enp175s0) stat: 134221 ( 134,221) <= tx20_cqes /sec
Ethtool(enp175s0) stat: 120787 ( 120,787) <= tx20_csum_none
/sec
Ethtool(enp175s0) stat: 13624 ( 13,624) <=
tx20_csum_partial /sec
Ethtool(enp175s0) stat: 2064 ( 2,064) <= tx20_nop /sec
Ethtool(enp175s0) stat: 165064 ( 165,064) <= tx20_packets /sec
Ethtool(enp175s0) stat: 54707259 ( 54,707,259) <= tx20_tso_bytes
/sec
Ethtool(enp175s0) stat: 8769 ( 8,769) <=
tx20_tso_packets /sec
Ethtool(enp175s0) stat: 188 ( 188) <= tx20_xmit_more
/sec
Ethtool(enp175s0) stat: 127029 ( 127,029) <=
tx21_added_vlan_packets /sec
Ethtool(enp175s0) stat: 98012630 ( 98,012,630) <= tx21_bytes /sec
Ethtool(enp175s0) stat: 126990 ( 126,990) <= tx21_cqes /sec
Ethtool(enp175s0) stat: 111553 ( 111,553) <= tx21_csum_none
/sec
Ethtool(enp175s0) stat: 15476 ( 15,476) <=
tx21_csum_partial /sec
Ethtool(enp175s0) stat: 2002 ( 2,002) <= tx21_nop /sec
Ethtool(enp175s0) stat: 159688 ( 159,688) <= tx21_packets /sec
Ethtool(enp175s0) stat: 59637304 ( 59,637,304) <= tx21_tso_bytes
/sec
Ethtool(enp175s0) stat: 9988 ( 9,988) <=
tx21_tso_packets /sec
Ethtool(enp175s0) stat: 39 ( 39) <= tx21_xmit_more
/sec
Ethtool(enp175s0) stat: 122610 ( 122,610) <=
tx22_added_vlan_packets /sec
Ethtool(enp175s0) stat: 70972052 ( 70,972,052) <= tx22_bytes /sec
Ethtool(enp175s0) stat: 122600 ( 122,600) <= tx22_cqes /sec
Ethtool(enp175s0) stat: 111526 ( 111,526) <= tx22_csum_none
/sec
Ethtool(enp175s0) stat: 11085 ( 11,085) <=
tx22_csum_partial /sec
Ethtool(enp175s0) stat: 1544 ( 1,544) <= tx22_nop /sec
Ethtool(enp175s0) stat: 144874 ( 144,874) <= tx22_packets /sec
Ethtool(enp175s0) stat: 40175057 ( 40,175,057) <= tx22_tso_bytes
/sec
Ethtool(enp175s0) stat: 6517 ( 6,517) <=
tx22_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx22_xmit_more
/sec
Ethtool(enp175s0) stat: 126809 ( 126,809) <=
tx23_added_vlan_packets /sec
Ethtool(enp175s0) stat: 76906314 ( 76,906,314) <= tx23_bytes /sec
Ethtool(enp175s0) stat: 126791 ( 126,791) <= tx23_cqes /sec
Ethtool(enp175s0) stat: 116656 ( 116,656) <= tx23_csum_none
/sec
Ethtool(enp175s0) stat: 10153 ( 10,153) <=
tx23_csum_partial /sec
Ethtool(enp175s0) stat: 1686 ( 1,686) <= tx23_nop /sec
Ethtool(enp175s0) stat: 148551 ( 148,551) <= tx23_packets /sec
Ethtool(enp175s0) stat: 39913764 ( 39,913,764) <= tx23_tso_bytes
/sec
Ethtool(enp175s0) stat: 6978 ( 6,978) <=
tx23_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx23_xmit_more
/sec
Ethtool(enp175s0) stat: 133910 ( 133,910) <=
tx24_added_vlan_packets /sec
Ethtool(enp175s0) stat: 69609913 ( 69,609,913) <= tx24_bytes /sec
Ethtool(enp175s0) stat: 133890 ( 133,890) <= tx24_cqes /sec
Ethtool(enp175s0) stat: 124462 ( 124,462) <= tx24_csum_none
/sec
Ethtool(enp175s0) stat: 9448 ( 9,448) <=
tx24_csum_partial /sec
Ethtool(enp175s0) stat: 1639 ( 1,639) <= tx24_nop /sec
Ethtool(enp175s0) stat: 154475 ( 154,475) <= tx24_packets /sec
Ethtool(enp175s0) stat: 37456148 ( 37,456,148) <= tx24_tso_bytes
/sec
Ethtool(enp175s0) stat: 6247 ( 6,247) <=
tx24_tso_packets /sec
Ethtool(enp175s0) stat: 20 ( 20) <= tx24_xmit_more
/sec
Ethtool(enp175s0) stat: 118528 ( 118,528) <=
tx25_added_vlan_packets /sec
Ethtool(enp175s0) stat: 62435525 ( 62,435,525) <= tx25_bytes /sec
Ethtool(enp175s0) stat: 118471 ( 118,471) <= tx25_cqes /sec
Ethtool(enp175s0) stat: 108887 ( 108,887) <= tx25_csum_none
/sec
Ethtool(enp175s0) stat: 9640 ( 9,640) <=
tx25_csum_partial /sec
Ethtool(enp175s0) stat: 1528 ( 1,528) <= tx25_nop /sec
Ethtool(enp175s0) stat: 138748 ( 138,748) <= tx25_packets /sec
Ethtool(enp175s0) stat: 36592045 ( 36,592,045) <= tx25_tso_bytes
/sec
Ethtool(enp175s0) stat: 5993 ( 5,993) <=
tx25_tso_packets /sec
Ethtool(enp175s0) stat: 44 ( 44) <= tx25_xmit_more
/sec
Ethtool(enp175s0) stat: 119890 ( 119,890) <=
tx26_added_vlan_packets /sec
Ethtool(enp175s0) stat: 76743929 ( 76,743,929) <= tx26_bytes /sec
Ethtool(enp175s0) stat: 119873 ( 119,873) <= tx26_cqes /sec
Ethtool(enp175s0) stat: 108181 ( 108,181) <= tx26_csum_none
/sec
Ethtool(enp175s0) stat: 11709 ( 11,709) <=
tx26_csum_partial /sec
Ethtool(enp175s0) stat: 1690 ( 1,690) <= tx26_nop /sec
Ethtool(enp175s0) stat: 144495 ( 144,495) <= tx26_packets /sec
Ethtool(enp175s0) stat: 43922304 ( 43,922,304) <= tx26_tso_bytes
/sec
Ethtool(enp175s0) stat: 7043 ( 7,043) <=
tx26_tso_packets /sec
Ethtool(enp175s0) stat: 17 ( 17) <= tx26_xmit_more
/sec
Ethtool(enp175s0) stat: 130825 ( 130,825) <=
tx27_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88723162 ( 88,723,162) <= tx27_bytes /sec
Ethtool(enp175s0) stat: 130769 ( 130,769) <= tx27_cqes /sec
Ethtool(enp175s0) stat: 116555 ( 116,555) <= tx27_csum_none
/sec
Ethtool(enp175s0) stat: 14270 ( 14,270) <=
tx27_csum_partial /sec
Ethtool(enp175s0) stat: 2042 ( 2,042) <= tx27_nop /sec
Ethtool(enp175s0) stat: 161228 ( 161,228) <= tx27_packets /sec
Ethtool(enp175s0) stat: 55637023 ( 55,637,023) <= tx27_tso_bytes
/sec
Ethtool(enp175s0) stat: 9707 ( 9,707) <=
tx27_tso_packets /sec
Ethtool(enp175s0) stat: 71 ( 71) <= tx27_xmit_more
/sec
Ethtool(enp175s0) stat: 166973 ( 166,973) <=
tx2_added_vlan_packets /sec
Ethtool(enp175s0) stat: 103659503 ( 103,659,503) <= tx2_bytes /sec
Ethtool(enp175s0) stat: 166941 ( 166,941) <= tx2_cqes /sec
Ethtool(enp175s0) stat: 151389 ( 151,389) <= tx2_csum_none /sec
Ethtool(enp175s0) stat: 15585 ( 15,585) <=
tx2_csum_partial /sec
Ethtool(enp175s0) stat: 2455 ( 2,455) <= tx2_nop /sec
Ethtool(enp175s0) stat: 201854 ( 201,854) <= tx2_packets /sec
Ethtool(enp175s0) stat: 65384298 ( 65,384,298) <= tx2_tso_bytes /sec
Ethtool(enp175s0) stat: 11809 ( 11,809) <=
tx2_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx2_xmit_more /sec
Ethtool(enp175s0) stat: 172353 ( 172,353) <=
tx3_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88248541 ( 88,248,541) <= tx3_bytes /sec
Ethtool(enp175s0) stat: 172277 ( 172,277) <= tx3_cqes /sec
Ethtool(enp175s0) stat: 160714 ( 160,714) <= tx3_csum_none /sec
Ethtool(enp175s0) stat: 11639 ( 11,639) <=
tx3_csum_partial /sec
Ethtool(enp175s0) stat: 2258 ( 2,258) <= tx3_nop /sec
Ethtool(enp175s0) stat: 199853 ( 199,853) <= tx3_packets /sec
Ethtool(enp175s0) stat: 49776463 ( 49,776,463) <= tx3_tso_bytes /sec
Ethtool(enp175s0) stat: 8353 ( 8,353) <=
tx3_tso_packets /sec
Ethtool(enp175s0) stat: 76 ( 76) <= tx3_xmit_more /sec
Ethtool(enp175s0) stat: 157527 ( 157,527) <=
tx4_added_vlan_packets /sec
Ethtool(enp175s0) stat: 110770979 ( 110,770,979) <= tx4_bytes /sec
Ethtool(enp175s0) stat: 157492 ( 157,492) <= tx4_cqes /sec
Ethtool(enp175s0) stat: 141858 ( 141,858) <= tx4_csum_none /sec
Ethtool(enp175s0) stat: 15670 ( 15,670) <=
tx4_csum_partial /sec
Ethtool(enp175s0) stat: 2320 ( 2,320) <= tx4_nop /sec
Ethtool(enp175s0) stat: 192429 ( 192,429) <= tx4_packets /sec
Ethtool(enp175s0) stat: 64689503 ( 64,689,503) <= tx4_tso_bytes /sec
Ethtool(enp175s0) stat: 11367 ( 11,367) <=
tx4_tso_packets /sec
Ethtool(enp175s0) stat: 35 ( 35) <= tx4_xmit_more /sec
Ethtool(enp175s0) stat: 169077 ( 169,077) <=
tx5_added_vlan_packets /sec
Ethtool(enp175s0) stat: 121536690 ( 121,536,690) <= tx5_bytes /sec
Ethtool(enp175s0) stat: 168906 ( 168,906) <= tx5_cqes /sec
Ethtool(enp175s0) stat: 150099 ( 150,099) <= tx5_csum_none /sec
Ethtool(enp175s0) stat: 18978 ( 18,978) <=
tx5_csum_partial /sec
Ethtool(enp175s0) stat: 2678 ( 2,678) <= tx5_nop /sec
Ethtool(enp175s0) stat: 210733 ( 210,733) <= tx5_packets /sec
Ethtool(enp175s0) stat: 76448346 ( 76,448,346) <= tx5_tso_bytes /sec
Ethtool(enp175s0) stat: 13238 ( 13,238) <=
tx5_tso_packets /sec
Ethtool(enp175s0) stat: 171 ( 171) <= tx5_xmit_more /sec
Ethtool(enp175s0) stat: 156881 ( 156,881) <=
tx6_added_vlan_packets /sec
Ethtool(enp175s0) stat: 85752393 ( 85,752,393) <= tx6_bytes /sec
Ethtool(enp175s0) stat: 156859 ( 156,859) <= tx6_cqes /sec
Ethtool(enp175s0) stat: 144843 ( 144,843) <= tx6_csum_none /sec
Ethtool(enp175s0) stat: 12038 ( 12,038) <=
tx6_csum_partial /sec
Ethtool(enp175s0) stat: 2034 ( 2,034) <= tx6_nop /sec
Ethtool(enp175s0) stat: 181315 ( 181,315) <= tx6_packets /sec
Ethtool(enp175s0) stat: 44625307 ( 44,625,307) <= tx6_tso_bytes /sec
Ethtool(enp175s0) stat: 7752 ( 7,752) <=
tx6_tso_packets /sec
Ethtool(enp175s0) stat: 24 ( 24) <= tx6_xmit_more /sec
Ethtool(enp175s0) stat: 157744 ( 157,744) <=
tx7_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88795890 ( 88,795,890) <= tx7_bytes /sec
Ethtool(enp175s0) stat: 157666 ( 157,666) <= tx7_cqes /sec
Ethtool(enp175s0) stat: 146023 ( 146,023) <= tx7_csum_none /sec
Ethtool(enp175s0) stat: 11720 ( 11,720) <=
tx7_csum_partial /sec
Ethtool(enp175s0) stat: 2190 ( 2,190) <= tx7_nop /sec
Ethtool(enp175s0) stat: 184845 ( 184,845) <= tx7_packets /sec
Ethtool(enp175s0) stat: 48986735 ( 48,986,735) <= tx7_tso_bytes /sec
Ethtool(enp175s0) stat: 8401 ( 8,401) <=
tx7_tso_packets /sec
Ethtool(enp175s0) stat: 78 ( 78) <= tx7_xmit_more /sec
Ethtool(enp175s0) stat: 165190 ( 165,190) <=
tx8_added_vlan_packets /sec
Ethtool(enp175s0) stat: 108303283 ( 108,303,283) <= tx8_bytes /sec
Ethtool(enp175s0) stat: 165145 ( 165,145) <= tx8_cqes /sec
Ethtool(enp175s0) stat: 150047 ( 150,047) <= tx8_csum_none /sec
Ethtool(enp175s0) stat: 15143 ( 15,143) <=
tx8_csum_partial /sec
Ethtool(enp175s0) stat: 2497 ( 2,497) <= tx8_nop /sec
Ethtool(enp175s0) stat: 201615 ( 201,615) <= tx8_packets /sec
Ethtool(enp175s0) stat: 67575199 ( 67,575,199) <= tx8_tso_bytes /sec
Ethtool(enp175s0) stat: 11828 ( 11,828) <=
tx8_tso_packets /sec
Ethtool(enp175s0) stat: 45 ( 45) <= tx8_xmit_more /sec
Ethtool(enp175s0) stat: 168256 ( 168,256) <=
tx9_added_vlan_packets /sec
Ethtool(enp175s0) stat: 115981227 ( 115,981,227) <= tx9_bytes /sec
Ethtool(enp175s0) stat: 168225 ( 168,225) <= tx9_cqes /sec
Ethtool(enp175s0) stat: 147980 ( 147,980) <= tx9_csum_none /sec
Ethtool(enp175s0) stat: 20275 ( 20,275) <=
tx9_csum_partial /sec
Ethtool(enp175s0) stat: 2522 ( 2,522) <= tx9_nop /sec
Ethtool(enp175s0) stat: 203398 ( 203,398) <= tx9_packets /sec
Ethtool(enp175s0) stat: 63891360 ( 63,891,360) <= tx9_tso_bytes /sec
Ethtool(enp175s0) stat: 10610 ( 10,610) <=
tx9_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx9_xmit_more /sec
Ethtool(enp175s0) stat: 4121491 ( 4,121,491) <=
tx_added_vlan_packets /sec
Ethtool(enp175s0) stat: 1 ( 1) <=
tx_broadcast_phy /sec
Ethtool(enp175s0) stat: 2591851137 ( 2,591,851,137) <= tx_bytes /sec
Ethtool(enp175s0) stat: 2626884154 ( 2,626,884,154) <= tx_bytes_phy /sec
Ethtool(enp175s0) stat: 4118829 ( 4,118,829) <= tx_cqes /sec
Ethtool(enp175s0) stat: 3741682 ( 3,741,682) <= tx_csum_none /sec
Ethtool(enp175s0) stat: 379810 ( 379,810) <=
tx_csum_partial /sec
Ethtool(enp175s0) stat: 58718 ( 58,718) <= tx_nop /sec
Ethtool(enp175s0) stat: 4956995 ( 4,956,995) <= tx_packets /sec
Ethtool(enp175s0) stat: 4956579 ( 4,956,579) <= tx_packets_phy
/sec
Ethtool(enp175s0) stat: 2627152852 ( 2,627,152,852) <= tx_prio0_bytes
/sec
Ethtool(enp175s0) stat: 4957649 ( 4,957,649) <=
tx_prio0_packets /sec
Ethtool(enp175s0) stat: 1518383905 ( 1,518,383,905) <= tx_tso_bytes /sec
Ethtool(enp175s0) stat: 253526 ( 253,526) <= tx_tso_packets
/sec
Ethtool(enp175s0) stat: 57 ( 57) <=
tx_vport_broadcast_bytes /sec
Ethtool(enp175s0) stat: 1 ( 1) <=
tx_vport_broadcast_packets /sec
Ethtool(enp175s0) stat: 2607061269 ( 2,607,061,269) <=
tx_vport_unicast_bytes /sec
Ethtool(enp175s0) stat: 4956492 ( 4,956,492) <=
tx_vport_unicast_packets /sec
Ethtool(enp175s0) stat: 2630 ( 2,630) <= tx_xmit_more /sec
Ethtool(enp216s0) stat: 6923 ( 6,923) <= ch0_arm /sec
Ethtool(enp216s0) stat: 8367 ( 8,367) <= ch0_events /sec
Ethtool(enp216s0) stat: 9797 ( 9,797) <= ch0_poll /sec
Ethtool(enp216s0) stat: 7703 ( 7,703) <= ch10_arm /sec
Ethtool(enp216s0) stat: 9253 ( 9,253) <= ch10_events /sec
Ethtool(enp216s0) stat: 10807 ( 10,807) <= ch10_poll /sec
Ethtool(enp216s0) stat: 7886 ( 7,886) <= ch11_arm /sec
Ethtool(enp216s0) stat: 9391 ( 9,391) <= ch11_events /sec
Ethtool(enp216s0) stat: 10896 ( 10,896) <= ch11_poll /sec
Ethtool(enp216s0) stat: 9793 ( 9,793) <= ch12_arm /sec
Ethtool(enp216s0) stat: 11299 ( 11,299) <= ch12_events /sec
Ethtool(enp216s0) stat: 12637 ( 12,637) <= ch12_poll /sec
Ethtool(enp216s0) stat: 9119 ( 9,119) <= ch13_arm /sec
Ethtool(enp216s0) stat: 10671 ( 10,671) <= ch13_events /sec
Ethtool(enp216s0) stat: 12205 ( 12,205) <= ch13_poll /sec
Ethtool(enp216s0) stat: 8784 ( 8,784) <= ch14_arm /sec
Ethtool(enp216s0) stat: 10189 ( 10,189) <= ch14_events /sec
Ethtool(enp216s0) stat: 11565 ( 11,565) <= ch14_poll /sec
Ethtool(enp216s0) stat: 9248 ( 9,248) <= ch15_arm /sec
Ethtool(enp216s0) stat: 10739 ( 10,739) <= ch15_events /sec
Ethtool(enp216s0) stat: 12238 ( 12,238) <= ch15_poll /sec
Ethtool(enp216s0) stat: 5201 ( 5,201) <= ch16_arm /sec
Ethtool(enp216s0) stat: 6554 ( 6,554) <= ch16_events /sec
Ethtool(enp216s0) stat: 7667 ( 7,667) <= ch16_poll /sec
Ethtool(enp216s0) stat: 7799 ( 7,799) <= ch17_arm /sec
Ethtool(enp216s0) stat: 9120 ( 9,120) <= ch17_events /sec
Ethtool(enp216s0) stat: 10194 ( 10,194) <= ch17_poll /sec
Ethtool(enp216s0) stat: 6889 ( 6,889) <= ch18_arm /sec
Ethtool(enp216s0) stat: 8266 ( 8,266) <= ch18_events /sec
Ethtool(enp216s0) stat: 9332 ( 9,332) <= ch18_poll /sec
Ethtool(enp216s0) stat: 5494 ( 5,494) <= ch19_arm /sec
Ethtool(enp216s0) stat: 6813 ( 6,813) <= ch19_events /sec
Ethtool(enp216s0) stat: 7837 ( 7,837) <= ch19_poll /sec
Ethtool(enp216s0) stat: 6672 ( 6,672) <= ch1_arm /sec
Ethtool(enp216s0) stat: 8124 ( 8,124) <= ch1_events /sec
Ethtool(enp216s0) stat: 9604 ( 9,604) <= ch1_poll /sec
Ethtool(enp216s0) stat: 7705 ( 7,705) <= ch20_arm /sec
Ethtool(enp216s0) stat: 9102 ( 9,102) <= ch20_events /sec
Ethtool(enp216s0) stat: 10287 ( 10,287) <= ch20_poll /sec
Ethtool(enp216s0) stat: 5929 ( 5,929) <= ch21_arm /sec
Ethtool(enp216s0) stat: 7333 ( 7,333) <= ch21_events /sec
Ethtool(enp216s0) stat: 8463 ( 8,463) <= ch21_poll /sec
Ethtool(enp216s0) stat: 5495 ( 5,495) <= ch22_arm /sec
Ethtool(enp216s0) stat: 6813 ( 6,813) <= ch22_events /sec
Ethtool(enp216s0) stat: 7843 ( 7,843) <= ch22_poll /sec
Ethtool(enp216s0) stat: 7091 ( 7,091) <= ch23_arm /sec
Ethtool(enp216s0) stat: 8367 ( 8,367) <= ch23_events /sec
Ethtool(enp216s0) stat: 9344 ( 9,344) <= ch23_poll /sec
Ethtool(enp216s0) stat: 5481 ( 5,481) <= ch24_arm /sec
Ethtool(enp216s0) stat: 6879 ( 6,879) <= ch24_events /sec
Ethtool(enp216s0) stat: 7995 ( 7,995) <= ch24_poll /sec
Ethtool(enp216s0) stat: 5642 ( 5,642) <= ch25_arm /sec
Ethtool(enp216s0) stat: 6959 ( 6,959) <= ch25_events /sec
Ethtool(enp216s0) stat: 7927 ( 7,927) <= ch25_poll /sec
Ethtool(enp216s0) stat: 5289 ( 5,289) <= ch26_arm /sec
Ethtool(enp216s0) stat: 6643 ( 6,643) <= ch26_events /sec
Ethtool(enp216s0) stat: 7691 ( 7,691) <= ch26_poll /sec
Ethtool(enp216s0) stat: 7313 ( 7,313) <= ch27_arm /sec
Ethtool(enp216s0) stat: 8719 ( 8,719) <= ch27_events /sec
Ethtool(enp216s0) stat: 9876 ( 9,876) <= ch27_poll /sec
Ethtool(enp216s0) stat: 7791 ( 7,791) <= ch2_arm /sec
Ethtool(enp216s0) stat: 9328 ( 9,328) <= ch2_events /sec
Ethtool(enp216s0) stat: 10838 ( 10,838) <= ch2_poll /sec
Ethtool(enp216s0) stat: 9595 ( 9,595) <= ch3_arm /sec
Ethtool(enp216s0) stat: 11015 ( 11,015) <= ch3_events /sec
Ethtool(enp216s0) stat: 12402 ( 12,402) <= ch3_poll /sec
Ethtool(enp216s0) stat: 9653 ( 9,653) <= ch4_arm /sec
Ethtool(enp216s0) stat: 11163 ( 11,163) <= ch4_events /sec
Ethtool(enp216s0) stat: 12553 ( 12,553) <= ch4_poll /sec
Ethtool(enp216s0) stat: 10269 ( 10,269) <= ch5_arm /sec
Ethtool(enp216s0) stat: 11727 ( 11,727) <= ch5_events /sec
Ethtool(enp216s0) stat: 13160 ( 13,160) <= ch5_poll /sec
Ethtool(enp216s0) stat: 8806 ( 8,806) <= ch6_arm /sec
Ethtool(enp216s0) stat: 10191 ( 10,191) <= ch6_events /sec
Ethtool(enp216s0) stat: 11431 ( 11,431) <= ch6_poll /sec
Ethtool(enp216s0) stat: 6866 ( 6,866) <= ch7_arm /sec
Ethtool(enp216s0) stat: 8412 ( 8,412) <= ch7_events /sec
Ethtool(enp216s0) stat: 9815 ( 9,815) <= ch7_poll /sec
Ethtool(enp216s0) stat: 10533 ( 10,533) <= ch8_arm /sec
Ethtool(enp216s0) stat: 11945 ( 11,945) <= ch8_events /sec
Ethtool(enp216s0) stat: 13286 ( 13,286) <= ch8_poll /sec
Ethtool(enp216s0) stat: 7126 ( 7,126) <= ch9_arm /sec
Ethtool(enp216s0) stat: 8643 ( 8,643) <= ch9_events /sec
Ethtool(enp216s0) stat: 10210 ( 10,210) <= ch9_poll /sec
Ethtool(enp216s0) stat: 212206 ( 212,206) <= ch_arm /sec
Ethtool(enp216s0) stat: 252118 ( 252,118) <= ch_events /sec
Ethtool(enp216s0) stat: 288009 ( 288,009) <= ch_poll /sec
Ethtool(enp216s0) stat: 93281134 ( 93,281,134) <= rx0_bytes /sec
Ethtool(enp216s0) stat: 94200 ( 94,200) <=
rx0_cache_reuse /sec
Ethtool(enp216s0) stat: 188327 ( 188,327) <=
rx0_csum_complete /sec
Ethtool(enp216s0) stat: 7 ( 7) <= rx0_csum_none /sec
Ethtool(enp216s0) stat: 188334 ( 188,334) <= rx0_packets /sec
Ethtool(enp216s0) stat: 188334 ( 188,334) <=
rx0_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 102443052 ( 102,443,052) <= rx10_bytes /sec
Ethtool(enp216s0) stat: 99816 ( 99,816) <=
rx10_cache_reuse /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <=
rx10_csum_complete /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <= rx10_packets /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <=
rx10_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 97655592 ( 97,655,592) <= rx11_bytes /sec
Ethtool(enp216s0) stat: 100854 ( 100,854) <=
rx11_cache_reuse /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <=
rx11_csum_complete /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <= rx11_packets /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <=
rx11_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 97076715 ( 97,076,715) <= rx12_bytes /sec
Ethtool(enp216s0) stat: 94078 ( 94,078) <=
rx12_cache_reuse /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <=
rx12_csum_complete /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <= rx12_packets /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <=
rx12_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 120547921 ( 120,547,921) <= rx13_bytes /sec
Ethtool(enp216s0) stat: 101709 ( 101,709) <=
rx13_cache_reuse /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <=
rx13_csum_complete /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <= rx13_packets /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <=
rx13_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 98367936 ( 98,367,936) <= rx14_bytes /sec
Ethtool(enp216s0) stat: 97741 ( 97,741) <=
rx14_cache_reuse /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <=
rx14_csum_complete /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <= rx14_packets /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <=
rx14_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 106726542 ( 106,726,542) <= rx15_bytes /sec
Ethtool(enp216s0) stat: 99694 ( 99,694) <=
rx15_cache_reuse /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <=
rx15_csum_complete /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <= rx15_packets /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <=
rx15_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 81928969 ( 81,928,969) <= rx16_bytes /sec
Ethtool(enp216s0) stat: 75580 ( 75,580) <=
rx16_cache_reuse /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <=
rx16_csum_complete /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <= rx16_packets /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <=
rx16_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 90509227 ( 90,509,227) <= rx17_bytes /sec
Ethtool(enp216s0) stat: 81196 ( 81,196) <=
rx17_cache_reuse /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <=
rx17_csum_complete /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <= rx17_packets /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <=
rx17_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 87854385 ( 87,854,385) <= rx18_bytes /sec
Ethtool(enp216s0) stat: 76923 ( 76,923) <=
rx18_cache_reuse /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <=
rx18_csum_complete /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <= rx18_packets /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <=
rx18_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 67849725 ( 67,849,725) <= rx19_bytes /sec
Ethtool(enp216s0) stat: 71001 ( 71,001) <=
rx19_cache_reuse /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <=
rx19_csum_complete /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <= rx19_packets /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <=
rx19_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 92611021 ( 92,611,021) <= rx1_bytes /sec
Ethtool(enp216s0) stat: 92307 ( 92,307) <=
rx1_cache_reuse /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <=
rx1_csum_complete /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <= rx1_packets /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <=
rx1_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88902617 ( 88,902,617) <= rx20_bytes /sec
Ethtool(enp216s0) stat: 82844 ( 82,844) <=
rx20_cache_reuse /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <=
rx20_csum_complete /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <= rx20_packets /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <=
rx20_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 98096942 ( 98,096,942) <= rx21_bytes /sec
Ethtool(enp216s0) stat: 79975 ( 79,975) <=
rx21_cache_reuse /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <=
rx21_csum_complete /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <= rx21_packets /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <=
rx21_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 71885445 ( 71,885,445) <= rx22_bytes /sec
Ethtool(enp216s0) stat: 73565 ( 73,565) <=
rx22_cache_reuse /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <=
rx22_csum_complete /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <= rx22_packets /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <=
rx22_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 77030721 ( 77,030,721) <= rx23_bytes /sec
Ethtool(enp216s0) stat: 74481 ( 74,481) <=
rx23_cache_reuse /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <=
rx23_csum_complete /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <= rx23_packets /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <=
rx23_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 69603951 ( 69,603,951) <= rx24_bytes /sec
Ethtool(enp216s0) stat: 77472 ( 77,472) <=
rx24_cache_reuse /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <=
rx24_csum_complete /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <= rx24_packets /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <=
rx24_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 62277522 ( 62,277,522) <= rx25_bytes /sec
Ethtool(enp216s0) stat: 69414 ( 69,414) <=
rx25_cache_reuse /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <=
rx25_csum_complete /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <= rx25_packets /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <=
rx25_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 77498258 ( 77,498,258) <= rx26_bytes /sec
Ethtool(enp216s0) stat: 72466 ( 72,466) <=
rx26_cache_reuse /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <=
rx26_csum_complete /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <= rx26_packets /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <=
rx26_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88962856 ( 88,962,856) <= rx27_bytes /sec
Ethtool(enp216s0) stat: 80952 ( 80,952) <=
rx27_cache_reuse /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <=
rx27_csum_complete /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <= rx27_packets /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <=
rx27_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 103624977 ( 103,624,977) <= rx2_bytes /sec
Ethtool(enp216s0) stat: 101098 ( 101,098) <=
rx2_cache_reuse /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <=
rx2_csum_complete /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <= rx2_packets /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <=
rx2_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 87213368 ( 87,213,368) <= rx3_bytes /sec
Ethtool(enp216s0) stat: 99877 ( 99,877) <=
rx3_cache_reuse /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <=
rx3_csum_complete /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <= rx3_packets /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <=
rx3_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 110458845 ( 110,458,845) <= rx4_bytes /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx4_cache_busy
/sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx4_cache_full
/sec
Ethtool(enp216s0) stat: 95523 ( 95,523) <=
rx4_cache_reuse /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <=
rx4_csum_complete /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <= rx4_packets /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <=
rx4_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 121222062 ( 121,222,062) <= rx5_bytes /sec
Ethtool(enp216s0) stat: 105616 ( 105,616) <=
rx5_cache_reuse /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <=
rx5_csum_complete /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <= rx5_packets /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <=
rx5_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 85541470 ( 85,541,470) <= rx6_bytes /sec
Ethtool(enp216s0) stat: 91147 ( 91,147) <=
rx6_cache_reuse /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <=
rx6_csum_complete /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <= rx6_packets /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <=
rx6_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88443649 ( 88,443,649) <= rx7_bytes /sec
Ethtool(enp216s0) stat: 92368 ( 92,368) <=
rx7_cache_reuse /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <=
rx7_csum_complete /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <= rx7_packets /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <=
rx7_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 108419072 ( 108,419,072) <= rx8_bytes /sec
Ethtool(enp216s0) stat: 101098 ( 101,098) <=
rx8_cache_reuse /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <=
rx8_csum_complete /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <= rx8_packets /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <=
rx8_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 116210958 ( 116,210,958) <= rx9_bytes /sec
Ethtool(enp216s0) stat: 102014 ( 102,014) <=
rx9_cache_reuse /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <=
rx9_csum_complete /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <= rx9_packets /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <=
rx9_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 1065697 ( 1,065,697) <=
rx_1024_to_1518_bytes_phy /sec
Ethtool(enp216s0) stat: 141705 ( 141,705) <=
rx_128_to_255_bytes_phy /sec
Ethtool(enp216s0) stat: 512465 ( 512,465) <=
rx_1519_to_2047_bytes_phy /sec
Ethtool(enp216s0) stat: 49523 ( 49,523) <=
rx_256_to_511_bytes_phy /sec
Ethtool(enp216s0) stat: 56321 ( 56,321) <=
rx_512_to_1023_bytes_phy /sec
Ethtool(enp216s0) stat: 957286 ( 957,286) <=
rx_64_bytes_phy /sec
Ethtool(enp216s0) stat: 2193644 ( 2,193,644) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_broadcast_phy /sec
Ethtool(enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
Ethtool(enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx_cache_busy /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx_cache_full /sec
Ethtool(enp216s0) stat: 2484928 ( 2,484,928) <= rx_cache_reuse
/sec
Ethtool(enp216s0) stat: 4971670 ( 4,971,670) <=
rx_csum_complete /sec
Ethtool(enp216s0) stat: 7 ( 7) <= rx_csum_none /sec
Ethtool(enp216s0) stat: 464 ( 464) <=
rx_discards_phy /sec
Ethtool(enp216s0) stat: 1376 ( 1,376) <=
rx_multicast_phy /sec
Ethtool(enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
Ethtool(enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy
/sec
Ethtool(enp216s0) stat: 2634727119 ( 2,634,727,119) <= rx_prio0_bytes
/sec
Ethtool(enp216s0) stat: 4975012 ( 4,975,012) <=
rx_prio0_packets /sec
Ethtool(enp216s0) stat: 61 ( 61) <= rx_prio1_bytes
/sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_prio1_packets /sec
Ethtool(enp216s0) stat: 2003 ( 2,003) <= rx_prio3_bytes
/sec
Ethtool(enp216s0) stat: 7 ( 7) <=
rx_prio3_packets /sec
Ethtool(enp216s0) stat: 2497 ( 2,497) <= rx_prio4_bytes
/sec
Ethtool(enp216s0) stat: 3 ( 3) <=
rx_prio4_packets /sec
Ethtool(enp216s0) stat: 10697 ( 10,697) <= rx_prio5_bytes
/sec
Ethtool(enp216s0) stat: 163 ( 163) <=
rx_prio5_packets /sec
Ethtool(enp216s0) stat: 3519 ( 3,519) <= rx_prio6_bytes
/sec
Ethtool(enp216s0) stat: 26 ( 26) <=
rx_prio6_packets /sec
Ethtool(enp216s0) stat: 95583 ( 95,583) <= rx_prio7_bytes
/sec
Ethtool(enp216s0) stat: 1365 ( 1,365) <=
rx_prio7_packets /sec
Ethtool(enp216s0) stat: 4971677 ( 4,971,677) <=
rx_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 1377 ( 1,377) <=
rx_steer_missed_packets /sec
Ethtool(enp216s0) stat: 150 ( 150) <=
rx_vport_broadcast_bytes /sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_vport_broadcast_packets /sec
Ethtool(enp216s0) stat: 90890 ( 90,890) <=
rx_vport_multicast_bytes /sec
Ethtool(enp216s0) stat: 1376 ( 1,376) <=
rx_vport_multicast_packets /sec
Ethtool(enp216s0) stat: 2612989221 ( 2,612,989,221) <=
rx_vport_unicast_bytes /sec
Ethtool(enp216s0) stat: 4973191 ( 4,973,191) <=
rx_vport_unicast_packets /sec
Ethtool(enp216s0) stat: 82580 ( 82,580) <=
tx0_added_vlan_packets /sec
Ethtool(enp216s0) stat: 186927644 ( 186,927,644) <= tx0_bytes /sec
Ethtool(enp216s0) stat: 81347 ( 81,347) <= tx0_cqes /sec
Ethtool(enp216s0) stat: 49836 ( 49,836) <= tx0_csum_none /sec
Ethtool(enp216s0) stat: 32745 ( 32,745) <=
tx0_csum_partial /sec
Ethtool(enp216s0) stat: 2093 ( 2,093) <= tx0_nop /sec
Ethtool(enp216s0) stat: 146450 ( 146,450) <= tx0_packets /sec
Ethtool(enp216s0) stat: 131547357 ( 131,547,357) <= tx0_tso_bytes /sec
Ethtool(enp216s0) stat: 28323 ( 28,323) <=
tx0_tso_packets /sec
Ethtool(enp216s0) stat: 1226 ( 1,226) <= tx0_xmit_more /sec
Ethtool(enp216s0) stat: 85851 ( 85,851) <=
tx10_added_vlan_packets /sec
Ethtool(enp216s0) stat: 198440153 ( 198,440,153) <= tx10_bytes /sec
Ethtool(enp216s0) stat: 84107 ( 84,107) <= tx10_cqes /sec
Ethtool(enp216s0) stat: 51644 ( 51,644) <= tx10_csum_none
/sec
Ethtool(enp216s0) stat: 34207 ( 34,207) <=
tx10_csum_partial /sec
Ethtool(enp216s0) stat: 2256 ( 2,256) <= tx10_nop /sec
Ethtool(enp216s0) stat: 151642 ( 151,642) <= tx10_packets /sec
Ethtool(enp216s0) stat: 135145744 ( 135,145,744) <= tx10_tso_bytes
/sec
Ethtool(enp216s0) stat: 29140 ( 29,140) <=
tx10_tso_packets /sec
Ethtool(enp216s0) stat: 1739 ( 1,739) <= tx10_xmit_more
/sec
Ethtool(enp216s0) stat: 85526 ( 85,526) <=
tx11_added_vlan_packets /sec
Ethtool(enp216s0) stat: 199463985 ( 199,463,985) <= tx11_bytes /sec
Ethtool(enp216s0) stat: 83793 ( 83,793) <= tx11_cqes /sec
Ethtool(enp216s0) stat: 50188 ( 50,188) <= tx11_csum_none
/sec
Ethtool(enp216s0) stat: 35338 ( 35,338) <=
tx11_csum_partial /sec
Ethtool(enp216s0) stat: 2227 ( 2,227) <= tx11_nop /sec
Ethtool(enp216s0) stat: 152815 ( 152,815) <= tx11_packets /sec
Ethtool(enp216s0) stat: 138134263 ( 138,134,263) <= tx11_tso_bytes
/sec
Ethtool(enp216s0) stat: 29948 ( 29,948) <=
tx11_tso_packets /sec
Ethtool(enp216s0) stat: 1733 ( 1,733) <= tx11_xmit_more
/sec
Ethtool(enp216s0) stat: 77464 ( 77,464) <=
tx12_added_vlan_packets /sec
Ethtool(enp216s0) stat: 185723223 ( 185,723,223) <= tx12_bytes /sec
Ethtool(enp216s0) stat: 75907 ( 75,907) <= tx12_cqes /sec
Ethtool(enp216s0) stat: 43727 ( 43,727) <= tx12_csum_none
/sec
Ethtool(enp216s0) stat: 33738 ( 33,738) <=
tx12_csum_partial /sec
Ethtool(enp216s0) stat: 2060 ( 2,060) <= tx12_nop /sec
Ethtool(enp216s0) stat: 141903 ( 141,903) <= tx12_packets /sec
Ethtool(enp216s0) stat: 133227574 ( 133,227,574) <= tx12_tso_bytes
/sec
Ethtool(enp216s0) stat: 29139 ( 29,139) <=
tx12_tso_packets /sec
Ethtool(enp216s0) stat: 1559 ( 1,559) <= tx12_xmit_more
/sec
Ethtool(enp216s0) stat: 79682 ( 79,682) <=
tx13_added_vlan_packets /sec
Ethtool(enp216s0) stat: 189943899 ( 189,943,899) <= tx13_bytes /sec
Ethtool(enp216s0) stat: 78110 ( 78,110) <= tx13_cqes /sec
Ethtool(enp216s0) stat: 46294 ( 46,294) <= tx13_csum_none
/sec
Ethtool(enp216s0) stat: 33388 ( 33,388) <=
tx13_csum_partial /sec
Ethtool(enp216s0) stat: 2109 ( 2,109) <= tx13_nop /sec
Ethtool(enp216s0) stat: 145877 ( 145,877) <= tx13_packets /sec
Ethtool(enp216s0) stat: 136076991 ( 136,076,991) <= tx13_tso_bytes
/sec
Ethtool(enp216s0) stat: 29367 ( 29,367) <=
tx13_tso_packets /sec
Ethtool(enp216s0) stat: 1572 ( 1,572) <= tx13_xmit_more
/sec
Ethtool(enp216s0) stat: 86096 ( 86,096) <=
tx14_added_vlan_packets /sec
Ethtool(enp216s0) stat: 199318635 ( 199,318,635) <= tx14_bytes /sec
Ethtool(enp216s0) stat: 84335 ( 84,335) <= tx14_cqes /sec
Ethtool(enp216s0) stat: 50934 ( 50,934) <= tx14_csum_none
/sec
Ethtool(enp216s0) stat: 35163 ( 35,163) <=
tx14_csum_partial /sec
Ethtool(enp216s0) stat: 2263 ( 2,263) <= tx14_nop /sec
Ethtool(enp216s0) stat: 156445 ( 156,445) <= tx14_packets /sec
Ethtool(enp216s0) stat: 142293555 ( 142,293,555) <= tx14_tso_bytes
/sec
Ethtool(enp216s0) stat: 29703 ( 29,703) <=
tx14_tso_packets /sec
Ethtool(enp216s0) stat: 1761 ( 1,761) <= tx14_xmit_more
/sec
Ethtool(enp216s0) stat: 79698 ( 79,698) <=
tx15_added_vlan_packets /sec
Ethtool(enp216s0) stat: 189620424 ( 189,620,424) <= tx15_bytes /sec
Ethtool(enp216s0) stat: 78356 ( 78,356) <= tx15_cqes /sec
Ethtool(enp216s0) stat: 45528 ( 45,528) <= tx15_csum_none
/sec
Ethtool(enp216s0) stat: 34170 ( 34,170) <=
tx15_csum_partial /sec
Ethtool(enp216s0) stat: 2156 ( 2,156) <= tx15_nop /sec
Ethtool(enp216s0) stat: 144935 ( 144,935) <= tx15_packets /sec
Ethtool(enp216s0) stat: 135023821 ( 135,023,821) <= tx15_tso_bytes
/sec
Ethtool(enp216s0) stat: 29400 ( 29,400) <=
tx15_tso_packets /sec
Ethtool(enp216s0) stat: 1344 ( 1,344) <= tx15_xmit_more
/sec
Ethtool(enp216s0) stat: 59598 ( 59,598) <=
tx16_added_vlan_packets /sec
Ethtool(enp216s0) stat: 143187495 ( 143,187,495) <= tx16_bytes /sec
Ethtool(enp216s0) stat: 58408 ( 58,408) <= tx16_cqes /sec
Ethtool(enp216s0) stat: 35002 ( 35,002) <= tx16_csum_none
/sec
Ethtool(enp216s0) stat: 24595 ( 24,595) <=
tx16_csum_partial /sec
Ethtool(enp216s0) stat: 1585 ( 1,585) <= tx16_nop /sec
Ethtool(enp216s0) stat: 110250 ( 110,250) <= tx16_packets /sec
Ethtool(enp216s0) stat: 101339613 ( 101,339,613) <= tx16_tso_bytes
/sec
Ethtool(enp216s0) stat: 20698 ( 20,698) <=
tx16_tso_packets /sec
Ethtool(enp216s0) stat: 1179 ( 1,179) <= tx16_xmit_more
/sec
Ethtool(enp216s0) stat: 69504 ( 69,504) <=
tx17_added_vlan_packets /sec
Ethtool(enp216s0) stat: 171534675 ( 171,534,675) <= tx17_bytes /sec
Ethtool(enp216s0) stat: 68155 ( 68,155) <= tx17_cqes /sec
Ethtool(enp216s0) stat: 39445 ( 39,445) <= tx17_csum_none
/sec
Ethtool(enp216s0) stat: 30059 ( 30,059) <=
tx17_csum_partial /sec
Ethtool(enp216s0) stat: 1886 ( 1,886) <= tx17_nop /sec
Ethtool(enp216s0) stat: 130910 ( 130,910) <= tx17_packets /sec
Ethtool(enp216s0) stat: 123978012 ( 123,978,012) <= tx17_tso_bytes
/sec
Ethtool(enp216s0) stat: 26215 ( 26,215) <=
tx17_tso_packets /sec
Ethtool(enp216s0) stat: 1349 ( 1,349) <= tx17_xmit_more
/sec
Ethtool(enp216s0) stat: 58880 ( 58,880) <=
tx18_added_vlan_packets /sec
Ethtool(enp216s0) stat: 141863299 ( 141,863,299) <= tx18_bytes /sec
Ethtool(enp216s0) stat: 57755 ( 57,755) <= tx18_cqes /sec
Ethtool(enp216s0) stat: 33875 ( 33,875) <= tx18_csum_none
/sec
Ethtool(enp216s0) stat: 25005 ( 25,005) <=
tx18_csum_partial /sec
Ethtool(enp216s0) stat: 1592 ( 1,592) <= tx18_nop /sec
Ethtool(enp216s0) stat: 110248 ( 110,248) <= tx18_packets /sec
Ethtool(enp216s0) stat: 103105544 ( 103,105,544) <= tx18_tso_bytes
/sec
Ethtool(enp216s0) stat: 21243 ( 21,243) <=
tx18_tso_packets /sec
Ethtool(enp216s0) stat: 1108 ( 1,108) <= tx18_xmit_more
/sec
Ethtool(enp216s0) stat: 69804 ( 69,804) <=
tx19_added_vlan_packets /sec
Ethtool(enp216s0) stat: 164225730 ( 164,225,730) <= tx19_bytes /sec
Ethtool(enp216s0) stat: 68756 ( 68,756) <= tx19_cqes /sec
Ethtool(enp216s0) stat: 42003 ( 42,003) <= tx19_csum_none
/sec
Ethtool(enp216s0) stat: 27801 ( 27,801) <=
tx19_csum_partial /sec
Ethtool(enp216s0) stat: 1790 ( 1,790) <= tx19_nop /sec
Ethtool(enp216s0) stat: 124794 ( 124,794) <= tx19_packets /sec
Ethtool(enp216s0) stat: 110814620 ( 110,814,620) <= tx19_tso_bytes
/sec
Ethtool(enp216s0) stat: 23278 ( 23,278) <=
tx19_tso_packets /sec
Ethtool(enp216s0) stat: 1045 ( 1,045) <= tx19_xmit_more
/sec
Ethtool(enp216s0) stat: 79346 ( 79,346) <=
tx1_added_vlan_packets /sec
Ethtool(enp216s0) stat: 181179251 ( 181,179,251) <= tx1_bytes /sec
Ethtool(enp216s0) stat: 78062 ( 78,062) <= tx1_cqes /sec
Ethtool(enp216s0) stat: 46525 ( 46,525) <= tx1_csum_none /sec
Ethtool(enp216s0) stat: 32821 ( 32,821) <=
tx1_csum_partial /sec
Ethtool(enp216s0) stat: 1996 ( 1,996) <= tx1_nop /sec
Ethtool(enp216s0) stat: 142716 ( 142,716) <= tx1_packets /sec
Ethtool(enp216s0) stat: 129507562 ( 129,507,562) <= tx1_tso_bytes /sec
Ethtool(enp216s0) stat: 27579 ( 27,579) <=
tx1_tso_packets /sec
Ethtool(enp216s0) stat: 1281 ( 1,281) <= tx1_xmit_more /sec
Ethtool(enp216s0) stat: 66641 ( 66,641) <=
tx20_added_vlan_packets /sec
Ethtool(enp216s0) stat: 161452661 ( 161,452,661) <= tx20_bytes /sec
Ethtool(enp216s0) stat: 65374 ( 65,374) <= tx20_cqes /sec
Ethtool(enp216s0) stat: 37657 ( 37,657) <= tx20_csum_none
/sec
Ethtool(enp216s0) stat: 28983 ( 28,983) <=
tx20_csum_partial /sec
Ethtool(enp216s0) stat: 1739 ( 1,739) <= tx20_nop /sec
Ethtool(enp216s0) stat: 122824 ( 122,824) <= tx20_packets /sec
Ethtool(enp216s0) stat: 115707977 ( 115,707,977) <= tx20_tso_bytes
/sec
Ethtool(enp216s0) stat: 24823 ( 24,823) <=
tx20_tso_packets /sec
Ethtool(enp216s0) stat: 1263 ( 1,263) <= tx20_xmit_more
/sec
Ethtool(enp216s0) stat: 60564 ( 60,564) <=
tx21_added_vlan_packets /sec
Ethtool(enp216s0) stat: 138260273 ( 138,260,273) <= tx21_bytes /sec
Ethtool(enp216s0) stat: 59611 ( 59,611) <= tx21_cqes /sec
Ethtool(enp216s0) stat: 36105 ( 36,105) <= tx21_csum_none
/sec
Ethtool(enp216s0) stat: 24459 ( 24,459) <=
tx21_csum_partial /sec
Ethtool(enp216s0) stat: 1490 ( 1,490) <= tx21_nop /sec
Ethtool(enp216s0) stat: 106913 ( 106,913) <= tx21_packets /sec
Ethtool(enp216s0) stat: 97216016 ( 97,216,016) <= tx21_tso_bytes
/sec
Ethtool(enp216s0) stat: 21944 ( 21,944) <=
tx21_tso_packets /sec
Ethtool(enp216s0) stat: 930 ( 930) <= tx21_xmit_more
/sec
Ethtool(enp216s0) stat: 66527 ( 66,527) <=
tx22_added_vlan_packets /sec
Ethtool(enp216s0) stat: 149686537 ( 149,686,537) <= tx22_bytes /sec
Ethtool(enp216s0) stat: 65262 ( 65,262) <= tx22_cqes /sec
Ethtool(enp216s0) stat: 40609 ( 40,609) <= tx22_csum_none
/sec
Ethtool(enp216s0) stat: 25918 ( 25,918) <=
tx22_csum_partial /sec
Ethtool(enp216s0) stat: 1676 ( 1,676) <= tx22_nop /sec
Ethtool(enp216s0) stat: 118071 ( 118,071) <= tx22_packets /sec
Ethtool(enp216s0) stat: 104822520 ( 104,822,520) <= tx22_tso_bytes
/sec
Ethtool(enp216s0) stat: 22046 ( 22,046) <=
tx22_tso_packets /sec
Ethtool(enp216s0) stat: 1265 ( 1,265) <= tx22_xmit_more
/sec
Ethtool(enp216s0) stat: 59973 ( 59,973) <=
tx23_added_vlan_packets /sec
Ethtool(enp216s0) stat: 147788578 ( 147,788,578) <= tx23_bytes /sec
Ethtool(enp216s0) stat: 58827 ( 58,827) <= tx23_cqes /sec
Ethtool(enp216s0) stat: 33197 ( 33,197) <= tx23_csum_none
/sec
Ethtool(enp216s0) stat: 26776 ( 26,776) <=
tx23_csum_partial /sec
Ethtool(enp216s0) stat: 1669 ( 1,669) <= tx23_nop /sec
Ethtool(enp216s0) stat: 112053 ( 112,053) <= tx23_packets /sec
Ethtool(enp216s0) stat: 106098597 ( 106,098,597) <= tx23_tso_bytes
/sec
Ethtool(enp216s0) stat: 22433 ( 22,433) <=
tx23_tso_packets /sec
Ethtool(enp216s0) stat: 1126 ( 1,126) <= tx23_xmit_more
/sec
Ethtool(enp216s0) stat: 58819 ( 58,819) <=
tx24_added_vlan_packets /sec
Ethtool(enp216s0) stat: 146231570 ( 146,231,570) <= tx24_bytes /sec
Ethtool(enp216s0) stat: 57661 ( 57,661) <= tx24_cqes /sec
Ethtool(enp216s0) stat: 32150 ( 32,150) <= tx24_csum_none
/sec
Ethtool(enp216s0) stat: 26669 ( 26,669) <=
tx24_csum_partial /sec
Ethtool(enp216s0) stat: 1578 ( 1,578) <= tx24_nop /sec
Ethtool(enp216s0) stat: 111230 ( 111,230) <= tx24_packets /sec
Ethtool(enp216s0) stat: 106359530 ( 106,359,530) <= tx24_tso_bytes
/sec
Ethtool(enp216s0) stat: 22402 ( 22,402) <=
tx24_tso_packets /sec
Ethtool(enp216s0) stat: 1158 ( 1,158) <= tx24_xmit_more
/sec
Ethtool(enp216s0) stat: 64116 ( 64,116) <=
tx25_added_vlan_packets /sec
Ethtool(enp216s0) stat: 156132090 ( 156,132,090) <= tx25_bytes /sec
Ethtool(enp216s0) stat: 62901 ( 62,901) <= tx25_cqes /sec
Ethtool(enp216s0) stat: 36357 ( 36,357) <= tx25_csum_none
/sec
Ethtool(enp216s0) stat: 27759 ( 27,759) <=
tx25_csum_partial /sec
Ethtool(enp216s0) stat: 1717 ( 1,717) <= tx25_nop /sec
Ethtool(enp216s0) stat: 118618 ( 118,618) <= tx25_packets /sec
Ethtool(enp216s0) stat: 110752893 ( 110,752,893) <= tx25_tso_bytes
/sec
Ethtool(enp216s0) stat: 23282 ( 23,282) <=
tx25_tso_packets /sec
Ethtool(enp216s0) stat: 1212 ( 1,212) <= tx25_xmit_more
/sec
Ethtool(enp216s0) stat: 62028 ( 62,028) <=
tx26_added_vlan_packets /sec
Ethtool(enp216s0) stat: 144966495 ( 144,966,495) <= tx26_bytes /sec
Ethtool(enp216s0) stat: 60906 ( 60,906) <= tx26_cqes /sec
Ethtool(enp216s0) stat: 36345 ( 36,345) <= tx26_csum_none
/sec
Ethtool(enp216s0) stat: 25684 ( 25,684) <=
tx26_csum_partial /sec
Ethtool(enp216s0) stat: 1615 ( 1,615) <= tx26_nop /sec
Ethtool(enp216s0) stat: 112487 ( 112,487) <= tx26_packets /sec
Ethtool(enp216s0) stat: 102083200 ( 102,083,200) <= tx26_tso_bytes
/sec
Ethtool(enp216s0) stat: 21598 ( 21,598) <=
tx26_tso_packets /sec
Ethtool(enp216s0) stat: 1123 ( 1,123) <= tx26_xmit_more
/sec
Ethtool(enp216s0) stat: 64029 ( 64,029) <=
tx27_added_vlan_packets /sec
Ethtool(enp216s0) stat: 154747343 ( 154,747,343) <= tx27_bytes /sec
Ethtool(enp216s0) stat: 62881 ( 62,881) <= tx27_cqes /sec
Ethtool(enp216s0) stat: 35343 ( 35,343) <= tx27_csum_none
/sec
Ethtool(enp216s0) stat: 28686 ( 28,686) <=
tx27_csum_partial /sec
Ethtool(enp216s0) stat: 1707 ( 1,707) <= tx27_nop /sec
Ethtool(enp216s0) stat: 118074 ( 118,074) <= tx27_packets /sec
Ethtool(enp216s0) stat: 110960676 ( 110,960,676) <= tx27_tso_bytes
/sec
Ethtool(enp216s0) stat: 24139 ( 24,139) <=
tx27_tso_packets /sec
Ethtool(enp216s0) stat: 1148 ( 1,148) <= tx27_xmit_more
/sec
Ethtool(enp216s0) stat: 82542 ( 82,542) <=
tx2_added_vlan_packets /sec
Ethtool(enp216s0) stat: 183007051 ( 183,007,051) <= tx2_bytes /sec
Ethtool(enp216s0) stat: 81320 ( 81,320) <= tx2_cqes /sec
Ethtool(enp216s0) stat: 49989 ( 49,989) <= tx2_csum_none /sec
Ethtool(enp216s0) stat: 32553 ( 32,553) <=
tx2_csum_partial /sec
Ethtool(enp216s0) stat: 2142 ( 2,142) <= tx2_nop /sec
Ethtool(enp216s0) stat: 143613 ( 143,613) <= tx2_packets /sec
Ethtool(enp216s0) stat: 126895404 ( 126,895,404) <= tx2_tso_bytes /sec
Ethtool(enp216s0) stat: 28173 ( 28,173) <=
tx2_tso_packets /sec
Ethtool(enp216s0) stat: 1220 ( 1,220) <= tx2_xmit_more /sec
Ethtool(enp216s0) stat: 78737 ( 78,737) <=
tx3_added_vlan_packets /sec
Ethtool(enp216s0) stat: 188546556 ( 188,546,556) <= tx3_bytes /sec
Ethtool(enp216s0) stat: 77389 ( 77,389) <= tx3_cqes /sec
Ethtool(enp216s0) stat: 46147 ( 46,147) <= tx3_csum_none /sec
Ethtool(enp216s0) stat: 32590 ( 32,590) <=
tx3_csum_partial /sec
Ethtool(enp216s0) stat: 2171 ( 2,171) <= tx3_nop /sec
Ethtool(enp216s0) stat: 144570 ( 144,570) <= tx3_packets /sec
Ethtool(enp216s0) stat: 134231500 ( 134,231,500) <= tx3_tso_bytes /sec
Ethtool(enp216s0) stat: 28857 ( 28,857) <=
tx3_tso_packets /sec
Ethtool(enp216s0) stat: 1348 ( 1,348) <= tx3_xmit_more /sec
Ethtool(enp216s0) stat: 76284 ( 76,284) <=
tx4_added_vlan_packets /sec
Ethtool(enp216s0) stat: 185633403 ( 185,633,403) <= tx4_bytes /sec
Ethtool(enp216s0) stat: 74710 ( 74,710) <= tx4_cqes /sec
Ethtool(enp216s0) stat: 42803 ( 42,803) <= tx4_csum_none /sec
Ethtool(enp216s0) stat: 33482 ( 33,482) <=
tx4_csum_partial /sec
Ethtool(enp216s0) stat: 2005 ( 2,005) <= tx4_nop /sec
Ethtool(enp216s0) stat: 139978 ( 139,978) <= tx4_packets /sec
Ethtool(enp216s0) stat: 131762891 ( 131,762,891) <= tx4_tso_bytes /sec
Ethtool(enp216s0) stat: 28843 ( 28,843) <=
tx4_tso_packets /sec
Ethtool(enp216s0) stat: 1563 ( 1,563) <= tx4_xmit_more /sec
Ethtool(enp216s0) stat: 83878 ( 83,878) <=
tx5_added_vlan_packets /sec
Ethtool(enp216s0) stat: 197920426 ( 197,920,426) <= tx5_bytes /sec
Ethtool(enp216s0) stat: 82146 ( 82,146) <= tx5_cqes /sec
Ethtool(enp216s0) stat: 48122 ( 48,122) <= tx5_csum_none /sec
Ethtool(enp216s0) stat: 35755 ( 35,755) <=
tx5_csum_partial /sec
Ethtool(enp216s0) stat: 2227 ( 2,227) <= tx5_nop /sec
Ethtool(enp216s0) stat: 151214 ( 151,214) <= tx5_packets /sec
Ethtool(enp216s0) stat: 139113072 ( 139,113,072) <= tx5_tso_bytes /sec
Ethtool(enp216s0) stat: 30390 ( 30,390) <=
tx5_tso_packets /sec
Ethtool(enp216s0) stat: 1730 ( 1,730) <= tx5_xmit_more /sec
Ethtool(enp216s0) stat: 76307 ( 76,307) <=
tx6_added_vlan_packets /sec
Ethtool(enp216s0) stat: 182153538 ( 182,153,538) <= tx6_bytes /sec
Ethtool(enp216s0) stat: 74744 ( 74,744) <= tx6_cqes /sec
Ethtool(enp216s0) stat: 44381 ( 44,381) <= tx6_csum_none /sec
Ethtool(enp216s0) stat: 31926 ( 31,926) <=
tx6_csum_partial /sec
Ethtool(enp216s0) stat: 2068 ( 2,068) <= tx6_nop /sec
Ethtool(enp216s0) stat: 139745 ( 139,745) <= tx6_packets /sec
Ethtool(enp216s0) stat: 129175625 ( 129,175,625) <= tx6_tso_bytes /sec
Ethtool(enp216s0) stat: 27349 ( 27,349) <=
tx6_tso_packets /sec
Ethtool(enp216s0) stat: 1562 ( 1,562) <= tx6_xmit_more /sec
Ethtool(enp216s0) stat: 72151 ( 72,151) <=
tx7_added_vlan_packets /sec
Ethtool(enp216s0) stat: 175340247 ( 175,340,247) <= tx7_bytes /sec
Ethtool(enp216s0) stat: 70565 ( 70,565) <= tx7_cqes /sec
Ethtool(enp216s0) stat: 40824 ( 40,824) <= tx7_csum_none /sec
Ethtool(enp216s0) stat: 31327 ( 31,327) <=
tx7_csum_partial /sec
Ethtool(enp216s0) stat: 1962 ( 1,962) <= tx7_nop /sec
Ethtool(enp216s0) stat: 133790 ( 133,790) <= tx7_packets /sec
Ethtool(enp216s0) stat: 124931972 ( 124,931,972) <= tx7_tso_bytes /sec
Ethtool(enp216s0) stat: 26314 ( 26,314) <=
tx7_tso_packets /sec
Ethtool(enp216s0) stat: 1586 ( 1,586) <= tx7_xmit_more /sec
Ethtool(enp216s0) stat: 83983 ( 83,983) <=
tx8_added_vlan_packets /sec
Ethtool(enp216s0) stat: 203970262 ( 203,970,262) <= tx8_bytes /sec
Ethtool(enp216s0) stat: 82475 ( 82,475) <= tx8_cqes /sec
Ethtool(enp216s0) stat: 47937 ( 47,937) <= tx8_csum_none /sec
Ethtool(enp216s0) stat: 36046 ( 36,046) <=
tx8_csum_partial /sec
Ethtool(enp216s0) stat: 2203 ( 2,203) <= tx8_nop /sec
Ethtool(enp216s0) stat: 153743 ( 153,743) <= tx8_packets /sec
Ethtool(enp216s0) stat: 143296525 ( 143,296,525) <= tx8_tso_bytes /sec
Ethtool(enp216s0) stat: 30936 ( 30,936) <=
tx8_tso_packets /sec
Ethtool(enp216s0) stat: 1500 ( 1,500) <= tx8_xmit_more /sec
Ethtool(enp216s0) stat: 79406 ( 79,406) <=
tx9_added_vlan_packets /sec
Ethtool(enp216s0) stat: 183286836 ( 183,286,836) <= tx9_bytes /sec
Ethtool(enp216s0) stat: 77769 ( 77,769) <= tx9_cqes /sec
Ethtool(enp216s0) stat: 48014 ( 48,014) <= tx9_csum_none /sec
Ethtool(enp216s0) stat: 31392 ( 31,392) <=
tx9_csum_partial /sec
Ethtool(enp216s0) stat: 2146 ( 2,146) <= tx9_nop /sec
Ethtool(enp216s0) stat: 142229 ( 142,229) <= tx9_packets /sec
Ethtool(enp216s0) stat: 127419807 ( 127,419,807) <= tx9_tso_bytes /sec
Ethtool(enp216s0) stat: 26644 ( 26,644) <=
tx9_tso_packets /sec
Ethtool(enp216s0) stat: 1633 ( 1,633) <= tx9_xmit_more /sec
Ethtool(enp216s0) stat: 2050335 ( 2,050,335) <=
tx_added_vlan_packets /sec
Ethtool(enp216s0) stat: 4851410368 ( 4,851,410,368) <= tx_bytes /sec
Ethtool(enp216s0) stat: 4881854912 ( 4,881,854,912) <= tx_bytes_phy /sec
Ethtool(enp216s0) stat: 2011989 ( 2,011,989) <= tx_cqes /sec
Ethtool(enp216s0) stat: 1191137 ( 1,191,137) <= tx_csum_none /sec
Ethtool(enp216s0) stat: 859197 ( 859,197) <=
tx_csum_partial /sec
Ethtool(enp216s0) stat: 54140 ( 54,140) <= tx_nop /sec
Ethtool(enp216s0) stat: 3728759 ( 3,728,759) <= tx_packets /sec
Ethtool(enp216s0) stat: 3729416 ( 3,729,416) <= tx_packets_phy
/sec
Ethtool(enp216s0) stat: 4882774572 ( 4,882,774,572) <= tx_prio0_bytes
/sec
Ethtool(enp216s0) stat: 3730195 ( 3,730,195) <=
tx_prio0_packets /sec
Ethtool(enp216s0) stat: 3431642829 ( 3,431,642,829) <= tx_tso_bytes /sec
Ethtool(enp216s0) stat: 734343 ( 734,343) <= tx_tso_packets
/sec
Ethtool(enp216s0) stat: 4866327192 ( 4,866,327,192) <=
tx_vport_unicast_bytes /sec
Ethtool(enp216s0) stat: 3728966 ( 3,728,966) <=
tx_vport_unicast_packets /sec
Ethtool(enp216s0) stat: 38268 ( 38,268) <= tx_xmit_more /sec
>>
>> Can you give output put from:
>> $ ethtool --show-priv-flag DEVICE
>>
>> I want you to experiment with:
> ethtool --show-priv-flags enp175s0
> Private flags for enp175s0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : off
> rx_striding_rq : on
> rx_no_csum_complete: off
>
>>
>> ethtool --set-priv-flags DEVICE rx_striding_rq off
> ok i will first check on test server if this will reset my interface
> and will not produce kernel panic :)
>>
>> I think you already have played with 'rx_cqe_compress', right.
> yes - and compress increasing number of irq's but doing not much for
> bandwidth same limit 60-64Gbit/s total RX+TX on one 100G port
>
> And what is weird - that limit is in overall symetric - cause if for
> example 100G port is receiving 42G traffic and transmitting 20G
> traffic - when i flood rx side with pktgen or other for example icmp
> traffic 1/2/3/4/5G - then receiving side increase with 1/2/3/4/5Gbit
> of traffic but transmitting is going down for same lvl's
>
>
>
>
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 20:02 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <20181110203409.482f39ec@redhat.com>
W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> I want you to experiment with:
>
> ethtool --set-priv-flags DEVICE rx_striding_rq off
just checked that previously connectx4 was have thos disabled:
ethtool --show-priv-flags enp175s0f0
Private flags for enp175s0f0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : off
rx_striding_rq : off
rx_no_csum_complete: off
So now we are on connectx5 and we have enabled - for sure connectx5
changed cpu load - where i have now max 50/60% cpu where with connectx4
there was sometimes near 100% with same configuration.
^ permalink raw reply
* From Mr Hamzak Wadrago;
From: Hamzak Wadrago @ 2018-11-10 20:56 UTC (permalink / raw)
I am Mr hamzak wadrago, the chief operating officer with my bank and I
want to inform you that an amount of US$15.5 million will be moved on
your name as the Foreign Business Partner to our late deceased
customer. I need your help to receive this money as we shall share the
money in the ratio of 60:40%. You will receive this amount through a
bank wire transfer.
Please send your full names, direct telephone numbers, and physical
address, more details and text of application form will be given to
you upon your reply.
Your quick response will be highly appreciated.
Yours sincerely,
Mr. Hamzak Wadrago,
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Jesper Dangaard Brouer @ 2018-11-10 21:01 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev@vger.kernel.org, brouer
In-Reply-To: <5d44e0d9-d8d3-1b07-5392-838e4096ca74@itcare.pl>
On Sat, 10 Nov 2018 21:02:10 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> > I want you to experiment with:
> >
> > ethtool --set-priv-flags DEVICE rx_striding_rq off
>
> just checked that previously connectx4 was have thos disabled:
> ethtool --show-priv-flags enp175s0f0
>
> Private flags for enp175s0f0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : off
> rx_striding_rq : off
> rx_no_csum_complete: off
>
The CX4 hardware does not have this feature (p.s. the CX4-Lx does).
> So now we are on connectx5 and we have enabled - for sure connectx5
> changed cpu load - where i have now max 50/60% cpu where with connectx4
> there was sometimes near 100% with same configuration.
I (strongly) believe the CPU load was related to the page-alloactor
lock congestion, that Aaron fixed.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox