* Re: [RFC PATCH] lib: Introduce generic __cmpxchg_u64() and use it where needed
From: Dmitry Vyukov @ 2018-11-01 17:46 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Paul E. McKenney, Trond Myklebust, mark.rutland@arm.com,
linux-kernel@vger.kernel.org, ralf@linux-mips.org,
jlayton@kernel.org, linuxppc-dev@lists.ozlabs.org,
bfields@fieldses.org, linux-mips@linux-mips.org,
linux@roeck-us.net, linux-nfs@vger.kernel.org,
akpm@linux-foundation.org, will.deacon@arm.com,
boqun.feng@gmail.com, paul.burton@mips.com,
"anna.schumaker@netapp.com
In-Reply-To: <20181101171846.GI3178@hirez.programming.kicks-ass.net>
On Thu, Nov 1, 2018 at 6:18 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> > > > > > My one question (and the reason why I went with cmpxchg() in the
>> > > > > > first place) would be about the overflow behaviour for
>> > > > > > atomic_fetch_inc() and friends. I believe those functions should
>> > > > > > be OK on x86, so that when we overflow the counter, it behaves
>> > > > > > like an unsigned value and wraps back around. Is that the case
>> > > > > > for all architectures?
>> > > > > >
>> > > > > > i.e. are atomic_t/atomic64_t always guaranteed to behave like
>> > > > > > u32/u64 on increment?
>> > > > > >
>> > > > > > I could not find any documentation that explicitly stated that
>> > > > > > they should.
>> > > > >
>> > > > > Peter, Will, I understand that the atomic_t/atomic64_t ops are
>> > > > > required to wrap per 2's-complement. IIUC the refcount code relies
>> > > > > on this.
>> > > > >
>> > > > > Can you confirm?
>> > > >
>> > > > There is quite a bit of core code that hard assumes 2s-complement.
>> > > > Not only for atomics but for any signed integer type. Also see the
>> > > > kernel using -fno-strict-overflow which implies -fwrapv, which
>> > > > defines signed overflow to behave like 2s-complement (and rids us of
>> > > > that particular UB).
>> > >
>> > > Fair enough, but there have also been bugfixes to explicitly fix unsafe
>> > > C standards assumptions for signed integers. See, for instance commit
>> > > 5a581b367b5d "jiffies: Avoid undefined behavior from signed overflow"
>> > > from Paul McKenney.
>> >
>> > Yes, I feel Paul has been to too many C/C++ committee meetings and got
>> > properly paranoid. Which isn't always a bad thing :-)
>>
>> Even the C standard defines 2s complement for atomics.
>
> Ooh good to know.
>
>> Just not for
>> normal arithmetic, where yes, signed overflow is UB. And yes, I do
>> know about -fwrapv, but I would like to avoid at least some copy-pasta
>> UB from my kernel code to who knows what user-mode environment. :-/
>>
>> At least where it is reasonably easy to do so.
>
> Fair enough I suppose; I just always make sure to include the same
> -fknobs for the userspace thing when I lift code.
>
>> And there is a push to define C++ signed arithmetic as 2s complement,
>> but there are still 1s complement systems with C compilers. Just not
>> C++ compilers. Legacy...
>
> *groan*; how about those ancient hardwares keep using ancient compilers
> and we all move on to the 70s :-)
>
>> > But for us using -fno-strict-overflow which actually defines signed
>> > overflow, I myself am really not worried. I'm also not sure if KASAN has
>> > been taught about this, or if it will still (incorrectly) warn about UB
>> > for signed types.
>>
>> UBSAN gave me a signed-overflow warning a few days ago. Which I have
>> fixed, even though 2s complement did the right thing. I am also taking
>> advantage of the change to use better naming.
>
> Oh too many *SANs I suppose; and yes, if you can make the code better,
> why not.
If there is a warning that we don't want to see at all, then we can
disable it. It supposed to be a useful tool, rather than a thing in
itself that lives own life. We already I think removed 1 particularly
noisy warning and made another optional via a config.
But the thing with overflows is that, even if it's defined, it's not
necessary the intended behavior. For example, take allocation size
calculation done via unsigned size_t. If it overflows it does not help
if C defines result or not, it still gives a user controlled write
primitive. We've seen similar cases with timeout/deadline calculation
in kernel, we really don't want it to just wrap modulo-2, right. Some
user-space projects even test with unsigned overflow warnings or
implicit truncation warnings, which are formally legal, but frequently
bugs.
^ permalink raw reply
* [PATCH] SUNRPC: Use atomic(64)_t for seq_send(64)
From: Paul Burton @ 2018-11-01 17:51 UTC (permalink / raw)
To: linux-nfs@vger.kernel.org, Trond Myklebust, Anna Schumaker
Cc: linux-kernel@vger.kernel.org, Paul Burton, J . Bruce Fields,
Jeff Layton, David S . Miller, netdev@vger.kernel.org
In-Reply-To: <20181101145926.GE3178@hirez.programming.kicks-ass.net>
The seq_send & seq_send64 fields in struct krb5_ctx are used as
atomically incrementing counters. This is implemented using cmpxchg() &
cmpxchg64() to implement what amount to custom versions of
atomic_fetch_inc() & atomic64_fetch_inc().
Besides the duplication, using cmpxchg64() has another major drawback in
that some 32 bit architectures don't provide it. As such commit
571ed1fd2390 ("SUNRPC: Replace krb5_seq_lock with a lockless scheme")
resulted in build failures for some architectures.
Change seq_send to be an atomic_t and seq_send64 to be an atomic64_t,
then use atomic(64)_* functions to manipulate the values. The atomic64_t
type & associated functions are provided even on architectures which
lack real 64 bit atomic memory access via CONFIG_GENERIC_ATOMIC64 which
uses spinlocks to serialize access. This fixes the build failures for
architectures lacking cmpxchg64().
A potential alternative that was raised would be to provide cmpxchg64()
on the 32 bit architectures that currently lack it, using spinlocks.
However this would provide a version of cmpxchg64() with semantics a
little different to the implementations on architectures with real 64
bit atomics - the spinlock-based implementation would only work if all
access to the memory used with cmpxchg64() is *always* performed using
cmpxchg64(). That is not currently a requirement for users of
cmpxchg64(), and making it one seems questionable. As such avoiding
cmpxchg64() outside of architecture-specific code seems best,
particularly in cases where atomic64_t seems like a better fit anyway.
The CONFIG_GENERIC_ATOMIC64 implementation of atomic64_* functions will
use spinlocks & so faces the same issue, but with the key difference
that the memory backing an atomic64_t ought to always be accessed via
the atomic64_* functions anyway making the issue moot.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Fixes: 571ed1fd2390 ("SUNRPC: Replace krb5_seq_lock with a lockless scheme")
Cc: Trond Myklebust <trond.myklebust@hammerspace.com>
Cc: Anna Schumaker <anna.schumaker@netapp.com>
Cc: J. Bruce Fields <bfields@fieldses.org>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: linux-nfs@vger.kernel.org
Cc: netdev@vger.kernel.org
---
include/linux/sunrpc/gss_krb5.h | 7 ++-----
net/sunrpc/auth_gss/gss_krb5_mech.c | 16 ++++++++++------
net/sunrpc/auth_gss/gss_krb5_seal.c | 28 ++--------------------------
net/sunrpc/auth_gss/gss_krb5_wrap.c | 4 ++--
4 files changed, 16 insertions(+), 39 deletions(-)
diff --git a/include/linux/sunrpc/gss_krb5.h b/include/linux/sunrpc/gss_krb5.h
index 131424cefc6a..02c0412e368c 100644
--- a/include/linux/sunrpc/gss_krb5.h
+++ b/include/linux/sunrpc/gss_krb5.h
@@ -107,8 +107,8 @@ struct krb5_ctx {
u8 Ksess[GSS_KRB5_MAX_KEYLEN]; /* session key */
u8 cksum[GSS_KRB5_MAX_KEYLEN];
s32 endtime;
- u32 seq_send;
- u64 seq_send64;
+ atomic_t seq_send;
+ atomic64_t seq_send64;
struct xdr_netobj mech_used;
u8 initiator_sign[GSS_KRB5_MAX_KEYLEN];
u8 acceptor_sign[GSS_KRB5_MAX_KEYLEN];
@@ -118,9 +118,6 @@ struct krb5_ctx {
u8 acceptor_integ[GSS_KRB5_MAX_KEYLEN];
};
-extern u32 gss_seq_send_fetch_and_inc(struct krb5_ctx *ctx);
-extern u64 gss_seq_send64_fetch_and_inc(struct krb5_ctx *ctx);
-
/* The length of the Kerberos GSS token header */
#define GSS_KRB5_TOK_HDR_LEN (16)
diff --git a/net/sunrpc/auth_gss/gss_krb5_mech.c b/net/sunrpc/auth_gss/gss_krb5_mech.c
index 7f0424dfa8f6..eab71fc7af3e 100644
--- a/net/sunrpc/auth_gss/gss_krb5_mech.c
+++ b/net/sunrpc/auth_gss/gss_krb5_mech.c
@@ -274,6 +274,7 @@ get_key(const void *p, const void *end,
static int
gss_import_v1_context(const void *p, const void *end, struct krb5_ctx *ctx)
{
+ u32 seq_send;
int tmp;
p = simple_get_bytes(p, end, &ctx->initiate, sizeof(ctx->initiate));
@@ -315,9 +316,10 @@ gss_import_v1_context(const void *p, const void *end, struct krb5_ctx *ctx)
p = simple_get_bytes(p, end, &ctx->endtime, sizeof(ctx->endtime));
if (IS_ERR(p))
goto out_err;
- p = simple_get_bytes(p, end, &ctx->seq_send, sizeof(ctx->seq_send));
+ p = simple_get_bytes(p, end, &seq_send, sizeof(seq_send));
if (IS_ERR(p))
goto out_err;
+ atomic_set(&ctx->seq_send, seq_send);
p = simple_get_netobj(p, end, &ctx->mech_used);
if (IS_ERR(p))
goto out_err;
@@ -607,6 +609,7 @@ static int
gss_import_v2_context(const void *p, const void *end, struct krb5_ctx *ctx,
gfp_t gfp_mask)
{
+ u64 seq_send64;
int keylen;
p = simple_get_bytes(p, end, &ctx->flags, sizeof(ctx->flags));
@@ -617,14 +620,15 @@ gss_import_v2_context(const void *p, const void *end, struct krb5_ctx *ctx,
p = simple_get_bytes(p, end, &ctx->endtime, sizeof(ctx->endtime));
if (IS_ERR(p))
goto out_err;
- p = simple_get_bytes(p, end, &ctx->seq_send64, sizeof(ctx->seq_send64));
+ p = simple_get_bytes(p, end, &seq_send64, sizeof(seq_send64));
if (IS_ERR(p))
goto out_err;
+ atomic64_set(&ctx->seq_send64, seq_send64);
/* set seq_send for use by "older" enctypes */
- ctx->seq_send = ctx->seq_send64;
- if (ctx->seq_send64 != ctx->seq_send) {
- dprintk("%s: seq_send64 %lx, seq_send %x overflow?\n", __func__,
- (unsigned long)ctx->seq_send64, ctx->seq_send);
+ atomic_set(&ctx->seq_send, seq_send64);
+ if (seq_send64 != atomic_read(&ctx->seq_send)) {
+ dprintk("%s: seq_send64 %llx, seq_send %x overflow?\n", __func__,
+ seq_send64, atomic_read(&ctx->seq_send));
p = ERR_PTR(-EINVAL);
goto out_err;
}
diff --git a/net/sunrpc/auth_gss/gss_krb5_seal.c b/net/sunrpc/auth_gss/gss_krb5_seal.c
index b4adeb06660b..48fe4a591b54 100644
--- a/net/sunrpc/auth_gss/gss_krb5_seal.c
+++ b/net/sunrpc/auth_gss/gss_krb5_seal.c
@@ -123,30 +123,6 @@ setup_token_v2(struct krb5_ctx *ctx, struct xdr_netobj *token)
return krb5_hdr;
}
-u32
-gss_seq_send_fetch_and_inc(struct krb5_ctx *ctx)
-{
- u32 old, seq_send = READ_ONCE(ctx->seq_send);
-
- do {
- old = seq_send;
- seq_send = cmpxchg(&ctx->seq_send, old, old + 1);
- } while (old != seq_send);
- return seq_send;
-}
-
-u64
-gss_seq_send64_fetch_and_inc(struct krb5_ctx *ctx)
-{
- u64 old, seq_send = READ_ONCE(ctx->seq_send);
-
- do {
- old = seq_send;
- seq_send = cmpxchg64(&ctx->seq_send64, old, old + 1);
- } while (old != seq_send);
- return seq_send;
-}
-
static u32
gss_get_mic_v1(struct krb5_ctx *ctx, struct xdr_buf *text,
struct xdr_netobj *token)
@@ -177,7 +153,7 @@ gss_get_mic_v1(struct krb5_ctx *ctx, struct xdr_buf *text,
memcpy(ptr + GSS_KRB5_TOK_HDR_LEN, md5cksum.data, md5cksum.len);
- seq_send = gss_seq_send_fetch_and_inc(ctx);
+ seq_send = atomic_fetch_inc(&ctx->seq_send);
if (krb5_make_seq_num(ctx, ctx->seq, ctx->initiate ? 0 : 0xff,
seq_send, ptr + GSS_KRB5_TOK_HDR_LEN, ptr + 8))
@@ -205,7 +181,7 @@ gss_get_mic_v2(struct krb5_ctx *ctx, struct xdr_buf *text,
/* Set up the sequence number. Now 64-bits in clear
* text and w/o direction indicator */
- seq_send_be64 = cpu_to_be64(gss_seq_send64_fetch_and_inc(ctx));
+ seq_send_be64 = cpu_to_be64(atomic64_fetch_inc(&ctx->seq_send64));
memcpy(krb5_hdr + 8, (char *) &seq_send_be64, 8);
if (ctx->initiate) {
diff --git a/net/sunrpc/auth_gss/gss_krb5_wrap.c b/net/sunrpc/auth_gss/gss_krb5_wrap.c
index 962fa84e6db1..5cdde6cb703a 100644
--- a/net/sunrpc/auth_gss/gss_krb5_wrap.c
+++ b/net/sunrpc/auth_gss/gss_krb5_wrap.c
@@ -228,7 +228,7 @@ gss_wrap_kerberos_v1(struct krb5_ctx *kctx, int offset,
memcpy(ptr + GSS_KRB5_TOK_HDR_LEN, md5cksum.data, md5cksum.len);
- seq_send = gss_seq_send_fetch_and_inc(kctx);
+ seq_send = atomic_fetch_inc(&kctx->seq_send);
/* XXX would probably be more efficient to compute checksum
* and encrypt at the same time: */
@@ -475,7 +475,7 @@ gss_wrap_kerberos_v2(struct krb5_ctx *kctx, u32 offset,
*be16ptr++ = 0;
be64ptr = (__be64 *)be16ptr;
- *be64ptr = cpu_to_be64(gss_seq_send64_fetch_and_inc(kctx));
+ *be64ptr = cpu_to_be64(atomic64_fetch_inc(&kctx->seq_send64));
err = (*kctx->gk5e->encrypt_v2)(kctx, offset, buf, pages);
if (err)
--
2.19.1
^ permalink raw reply related
* Re: [RFC PATCH] lib: Introduce generic __cmpxchg_u64() and use it where needed
From: Paul Burton @ 2018-11-01 17:54 UTC (permalink / raw)
To: Trond Myklebust
Cc: linux@roeck-us.net, linux-kernel@vger.kernel.org,
ralf@linux-mips.org, jlayton@kernel.org,
linuxppc-dev@lists.ozlabs.org, bfields@fieldses.org,
linux-mips@linux-mips.org, linux-nfs@vger.kernel.org,
akpm@linux-foundation.org, anna.schumaker@netapp.com,
jhogan@kernel.org, netdev@vger.kernel.org, davem@davemloft.net,
arnd@arndb.de, paulus@samba.org, mpe@ellerman.id.au
In-Reply-To: <4e2438a23d2edf03368950a72ec058d1d299c32e.camel@hammerspace.com>
Hi Trond,
On Thu, Nov 01, 2018 at 12:17:31AM +0000, Trond Myklebust wrote:
> On Wed, 2018-10-31 at 23:32 +0000, Paul Burton wrote:
> > In this particular case I have no idea why
> > net/sunrpc/auth_gss/gss_krb5_seal.c is using cmpxchg64() at all. It's
> > essentially reinventing atomic64_fetch_inc() which is already
> > provided
> > everywhere via CONFIG_GENERIC_ATOMIC64 & the spinlock approach. At
> > least
> > for atomic64_* functions the assumption that all access will be
> > performed using those same functions seems somewhat reasonable.
> >
> > So how does the below look? Trond?
>
> My one question (and the reason why I went with cmpxchg() in the first
> place) would be about the overflow behaviour for atomic_fetch_inc() and
> friends. I believe those functions should be OK on x86, so that when we
> overflow the counter, it behaves like an unsigned value and wraps back
> around. Is that the case for all architectures?
>
> i.e. are atomic_t/atomic64_t always guaranteed to behave like u32/u64
> on increment?
>
> I could not find any documentation that explicitly stated that they
> should.
Based on other replies it seems like it's at least implicitly assumed by
other code, even if not explicitly stated.
>From a MIPS perspective where atomics are implemented using load-linked
& store-conditional instructions the actual addition will be performed
using the same addu instruction that a plain integer addition would
generate (regardless of signedness), so there'll be absolutely no
difference in arithmetic between your gss_seq_send64_fetch_and_inc()
function and atomic64_fetch_inc(). I'd expect the same to be true for
other architectures with load-linked & store-conditional style atomics.
In any case, for the benefit of anyone interested who I didn't copy on
the patch submission, here it is:
https://lore.kernel.org/lkml/20181101175109.8621-1-paul.burton@mips.com/
Thanks,
Paul
^ permalink raw reply
* Re: [PATCH] SUNRPC: Use atomic(64)_t for seq_send(64)
From: Trond Myklebust @ 2018-11-01 17:57 UTC (permalink / raw)
To: anna.schumaker@netapp.com, linux-nfs@vger.kernel.org,
paul.burton@mips.com
Cc: bfields@fieldses.org, linux-kernel@vger.kernel.org,
pburton@wavecomp.com, davem@davemloft.net, jlayton@kernel.org,
netdev@vger.kernel.org
In-Reply-To: <20181101175109.8621-1-paul.burton@mips.com>
On Thu, 2018-11-01 at 17:51 +0000, Paul Burton wrote:
> The seq_send & seq_send64 fields in struct krb5_ctx are used as
> atomically incrementing counters. This is implemented using cmpxchg()
> &
> cmpxchg64() to implement what amount to custom versions of
> atomic_fetch_inc() & atomic64_fetch_inc().
>
> Besides the duplication, using cmpxchg64() has another major drawback
> in
> that some 32 bit architectures don't provide it. As such commit
> 571ed1fd2390 ("SUNRPC: Replace krb5_seq_lock with a lockless scheme")
> resulted in build failures for some architectures.
>
> Change seq_send to be an atomic_t and seq_send64 to be an atomic64_t,
> then use atomic(64)_* functions to manipulate the values. The
> atomic64_t
> type & associated functions are provided even on architectures which
> lack real 64 bit atomic memory access via CONFIG_GENERIC_ATOMIC64
> which
> uses spinlocks to serialize access. This fixes the build failures for
> architectures lacking cmpxchg64().
>
> A potential alternative that was raised would be to provide
> cmpxchg64()
> on the 32 bit architectures that currently lack it, using spinlocks.
> However this would provide a version of cmpxchg64() with semantics a
> little different to the implementations on architectures with real 64
> bit atomics - the spinlock-based implementation would only work if
> all
> access to the memory used with cmpxchg64() is *always* performed
> using
> cmpxchg64(). That is not currently a requirement for users of
> cmpxchg64(), and making it one seems questionable. As such avoiding
> cmpxchg64() outside of architecture-specific code seems best,
> particularly in cases where atomic64_t seems like a better fit
> anyway.
>
> The CONFIG_GENERIC_ATOMIC64 implementation of atomic64_* functions
> will
> use spinlocks & so faces the same issue, but with the key difference
> that the memory backing an atomic64_t ought to always be accessed via
> the atomic64_* functions anyway making the issue moot.
>
> Signed-off-by: Paul Burton <paul.burton@mips.com>
> Fixes: 571ed1fd2390 ("SUNRPC: Replace krb5_seq_lock with a lockless
> scheme")
> Cc: Trond Myklebust <trond.myklebust@hammerspace.com>
> Cc: Anna Schumaker <anna.schumaker@netapp.com>
> Cc: J. Bruce Fields <bfields@fieldses.org>
> Cc: Jeff Layton <jlayton@kernel.org>
> Cc: David S. Miller <davem@davemloft.net>
> Cc: linux-nfs@vger.kernel.org
> Cc: netdev@vger.kernel.org
> ---
> include/linux/sunrpc/gss_krb5.h | 7 ++-----
> net/sunrpc/auth_gss/gss_krb5_mech.c | 16 ++++++++++------
> net/sunrpc/auth_gss/gss_krb5_seal.c | 28 ++-------------------------
> -
> net/sunrpc/auth_gss/gss_krb5_wrap.c | 4 ++--
> 4 files changed, 16 insertions(+), 39 deletions(-)
>
Thanks Paul! ...and thanks for your patience in working out the
atomicity wraparound issues. Applied..
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com
^ permalink raw reply
* Re: [LKP] [tools] 6534770d6f: kernel_selftests.bpf.test_lwt_seg6local.fail
From: Jakub Kicinski @ 2018-11-01 18:18 UTC (permalink / raw)
To: kernel test robot
Cc: David S. Miller, LKML, Linus Torvalds, lkp, Martin KaFai Lau,
Okash Khawaja, Daniel Borkmann, Alexei Starovoitov,
netdev@vger.kernel.org
In-Reply-To: <20181101032643.GB24195@shao2-debian>
On Thu, 1 Nov 2018 11:26:44 +0800, kernel test robot wrote:
> FYI, we noticed the following commit (built with gcc-7):
>
> commit: 6534770d6f176093b50896961107b2d545ef38f0 ("tools: bpf: fix BTF code added twice to different trees")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
That's a fix to a merge commit, perhaps something before broke it?
> in testcase: kernel_selftests
> with following parameters:
>
> group: kselftests-00
>
> test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel.
> test-url: https://www.kernel.org/doc/Documentation/kselftest.txt
>
>
> on test machine: 80 threads Skylake with 64G memory
>
> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
>
>
> commit:
> 6b431d50d2a8acd1c418b998b856a055252ebc3a
> 6534770d6f176093b50896961107b2d545ef38f0
>
> 6b431d50d2a8acd1 6534770d6f176093b508969611
> ---------------- --------------------------
> fail:runs %reproduction fail:runs
> | | |
> 443:8 14689% 1618:8 dmesg.timestamp:last
> :8 100% 8:8 kernel_selftests.bpf.test_lwt_seg6local.fail
> :8 100% 8:8 kernel_selftests.bpf.test_lwt_seg6local.sh.fail
>
>
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
>
>
> Thanks,
> Rong Chen
^ permalink raw reply
* Re: [PATCH v2 net 3/3] net/mlx4_en: use __netdev_tx_sent_queue()
From: Tariq Toukan @ 2018-11-01 9:19 UTC (permalink / raw)
To: Eric Dumazet, David S . Miller
Cc: netdev, Tariq Toukan, Willem de Bruijn, Eric Dumazet
In-Reply-To: <20181031153914.132127-4-edumazet@google.com>
On 31/10/2018 5:39 PM, Eric Dumazet wrote:
> doorbell only depends on xmit_more and netif_tx_queue_stopped()
>
> Using __netdev_tx_sent_queue() avoids messing with BQL stop flag,
> and is more generic.
>
> This patch increases performance on GSO workload by keeping
> doorbells to the minimum required.
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Cc: Tariq Toukan <tariqt@mellanox.com>
> ---
> drivers/net/ethernet/mellanox/mlx4/en_tx.c | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> index 1857ee0f0871d48285a6d3711f7c3e9a1e08a05f..6f5153afcab4dfc331c099da854c54f1b9500887 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
> @@ -1006,7 +1006,6 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
> ring->packets++;
> }
> ring->bytes += tx_info->nr_bytes;
> - netdev_tx_sent_queue(ring->tx_queue, tx_info->nr_bytes);
> AVG_PERF_COUNTER(priv->pstats.tx_pktsz_avg, skb->len);
>
> if (tx_info->inl)
> @@ -1044,7 +1043,10 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
> netif_tx_stop_queue(ring->tx_queue);
> ring->queue_stopped++;
> }
> - send_doorbell = !skb->xmit_more || netif_xmit_stopped(ring->tx_queue);
> +
> + send_doorbell = __netdev_tx_sent_queue(ring->tx_queue,
> + tx_info->nr_bytes,
> + skb->xmit_more);
>
> real_size = (real_size / 16) & 0x3f;
>
>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Looks good to me.
Thanks.
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Jesper Dangaard Brouer @ 2018-11-01 9:22 UTC (permalink / raw)
To: Paweł Staszewski
Cc: brouer, Eric Dumazet, netdev, Tariq Toukan, Ilias Apalodimas,
Yoel Caspersen, Mel Gorman, Aaron Lu
In-Reply-To: <8e10bf68-f3b3-98f2-91a5-25b151756dd6@itcare.pl>
On Wed, 31 Oct 2018 23:20:01 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 31.10.2018 o 23:09, Eric Dumazet pisze:
> >
> > On 10/31/2018 02:57 PM, Paweł Staszewski wrote:
> >> Hi
> >>
> >> So maybee someone will be interested how linux kernel handles
> >> normal traffic (not pktgen :) )
Pawel is this live production traffic?
I know Yoel (Cc) is very interested to know the real-life limitation of
Linux as a router, especially with VLANs like you use.
> >>
> >> Server HW configuration:
> >>
> >> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
> >>
> >> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
> >>
> >>
> >> Server software:
> >>
> >> FRR - as routing daemon
> >>
> >> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa node)
> >>
> >> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
> >>
> >>
> >> Maximum traffic that server can handle:
> >>
> >> Bandwidth
> >>
> >> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> >> input: /proc/net/dev type: rate
> >> \ iface Rx Tx Total
> >> ==============================================================================
> >> enp175s0f1: 28.51 Gb/s 37.24 Gb/s 65.74 Gb/s
> >> enp175s0f0: 38.07 Gb/s 28.44 Gb/s 66.51 Gb/s
> >> ------------------------------------------------------------------------------
> >> total: 66.58 Gb/s 65.67 Gb/s 132.25 Gb/s
> >>
Actually rather impressive number for a Linux router.
> >>
> >> Packets per second:
> >>
> >> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> >> input: /proc/net/dev type: rate
> >> - iface Rx Tx Total
> >> ==============================================================================
> >> enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
> >> enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
> >> ------------------------------------------------------------------------------
> >> total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
> >>
Average packet size:
(28.51*10^9/8)/5248589 = 678.99 bytes
(38.07*10^9/8)/3557944 = 1337.49 bytes
> >> After reaching that limits nics on the upstream side (more RX
> >> traffic) start to drop packets
> >>
> >>
> >> I just dont understand that server can't handle more bandwidth
> >> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
> >> RX side are increasing.
> >>
> >> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
> >> is 126Gbit - and also when testing with pktgen i can reach more bw
> >> and pps (like 4x more comparing to normal internet traffic)
> >>
> >> And wondering if there is something that can be improved here.
> >>
> >>
> >>
> >> Some more informations / counters / stats and perf top below:
> >>
> >> Perf top flame graph:
> >>
> >> https://uploadfiles.io/7zo6u
Thanks a lot for the flame graph!
> >>
> >> System configuration(long):
> >>
> >>
> >> cat /sys/devices/system/node/node1/cpulist
> >> 14-27,42-55
> >> cat /sys/class/net/enp175s0f0/device/numa_node
> >> 1
> >> cat /sys/class/net/enp175s0f1/device/numa_node
> >> 1
> >>
Hint grep can give you nicer output that cat:
$ grep -H . /sys/class/net/*/device/numa_node
> >>
> >>
> >>
> >>
> >> ip -s -d link ls dev enp175s0f0
> >> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
> >> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
> >> RX: bytes packets errors dropped overrun mcast
> >> 184142375840858 141347715974 2 2806325 0 85050528
> >> TX: bytes packets errors dropped carrier collsns
> >> 99270697277430 172227994003 0 0 0 0
> >>
> >> ip -s -d link ls dev enp175s0f1
> >> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
> >> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
> >> RX: bytes packets errors dropped overrun mcast
> >> 99686284170801 173507590134 61 669685 0 100304421
> >> TX: bytes packets errors dropped carrier collsns
> >> 184435107970545 142383178304 0 0 0 0
> >>
You have increased the default (1000) qlen to 8192, why?
What default qdisc do you run?... looking through your very detail main
email report (I do love the details you give!). You run
pfifo_fast_dequeue, thus this 8192 qlen is actually having effect.
I would like to know if and how much qdisc_dequeue bulking is happening
in this setup? Can you run:
perf-stat-hist -m 8192 -P2 qdisc:qdisc_dequeue packets
The perf-stat-hist is from Brendan Gregg's git-tree:
https://github.com/brendangregg/perf-tools
https://github.com/brendangregg/perf-tools/blob/master/misc/perf-stat-hist
> >> ./softnet.sh
> >> cpu total dropped squeezed collision rps flow_limit
> >>
> >>
> >>
> >>
> >> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
> >> ------------------------------------------------------------------------------------------
> >>
> >> 26.78% [kernel] [k] queued_spin_lock_slowpath
> >
> > This is highly suspect.
> >
I agree! -- 26.78% spend in queued_spin_lock_slowpath. Hint if you see
_raw_spin_lock then it is likely not a contended lock, but if you see
queued_spin_lock_slowpath in a perf-report your workload is likely in
trouble.
> > A call graph (perf record -a -g sleep 1; perf report --stdio)
> > would tell what is going on.
>
> perf report:
> https://ufile.io/rqp0h
>
Thanks for the output (my 30" screen is just large enough to see the
full output). Together with the flame-graph, it is clear that this
lock happens in the page allocator code.
Section copied out:
mlx5e_poll_tx_cq
|
--16.34%--napi_consume_skb
|
|--12.65%--__free_pages_ok
| |
| --11.86%--free_one_page
| |
| |--10.10%--queued_spin_lock_slowpath
| |
| --0.65%--_raw_spin_lock
|
|--1.55%--page_frag_free
|
--1.44%--skb_release_data
Let me explain what (I think) happens. The mlx5 driver RX-page recycle
mechanism is not effective in this workload, and pages have to go
through the page allocator. The lock contention happens during mlx5
DMA TX completion cycle. And the page allocator cannot keep up at
these speeds.
One solution is extend page allocator with a bulk free API. (This have
been on my TODO list for a long time, but I don't have a
micro-benchmark that trick the driver page-recycle to fail). It should
fit nicely, as I can see that kmem_cache_free_bulk() does get
activated (bulk freeing SKBs), which means that DMA TX completion do
have a bulk of packets.
We can (and should) also improve the page recycle scheme in the driver.
After LPC, I have a project with Tariq and Ilias (Cc'ed) to improve the
page_pool, and we will (attempt) to generalize this, for both high-end
mlx5 and more low-end ARM64-boards (macchiatobin and espressobin).
The MM-people is in parallel working to improve the performance of
order-0 page returns. Thus, the explicit page bulk free API might
actually become less important. I actually think (Cc.) Aaron have a
patchset he would like you to test, which removes the (zone->)lock
you hit in free_one_page().
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: [PATCH resend] macvlan: use per-cpu queues for broadcast and multicast packets
From: David Miller @ 2018-11-01 18:53 UTC (permalink / raw)
To: khlebnikov; +Cc: netdev, linux-kernel, vfedorenko
In-Reply-To: <154108883150.919735.10620997593741055347.stgit@buzz>
From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Date: Thu, 01 Nov 2018 19:13:51 +0300
> Currently macvlan has single per-port queue for broadcast and multicast.
> This disrupts order of packets when flows from different cpus are mixed.
>
> This patch replaces this queue with single set of per-cpu queues.
> Pointer to macvlan port is passed in skb control block.
>
> Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> Reported-add-tested-by: Vadim Fedorenko <vfedorenko@yandex-team.ru>
Konstantin, all of your postings to vger.kernel.org are being rejected.
They are because of the:
"List-Unsubscribe: <https://ml.yandex-team.ru/lists/vfedorenko@yandex-team.ru/unsubscribe-click>"
and similar strings that appear in your email headers.
Please disable whatever is producing these email headers, otherwise
nobody (including me) will see and process your postings.
Thank you.
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Saeed Mahameed @ 2018-11-01 9:50 UTC (permalink / raw)
To: pstaszewski@itcare.pl, netdev@vger.kernel.org
In-Reply-To: <61697e49-e839-befc-8330-fc00187c48ee@itcare.pl>
On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
> Hi
>
> So maybee someone will be interested how linux kernel handles normal
> traffic (not pktgen :) )
>
>
> Server HW configuration:
>
> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>
> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>
>
> Server software:
>
> FRR - as routing daemon
>
> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local
> numa
> node)
>
> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa
> node)
>
>
> Maximum traffic that server can handle:
>
> Bandwidth
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> \ iface Rx Tx Total
> =====================================================================
> =========
> enp175s0f1: 28.51 Gb/s 37.24
> Gb/s
> 65.74 Gb/s
> enp175s0f0: 38.07 Gb/s 28.44
> Gb/s
> 66.51 Gb/s
> -------------------------------------------------------------------
> -----------
> total: 66.58 Gb/s 65.67
> Gb/s
> 132.25 Gb/s
>
>
> Packets per second:
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> - iface Rx Tx Total
> =====================================================================
> =========
> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
> 8735207.00 P/s
> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
> 8790460.00 P/s
> -------------------------------------------------------------------
> -----------
> total: 8806533.00 P/s 8719134.00 P/s
> 17525668.00 P/s
>
>
> After reaching that limits nics on the upstream side (more RX
> traffic)
> start to drop packets
>
>
> I just dont understand that server can't handle more bandwidth
> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
> side are increasing.
>
Where do you see 40 Gb/s ? you showed that both ports on the same NIC (
same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) = 132.25
Gb/s which aligns with your pcie link limit, what am i missing ?
> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
> is
> 126Gbit - and also when testing with pktgen i can reach more bw and
> pps
> (like 4x more comparing to normal internet traffic)
>
Are you forwarding when using pktgen as well or you just testing the RX
side pps ?
> And wondering if there is something that can be improved here.
>
>
>
> Some more informations / counters / stats and perf top below:
>
> Perf top flame graph:
>
> https://uploadfiles.io/7zo6u
>
>
>
> System configuration(long):
>
>
> cat /sys/devices/system/node/node1/cpulist
> 14-27,42-55
> cat /sys/class/net/enp175s0f0/device/numa_node
> 1
> cat /sys/class/net/enp175s0f1/device/numa_node
> 1
>
>
>
>
>
> ip -s -d link ls dev enp175s0f0
> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state
> UP mode DEFAULT group default qlen 8192
> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity
> 0
> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
> gso_max_segs 65535
> RX: bytes packets errors dropped overrun mcast
> 184142375840858 141347715974 2 2806325 0 85050528
> TX: bytes packets errors dropped carrier collsns
> 99270697277430 172227994003 0 0 0 0
>
> ip -s -d link ls dev enp175s0f1
> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state
> UP mode DEFAULT group default qlen 8192
> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity
> 0
> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
> gso_max_segs 65535
> RX: bytes packets errors dropped overrun mcast
> 99686284170801 173507590134 61 669685 0 100304421
> TX: bytes packets errors dropped carrier collsns
> 184435107970545 142383178304 0 0 0 0
>
>
> ./softnet.sh
> cpu total dropped squeezed collision rps flow_limit
> 0 3961392822 0 1221478 0 0 0
> 1 3701952251 0 1258234 0 0 0
> 2 3879522030 0 1584282 0 0 0
> 3 3731349789 0 1529029 0 0 0
> 4 1323956701 0 2176371 0 0 0
> 5 420528963 0 1880146 0 0 0
> 6 348720322 0 1830142 0 0 0
> 7 372736328 0 1820891 0 0 0
> 8 567888751 0 1414763 0 0 0
> 9 476075775 0 1868150 0 0 0
> 10 468946725 0 1841428 0 0 0
> 11 676591958 0 1900160 0 0 0
> 12 346803472 0 1834600 0 0 0
> 13 457960872 0 1874529 0 0 0
> 14 1990279665 0 4699000 0 0 0
> 15 1211873601 0 4541281 0 0 0
> 16 1123871928 0 4544712 0 0 0
> 17 1014957263 0 4152355 0 0 0
> 18 2603779724 0 4593869 0 0 0
> 19 2181924054 0 4930618 0 0 0
> 20 2273502182 0 4894627 0 0 0
> 21 2232030947 0 4860048 0 0 0
> 22 2203555394 0 4603830 0 0 0
> 23 2194756800 0 4921294 0 0 0
> 24 2347158294 0 4818354 0 0 0
> 25 2291097883 0 4744469 0 0 0
> 26 2206945011 0 4836483 0 0 0
> 27 2318530217 0 4917617 0 0 0
> 28 512797543 0 1895200 0 0 0
> 29 597279474 0 1532134 0 0 0
> 30 475317503 0 1451523 0 0 0
> 31 499172796 0 1901207 0 0 0
> 32 493874745 0 1915382 0 0 0
> 33 296056288 0 1865535 0 0 0
> 34 3905097041 0 1580822 0 0 0
> 35 3905112345 0 1536105 0 0 0
> 36 3900358950 0 1166319 0 0 0
> 37 3940978093 0 1600219 0 0 0
> 38 3878632215 0 1180389 0 0 0
> 39 3814804736 0 1584925 0 0 0
> 40 4152934337 0 1663660 0 0 0
> 41 3855273904 0 1552219 0 0 0
> 42 2319538182 0 4884480 0 0 0
> 43 2448606991 0 4387456 0 0 0
> 44 1436136753 0 4485073 0 0 0
> 45 1200500141 0 4537284 0 0 0
> 46 1307799923 0 4534156 0 0 0
> 47 1586575293 0 4272997 0 0 0
> 48 3852574 0 4162653 0 0 0
> 49 391449390 0 3935202 0 0 0
> 50 791388200 0 4290738 0 0 0
> 51 127107573 0 3907750 0 0 0
> 52 115622148 0 4012843 0 0 0
> 53 71098871 0 4200625 0 0 0
> 54 305121466 0 4365614 0 0 0
> 55 10914257 0 4369426 0 0 0
>
>
>
>
> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> -------------------------------------------------------------------
> -------------------------------------------------------------------
> -------------------------------------------------------------------
> ------
>
> 26.78% [kernel] [k] queued_spin_lock_slowpath
> 9.09% [kernel] [k] mlx5e_skb_from_cqe_linear
> 4.94% [kernel] [k] mlx5e_sq_xmit
> 3.63% [kernel] [k] memcpy_erms
> 3.30% [kernel] [k] fib_table_lookup
> 3.26% [kernel] [k] build_skb
> 2.41% [kernel] [k] mlx5e_poll_tx_cq
> 2.11% [kernel] [k] get_page_from_freelist
> 1.51% [kernel] [k] vlan_do_receive
> 1.51% [kernel] [k] _raw_spin_lock
> 1.43% [kernel] [k] __dev_queue_xmit
> 1.41% [kernel] [k] dev_gro_receive
> 1.34% [kernel] [k] mlx5e_poll_rx_cq
> 1.26% [kernel] [k] tcp_gro_receive
> 1.21% [kernel] [k] free_one_page
> 1.13% [kernel] [k] swiotlb_map_page
> 1.13% [kernel] [k] mlx5e_post_rx_wqes
> 1.05% [kernel] [k] pfifo_fast_dequeue
> 1.05% [kernel] [k] mlx5e_handle_rx_cqe
> 1.03% [kernel] [k] ip_finish_output2
> 1.02% [kernel] [k] ipt_do_table
> 0.96% [kernel] [k] inet_gro_receive
> 0.91% [kernel] [k] mlx5_eq_int
> 0.88% [kernel] [k] __slab_free.isra.79
> 0.86% [kernel] [k] __build_skb
> 0.84% [kernel] [k] page_frag_free
> 0.76% [kernel] [k] skb_release_data
> 0.75% [kernel] [k] __netif_receive_skb_core
> 0.75% [kernel] [k] irq_entries_start
> 0.71% [kernel] [k] ip_route_input_rcu
> 0.65% [kernel] [k] vlan_dev_hard_start_xmit
> 0.56% [kernel] [k] ip_forward
> 0.56% [kernel] [k] __memcpy
> 0.52% [kernel] [k] kmem_cache_alloc
> 0.52% [kernel] [k] kmem_cache_free_bulk
> 0.49% [kernel] [k] mlx5e_page_release
> 0.47% [kernel] [k] netif_skb_features
> 0.47% [kernel] [k] mlx5e_build_rx_skb
> 0.47% [kernel] [k] dev_hard_start_xmit
> 0.43% [kernel] [k] __page_pool_put_page
> 0.43% [kernel] [k] __netif_schedule
> 0.43% [kernel] [k] mlx5e_xmit
> 0.41% [kernel] [k] __qdisc_run
> 0.41% [kernel] [k] validate_xmit_skb.isra.142
> 0.41% [kernel] [k] swiotlb_unmap_page
> 0.40% [kernel] [k] inet_lookup_ifaddr_rcu
> 0.34% [kernel] [k] ip_rcv_core.isra.20.constprop.25
> 0.34% [kernel] [k] tcp4_gro_receive
> 0.29% [kernel] [k] _raw_spin_lock_irqsave
> 0.29% [kernel] [k] napi_consume_skb
> 0.29% [kernel] [k] skb_gro_receive
> 0.29% [kernel] [k] ___slab_alloc.isra.80
> 0.27% [kernel] [k] eth_type_trans
> 0.26% [kernel] [k] __free_pages_ok
> 0.26% [kernel] [k] __get_xps_queue_idx
> 0.24% [kernel] [k] _raw_spin_trylock
> 0.23% [kernel] [k] __local_bh_enable_ip
> 0.22% [kernel] [k] pfifo_fast_enqueue
> 0.21% [kernel] [k] tasklet_action_common.isra.21
> 0.21% [kernel] [k] sch_direct_xmit
> 0.21% [kernel] [k] skb_network_protocol
> 0.21% [kernel] [k] kmem_cache_free
> 0.20% [kernel] [k] netdev_pick_tx
> 0.18% [kernel] [k] napi_gro_complete
> 0.18% [kernel] [k] __sched_text_start
> 0.18% [kernel] [k] mlx5e_xdp_handle
> 0.17% [kernel] [k] ip_finish_output
> 0.16% [kernel] [k] napi_gro_flush
> 0.16% [kernel] [k] vlan_passthru_hard_header
> 0.16% [kernel] [k] skb_segment
> 0.15% [kernel] [k] __alloc_pages_nodemask
> 0.15% [kernel] [k] mlx5e_features_check
> 0.15% [kernel] [k] mlx5e_napi_poll
> 0.15% [kernel] [k] napi_gro_receive
> 0.14% [kernel] [k] fib_validate_source
> 0.14% [kernel] [k] _raw_spin_lock_irq
> 0.14% [kernel] [k] inet_gro_complete
> 0.14% [kernel] [k] get_partial_node.isra.78
> 0.13% [kernel] [k] napi_complete_done
> 0.13% [kernel] [k] ip_rcv_finish_core.isra.17
> 0.13% [kernel] [k] cmd_exec
>
>
>
> ethtool -S enp175s0f1
> NIC statistics:
> rx_packets: 173730800927
> rx_bytes: 99827422751332
> tx_packets: 142532009512
> tx_bytes: 184633045911222
> tx_tso_packets: 25989113891
> tx_tso_bytes: 132933363384458
> tx_tso_inner_packets: 0
> tx_tso_inner_bytes: 0
> tx_added_vlan_packets: 74630239613
> tx_nop: 2029817748
> rx_lro_packets: 0
> rx_lro_bytes: 0
> rx_ecn_mark: 0
> rx_removed_vlan_packets: 173730800927
> rx_csum_unnecessary: 0
> rx_csum_none: 434357
> rx_csum_complete: 173730366570
> rx_csum_unnecessary_inner: 0
> rx_xdp_drop: 0
> rx_xdp_redirect: 0
> rx_xdp_tx_xmit: 0
> rx_xdp_tx_full: 0
> rx_xdp_tx_err: 0
> rx_xdp_tx_cqe: 0
> tx_csum_none: 38260960853
> tx_csum_partial: 36369278774
> tx_csum_partial_inner: 0
> tx_queue_stopped: 1
> tx_queue_dropped: 0
> tx_xmit_more: 748638099
> tx_recover: 0
> tx_cqes: 73881645031
> tx_queue_wake: 1
> tx_udp_seg_rem: 0
> tx_cqe_err: 0
> tx_xdp_xmit: 0
> tx_xdp_full: 0
> tx_xdp_err: 0
> tx_xdp_cqes: 0
> rx_wqe_err: 0
> rx_mpwqe_filler_cqes: 0
> rx_mpwqe_filler_strides: 0
> rx_buff_alloc_err: 0
> rx_cqe_compress_blks: 0
> rx_cqe_compress_pkts: 0
If this is a pcie bottleneck it might be useful to enable CQE
compression (to reduce PCIe completion descriptors transactions)
you should see the above rx_cqe_compress_pkts increasing when enabled.
$ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
$ ethtool --show-priv-flags enp175s0f1
Private flags for p6p1:
rx_cqe_moder : on
cqe_moder : off
rx_cqe_compress : on
...
try this on both interfaces.
> rx_page_reuse: 0
> rx_cache_reuse: 14441066823
> rx_cache_full: 51126004413
> rx_cache_empty: 21297344082
> rx_cache_busy: 51127247487
> rx_cache_waive: 21298322293
> rx_congst_umr: 0
> rx_arfs_err: 0
> ch_events: 24603119858
> ch_poll: 25180949074
> ch_arm: 24480437587
> ch_aff_change: 75
> ch_eq_rearm: 0
> rx_out_of_buffer: 669685
comparing this to rx_vport_unicast_packets, it is a very small
percentage of dropped packets due to stalled rx cpu, so rx cpu is not a
bottleneck, at least for the driver rx rings.
> rx_if_down_packets: 61
> rx_vport_unicast_packets: 173731641945
> rx_vport_unicast_bytes: 100522745036693
> tx_vport_unicast_packets: 142531901313
> tx_vport_unicast_bytes: 185189071776429
> rx_vport_multicast_packets: 100360886
> rx_vport_multicast_bytes: 6639236688
> tx_vport_multicast_packets: 32837
> tx_vport_multicast_bytes: 2978810
> rx_vport_broadcast_packets: 44854
> rx_vport_broadcast_bytes: 6313510
> tx_vport_broadcast_packets: 72258
> tx_vport_broadcast_bytes: 4335480
> rx_vport_rdma_unicast_packets: 0
> rx_vport_rdma_unicast_bytes: 0
> tx_vport_rdma_unicast_packets: 0
> tx_vport_rdma_unicast_bytes: 0
> rx_vport_rdma_multicast_packets: 0
> rx_vport_rdma_multicast_bytes: 0
> tx_vport_rdma_multicast_packets: 0
> tx_vport_rdma_multicast_bytes: 0
> tx_packets_phy: 142532004669
> rx_packets_phy: 173980375752
> rx_crc_errors_phy: 0
> tx_bytes_phy: 185759204762903
> rx_bytes_phy: 101326109361379
> tx_multicast_phy: 32837
> tx_broadcast_phy: 72258
> rx_multicast_phy: 100360885
> rx_broadcast_phy: 44854
> rx_in_range_len_errors_phy: 2
> rx_out_of_range_len_phy: 0
> rx_oversize_pkts_phy: 59
> rx_symbol_err_phy: 0
> tx_mac_control_phy: 0
> rx_mac_control_phy: 0
> rx_unsupported_op_phy: 0
> rx_pause_ctrl_phy: 0
> tx_pause_ctrl_phy: 0
> rx_discards_phy: 148328738
> tx_discards_phy: 0
> tx_errors_phy: 0
> rx_undersize_pkts_phy: 0
> rx_fragments_phy: 0
> rx_jabbers_phy: 0
> rx_64_bytes_phy: 36551843112
> rx_65_to_127_bytes_phy: 65102131735
> rx_128_to_255_bytes_phy: 5755731137
> rx_256_to_511_bytes_phy: 2475619839
> rx_512_to_1023_bytes_phy: 2826971156
> rx_1024_to_1518_bytes_phy: 42474023107
> rx_1519_to_2047_bytes_phy: 18794051270
> rx_2048_to_4095_bytes_phy: 0
> rx_4096_to_8191_bytes_phy: 0
> rx_8192_to_10239_bytes_phy: 0
> link_down_events_phy: 0
> rx_pcs_symbol_err_phy: 0
> rx_corrected_bits_phy: 0
> rx_pci_signal_integrity: 0
> tx_pci_signal_integrity: 48
> rx_prio0_bytes: 101316322498995
> rx_prio0_packets: 173711151686
> tx_prio0_bytes: 185759176566814
> tx_prio0_packets: 142531983704
> rx_prio1_bytes: 47062768
> rx_prio1_packets: 228932
> tx_prio1_bytes: 0
> tx_prio1_packets: 0
> rx_prio2_bytes: 12434759
> rx_prio2_packets: 83773
> tx_prio2_bytes: 0
> tx_prio2_packets: 0
> rx_prio3_bytes: 288843134
> rx_prio3_packets: 982102
> tx_prio3_bytes: 0
> tx_prio3_packets: 0
> rx_prio4_bytes: 699797236
> rx_prio4_packets: 8109231
> tx_prio4_bytes: 0
> tx_prio4_packets: 0
> rx_prio5_bytes: 1385386738
> rx_prio5_packets: 9661187
> tx_prio5_bytes: 0
> tx_prio5_packets: 0
> rx_prio6_bytes: 317092102
> rx_prio6_packets: 1951538
> tx_prio6_bytes: 0
> tx_prio6_packets: 0
> rx_prio7_bytes: 7015734695
> rx_prio7_packets: 99847456
> tx_prio7_bytes: 0
> tx_prio7_packets: 0
> module_unplug: 0
> module_bus_stuck: 0
> module_high_temp: 0
> module_bad_shorted: 0
> ch0_events: 936264703
> ch0_poll: 963766474
> ch0_arm: 930246079
> ch0_aff_change: 0
> ch0_eq_rearm: 0
> ch1_events: 869408429
> ch1_poll: 896099392
> ch1_arm: 864336861
> ch1_aff_change: 0
> ch1_eq_rearm: 0
> ch2_events: 843345698
> ch2_poll: 869749522
> ch2_arm: 838186113
> ch2_aff_change: 2
> ch2_eq_rearm: 0
> ch3_events: 850261340
> ch3_poll: 876721111
> ch3_arm: 845295235
> ch3_aff_change: 3
> ch3_eq_rearm: 0
> ch4_events: 974985780
> ch4_poll: 997781915
> ch4_arm: 969618250
> ch4_aff_change: 3
> ch4_eq_rearm: 0
> ch5_events: 888559089
> ch5_poll: 912783615
> ch5_arm: 883826078
> ch5_aff_change: 2
> ch5_eq_rearm: 0
> ch6_events: 873730730
> ch6_poll: 899635752
> ch6_arm: 868677574
> ch6_aff_change: 4
> ch6_eq_rearm: 0
> ch7_events: 873478411
> ch7_poll: 899216716
> ch7_arm: 868693645
> ch7_aff_change: 3
> ch7_eq_rearm: 0
> ch8_events: 871900967
> ch8_poll: 898575518
> ch8_arm: 866763693
> ch8_aff_change: 3
> ch8_eq_rearm: 0
> ch9_events: 880325565
> ch9_poll: 904983269
> ch9_arm: 875643922
> ch9_aff_change: 2
> ch9_eq_rearm: 0
> ch10_events: 889919775
> ch10_poll: 915335809
> ch10_arm: 885110225
> ch10_aff_change: 4
> ch10_eq_rearm: 0
> ch11_events: 962709175
> ch11_poll: 983963451
> ch11_arm: 958117526
> ch11_aff_change: 2
> ch11_eq_rearm: 0
> ch12_events: 941333837
> ch12_poll: 964625523
> ch12_arm: 936409706
> ch12_aff_change: 2
> ch12_eq_rearm: 0
> ch13_events: 914996974
> ch13_poll: 937441049
> ch13_arm: 910478393
> ch13_aff_change: 4
> ch13_eq_rearm: 0
> ch14_events: 888050001
> ch14_poll: 911818008
> ch14_arm: 883465035
> ch14_aff_change: 4
> ch14_eq_rearm: 0
> ch15_events: 947547704
> ch15_poll: 969073194
> ch15_arm: 942686515
> ch15_aff_change: 4
> ch15_eq_rearm: 0
> ch16_events: 825804904
> ch16_poll: 840630747
> ch16_arm: 822227488
> ch16_aff_change: 2
> ch16_eq_rearm: 0
> ch17_events: 861673823
> ch17_poll: 874754041
> ch17_arm: 858520448
> ch17_aff_change: 2
> ch17_eq_rearm: 0
> ch18_events: 879413440
> ch18_poll: 893962529
> ch18_arm: 875983204
> ch18_aff_change: 4
> ch18_eq_rearm: 0
> ch19_events: 896073709
> ch19_poll: 909216857
> ch19_arm: 893022121
> ch19_aff_change: 4
> ch19_eq_rearm: 0
> ch20_events: 865188535
> ch20_poll: 880692345
> ch20_arm: 861440265
> ch20_aff_change: 3
> ch20_eq_rearm: 0
> ch21_events: 862709303
> ch21_poll: 878104242
> ch21_arm: 859041767
> ch21_aff_change: 2
> ch21_eq_rearm: 0
> ch22_events: 887720551
> ch22_poll: 904122074
> ch22_arm: 883983794
> ch22_aff_change: 2
> ch22_eq_rearm: 0
> ch23_events: 813355027
> ch23_poll: 828074467
> ch23_arm: 809912398
> ch23_aff_change: 4
> ch23_eq_rearm: 0
> ch24_events: 822366675
> ch24_poll: 839917937
> ch24_arm: 818422754
> ch24_aff_change: 2
> ch24_eq_rearm: 0
> ch25_events: 826642292
> ch25_poll: 842630121
> ch25_arm: 822642618
> ch25_aff_change: 2
> ch25_eq_rearm: 0
> ch26_events: 826392584
> ch26_poll: 843406973
> ch26_arm: 822455000
> ch26_aff_change: 3
> ch26_eq_rearm: 0
> ch27_events: 828960899
> ch27_poll: 843866518
> ch27_arm: 825230937
> ch27_aff_change: 3
> ch27_eq_rearm: 0
> ch28_events: 7
> ch28_poll: 7
> ch28_arm: 7
> ch28_aff_change: 0
> ch28_eq_rearm: 0
> ch29_events: 4
> ch29_poll: 4
> ch29_arm: 4
> ch29_aff_change: 0
> ch29_eq_rearm: 0
> ch30_events: 4
> ch30_poll: 4
> ch30_arm: 4
> ch30_aff_change: 0
> ch30_eq_rearm: 0
> ch31_events: 4
> ch31_poll: 4
> ch31_arm: 4
> ch31_aff_change: 0
> ch31_eq_rearm: 0
> ch32_events: 4
> ch32_poll: 4
> ch32_arm: 4
> ch32_aff_change: 0
> ch32_eq_rearm: 0
> ch33_events: 4
> ch33_poll: 4
> ch33_arm: 4
> ch33_aff_change: 0
> ch33_eq_rearm: 0
> ch34_events: 4
> ch34_poll: 4
> ch34_arm: 4
> ch34_aff_change: 0
> ch34_eq_rearm: 0
> ch35_events: 4
> ch35_poll: 4
> ch35_arm: 4
> ch35_aff_change: 0
> ch35_eq_rearm: 0
> ch36_events: 4
> ch36_poll: 4
> ch36_arm: 4
> ch36_aff_change: 0
> ch36_eq_rearm: 0
> ch37_events: 4
> ch37_poll: 4
> ch37_arm: 4
> ch37_aff_change: 0
> ch37_eq_rearm: 0
> ch38_events: 4
> ch38_poll: 4
> ch38_arm: 4
> ch38_aff_change: 0
> ch38_eq_rearm: 0
> ch39_events: 4
> ch39_poll: 4
> ch39_arm: 4
> ch39_aff_change: 0
> ch39_eq_rearm: 0
> ch40_events: 4
> ch40_poll: 4
> ch40_arm: 4
> ch40_aff_change: 0
> ch40_eq_rearm: 0
> ch41_events: 4
> ch41_poll: 4
> ch41_arm: 4
> ch41_aff_change: 0
> ch41_eq_rearm: 0
> ch42_events: 4
> ch42_poll: 4
> ch42_arm: 4
> ch42_aff_change: 0
> ch42_eq_rearm: 0
> ch43_events: 4
> ch43_poll: 4
> ch43_arm: 4
> ch43_aff_change: 0
> ch43_eq_rearm: 0
> ch44_events: 4
> ch44_poll: 4
> ch44_arm: 4
> ch44_aff_change: 0
> ch44_eq_rearm: 0
> ch45_events: 4
> ch45_poll: 4
> ch45_arm: 4
> ch45_aff_change: 0
> ch45_eq_rearm: 0
> ch46_events: 4
> ch46_poll: 4
> ch46_arm: 4
> ch46_aff_change: 0
> ch46_eq_rearm: 0
> ch47_events: 4
> ch47_poll: 4
> ch47_arm: 4
> ch47_aff_change: 0
> ch47_eq_rearm: 0
> ch48_events: 4
> ch48_poll: 4
> ch48_arm: 4
> ch48_aff_change: 0
> ch48_eq_rearm: 0
> ch49_events: 4
> ch49_poll: 4
> ch49_arm: 4
> ch49_aff_change: 0
> ch49_eq_rearm: 0
> ch50_events: 4
> ch50_poll: 4
> ch50_arm: 4
> ch50_aff_change: 0
> ch50_eq_rearm: 0
> ch51_events: 4
> ch51_poll: 4
> ch51_arm: 4
> ch51_aff_change: 0
> ch51_eq_rearm: 0
> ch52_events: 4
> ch52_poll: 4
> ch52_arm: 4
> ch52_aff_change: 0
> ch52_eq_rearm: 0
> ch53_events: 4
> ch53_poll: 4
> ch53_arm: 4
> ch53_aff_change: 0
> ch53_eq_rearm: 0
> ch54_events: 4
> ch54_poll: 4
> ch54_arm: 4
> ch54_aff_change: 0
> ch54_eq_rearm: 0
> ch55_events: 4
> ch55_poll: 4
> ch55_arm: 4
> ch55_aff_change: 0
> ch55_eq_rearm: 0
> rx0_packets: 7284057433
> rx0_bytes: 4330611281319
> rx0_csum_complete: 7283623076
> rx0_csum_unnecessary: 0
> rx0_csum_unnecessary_inner: 0
> rx0_csum_none: 434357
> rx0_xdp_drop: 0
> rx0_xdp_redirect: 0
> rx0_lro_packets: 0
> rx0_lro_bytes: 0
> rx0_ecn_mark: 0
> rx0_removed_vlan_packets: 7284057433
> rx0_wqe_err: 0
> rx0_mpwqe_filler_cqes: 0
> rx0_mpwqe_filler_strides: 0
> rx0_buff_alloc_err: 0
> rx0_cqe_compress_blks: 0
> rx0_cqe_compress_pkts: 0
> rx0_page_reuse: 0
> rx0_cache_reuse: 1989731589
> rx0_cache_full: 28213297
> rx0_cache_empty: 1624089822
> rx0_cache_busy: 28213961
> rx0_cache_waive: 1624083610
> rx0_congst_umr: 0
> rx0_arfs_err: 0
> rx0_xdp_tx_xmit: 0
> rx0_xdp_tx_full: 0
> rx0_xdp_tx_err: 0
> rx0_xdp_tx_cqes: 0
> rx1_packets: 6691319211
> rx1_bytes: 3799580210608
> rx1_csum_complete: 6691319211
> rx1_csum_unnecessary: 0
> rx1_csum_unnecessary_inner: 0
> rx1_csum_none: 0
> rx1_xdp_drop: 0
> rx1_xdp_redirect: 0
> rx1_lro_packets: 0
> rx1_lro_bytes: 0
> rx1_ecn_mark: 0
> rx1_removed_vlan_packets: 6691319211
> rx1_wqe_err: 0
> rx1_mpwqe_filler_cqes: 0
> rx1_mpwqe_filler_strides: 0
> rx1_buff_alloc_err: 0
> rx1_cqe_compress_blks: 0
> rx1_cqe_compress_pkts: 0
> rx1_page_reuse: 0
> rx1_cache_reuse: 2270019
> rx1_cache_full: 3343389331
> rx1_cache_empty: 6656
> rx1_cache_busy: 3343389585
> rx1_cache_waive: 0
> rx1_congst_umr: 0
> rx1_arfs_err: 0
> rx1_xdp_tx_xmit: 0
> rx1_xdp_tx_full: 0
> rx1_xdp_tx_err: 0
> rx1_xdp_tx_cqes: 0
> rx2_packets: 6618370416
> rx2_bytes: 3762508364015
> rx2_csum_complete: 6618370416
> rx2_csum_unnecessary: 0
> rx2_csum_unnecessary_inner: 0
> rx2_csum_none: 0
> rx2_xdp_drop: 0
> rx2_xdp_redirect: 0
> rx2_lro_packets: 0
> rx2_lro_bytes: 0
> rx2_ecn_mark: 0
> rx2_removed_vlan_packets: 6618370416
> rx2_wqe_err: 0
> rx2_mpwqe_filler_cqes: 0
> rx2_mpwqe_filler_strides: 0
> rx2_buff_alloc_err: 0
> rx2_cqe_compress_blks: 0
> rx2_cqe_compress_pkts: 0
> rx2_page_reuse: 0
> rx2_cache_reuse: 111419328
> rx2_cache_full: 1807563903
> rx2_cache_empty: 1390208158
> rx2_cache_busy: 1807564378
> rx2_cache_waive: 1390201722
> rx2_congst_umr: 0
> rx2_arfs_err: 0
> rx2_xdp_tx_xmit: 0
> rx2_xdp_tx_full: 0
> rx2_xdp_tx_err: 0
> rx2_xdp_tx_cqes: 0
> rx3_packets: 6665308976
> rx3_bytes: 3828546206006
> rx3_csum_complete: 6665308976
> rx3_csum_unnecessary: 0
> rx3_csum_unnecessary_inner: 0
> rx3_csum_none: 0
> rx3_xdp_drop: 0
> rx3_xdp_redirect: 0
> rx3_lro_packets: 0
> rx3_lro_bytes: 0
> rx3_ecn_mark: 0
> rx3_removed_vlan_packets: 6665308976
> rx3_wqe_err: 0
> rx3_mpwqe_filler_cqes: 0
> rx3_mpwqe_filler_strides: 0
> rx3_buff_alloc_err: 0
> rx3_cqe_compress_blks: 0
> rx3_cqe_compress_pkts: 0
> rx3_page_reuse: 0
> rx3_cache_reuse: 215779091
> rx3_cache_full: 1720040649
> rx3_cache_empty: 1396840926
> rx3_cache_busy: 1720041127
> rx3_cache_waive: 1396834493
> rx3_congst_umr: 0
> rx3_arfs_err: 0
> rx3_xdp_tx_xmit: 0
> rx3_xdp_tx_full: 0
> rx3_xdp_tx_err: 0
> rx3_xdp_tx_cqes: 0
> rx4_packets: 6764448165
> rx4_bytes: 3883101339142
> rx4_csum_complete: 6764448165
> rx4_csum_unnecessary: 0
> rx4_csum_unnecessary_inner: 0
> rx4_csum_none: 0
> rx4_xdp_drop: 0
> rx4_xdp_redirect: 0
> rx4_lro_packets: 0
> rx4_lro_bytes: 0
> rx4_ecn_mark: 0
> rx4_removed_vlan_packets: 6764448165
> rx4_wqe_err: 0
> rx4_mpwqe_filler_cqes: 0
> rx4_mpwqe_filler_strides: 0
> rx4_buff_alloc_err: 0
> rx4_cqe_compress_blks: 0
> rx4_cqe_compress_pkts: 0
> rx4_page_reuse: 0
> rx4_cache_reuse: 1930710653
> rx4_cache_full: 6490815
> rx4_cache_empty: 1445028605
> rx4_cache_busy: 6491478
> rx4_cache_waive: 1445022392
> rx4_congst_umr: 0
> rx4_arfs_err: 0
> rx4_xdp_tx_xmit: 0
> rx4_xdp_tx_full: 0
> rx4_xdp_tx_err: 0
> rx4_xdp_tx_cqes: 0
> rx5_packets: 6736853264
> rx5_bytes: 3925186068552
> rx5_csum_complete: 6736853264
> rx5_csum_unnecessary: 0
> rx5_csum_unnecessary_inner: 0
> rx5_csum_none: 0
> rx5_xdp_drop: 0
> rx5_xdp_redirect: 0
> rx5_lro_packets: 0
> rx5_lro_bytes: 0
> rx5_ecn_mark: 0
> rx5_removed_vlan_packets: 6736853264
> rx5_wqe_err: 0
> rx5_mpwqe_filler_cqes: 0
> rx5_mpwqe_filler_strides: 0
> rx5_buff_alloc_err: 0
> rx5_cqe_compress_blks: 0
> rx5_cqe_compress_pkts: 0
> rx5_page_reuse: 0
> rx5_cache_reuse: 7283914
> rx5_cache_full: 3361142463
> rx5_cache_empty: 6656
> rx5_cache_busy: 3361142718
> rx5_cache_waive: 0
> rx5_congst_umr: 0
> rx5_arfs_err: 0
> rx5_xdp_tx_xmit: 0
> rx5_xdp_tx_full: 0
> rx5_xdp_tx_err: 0
> rx5_xdp_tx_cqes: 0
> rx6_packets: 6751588828
> rx6_bytes: 3860537598885
> rx6_csum_complete: 6751588828
> rx6_csum_unnecessary: 0
> rx6_csum_unnecessary_inner: 0
> rx6_csum_none: 0
> rx6_xdp_drop: 0
> rx6_xdp_redirect: 0
> rx6_lro_packets: 0
> rx6_lro_bytes: 0
> rx6_ecn_mark: 0
> rx6_removed_vlan_packets: 6751588828
> rx6_wqe_err: 0
> rx6_mpwqe_filler_cqes: 0
> rx6_mpwqe_filler_strides: 0
> rx6_buff_alloc_err: 0
> rx6_cqe_compress_blks: 0
> rx6_cqe_compress_pkts: 0
> rx6_page_reuse: 0
> rx6_cache_reuse: 96032126
> rx6_cache_full: 1857890923
> rx6_cache_empty: 1421877543
> rx6_cache_busy: 1857891399
> rx6_cache_waive: 1421871110
> rx6_congst_umr: 0
> rx6_arfs_err: 0
> rx6_xdp_tx_xmit: 0
> rx6_xdp_tx_full: 0
> rx6_xdp_tx_err: 0
> rx6_xdp_tx_cqes: 0
> rx7_packets: 6935300074
> rx7_bytes: 4004713524388
> rx7_csum_complete: 6935300074
> rx7_csum_unnecessary: 0
> rx7_csum_unnecessary_inner: 0
> rx7_csum_none: 0
> rx7_xdp_drop: 0
> rx7_xdp_redirect: 0
> rx7_lro_packets: 0
> rx7_lro_bytes: 0
> rx7_ecn_mark: 0
> rx7_removed_vlan_packets: 6935300074
> rx7_wqe_err: 0
> rx7_mpwqe_filler_cqes: 0
> rx7_mpwqe_filler_strides: 0
> rx7_buff_alloc_err: 0
> rx7_cqe_compress_blks: 0
> rx7_cqe_compress_pkts: 0
> rx7_page_reuse: 0
> rx7_cache_reuse: 17555187
> rx7_cache_full: 3450094595
> rx7_cache_empty: 6656
> rx7_cache_busy: 3450094849
> rx7_cache_waive: 0
> rx7_congst_umr: 0
> rx7_arfs_err: 0
> rx7_xdp_tx_xmit: 0
> rx7_xdp_tx_full: 0
> rx7_xdp_tx_err: 0
> rx7_xdp_tx_cqes: 0
> rx8_packets: 6678640094
> rx8_bytes: 3783722686028
> rx8_csum_complete: 6678640094
> rx8_csum_unnecessary: 0
> rx8_csum_unnecessary_inner: 0
> rx8_csum_none: 0
> rx8_xdp_drop: 0
> rx8_xdp_redirect: 0
> rx8_lro_packets: 0
> rx8_lro_bytes: 0
> rx8_ecn_mark: 0
> rx8_removed_vlan_packets: 6678640094
> rx8_wqe_err: 0
> rx8_mpwqe_filler_cqes: 0
> rx8_mpwqe_filler_strides: 0
> rx8_buff_alloc_err: 0
> rx8_cqe_compress_blks: 0
> rx8_cqe_compress_pkts: 0
> rx8_page_reuse: 0
> rx8_cache_reuse: 71006578
> rx8_cache_full: 1879380649
> rx8_cache_empty: 1388938999
> rx8_cache_busy: 1879381123
> rx8_cache_waive: 1388932565
> rx8_congst_umr: 0
> rx8_arfs_err: 0
> rx8_xdp_tx_xmit: 0
> rx8_xdp_tx_full: 0
> rx8_xdp_tx_err: 0
> rx8_xdp_tx_cqes: 0
> rx9_packets: 6709855557
> rx9_bytes: 3849522227880
> rx9_csum_complete: 6709855557
> rx9_csum_unnecessary: 0
> rx9_csum_unnecessary_inner: 0
> rx9_csum_none: 0
> rx9_xdp_drop: 0
> rx9_xdp_redirect: 0
> rx9_lro_packets: 0
> rx9_lro_bytes: 0
> rx9_ecn_mark: 0
> rx9_removed_vlan_packets: 6709855557
> rx9_wqe_err: 0
> rx9_mpwqe_filler_cqes: 0
> rx9_mpwqe_filler_strides: 0
> rx9_buff_alloc_err: 0
> rx9_cqe_compress_blks: 0
> rx9_cqe_compress_pkts: 0
> rx9_page_reuse: 0
> rx9_cache_reuse: 108980215
> rx9_cache_full: 1822730121
> rx9_cache_empty: 1423223623
> rx9_cache_busy: 1822730594
> rx9_cache_waive: 1423217187
> rx9_congst_umr: 0
> rx9_arfs_err: 0
> rx9_xdp_tx_xmit: 0
> rx9_xdp_tx_full: 0
> rx9_xdp_tx_err: 0
> rx9_xdp_tx_cqes: 0
> rx10_packets: 6761861066
> rx10_bytes: 3816266733385
> rx10_csum_complete: 6761861066
> rx10_csum_unnecessary: 0
> rx10_csum_unnecessary_inner: 0
> rx10_csum_none: 0
> rx10_xdp_drop: 0
> rx10_xdp_redirect: 0
> rx10_lro_packets: 0
> rx10_lro_bytes: 0
> rx10_ecn_mark: 0
> rx10_removed_vlan_packets: 6761861066
> rx10_wqe_err: 0
> rx10_mpwqe_filler_cqes: 0
> rx10_mpwqe_filler_strides: 0
> rx10_buff_alloc_err: 0
> rx10_cqe_compress_blks: 0
> rx10_cqe_compress_pkts: 0
> rx10_page_reuse: 0
> rx10_cache_reuse: 3489300
> rx10_cache_full: 3377440977
> rx10_cache_empty: 6656
> rx10_cache_busy: 3377441216
> rx10_cache_waive: 0
> rx10_congst_umr: 0
> rx10_arfs_err: 0
> rx10_xdp_tx_xmit: 0
> rx10_xdp_tx_full: 0
> rx10_xdp_tx_err: 0
> rx10_xdp_tx_cqes: 0
> rx11_packets: 6868113938
> rx11_bytes: 4048196300710
> rx11_csum_complete: 6868113938
> rx11_csum_unnecessary: 0
> rx11_csum_unnecessary_inner: 0
> rx11_csum_none: 0
> rx11_xdp_drop: 0
> rx11_xdp_redirect: 0
> rx11_lro_packets: 0
> rx11_lro_bytes: 0
> rx11_ecn_mark: 0
> rx11_removed_vlan_packets: 6868113938
> rx11_wqe_err: 0
> rx11_mpwqe_filler_cqes: 0
> rx11_mpwqe_filler_strides: 0
> rx11_buff_alloc_err: 0
> rx11_cqe_compress_blks: 0
> rx11_cqe_compress_pkts: 0
> rx11_page_reuse: 0
> rx11_cache_reuse: 1948516819
> rx11_cache_full: 17132157
> rx11_cache_empty: 1468413985
> rx11_cache_busy: 17132820
> rx11_cache_waive: 1468407772
> rx11_congst_umr: 0
> rx11_arfs_err: 0
> rx11_xdp_tx_xmit: 0
> rx11_xdp_tx_full: 0
> rx11_xdp_tx_err: 0
> rx11_xdp_tx_cqes: 0
> rx12_packets: 6742955386
> rx12_bytes: 3865747629271
> rx12_csum_complete: 6742955386
> rx12_csum_unnecessary: 0
> rx12_csum_unnecessary_inner: 0
> rx12_csum_none: 0
> rx12_xdp_drop: 0
> rx12_xdp_redirect: 0
> rx12_lro_packets: 0
> rx12_lro_bytes: 0
> rx12_ecn_mark: 0
> rx12_removed_vlan_packets: 6742955386
> rx12_wqe_err: 0
> rx12_mpwqe_filler_cqes: 0
> rx12_mpwqe_filler_strides: 0
> rx12_buff_alloc_err: 0
> rx12_cqe_compress_blks: 0
> rx12_cqe_compress_pkts: 0
> rx12_page_reuse: 0
> rx12_cache_reuse: 30809331
> rx12_cache_full: 3340668106
> rx12_cache_empty: 6656
> rx12_cache_busy: 3340668333
> rx12_cache_waive: 0
> rx12_congst_umr: 0
> rx12_arfs_err: 0
> rx12_xdp_tx_xmit: 0
> rx12_xdp_tx_full: 0
> rx12_xdp_tx_err: 0
> rx12_xdp_tx_cqes: 0
> rx13_packets: 6707028036
> rx13_bytes: 3813462190623
> rx13_csum_complete: 6707028036
> rx13_csum_unnecessary: 0
> rx13_csum_unnecessary_inner: 0
> rx13_csum_none: 0
> rx13_xdp_drop: 0
> rx13_xdp_redirect: 0
> rx13_lro_packets: 0
> rx13_lro_bytes: 0
> rx13_ecn_mark: 0
> rx13_removed_vlan_packets: 6707028036
> rx13_wqe_err: 0
> rx13_mpwqe_filler_cqes: 0
> rx13_mpwqe_filler_strides: 0
> rx13_buff_alloc_err: 0
> rx13_cqe_compress_blks: 0
> rx13_cqe_compress_pkts: 0
> rx13_page_reuse: 0
> rx13_cache_reuse: 14951053
> rx13_cache_full: 3338562710
> rx13_cache_empty: 6656
> rx13_cache_busy: 3338562963
> rx13_cache_waive: 0
> rx13_congst_umr: 0
> rx13_arfs_err: 0
> rx13_xdp_tx_xmit: 0
> rx13_xdp_tx_full: 0
> rx13_xdp_tx_err: 0
> rx13_xdp_tx_cqes: 0
> rx14_packets: 6737074410
> rx14_bytes: 3868905276119
> rx14_csum_complete: 6737074410
> rx14_csum_unnecessary: 0
> rx14_csum_unnecessary_inner: 0
> rx14_csum_none: 0
> rx14_xdp_drop: 0
> rx14_xdp_redirect: 0
> rx14_lro_packets: 0
> rx14_lro_bytes: 0
> rx14_ecn_mark: 0
> rx14_removed_vlan_packets: 6737074410
> rx14_wqe_err: 0
> rx14_mpwqe_filler_cqes: 0
> rx14_mpwqe_filler_strides: 0
> rx14_buff_alloc_err: 0
> rx14_cqe_compress_blks: 0
> rx14_cqe_compress_pkts: 0
> rx14_page_reuse: 0
> rx14_cache_reuse: 967799432
> rx14_cache_full: 982704312
> rx14_cache_empty: 1418039639
> rx14_cache_busy: 982704789
> rx14_cache_waive: 1418033206
> rx14_congst_umr: 0
> rx14_arfs_err: 0
> rx14_xdp_tx_xmit: 0
> rx14_xdp_tx_full: 0
> rx14_xdp_tx_err: 0
> rx14_xdp_tx_cqes: 0
> rx15_packets: 6641887441
> rx15_bytes: 3742874400402
> rx15_csum_complete: 6641887441
> rx15_csum_unnecessary: 0
> rx15_csum_unnecessary_inner: 0
> rx15_csum_none: 0
> rx15_xdp_drop: 0
> rx15_xdp_redirect: 0
> rx15_lro_packets: 0
> rx15_lro_bytes: 0
> rx15_ecn_mark: 0
> rx15_removed_vlan_packets: 6641887441
> rx15_wqe_err: 0
> rx15_mpwqe_filler_cqes: 0
> rx15_mpwqe_filler_strides: 0
> rx15_buff_alloc_err: 0
> rx15_cqe_compress_blks: 0
> rx15_cqe_compress_pkts: 0
> rx15_page_reuse: 0
> rx15_cache_reuse: 1920227538
> rx15_cache_full: 19386129
> rx15_cache_empty: 1381335137
> rx15_cache_busy: 19387693
> rx15_cache_waive: 1381329825
> rx15_congst_umr: 0
> rx15_arfs_err: 0
> rx15_xdp_tx_xmit: 0
> rx15_xdp_tx_full: 0
> rx15_xdp_tx_err: 0
> rx15_xdp_tx_cqes: 0
> rx16_packets: 5420472874
> rx16_bytes: 3079293332581
> rx16_csum_complete: 5420472874
> rx16_csum_unnecessary: 0
> rx16_csum_unnecessary_inner: 0
> rx16_csum_none: 0
> rx16_xdp_drop: 0
> rx16_xdp_redirect: 0
> rx16_lro_packets: 0
> rx16_lro_bytes: 0
> rx16_ecn_mark: 0
> rx16_removed_vlan_packets: 5420472874
> rx16_wqe_err: 0
> rx16_mpwqe_filler_cqes: 0
> rx16_mpwqe_filler_strides: 0
> rx16_buff_alloc_err: 0
> rx16_cqe_compress_blks: 0
> rx16_cqe_compress_pkts: 0
> rx16_page_reuse: 0
> rx16_cache_reuse: 2361079
> rx16_cache_full: 2707875103
> rx16_cache_empty: 6656
> rx16_cache_busy: 2707875349
> rx16_cache_waive: 0
> rx16_congst_umr: 0
> rx16_arfs_err: 0
> rx16_xdp_tx_xmit: 0
> rx16_xdp_tx_full: 0
> rx16_xdp_tx_err: 0
> rx16_xdp_tx_cqes: 0
> rx17_packets: 5428380986
> rx17_bytes: 3080981893118
> rx17_csum_complete: 5428380986
> rx17_csum_unnecessary: 0
> rx17_csum_unnecessary_inner: 0
> rx17_csum_none: 0
> rx17_xdp_drop: 0
> rx17_xdp_redirect: 0
> rx17_lro_packets: 0
> rx17_lro_bytes: 0
> rx17_ecn_mark: 0
> rx17_removed_vlan_packets: 5428380986
> rx17_wqe_err: 0
> rx17_mpwqe_filler_cqes: 0
> rx17_mpwqe_filler_strides: 0
> rx17_buff_alloc_err: 0
> rx17_cqe_compress_blks: 0
> rx17_cqe_compress_pkts: 0
> rx17_page_reuse: 0
> rx17_cache_reuse: 1552266402
> rx17_cache_full: 5947505
> rx17_cache_empty: 1155981856
> rx17_cache_busy: 5948870
> rx17_cache_waive: 1155976345
> rx17_congst_umr: 0
> rx17_arfs_err: 0
> rx17_xdp_tx_xmit: 0
> rx17_xdp_tx_full: 0
> rx17_xdp_tx_err: 0
> rx17_xdp_tx_cqes: 0
> rx18_packets: 5529118410
> rx18_bytes: 3254749573833
> rx18_csum_complete: 5529118410
> rx18_csum_unnecessary: 0
> rx18_csum_unnecessary_inner: 0
> rx18_csum_none: 0
> rx18_xdp_drop: 0
> rx18_xdp_redirect: 0
> rx18_lro_packets: 0
> rx18_lro_bytes: 0
> rx18_ecn_mark: 0
> rx18_removed_vlan_packets: 5529118410
> rx18_wqe_err: 0
> rx18_mpwqe_filler_cqes: 0
> rx18_mpwqe_filler_strides: 0
> rx18_buff_alloc_err: 0
> rx18_cqe_compress_blks: 0
> rx18_cqe_compress_pkts: 0
> rx18_page_reuse: 0
> rx18_cache_reuse: 67438840
> rx18_cache_full: 1536718472
> rx18_cache_empty: 1160408072
> rx18_cache_busy: 1536718932
> rx18_cache_waive: 1160401638
> rx18_congst_umr: 0
> rx18_arfs_err: 0
> rx18_xdp_tx_xmit: 0
> rx18_xdp_tx_full: 0
> rx18_xdp_tx_err: 0
> rx18_xdp_tx_cqes: 0
> rx19_packets: 5449932653
> rx19_bytes: 3148726579411
> rx19_csum_complete: 5449932653
> rx19_csum_unnecessary: 0
> rx19_csum_unnecessary_inner: 0
> rx19_csum_none: 0
> rx19_xdp_drop: 0
> rx19_xdp_redirect: 0
> rx19_lro_packets: 0
> rx19_lro_bytes: 0
> rx19_ecn_mark: 0
> rx19_removed_vlan_packets: 5449932653
> rx19_wqe_err: 0
> rx19_mpwqe_filler_cqes: 0
> rx19_mpwqe_filler_strides: 0
> rx19_buff_alloc_err: 0
> rx19_cqe_compress_blks: 0
> rx19_cqe_compress_pkts: 0
> rx19_page_reuse: 0
> rx19_cache_reuse: 1537841743
> rx19_cache_full: 9920960
> rx19_cache_empty: 1177208938
> rx19_cache_busy: 9922299
> rx19_cache_waive: 1177203401
> rx19_congst_umr: 0
> rx19_arfs_err: 0
> rx19_xdp_tx_xmit: 0
> rx19_xdp_tx_full: 0
> rx19_xdp_tx_err: 0
> rx19_xdp_tx_cqes: 0
> rx20_packets: 5407910071
> rx20_bytes: 3123560861922
> rx20_csum_complete: 5407910071
> rx20_csum_unnecessary: 0
> rx20_csum_unnecessary_inner: 0
> rx20_csum_none: 0
> rx20_xdp_drop: 0
> rx20_xdp_redirect: 0
> rx20_lro_packets: 0
> rx20_lro_bytes: 0
> rx20_ecn_mark: 0
> rx20_removed_vlan_packets: 5407910071
> rx20_wqe_err: 0
> rx20_mpwqe_filler_cqes: 0
> rx20_mpwqe_filler_strides: 0
> rx20_buff_alloc_err: 0
> rx20_cqe_compress_blks: 0
> rx20_cqe_compress_pkts: 0
> rx20_page_reuse: 0
> rx20_cache_reuse: 10255209
> rx20_cache_full: 2693699571
> rx20_cache_empty: 6656
> rx20_cache_busy: 2693699823
> rx20_cache_waive: 0
> rx20_congst_umr: 0
> rx20_arfs_err: 0
> rx20_xdp_tx_xmit: 0
> rx20_xdp_tx_full: 0
> rx20_xdp_tx_err: 0
> rx20_xdp_tx_cqes: 0
> rx21_packets: 5417498508
> rx21_bytes: 3131335892379
> rx21_csum_complete: 5417498508
> rx21_csum_unnecessary: 0
> rx21_csum_unnecessary_inner: 0
> rx21_csum_none: 0
> rx21_xdp_drop: 0
> rx21_xdp_redirect: 0
> rx21_lro_packets: 0
> rx21_lro_bytes: 0
> rx21_ecn_mark: 0
> rx21_removed_vlan_packets: 5417498508
> rx21_wqe_err: 0
> rx21_mpwqe_filler_cqes: 0
> rx21_mpwqe_filler_strides: 0
> rx21_buff_alloc_err: 0
> rx21_cqe_compress_blks: 0
> rx21_cqe_compress_pkts: 0
> rx21_page_reuse: 0
> rx21_cache_reuse: 192662917
> rx21_cache_full: 1374120417
> rx21_cache_empty: 1141972100
> rx21_cache_busy: 1374120891
> rx21_cache_waive: 1141965665
> rx21_congst_umr: 0
> rx21_arfs_err: 0
> rx21_xdp_tx_xmit: 0
> rx21_xdp_tx_full: 0
> rx21_xdp_tx_err: 0
> rx21_xdp_tx_cqes: 0
> rx22_packets: 5613634706
> rx22_bytes: 3240055099058
> rx22_csum_complete: 5613634706
> rx22_csum_unnecessary: 0
> rx22_csum_unnecessary_inner: 0
> rx22_csum_none: 0
> rx22_xdp_drop: 0
> rx22_xdp_redirect: 0
> rx22_lro_packets: 0
> rx22_lro_bytes: 0
> rx22_ecn_mark: 0
> rx22_removed_vlan_packets: 5613634706
> rx22_wqe_err: 0
> rx22_mpwqe_filler_cqes: 0
> rx22_mpwqe_filler_strides: 0
> rx22_buff_alloc_err: 0
> rx22_cqe_compress_blks: 0
> rx22_cqe_compress_pkts: 0
> rx22_page_reuse: 0
> rx22_cache_reuse: 12161531
> rx22_cache_full: 2794655567
> rx22_cache_empty: 6656
> rx22_cache_busy: 2794655821
> rx22_cache_waive: 0
> rx22_congst_umr: 0
> rx22_arfs_err: 0
> rx22_xdp_tx_xmit: 0
> rx22_xdp_tx_full: 0
> rx22_xdp_tx_err: 0
> rx22_xdp_tx_cqes: 0
> rx23_packets: 5389977167
> rx23_bytes: 3054270771559
> rx23_csum_complete: 5389977167
> rx23_csum_unnecessary: 0
> rx23_csum_unnecessary_inner: 0
> rx23_csum_none: 0
> rx23_xdp_drop: 0
> rx23_xdp_redirect: 0
> rx23_lro_packets: 0
> rx23_lro_bytes: 0
> rx23_ecn_mark: 0
> rx23_removed_vlan_packets: 5389977167
> rx23_wqe_err: 0
> rx23_mpwqe_filler_cqes: 0
> rx23_mpwqe_filler_strides: 0
> rx23_buff_alloc_err: 0
> rx23_cqe_compress_blks: 0
> rx23_cqe_compress_pkts: 0
> rx23_page_reuse: 0
> rx23_cache_reuse: 709328
> rx23_cache_full: 2694279000
> rx23_cache_empty: 6656
> rx23_cache_busy: 2694279252
> rx23_cache_waive: 0
> rx23_congst_umr: 0
> rx23_arfs_err: 0
> rx23_xdp_tx_xmit: 0
> rx23_xdp_tx_full: 0
> rx23_xdp_tx_err: 0
> rx23_xdp_tx_cqes: 0
> rx24_packets: 5547561932
> rx24_bytes: 3166602453443
> rx24_csum_complete: 5547561932
> rx24_csum_unnecessary: 0
> rx24_csum_unnecessary_inner: 0
> rx24_csum_none: 0
> rx24_xdp_drop: 0
> rx24_xdp_redirect: 0
> rx24_lro_packets: 0
> rx24_lro_bytes: 0
> rx24_ecn_mark: 0
> rx24_removed_vlan_packets: 5547561932
> rx24_wqe_err: 0
> rx24_mpwqe_filler_cqes: 0
> rx24_mpwqe_filler_strides: 0
> rx24_buff_alloc_err: 0
> rx24_cqe_compress_blks: 0
> rx24_cqe_compress_pkts: 0
> rx24_page_reuse: 0
> rx24_cache_reuse: 57885119
> rx24_cache_full: 1529450077
> rx24_cache_empty: 1186451948
> rx24_cache_busy: 1529450553
> rx24_cache_waive: 1186445515
> rx24_congst_umr: 0
> rx24_arfs_err: 0
> rx24_xdp_tx_xmit: 0
> rx24_xdp_tx_full: 0
> rx24_xdp_tx_err: 0
> rx24_xdp_tx_cqes: 0
> rx25_packets: 5414569326
> rx25_bytes: 3184757708091
> rx25_csum_complete: 5414569326
> rx25_csum_unnecessary: 0
> rx25_csum_unnecessary_inner: 0
> rx25_csum_none: 0
> rx25_xdp_drop: 0
> rx25_xdp_redirect: 0
> rx25_lro_packets: 0
> rx25_lro_bytes: 0
> rx25_ecn_mark: 0
> rx25_removed_vlan_packets: 5414569326
> rx25_wqe_err: 0
> rx25_mpwqe_filler_cqes: 0
> rx25_mpwqe_filler_strides: 0
> rx25_buff_alloc_err: 0
> rx25_cqe_compress_blks: 0
> rx25_cqe_compress_pkts: 0
> rx25_page_reuse: 0
> rx25_cache_reuse: 5080853
> rx25_cache_full: 2702203555
> rx25_cache_empty: 6656
> rx25_cache_busy: 2702203807
> rx25_cache_waive: 0
> rx25_congst_umr: 0
> rx25_arfs_err: 0
> rx25_xdp_tx_xmit: 0
> rx25_xdp_tx_full: 0
> rx25_xdp_tx_err: 0
> rx25_xdp_tx_cqes: 0
> rx26_packets: 5479972151
> rx26_bytes: 3110642276239
> rx26_csum_complete: 5479972151
> rx26_csum_unnecessary: 0
> rx26_csum_unnecessary_inner: 0
> rx26_csum_none: 0
> rx26_xdp_drop: 0
> rx26_xdp_redirect: 0
> rx26_lro_packets: 0
> rx26_lro_bytes: 0
> rx26_ecn_mark: 0
> rx26_removed_vlan_packets: 5479972151
> rx26_wqe_err: 0
> rx26_mpwqe_filler_cqes: 0
> rx26_mpwqe_filler_strides: 0
> rx26_buff_alloc_err: 0
> rx26_cqe_compress_blks: 0
> rx26_cqe_compress_pkts: 0
> rx26_page_reuse: 0
> rx26_cache_reuse: 26543335
> rx26_cache_full: 2713442485
> rx26_cache_empty: 6656
> rx26_cache_busy: 2713442737
> rx26_cache_waive: 0
> rx26_congst_umr: 0
> rx26_arfs_err: 0
> rx26_xdp_tx_xmit: 0
> rx26_xdp_tx_full: 0
> rx26_xdp_tx_err: 0
> rx26_xdp_tx_cqes: 0
> rx27_packets: 5337113900
> rx27_bytes: 3068966906075
> rx27_csum_complete: 5337113900
> rx27_csum_unnecessary: 0
> rx27_csum_unnecessary_inner: 0
> rx27_csum_none: 0
> rx27_xdp_drop: 0
> rx27_xdp_redirect: 0
> rx27_lro_packets: 0
> rx27_lro_bytes: 0
> rx27_ecn_mark: 0
> rx27_removed_vlan_packets: 5337113900
> rx27_wqe_err: 0
> rx27_mpwqe_filler_cqes: 0
> rx27_mpwqe_filler_strides: 0
> rx27_buff_alloc_err: 0
> rx27_cqe_compress_blks: 0
> rx27_cqe_compress_pkts: 0
> rx27_page_reuse: 0
> rx27_cache_reuse: 1539298962
> rx27_cache_full: 10861919
> rx27_cache_empty: 1117173179
> rx27_cache_busy: 12091463
> rx27_cache_waive: 1118395847
> rx27_congst_umr: 0
> rx27_arfs_err: 0
> rx27_xdp_tx_xmit: 0
> rx27_xdp_tx_full: 0
> rx27_xdp_tx_err: 0
> rx27_xdp_tx_cqes: 0
> rx28_packets: 0
> rx28_bytes: 0
> rx28_csum_complete: 0
> rx28_csum_unnecessary: 0
> rx28_csum_unnecessary_inner: 0
> rx28_csum_none: 0
> rx28_xdp_drop: 0
> rx28_xdp_redirect: 0
> rx28_lro_packets: 0
> rx28_lro_bytes: 0
> rx28_ecn_mark: 0
> rx28_removed_vlan_packets: 0
> rx28_wqe_err: 0
> rx28_mpwqe_filler_cqes: 0
> rx28_mpwqe_filler_strides: 0
> rx28_buff_alloc_err: 0
> rx28_cqe_compress_blks: 0
> rx28_cqe_compress_pkts: 0
> rx28_page_reuse: 0
> rx28_cache_reuse: 0
> rx28_cache_full: 0
> rx28_cache_empty: 2560
> rx28_cache_busy: 0
> rx28_cache_waive: 0
> rx28_congst_umr: 0
> rx28_arfs_err: 0
> rx28_xdp_tx_xmit: 0
> rx28_xdp_tx_full: 0
> rx28_xdp_tx_err: 0
> rx28_xdp_tx_cqes: 0
> rx29_packets: 0
> rx29_bytes: 0
> rx29_csum_complete: 0
> rx29_csum_unnecessary: 0
> rx29_csum_unnecessary_inner: 0
> rx29_csum_none: 0
> rx29_xdp_drop: 0
> rx29_xdp_redirect: 0
> rx29_lro_packets: 0
> rx29_lro_bytes: 0
> rx29_ecn_mark: 0
> rx29_removed_vlan_packets: 0
> rx29_wqe_err: 0
> rx29_mpwqe_filler_cqes: 0
> rx29_mpwqe_filler_strides: 0
> rx29_buff_alloc_err: 0
> rx29_cqe_compress_blks: 0
> rx29_cqe_compress_pkts: 0
> rx29_page_reuse: 0
> rx29_cache_reuse: 0
> rx29_cache_full: 0
> rx29_cache_empty: 2560
> rx29_cache_busy: 0
> rx29_cache_waive: 0
> rx29_congst_umr: 0
> rx29_arfs_err: 0
> rx29_xdp_tx_xmit: 0
> rx29_xdp_tx_full: 0
> rx29_xdp_tx_err: 0
> rx29_xdp_tx_cqes: 0
> rx30_packets: 0
> rx30_bytes: 0
> rx30_csum_complete: 0
> rx30_csum_unnecessary: 0
> rx30_csum_unnecessary_inner: 0
> rx30_csum_none: 0
> rx30_xdp_drop: 0
> rx30_xdp_redirect: 0
> rx30_lro_packets: 0
> rx30_lro_bytes: 0
> rx30_ecn_mark: 0
> rx30_removed_vlan_packets: 0
> rx30_wqe_err: 0
> rx30_mpwqe_filler_cqes: 0
> rx30_mpwqe_filler_strides: 0
> rx30_buff_alloc_err: 0
> rx30_cqe_compress_blks: 0
> rx30_cqe_compress_pkts: 0
> rx30_page_reuse: 0
> rx30_cache_reuse: 0
> rx30_cache_full: 0
> rx30_cache_empty: 2560
> rx30_cache_busy: 0
> rx30_cache_waive: 0
> rx30_congst_umr: 0
> rx30_arfs_err: 0
> rx30_xdp_tx_xmit: 0
> rx30_xdp_tx_full: 0
> rx30_xdp_tx_err: 0
> rx30_xdp_tx_cqes: 0
> rx31_packets: 0
> rx31_bytes: 0
> rx31_csum_complete: 0
> rx31_csum_unnecessary: 0
> rx31_csum_unnecessary_inner: 0
> rx31_csum_none: 0
> rx31_xdp_drop: 0
> rx31_xdp_redirect: 0
> rx31_lro_packets: 0
> rx31_lro_bytes: 0
> rx31_ecn_mark: 0
> rx31_removed_vlan_packets: 0
> rx31_wqe_err: 0
> rx31_mpwqe_filler_cqes: 0
> rx31_mpwqe_filler_strides: 0
> rx31_buff_alloc_err: 0
> rx31_cqe_compress_blks: 0
> rx31_cqe_compress_pkts: 0
> rx31_page_reuse: 0
> rx31_cache_reuse: 0
> rx31_cache_full: 0
> rx31_cache_empty: 2560
> rx31_cache_busy: 0
> rx31_cache_waive: 0
> rx31_congst_umr: 0
> rx31_arfs_err: 0
> rx31_xdp_tx_xmit: 0
> rx31_xdp_tx_full: 0
> rx31_xdp_tx_err: 0
> rx31_xdp_tx_cqes: 0
> rx32_packets: 0
> rx32_bytes: 0
> rx32_csum_complete: 0
> rx32_csum_unnecessary: 0
> rx32_csum_unnecessary_inner: 0
> rx32_csum_none: 0
> rx32_xdp_drop: 0
> rx32_xdp_redirect: 0
> rx32_lro_packets: 0
> rx32_lro_bytes: 0
> rx32_ecn_mark: 0
> rx32_removed_vlan_packets: 0
> rx32_wqe_err: 0
> rx32_mpwqe_filler_cqes: 0
> rx32_mpwqe_filler_strides: 0
> rx32_buff_alloc_err: 0
> rx32_cqe_compress_blks: 0
> rx32_cqe_compress_pkts: 0
> rx32_page_reuse: 0
> rx32_cache_reuse: 0
> rx32_cache_full: 0
> rx32_cache_empty: 2560
> rx32_cache_busy: 0
> rx32_cache_waive: 0
> rx32_congst_umr: 0
> rx32_arfs_err: 0
> rx32_xdp_tx_xmit: 0
> rx32_xdp_tx_full: 0
> rx32_xdp_tx_err: 0
> rx32_xdp_tx_cqes: 0
> rx33_packets: 0
> rx33_bytes: 0
> rx33_csum_complete: 0
> rx33_csum_unnecessary: 0
> rx33_csum_unnecessary_inner: 0
> rx33_csum_none: 0
> rx33_xdp_drop: 0
> rx33_xdp_redirect: 0
> rx33_lro_packets: 0
> rx33_lro_bytes: 0
> rx33_ecn_mark: 0
> rx33_removed_vlan_packets: 0
> rx33_wqe_err: 0
> rx33_mpwqe_filler_cqes: 0
> rx33_mpwqe_filler_strides: 0
> rx33_buff_alloc_err: 0
> rx33_cqe_compress_blks: 0
> rx33_cqe_compress_pkts: 0
> rx33_page_reuse: 0
> rx33_cache_reuse: 0
> rx33_cache_full: 0
> rx33_cache_empty: 2560
> rx33_cache_busy: 0
> rx33_cache_waive: 0
> rx33_congst_umr: 0
> rx33_arfs_err: 0
> rx33_xdp_tx_xmit: 0
> rx33_xdp_tx_full: 0
> rx33_xdp_tx_err: 0
> rx33_xdp_tx_cqes: 0
> rx34_packets: 0
> rx34_bytes: 0
> rx34_csum_complete: 0
> rx34_csum_unnecessary: 0
> rx34_csum_unnecessary_inner: 0
> rx34_csum_none: 0
> rx34_xdp_drop: 0
> rx34_xdp_redirect: 0
> rx34_lro_packets: 0
> rx34_lro_bytes: 0
> rx34_ecn_mark: 0
> rx34_removed_vlan_packets: 0
> rx34_wqe_err: 0
> rx34_mpwqe_filler_cqes: 0
> rx34_mpwqe_filler_strides: 0
> rx34_buff_alloc_err: 0
> rx34_cqe_compress_blks: 0
> rx34_cqe_compress_pkts: 0
> rx34_page_reuse: 0
> rx34_cache_reuse: 0
> rx34_cache_full: 0
> rx34_cache_empty: 2560
> rx34_cache_busy: 0
> rx34_cache_waive: 0
> rx34_congst_umr: 0
> rx34_arfs_err: 0
> rx34_xdp_tx_xmit: 0
> rx34_xdp_tx_full: 0
> rx34_xdp_tx_err: 0
> rx34_xdp_tx_cqes: 0
> rx35_packets: 0
> rx35_bytes: 0
> rx35_csum_complete: 0
> rx35_csum_unnecessary: 0
> rx35_csum_unnecessary_inner: 0
> rx35_csum_none: 0
> rx35_xdp_drop: 0
> rx35_xdp_redirect: 0
> rx35_lro_packets: 0
> rx35_lro_bytes: 0
> rx35_ecn_mark: 0
> rx35_removed_vlan_packets: 0
> rx35_wqe_err: 0
> rx35_mpwqe_filler_cqes: 0
> rx35_mpwqe_filler_strides: 0
> rx35_buff_alloc_err: 0
> rx35_cqe_compress_blks: 0
> rx35_cqe_compress_pkts: 0
> rx35_page_reuse: 0
> rx35_cache_reuse: 0
> rx35_cache_full: 0
> rx35_cache_empty: 2560
> rx35_cache_busy: 0
> rx35_cache_waive: 0
> rx35_congst_umr: 0
> rx35_arfs_err: 0
> rx35_xdp_tx_xmit: 0
> rx35_xdp_tx_full: 0
> rx35_xdp_tx_err: 0
> rx35_xdp_tx_cqes: 0
> rx36_packets: 0
> rx36_bytes: 0
> rx36_csum_complete: 0
> rx36_csum_unnecessary: 0
> rx36_csum_unnecessary_inner: 0
> rx36_csum_none: 0
> rx36_xdp_drop: 0
> rx36_xdp_redirect: 0
> rx36_lro_packets: 0
> rx36_lro_bytes: 0
> rx36_ecn_mark: 0
> rx36_removed_vlan_packets: 0
> rx36_wqe_err: 0
> rx36_mpwqe_filler_cqes: 0
> rx36_mpwqe_filler_strides: 0
> rx36_buff_alloc_err: 0
> rx36_cqe_compress_blks: 0
> rx36_cqe_compress_pkts: 0
> rx36_page_reuse: 0
> rx36_cache_reuse: 0
> rx36_cache_full: 0
> rx36_cache_empty: 2560
> rx36_cache_busy: 0
> rx36_cache_waive: 0
> rx36_congst_umr: 0
> rx36_arfs_err: 0
> rx36_xdp_tx_xmit: 0
> rx36_xdp_tx_full: 0
> rx36_xdp_tx_err: 0
> rx36_xdp_tx_cqes: 0
> rx37_packets: 0
> rx37_bytes: 0
> rx37_csum_complete: 0
> rx37_csum_unnecessary: 0
> rx37_csum_unnecessary_inner: 0
> rx37_csum_none: 0
> rx37_xdp_drop: 0
> rx37_xdp_redirect: 0
> rx37_lro_packets: 0
> rx37_lro_bytes: 0
> rx37_ecn_mark: 0
> rx37_removed_vlan_packets: 0
> rx37_wqe_err: 0
> rx37_mpwqe_filler_cqes: 0
> rx37_mpwqe_filler_strides: 0
> rx37_buff_alloc_err: 0
> rx37_cqe_compress_blks: 0
> rx37_cqe_compress_pkts: 0
> rx37_page_reuse: 0
> rx37_cache_reuse: 0
> rx37_cache_full: 0
> rx37_cache_empty: 2560
> rx37_cache_busy: 0
> rx37_cache_waive: 0
> rx37_congst_umr: 0
> rx37_arfs_err: 0
> rx37_xdp_tx_xmit: 0
> rx37_xdp_tx_full: 0
> rx37_xdp_tx_err: 0
> rx37_xdp_tx_cqes: 0
> rx38_packets: 0
> rx38_bytes: 0
> rx38_csum_complete: 0
> rx38_csum_unnecessary: 0
> rx38_csum_unnecessary_inner: 0
> rx38_csum_none: 0
> rx38_xdp_drop: 0
> rx38_xdp_redirect: 0
> rx38_lro_packets: 0
> rx38_lro_bytes: 0
> rx38_ecn_mark: 0
> rx38_removed_vlan_packets: 0
> rx38_wqe_err: 0
> rx38_mpwqe_filler_cqes: 0
> rx38_mpwqe_filler_strides: 0
> rx38_buff_alloc_err: 0
> rx38_cqe_compress_blks: 0
> rx38_cqe_compress_pkts: 0
> rx38_page_reuse: 0
> rx38_cache_reuse: 0
> rx38_cache_full: 0
> rx38_cache_empty: 2560
> rx38_cache_busy: 0
> rx38_cache_waive: 0
> rx38_congst_umr: 0
> rx38_arfs_err: 0
> rx38_xdp_tx_xmit: 0
> rx38_xdp_tx_full: 0
> rx38_xdp_tx_err: 0
> rx38_xdp_tx_cqes: 0
> rx39_packets: 0
> rx39_bytes: 0
> rx39_csum_complete: 0
> rx39_csum_unnecessary: 0
> rx39_csum_unnecessary_inner: 0
> rx39_csum_none: 0
> rx39_xdp_drop: 0
> rx39_xdp_redirect: 0
> rx39_lro_packets: 0
> rx39_lro_bytes: 0
> rx39_ecn_mark: 0
> rx39_removed_vlan_packets: 0
> rx39_wqe_err: 0
> rx39_mpwqe_filler_cqes: 0
> rx39_mpwqe_filler_strides: 0
> rx39_buff_alloc_err: 0
> rx39_cqe_compress_blks: 0
> rx39_cqe_compress_pkts: 0
> rx39_page_reuse: 0
> rx39_cache_reuse: 0
> rx39_cache_full: 0
> rx39_cache_empty: 2560
> rx39_cache_busy: 0
> rx39_cache_waive: 0
> rx39_congst_umr: 0
> rx39_arfs_err: 0
> rx39_xdp_tx_xmit: 0
> rx39_xdp_tx_full: 0
> rx39_xdp_tx_err: 0
> rx39_xdp_tx_cqes: 0
> rx40_packets: 0
> rx40_bytes: 0
> rx40_csum_complete: 0
> rx40_csum_unnecessary: 0
> rx40_csum_unnecessary_inner: 0
> rx40_csum_none: 0
> rx40_xdp_drop: 0
> rx40_xdp_redirect: 0
> rx40_lro_packets: 0
> rx40_lro_bytes: 0
> rx40_ecn_mark: 0
> rx40_removed_vlan_packets: 0
> rx40_wqe_err: 0
> rx40_mpwqe_filler_cqes: 0
> rx40_mpwqe_filler_strides: 0
> rx40_buff_alloc_err: 0
> rx40_cqe_compress_blks: 0
> rx40_cqe_compress_pkts: 0
> rx40_page_reuse: 0
> rx40_cache_reuse: 0
> rx40_cache_full: 0
> rx40_cache_empty: 2560
> rx40_cache_busy: 0
> rx40_cache_waive: 0
> rx40_congst_umr: 0
> rx40_arfs_err: 0
> rx40_xdp_tx_xmit: 0
> rx40_xdp_tx_full: 0
> rx40_xdp_tx_err: 0
> rx40_xdp_tx_cqes: 0
> rx41_packets: 0
> rx41_bytes: 0
> rx41_csum_complete: 0
> rx41_csum_unnecessary: 0
> rx41_csum_unnecessary_inner: 0
> rx41_csum_none: 0
> rx41_xdp_drop: 0
> rx41_xdp_redirect: 0
> rx41_lro_packets: 0
> rx41_lro_bytes: 0
> rx41_ecn_mark: 0
> rx41_removed_vlan_packets: 0
> rx41_wqe_err: 0
> rx41_mpwqe_filler_cqes: 0
> rx41_mpwqe_filler_strides: 0
> rx41_buff_alloc_err: 0
> rx41_cqe_compress_blks: 0
> rx41_cqe_compress_pkts: 0
> rx41_page_reuse: 0
> rx41_cache_reuse: 0
> rx41_cache_full: 0
> rx41_cache_empty: 2560
> rx41_cache_busy: 0
> rx41_cache_waive: 0
> rx41_congst_umr: 0
> rx41_arfs_err: 0
> rx41_xdp_tx_xmit: 0
> rx41_xdp_tx_full: 0
> rx41_xdp_tx_err: 0
> rx41_xdp_tx_cqes: 0
> rx42_packets: 0
> rx42_bytes: 0
> rx42_csum_complete: 0
> rx42_csum_unnecessary: 0
> rx42_csum_unnecessary_inner: 0
> rx42_csum_none: 0
> rx42_xdp_drop: 0
> rx42_xdp_redirect: 0
> rx42_lro_packets: 0
> rx42_lro_bytes: 0
> rx42_ecn_mark: 0
> rx42_removed_vlan_packets: 0
> rx42_wqe_err: 0
> rx42_mpwqe_filler_cqes: 0
> rx42_mpwqe_filler_strides: 0
> rx42_buff_alloc_err: 0
> rx42_cqe_compress_blks: 0
> rx42_cqe_compress_pkts: 0
> rx42_page_reuse: 0
> rx42_cache_reuse: 0
> rx42_cache_full: 0
> rx42_cache_empty: 2560
> rx42_cache_busy: 0
> rx42_cache_waive: 0
> rx42_congst_umr: 0
> rx42_arfs_err: 0
> rx42_xdp_tx_xmit: 0
> rx42_xdp_tx_full: 0
> rx42_xdp_tx_err: 0
> rx42_xdp_tx_cqes: 0
> rx43_packets: 0
> rx43_bytes: 0
> rx43_csum_complete: 0
> rx43_csum_unnecessary: 0
> rx43_csum_unnecessary_inner: 0
> rx43_csum_none: 0
> rx43_xdp_drop: 0
> rx43_xdp_redirect: 0
> rx43_lro_packets: 0
> rx43_lro_bytes: 0
> rx43_ecn_mark: 0
> rx43_removed_vlan_packets: 0
> rx43_wqe_err: 0
> rx43_mpwqe_filler_cqes: 0
> rx43_mpwqe_filler_strides: 0
> rx43_buff_alloc_err: 0
> rx43_cqe_compress_blks: 0
> rx43_cqe_compress_pkts: 0
> rx43_page_reuse: 0
> rx43_cache_reuse: 0
> rx43_cache_full: 0
> rx43_cache_empty: 2560
> rx43_cache_busy: 0
> rx43_cache_waive: 0
> rx43_congst_umr: 0
> rx43_arfs_err: 0
> rx43_xdp_tx_xmit: 0
> rx43_xdp_tx_full: 0
> rx43_xdp_tx_err: 0
> rx43_xdp_tx_cqes: 0
> rx44_packets: 0
> rx44_bytes: 0
> rx44_csum_complete: 0
> rx44_csum_unnecessary: 0
> rx44_csum_unnecessary_inner: 0
> rx44_csum_none: 0
> rx44_xdp_drop: 0
> rx44_xdp_redirect: 0
> rx44_lro_packets: 0
> rx44_lro_bytes: 0
> rx44_ecn_mark: 0
> rx44_removed_vlan_packets: 0
> rx44_wqe_err: 0
> rx44_mpwqe_filler_cqes: 0
> rx44_mpwqe_filler_strides: 0
> rx44_buff_alloc_err: 0
> rx44_cqe_compress_blks: 0
> rx44_cqe_compress_pkts: 0
> rx44_page_reuse: 0
> rx44_cache_reuse: 0
> rx44_cache_full: 0
> rx44_cache_empty: 2560
> rx44_cache_busy: 0
> rx44_cache_waive: 0
> rx44_congst_umr: 0
> rx44_arfs_err: 0
> rx44_xdp_tx_xmit: 0
> rx44_xdp_tx_full: 0
> rx44_xdp_tx_err: 0
> rx44_xdp_tx_cqes: 0
> rx45_packets: 0
> rx45_bytes: 0
> rx45_csum_complete: 0
> rx45_csum_unnecessary: 0
> rx45_csum_unnecessary_inner: 0
> rx45_csum_none: 0
> rx45_xdp_drop: 0
> rx45_xdp_redirect: 0
> rx45_lro_packets: 0
> rx45_lro_bytes: 0
> rx45_ecn_mark: 0
> rx45_removed_vlan_packets: 0
> rx45_wqe_err: 0
> rx45_mpwqe_filler_cqes: 0
> rx45_mpwqe_filler_strides: 0
> rx45_buff_alloc_err: 0
> rx45_cqe_compress_blks: 0
> rx45_cqe_compress_pkts: 0
> rx45_page_reuse: 0
> rx45_cache_reuse: 0
> rx45_cache_full: 0
> rx45_cache_empty: 2560
> rx45_cache_busy: 0
> rx45_cache_waive: 0
> rx45_congst_umr: 0
> rx45_arfs_err: 0
> rx45_xdp_tx_xmit: 0
> rx45_xdp_tx_full: 0
> rx45_xdp_tx_err: 0
> rx45_xdp_tx_cqes: 0
> rx46_packets: 0
> rx46_bytes: 0
> rx46_csum_complete: 0
> rx46_csum_unnecessary: 0
> rx46_csum_unnecessary_inner: 0
> rx46_csum_none: 0
> rx46_xdp_drop: 0
> rx46_xdp_redirect: 0
> rx46_lro_packets: 0
> rx46_lro_bytes: 0
> rx46_ecn_mark: 0
> rx46_removed_vlan_packets: 0
> rx46_wqe_err: 0
> rx46_mpwqe_filler_cqes: 0
> rx46_mpwqe_filler_strides: 0
> rx46_buff_alloc_err: 0
> rx46_cqe_compress_blks: 0
> rx46_cqe_compress_pkts: 0
> rx46_page_reuse: 0
> rx46_cache_reuse: 0
> rx46_cache_full: 0
> rx46_cache_empty: 2560
> rx46_cache_busy: 0
> rx46_cache_waive: 0
> rx46_congst_umr: 0
> rx46_arfs_err: 0
> rx46_xdp_tx_xmit: 0
> rx46_xdp_tx_full: 0
> rx46_xdp_tx_err: 0
> rx46_xdp_tx_cqes: 0
> rx47_packets: 0
> rx47_bytes: 0
> rx47_csum_complete: 0
> rx47_csum_unnecessary: 0
> rx47_csum_unnecessary_inner: 0
> rx47_csum_none: 0
> rx47_xdp_drop: 0
> rx47_xdp_redirect: 0
> rx47_lro_packets: 0
> rx47_lro_bytes: 0
> rx47_ecn_mark: 0
> rx47_removed_vlan_packets: 0
> rx47_wqe_err: 0
> rx47_mpwqe_filler_cqes: 0
> rx47_mpwqe_filler_strides: 0
> rx47_buff_alloc_err: 0
> rx47_cqe_compress_blks: 0
> rx47_cqe_compress_pkts: 0
> rx47_page_reuse: 0
> rx47_cache_reuse: 0
> rx47_cache_full: 0
> rx47_cache_empty: 2560
> rx47_cache_busy: 0
> rx47_cache_waive: 0
> rx47_congst_umr: 0
> rx47_arfs_err: 0
> rx47_xdp_tx_xmit: 0
> rx47_xdp_tx_full: 0
> rx47_xdp_tx_err: 0
> rx47_xdp_tx_cqes: 0
> rx48_packets: 0
> rx48_bytes: 0
> rx48_csum_complete: 0
> rx48_csum_unnecessary: 0
> rx48_csum_unnecessary_inner: 0
> rx48_csum_none: 0
> rx48_xdp_drop: 0
> rx48_xdp_redirect: 0
> rx48_lro_packets: 0
> rx48_lro_bytes: 0
> rx48_ecn_mark: 0
> rx48_removed_vlan_packets: 0
> rx48_wqe_err: 0
> rx48_mpwqe_filler_cqes: 0
> rx48_mpwqe_filler_strides: 0
> rx48_buff_alloc_err: 0
> rx48_cqe_compress_blks: 0
> rx48_cqe_compress_pkts: 0
> rx48_page_reuse: 0
> rx48_cache_reuse: 0
> rx48_cache_full: 0
> rx48_cache_empty: 2560
> rx48_cache_busy: 0
> rx48_cache_waive: 0
> rx48_congst_umr: 0
> rx48_arfs_err: 0
> rx48_xdp_tx_xmit: 0
> rx48_xdp_tx_full: 0
> rx48_xdp_tx_err: 0
> rx48_xdp_tx_cqes: 0
> rx49_packets: 0
> rx49_bytes: 0
> rx49_csum_complete: 0
> rx49_csum_unnecessary: 0
> rx49_csum_unnecessary_inner: 0
> rx49_csum_none: 0
> rx49_xdp_drop: 0
> rx49_xdp_redirect: 0
> rx49_lro_packets: 0
> rx49_lro_bytes: 0
> rx49_ecn_mark: 0
> rx49_removed_vlan_packets: 0
> rx49_wqe_err: 0
> rx49_mpwqe_filler_cqes: 0
> rx49_mpwqe_filler_strides: 0
> rx49_buff_alloc_err: 0
> rx49_cqe_compress_blks: 0
> rx49_cqe_compress_pkts: 0
> rx49_page_reuse: 0
> rx49_cache_reuse: 0
> rx49_cache_full: 0
> rx49_cache_empty: 2560
> rx49_cache_busy: 0
> rx49_cache_waive: 0
> rx49_congst_umr: 0
> rx49_arfs_err: 0
> rx49_xdp_tx_xmit: 0
> rx49_xdp_tx_full: 0
> rx49_xdp_tx_err: 0
> rx49_xdp_tx_cqes: 0
> rx50_packets: 0
> rx50_bytes: 0
> rx50_csum_complete: 0
> rx50_csum_unnecessary: 0
> rx50_csum_unnecessary_inner: 0
> rx50_csum_none: 0
> rx50_xdp_drop: 0
> rx50_xdp_redirect: 0
> rx50_lro_packets: 0
> rx50_lro_bytes: 0
> rx50_ecn_mark: 0
> rx50_removed_vlan_packets: 0
> rx50_wqe_err: 0
> rx50_mpwqe_filler_cqes: 0
> rx50_mpwqe_filler_strides: 0
> rx50_buff_alloc_err: 0
> rx50_cqe_compress_blks: 0
> rx50_cqe_compress_pkts: 0
> rx50_page_reuse: 0
> rx50_cache_reuse: 0
> rx50_cache_full: 0
> rx50_cache_empty: 2560
> rx50_cache_busy: 0
> rx50_cache_waive: 0
> rx50_congst_umr: 0
> rx50_arfs_err: 0
> rx50_xdp_tx_xmit: 0
> rx50_xdp_tx_full: 0
> rx50_xdp_tx_err: 0
> rx50_xdp_tx_cqes: 0
> rx51_packets: 0
> rx51_bytes: 0
> rx51_csum_complete: 0
> rx51_csum_unnecessary: 0
> rx51_csum_unnecessary_inner: 0
> rx51_csum_none: 0
> rx51_xdp_drop: 0
> rx51_xdp_redirect: 0
> rx51_lro_packets: 0
> rx51_lro_bytes: 0
> rx51_ecn_mark: 0
> rx51_removed_vlan_packets: 0
> rx51_wqe_err: 0
> rx51_mpwqe_filler_cqes: 0
> rx51_mpwqe_filler_strides: 0
> rx51_buff_alloc_err: 0
> rx51_cqe_compress_blks: 0
> rx51_cqe_compress_pkts: 0
> rx51_page_reuse: 0
> rx51_cache_reuse: 0
> rx51_cache_full: 0
> rx51_cache_empty: 2560
> rx51_cache_busy: 0
> rx51_cache_waive: 0
> rx51_congst_umr: 0
> rx51_arfs_err: 0
> rx51_xdp_tx_xmit: 0
> rx51_xdp_tx_full: 0
> rx51_xdp_tx_err: 0
> rx51_xdp_tx_cqes: 0
> rx52_packets: 0
> rx52_bytes: 0
> rx52_csum_complete: 0
> rx52_csum_unnecessary: 0
> rx52_csum_unnecessary_inner: 0
> rx52_csum_none: 0
> rx52_xdp_drop: 0
> rx52_xdp_redirect: 0
> rx52_lro_packets: 0
> rx52_lro_bytes: 0
> rx52_ecn_mark: 0
> rx52_removed_vlan_packets: 0
> rx52_wqe_err: 0
> rx52_mpwqe_filler_cqes: 0
> rx52_mpwqe_filler_strides: 0
> rx52_buff_alloc_err: 0
> rx52_cqe_compress_blks: 0
> rx52_cqe_compress_pkts: 0
> rx52_page_reuse: 0
> rx52_cache_reuse: 0
> rx52_cache_full: 0
> rx52_cache_empty: 2560
> rx52_cache_busy: 0
> rx52_cache_waive: 0
> rx52_congst_umr: 0
> rx52_arfs_err: 0
> rx52_xdp_tx_xmit: 0
> rx52_xdp_tx_full: 0
> rx52_xdp_tx_err: 0
> rx52_xdp_tx_cqes: 0
> rx53_packets: 0
> rx53_bytes: 0
> rx53_csum_complete: 0
> rx53_csum_unnecessary: 0
> rx53_csum_unnecessary_inner: 0
> rx53_csum_none: 0
> rx53_xdp_drop: 0
> rx53_xdp_redirect: 0
> rx53_lro_packets: 0
> rx53_lro_bytes: 0
> rx53_ecn_mark: 0
> rx53_removed_vlan_packets: 0
> rx53_wqe_err: 0
> rx53_mpwqe_filler_cqes: 0
> rx53_mpwqe_filler_strides: 0
> rx53_buff_alloc_err: 0
> rx53_cqe_compress_blks: 0
> rx53_cqe_compress_pkts: 0
> rx53_page_reuse: 0
> rx53_cache_reuse: 0
> rx53_cache_full: 0
> rx53_cache_empty: 2560
> rx53_cache_busy: 0
> rx53_cache_waive: 0
> rx53_congst_umr: 0
> rx53_arfs_err: 0
> rx53_xdp_tx_xmit: 0
> rx53_xdp_tx_full: 0
> rx53_xdp_tx_err: 0
> rx53_xdp_tx_cqes: 0
> rx54_packets: 0
> rx54_bytes: 0
> rx54_csum_complete: 0
> rx54_csum_unnecessary: 0
> rx54_csum_unnecessary_inner: 0
> rx54_csum_none: 0
> rx54_xdp_drop: 0
> rx54_xdp_redirect: 0
> rx54_lro_packets: 0
> rx54_lro_bytes: 0
> rx54_ecn_mark: 0
> rx54_removed_vlan_packets: 0
> rx54_wqe_err: 0
> rx54_mpwqe_filler_cqes: 0
> rx54_mpwqe_filler_strides: 0
> rx54_buff_alloc_err: 0
> rx54_cqe_compress_blks: 0
> rx54_cqe_compress_pkts: 0
> rx54_page_reuse: 0
> rx54_cache_reuse: 0
> rx54_cache_full: 0
> rx54_cache_empty: 2560
> rx54_cache_busy: 0
> rx54_cache_waive: 0
> rx54_congst_umr: 0
> rx54_arfs_err: 0
> rx54_xdp_tx_xmit: 0
> rx54_xdp_tx_full: 0
> rx54_xdp_tx_err: 0
> rx54_xdp_tx_cqes: 0
> rx55_packets: 0
> rx55_bytes: 0
> rx55_csum_complete: 0
> rx55_csum_unnecessary: 0
> rx55_csum_unnecessary_inner: 0
> rx55_csum_none: 0
> rx55_xdp_drop: 0
> rx55_xdp_redirect: 0
> rx55_lro_packets: 0
> rx55_lro_bytes: 0
> rx55_ecn_mark: 0
> rx55_removed_vlan_packets: 0
> rx55_wqe_err: 0
> rx55_mpwqe_filler_cqes: 0
> rx55_mpwqe_filler_strides: 0
> rx55_buff_alloc_err: 0
> rx55_cqe_compress_blks: 0
> rx55_cqe_compress_pkts: 0
> rx55_page_reuse: 0
> rx55_cache_reuse: 0
> rx55_cache_full: 0
> rx55_cache_empty: 2560
> rx55_cache_busy: 0
> rx55_cache_waive: 0
> rx55_congst_umr: 0
> rx55_arfs_err: 0
> rx55_xdp_tx_xmit: 0
> rx55_xdp_tx_full: 0
> rx55_xdp_tx_err: 0
> rx55_xdp_tx_cqes: 0
> tx0_packets: 5868971166
> tx0_bytes: 7384241881537
> tx0_tso_packets: 1005089669
> tx0_tso_bytes: 5138882499687
> tx0_tso_inner_packets: 0
> tx0_tso_inner_bytes: 0
> tx0_csum_partial: 1405330470
> tx0_csum_partial_inner: 0
> tx0_added_vlan_packets: 3247061022
> tx0_nop: 83925216
> tx0_csum_none: 1841730552
> tx0_stopped: 0
> tx0_dropped: 0
> tx0_xmit_more: 29664303
> tx0_recover: 0
> tx0_cqes: 3217398842
> tx0_wake: 0
> tx0_cqe_err: 0
> tx1_packets: 5599378674
> tx1_bytes: 7272236466962
> tx1_tso_packets: 1024612268
> tx1_tso_bytes: 5244192050917
> tx1_tso_inner_packets: 0
> tx1_tso_inner_bytes: 0
> tx1_csum_partial: 1438007932
> tx1_csum_partial_inner: 0
> tx1_added_vlan_packets: 2919765857
> tx1_nop: 79661231
> tx1_csum_none: 1481757925
> tx1_stopped: 0
> tx1_dropped: 0
> tx1_xmit_more: 29485355
> tx1_recover: 0
> tx1_cqes: 2890282176
> tx1_wake: 0
> tx1_cqe_err: 0
> tx2_packets: 5413821094
> tx2_bytes: 7033951631334
> tx2_tso_packets: 1002868589
> tx2_tso_bytes: 5089549008985
> tx2_tso_inner_packets: 0
> tx2_tso_inner_bytes: 0
> tx2_csum_partial: 1404186175
> tx2_csum_partial_inner: 0
> tx2_added_vlan_packets: 2822670460
> tx2_nop: 77115408
> tx2_csum_none: 1418484285
> tx2_stopped: 0
> tx2_dropped: 0
> tx2_xmit_more: 29321129
> tx2_recover: 0
> tx2_cqes: 2793351019
> tx2_wake: 0
> tx2_cqe_err: 0
> tx3_packets: 5479609727
> tx3_bytes: 7116904107659
> tx3_tso_packets: 1002992639
> tx3_tso_bytes: 5154225081979
> tx3_tso_inner_packets: 0
> tx3_tso_inner_bytes: 0
> tx3_csum_partial: 1415739849
> tx3_csum_partial_inner: 0
> tx3_added_vlan_packets: 2842823811
> tx3_nop: 78060813
> tx3_csum_none: 1427083971
> tx3_stopped: 0
> tx3_dropped: 0
> tx3_xmit_more: 28575040
> tx3_recover: 0
> tx3_cqes: 2814250785
> tx3_wake: 0
> tx3_cqe_err: 0
> tx4_packets: 5508297397
> tx4_bytes: 7127659369902
> tx4_tso_packets: 1007356432
> tx4_tso_bytes: 5145975736034
> tx4_tso_inner_packets: 0
> tx4_tso_inner_bytes: 0
> tx4_csum_partial: 1411271000
> tx4_csum_partial_inner: 0
> tx4_added_vlan_packets: 2882086825
> tx4_nop: 78433610
> tx4_csum_none: 1470815825
> tx4_stopped: 0
> tx4_dropped: 0
> tx4_xmit_more: 28632444
> tx4_recover: 0
> tx4_cqes: 2853456464
> tx4_wake: 0
> tx4_cqe_err: 0
> tx5_packets: 5513864156
> tx5_bytes: 7165864145517
> tx5_tso_packets: 1014046485
> tx5_tso_bytes: 5192635614477
> tx5_tso_inner_packets: 0
> tx5_tso_inner_bytes: 0
> tx5_csum_partial: 1420810473
> tx5_csum_partial_inner: 0
> tx5_added_vlan_packets: 2861370556
> tx5_nop: 78481355
> tx5_csum_none: 1440560083
> tx5_stopped: 0
> tx5_dropped: 0
> tx5_xmit_more: 28222467
> tx5_recover: 0
> tx5_cqes: 2833149758
> tx5_wake: 0
> tx5_cqe_err: 0
> tx6_packets: 5560724761
> tx6_bytes: 7210309972086
> tx6_tso_packets: 994050514
> tx6_tso_bytes: 5171393741595
> tx6_tso_inner_packets: 0
> tx6_tso_inner_bytes: 0
> tx6_csum_partial: 1414303265
> tx6_csum_partial_inner: 0
> tx6_added_vlan_packets: 2905794177
> tx6_nop: 79353318
> tx6_csum_none: 1491490912
> tx6_stopped: 0
> tx6_dropped: 0
> tx6_xmit_more: 31246664
> tx6_recover: 0
> tx6_cqes: 2874549217
> tx6_wake: 0
> tx6_cqe_err: 0
> tx7_packets: 5557594170
> tx7_bytes: 7223138778685
> tx7_tso_packets: 1013475396
> tx7_tso_bytes: 5241530065484
> tx7_tso_inner_packets: 0
> tx7_tso_inner_bytes: 0
> tx7_csum_partial: 1438604314
> tx7_csum_partial_inner: 0
> tx7_added_vlan_packets: 2873917552
> tx7_nop: 79057059
> tx7_csum_none: 1435313239
> tx7_stopped: 0
> tx7_dropped: 0
> tx7_xmit_more: 29258761
> tx7_recover: 0
> tx7_cqes: 2844660578
> tx7_wake: 0
> tx7_cqe_err: 0
> tx8_packets: 5521254733
> tx8_bytes: 7208043146297
> tx8_tso_packets: 1014670801
> tx8_tso_bytes: 5185842447246
> tx8_tso_inner_packets: 0
> tx8_tso_inner_bytes: 0
> tx8_csum_partial: 1431631562
> tx8_csum_partial_inner: 0
> tx8_added_vlan_packets: 2872641129
> tx8_nop: 78545776
> tx8_csum_none: 1441009567
> tx8_stopped: 0
> tx8_dropped: 0
> tx8_xmit_more: 29106291
> tx8_recover: 0
> tx8_cqes: 2843536748
> tx8_wake: 0
> tx8_cqe_err: 0
> tx9_packets: 5528889957
> tx9_bytes: 7191793816058
> tx9_tso_packets: 1015955476
> tx9_tso_bytes: 5207232047828
> tx9_tso_inner_packets: 0
> tx9_tso_inner_bytes: 0
> tx9_csum_partial: 1421266796
> tx9_csum_partial_inner: 0
> tx9_added_vlan_packets: 2869523921
> tx9_nop: 78586218
> tx9_csum_none: 1448257125
> tx9_stopped: 0
> tx9_dropped: 0
> tx9_xmit_more: 29483347
> tx9_recover: 0
> tx9_cqes: 2840042245
> tx9_wake: 0
> tx9_cqe_err: 0
> tx10_packets: 5556351222
> tx10_bytes: 7254798330757
> tx10_tso_packets: 1028554460
> tx10_tso_bytes: 5246179615774
> tx10_tso_inner_packets: 0
> tx10_tso_inner_bytes: 0
> tx10_csum_partial: 1430459021
> tx10_csum_partial_inner: 0
> tx10_added_vlan_packets: 2881683382
> tx10_nop: 79139584
> tx10_csum_none: 1451224361
> tx10_stopped: 0
> tx10_dropped: 0
> tx10_xmit_more: 29217190
> tx10_recover: 0
> tx10_cqes: 2852467898
> tx10_wake: 0
> tx10_cqe_err: 0
> tx11_packets: 5455631854
> tx11_bytes: 7061121713772
> tx11_tso_packets: 992133383
> tx11_tso_bytes: 5089419722682
> tx11_tso_inner_packets: 0
> tx11_tso_inner_bytes: 0
> tx11_csum_partial: 1395542033
> tx11_csum_partial_inner: 0
> tx11_added_vlan_packets: 2852589093
> tx11_nop: 77799857
> tx11_csum_none: 1457047060
> tx11_stopped: 0
> tx11_dropped: 0
> tx11_xmit_more: 29559927
> tx11_recover: 0
> tx11_cqes: 2823031110
> tx11_wake: 0
> tx11_cqe_err: 0
> tx12_packets: 5488286808
> tx12_bytes: 7137087569303
> tx12_tso_packets: 1006435537
> tx12_tso_bytes: 5163371416750
> tx12_tso_inner_packets: 0
> tx12_tso_inner_bytes: 0
> tx12_csum_partial: 1414799411
> tx12_csum_partial_inner: 0
> tx12_added_vlan_packets: 2841679543
> tx12_nop: 78387039
> tx12_csum_none: 1426880132
> tx12_stopped: 0
> tx12_dropped: 0
> tx12_xmit_more: 28607526
> tx12_recover: 0
> tx12_cqes: 2813073557
> tx12_wake: 0
> tx12_cqe_err: 0
> tx13_packets: 5594132290
> tx13_bytes: 7251106284829
> tx13_tso_packets: 1035172061
> tx13_tso_bytes: 5251200286298
> tx13_tso_inner_packets: 0
> tx13_tso_inner_bytes: 0
> tx13_csum_partial: 1443665981
> tx13_csum_partial_inner: 0
> tx13_added_vlan_packets: 2916604799
> tx13_nop: 79670465
> tx13_csum_none: 1472938818
> tx13_stopped: 0
> tx13_dropped: 0
> tx13_xmit_more: 27797067
> tx13_recover: 0
> tx13_cqes: 2888809352
> tx13_wake: 0
> tx13_cqe_err: 0
> tx14_packets: 5548790952
> tx14_bytes: 7194211868411
> tx14_tso_packets: 1021015561
> tx14_tso_bytes: 5231483708869
> tx14_tso_inner_packets: 0
> tx14_tso_inner_bytes: 0
> tx14_csum_partial: 1427711576
> tx14_csum_partial_inner: 0
> tx14_added_vlan_packets: 2875288572
> tx14_nop: 78900224
> tx14_csum_none: 1447576996
> tx14_stopped: 0
> tx14_dropped: 0
> tx14_xmit_more: 30003496
> tx14_recover: 0
> tx14_cqes: 2845286732
> tx14_wake: 0
> tx14_cqe_err: 0
> tx15_packets: 5609310963
> tx15_bytes: 7271380831798
> tx15_tso_packets: 1027830118
> tx15_tso_bytes: 5229697431506
> tx15_tso_inner_packets: 0
> tx15_tso_inner_bytes: 0
> tx15_csum_partial: 1429209941
> tx15_csum_partial_inner: 0
> tx15_added_vlan_packets: 2940315402
> tx15_nop: 79950883
> tx15_csum_none: 1511105462
> tx15_stopped: 0
> tx15_dropped: 0
> tx15_xmit_more: 28820740
> tx15_recover: 0
> tx15_cqes: 2911496633
> tx15_wake: 0
> tx15_cqe_err: 0
> tx16_packets: 4465363036
> tx16_bytes: 5769771803704
> tx16_tso_packets: 817101913
> tx16_tso_bytes: 4180172833814
> tx16_tso_inner_packets: 0
> tx16_tso_inner_bytes: 0
> tx16_csum_partial: 1136731404
> tx16_csum_partial_inner: 0
> tx16_added_vlan_packets: 2332178232
> tx16_nop: 63458573
> tx16_csum_none: 1195446828
> tx16_stopped: 0
> tx16_dropped: 0
> tx16_xmit_more: 23756254
> tx16_recover: 0
> tx16_cqes: 2308423025
> tx16_wake: 0
> tx16_cqe_err: 0
> tx17_packets: 4380386348
> tx17_bytes: 5708702994526
> tx17_tso_packets: 813638023
> tx17_tso_bytes: 4130806014947
> tx17_tso_inner_packets: 0
> tx17_tso_inner_bytes: 0
> tx17_csum_partial: 1133007164
> tx17_csum_partial_inner: 0
> tx17_added_vlan_packets: 2277314787
> tx17_nop: 62377372
> tx17_csum_none: 1144307623
> tx17_stopped: 0
> tx17_dropped: 0
> tx17_xmit_more: 23731361
> tx17_recover: 0
> tx17_cqes: 2253584638
> tx17_wake: 0
> tx17_cqe_err: 0
> tx18_packets: 4450359743
> tx18_bytes: 5758968674820
> tx18_tso_packets: 815791601
> tx18_tso_bytes: 4179942688909
> tx18_tso_inner_packets: 0
> tx18_tso_inner_bytes: 0
> tx18_csum_partial: 1137649257
> tx18_csum_partial_inner: 0
> tx18_added_vlan_packets: 2314556550
> tx18_nop: 63271085
> tx18_csum_none: 1176907293
> tx18_stopped: 0
> tx18_dropped: 0
> tx18_xmit_more: 23055770
> tx18_recover: 0
> tx18_cqes: 2291501928
> tx18_wake: 0
> tx18_cqe_err: 0
> tx19_packets: 4596064378
> tx19_bytes: 5916675706535
> tx19_tso_packets: 825788649
> tx19_tso_bytes: 4208046929921
> tx19_tso_inner_packets: 0
> tx19_tso_inner_bytes: 0
> tx19_csum_partial: 1150666569
> tx19_csum_partial_inner: 0
> tx19_added_vlan_packets: 2450567026
> tx19_nop: 65468504
> tx19_csum_none: 1299900457
> tx19_stopped: 0
> tx19_dropped: 0
> tx19_xmit_more: 23846250
> tx19_recover: 0
> tx19_cqes: 2426722127
> tx19_wake: 0
> tx19_cqe_err: 0
> tx20_packets: 4424935388
> tx20_bytes: 5757631205901
> tx20_tso_packets: 804875006
> tx20_tso_bytes: 4156262736109
> tx20_tso_inner_packets: 0
> tx20_tso_inner_bytes: 0
> tx20_csum_partial: 1134144916
> tx20_csum_partial_inner: 0
> tx20_added_vlan_packets: 2294839665
> tx20_nop: 63023986
> tx20_csum_none: 1160694749
> tx20_stopped: 0
> tx20_dropped: 0
> tx20_xmit_more: 23393201
> tx20_recover: 0
> tx20_cqes: 2271447623
> tx20_wake: 0
> tx20_cqe_err: 0
> tx21_packets: 4595062285
> tx21_bytes: 5958671993467
> tx21_tso_packets: 821936215
> tx21_tso_bytes: 4187977870684
> tx21_tso_inner_packets: 0
> tx21_tso_inner_bytes: 0
> tx21_csum_partial: 1143339787
> tx21_csum_partial_inner: 0
> tx21_added_vlan_packets: 2457167412
> tx21_nop: 65697763
> tx21_csum_none: 1313827625
> tx21_stopped: 0
> tx21_dropped: 0
> tx21_xmit_more: 23858345
> tx21_recover: 0
> tx21_cqes: 2433310348
> tx21_wake: 0
> tx21_cqe_err: 0
> tx22_packets: 4664446513
> tx22_bytes: 5931429292082
> tx22_tso_packets: 814457881
> tx22_tso_bytes: 4148607956533
> tx22_tso_inner_packets: 0
> tx22_tso_inner_bytes: 0
> tx22_csum_partial: 1127284783
> tx22_csum_partial_inner: 0
> tx22_added_vlan_packets: 2548650146
> tx22_nop: 66299909
> tx22_csum_none: 1421365363
> tx22_stopped: 0
> tx22_dropped: 0
> tx22_xmit_more: 23800911
> tx22_recover: 0
> tx22_cqes: 2524850415
> tx22_wake: 0
> tx22_cqe_err: 0
> tx23_packets: 4416221747
> tx23_bytes: 5721472587985
> tx23_tso_packets: 823538520
> tx23_tso_bytes: 4163520218617
> tx23_tso_inner_packets: 0
> tx23_tso_inner_bytes: 0
> tx23_csum_partial: 1135996006
> tx23_csum_partial_inner: 0
> tx23_added_vlan_packets: 2292404120
> tx23_nop: 62709432
> tx23_csum_none: 1156408114
> tx23_stopped: 0
> tx23_dropped: 0
> tx23_xmit_more: 22299889
> tx23_recover: 0
> tx23_cqes: 2270105487
> tx23_wake: 0
> tx23_cqe_err: 0
> tx24_packets: 4420014824
> tx24_bytes: 5740767318521
> tx24_tso_packets: 820838072
> tx24_tso_bytes: 4183722948422
> tx24_tso_inner_packets: 0
> tx24_tso_inner_bytes: 0
> tx24_csum_partial: 1138070059
> tx24_csum_partial_inner: 0
> tx24_added_vlan_packets: 2289043946
> tx24_nop: 62797341
> tx24_csum_none: 1150973887
> tx24_stopped: 0
> tx24_dropped: 0
> tx24_xmit_more: 22744690
> tx24_recover: 0
> tx24_cqes: 2266300568
> tx24_wake: 0
> tx24_cqe_err: 0
> tx25_packets: 4413225545
> tx25_bytes: 5716162617155
> tx25_tso_packets: 808274341
> tx25_tso_bytes: 4138408857714
> tx25_tso_inner_packets: 0
> tx25_tso_inner_bytes: 0
> tx25_csum_partial: 1134587898
> tx25_csum_partial_inner: 0
> tx25_added_vlan_packets: 2297149310
> tx25_nop: 62958238
> tx25_csum_none: 1162561412
> tx25_stopped: 0
> tx25_dropped: 0
> tx25_xmit_more: 24463552
> tx25_recover: 0
> tx25_cqes: 2272686971
> tx25_wake: 0
> tx25_cqe_err: 0
> tx26_packets: 4524907591
> tx26_bytes: 5865394280699
> tx26_tso_packets: 807270022
> tx26_tso_bytes: 4148754705317
> tx26_tso_inner_packets: 0
> tx26_tso_inner_bytes: 0
> tx26_csum_partial: 1130306933
> tx26_csum_partial_inner: 0
> tx26_added_vlan_packets: 2402682460
> tx26_nop: 64474322
> tx26_csum_none: 1272375527
> tx26_stopped: 1
> tx26_dropped: 0
> tx26_xmit_more: 23316186
> tx26_recover: 0
> tx26_cqes: 2379367502
> tx26_wake: 1
> tx26_cqe_err: 0
> tx27_packets: 4376114969
> tx27_bytes: 5683551238304
> tx27_tso_packets: 809344829
> tx27_tso_bytes: 4124331859270
> tx27_tso_inner_packets: 0
> tx27_tso_inner_bytes: 0
> tx27_csum_partial: 1124954937
> tx27_csum_partial_inner: 0
> tx27_added_vlan_packets: 2267871300
> tx27_nop: 62213214
> tx27_csum_none: 1142916363
> tx27_stopped: 0
> tx27_dropped: 0
> tx27_xmit_more: 23369974
> tx27_recover: 0
> tx27_cqes: 2244502686
> tx27_wake: 0
> tx27_cqe_err: 0
> tx28_packets: 3
> tx28_bytes: 266
> tx28_tso_packets: 0
> tx28_tso_bytes: 0
> tx28_tso_inner_packets: 0
> tx28_tso_inner_bytes: 0
> tx28_csum_partial: 0
> tx28_csum_partial_inner: 0
> tx28_added_vlan_packets: 0
> tx28_nop: 0
> tx28_csum_none: 3
> tx28_stopped: 0
> tx28_dropped: 0
> tx28_xmit_more: 0
> tx28_recover: 0
> tx28_cqes: 3
> tx28_wake: 0
> tx28_cqe_err: 0
> tx29_packets: 0
> tx29_bytes: 0
> tx29_tso_packets: 0
> tx29_tso_bytes: 0
> tx29_tso_inner_packets: 0
> tx29_tso_inner_bytes: 0
> tx29_csum_partial: 0
> tx29_csum_partial_inner: 0
> tx29_added_vlan_packets: 0
> tx29_nop: 0
> tx29_csum_none: 0
> tx29_stopped: 0
> tx29_dropped: 0
> tx29_xmit_more: 0
> tx29_recover: 0
> tx29_cqes: 0
> tx29_wake: 0
> tx29_cqe_err: 0
> tx30_packets: 0
> tx30_bytes: 0
> tx30_tso_packets: 0
> tx30_tso_bytes: 0
> tx30_tso_inner_packets: 0
> tx30_tso_inner_bytes: 0
> tx30_csum_partial: 0
> tx30_csum_partial_inner: 0
> tx30_added_vlan_packets: 0
> tx30_nop: 0
> tx30_csum_none: 0
> tx30_stopped: 0
> tx30_dropped: 0
> tx30_xmit_more: 0
> tx30_recover: 0
> tx30_cqes: 0
> tx30_wake: 0
> tx30_cqe_err: 0
> tx31_packets: 0
> tx31_bytes: 0
> tx31_tso_packets: 0
> tx31_tso_bytes: 0
> tx31_tso_inner_packets: 0
> tx31_tso_inner_bytes: 0
> tx31_csum_partial: 0
> tx31_csum_partial_inner: 0
> tx31_added_vlan_packets: 0
> tx31_nop: 0
> tx31_csum_none: 0
> tx31_stopped: 0
> tx31_dropped: 0
> tx31_xmit_more: 0
> tx31_recover: 0
> tx31_cqes: 0
> tx31_wake: 0
> tx31_cqe_err: 0
> tx32_packets: 0
> tx32_bytes: 0
> tx32_tso_packets: 0
> tx32_tso_bytes: 0
> tx32_tso_inner_packets: 0
> tx32_tso_inner_bytes: 0
> tx32_csum_partial: 0
> tx32_csum_partial_inner: 0
> tx32_added_vlan_packets: 0
> tx32_nop: 0
> tx32_csum_none: 0
> tx32_stopped: 0
> tx32_dropped: 0
> tx32_xmit_more: 0
> tx32_recover: 0
> tx32_cqes: 0
> tx32_wake: 0
> tx32_cqe_err: 0
> tx33_packets: 0
> tx33_bytes: 0
> tx33_tso_packets: 0
> tx33_tso_bytes: 0
> tx33_tso_inner_packets: 0
> tx33_tso_inner_bytes: 0
> tx33_csum_partial: 0
> tx33_csum_partial_inner: 0
> tx33_added_vlan_packets: 0
> tx33_nop: 0
> tx33_csum_none: 0
> tx33_stopped: 0
> tx33_dropped: 0
> tx33_xmit_more: 0
> tx33_recover: 0
> tx33_cqes: 0
> tx33_wake: 0
> tx33_cqe_err: 0
> tx34_packets: 0
> tx34_bytes: 0
> tx34_tso_packets: 0
> tx34_tso_bytes: 0
> tx34_tso_inner_packets: 0
> tx34_tso_inner_bytes: 0
> tx34_csum_partial: 0
> tx34_csum_partial_inner: 0
> tx34_added_vlan_packets: 0
> tx34_nop: 0
> tx34_csum_none: 0
> tx34_stopped: 0
> tx34_dropped: 0
> tx34_xmit_more: 0
> tx34_recover: 0
> tx34_cqes: 0
> tx34_wake: 0
> tx34_cqe_err: 0
> tx35_packets: 0
> tx35_bytes: 0
> tx35_tso_packets: 0
> tx35_tso_bytes: 0
> tx35_tso_inner_packets: 0
> tx35_tso_inner_bytes: 0
> tx35_csum_partial: 0
> tx35_csum_partial_inner: 0
> tx35_added_vlan_packets: 0
> tx35_nop: 0
> tx35_csum_none: 0
> tx35_stopped: 0
> tx35_dropped: 0
> tx35_xmit_more: 0
> tx35_recover: 0
> tx35_cqes: 0
> tx35_wake: 0
> tx35_cqe_err: 0
> tx36_packets: 0
> tx36_bytes: 0
> tx36_tso_packets: 0
> tx36_tso_bytes: 0
> tx36_tso_inner_packets: 0
> tx36_tso_inner_bytes: 0
> tx36_csum_partial: 0
> tx36_csum_partial_inner: 0
> tx36_added_vlan_packets: 0
> tx36_nop: 0
> tx36_csum_none: 0
> tx36_stopped: 0
> tx36_dropped: 0
> tx36_xmit_more: 0
> tx36_recover: 0
> tx36_cqes: 0
> tx36_wake: 0
> tx36_cqe_err: 0
> tx37_packets: 0
> tx37_bytes: 0
> tx37_tso_packets: 0
> tx37_tso_bytes: 0
> tx37_tso_inner_packets: 0
> tx37_tso_inner_bytes: 0
> tx37_csum_partial: 0
> tx37_csum_partial_inner: 0
> tx37_added_vlan_packets: 0
> tx37_nop: 0
> tx37_csum_none: 0
> tx37_stopped: 0
> tx37_dropped: 0
> tx37_xmit_more: 0
> tx37_recover: 0
> tx37_cqes: 0
> tx37_wake: 0
> tx37_cqe_err: 0
> tx38_packets: 0
> tx38_bytes: 0
> tx38_tso_packets: 0
> tx38_tso_bytes: 0
> tx38_tso_inner_packets: 0
> tx38_tso_inner_bytes: 0
> tx38_csum_partial: 0
> tx38_csum_partial_inner: 0
> tx38_added_vlan_packets: 0
> tx38_nop: 0
> tx38_csum_none: 0
> tx38_stopped: 0
> tx38_dropped: 0
> tx38_xmit_more: 0
> tx38_recover: 0
> tx38_cqes: 0
> tx38_wake: 0
> tx38_cqe_err: 0
> tx39_packets: 0
> tx39_bytes: 0
> tx39_tso_packets: 0
> tx39_tso_bytes: 0
> tx39_tso_inner_packets: 0
> tx39_tso_inner_bytes: 0
> tx39_csum_partial: 0
> tx39_csum_partial_inner: 0
> tx39_added_vlan_packets: 0
> tx39_nop: 0
> tx39_csum_none: 0
> tx39_stopped: 0
> tx39_dropped: 0
> tx39_xmit_more: 0
> tx39_recover: 0
> tx39_cqes: 0
> tx39_wake: 0
> tx39_cqe_err: 0
> tx40_packets: 0
> tx40_bytes: 0
> tx40_tso_packets: 0
> tx40_tso_bytes: 0
> tx40_tso_inner_packets: 0
> tx40_tso_inner_bytes: 0
> tx40_csum_partial: 0
> tx40_csum_partial_inner: 0
> tx40_added_vlan_packets: 0
> tx40_nop: 0
> tx40_csum_none: 0
> tx40_stopped: 0
> tx40_dropped: 0
> tx40_xmit_more: 0
> tx40_recover: 0
> tx40_cqes: 0
> tx40_wake: 0
> tx40_cqe_err: 0
> tx41_packets: 0
> tx41_bytes: 0
> tx41_tso_packets: 0
> tx41_tso_bytes: 0
> tx41_tso_inner_packets: 0
> tx41_tso_inner_bytes: 0
> tx41_csum_partial: 0
> tx41_csum_partial_inner: 0
> tx41_added_vlan_packets: 0
> tx41_nop: 0
> tx41_csum_none: 0
> tx41_stopped: 0
> tx41_dropped: 0
> tx41_xmit_more: 0
> tx41_recover: 0
> tx41_cqes: 0
> tx41_wake: 0
> tx41_cqe_err: 0
> tx42_packets: 0
> tx42_bytes: 0
> tx42_tso_packets: 0
> tx42_tso_bytes: 0
> tx42_tso_inner_packets: 0
> tx42_tso_inner_bytes: 0
> tx42_csum_partial: 0
> tx42_csum_partial_inner: 0
> tx42_added_vlan_packets: 0
> tx42_nop: 0
> tx42_csum_none: 0
> tx42_stopped: 0
> tx42_dropped: 0
> tx42_xmit_more: 0
> tx42_recover: 0
> tx42_cqes: 0
> tx42_wake: 0
> tx42_cqe_err: 0
> tx43_packets: 0
> tx43_bytes: 0
> tx43_tso_packets: 0
> tx43_tso_bytes: 0
> tx43_tso_inner_packets: 0
> tx43_tso_inner_bytes: 0
> tx43_csum_partial: 0
> tx43_csum_partial_inner: 0
> tx43_added_vlan_packets: 0
> tx43_nop: 0
> tx43_csum_none: 0
> tx43_stopped: 0
> tx43_dropped: 0
> tx43_xmit_more: 0
> tx43_recover: 0
> tx43_cqes: 0
> tx43_wake: 0
> tx43_cqe_err: 0
> tx44_packets: 0
> tx44_bytes: 0
> tx44_tso_packets: 0
> tx44_tso_bytes: 0
> tx44_tso_inner_packets: 0
> tx44_tso_inner_bytes: 0
> tx44_csum_partial: 0
> tx44_csum_partial_inner: 0
> tx44_added_vlan_packets: 0
> tx44_nop: 0
> tx44_csum_none: 0
> tx44_stopped: 0
> tx44_dropped: 0
> tx44_xmit_more: 0
> tx44_recover: 0
> tx44_cqes: 0
> tx44_wake: 0
> tx44_cqe_err: 0
> tx45_packets: 0
> tx45_bytes: 0
> tx45_tso_packets: 0
> tx45_tso_bytes: 0
> tx45_tso_inner_packets: 0
> tx45_tso_inner_bytes: 0
> tx45_csum_partial: 0
> tx45_csum_partial_inner: 0
> tx45_added_vlan_packets: 0
> tx45_nop: 0
> tx45_csum_none: 0
> tx45_stopped: 0
> tx45_dropped: 0
> tx45_xmit_more: 0
> tx45_recover: 0
> tx45_cqes: 0
> tx45_wake: 0
> tx45_cqe_err: 0
> tx46_packets: 0
> tx46_bytes: 0
> tx46_tso_packets: 0
> tx46_tso_bytes: 0
> tx46_tso_inner_packets: 0
> tx46_tso_inner_bytes: 0
> tx46_csum_partial: 0
> tx46_csum_partial_inner: 0
> tx46_added_vlan_packets: 0
> tx46_nop: 0
> tx46_csum_none: 0
> tx46_stopped: 0
> tx46_dropped: 0
> tx46_xmit_more: 0
> tx46_recover: 0
> tx46_cqes: 0
> tx46_wake: 0
> tx46_cqe_err: 0
> tx47_packets: 0
> tx47_bytes: 0
> tx47_tso_packets: 0
> tx47_tso_bytes: 0
> tx47_tso_inner_packets: 0
> tx47_tso_inner_bytes: 0
> tx47_csum_partial: 0
> tx47_csum_partial_inner: 0
> tx47_added_vlan_packets: 0
> tx47_nop: 0
> tx47_csum_none: 0
> tx47_stopped: 0
> tx47_dropped: 0
> tx47_xmit_more: 0
> tx47_recover: 0
> tx47_cqes: 0
> tx47_wake: 0
> tx47_cqe_err: 0
> tx48_packets: 0
> tx48_bytes: 0
> tx48_tso_packets: 0
> tx48_tso_bytes: 0
> tx48_tso_inner_packets: 0
> tx48_tso_inner_bytes: 0
> tx48_csum_partial: 0
> tx48_csum_partial_inner: 0
> tx48_added_vlan_packets: 0
> tx48_nop: 0
> tx48_csum_none: 0
> tx48_stopped: 0
> tx48_dropped: 0
> tx48_xmit_more: 0
> tx48_recover: 0
> tx48_cqes: 0
> tx48_wake: 0
> tx48_cqe_err: 0
> tx49_packets: 0
> tx49_bytes: 0
> tx49_tso_packets: 0
> tx49_tso_bytes: 0
> tx49_tso_inner_packets: 0
> tx49_tso_inner_bytes: 0
> tx49_csum_partial: 0
> tx49_csum_partial_inner: 0
> tx49_added_vlan_packets: 0
> tx49_nop: 0
> tx49_csum_none: 0
> tx49_stopped: 0
> tx49_dropped: 0
> tx49_xmit_more: 0
> tx49_recover: 0
> tx49_cqes: 0
> tx49_wake: 0
> tx49_cqe_err: 0
> tx50_packets: 0
> tx50_bytes: 0
> tx50_tso_packets: 0
> tx50_tso_bytes: 0
> tx50_tso_inner_packets: 0
> tx50_tso_inner_bytes: 0
> tx50_csum_partial: 0
> tx50_csum_partial_inner: 0
> tx50_added_vlan_packets: 0
> tx50_nop: 0
> tx50_csum_none: 0
> tx50_stopped: 0
> tx50_dropped: 0
> tx50_xmit_more: 0
> tx50_recover: 0
> tx50_cqes: 0
> tx50_wake: 0
> tx50_cqe_err: 0
> tx51_packets: 0
> tx51_bytes: 0
> tx51_tso_packets: 0
> tx51_tso_bytes: 0
> tx51_tso_inner_packets: 0
> tx51_tso_inner_bytes: 0
> tx51_csum_partial: 0
> tx51_csum_partial_inner: 0
> tx51_added_vlan_packets: 0
> tx51_nop: 0
> tx51_csum_none: 0
> tx51_stopped: 0
> tx51_dropped: 0
> tx51_xmit_more: 0
> tx51_recover: 0
> tx51_cqes: 0
> tx51_wake: 0
> tx51_cqe_err: 0
> tx52_packets: 0
> tx52_bytes: 0
> tx52_tso_packets: 0
> tx52_tso_bytes: 0
> tx52_tso_inner_packets: 0
> tx52_tso_inner_bytes: 0
> tx52_csum_partial: 0
> tx52_csum_partial_inner: 0
> tx52_added_vlan_packets: 0
> tx52_nop: 0
> tx52_csum_none: 0
> tx52_stopped: 0
> tx52_dropped: 0
> tx52_xmit_more: 0
> tx52_recover: 0
> tx52_cqes: 0
> tx52_wake: 0
> tx52_cqe_err: 0
> tx53_packets: 0
> tx53_bytes: 0
> tx53_tso_packets: 0
> tx53_tso_bytes: 0
> tx53_tso_inner_packets: 0
> tx53_tso_inner_bytes: 0
> tx53_csum_partial: 0
> tx53_csum_partial_inner: 0
> tx53_added_vlan_packets: 0
> tx53_nop: 0
> tx53_csum_none: 0
> tx53_stopped: 0
> tx53_dropped: 0
> tx53_xmit_more: 0
> tx53_recover: 0
> tx53_cqes: 0
> tx53_wake: 0
> tx53_cqe_err: 0
> tx54_packets: 0
> tx54_bytes: 0
> tx54_tso_packets: 0
> tx54_tso_bytes: 0
> tx54_tso_inner_packets: 0
> tx54_tso_inner_bytes: 0
> tx54_csum_partial: 0
> tx54_csum_partial_inner: 0
> tx54_added_vlan_packets: 0
> tx54_nop: 0
> tx54_csum_none: 0
> tx54_stopped: 0
> tx54_dropped: 0
> tx54_xmit_more: 0
> tx54_recover: 0
> tx54_cqes: 0
> tx54_wake: 0
> tx54_cqe_err: 0
> tx55_packets: 0
> tx55_bytes: 0
> tx55_tso_packets: 0
> tx55_tso_bytes: 0
> tx55_tso_inner_packets: 0
> tx55_tso_inner_bytes: 0
> tx55_csum_partial: 0
> tx55_csum_partial_inner: 0
> tx55_added_vlan_packets: 0
> tx55_nop: 0
> tx55_csum_none: 0
> tx55_stopped: 0
> tx55_dropped: 0
> tx55_xmit_more: 0
> tx55_recover: 0
> tx55_cqes: 0
> tx55_wake: 0
> tx55_cqe_err: 0
> tx0_xdp_xmit: 0
> tx0_xdp_full: 0
> tx0_xdp_err: 0
> tx0_xdp_cqes: 0
> tx1_xdp_xmit: 0
> tx1_xdp_full: 0
> tx1_xdp_err: 0
> tx1_xdp_cqes: 0
> tx2_xdp_xmit: 0
> tx2_xdp_full: 0
> tx2_xdp_err: 0
> tx2_xdp_cqes: 0
> tx3_xdp_xmit: 0
> tx3_xdp_full: 0
> tx3_xdp_err: 0
> tx3_xdp_cqes: 0
> tx4_xdp_xmit: 0
> tx4_xdp_full: 0
> tx4_xdp_err: 0
> tx4_xdp_cqes: 0
> tx5_xdp_xmit: 0
> tx5_xdp_full: 0
> tx5_xdp_err: 0
> tx5_xdp_cqes: 0
> tx6_xdp_xmit: 0
> tx6_xdp_full: 0
> tx6_xdp_err: 0
> tx6_xdp_cqes: 0
> tx7_xdp_xmit: 0
> tx7_xdp_full: 0
> tx7_xdp_err: 0
> tx7_xdp_cqes: 0
> tx8_xdp_xmit: 0
> tx8_xdp_full: 0
> tx8_xdp_err: 0
> tx8_xdp_cqes: 0
> tx9_xdp_xmit: 0
> tx9_xdp_full: 0
> tx9_xdp_err: 0
> tx9_xdp_cqes: 0
> tx10_xdp_xmit: 0
> tx10_xdp_full: 0
> tx10_xdp_err: 0
> tx10_xdp_cqes: 0
> tx11_xdp_xmit: 0
> tx11_xdp_full: 0
> tx11_xdp_err: 0
> tx11_xdp_cqes: 0
> tx12_xdp_xmit: 0
> tx12_xdp_full: 0
> tx12_xdp_err: 0
> tx12_xdp_cqes: 0
> tx13_xdp_xmit: 0
> tx13_xdp_full: 0
> tx13_xdp_err: 0
> tx13_xdp_cqes: 0
> tx14_xdp_xmit: 0
> tx14_xdp_full: 0
> tx14_xdp_err: 0
> tx14_xdp_cqes: 0
> tx15_xdp_xmit: 0
> tx15_xdp_full: 0
> tx15_xdp_err: 0
> tx15_xdp_cqes: 0
> tx16_xdp_xmit: 0
> tx16_xdp_full: 0
> tx16_xdp_err: 0
> tx16_xdp_cqes: 0
> tx17_xdp_xmit: 0
> tx17_xdp_full: 0
> tx17_xdp_err: 0
> tx17_xdp_cqes: 0
> tx18_xdp_xmit: 0
> tx18_xdp_full: 0
> tx18_xdp_err: 0
> tx18_xdp_cqes: 0
> tx19_xdp_xmit: 0
> tx19_xdp_full: 0
> tx19_xdp_err: 0
> tx19_xdp_cqes: 0
> tx20_xdp_xmit: 0
> tx20_xdp_full: 0
> tx20_xdp_err: 0
> tx20_xdp_cqes: 0
> tx21_xdp_xmit: 0
> tx21_xdp_full: 0
> tx21_xdp_err: 0
> tx21_xdp_cqes: 0
> tx22_xdp_xmit: 0
> tx22_xdp_full: 0
> tx22_xdp_err: 0
> tx22_xdp_cqes: 0
> tx23_xdp_xmit: 0
> tx23_xdp_full: 0
> tx23_xdp_err: 0
> tx23_xdp_cqes: 0
> tx24_xdp_xmit: 0
> tx24_xdp_full: 0
> tx24_xdp_err: 0
> tx24_xdp_cqes: 0
> tx25_xdp_xmit: 0
> tx25_xdp_full: 0
> tx25_xdp_err: 0
> tx25_xdp_cqes: 0
> tx26_xdp_xmit: 0
> tx26_xdp_full: 0
> tx26_xdp_err: 0
> tx26_xdp_cqes: 0
> tx27_xdp_xmit: 0
> tx27_xdp_full: 0
> tx27_xdp_err: 0
> tx27_xdp_cqes: 0
> tx28_xdp_xmit: 0
> tx28_xdp_full: 0
> tx28_xdp_err: 0
> tx28_xdp_cqes: 0
> tx29_xdp_xmit: 0
> tx29_xdp_full: 0
> tx29_xdp_err: 0
> tx29_xdp_cqes: 0
> tx30_xdp_xmit: 0
> tx30_xdp_full: 0
> tx30_xdp_err: 0
> tx30_xdp_cqes: 0
> tx31_xdp_xmit: 0
> tx31_xdp_full: 0
> tx31_xdp_err: 0
> tx31_xdp_cqes: 0
> tx32_xdp_xmit: 0
> tx32_xdp_full: 0
> tx32_xdp_err: 0
> tx32_xdp_cqes: 0
> tx33_xdp_xmit: 0
> tx33_xdp_full: 0
> tx33_xdp_err: 0
> tx33_xdp_cqes: 0
> tx34_xdp_xmit: 0
> tx34_xdp_full: 0
> tx34_xdp_err: 0
> tx34_xdp_cqes: 0
> tx35_xdp_xmit: 0
> tx35_xdp_full: 0
> tx35_xdp_err: 0
> tx35_xdp_cqes: 0
> tx36_xdp_xmit: 0
> tx36_xdp_full: 0
> tx36_xdp_err: 0
> tx36_xdp_cqes: 0
> tx37_xdp_xmit: 0
> tx37_xdp_full: 0
> tx37_xdp_err: 0
> tx37_xdp_cqes: 0
> tx38_xdp_xmit: 0
> tx38_xdp_full: 0
> tx38_xdp_err: 0
> tx38_xdp_cqes: 0
> tx39_xdp_xmit: 0
> tx39_xdp_full: 0
> tx39_xdp_err: 0
> tx39_xdp_cqes: 0
> tx40_xdp_xmit: 0
> tx40_xdp_full: 0
> tx40_xdp_err: 0
> tx40_xdp_cqes: 0
> tx41_xdp_xmit: 0
> tx41_xdp_full: 0
> tx41_xdp_err: 0
> tx41_xdp_cqes: 0
> tx42_xdp_xmit: 0
> tx42_xdp_full: 0
> tx42_xdp_err: 0
> tx42_xdp_cqes: 0
> tx43_xdp_xmit: 0
> tx43_xdp_full: 0
> tx43_xdp_err: 0
> tx43_xdp_cqes: 0
> tx44_xdp_xmit: 0
> tx44_xdp_full: 0
> tx44_xdp_err: 0
> tx44_xdp_cqes: 0
> tx45_xdp_xmit: 0
> tx45_xdp_full: 0
> tx45_xdp_err: 0
> tx45_xdp_cqes: 0
> tx46_xdp_xmit: 0
> tx46_xdp_full: 0
> tx46_xdp_err: 0
> tx46_xdp_cqes: 0
> tx47_xdp_xmit: 0
> tx47_xdp_full: 0
> tx47_xdp_err: 0
> tx47_xdp_cqes: 0
> tx48_xdp_xmit: 0
> tx48_xdp_full: 0
> tx48_xdp_err: 0
> tx48_xdp_cqes: 0
> tx49_xdp_xmit: 0
> tx49_xdp_full: 0
> tx49_xdp_err: 0
> tx49_xdp_cqes: 0
> tx50_xdp_xmit: 0
> tx50_xdp_full: 0
> tx50_xdp_err: 0
> tx50_xdp_cqes: 0
> tx51_xdp_xmit: 0
> tx51_xdp_full: 0
> tx51_xdp_err: 0
> tx51_xdp_cqes: 0
> tx52_xdp_xmit: 0
> tx52_xdp_full: 0
> tx52_xdp_err: 0
> tx52_xdp_cqes: 0
> tx53_xdp_xmit: 0
> tx53_xdp_full: 0
> tx53_xdp_err: 0
> tx53_xdp_cqes: 0
> tx54_xdp_xmit: 0
> tx54_xdp_full: 0
> tx54_xdp_err: 0
> tx54_xdp_cqes: 0
> tx55_xdp_xmit: 0
> tx55_xdp_full: 0
> tx55_xdp_err: 0
> tx55_xdp_cqes: 0
>
> ethtool -S enp175s0f0
> NIC statistics:
> rx_packets: 141574897253
> rx_bytes: 184445040406258
> tx_packets: 172569543894
> tx_bytes: 99486882076365
> tx_tso_packets: 9367664195
> tx_tso_bytes: 56435233992948
> tx_tso_inner_packets: 0
> tx_tso_inner_bytes: 0
> tx_added_vlan_packets: 141297671626
> tx_nop: 2102916272
> rx_lro_packets: 0
> rx_lro_bytes: 0
> rx_ecn_mark: 0
> rx_removed_vlan_packets: 141574897252
> rx_csum_unnecessary: 0
> rx_csum_none: 23135854
> rx_csum_complete: 141551761398
> rx_csum_unnecessary_inner: 0
> rx_xdp_drop: 0
> rx_xdp_redirect: 0
> rx_xdp_tx_xmit: 0
> rx_xdp_tx_full: 0
> rx_xdp_tx_err: 0
> rx_xdp_tx_cqe: 0
> tx_csum_none: 127934791664
It is a good idea to look into this, tx is not requesting hw tx
csumming for a lot of packets, maybe you are wasting a lot of cpu on
calculating csum, or maybe this is just the rx csum complete..
> tx_csum_partial: 13362879974
> tx_csum_partial_inner: 0
> tx_queue_stopped: 232561
TX queues are stalling, could be an indentation for the pcie
bottelneck.
> tx_queue_dropped: 0
> tx_xmit_more: 1266021946
> tx_recover: 0
> tx_cqes: 140031716469
> tx_queue_wake: 232561
> tx_udp_seg_rem: 0
> tx_cqe_err: 0
> tx_xdp_xmit: 0
> tx_xdp_full: 0
> tx_xdp_err: 0
> tx_xdp_cqes: 0
> rx_wqe_err: 0
> rx_mpwqe_filler_cqes: 0
> rx_mpwqe_filler_strides: 0
> rx_buff_alloc_err: 0
> rx_cqe_compress_blks: 0
> rx_cqe_compress_pkts: 0
> rx_page_reuse: 0
> rx_cache_reuse: 16625975793
> rx_cache_full: 54161465914
> rx_cache_empty: 258048
> rx_cache_busy: 54161472735
> rx_cache_waive: 0
> rx_congst_umr: 0
> rx_arfs_err: 0
> ch_events: 40572621887
> ch_poll: 40885650979
> ch_arm: 40429276692
> ch_aff_change: 0
> ch_eq_rearm: 0
> rx_out_of_buffer: 2791690
> rx_if_down_packets: 74
> rx_vport_unicast_packets: 141843476308
> rx_vport_unicast_bytes: 185421265403318
> tx_vport_unicast_packets: 172569484005
> tx_vport_unicast_bytes: 100019940094298
> rx_vport_multicast_packets: 85122935
> rx_vport_multicast_bytes: 5761316431
> tx_vport_multicast_packets: 6452
> tx_vport_multicast_bytes: 643540
> rx_vport_broadcast_packets: 22423624
> rx_vport_broadcast_bytes: 1390127090
> tx_vport_broadcast_packets: 22024
> tx_vport_broadcast_bytes: 1321440
> rx_vport_rdma_unicast_packets: 0
> rx_vport_rdma_unicast_bytes: 0
> tx_vport_rdma_unicast_packets: 0
> tx_vport_rdma_unicast_bytes: 0
> rx_vport_rdma_multicast_packets: 0
> rx_vport_rdma_multicast_bytes: 0
> tx_vport_rdma_multicast_packets: 0
> tx_vport_rdma_multicast_bytes: 0
> tx_packets_phy: 172569501577
> rx_packets_phy: 142871314588
> rx_crc_errors_phy: 0
> tx_bytes_phy: 100710212814151
> rx_bytes_phy: 187209224289564
> tx_multicast_phy: 6452
> tx_broadcast_phy: 22024
> rx_multicast_phy: 85122933
> rx_broadcast_phy: 22423623
> rx_in_range_len_errors_phy: 2
> rx_out_of_range_len_phy: 0
> rx_oversize_pkts_phy: 0
> rx_symbol_err_phy: 0
> tx_mac_control_phy: 0
> rx_mac_control_phy: 0
> rx_unsupported_op_phy: 0
> rx_pause_ctrl_phy: 0
> tx_pause_ctrl_phy: 0
> rx_discards_phy: 920161423
Ok, this port seem to be suffering more, RX is congested, maybe due to
the pcie bottleneck.
> tx_discards_phy: 0
> tx_errors_phy: 0
> rx_undersize_pkts_phy: 0
> rx_fragments_phy: 0
> rx_jabbers_phy: 0
> rx_64_bytes_phy: 412006326
> rx_65_to_127_bytes_phy: 11934371453
> rx_128_to_255_bytes_phy: 3415281165
> rx_256_to_511_bytes_phy: 2072955511
> rx_512_to_1023_bytes_phy: 2415393005
> rx_1024_to_1518_bytes_phy: 72182391608
> rx_1519_to_2047_bytes_phy: 50438902587
> rx_2048_to_4095_bytes_phy: 0
> rx_4096_to_8191_bytes_phy: 0
> rx_8192_to_10239_bytes_phy: 0
> link_down_events_phy: 0
> rx_pcs_symbol_err_phy: 0
> rx_corrected_bits_phy: 0
> rx_pci_signal_integrity: 0
> tx_pci_signal_integrity: 48
> rx_prio0_bytes: 186709842592642
> rx_prio0_packets: 141481966007
> tx_prio0_bytes: 100710171118138
> tx_prio0_packets: 172569437949
> rx_prio1_bytes: 492288152326
> rx_prio1_packets: 385996045
> tx_prio1_bytes: 0
> tx_prio1_packets: 0
> rx_prio2_bytes: 22119952
> rx_prio2_packets: 70788
> tx_prio2_bytes: 0
> tx_prio2_packets: 0
> rx_prio3_bytes: 546141102
> rx_prio3_packets: 681608
> tx_prio3_bytes: 0
> tx_prio3_packets: 0
> rx_prio4_bytes: 14665067
> rx_prio4_packets: 29486
> tx_prio4_bytes: 0
> tx_prio4_packets: 0
> rx_prio5_bytes: 158862504
> rx_prio5_packets: 965307
> tx_prio5_bytes: 0
> tx_prio5_packets: 0
> rx_prio6_bytes: 669337783
> rx_prio6_packets: 1475775
> tx_prio6_bytes: 0
> tx_prio6_packets: 0
> rx_prio7_bytes: 5623481349
> rx_prio7_packets: 79926412
> tx_prio7_bytes: 0
> tx_prio7_packets: 0
> module_unplug: 0
> module_bus_stuck: 0
> module_high_temp: 0
> module_bad_shorted: 0
> ch0_events: 1446162630
> ch0_poll: 1463312972
> ch0_arm: 1440728278
> ch0_aff_change: 0
> ch0_eq_rearm: 0
> ch1_events: 1384301405
> ch1_poll: 1399210915
> ch1_arm: 1378636486
> ch1_aff_change: 0
> ch1_eq_rearm: 0
> ch2_events: 1382788887
> ch2_poll: 1397231470
> ch2_arm: 1377058116
> ch2_aff_change: 0
> ch2_eq_rearm: 0
> ch3_events: 1461956995
> ch3_poll: 1475553146
> ch3_arm: 1456571625
> ch3_aff_change: 0
> ch3_eq_rearm: 0
> ch4_events: 1497359109
> ch4_poll: 1511021037
> ch4_arm: 1491733757
> ch4_aff_change: 0
> ch4_eq_rearm: 0
> ch5_events: 1387736262
> ch5_poll: 1400964615
> ch5_arm: 1382382834
> ch5_aff_change: 0
> ch5_eq_rearm: 0
> ch6_events: 1376772405
> ch6_poll: 1390851449
> ch6_arm: 1371551764
> ch6_aff_change: 0
> ch6_eq_rearm: 0
> ch7_events: 1431271514
> ch7_poll: 1445049729
> ch7_arm: 1425753718
> ch7_aff_change: 0
> ch7_eq_rearm: 0
> ch8_events: 1426976374
> ch8_poll: 1439938692
> ch8_arm: 1421392984
> ch8_aff_change: 0
> ch8_eq_rearm: 0
> ch9_events: 1456160031
> ch9_poll: 1468922870
> ch9_arm: 1450930446
> ch9_aff_change: 0
> ch9_eq_rearm: 0
> ch10_events: 1443640165
> ch10_poll: 1456812203
> ch10_arm: 1438425101
> ch10_aff_change: 0
> ch10_eq_rearm: 0
> ch11_events: 1381104776
> ch11_poll: 1393811057
> ch11_arm: 1376059326
> ch11_aff_change: 0
> ch11_eq_rearm: 0
> ch12_events: 1365223276
> ch12_poll: 1378406059
> ch12_arm: 1359950494
> ch12_aff_change: 0
> ch12_eq_rearm: 0
> ch13_events: 1421622259
> ch13_poll: 1434670996
> ch13_arm: 1416241801
> ch13_aff_change: 0
> ch13_eq_rearm: 0
> ch14_events: 1379084590
> ch14_poll: 1392425015
> ch14_arm: 1373675179
> ch14_aff_change: 0
> ch14_eq_rearm: 0
> ch15_events: 1531217338
> ch15_poll: 1543353833
> ch15_arm: 1526350453
> ch15_aff_change: 0
> ch15_eq_rearm: 0
> ch16_events: 1460469776
> ch16_poll: 1467995928
> ch16_arm: 1456010194
> ch16_aff_change: 0
> ch16_eq_rearm: 0
> ch17_events: 1494067670
> ch17_poll: 1500856680
> ch17_arm: 1489232674
> ch17_aff_change: 0
> ch17_eq_rearm: 0
> ch18_events: 1530126866
> ch18_poll: 1537293620
> ch18_arm: 1525476123
> ch18_aff_change: 0
> ch18_eq_rearm: 0
> ch19_events: 1499526149
> ch19_poll: 1506789309
> ch19_arm: 1495161602
> ch19_aff_change: 0
> ch19_eq_rearm: 0
> ch20_events: 1451479763
> ch20_poll: 1459767921
> ch20_arm: 1446360801
> ch20_aff_change: 0
> ch20_eq_rearm: 0
> ch21_events: 1521413613
> ch21_poll: 1529345146
> ch21_arm: 1517229314
> ch21_aff_change: 0
> ch21_eq_rearm: 0
> ch22_events: 1471950045
> ch22_poll: 1479746764
> ch22_arm: 1467681629
> ch22_aff_change: 0
> ch22_eq_rearm: 0
> ch23_events: 1502968393
> ch23_poll: 1510419909
> ch23_arm: 1498168438
> ch23_aff_change: 0
> ch23_eq_rearm: 0
> ch24_events: 1473451639
> ch24_poll: 1482606899
> ch24_arm: 1468212489
> ch24_aff_change: 0
> ch24_eq_rearm: 0
> ch25_events: 1440399182
> ch25_poll: 1448897475
> ch25_arm: 1435044786
> ch25_aff_change: 0
> ch25_eq_rearm: 0
> ch26_events: 1436831565
> ch26_poll: 1445485731
> ch26_arm: 1431827527
> ch26_aff_change: 0
> ch26_eq_rearm: 0
> ch27_events: 1516560621
> ch27_poll: 1524911010
> ch27_arm: 1511430164
> ch27_aff_change: 0
> ch27_eq_rearm: 0
> ch28_events: 4
> ch28_poll: 4
> ch28_arm: 4
> ch28_aff_change: 0
> ch28_eq_rearm: 0
> ch29_events: 6
> ch29_poll: 6
> ch29_arm: 6
> ch29_aff_change: 0
> ch29_eq_rearm: 0
> ch30_events: 4
> ch30_poll: 4
> ch30_arm: 4
> ch30_aff_change: 0
> ch30_eq_rearm: 0
> ch31_events: 4
> ch31_poll: 4
> ch31_arm: 4
> ch31_aff_change: 0
> ch31_eq_rearm: 0
> ch32_events: 4
> ch32_poll: 4
> ch32_arm: 4
> ch32_aff_change: 0
> ch32_eq_rearm: 0
> ch33_events: 4
> ch33_poll: 4
> ch33_arm: 4
> ch33_aff_change: 0
> ch33_eq_rearm: 0
> ch34_events: 4
> ch34_poll: 4
> ch34_arm: 4
> ch34_aff_change: 0
> ch34_eq_rearm: 0
> ch35_events: 4
> ch35_poll: 4
> ch35_arm: 4
> ch35_aff_change: 0
> ch35_eq_rearm: 0
> ch36_events: 4
> ch36_poll: 4
> ch36_arm: 4
> ch36_aff_change: 0
> ch36_eq_rearm: 0
> ch37_events: 4
> ch37_poll: 4
> ch37_arm: 4
> ch37_aff_change: 0
> ch37_eq_rearm: 0
> ch38_events: 4
> ch38_poll: 4
> ch38_arm: 4
> ch38_aff_change: 0
> ch38_eq_rearm: 0
> ch39_events: 4
> ch39_poll: 4
> ch39_arm: 4
> ch39_aff_change: 0
> ch39_eq_rearm: 0
> ch40_events: 4
> ch40_poll: 4
> ch40_arm: 4
> ch40_aff_change: 0
> ch40_eq_rearm: 0
> ch41_events: 4
> ch41_poll: 4
> ch41_arm: 4
> ch41_aff_change: 0
> ch41_eq_rearm: 0
> ch42_events: 4
> ch42_poll: 4
> ch42_arm: 4
> ch42_aff_change: 0
> ch42_eq_rearm: 0
> ch43_events: 4
> ch43_poll: 4
> ch43_arm: 4
> ch43_aff_change: 0
> ch43_eq_rearm: 0
> ch44_events: 4
> ch44_poll: 4
> ch44_arm: 4
> ch44_aff_change: 0
> ch44_eq_rearm: 0
> ch45_events: 4
> ch45_poll: 4
> ch45_arm: 4
> ch45_aff_change: 0
> ch45_eq_rearm: 0
> ch46_events: 4
> ch46_poll: 4
> ch46_arm: 4
> ch46_aff_change: 0
> ch46_eq_rearm: 0
> ch47_events: 4
> ch47_poll: 4
> ch47_arm: 4
> ch47_aff_change: 0
> ch47_eq_rearm: 0
> ch48_events: 4
> ch48_poll: 4
> ch48_arm: 4
> ch48_aff_change: 0
> ch48_eq_rearm: 0
> ch49_events: 4
> ch49_poll: 4
> ch49_arm: 4
> ch49_aff_change: 0
> ch49_eq_rearm: 0
> ch50_events: 4
> ch50_poll: 4
> ch50_arm: 4
> ch50_aff_change: 0
> ch50_eq_rearm: 0
> ch51_events: 4
> ch51_poll: 4
> ch51_arm: 4
> ch51_aff_change: 0
> ch51_eq_rearm: 0
> ch52_events: 4
> ch52_poll: 4
> ch52_arm: 4
> ch52_aff_change: 0
> ch52_eq_rearm: 0
> ch53_events: 4
> ch53_poll: 4
> ch53_arm: 4
> ch53_aff_change: 0
> ch53_eq_rearm: 0
> ch54_events: 4
> ch54_poll: 4
> ch54_arm: 4
> ch54_aff_change: 0
> ch54_eq_rearm: 0
> ch55_events: 4
> ch55_poll: 4
> ch55_arm: 4
> ch55_aff_change: 0
> ch55_eq_rearm: 0
> rx0_packets: 5861448653
> rx0_bytes: 7389128595728
> rx0_csum_complete: 5838312798
> rx0_csum_unnecessary: 0
> rx0_csum_unnecessary_inner: 0
> rx0_csum_none: 23135855
> rx0_xdp_drop: 0
> rx0_xdp_redirect: 0
> rx0_lro_packets: 0
> rx0_lro_bytes: 0
> rx0_ecn_mark: 0
> rx0_removed_vlan_packets: 5861448653
> rx0_wqe_err: 0
> rx0_mpwqe_filler_cqes: 0
> rx0_mpwqe_filler_strides: 0
> rx0_buff_alloc_err: 0
> rx0_cqe_compress_blks: 0
> rx0_cqe_compress_pkts: 0
> rx0_page_reuse: 0
> rx0_cache_reuse: 2559
> rx0_cache_full: 2930721512
> rx0_cache_empty: 6656
> rx0_cache_busy: 2930721765
> rx0_cache_waive: 0
> rx0_congst_umr: 0
> rx0_arfs_err: 0
> rx0_xdp_tx_xmit: 0
> rx0_xdp_tx_full: 0
> rx0_xdp_tx_err: 0
> rx0_xdp_tx_cqes: 0
> rx1_packets: 5550585106
> rx1_bytes: 7255635262803
> rx1_csum_complete: 5550585106
> rx1_csum_unnecessary: 0
> rx1_csum_unnecessary_inner: 0
> rx1_csum_none: 0
> rx1_xdp_drop: 0
> rx1_xdp_redirect: 0
> rx1_lro_packets: 0
> rx1_lro_bytes: 0
> rx1_ecn_mark: 0
> rx1_removed_vlan_packets: 5550585106
> rx1_wqe_err: 0
> rx1_mpwqe_filler_cqes: 0
> rx1_mpwqe_filler_strides: 0
> rx1_buff_alloc_err: 0
> rx1_cqe_compress_blks: 0
> rx1_cqe_compress_pkts: 0
> rx1_page_reuse: 0
> rx1_cache_reuse: 2918845
> rx1_cache_full: 2772373453
> rx1_cache_empty: 6656
> rx1_cache_busy: 2772373707
> rx1_cache_waive: 0
> rx1_congst_umr: 0
> rx1_arfs_err: 0
> rx1_xdp_tx_xmit: 0
> rx1_xdp_tx_full: 0
> rx1_xdp_tx_err: 0
> rx1_xdp_tx_cqes: 0
> rx2_packets: 5383874739
> rx2_bytes: 7031545423967
> rx2_csum_complete: 5383874739
> rx2_csum_unnecessary: 0
> rx2_csum_unnecessary_inner: 0
> rx2_csum_none: 0
> rx2_xdp_drop: 0
> rx2_xdp_redirect: 0
> rx2_lro_packets: 0
> rx2_lro_bytes: 0
> rx2_ecn_mark: 0
> rx2_removed_vlan_packets: 5383874739
> rx2_wqe_err: 0
> rx2_mpwqe_filler_cqes: 0
> rx2_mpwqe_filler_strides: 0
> rx2_buff_alloc_err: 0
> rx2_cqe_compress_blks: 0
> rx2_cqe_compress_pkts: 0
> rx2_page_reuse: 0
> rx2_cache_reuse: 2173370
> rx2_cache_full: 2689763744
> rx2_cache_empty: 6656
> rx2_cache_busy: 2689763998
> rx2_cache_waive: 0
> rx2_congst_umr: 0
> rx2_arfs_err: 0
> rx2_xdp_tx_xmit: 0
> rx2_xdp_tx_full: 0
> rx2_xdp_tx_err: 0
> rx2_xdp_tx_cqes: 0
> rx3_packets: 5456494012
> rx3_bytes: 7120241119485
> rx3_csum_complete: 5456494012
> rx3_csum_unnecessary: 0
> rx3_csum_unnecessary_inner: 0
> rx3_csum_none: 0
> rx3_xdp_drop: 0
> rx3_xdp_redirect: 0
> rx3_lro_packets: 0
> rx3_lro_bytes: 0
> rx3_ecn_mark: 0
> rx3_removed_vlan_packets: 5456494012
> rx3_wqe_err: 0
> rx3_mpwqe_filler_cqes: 0
> rx3_mpwqe_filler_strides: 0
> rx3_buff_alloc_err: 0
> rx3_cqe_compress_blks: 0
> rx3_cqe_compress_pkts: 0
> rx3_page_reuse: 0
> rx3_cache_reuse: 2120123
> rx3_cache_full: 2726126628
> rx3_cache_empty: 6656
> rx3_cache_busy: 2726126881
> rx3_cache_waive: 0
> rx3_congst_umr: 0
> rx3_arfs_err: 0
> rx3_xdp_tx_xmit: 0
> rx3_xdp_tx_full: 0
> rx3_xdp_tx_err: 0
> rx3_xdp_tx_cqes: 0
> rx4_packets: 5475216251
> rx4_bytes: 7123129170196
> rx4_csum_complete: 5475216251
> rx4_csum_unnecessary: 0
> rx4_csum_unnecessary_inner: 0
> rx4_csum_none: 0
> rx4_xdp_drop: 0
> rx4_xdp_redirect: 0
> rx4_lro_packets: 0
> rx4_lro_bytes: 0
> rx4_ecn_mark: 0
> rx4_removed_vlan_packets: 5475216251
> rx4_wqe_err: 0
> rx4_mpwqe_filler_cqes: 0
> rx4_mpwqe_filler_strides: 0
> rx4_buff_alloc_err: 0
> rx4_cqe_compress_blks: 0
> rx4_cqe_compress_pkts: 0
> rx4_page_reuse: 0
> rx4_cache_reuse: 2668296355
> rx4_cache_full: 69311549
> rx4_cache_empty: 6656
> rx4_cache_busy: 69311769
> rx4_cache_waive: 0
> rx4_congst_umr: 0
> rx4_arfs_err: 0
> rx4_xdp_tx_xmit: 0
> rx4_xdp_tx_full: 0
> rx4_xdp_tx_err: 0
> rx4_xdp_tx_cqes: 0
> rx5_packets: 5474372232
> rx5_bytes: 7159146801926
> rx5_csum_complete: 5474372232
> rx5_csum_unnecessary: 0
> rx5_csum_unnecessary_inner: 0
> rx5_csum_none: 0
> rx5_xdp_drop: 0
> rx5_xdp_redirect: 0
> rx5_lro_packets: 0
> rx5_lro_bytes: 0
> rx5_ecn_mark: 0
> rx5_removed_vlan_packets: 5474372232
> rx5_wqe_err: 0
> rx5_mpwqe_filler_cqes: 0
> rx5_mpwqe_filler_strides: 0
> rx5_buff_alloc_err: 0
> rx5_cqe_compress_blks: 0
> rx5_cqe_compress_pkts: 0
> rx5_page_reuse: 0
> rx5_cache_reuse: 626187
> rx5_cache_full: 2736559674
> rx5_cache_empty: 6656
> rx5_cache_busy: 2736559929
> rx5_cache_waive: 0
> rx5_congst_umr: 0
> rx5_arfs_err: 0
> rx5_xdp_tx_xmit: 0
> rx5_xdp_tx_full: 0
> rx5_xdp_tx_err: 0
> rx5_xdp_tx_cqes: 0
> rx6_packets: 5533622456
> rx6_bytes: 7207308809081
> rx6_csum_complete: 5533622456
> rx6_csum_unnecessary: 0
> rx6_csum_unnecessary_inner: 0
> rx6_csum_none: 0
> rx6_xdp_drop: 0
> rx6_xdp_redirect: 0
> rx6_lro_packets: 0
> rx6_lro_bytes: 0
> rx6_ecn_mark: 0
> rx6_removed_vlan_packets: 5533622456
> rx6_wqe_err: 0
> rx6_mpwqe_filler_cqes: 0
> rx6_mpwqe_filler_strides: 0
> rx6_buff_alloc_err: 0
> rx6_cqe_compress_blks: 0
> rx6_cqe_compress_pkts: 0
> rx6_page_reuse: 0
> rx6_cache_reuse: 2325217
> rx6_cache_full: 2764485756
> rx6_cache_empty: 6656
> rx6_cache_busy: 2764486011
> rx6_cache_waive: 0
> rx6_congst_umr: 0
> rx6_arfs_err: 0
> rx6_xdp_tx_xmit: 0
> rx6_xdp_tx_full: 0
> rx6_xdp_tx_err: 0
> rx6_xdp_tx_cqes: 0
> rx7_packets: 5533901822
> rx7_bytes: 7227441240536
> rx7_csum_complete: 5533901822
> rx7_csum_unnecessary: 0
> rx7_csum_unnecessary_inner: 0
> rx7_csum_none: 0
> rx7_xdp_drop: 0
> rx7_xdp_redirect: 0
> rx7_lro_packets: 0
> rx7_lro_bytes: 0
> rx7_ecn_mark: 0
> rx7_removed_vlan_packets: 5533901822
> rx7_wqe_err: 0
> rx7_mpwqe_filler_cqes: 0
> rx7_mpwqe_filler_strides: 0
> rx7_buff_alloc_err: 0
> rx7_cqe_compress_blks: 0
> rx7_cqe_compress_pkts: 0
> rx7_page_reuse: 0
> rx7_cache_reuse: 2372505
> rx7_cache_full: 2764578151
> rx7_cache_empty: 6656
> rx7_cache_busy: 2764578403
> rx7_cache_waive: 0
> rx7_congst_umr: 0
> rx7_arfs_err: 0
> rx7_xdp_tx_xmit: 0
> rx7_xdp_tx_full: 0
> rx7_xdp_tx_err: 0
> rx7_xdp_tx_cqes: 0
> rx8_packets: 5485670137
> rx8_bytes: 7203339989013
> rx8_csum_complete: 5485670137
> rx8_csum_unnecessary: 0
> rx8_csum_unnecessary_inner: 0
> rx8_csum_none: 0
> rx8_xdp_drop: 0
> rx8_xdp_redirect: 0
> rx8_lro_packets: 0
> rx8_lro_bytes: 0
> rx8_ecn_mark: 0
> rx8_removed_vlan_packets: 5485670137
> rx8_wqe_err: 0
> rx8_mpwqe_filler_cqes: 0
> rx8_mpwqe_filler_strides: 0
> rx8_buff_alloc_err: 0
> rx8_cqe_compress_blks: 0
> rx8_cqe_compress_pkts: 0
> rx8_page_reuse: 0
> rx8_cache_reuse: 7522232
> rx8_cache_full: 2735312581
> rx8_cache_empty: 6656
> rx8_cache_busy: 2735312836
> rx8_cache_waive: 0
> rx8_congst_umr: 0
> rx8_arfs_err: 0
> rx8_xdp_tx_xmit: 0
> rx8_xdp_tx_full: 0
> rx8_xdp_tx_err: 0
> rx8_xdp_tx_cqes: 0
> rx9_packets: 5482212354
> rx9_bytes: 7169663341718
> rx9_csum_complete: 5482212354
> rx9_csum_unnecessary: 0
> rx9_csum_unnecessary_inner: 0
> rx9_csum_none: 0
> rx9_xdp_drop: 0
> rx9_xdp_redirect: 0
> rx9_lro_packets: 0
> rx9_lro_bytes: 0
> rx9_ecn_mark: 0
> rx9_removed_vlan_packets: 5482212354
> rx9_wqe_err: 0
> rx9_mpwqe_filler_cqes: 0
> rx9_mpwqe_filler_strides: 0
> rx9_buff_alloc_err: 0
> rx9_cqe_compress_blks: 0
> rx9_cqe_compress_pkts: 0
> rx9_page_reuse: 0
> rx9_cache_reuse: 37279961
> rx9_cache_full: 2703825961
> rx9_cache_empty: 6656
> rx9_cache_busy: 2703826215
> rx9_cache_waive: 0
> rx9_congst_umr: 0
> rx9_arfs_err: 0
> rx9_xdp_tx_xmit: 0
> rx9_xdp_tx_full: 0
> rx9_xdp_tx_err: 0
> rx9_xdp_tx_cqes: 0
> rx10_packets: 5524679952
> rx10_bytes: 7248301275181
> rx10_csum_complete: 5524679952
> rx10_csum_unnecessary: 0
> rx10_csum_unnecessary_inner: 0
> rx10_csum_none: 0
> rx10_xdp_drop: 0
> rx10_xdp_redirect: 0
> rx10_lro_packets: 0
> rx10_lro_bytes: 0
> rx10_ecn_mark: 0
> rx10_removed_vlan_packets: 5524679952
> rx10_wqe_err: 0
> rx10_mpwqe_filler_cqes: 0
> rx10_mpwqe_filler_strides: 0
> rx10_buff_alloc_err: 0
> rx10_cqe_compress_blks: 0
> rx10_cqe_compress_pkts: 0
> rx10_page_reuse: 0
> rx10_cache_reuse: 2049666
> rx10_cache_full: 2760290055
> rx10_cache_empty: 6656
> rx10_cache_busy: 2760290310
> rx10_cache_waive: 0
> rx10_congst_umr: 0
> rx10_arfs_err: 0
> rx10_xdp_tx_xmit: 0
> rx10_xdp_tx_full: 0
> rx10_xdp_tx_err: 0
> rx10_xdp_tx_cqes: 0
> rx11_packets: 5394633545
> rx11_bytes: 7033509636092
> rx11_csum_complete: 5394633545
> rx11_csum_unnecessary: 0
> rx11_csum_unnecessary_inner: 0
> rx11_csum_none: 0
> rx11_xdp_drop: 0
> rx11_xdp_redirect: 0
> rx11_lro_packets: 0
> rx11_lro_bytes: 0
> rx11_ecn_mark: 0
> rx11_removed_vlan_packets: 5394633545
> rx11_wqe_err: 0
> rx11_mpwqe_filler_cqes: 0
> rx11_mpwqe_filler_strides: 0
> rx11_buff_alloc_err: 0
> rx11_cqe_compress_blks: 0
> rx11_cqe_compress_pkts: 0
> rx11_page_reuse: 0
> rx11_cache_reuse: 2617466268
> rx11_cache_full: 79850284
> rx11_cache_empty: 6656
> rx11_cache_busy: 79850504
> rx11_cache_waive: 0
> rx11_congst_umr: 0
> rx11_arfs_err: 0
> rx11_xdp_tx_xmit: 0
> rx11_xdp_tx_full: 0
> rx11_xdp_tx_err: 0
> rx11_xdp_tx_cqes: 0
> rx12_packets: 5458907385
> rx12_bytes: 7134867867515
> rx12_csum_complete: 5458907385
> rx12_csum_unnecessary: 0
> rx12_csum_unnecessary_inner: 0
> rx12_csum_none: 0
> rx12_xdp_drop: 0
> rx12_xdp_redirect: 0
> rx12_lro_packets: 0
> rx12_lro_bytes: 0
> rx12_ecn_mark: 0
> rx12_removed_vlan_packets: 5458907385
> rx12_wqe_err: 0
> rx12_mpwqe_filler_cqes: 0
> rx12_mpwqe_filler_strides: 0
> rx12_buff_alloc_err: 0
> rx12_cqe_compress_blks: 0
> rx12_cqe_compress_pkts: 0
> rx12_page_reuse: 0
> rx12_cache_reuse: 2650214169
> rx12_cache_full: 79239303
> rx12_cache_empty: 6656
> rx12_cache_busy: 79239523
> rx12_cache_waive: 0
> rx12_congst_umr: 0
> rx12_arfs_err: 0
> rx12_xdp_tx_xmit: 0
> rx12_xdp_tx_full: 0
> rx12_xdp_tx_err: 0
> rx12_xdp_tx_cqes: 0
> rx13_packets: 5549932912
> rx13_bytes: 7232548705586
> rx13_csum_complete: 5549932912
> rx13_csum_unnecessary: 0
> rx13_csum_unnecessary_inner: 0
> rx13_csum_none: 0
> rx13_xdp_drop: 0
> rx13_xdp_redirect: 0
> rx13_lro_packets: 0
> rx13_lro_bytes: 0
> rx13_ecn_mark: 0
> rx13_removed_vlan_packets: 5549932912
> rx13_wqe_err: 0
> rx13_mpwqe_filler_cqes: 0
> rx13_mpwqe_filler_strides: 0
> rx13_buff_alloc_err: 0
> rx13_cqe_compress_blks: 0
> rx13_cqe_compress_pkts: 0
> rx13_page_reuse: 0
> rx13_cache_reuse: 2417696
> rx13_cache_full: 2772548505
> rx13_cache_empty: 6656
> rx13_cache_busy: 2772548760
> rx13_cache_waive: 0
> rx13_congst_umr: 0
> rx13_arfs_err: 0
> rx13_xdp_tx_xmit: 0
> rx13_xdp_tx_full: 0
> rx13_xdp_tx_err: 0
> rx13_xdp_tx_cqes: 0
> rx14_packets: 5517712329
> rx14_bytes: 7192111965227
> rx14_csum_complete: 5517712329
> rx14_csum_unnecessary: 0
> rx14_csum_unnecessary_inner: 0
> rx14_csum_none: 0
> rx14_xdp_drop: 0
> rx14_xdp_redirect: 0
> rx14_lro_packets: 0
> rx14_lro_bytes: 0
> rx14_ecn_mark: 0
> rx14_removed_vlan_packets: 5517712329
> rx14_wqe_err: 0
> rx14_mpwqe_filler_cqes: 0
> rx14_mpwqe_filler_strides: 0
> rx14_buff_alloc_err: 0
> rx14_cqe_compress_blks: 0
> rx14_cqe_compress_pkts: 0
> rx14_page_reuse: 0
> rx14_cache_reuse: 1830206
> rx14_cache_full: 2757025703
> rx14_cache_empty: 6656
> rx14_cache_busy: 2757025958
> rx14_cache_waive: 0
> rx14_congst_umr: 0
> rx14_arfs_err: 0
> rx14_xdp_tx_xmit: 0
> rx14_xdp_tx_full: 0
> rx14_xdp_tx_err: 0
> rx14_xdp_tx_cqes: 0
> rx15_packets: 5578343373
> rx15_bytes: 7268484501219
> rx15_csum_complete: 5578343373
> rx15_csum_unnecessary: 0
> rx15_csum_unnecessary_inner: 0
> rx15_csum_none: 0
> rx15_xdp_drop: 0
> rx15_xdp_redirect: 0
> rx15_lro_packets: 0
> rx15_lro_bytes: 0
> rx15_ecn_mark: 0
> rx15_removed_vlan_packets: 5578343373
> rx15_wqe_err: 0
> rx15_mpwqe_filler_cqes: 0
> rx15_mpwqe_filler_strides: 0
> rx15_buff_alloc_err: 0
> rx15_cqe_compress_blks: 0
> rx15_cqe_compress_pkts: 0
> rx15_page_reuse: 0
> rx15_cache_reuse: 2317165
> rx15_cache_full: 2786854266
> rx15_cache_empty: 6656
> rx15_cache_busy: 2786854519
> rx15_cache_waive: 0
> rx15_congst_umr: 0
> rx15_arfs_err: 0
> rx15_xdp_tx_xmit: 0
> rx15_xdp_tx_full: 0
> rx15_xdp_tx_err: 0
> rx15_xdp_tx_cqes: 0
> rx16_packets: 4435773951
> rx16_bytes: 5766665272007
> rx16_csum_complete: 4435773951
> rx16_csum_unnecessary: 0
> rx16_csum_unnecessary_inner: 0
> rx16_csum_none: 0
> rx16_xdp_drop: 0
> rx16_xdp_redirect: 0
> rx16_lro_packets: 0
> rx16_lro_bytes: 0
> rx16_ecn_mark: 0
> rx16_removed_vlan_packets: 4435773951
> rx16_wqe_err: 0
> rx16_mpwqe_filler_cqes: 0
> rx16_mpwqe_filler_strides: 0
> rx16_buff_alloc_err: 0
> rx16_cqe_compress_blks: 0
> rx16_cqe_compress_pkts: 0
> rx16_page_reuse: 0
> rx16_cache_reuse: 2033793
> rx16_cache_full: 2215852927
> rx16_cache_empty: 6656
> rx16_cache_busy: 2215853179
> rx16_cache_waive: 0
> rx16_congst_umr: 0
> rx16_arfs_err: 0
> rx16_xdp_tx_xmit: 0
> rx16_xdp_tx_full: 0
> rx16_xdp_tx_err: 0
> rx16_xdp_tx_cqes: 0
> rx17_packets: 4344087587
> rx17_bytes: 5695006496323
> rx17_csum_complete: 4344087587
> rx17_csum_unnecessary: 0
> rx17_csum_unnecessary_inner: 0
> rx17_csum_none: 0
> rx17_xdp_drop: 0
> rx17_xdp_redirect: 0
> rx17_lro_packets: 0
> rx17_lro_bytes: 0
> rx17_ecn_mark: 0
> rx17_removed_vlan_packets: 4344087587
> rx17_wqe_err: 0
> rx17_mpwqe_filler_cqes: 0
> rx17_mpwqe_filler_strides: 0
> rx17_buff_alloc_err: 0
> rx17_cqe_compress_blks: 0
> rx17_cqe_compress_pkts: 0
> rx17_page_reuse: 0
> rx17_cache_reuse: 2652127
> rx17_cache_full: 2169391411
> rx17_cache_empty: 6656
> rx17_cache_busy: 2169391665
> rx17_cache_waive: 0
> rx17_congst_umr: 0
> rx17_arfs_err: 0
> rx17_xdp_tx_xmit: 0
> rx17_xdp_tx_full: 0
> rx17_xdp_tx_err: 0
> rx17_xdp_tx_cqes: 0
> rx18_packets: 4407422804
> rx18_bytes: 5741134634177
> rx18_csum_complete: 4407422804
> rx18_csum_unnecessary: 0
> rx18_csum_unnecessary_inner: 0
> rx18_csum_none: 0
> rx18_xdp_drop: 0
> rx18_xdp_redirect: 0
> rx18_lro_packets: 0
> rx18_lro_bytes: 0
> rx18_ecn_mark: 0
> rx18_removed_vlan_packets: 4407422804
> rx18_wqe_err: 0
> rx18_mpwqe_filler_cqes: 0
> rx18_mpwqe_filler_strides: 0
> rx18_buff_alloc_err: 0
> rx18_cqe_compress_blks: 0
> rx18_cqe_compress_pkts: 0
> rx18_page_reuse: 0
> rx18_cache_reuse: 2156080239
> rx18_cache_full: 47630941
> rx18_cache_empty: 6656
> rx18_cache_busy: 47631161
> rx18_cache_waive: 0
> rx18_congst_umr: 0
> rx18_arfs_err: 0
> rx18_xdp_tx_xmit: 0
> rx18_xdp_tx_full: 0
> rx18_xdp_tx_err: 0
> rx18_xdp_tx_cqes: 0
> rx19_packets: 4545554180
> rx19_bytes: 5905277503466
> rx19_csum_complete: 4545554180
> rx19_csum_unnecessary: 0
> rx19_csum_unnecessary_inner: 0
> rx19_csum_none: 0
> rx19_xdp_drop: 0
> rx19_xdp_redirect: 0
> rx19_lro_packets: 0
> rx19_lro_bytes: 0
> rx19_ecn_mark: 0
> rx19_removed_vlan_packets: 4545554180
> rx19_wqe_err: 0
> rx19_mpwqe_filler_cqes: 0
> rx19_mpwqe_filler_strides: 0
> rx19_buff_alloc_err: 0
> rx19_cqe_compress_blks: 0
> rx19_cqe_compress_pkts: 0
> rx19_page_reuse: 0
> rx19_cache_reuse: 11112455
> rx19_cache_full: 2261664379
> rx19_cache_empty: 6656
> rx19_cache_busy: 2261664601
> rx19_cache_waive: 0
> rx19_congst_umr: 0
> rx19_arfs_err: 0
> rx19_xdp_tx_xmit: 0
> rx19_xdp_tx_full: 0
> rx19_xdp_tx_err: 0
> rx19_xdp_tx_cqes: 0
> rx20_packets: 4397428553
> rx20_bytes: 5757329184301
> rx20_csum_complete: 4397428553
> rx20_csum_unnecessary: 0
> rx20_csum_unnecessary_inner: 0
> rx20_csum_none: 0
> rx20_xdp_drop: 0
> rx20_xdp_redirect: 0
> rx20_lro_packets: 0
> rx20_lro_bytes: 0
> rx20_ecn_mark: 0
> rx20_removed_vlan_packets: 4397428553
> rx20_wqe_err: 0
> rx20_mpwqe_filler_cqes: 0
> rx20_mpwqe_filler_strides: 0
> rx20_buff_alloc_err: 0
> rx20_cqe_compress_blks: 0
> rx20_cqe_compress_pkts: 0
> rx20_page_reuse: 0
> rx20_cache_reuse: 2168116995
> rx20_cache_full: 30597061
> rx20_cache_empty: 6656
> rx20_cache_busy: 30597281
> rx20_cache_waive: 0
> rx20_congst_umr: 0
> rx20_arfs_err: 0
> rx20_xdp_tx_xmit: 0
> rx20_xdp_tx_full: 0
> rx20_xdp_tx_err: 0
> rx20_xdp_tx_cqes: 0
> rx21_packets: 4552564821
> rx21_bytes: 5944840329249
> rx21_csum_complete: 4552564821
> rx21_csum_unnecessary: 0
> rx21_csum_unnecessary_inner: 0
> rx21_csum_none: 0
> rx21_xdp_drop: 0
> rx21_xdp_redirect: 0
> rx21_lro_packets: 0
> rx21_lro_bytes: 0
> rx21_ecn_mark: 0
> rx21_removed_vlan_packets: 4552564821
> rx21_wqe_err: 0
> rx21_mpwqe_filler_cqes: 0
> rx21_mpwqe_filler_strides: 0
> rx21_buff_alloc_err: 0
> rx21_cqe_compress_blks: 0
> rx21_cqe_compress_pkts: 0
> rx21_page_reuse: 0
> rx21_cache_reuse: 2295681
> rx21_cache_full: 2273986474
> rx21_cache_empty: 6656
> rx21_cache_busy: 2273986727
> rx21_cache_waive: 0
> rx21_congst_umr: 0
> rx21_arfs_err: 0
> rx21_xdp_tx_xmit: 0
> rx21_xdp_tx_full: 0
> rx21_xdp_tx_err: 0
> rx21_xdp_tx_cqes: 0
> rx22_packets: 4629499740
> rx22_bytes: 5924206566499
> rx22_csum_complete: 4629499740
> rx22_csum_unnecessary: 0
> rx22_csum_unnecessary_inner: 0
> rx22_csum_none: 0
> rx22_xdp_drop: 0
> rx22_xdp_redirect: 0
> rx22_lro_packets: 0
> rx22_lro_bytes: 0
> rx22_ecn_mark: 0
> rx22_removed_vlan_packets: 4629499740
> rx22_wqe_err: 0
> rx22_mpwqe_filler_cqes: 0
> rx22_mpwqe_filler_strides: 0
> rx22_buff_alloc_err: 0
> rx22_cqe_compress_blks: 0
> rx22_cqe_compress_pkts: 0
> rx22_page_reuse: 0
> rx22_cache_reuse: 1407527
> rx22_cache_full: 2313342088
> rx22_cache_empty: 6656
> rx22_cache_busy: 2313342341
> rx22_cache_waive: 0
> rx22_congst_umr: 0
> rx22_arfs_err: 0
> rx22_xdp_tx_xmit: 0
> rx22_xdp_tx_full: 0
> rx22_xdp_tx_err: 0
> rx22_xdp_tx_cqes: 0
> rx23_packets: 4387124505
> rx23_bytes: 5718118678470
> rx23_csum_complete: 4387124505
> rx23_csum_unnecessary: 0
> rx23_csum_unnecessary_inner: 0
> rx23_csum_none: 0
> rx23_xdp_drop: 0
> rx23_xdp_redirect: 0
> rx23_lro_packets: 0
> rx23_lro_bytes: 0
> rx23_ecn_mark: 0
> rx23_removed_vlan_packets: 4387124505
> rx23_wqe_err: 0
> rx23_mpwqe_filler_cqes: 0
> rx23_mpwqe_filler_strides: 0
> rx23_buff_alloc_err: 0
> rx23_cqe_compress_blks: 0
> rx23_cqe_compress_pkts: 0
> rx23_page_reuse: 0
> rx23_cache_reuse: 2013280
> rx23_cache_full: 2191548717
> rx23_cache_empty: 6656
> rx23_cache_busy: 2191548972
> rx23_cache_waive: 0
> rx23_congst_umr: 0
> rx23_arfs_err: 0
> rx23_xdp_tx_xmit: 0
> rx23_xdp_tx_full: 0
> rx23_xdp_tx_err: 0
> rx23_xdp_tx_cqes: 0
> rx24_packets: 4398791634
> rx24_bytes: 5744875564632
> rx24_csum_complete: 4398791634
> rx24_csum_unnecessary: 0
> rx24_csum_unnecessary_inner: 0
> rx24_csum_none: 0
> rx24_xdp_drop: 0
> rx24_xdp_redirect: 0
> rx24_lro_packets: 0
> rx24_lro_bytes: 0
> rx24_ecn_mark: 0
> rx24_removed_vlan_packets: 4398791634
> rx24_wqe_err: 0
> rx24_mpwqe_filler_cqes: 0
> rx24_mpwqe_filler_strides: 0
> rx24_buff_alloc_err: 0
> rx24_cqe_compress_blks: 0
> rx24_cqe_compress_pkts: 0
> rx24_page_reuse: 0
> rx24_cache_reuse: 2143926100
> rx24_cache_full: 55469496
> rx24_cache_empty: 6656
> rx24_cache_busy: 55469716
> rx24_cache_waive: 0
> rx24_congst_umr: 0
> rx24_arfs_err: 0
> rx24_xdp_tx_xmit: 0
> rx24_xdp_tx_full: 0
> rx24_xdp_tx_err: 0
> rx24_xdp_tx_cqes: 0
> rx25_packets: 4377204935
> rx25_bytes: 5710369124105
> rx25_csum_complete: 4377204935
> rx25_csum_unnecessary: 0
> rx25_csum_unnecessary_inner: 0
> rx25_csum_none: 0
> rx25_xdp_drop: 0
> rx25_xdp_redirect: 0
> rx25_lro_packets: 0
> rx25_lro_bytes: 0
> rx25_ecn_mark: 0
> rx25_removed_vlan_packets: 4377204935
> rx25_wqe_err: 0
> rx25_mpwqe_filler_cqes: 0
> rx25_mpwqe_filler_strides: 0
> rx25_buff_alloc_err: 0
> rx25_cqe_compress_blks: 0
> rx25_cqe_compress_pkts: 0
> rx25_page_reuse: 0
> rx25_cache_reuse: 2132658660
> rx25_cache_full: 55943584
> rx25_cache_empty: 6656
> rx25_cache_busy: 55943804
> rx25_cache_waive: 0
> rx25_congst_umr: 0
> rx25_arfs_err: 0
> rx25_xdp_tx_xmit: 0
> rx25_xdp_tx_full: 0
> rx25_xdp_tx_err: 0
> rx25_xdp_tx_cqes: 0
> rx26_packets: 4496003688
> rx26_bytes: 5862180715503
> rx26_csum_complete: 4496003688
> rx26_csum_unnecessary: 0
> rx26_csum_unnecessary_inner: 0
> rx26_csum_none: 0
> rx26_xdp_drop: 0
> rx26_xdp_redirect: 0
> rx26_lro_packets: 0
> rx26_lro_bytes: 0
> rx26_ecn_mark: 0
> rx26_removed_vlan_packets: 4496003688
> rx26_wqe_err: 0
> rx26_mpwqe_filler_cqes: 0
> rx26_mpwqe_filler_strides: 0
> rx26_buff_alloc_err: 0
> rx26_cqe_compress_blks: 0
> rx26_cqe_compress_pkts: 0
> rx26_page_reuse: 0
> rx26_cache_reuse: 8
> rx26_cache_full: 2248001581
> rx26_cache_empty: 6656
> rx26_cache_busy: 2248001836
> rx26_cache_waive: 0
> rx26_congst_umr: 0
> rx26_arfs_err: 0
> rx26_xdp_tx_xmit: 0
> rx26_xdp_tx_full: 0
> rx26_xdp_tx_err: 0
> rx26_xdp_tx_cqes: 0
> rx27_packets: 4341849333
> rx27_bytes: 5678653545018
> rx27_csum_complete: 4341849333
> rx27_csum_unnecessary: 0
> rx27_csum_unnecessary_inner: 0
> rx27_csum_none: 0
> rx27_xdp_drop: 0
> rx27_xdp_redirect: 0
> rx27_lro_packets: 0
> rx27_lro_bytes: 0
> rx27_ecn_mark: 0
> rx27_removed_vlan_packets: 4341849333
> rx27_wqe_err: 0
> rx27_mpwqe_filler_cqes: 0
> rx27_mpwqe_filler_strides: 0
> rx27_buff_alloc_err: 0
> rx27_cqe_compress_blks: 0
> rx27_cqe_compress_pkts: 0
> rx27_page_reuse: 0
> rx27_cache_reuse: 1748188
> rx27_cache_full: 2169176223
> rx27_cache_empty: 6656
> rx27_cache_busy: 2169176476
> rx27_cache_waive: 0
> rx27_congst_umr: 0
> rx27_arfs_err: 0
> rx27_xdp_tx_xmit: 0
> rx27_xdp_tx_full: 0
> rx27_xdp_tx_err: 0
> rx27_xdp_tx_cqes: 0
> rx28_packets: 0
> rx28_bytes: 0
> rx28_csum_complete: 0
> rx28_csum_unnecessary: 0
> rx28_csum_unnecessary_inner: 0
> rx28_csum_none: 0
> rx28_xdp_drop: 0
> rx28_xdp_redirect: 0
> rx28_lro_packets: 0
> rx28_lro_bytes: 0
> rx28_ecn_mark: 0
> rx28_removed_vlan_packets: 0
> rx28_wqe_err: 0
> rx28_mpwqe_filler_cqes: 0
> rx28_mpwqe_filler_strides: 0
> rx28_buff_alloc_err: 0
> rx28_cqe_compress_blks: 0
> rx28_cqe_compress_pkts: 0
> rx28_page_reuse: 0
> rx28_cache_reuse: 0
> rx28_cache_full: 0
> rx28_cache_empty: 2560
> rx28_cache_busy: 0
> rx28_cache_waive: 0
> rx28_congst_umr: 0
> rx28_arfs_err: 0
> rx28_xdp_tx_xmit: 0
> rx28_xdp_tx_full: 0
> rx28_xdp_tx_err: 0
> rx28_xdp_tx_cqes: 0
> rx29_packets: 0
> rx29_bytes: 0
> rx29_csum_complete: 0
> rx29_csum_unnecessary: 0
> rx29_csum_unnecessary_inner: 0
> rx29_csum_none: 0
> rx29_xdp_drop: 0
> rx29_xdp_redirect: 0
> rx29_lro_packets: 0
> rx29_lro_bytes: 0
> rx29_ecn_mark: 0
> rx29_removed_vlan_packets: 0
> rx29_wqe_err: 0
> rx29_mpwqe_filler_cqes: 0
> rx29_mpwqe_filler_strides: 0
> rx29_buff_alloc_err: 0
> rx29_cqe_compress_blks: 0
> rx29_cqe_compress_pkts: 0
> rx29_page_reuse: 0
> rx29_cache_reuse: 0
> rx29_cache_full: 0
> rx29_cache_empty: 2560
> rx29_cache_busy: 0
> rx29_cache_waive: 0
> rx29_congst_umr: 0
> rx29_arfs_err: 0
> rx29_xdp_tx_xmit: 0
> rx29_xdp_tx_full: 0
> rx29_xdp_tx_err: 0
> rx29_xdp_tx_cqes: 0
> rx30_packets: 0
> rx30_bytes: 0
> rx30_csum_complete: 0
> rx30_csum_unnecessary: 0
> rx30_csum_unnecessary_inner: 0
> rx30_csum_none: 0
> rx30_xdp_drop: 0
> rx30_xdp_redirect: 0
> rx30_lro_packets: 0
> rx30_lro_bytes: 0
> rx30_ecn_mark: 0
> rx30_removed_vlan_packets: 0
> rx30_wqe_err: 0
> rx30_mpwqe_filler_cqes: 0
> rx30_mpwqe_filler_strides: 0
> rx30_buff_alloc_err: 0
> rx30_cqe_compress_blks: 0
> rx30_cqe_compress_pkts: 0
> rx30_page_reuse: 0
> rx30_cache_reuse: 0
> rx30_cache_full: 0
> rx30_cache_empty: 2560
> rx30_cache_busy: 0
> rx30_cache_waive: 0
> rx30_congst_umr: 0
> rx30_arfs_err: 0
> rx30_xdp_tx_xmit: 0
> rx30_xdp_tx_full: 0
> rx30_xdp_tx_err: 0
> rx30_xdp_tx_cqes: 0
> rx31_packets: 0
> rx31_bytes: 0
> rx31_csum_complete: 0
> rx31_csum_unnecessary: 0
> rx31_csum_unnecessary_inner: 0
> rx31_csum_none: 0
> rx31_xdp_drop: 0
> rx31_xdp_redirect: 0
> rx31_lro_packets: 0
> rx31_lro_bytes: 0
> rx31_ecn_mark: 0
> rx31_removed_vlan_packets: 0
> rx31_wqe_err: 0
> rx31_mpwqe_filler_cqes: 0
> rx31_mpwqe_filler_strides: 0
> rx31_buff_alloc_err: 0
> rx31_cqe_compress_blks: 0
> rx31_cqe_compress_pkts: 0
> rx31_page_reuse: 0
> rx31_cache_reuse: 0
> rx31_cache_full: 0
> rx31_cache_empty: 2560
> rx31_cache_busy: 0
> rx31_cache_waive: 0
> rx31_congst_umr: 0
> rx31_arfs_err: 0
> rx31_xdp_tx_xmit: 0
> rx31_xdp_tx_full: 0
> rx31_xdp_tx_err: 0
> rx31_xdp_tx_cqes: 0
> rx32_packets: 0
> rx32_bytes: 0
> rx32_csum_complete: 0
> rx32_csum_unnecessary: 0
> rx32_csum_unnecessary_inner: 0
> rx32_csum_none: 0
> rx32_xdp_drop: 0
> rx32_xdp_redirect: 0
> rx32_lro_packets: 0
> rx32_lro_bytes: 0
> rx32_ecn_mark: 0
> rx32_removed_vlan_packets: 0
> rx32_wqe_err: 0
> rx32_mpwqe_filler_cqes: 0
> rx32_mpwqe_filler_strides: 0
> rx32_buff_alloc_err: 0
> rx32_cqe_compress_blks: 0
> rx32_cqe_compress_pkts: 0
> rx32_page_reuse: 0
> rx32_cache_reuse: 0
> rx32_cache_full: 0
> rx32_cache_empty: 2560
> rx32_cache_busy: 0
> rx32_cache_waive: 0
> rx32_congst_umr: 0
> rx32_arfs_err: 0
> rx32_xdp_tx_xmit: 0
> rx32_xdp_tx_full: 0
> rx32_xdp_tx_err: 0
> rx32_xdp_tx_cqes: 0
> rx33_packets: 0
> rx33_bytes: 0
> rx33_csum_complete: 0
> rx33_csum_unnecessary: 0
> rx33_csum_unnecessary_inner: 0
> rx33_csum_none: 0
> rx33_xdp_drop: 0
> rx33_xdp_redirect: 0
> rx33_lro_packets: 0
> rx33_lro_bytes: 0
> rx33_ecn_mark: 0
> rx33_removed_vlan_packets: 0
> rx33_wqe_err: 0
> rx33_mpwqe_filler_cqes: 0
> rx33_mpwqe_filler_strides: 0
> rx33_buff_alloc_err: 0
> rx33_cqe_compress_blks: 0
> rx33_cqe_compress_pkts: 0
> rx33_page_reuse: 0
> rx33_cache_reuse: 0
> rx33_cache_full: 0
> rx33_cache_empty: 2560
> rx33_cache_busy: 0
> rx33_cache_waive: 0
> rx33_congst_umr: 0
> rx33_arfs_err: 0
> rx33_xdp_tx_xmit: 0
> rx33_xdp_tx_full: 0
> rx33_xdp_tx_err: 0
> rx33_xdp_tx_cqes: 0
> rx34_packets: 0
> rx34_bytes: 0
> rx34_csum_complete: 0
> rx34_csum_unnecessary: 0
> rx34_csum_unnecessary_inner: 0
> rx34_csum_none: 0
> rx34_xdp_drop: 0
> rx34_xdp_redirect: 0
> rx34_lro_packets: 0
> rx34_lro_bytes: 0
> rx34_ecn_mark: 0
> rx34_removed_vlan_packets: 0
> rx34_wqe_err: 0
> rx34_mpwqe_filler_cqes: 0
> rx34_mpwqe_filler_strides: 0
> rx34_buff_alloc_err: 0
> rx34_cqe_compress_blks: 0
> rx34_cqe_compress_pkts: 0
> rx34_page_reuse: 0
> rx34_cache_reuse: 0
> rx34_cache_full: 0
> rx34_cache_empty: 2560
> rx34_cache_busy: 0
> rx34_cache_waive: 0
> rx34_congst_umr: 0
> rx34_arfs_err: 0
> rx34_xdp_tx_xmit: 0
> rx34_xdp_tx_full: 0
> rx34_xdp_tx_err: 0
> rx34_xdp_tx_cqes: 0
> rx35_packets: 0
> rx35_bytes: 0
> rx35_csum_complete: 0
> rx35_csum_unnecessary: 0
> rx35_csum_unnecessary_inner: 0
> rx35_csum_none: 0
> rx35_xdp_drop: 0
> rx35_xdp_redirect: 0
> rx35_lro_packets: 0
> rx35_lro_bytes: 0
> rx35_ecn_mark: 0
> rx35_removed_vlan_packets: 0
> rx35_wqe_err: 0
> rx35_mpwqe_filler_cqes: 0
> rx35_mpwqe_filler_strides: 0
> rx35_buff_alloc_err: 0
> rx35_cqe_compress_blks: 0
> rx35_cqe_compress_pkts: 0
> rx35_page_reuse: 0
> rx35_cache_reuse: 0
> rx35_cache_full: 0
> rx35_cache_empty: 2560
> rx35_cache_busy: 0
> rx35_cache_waive: 0
> rx35_congst_umr: 0
> rx35_arfs_err: 0
> rx35_xdp_tx_xmit: 0
> rx35_xdp_tx_full: 0
> rx35_xdp_tx_err: 0
> rx35_xdp_tx_cqes: 0
> rx36_packets: 0
> rx36_bytes: 0
> rx36_csum_complete: 0
> rx36_csum_unnecessary: 0
> rx36_csum_unnecessary_inner: 0
> rx36_csum_none: 0
> rx36_xdp_drop: 0
> rx36_xdp_redirect: 0
> rx36_lro_packets: 0
> rx36_lro_bytes: 0
> rx36_ecn_mark: 0
> rx36_removed_vlan_packets: 0
> rx36_wqe_err: 0
> rx36_mpwqe_filler_cqes: 0
> rx36_mpwqe_filler_strides: 0
> rx36_buff_alloc_err: 0
> rx36_cqe_compress_blks: 0
> rx36_cqe_compress_pkts: 0
> rx36_page_reuse: 0
> rx36_cache_reuse: 0
> rx36_cache_full: 0
> rx36_cache_empty: 2560
> rx36_cache_busy: 0
> rx36_cache_waive: 0
> rx36_congst_umr: 0
> rx36_arfs_err: 0
> rx36_xdp_tx_xmit: 0
> rx36_xdp_tx_full: 0
> rx36_xdp_tx_err: 0
> rx36_xdp_tx_cqes: 0
> rx37_packets: 0
> rx37_bytes: 0
> rx37_csum_complete: 0
> rx37_csum_unnecessary: 0
> rx37_csum_unnecessary_inner: 0
> rx37_csum_none: 0
> rx37_xdp_drop: 0
> rx37_xdp_redirect: 0
> rx37_lro_packets: 0
> rx37_lro_bytes: 0
> rx37_ecn_mark: 0
> rx37_removed_vlan_packets: 0
> rx37_wqe_err: 0
> rx37_mpwqe_filler_cqes: 0
> rx37_mpwqe_filler_strides: 0
> rx37_buff_alloc_err: 0
> rx37_cqe_compress_blks: 0
> rx37_cqe_compress_pkts: 0
> rx37_page_reuse: 0
> rx37_cache_reuse: 0
> rx37_cache_full: 0
> rx37_cache_empty: 2560
> rx37_cache_busy: 0
> rx37_cache_waive: 0
> rx37_congst_umr: 0
> rx37_arfs_err: 0
> rx37_xdp_tx_xmit: 0
> rx37_xdp_tx_full: 0
> rx37_xdp_tx_err: 0
> rx37_xdp_tx_cqes: 0
> rx38_packets: 0
> rx38_bytes: 0
> rx38_csum_complete: 0
> rx38_csum_unnecessary: 0
> rx38_csum_unnecessary_inner: 0
> rx38_csum_none: 0
> rx38_xdp_drop: 0
> rx38_xdp_redirect: 0
> rx38_lro_packets: 0
> rx38_lro_bytes: 0
> rx38_ecn_mark: 0
> rx38_removed_vlan_packets: 0
> rx38_wqe_err: 0
> rx38_mpwqe_filler_cqes: 0
> rx38_mpwqe_filler_strides: 0
> rx38_buff_alloc_err: 0
> rx38_cqe_compress_blks: 0
> rx38_cqe_compress_pkts: 0
> rx38_page_reuse: 0
> rx38_cache_reuse: 0
> rx38_cache_full: 0
> rx38_cache_empty: 2560
> rx38_cache_busy: 0
> rx38_cache_waive: 0
> rx38_congst_umr: 0
> rx38_arfs_err: 0
> rx38_xdp_tx_xmit: 0
> rx38_xdp_tx_full: 0
> rx38_xdp_tx_err: 0
> rx38_xdp_tx_cqes: 0
> rx39_packets: 0
> rx39_bytes: 0
> rx39_csum_complete: 0
> rx39_csum_unnecessary: 0
> rx39_csum_unnecessary_inner: 0
> rx39_csum_none: 0
> rx39_xdp_drop: 0
> rx39_xdp_redirect: 0
> rx39_lro_packets: 0
> rx39_lro_bytes: 0
> rx39_ecn_mark: 0
> rx39_removed_vlan_packets: 0
> rx39_wqe_err: 0
> rx39_mpwqe_filler_cqes: 0
> rx39_mpwqe_filler_strides: 0
> rx39_buff_alloc_err: 0
> rx39_cqe_compress_blks: 0
> rx39_cqe_compress_pkts: 0
> rx39_page_reuse: 0
> rx39_cache_reuse: 0
> rx39_cache_full: 0
> rx39_cache_empty: 2560
> rx39_cache_busy: 0
> rx39_cache_waive: 0
> rx39_congst_umr: 0
> rx39_arfs_err: 0
> rx39_xdp_tx_xmit: 0
> rx39_xdp_tx_full: 0
> rx39_xdp_tx_err: 0
> rx39_xdp_tx_cqes: 0
> rx40_packets: 0
> rx40_bytes: 0
> rx40_csum_complete: 0
> rx40_csum_unnecessary: 0
> rx40_csum_unnecessary_inner: 0
> rx40_csum_none: 0
> rx40_xdp_drop: 0
> rx40_xdp_redirect: 0
> rx40_lro_packets: 0
> rx40_lro_bytes: 0
> rx40_ecn_mark: 0
> rx40_removed_vlan_packets: 0
> rx40_wqe_err: 0
> rx40_mpwqe_filler_cqes: 0
> rx40_mpwqe_filler_strides: 0
> rx40_buff_alloc_err: 0
> rx40_cqe_compress_blks: 0
> rx40_cqe_compress_pkts: 0
> rx40_page_reuse: 0
> rx40_cache_reuse: 0
> rx40_cache_full: 0
> rx40_cache_empty: 2560
> rx40_cache_busy: 0
> rx40_cache_waive: 0
> rx40_congst_umr: 0
> rx40_arfs_err: 0
> rx40_xdp_tx_xmit: 0
> rx40_xdp_tx_full: 0
> rx40_xdp_tx_err: 0
> rx40_xdp_tx_cqes: 0
> rx41_packets: 0
> rx41_bytes: 0
> rx41_csum_complete: 0
> rx41_csum_unnecessary: 0
> rx41_csum_unnecessary_inner: 0
> rx41_csum_none: 0
> rx41_xdp_drop: 0
> rx41_xdp_redirect: 0
> rx41_lro_packets: 0
> rx41_lro_bytes: 0
> rx41_ecn_mark: 0
> rx41_removed_vlan_packets: 0
> rx41_wqe_err: 0
> rx41_mpwqe_filler_cqes: 0
> rx41_mpwqe_filler_strides: 0
> rx41_buff_alloc_err: 0
> rx41_cqe_compress_blks: 0
> rx41_cqe_compress_pkts: 0
> rx41_page_reuse: 0
> rx41_cache_reuse: 0
> rx41_cache_full: 0
> rx41_cache_empty: 2560
> rx41_cache_busy: 0
> rx41_cache_waive: 0
> rx41_congst_umr: 0
> rx41_arfs_err: 0
> rx41_xdp_tx_xmit: 0
> rx41_xdp_tx_full: 0
> rx41_xdp_tx_err: 0
> rx41_xdp_tx_cqes: 0
> rx42_packets: 0
> rx42_bytes: 0
> rx42_csum_complete: 0
> rx42_csum_unnecessary: 0
> rx42_csum_unnecessary_inner: 0
> rx42_csum_none: 0
> rx42_xdp_drop: 0
> rx42_xdp_redirect: 0
> rx42_lro_packets: 0
> rx42_lro_bytes: 0
> rx42_ecn_mark: 0
> rx42_removed_vlan_packets: 0
> rx42_wqe_err: 0
> rx42_mpwqe_filler_cqes: 0
> rx42_mpwqe_filler_strides: 0
> rx42_buff_alloc_err: 0
> rx42_cqe_compress_blks: 0
> rx42_cqe_compress_pkts: 0
> rx42_page_reuse: 0
> rx42_cache_reuse: 0
> rx42_cache_full: 0
> rx42_cache_empty: 2560
> rx42_cache_busy: 0
> rx42_cache_waive: 0
> rx42_congst_umr: 0
> rx42_arfs_err: 0
> rx42_xdp_tx_xmit: 0
> rx42_xdp_tx_full: 0
> rx42_xdp_tx_err: 0
> rx42_xdp_tx_cqes: 0
> rx43_packets: 0
> rx43_bytes: 0
> rx43_csum_complete: 0
> rx43_csum_unnecessary: 0
> rx43_csum_unnecessary_inner: 0
> rx43_csum_none: 0
> rx43_xdp_drop: 0
> rx43_xdp_redirect: 0
> rx43_lro_packets: 0
> rx43_lro_bytes: 0
> rx43_ecn_mark: 0
> rx43_removed_vlan_packets: 0
> rx43_wqe_err: 0
> rx43_mpwqe_filler_cqes: 0
> rx43_mpwqe_filler_strides: 0
> rx43_buff_alloc_err: 0
> rx43_cqe_compress_blks: 0
> rx43_cqe_compress_pkts: 0
> rx43_page_reuse: 0
> rx43_cache_reuse: 0
> rx43_cache_full: 0
> rx43_cache_empty: 2560
> rx43_cache_busy: 0
> rx43_cache_waive: 0
> rx43_congst_umr: 0
> rx43_arfs_err: 0
> rx43_xdp_tx_xmit: 0
> rx43_xdp_tx_full: 0
> rx43_xdp_tx_err: 0
> rx43_xdp_tx_cqes: 0
> rx44_packets: 0
> rx44_bytes: 0
> rx44_csum_complete: 0
> rx44_csum_unnecessary: 0
> rx44_csum_unnecessary_inner: 0
> rx44_csum_none: 0
> rx44_xdp_drop: 0
> rx44_xdp_redirect: 0
> rx44_lro_packets: 0
> rx44_lro_bytes: 0
> rx44_ecn_mark: 0
> rx44_removed_vlan_packets: 0
> rx44_wqe_err: 0
> rx44_mpwqe_filler_cqes: 0
> rx44_mpwqe_filler_strides: 0
> rx44_buff_alloc_err: 0
> rx44_cqe_compress_blks: 0
> rx44_cqe_compress_pkts: 0
> rx44_page_reuse: 0
> rx44_cache_reuse: 0
> rx44_cache_full: 0
> rx44_cache_empty: 2560
> rx44_cache_busy: 0
> rx44_cache_waive: 0
> rx44_congst_umr: 0
> rx44_arfs_err: 0
> rx44_xdp_tx_xmit: 0
> rx44_xdp_tx_full: 0
> rx44_xdp_tx_err: 0
> rx44_xdp_tx_cqes: 0
> rx45_packets: 0
> rx45_bytes: 0
> rx45_csum_complete: 0
> rx45_csum_unnecessary: 0
> rx45_csum_unnecessary_inner: 0
> rx45_csum_none: 0
> rx45_xdp_drop: 0
> rx45_xdp_redirect: 0
> rx45_lro_packets: 0
> rx45_lro_bytes: 0
> rx45_ecn_mark: 0
> rx45_removed_vlan_packets: 0
> rx45_wqe_err: 0
> rx45_mpwqe_filler_cqes: 0
> rx45_mpwqe_filler_strides: 0
> rx45_buff_alloc_err: 0
> rx45_cqe_compress_blks: 0
> rx45_cqe_compress_pkts: 0
> rx45_page_reuse: 0
> rx45_cache_reuse: 0
> rx45_cache_full: 0
> rx45_cache_empty: 2560
> rx45_cache_busy: 0
> rx45_cache_waive: 0
> rx45_congst_umr: 0
> rx45_arfs_err: 0
> rx45_xdp_tx_xmit: 0
> rx45_xdp_tx_full: 0
> rx45_xdp_tx_err: 0
> rx45_xdp_tx_cqes: 0
> rx46_packets: 0
> rx46_bytes: 0
> rx46_csum_complete: 0
> rx46_csum_unnecessary: 0
> rx46_csum_unnecessary_inner: 0
> rx46_csum_none: 0
> rx46_xdp_drop: 0
> rx46_xdp_redirect: 0
> rx46_lro_packets: 0
> rx46_lro_bytes: 0
> rx46_ecn_mark: 0
> rx46_removed_vlan_packets: 0
> rx46_wqe_err: 0
> rx46_mpwqe_filler_cqes: 0
> rx46_mpwqe_filler_strides: 0
> rx46_buff_alloc_err: 0
> rx46_cqe_compress_blks: 0
> rx46_cqe_compress_pkts: 0
> rx46_page_reuse: 0
> rx46_cache_reuse: 0
> rx46_cache_full: 0
> rx46_cache_empty: 2560
> rx46_cache_busy: 0
> rx46_cache_waive: 0
> rx46_congst_umr: 0
> rx46_arfs_err: 0
> rx46_xdp_tx_xmit: 0
> rx46_xdp_tx_full: 0
> rx46_xdp_tx_err: 0
> rx46_xdp_tx_cqes: 0
> rx47_packets: 0
> rx47_bytes: 0
> rx47_csum_complete: 0
> rx47_csum_unnecessary: 0
> rx47_csum_unnecessary_inner: 0
> rx47_csum_none: 0
> rx47_xdp_drop: 0
> rx47_xdp_redirect: 0
> rx47_lro_packets: 0
> rx47_lro_bytes: 0
> rx47_ecn_mark: 0
> rx47_removed_vlan_packets: 0
> rx47_wqe_err: 0
> rx47_mpwqe_filler_cqes: 0
> rx47_mpwqe_filler_strides: 0
> rx47_buff_alloc_err: 0
> rx47_cqe_compress_blks: 0
> rx47_cqe_compress_pkts: 0
> rx47_page_reuse: 0
> rx47_cache_reuse: 0
> rx47_cache_full: 0
> rx47_cache_empty: 2560
> rx47_cache_busy: 0
> rx47_cache_waive: 0
> rx47_congst_umr: 0
> rx47_arfs_err: 0
> rx47_xdp_tx_xmit: 0
> rx47_xdp_tx_full: 0
> rx47_xdp_tx_err: 0
> rx47_xdp_tx_cqes: 0
> rx48_packets: 0
> rx48_bytes: 0
> rx48_csum_complete: 0
> rx48_csum_unnecessary: 0
> rx48_csum_unnecessary_inner: 0
> rx48_csum_none: 0
> rx48_xdp_drop: 0
> rx48_xdp_redirect: 0
> rx48_lro_packets: 0
> rx48_lro_bytes: 0
> rx48_ecn_mark: 0
> rx48_removed_vlan_packets: 0
> rx48_wqe_err: 0
> rx48_mpwqe_filler_cqes: 0
> rx48_mpwqe_filler_strides: 0
> rx48_buff_alloc_err: 0
> rx48_cqe_compress_blks: 0
> rx48_cqe_compress_pkts: 0
> rx48_page_reuse: 0
> rx48_cache_reuse: 0
> rx48_cache_full: 0
> rx48_cache_empty: 2560
> rx48_cache_busy: 0
> rx48_cache_waive: 0
> rx48_congst_umr: 0
> rx48_arfs_err: 0
> rx48_xdp_tx_xmit: 0
> rx48_xdp_tx_full: 0
> rx48_xdp_tx_err: 0
> rx48_xdp_tx_cqes: 0
> rx49_packets: 0
> rx49_bytes: 0
> rx49_csum_complete: 0
> rx49_csum_unnecessary: 0
> rx49_csum_unnecessary_inner: 0
> rx49_csum_none: 0
> rx49_xdp_drop: 0
> rx49_xdp_redirect: 0
> rx49_lro_packets: 0
> rx49_lro_bytes: 0
> rx49_ecn_mark: 0
> rx49_removed_vlan_packets: 0
> rx49_wqe_err: 0
> rx49_mpwqe_filler_cqes: 0
> rx49_mpwqe_filler_strides: 0
> rx49_buff_alloc_err: 0
> rx49_cqe_compress_blks: 0
> rx49_cqe_compress_pkts: 0
> rx49_page_reuse: 0
> rx49_cache_reuse: 0
> rx49_cache_full: 0
> rx49_cache_empty: 2560
> rx49_cache_busy: 0
> rx49_cache_waive: 0
> rx49_congst_umr: 0
> rx49_arfs_err: 0
> rx49_xdp_tx_xmit: 0
> rx49_xdp_tx_full: 0
> rx49_xdp_tx_err: 0
> rx49_xdp_tx_cqes: 0
> rx50_packets: 0
> rx50_bytes: 0
> rx50_csum_complete: 0
> rx50_csum_unnecessary: 0
> rx50_csum_unnecessary_inner: 0
> rx50_csum_none: 0
> rx50_xdp_drop: 0
> rx50_xdp_redirect: 0
> rx50_lro_packets: 0
> rx50_lro_bytes: 0
> rx50_ecn_mark: 0
> rx50_removed_vlan_packets: 0
> rx50_wqe_err: 0
> rx50_mpwqe_filler_cqes: 0
> rx50_mpwqe_filler_strides: 0
> rx50_buff_alloc_err: 0
> rx50_cqe_compress_blks: 0
> rx50_cqe_compress_pkts: 0
> rx50_page_reuse: 0
> rx50_cache_reuse: 0
> rx50_cache_full: 0
> rx50_cache_empty: 2560
> rx50_cache_busy: 0
> rx50_cache_waive: 0
> rx50_congst_umr: 0
> rx50_arfs_err: 0
> rx50_xdp_tx_xmit: 0
> rx50_xdp_tx_full: 0
> rx50_xdp_tx_err: 0
> rx50_xdp_tx_cqes: 0
> rx51_packets: 0
> rx51_bytes: 0
> rx51_csum_complete: 0
> rx51_csum_unnecessary: 0
> rx51_csum_unnecessary_inner: 0
> rx51_csum_none: 0
> rx51_xdp_drop: 0
> rx51_xdp_redirect: 0
> rx51_lro_packets: 0
> rx51_lro_bytes: 0
> rx51_ecn_mark: 0
> rx51_removed_vlan_packets: 0
> rx51_wqe_err: 0
> rx51_mpwqe_filler_cqes: 0
> rx51_mpwqe_filler_strides: 0
> rx51_buff_alloc_err: 0
> rx51_cqe_compress_blks: 0
> rx51_cqe_compress_pkts: 0
> rx51_page_reuse: 0
> rx51_cache_reuse: 0
> rx51_cache_full: 0
> rx51_cache_empty: 2560
> rx51_cache_busy: 0
> rx51_cache_waive: 0
> rx51_congst_umr: 0
> rx51_arfs_err: 0
> rx51_xdp_tx_xmit: 0
> rx51_xdp_tx_full: 0
> rx51_xdp_tx_err: 0
> rx51_xdp_tx_cqes: 0
> rx52_packets: 0
> rx52_bytes: 0
> rx52_csum_complete: 0
> rx52_csum_unnecessary: 0
> rx52_csum_unnecessary_inner: 0
> rx52_csum_none: 0
> rx52_xdp_drop: 0
> rx52_xdp_redirect: 0
> rx52_lro_packets: 0
> rx52_lro_bytes: 0
> rx52_ecn_mark: 0
> rx52_removed_vlan_packets: 0
> rx52_wqe_err: 0
> rx52_mpwqe_filler_cqes: 0
> rx52_mpwqe_filler_strides: 0
> rx52_buff_alloc_err: 0
> rx52_cqe_compress_blks: 0
> rx52_cqe_compress_pkts: 0
> rx52_page_reuse: 0
> rx52_cache_reuse: 0
> rx52_cache_full: 0
> rx52_cache_empty: 2560
> rx52_cache_busy: 0
> rx52_cache_waive: 0
> rx52_congst_umr: 0
> rx52_arfs_err: 0
> rx52_xdp_tx_xmit: 0
> rx52_xdp_tx_full: 0
> rx52_xdp_tx_err: 0
> rx52_xdp_tx_cqes: 0
> rx53_packets: 0
> rx53_bytes: 0
> rx53_csum_complete: 0
> rx53_csum_unnecessary: 0
> rx53_csum_unnecessary_inner: 0
> rx53_csum_none: 0
> rx53_xdp_drop: 0
> rx53_xdp_redirect: 0
> rx53_lro_packets: 0
> rx53_lro_bytes: 0
> rx53_ecn_mark: 0
> rx53_removed_vlan_packets: 0
> rx53_wqe_err: 0
> rx53_mpwqe_filler_cqes: 0
> rx53_mpwqe_filler_strides: 0
> rx53_buff_alloc_err: 0
> rx53_cqe_compress_blks: 0
> rx53_cqe_compress_pkts: 0
> rx53_page_reuse: 0
> rx53_cache_reuse: 0
> rx53_cache_full: 0
> rx53_cache_empty: 2560
> rx53_cache_busy: 0
> rx53_cache_waive: 0
> rx53_congst_umr: 0
> rx53_arfs_err: 0
> rx53_xdp_tx_xmit: 0
> rx53_xdp_tx_full: 0
> rx53_xdp_tx_err: 0
> rx53_xdp_tx_cqes: 0
> rx54_packets: 0
> rx54_bytes: 0
> rx54_csum_complete: 0
> rx54_csum_unnecessary: 0
> rx54_csum_unnecessary_inner: 0
> rx54_csum_none: 0
> rx54_xdp_drop: 0
> rx54_xdp_redirect: 0
> rx54_lro_packets: 0
> rx54_lro_bytes: 0
> rx54_ecn_mark: 0
> rx54_removed_vlan_packets: 0
> rx54_wqe_err: 0
> rx54_mpwqe_filler_cqes: 0
> rx54_mpwqe_filler_strides: 0
> rx54_buff_alloc_err: 0
> rx54_cqe_compress_blks: 0
> rx54_cqe_compress_pkts: 0
> rx54_page_reuse: 0
> rx54_cache_reuse: 0
> rx54_cache_full: 0
> rx54_cache_empty: 2560
> rx54_cache_busy: 0
> rx54_cache_waive: 0
> rx54_congst_umr: 0
> rx54_arfs_err: 0
> rx54_xdp_tx_xmit: 0
> rx54_xdp_tx_full: 0
> rx54_xdp_tx_err: 0
> rx54_xdp_tx_cqes: 0
> rx55_packets: 0
> rx55_bytes: 0
> rx55_csum_complete: 0
> rx55_csum_unnecessary: 0
> rx55_csum_unnecessary_inner: 0
> rx55_csum_none: 0
> rx55_xdp_drop: 0
> rx55_xdp_redirect: 0
> rx55_lro_packets: 0
> rx55_lro_bytes: 0
> rx55_ecn_mark: 0
> rx55_removed_vlan_packets: 0
> rx55_wqe_err: 0
> rx55_mpwqe_filler_cqes: 0
> rx55_mpwqe_filler_strides: 0
> rx55_buff_alloc_err: 0
> rx55_cqe_compress_blks: 0
> rx55_cqe_compress_pkts: 0
> rx55_page_reuse: 0
> rx55_cache_reuse: 0
> rx55_cache_full: 0
> rx55_cache_empty: 2560
> rx55_cache_busy: 0
> rx55_cache_waive: 0
> rx55_congst_umr: 0
> rx55_arfs_err: 0
> rx55_xdp_tx_xmit: 0
> rx55_xdp_tx_full: 0
> rx55_xdp_tx_err: 0
> rx55_xdp_tx_cqes: 0
> tx0_packets: 6019477917
> tx0_bytes: 3445238940825
> tx0_tso_packets: 311304622
> tx0_tso_bytes: 1897094773213
> tx0_tso_inner_packets: 0
> tx0_tso_inner_bytes: 0
> tx0_csum_partial: 457981794
> tx0_csum_partial_inner: 0
> tx0_added_vlan_packets: 4965567654
> tx0_nop: 72290329
> tx0_csum_none: 4507585860
> tx0_stopped: 9118
> tx0_dropped: 0
> tx0_xmit_more: 51651593
> tx0_recover: 0
> tx0_cqes: 4913918402
> tx0_wake: 9118
> tx0_cqe_err: 0
> tx1_packets: 5700413414
> tx1_bytes: 3340870662350
> tx1_tso_packets: 318201557
> tx1_tso_bytes: 1915233462303
> tx1_tso_inner_packets: 0
> tx1_tso_inner_bytes: 0
> tx1_csum_partial: 461736722
> tx1_csum_partial_inner: 0
> tx1_added_vlan_packets: 4638708749
> tx1_nop: 70061796
> tx1_csum_none: 4176972027
> tx1_stopped: 9248
> tx1_dropped: 0
> tx1_xmit_more: 39531959
> tx1_recover: 0
> tx1_cqes: 4599179178
> tx1_wake: 9248
> tx1_cqe_err: 0
> tx2_packets: 5795960848
> tx2_bytes: 3394876820271
> tx2_tso_packets: 322935065
> tx2_tso_bytes: 1910825901109
> tx2_tso_inner_packets: 0
> tx2_tso_inner_bytes: 0
> tx2_csum_partial: 460747092
> tx2_csum_partial_inner: 0
> tx2_added_vlan_packets: 4743705654
> tx2_nop: 72722430
> tx2_csum_none: 4282958562
> tx2_stopped: 8938
> tx2_dropped: 0
> tx2_xmit_more: 44084718
> tx2_recover: 0
> tx2_cqes: 4699623410
> tx2_wake: 8938
> tx2_cqe_err: 0
> tx3_packets: 5580215878
> tx3_bytes: 3191677257787
> tx3_tso_packets: 305771141
> tx3_tso_bytes: 1823265793476
> tx3_tso_inner_packets: 0
> tx3_tso_inner_bytes: 0
> tx3_csum_partial: 434976070
> tx3_csum_partial_inner: 0
> tx3_added_vlan_packets: 4569899956
> tx3_nop: 68184348
> tx3_csum_none: 4134923886
> tx3_stopped: 8383
> tx3_dropped: 0
> tx3_xmit_more: 41940375
> tx3_recover: 0
> tx3_cqes: 4527961924
> tx3_wake: 8383
> tx3_cqe_err: 0
> tx4_packets: 6795007068
> tx4_bytes: 3963890025270
> tx4_tso_packets: 358437617
> tx4_tso_bytes: 2154747995355
> tx4_tso_inner_packets: 0
> tx4_tso_inner_bytes: 0
> tx4_csum_partial: 504764524
> tx4_csum_partial_inner: 0
> tx4_added_vlan_packets: 5602510191
> tx4_nop: 81345604
> tx4_csum_none: 5097745667
> tx4_stopped: 10248
> tx4_dropped: 0
> tx4_xmit_more: 49068571
> tx4_recover: 0
> tx4_cqes: 5553444276
> tx4_wake: 10248
> tx4_cqe_err: 0
> tx5_packets: 6408089261
> tx5_bytes: 3676275848279
> tx5_tso_packets: 345129329
> tx5_tso_bytes: 2108447877473
> tx5_tso_inner_packets: 0
> tx5_tso_inner_bytes: 0
> tx5_csum_partial: 494705894
> tx5_csum_partial_inner: 0
> tx5_added_vlan_packets: 5235998343
> tx5_nop: 77694627
> tx5_csum_none: 4741292449
> tx5_stopped: 46
> tx5_dropped: 0
> tx5_xmit_more: 46675831
> tx5_recover: 0
> tx5_cqes: 5189323550
> tx5_wake: 46
> tx5_cqe_err: 0
> tx6_packets: 6382289663
> tx6_bytes: 3670991418150
> tx6_tso_packets: 342927826
> tx6_tso_bytes: 2075049679904
> tx6_tso_inner_packets: 0
> tx6_tso_inner_bytes: 0
> tx6_csum_partial: 490369221
> tx6_csum_partial_inner: 0
> tx6_added_vlan_packets: 5232144528
> tx6_nop: 77391246
> tx6_csum_none: 4741775307
> tx6_stopped: 10823
> tx6_dropped: 0
> tx6_xmit_more: 44487607
> tx6_recover: 0
> tx6_cqes: 5187659877
> tx6_wake: 10823
> tx6_cqe_err: 0
> tx7_packets: 6456378284
> tx7_bytes: 3758013320518
> tx7_tso_packets: 350958294
> tx7_tso_bytes: 2126833408524
> tx7_tso_inner_packets: 0
> tx7_tso_inner_bytes: 0
> tx7_csum_partial: 501804109
> tx7_csum_partial_inner: 0
> tx7_added_vlan_packets: 5275635204
> tx7_nop: 79010883
> tx7_csum_none: 4773831096
> tx7_stopped: 14684
> tx7_dropped: 0
> tx7_xmit_more: 44447469
> tx7_recover: 0
> tx7_cqes: 5231191770
> tx7_wake: 14684
> tx7_cqe_err: 0
> tx8_packets: 6401799768
> tx8_bytes: 3681210808766
> tx8_tso_packets: 342878228
> tx8_tso_bytes: 2089688012191
> tx8_tso_inner_packets: 0
> tx8_tso_inner_bytes: 0
> tx8_csum_partial: 494865145
> tx8_csum_partial_inner: 0
> tx8_added_vlan_packets: 5242288908
> tx8_nop: 77250910
> tx8_csum_none: 4747423763
> tx8_stopped: 2
> tx8_dropped: 0
> tx8_xmit_more: 44191737
> tx8_recover: 0
> tx8_cqes: 5198098454
> tx8_wake: 2
> tx8_cqe_err: 0
> tx9_packets: 6632882888
> tx9_bytes: 3820110338309
> tx9_tso_packets: 354189056
> tx9_tso_bytes: 2187883597128
> tx9_tso_inner_packets: 0
> tx9_tso_inner_bytes: 0
> tx9_csum_partial: 511108218
> tx9_csum_partial_inner: 0
> tx9_added_vlan_packets: 5413836353
> tx9_nop: 80560668
> tx9_csum_none: 4902728135
> tx9_stopped: 9091
> tx9_dropped: 0
> tx9_xmit_more: 54501293
> tx9_recover: 0
> tx9_cqes: 5359337638
> tx9_wake: 9091
> tx9_cqe_err: 0
> tx10_packets: 6421786406
> tx10_bytes: 3692798413429
> tx10_tso_packets: 346878943
> tx10_tso_bytes: 2111921062110
> tx10_tso_inner_packets: 0
> tx10_tso_inner_bytes: 0
> tx10_csum_partial: 494356645
> tx10_csum_partial_inner: 0
> tx10_added_vlan_packets: 5248274374
> tx10_nop: 77922624
> tx10_csum_none: 4753917730
> tx10_stopped: 9617
> tx10_dropped: 0
> tx10_xmit_more: 44473939
> tx10_recover: 0
> tx10_cqes: 5203802547
> tx10_wake: 9617
> tx10_cqe_err: 0
> tx11_packets: 6406750938
> tx11_bytes: 3660343565126
> tx11_tso_packets: 355917271
> tx11_tso_bytes: 2130812246956
> tx11_tso_inner_packets: 0
> tx11_tso_inner_bytes: 0
> tx11_csum_partial: 500336369
> tx11_csum_partial_inner: 0
> tx11_added_vlan_packets: 5228267547
> tx11_nop: 78906315
> tx11_csum_none: 4727931178
> tx11_stopped: 9607
> tx11_dropped: 0
> tx11_xmit_more: 40041492
> tx11_recover: 0
> tx11_cqes: 5188228290
> tx11_wake: 9607
> tx11_cqe_err: 0
> tx12_packets: 6422347846
> tx12_bytes: 3718772753227
> tx12_tso_packets: 355397223
> tx12_tso_bytes: 2162614059758
> tx12_tso_inner_packets: 0
> tx12_tso_inner_bytes: 0
> tx12_csum_partial: 511437844
> tx12_csum_partial_inner: 0
> tx12_added_vlan_packets: 5221373746
> tx12_nop: 78866779
> tx12_csum_none: 4709935902
> tx12_stopped: 10280
> tx12_dropped: 0
> tx12_xmit_more: 42189399
> tx12_recover: 0
> tx12_cqes: 5179187154
> tx12_wake: 10280
> tx12_cqe_err: 0
> tx13_packets: 6429383816
> tx13_bytes: 3725679445046
> tx13_tso_packets: 360934759
> tx13_tso_bytes: 2148016411436
> tx13_tso_inner_packets: 0
> tx13_tso_inner_bytes: 0
> tx13_csum_partial: 505245849
> tx13_csum_partial_inner: 0
> tx13_added_vlan_packets: 5240267441
> tx13_nop: 80295637
> tx13_csum_none: 4735021592
> tx13_stopped: 84
> tx13_dropped: 0
> tx13_xmit_more: 43118045
> tx13_recover: 0
> tx13_cqes: 5197150348
> tx13_wake: 84
> tx13_cqe_err: 0
> tx14_packets: 6375279148
> tx14_bytes: 3624267203336
> tx14_tso_packets: 344388148
> tx14_tso_bytes: 2094966273548
> tx14_tso_inner_packets: 0
> tx14_tso_inner_bytes: 0
> tx14_csum_partial: 494129407
> tx14_csum_partial_inner: 0
> tx14_added_vlan_packets: 5210749337
> tx14_nop: 77280615
> tx14_csum_none: 4716619930
> tx14_stopped: 13057
> tx14_dropped: 0
> tx14_xmit_more: 40849682
> tx14_recover: 0
> tx14_cqes: 5169902694
> tx14_wake: 13057
> tx14_cqe_err: 0
> tx15_packets: 6489306520
> tx15_bytes: 3775716194795
> tx15_tso_packets: 368716406
> tx15_tso_bytes: 2165876423354
> tx15_tso_inner_packets: 0
> tx15_tso_inner_bytes: 0
> tx15_csum_partial: 509887864
> tx15_csum_partial_inner: 0
> tx15_added_vlan_packets: 5296767390
> tx15_nop: 80803468
> tx15_csum_none: 4786879529
> tx15_stopped: 1
> tx15_dropped: 0
> tx15_xmit_more: 46979676
> tx15_recover: 0
> tx15_cqes: 5249789328
> tx15_wake: 1
> tx15_cqe_err: 0
> tx16_packets: 6559857761
> tx16_bytes: 3724080573905
> tx16_tso_packets: 350864176
> tx16_tso_bytes: 2099634006033
> tx16_tso_inner_packets: 0
> tx16_tso_inner_bytes: 0
> tx16_csum_partial: 489397232
> tx16_csum_partial_inner: 0
> tx16_added_vlan_packets: 5398869334
> tx16_nop: 79046075
> tx16_csum_none: 4909472106
> tx16_stopped: 4480
> tx16_dropped: 0
> tx16_xmit_more: 47273286
> tx16_recover: 0
> tx16_cqes: 5351598315
> tx16_wake: 4480
> tx16_cqe_err: 0
> tx17_packets: 6358711533
> tx17_bytes: 3650180865573
> tx17_tso_packets: 350723136
> tx17_tso_bytes: 2109426587128
> tx17_tso_inner_packets: 0
> tx17_tso_inner_bytes: 0
> tx17_csum_partial: 494719487
> tx17_csum_partial_inner: 0
> tx17_added_vlan_packets: 5190068796
> tx17_nop: 77285612
> tx17_csum_none: 4695349309
> tx17_stopped: 10443
> tx17_dropped: 0
> tx17_xmit_more: 45582108
> tx17_recover: 0
> tx17_cqes: 5144489363
> tx17_wake: 10443
> tx17_cqe_err: 0
> tx18_packets: 6655328437
> tx18_bytes: 3801768461807
> tx18_tso_packets: 356516373
> tx18_tso_bytes: 2164829247550
> tx18_tso_inner_packets: 0
> tx18_tso_inner_bytes: 0
> tx18_csum_partial: 500508446
> tx18_csum_partial_inner: 0
> tx18_added_vlan_packets: 5454166840
> tx18_nop: 80423007
> tx18_csum_none: 4953658394
> tx18_stopped: 14760
> tx18_dropped: 0
> tx18_xmit_more: 50837465
> tx18_recover: 0
> tx18_cqes: 5403332553
> tx18_wake: 14760
> tx18_cqe_err: 0
> tx19_packets: 6408680611
> tx19_bytes: 3644119934372
> tx19_tso_packets: 350727530
> tx19_tso_bytes: 2089896715365
> tx19_tso_inner_packets: 0
> tx19_tso_inner_bytes: 0
> tx19_csum_partial: 486536490
> tx19_csum_partial_inner: 0
> tx19_added_vlan_packets: 5255839020
> tx19_nop: 78525198
> tx19_csum_none: 4769302530
> tx19_stopped: 8614
> tx19_dropped: 0
> tx19_xmit_more: 43605232
> tx19_recover: 0
> tx19_cqes: 5212236833
> tx19_wake: 8614
> tx19_cqe_err: 0
> tx20_packets: 5609275141
> tx20_bytes: 3187279031581
> tx20_tso_packets: 298609303
> tx20_tso_bytes: 1794382229379
> tx20_tso_inner_packets: 0
> tx20_tso_inner_bytes: 0
> tx20_csum_partial: 430691178
> tx20_csum_partial_inner: 0
> tx20_added_vlan_packets: 4616844286
> tx20_nop: 67450040
> tx20_csum_none: 4186153108
> tx20_stopped: 9099
> tx20_dropped: 0
> tx20_xmit_more: 42040991
> tx20_recover: 0
> tx20_cqes: 4574805846
> tx20_wake: 9099
> tx20_cqe_err: 0
> tx21_packets: 5641621183
> tx21_bytes: 3279282331124
> tx21_tso_packets: 311297057
> tx21_tso_bytes: 1875735401012
> tx21_tso_inner_packets: 0
> tx21_tso_inner_bytes: 0
> tx21_csum_partial: 444333894
> tx21_csum_partial_inner: 0
> tx21_added_vlan_packets: 4603527701
> tx21_nop: 68857983
> tx21_csum_none: 4159193807
> tx21_stopped: 10082
> tx21_dropped: 0
> tx21_xmit_more: 43988081
> tx21_recover: 0
> tx21_cqes: 4559542410
> tx21_wake: 10082
> tx21_cqe_err: 0
> tx22_packets: 5822168288
> tx22_bytes: 3452026726862
> tx22_tso_packets: 308230791
> tx22_tso_bytes: 1859686450671
> tx22_tso_inner_packets: 0
> tx22_tso_inner_bytes: 0
> tx22_csum_partial: 442751518
> tx22_csum_partial_inner: 0
> tx22_added_vlan_packets: 4792100335
> tx22_nop: 70631706
> tx22_csum_none: 4349348817
> tx22_stopped: 9355
> tx22_dropped: 0
> tx22_xmit_more: 45165994
> tx22_recover: 0
> tx22_cqes: 4746936601
> tx22_wake: 9355
> tx22_cqe_err: 0
> tx23_packets: 5664896066
> tx23_bytes: 3207724186946
> tx23_tso_packets: 300418757
> tx23_tso_bytes: 1794180478679
> tx23_tso_inner_packets: 0
> tx23_tso_inner_bytes: 0
> tx23_csum_partial: 429898848
> tx23_csum_partial_inner: 0
> tx23_added_vlan_packets: 4674317320
> tx23_nop: 67899896
> tx23_csum_none: 4244418472
> tx23_stopped: 11684
> tx23_dropped: 0
> tx23_xmit_more: 43351132
> tx23_recover: 0
> tx23_cqes: 4630969028
> tx23_wake: 11684
> tx23_cqe_err: 0
> tx24_packets: 5663326601
> tx24_bytes: 3250127095110
> tx24_tso_packets: 301327422
> tx24_tso_bytes: 1831260534157
> tx24_tso_inner_packets: 0
> tx24_tso_inner_bytes: 0
> tx24_csum_partial: 438757312
> tx24_csum_partial_inner: 0
> tx24_added_vlan_packets: 4646014986
> tx24_nop: 68431153
> tx24_csum_none: 4207257674
> tx24_stopped: 9240
> tx24_dropped: 0
> tx24_xmit_more: 47699542
> tx24_recover: 0
> tx24_cqes: 4598317913
> tx24_wake: 9240
> tx24_cqe_err: 0
> tx25_packets: 5703883962
> tx25_bytes: 3291856915695
> tx25_tso_packets: 308900318
> tx25_tso_bytes: 1855516128386
> tx25_tso_inner_packets: 0
> tx25_tso_inner_bytes: 0
> tx25_csum_partial: 444753744
> tx25_csum_partial_inner: 0
> tx25_added_vlan_packets: 4676528924
> tx25_nop: 69230967
> tx25_csum_none: 4231775180
> tx25_stopped: 1140
> tx25_dropped: 0
> tx25_xmit_more: 40819195
> tx25_recover: 0
> tx25_cqes: 4635710966
> tx25_wake: 1140
> tx25_cqe_err: 0
> tx26_packets: 5803495984
> tx26_bytes: 3413564272139
> tx26_tso_packets: 319986230
> tx26_tso_bytes: 1929042839677
> tx26_tso_inner_packets: 0
> tx26_tso_inner_bytes: 0
> tx26_csum_partial: 464771163
> tx26_csum_partial_inner: 0
> tx26_added_vlan_packets: 4734767280
> tx26_nop: 71345080
> tx26_csum_none: 4269996117
> tx26_stopped: 10972
> tx26_dropped: 0
> tx26_xmit_more: 43793424
> tx26_recover: 0
> tx26_cqes: 4690976400
> tx26_wake: 10972
> tx26_cqe_err: 0
> tx27_packets: 5960955343
> tx27_bytes: 3444156164526
> tx27_tso_packets: 325099639
> tx27_tso_bytes: 1928378678784
> tx27_tso_inner_packets: 0
> tx27_tso_inner_bytes: 0
> tx27_csum_partial: 467310289
> tx27_csum_partial_inner: 0
> tx27_added_vlan_packets: 4888651368
> tx27_nop: 73201664
> tx27_csum_none: 4421341079
> tx27_stopped: 9465
> tx27_dropped: 0
> tx27_xmit_more: 53632121
> tx27_recover: 0
> tx27_cqes: 4835021398
> tx27_wake: 9465
> tx27_cqe_err: 0
> tx28_packets: 0
> tx28_bytes: 0
> tx28_tso_packets: 0
> tx28_tso_bytes: 0
> tx28_tso_inner_packets: 0
> tx28_tso_inner_bytes: 0
> tx28_csum_partial: 0
> tx28_csum_partial_inner: 0
> tx28_added_vlan_packets: 0
> tx28_nop: 0
> tx28_csum_none: 0
> tx28_stopped: 0
> tx28_dropped: 0
> tx28_xmit_more: 0
> tx28_recover: 0
> tx28_cqes: 0
> tx28_wake: 0
> tx28_cqe_err: 0
> tx29_packets: 3
> tx29_bytes: 266
> tx29_tso_packets: 0
> tx29_tso_bytes: 0
> tx29_tso_inner_packets: 0
> tx29_tso_inner_bytes: 0
> tx29_csum_partial: 0
> tx29_csum_partial_inner: 0
> tx29_added_vlan_packets: 0
> tx29_nop: 0
> tx29_csum_none: 3
> tx29_stopped: 0
> tx29_dropped: 0
> tx29_xmit_more: 1
> tx29_recover: 0
> tx29_cqes: 2
> tx29_wake: 0
> tx29_cqe_err: 0
> tx30_packets: 0
> tx30_bytes: 0
> tx30_tso_packets: 0
> tx30_tso_bytes: 0
> tx30_tso_inner_packets: 0
> tx30_tso_inner_bytes: 0
> tx30_csum_partial: 0
> tx30_csum_partial_inner: 0
> tx30_added_vlan_packets: 0
> tx30_nop: 0
> tx30_csum_none: 0
> tx30_stopped: 0
> tx30_dropped: 0
> tx30_xmit_more: 0
> tx30_recover: 0
> tx30_cqes: 0
> tx30_wake: 0
> tx30_cqe_err: 0
> tx31_packets: 0
> tx31_bytes: 0
> tx31_tso_packets: 0
> tx31_tso_bytes: 0
> tx31_tso_inner_packets: 0
> tx31_tso_inner_bytes: 0
> tx31_csum_partial: 0
> tx31_csum_partial_inner: 0
> tx31_added_vlan_packets: 0
> tx31_nop: 0
> tx31_csum_none: 0
> tx31_stopped: 0
> tx31_dropped: 0
> tx31_xmit_more: 0
> tx31_recover: 0
> tx31_cqes: 0
> tx31_wake: 0
> tx31_cqe_err: 0
> tx32_packets: 0
> tx32_bytes: 0
> tx32_tso_packets: 0
> tx32_tso_bytes: 0
> tx32_tso_inner_packets: 0
> tx32_tso_inner_bytes: 0
> tx32_csum_partial: 0
> tx32_csum_partial_inner: 0
> tx32_added_vlan_packets: 0
> tx32_nop: 0
> tx32_csum_none: 0
> tx32_stopped: 0
> tx32_dropped: 0
> tx32_xmit_more: 0
> tx32_recover: 0
> tx32_cqes: 0
> tx32_wake: 0
> tx32_cqe_err: 0
> tx33_packets: 0
> tx33_bytes: 0
> tx33_tso_packets: 0
> tx33_tso_bytes: 0
> tx33_tso_inner_packets: 0
> tx33_tso_inner_bytes: 0
> tx33_csum_partial: 0
> tx33_csum_partial_inner: 0
> tx33_added_vlan_packets: 0
> tx33_nop: 0
> tx33_csum_none: 0
> tx33_stopped: 0
> tx33_dropped: 0
> tx33_xmit_more: 0
> tx33_recover: 0
> tx33_cqes: 0
> tx33_wake: 0
> tx33_cqe_err: 0
> tx34_packets: 0
> tx34_bytes: 0
> tx34_tso_packets: 0
> tx34_tso_bytes: 0
> tx34_tso_inner_packets: 0
> tx34_tso_inner_bytes: 0
> tx34_csum_partial: 0
> tx34_csum_partial_inner: 0
> tx34_added_vlan_packets: 0
> tx34_nop: 0
> tx34_csum_none: 0
> tx34_stopped: 0
> tx34_dropped: 0
> tx34_xmit_more: 0
> tx34_recover: 0
> tx34_cqes: 0
> tx34_wake: 0
> tx34_cqe_err: 0
> tx35_packets: 0
> tx35_bytes: 0
> tx35_tso_packets: 0
> tx35_tso_bytes: 0
> tx35_tso_inner_packets: 0
> tx35_tso_inner_bytes: 0
> tx35_csum_partial: 0
> tx35_csum_partial_inner: 0
> tx35_added_vlan_packets: 0
> tx35_nop: 0
> tx35_csum_none: 0
> tx35_stopped: 0
> tx35_dropped: 0
> tx35_xmit_more: 0
> tx35_recover: 0
> tx35_cqes: 0
> tx35_wake: 0
> tx35_cqe_err: 0
> tx36_packets: 0
> tx36_bytes: 0
> tx36_tso_packets: 0
> tx36_tso_bytes: 0
> tx36_tso_inner_packets: 0
> tx36_tso_inner_bytes: 0
> tx36_csum_partial: 0
> tx36_csum_partial_inner: 0
> tx36_added_vlan_packets: 0
> tx36_nop: 0
> tx36_csum_none: 0
> tx36_stopped: 0
> tx36_dropped: 0
> tx36_xmit_more: 0
> tx36_recover: 0
> tx36_cqes: 0
> tx36_wake: 0
> tx36_cqe_err: 0
> tx37_packets: 0
> tx37_bytes: 0
> tx37_tso_packets: 0
> tx37_tso_bytes: 0
> tx37_tso_inner_packets: 0
> tx37_tso_inner_bytes: 0
> tx37_csum_partial: 0
> tx37_csum_partial_inner: 0
> tx37_added_vlan_packets: 0
> tx37_nop: 0
> tx37_csum_none: 0
> tx37_stopped: 0
> tx37_dropped: 0
> tx37_xmit_more: 0
> tx37_recover: 0
> tx37_cqes: 0
> tx37_wake: 0
> tx37_cqe_err: 0
> tx38_packets: 0
> tx38_bytes: 0
> tx38_tso_packets: 0
> tx38_tso_bytes: 0
> tx38_tso_inner_packets: 0
> tx38_tso_inner_bytes: 0
> tx38_csum_partial: 0
> tx38_csum_partial_inner: 0
> tx38_added_vlan_packets: 0
> tx38_nop: 0
> tx38_csum_none: 0
> tx38_stopped: 0
> tx38_dropped: 0
> tx38_xmit_more: 0
> tx38_recover: 0
> tx38_cqes: 0
> tx38_wake: 0
> tx38_cqe_err: 0
> tx39_packets: 0
> tx39_bytes: 0
> tx39_tso_packets: 0
> tx39_tso_bytes: 0
> tx39_tso_inner_packets: 0
> tx39_tso_inner_bytes: 0
> tx39_csum_partial: 0
> tx39_csum_partial_inner: 0
> tx39_added_vlan_packets: 0
> tx39_nop: 0
> tx39_csum_none: 0
> tx39_stopped: 0
> tx39_dropped: 0
> tx39_xmit_more: 0
> tx39_recover: 0
> tx39_cqes: 0
> tx39_wake: 0
> tx39_cqe_err: 0
> tx40_packets: 0
> tx40_bytes: 0
> tx40_tso_packets: 0
> tx40_tso_bytes: 0
> tx40_tso_inner_packets: 0
> tx40_tso_inner_bytes: 0
> tx40_csum_partial: 0
> tx40_csum_partial_inner: 0
> tx40_added_vlan_packets: 0
> tx40_nop: 0
> tx40_csum_none: 0
> tx40_stopped: 0
> tx40_dropped: 0
> tx40_xmit_more: 0
> tx40_recover: 0
> tx40_cqes: 0
> tx40_wake: 0
> tx40_cqe_err: 0
> tx41_packets: 0
> tx41_bytes: 0
> tx41_tso_packets: 0
> tx41_tso_bytes: 0
> tx41_tso_inner_packets: 0
> tx41_tso_inner_bytes: 0
> tx41_csum_partial: 0
> tx41_csum_partial_inner: 0
> tx41_added_vlan_packets: 0
> tx41_nop: 0
> tx41_csum_none: 0
> tx41_stopped: 0
> tx41_dropped: 0
> tx41_xmit_more: 0
> tx41_recover: 0
> tx41_cqes: 0
> tx41_wake: 0
> tx41_cqe_err: 0
> tx42_packets: 0
> tx42_bytes: 0
> tx42_tso_packets: 0
> tx42_tso_bytes: 0
> tx42_tso_inner_packets: 0
> tx42_tso_inner_bytes: 0
> tx42_csum_partial: 0
> tx42_csum_partial_inner: 0
> tx42_added_vlan_packets: 0
> tx42_nop: 0
> tx42_csum_none: 0
> tx42_stopped: 0
> tx42_dropped: 0
> tx42_xmit_more: 0
> tx42_recover: 0
> tx42_cqes: 0
> tx42_wake: 0
> tx42_cqe_err: 0
> tx43_packets: 0
> tx43_bytes: 0
> tx43_tso_packets: 0
> tx43_tso_bytes: 0
> tx43_tso_inner_packets: 0
> tx43_tso_inner_bytes: 0
> tx43_csum_partial: 0
> tx43_csum_partial_inner: 0
> tx43_added_vlan_packets: 0
> tx43_nop: 0
> tx43_csum_none: 0
> tx43_stopped: 0
> tx43_dropped: 0
> tx43_xmit_more: 0
> tx43_recover: 0
> tx43_cqes: 0
> tx43_wake: 0
> tx43_cqe_err: 0
> tx44_packets: 0
> tx44_bytes: 0
> tx44_tso_packets: 0
> tx44_tso_bytes: 0
> tx44_tso_inner_packets: 0
> tx44_tso_inner_bytes: 0
> tx44_csum_partial: 0
> tx44_csum_partial_inner: 0
> tx44_added_vlan_packets: 0
> tx44_nop: 0
> tx44_csum_none: 0
> tx44_stopped: 0
> tx44_dropped: 0
> tx44_xmit_more: 0
> tx44_recover: 0
> tx44_cqes: 0
> tx44_wake: 0
> tx44_cqe_err: 0
> tx45_packets: 0
> tx45_bytes: 0
> tx45_tso_packets: 0
> tx45_tso_bytes: 0
> tx45_tso_inner_packets: 0
> tx45_tso_inner_bytes: 0
> tx45_csum_partial: 0
> tx45_csum_partial_inner: 0
> tx45_added_vlan_packets: 0
> tx45_nop: 0
> tx45_csum_none: 0
> tx45_stopped: 0
> tx45_dropped: 0
> tx45_xmit_more: 0
> tx45_recover: 0
> tx45_cqes: 0
> tx45_wake: 0
> tx45_cqe_err: 0
> tx46_packets: 0
> tx46_bytes: 0
> tx46_tso_packets: 0
> tx46_tso_bytes: 0
> tx46_tso_inner_packets: 0
> tx46_tso_inner_bytes: 0
> tx46_csum_partial: 0
> tx46_csum_partial_inner: 0
> tx46_added_vlan_packets: 0
> tx46_nop: 0
> tx46_csum_none: 0
> tx46_stopped: 0
> tx46_dropped: 0
> tx46_xmit_more: 0
> tx46_recover: 0
> tx46_cqes: 0
> tx46_wake: 0
> tx46_cqe_err: 0
> tx47_packets: 0
> tx47_bytes: 0
> tx47_tso_packets: 0
> tx47_tso_bytes: 0
> tx47_tso_inner_packets: 0
> tx47_tso_inner_bytes: 0
> tx47_csum_partial: 0
> tx47_csum_partial_inner: 0
> tx47_added_vlan_packets: 0
> tx47_nop: 0
> tx47_csum_none: 0
> tx47_stopped: 0
> tx47_dropped: 0
> tx47_xmit_more: 0
> tx47_recover: 0
> tx47_cqes: 0
> tx47_wake: 0
> tx47_cqe_err: 0
> tx48_packets: 0
> tx48_bytes: 0
> tx48_tso_packets: 0
> tx48_tso_bytes: 0
> tx48_tso_inner_packets: 0
> tx48_tso_inner_bytes: 0
> tx48_csum_partial: 0
> tx48_csum_partial_inner: 0
> tx48_added_vlan_packets: 0
> tx48_nop: 0
> tx48_csum_none: 0
> tx48_stopped: 0
> tx48_dropped: 0
> tx48_xmit_more: 0
> tx48_recover: 0
> tx48_cqes: 0
> tx48_wake: 0
> tx48_cqe_err: 0
> tx49_packets: 0
> tx49_bytes: 0
> tx49_tso_packets: 0
> tx49_tso_bytes: 0
> tx49_tso_inner_packets: 0
> tx49_tso_inner_bytes: 0
> tx49_csum_partial: 0
> tx49_csum_partial_inner: 0
> tx49_added_vlan_packets: 0
> tx49_nop: 0
> tx49_csum_none: 0
> tx49_stopped: 0
> tx49_dropped: 0
> tx49_xmit_more: 0
> tx49_recover: 0
> tx49_cqes: 0
> tx49_wake: 0
> tx49_cqe_err: 0
> tx50_packets: 0
> tx50_bytes: 0
> tx50_tso_packets: 0
> tx50_tso_bytes: 0
> tx50_tso_inner_packets: 0
> tx50_tso_inner_bytes: 0
> tx50_csum_partial: 0
> tx50_csum_partial_inner: 0
> tx50_added_vlan_packets: 0
> tx50_nop: 0
> tx50_csum_none: 0
> tx50_stopped: 0
> tx50_dropped: 0
> tx50_xmit_more: 0
> tx50_recover: 0
> tx50_cqes: 0
> tx50_wake: 0
> tx50_cqe_err: 0
> tx51_packets: 0
> tx51_bytes: 0
> tx51_tso_packets: 0
> tx51_tso_bytes: 0
> tx51_tso_inner_packets: 0
> tx51_tso_inner_bytes: 0
> tx51_csum_partial: 0
> tx51_csum_partial_inner: 0
> tx51_added_vlan_packets: 0
> tx51_nop: 0
> tx51_csum_none: 0
> tx51_stopped: 0
> tx51_dropped: 0
> tx51_xmit_more: 0
> tx51_recover: 0
> tx51_cqes: 0
> tx51_wake: 0
> tx51_cqe_err: 0
> tx52_packets: 0
> tx52_bytes: 0
> tx52_tso_packets: 0
> tx52_tso_bytes: 0
> tx52_tso_inner_packets: 0
> tx52_tso_inner_bytes: 0
> tx52_csum_partial: 0
> tx52_csum_partial_inner: 0
> tx52_added_vlan_packets: 0
> tx52_nop: 0
> tx52_csum_none: 0
> tx52_stopped: 0
> tx52_dropped: 0
> tx52_xmit_more: 0
> tx52_recover: 0
> tx52_cqes: 0
> tx52_wake: 0
> tx52_cqe_err: 0
> tx53_packets: 0
> tx53_bytes: 0
> tx53_tso_packets: 0
> tx53_tso_bytes: 0
> tx53_tso_inner_packets: 0
> tx53_tso_inner_bytes: 0
> tx53_csum_partial: 0
> tx53_csum_partial_inner: 0
> tx53_added_vlan_packets: 0
> tx53_nop: 0
> tx53_csum_none: 0
> tx53_stopped: 0
> tx53_dropped: 0
> tx53_xmit_more: 0
> tx53_recover: 0
> tx53_cqes: 0
> tx53_wake: 0
> tx53_cqe_err: 0
> tx54_packets: 0
> tx54_bytes: 0
> tx54_tso_packets: 0
> tx54_tso_bytes: 0
> tx54_tso_inner_packets: 0
> tx54_tso_inner_bytes: 0
> tx54_csum_partial: 0
> tx54_csum_partial_inner: 0
> tx54_added_vlan_packets: 0
> tx54_nop: 0
> tx54_csum_none: 0
> tx54_stopped: 0
> tx54_dropped: 0
> tx54_xmit_more: 0
> tx54_recover: 0
> tx54_cqes: 0
> tx54_wake: 0
> tx54_cqe_err: 0
> tx55_packets: 0
> tx55_bytes: 0
> tx55_tso_packets: 0
> tx55_tso_bytes: 0
> tx55_tso_inner_packets: 0
> tx55_tso_inner_bytes: 0
> tx55_csum_partial: 0
> tx55_csum_partial_inner: 0
> tx55_added_vlan_packets: 0
> tx55_nop: 0
> tx55_csum_none: 0
> tx55_stopped: 0
> tx55_dropped: 0
> tx55_xmit_more: 0
> tx55_recover: 0
> tx55_cqes: 0
> tx55_wake: 0
> tx55_cqe_err: 0
> tx0_xdp_xmit: 0
> tx0_xdp_full: 0
> tx0_xdp_err: 0
> tx0_xdp_cqes: 0
> tx1_xdp_xmit: 0
> tx1_xdp_full: 0
> tx1_xdp_err: 0
> tx1_xdp_cqes: 0
> tx2_xdp_xmit: 0
> tx2_xdp_full: 0
> tx2_xdp_err: 0
> tx2_xdp_cqes: 0
> tx3_xdp_xmit: 0
> tx3_xdp_full: 0
> tx3_xdp_err: 0
> tx3_xdp_cqes: 0
> tx4_xdp_xmit: 0
> tx4_xdp_full: 0
> tx4_xdp_err: 0
> tx4_xdp_cqes: 0
> tx5_xdp_xmit: 0
> tx5_xdp_full: 0
> tx5_xdp_err: 0
> tx5_xdp_cqes: 0
> tx6_xdp_xmit: 0
> tx6_xdp_full: 0
> tx6_xdp_err: 0
> tx6_xdp_cqes: 0
> tx7_xdp_xmit: 0
> tx7_xdp_full: 0
> tx7_xdp_err: 0
> tx7_xdp_cqes: 0
> tx8_xdp_xmit: 0
> tx8_xdp_full: 0
> tx8_xdp_err: 0
> tx8_xdp_cqes: 0
> tx9_xdp_xmit: 0
> tx9_xdp_full: 0
> tx9_xdp_err: 0
> tx9_xdp_cqes: 0
> tx10_xdp_xmit: 0
> tx10_xdp_full: 0
> tx10_xdp_err: 0
> tx10_xdp_cqes: 0
> tx11_xdp_xmit: 0
> tx11_xdp_full: 0
> tx11_xdp_err: 0
> tx11_xdp_cqes: 0
> tx12_xdp_xmit: 0
> tx12_xdp_full: 0
> tx12_xdp_err: 0
> tx12_xdp_cqes: 0
> tx13_xdp_xmit: 0
> tx13_xdp_full: 0
> tx13_xdp_err: 0
> tx13_xdp_cqes: 0
> tx14_xdp_xmit: 0
> tx14_xdp_full: 0
> tx14_xdp_err: 0
> tx14_xdp_cqes: 0
> tx15_xdp_xmit: 0
> tx15_xdp_full: 0
> tx15_xdp_err: 0
> tx15_xdp_cqes: 0
> tx16_xdp_xmit: 0
> tx16_xdp_full: 0
> tx16_xdp_err: 0
> tx16_xdp_cqes: 0
> tx17_xdp_xmit: 0
> tx17_xdp_full: 0
> tx17_xdp_err: 0
> tx17_xdp_cqes: 0
> tx18_xdp_xmit: 0
> tx18_xdp_full: 0
> tx18_xdp_err: 0
> tx18_xdp_cqes: 0
> tx19_xdp_xmit: 0
> tx19_xdp_full: 0
> tx19_xdp_err: 0
> tx19_xdp_cqes: 0
> tx20_xdp_xmit: 0
> tx20_xdp_full: 0
> tx20_xdp_err: 0
> tx20_xdp_cqes: 0
> tx21_xdp_xmit: 0
> tx21_xdp_full: 0
> tx21_xdp_err: 0
> tx21_xdp_cqes: 0
> tx22_xdp_xmit: 0
> tx22_xdp_full: 0
> tx22_xdp_err: 0
> tx22_xdp_cqes: 0
> tx23_xdp_xmit: 0
> tx23_xdp_full: 0
> tx23_xdp_err: 0
> tx23_xdp_cqes: 0
> tx24_xdp_xmit: 0
> tx24_xdp_full: 0
> tx24_xdp_err: 0
> tx24_xdp_cqes: 0
> tx25_xdp_xmit: 0
> tx25_xdp_full: 0
> tx25_xdp_err: 0
> tx25_xdp_cqes: 0
> tx26_xdp_xmit: 0
> tx26_xdp_full: 0
> tx26_xdp_err: 0
> tx26_xdp_cqes: 0
> tx27_xdp_xmit: 0
> tx27_xdp_full: 0
> tx27_xdp_err: 0
> tx27_xdp_cqes: 0
> tx28_xdp_xmit: 0
> tx28_xdp_full: 0
> tx28_xdp_err: 0
> tx28_xdp_cqes: 0
> tx29_xdp_xmit: 0
> tx29_xdp_full: 0
> tx29_xdp_err: 0
> tx29_xdp_cqes: 0
> tx30_xdp_xmit: 0
> tx30_xdp_full: 0
> tx30_xdp_err: 0
> tx30_xdp_cqes: 0
> tx31_xdp_xmit: 0
> tx31_xdp_full: 0
> tx31_xdp_err: 0
> tx31_xdp_cqes: 0
> tx32_xdp_xmit: 0
> tx32_xdp_full: 0
> tx32_xdp_err: 0
> tx32_xdp_cqes: 0
> tx33_xdp_xmit: 0
> tx33_xdp_full: 0
> tx33_xdp_err: 0
> tx33_xdp_cqes: 0
> tx34_xdp_xmit: 0
> tx34_xdp_full: 0
> tx34_xdp_err: 0
> tx34_xdp_cqes: 0
> tx35_xdp_xmit: 0
> tx35_xdp_full: 0
> tx35_xdp_err: 0
> tx35_xdp_cqes: 0
> tx36_xdp_xmit: 0
> tx36_xdp_full: 0
> tx36_xdp_err: 0
> tx36_xdp_cqes: 0
> tx37_xdp_xmit: 0
> tx37_xdp_full: 0
> tx37_xdp_err: 0
> tx37_xdp_cqes: 0
> tx38_xdp_xmit: 0
> tx38_xdp_full: 0
> tx38_xdp_err: 0
> tx38_xdp_cqes: 0
> tx39_xdp_xmit: 0
> tx39_xdp_full: 0
> tx39_xdp_err: 0
> tx39_xdp_cqes: 0
> tx40_xdp_xmit: 0
> tx40_xdp_full: 0
> tx40_xdp_err: 0
> tx40_xdp_cqes: 0
> tx41_xdp_xmit: 0
> tx41_xdp_full: 0
> tx41_xdp_err: 0
> tx41_xdp_cqes: 0
> tx42_xdp_xmit: 0
> tx42_xdp_full: 0
> tx42_xdp_err: 0
> tx42_xdp_cqes: 0
> tx43_xdp_xmit: 0
> tx43_xdp_full: 0
> tx43_xdp_err: 0
> tx43_xdp_cqes: 0
> tx44_xdp_xmit: 0
> tx44_xdp_full: 0
> tx44_xdp_err: 0
> tx44_xdp_cqes: 0
> tx45_xdp_xmit: 0
> tx45_xdp_full: 0
> tx45_xdp_err: 0
> tx45_xdp_cqes: 0
> tx46_xdp_xmit: 0
> tx46_xdp_full: 0
> tx46_xdp_err: 0
> tx46_xdp_cqes: 0
> tx47_xdp_xmit: 0
> tx47_xdp_full: 0
> tx47_xdp_err: 0
> tx47_xdp_cqes: 0
> tx48_xdp_xmit: 0
> tx48_xdp_full: 0
> tx48_xdp_err: 0
> tx48_xdp_cqes: 0
> tx49_xdp_xmit: 0
> tx49_xdp_full: 0
> tx49_xdp_err: 0
> tx49_xdp_cqes: 0
> tx50_xdp_xmit: 0
> tx50_xdp_full: 0
> tx50_xdp_err: 0
> tx50_xdp_cqes: 0
> tx51_xdp_xmit: 0
> tx51_xdp_full: 0
> tx51_xdp_err: 0
> tx51_xdp_cqes: 0
> tx52_xdp_xmit: 0
> tx52_xdp_full: 0
> tx52_xdp_err: 0
> tx52_xdp_cqes: 0
> tx53_xdp_xmit: 0
> tx53_xdp_full: 0
> tx53_xdp_err: 0
> tx53_xdp_cqes: 0
> tx54_xdp_xmit: 0
> tx54_xdp_full: 0
> tx54_xdp_err: 0
> tx54_xdp_cqes: 0
> tx55_xdp_xmit: 0
> tx55_xdp_full: 0
> tx55_xdp_err: 0
> tx55_xdp_cqes: 0
>
>
> mpstat -P ALL 1 10
> Average: CPU %usr %nice %sys %iowait %irq %soft
> %steal
> %guest %gnice %idle
> Average: all 0.04 0.00 6.94 0.02 0.00 32.00
> 0.00 0.00 0.00 61.00
> Average: 0 0.00 0.00 1.20 0.00 0.00 0.00
> 0.00
> 0.00 0.00 98.80
> Average: 1 0.00 0.00 2.30 0.00 0.00 0.00
> 0.00
> 0.00 0.00 97.70
> Average: 2 0.10 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.90
> Average: 3 0.10 0.00 1.50 0.00 0.00 0.00
> 0.00
> 0.00 0.00 98.40
> Average: 4 0.50 0.00 2.50 0.00 0.00 0.00
> 0.00
> 0.00 0.00 97.00
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 6 0.90 0.00 10.20 0.00 0.00 0.00
> 0.00
> 0.00 0.00 88.90
> Average: 7 0.00 0.00 0.00 1.40 0.00 0.00
> 0.00
> 0.00 0.00 98.60
> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 11 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 14 0.00 0.00 12.99 0.00 0.00 62.64
> 0.00 0.00 0.00 24.38
> Average: 15 0.00 0.00 12.70 0.00 0.00 63.40
> 0.00 0.00 0.00 23.90
> Average: 16 0.00 0.00 11.20 0.00 0.00 66.40
> 0.00 0.00 0.00 22.40
> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
> 0.00 0.00 0.00 31.30
> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
> 0.00 0.00 0.00 24.90
> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
> 0.00 0.00 0.00 19.68
> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
> 0.00 0.00 0.00 18.00
> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
> 0.00 0.00 0.00 17.40
> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
> 0.00 0.00 0.00 26.03
> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
> 0.00 0.00 0.00 17.52
> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
> 0.00 0.00 0.00 18.20
> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
> 0.00 0.00 0.00 17.68
> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
> 0.00 0.00 0.00 18.20
> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
> 0.00 0.00 0.00 20.42
The numa cores are not at 100% util, you have around 20% of idle on
each one.
> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
> 0.00
> 0.00 0.00 96.10
> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.70
> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.50
> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00
> 0.00 0.00 100.00
> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
> 0.00
> 0.00 0.00 97.40
> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.10
> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
> 0.00
> 0.00 0.00 99.40
> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
> 0.00 0.00 0.00 19.42
> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
> 0.00 0.00 0.00 26.60
> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
> 0.00 0.00 0.00 21.60
> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
> 0.00 0.00 0.00 20.50
> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
> 0.00 0.00 0.00 21.60
> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
> 0.00 0.00 0.00 24.58
> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
> 0.00 0.00 0.00 24.68
> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
> 0.00 0.00 0.00 28.34
> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
> 0.00 0.00 0.00 25.83
> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
> 0.00 0.00 0.00 27.20
> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
> 0.00 0.00 0.00 28.03
> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
> 0.00 0.00 0.00 24.70
> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
> 0.00 0.00 0.00 25.30
> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
> 0.00 0.00 0.00 26.77
>
>
> ethtool -k enp175s0f0
> Features for enp175s0f0:
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: on
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: on
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: on
> highdma: on [fixed]
> rx-vlan-filter: on
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: on
> tx-gre-csum-segmentation: on
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: on
> tx-udp_tnl-csum-segmentation: on
> tx-gso-partial: on
> tx-sctp-segmentation: off [fixed]
> tx-esp-segmentation: off [fixed]
> tx-udp-segmentation: on
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off
> rx-all: off
> tx-vlan-stag-hw-insert: on
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: on [fixed]
> l2-fwd-offload: off [fixed]
> hw-tc-offload: off
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> rx-udp_tunnel-port-offload: on
> tls-hw-tx-offload: off [fixed]
> tls-hw-rx-offload: off [fixed]
> rx-gro-hw: off [fixed]
> tls-hw-record: off [fixed]
>
> ethtool -c enp175s0f0
> Coalesce parameters for enp175s0f0:
> Adaptive RX: off TX: on
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
> dmac: 32703
>
> rx-usecs: 256
> rx-frames: 128
> rx-usecs-irq: 0
> rx-frames-irq: 0
>
> tx-usecs: 8
> tx-frames: 128
> tx-usecs-irq: 0
> tx-frames-irq: 0
>
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
>
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
>
> ethtool -g enp175s0f0
> Ring parameters for enp175s0f0:
> Pre-set maximums:
> RX: 8192
> RX Mini: 0
> RX Jumbo: 0
> TX: 8192
> Current hardware settings:
> RX: 4096
> RX Mini: 0
> RX Jumbo: 0
> TX: 4096
>
>
>
>
>
>
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-01 10:34 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Eric Dumazet, netdev, Tariq Toukan, Ilias Apalodimas,
Yoel Caspersen, Mel Gorman, Aaron Lu
In-Reply-To: <20181101102213.2fa2643d@redhat.com>
W dniu 01.11.2018 o 10:22, Jesper Dangaard Brouer pisze:
> On Wed, 31 Oct 2018 23:20:01 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 31.10.2018 o 23:09, Eric Dumazet pisze:
>>> On 10/31/2018 02:57 PM, Paweł Staszewski wrote:
>>>> Hi
>>>>
>>>> So maybee someone will be interested how linux kernel handles
>>>> normal traffic (not pktgen :) )
> Pawel is this live production traffic?
Yes moved server from testlab to production to check (risking a little -
but this is traffic switched to backup router : ) )
>
> I know Yoel (Cc) is very interested to know the real-life limitation of
> Linux as a router, especially with VLANs like you use.
So yes this is real-life traffic , real users - normal mixed internet
traffic forwarded (including ddos-es :) )
>
>
>>>> Server HW configuration:
>>>>
>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>
>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>
>>>>
>>>> Server software:
>>>>
>>>> FRR - as routing daemon
>>>>
>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local numa node)
>>>>
>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa node)
>>>>
>>>>
>>>> Maximum traffic that server can handle:
>>>>
>>>> Bandwidth
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> \ iface Rx Tx Total
>>>> ==============================================================================
>>>> enp175s0f1: 28.51 Gb/s 37.24 Gb/s 65.74 Gb/s
>>>> enp175s0f0: 38.07 Gb/s 28.44 Gb/s 66.51 Gb/s
>>>> ------------------------------------------------------------------------------
>>>> total: 66.58 Gb/s 65.67 Gb/s 132.25 Gb/s
>>>>
> Actually rather impressive number for a Linux router.
>
>>>> Packets per second:
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> ==============================================================================
>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s 8735207.00 P/s
>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s 8790460.00 P/s
>>>> ------------------------------------------------------------------------------
>>>> total: 8806533.00 P/s 8719134.00 P/s 17525668.00 P/s
>>>>
> Average packet size:
> (28.51*10^9/8)/5248589 = 678.99 bytes
> (38.07*10^9/8)/3557944 = 1337.49 bytes
>
>
>>>> After reaching that limits nics on the upstream side (more RX
>>>> traffic) start to drop packets
>>>>
>>>>
>>>> I just dont understand that server can't handle more bandwidth
>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>> RX side are increasing.
>>>>
>>>> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
>>>> is 126Gbit - and also when testing with pktgen i can reach more bw
>>>> and pps (like 4x more comparing to normal internet traffic)
>>>>
>>>> And wondering if there is something that can be improved here.
>>>>
>>>>
>>>>
>>>> Some more informations / counters / stats and perf top below:
>>>>
>>>> Perf top flame graph:
>>>>
>>>> https://uploadfiles.io/7zo6u
> Thanks a lot for the flame graph!
>
>>>> System configuration(long):
>>>>
>>>>
>>>> cat /sys/devices/system/node/node1/cpulist
>>>> 14-27,42-55
>>>> cat /sys/class/net/enp175s0f0/device/numa_node
>>>> 1
>>>> cat /sys/class/net/enp175s0f1/device/numa_node
>>>> 1
>>>>
> Hint grep can give you nicer output that cat:
>
> $ grep -H . /sys/class/net/*/device/numa_node
Sure:
grep -H . /sys/class/net/*/device/numa_node
/sys/class/net/enp175s0f0/device/numa_node:1
/sys/class/net/enp175s0f1/device/numa_node:1
>
>>>>
>>>>
>>>>
>>>> ip -s -d link ls dev enp175s0f0
>>>> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
>>>> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
>>>> RX: bytes packets errors dropped overrun mcast
>>>> 184142375840858 141347715974 2 2806325 0 85050528
>>>> TX: bytes packets errors dropped carrier collsns
>>>> 99270697277430 172227994003 0 0 0 0
>>>>
>>>> ip -s -d link ls dev enp175s0f1
>>>> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 8192
>>>> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536 gso_max_segs 65535
>>>> RX: bytes packets errors dropped overrun mcast
>>>> 99686284170801 173507590134 61 669685 0 100304421
>>>> TX: bytes packets errors dropped carrier collsns
>>>> 184435107970545 142383178304 0 0 0 0
>>>>
> You have increased the default (1000) qlen to 8192, why?
Was checking if higher txq will change anything
But no change for settings 1000,4096,8192
But yes i do not use there any traffic shaping like hfsc/hdb etc
- just default qdisc mq 0:
root pfifp_fast
tc qdisc show dev enp175s0f1
qdisc mq 0: root
qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
qdisc pfifo_fast 0: parent :36 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
...
...
And vlans are noqueue
tc -s -d qdisc show dev vlan1521
qdisc noqueue 0: root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
Weird is that no counters increasing but there is traffic in/out on that
vlans
ip -s -d link ls dev vlan1521
87: vlan1521@enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity 0
vlan protocol 802.1Q id 1521 <REORDER_HDR> addrgenmode eui64
numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
RX: bytes packets errors dropped overrun mcast
562964218394 1639370761 0 0 0 0
TX: bytes packets errors dropped carrier collsns
1417648713052 618271312 0 0 0 0
>
> What default qdisc do you run?... looking through your very detail main
> email report (I do love the details you give!). You run
> pfifo_fast_dequeue, thus this 8192 qlen is actually having effect.
>
> I would like to know if and how much qdisc_dequeue bulking is happening
> in this setup? Can you run:
>
> perf-stat-hist -m 8192 -P2 qdisc:qdisc_dequeue packets
>
> The perf-stat-hist is from Brendan Gregg's git-tree:
> https://github.com/brendangregg/perf-tools
> https://github.com/brendangregg/perf-tools/blob/master/misc/perf-stat-hist
>
./perf-stat-hist -m 8192 -P2 qdisc:qdisc_dequeue packets
Tracing qdisc:qdisc_dequeue, power-of-2, max 8192, until Ctrl-C...
^C
Range : Count Distribution
-> -1 : 0 | |
0 -> 0 : 43768349
|######################################|
1 -> 1 : 43895249
|######################################|
2 -> 3 : 352 |# |
4 -> 7 : 228 |# |
8 -> 15 : 135 |# |
16 -> 31 : 73 |# |
32 -> 63 : 7 |# |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 0 | |
4096 -> 8191 : 0 | |
8192 -> : 0 | |
>>>> ./softnet.sh
>>>> cpu total dropped squeezed collision rps flow_limit
>>>>
>>>>
>>>>
>>>>
>>>> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz cycles], (all, 56 CPUs)
>>>> ------------------------------------------------------------------------------------------
>>>>
>>>> 26.78% [kernel] [k] queued_spin_lock_slowpath
>>> This is highly suspect.
>>>
> I agree! -- 26.78% spend in queued_spin_lock_slowpath. Hint if you see
> _raw_spin_lock then it is likely not a contended lock, but if you see
> queued_spin_lock_slowpath in a perf-report your workload is likely in
> trouble.
>
>
>>> A call graph (perf record -a -g sleep 1; perf report --stdio)
>>> would tell what is going on.
>> perf report:
>> https://ufile.io/rqp0h
>>
> Thanks for the output (my 30" screen is just large enough to see the
> full output). Together with the flame-graph, it is clear that this
> lock happens in the page allocator code.
>
> Section copied out:
>
> mlx5e_poll_tx_cq
> |
> --16.34%--napi_consume_skb
> |
> |--12.65%--__free_pages_ok
> | |
> | --11.86%--free_one_page
> | |
> | |--10.10%--queued_spin_lock_slowpath
> | |
> | --0.65%--_raw_spin_lock
> |
> |--1.55%--page_frag_free
> |
> --1.44%--skb_release_data
>
>
> Let me explain what (I think) happens. The mlx5 driver RX-page recycle
> mechanism is not effective in this workload, and pages have to go
> through the page allocator. The lock contention happens during mlx5
> DMA TX completion cycle. And the page allocator cannot keep up at
> these speeds.
>
> One solution is extend page allocator with a bulk free API. (This have
> been on my TODO list for a long time, but I don't have a
> micro-benchmark that trick the driver page-recycle to fail). It should
> fit nicely, as I can see that kmem_cache_free_bulk() does get
> activated (bulk freeing SKBs), which means that DMA TX completion do
> have a bulk of packets.
>
> We can (and should) also improve the page recycle scheme in the driver.
> After LPC, I have a project with Tariq and Ilias (Cc'ed) to improve the
> page_pool, and we will (attempt) to generalize this, for both high-end
> mlx5 and more low-end ARM64-boards (macchiatobin and espressobin).
>
> The MM-people is in parallel working to improve the performance of
> order-0 page returns. Thus, the explicit page bulk free API might
> actually become less important. I actually think (Cc.) Aaron have a
> patchset he would like you to test, which removes the (zone->)lock
> you hit in free_one_page().
>
Ok - Thank You Jesper
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Jesper Dangaard Brouer @ 2018-11-01 10:55 UTC (permalink / raw)
To: David Ahern; +Cc: brouer, Paweł Staszewski, netdev, Yoel Caspersen
In-Reply-To: <3a88bb53-9d17-3e85-638e-a605f5bfe0fb@gmail.com>
On Wed, 31 Oct 2018 21:37:16 -0600 David Ahern <dsahern@gmail.com> wrote:
> This is mainly a forwarding use case? Seems so based on the perf report.
> I suspect forwarding with XDP would show pretty good improvement.
Yes, significant performance improvements.
Notice Davids talk: "Leveraging Kernel Tables with XDP"
http://vger.kernel.org/lpc-networking2018.html#session-1
It looks like that you are doing "pure" IP-routing, without any
iptables conntrack stuff (from your perf report data). That will
actually be a really good use-case for accelerating this with XDP.
I want you to understand the philosophy behind how David and I want
people to leverage XDP. Think of XDP as a software offload layer for
the kernel network stack. Setup and use Linux kernel network stack, but
accelerate parts of it with XDP, e.g. the route FIB lookup.
Sample code avail here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/bpf/xdp_fwd_kern.c
(I do warn, what we just found a bug/crash in setup+tairdown for the
mlx5 driver you are using, that we/mlx _will_ fix soon)
> You need the vlan changes I have queued up though.
I know Yoel will be very interested in those changes too! I've
convinced Yoel to write an XDP program for his Border Network Gateway
(BNG) production system[1], and his is a heavy VLAN user. And the plan
is to Open Source this when he have-something-working.
[1] https://www.version2.dk/blog/software-router-del-5-linux-bng-1086060
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-01 11:09 UTC (permalink / raw)
To: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <659fbf4b481c815f45a58b2351481cc9f761445b.camel@mellanox.com>
W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>> Hi
>>
>> So maybee someone will be interested how linux kernel handles normal
>> traffic (not pktgen :) )
>>
>>
>> Server HW configuration:
>>
>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>
>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>
>>
>> Server software:
>>
>> FRR - as routing daemon
>>
>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to local
>> numa
>> node)
>>
>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local numa
>> node)
>>
>>
>> Maximum traffic that server can handle:
>>
>> Bandwidth
>>
>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>> input: /proc/net/dev type: rate
>> \ iface Rx Tx Total
>> =====================================================================
>> =========
>> enp175s0f1: 28.51 Gb/s 37.24
>> Gb/s
>> 65.74 Gb/s
>> enp175s0f0: 38.07 Gb/s 28.44
>> Gb/s
>> 66.51 Gb/s
>> -------------------------------------------------------------------
>> -----------
>> total: 66.58 Gb/s 65.67
>> Gb/s
>> 132.25 Gb/s
>>
>>
>> Packets per second:
>>
>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>> input: /proc/net/dev type: rate
>> - iface Rx Tx Total
>> =====================================================================
>> =========
>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>> 8735207.00 P/s
>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>> 8790460.00 P/s
>> -------------------------------------------------------------------
>> -----------
>> total: 8806533.00 P/s 8719134.00 P/s
>> 17525668.00 P/s
>>
>>
>> After reaching that limits nics on the upstream side (more RX
>> traffic)
>> start to drop packets
>>
>>
>> I just dont understand that server can't handle more bandwidth
>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on RX
>> side are increasing.
>>
> Where do you see 40 Gb/s ? you showed that both ports on the same NIC (
> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) = 132.25
> Gb/s which aligns with your pcie link limit, what am i missing ?
hmm yes that was my concern also - cause cant find anywhere informations
about that bandwidth is uni or bidirectional - so if 126Gbit for x16 8GT
is unidir - then bidir will be 126/2 ~68Gbit - which will fit total bw
on both ports
This can explain maybee also why cpuload is rising rapidly from
120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net - so
there can be some error in reading them when offloading (gro/gso/tso) on
nic's is enabled that is why
>
>> Was thinking that maybee reached some pcie x16 limit - but x16 8GT
>> is
>> 126Gbit - and also when testing with pktgen i can reach more bw and
>> pps
>> (like 4x more comparing to normal internet traffic)
>>
> Are you forwarding when using pktgen as well or you just testing the RX
> side pps ?
Yes pktgen was tested on single port RX
Can check also forwarding to eliminate pciex limits
>
>> And wondering if there is something that can be improved here.
>>
>>
>>
>> Some more informations / counters / stats and perf top below:
>>
>> Perf top flame graph:
>>
>> https://uploadfiles.io/7zo6u
>>
>>
>>
>> System configuration(long):
>>
>>
>> cat /sys/devices/system/node/node1/cpulist
>> 14-27,42-55
>> cat /sys/class/net/enp175s0f0/device/numa_node
>> 1
>> cat /sys/class/net/enp175s0f1/device/numa_node
>> 1
>>
>>
>>
>>
>>
>> ip -s -d link ls dev enp175s0f0
>> 6: enp175s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>> state
>> UP mode DEFAULT group default qlen 8192
>> link/ether 0c:c4:7a:d8:5d:1c brd ff:ff:ff:ff:ff:ff promiscuity
>> 0
>> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
>> gso_max_segs 65535
>> RX: bytes packets errors dropped overrun mcast
>> 184142375840858 141347715974 2 2806325 0 85050528
>> TX: bytes packets errors dropped carrier collsns
>> 99270697277430 172227994003 0 0 0 0
>>
>> ip -s -d link ls dev enp175s0f1
>> 7: enp175s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
>> state
>> UP mode DEFAULT group default qlen 8192
>> link/ether 0c:c4:7a:d8:5d:1d brd ff:ff:ff:ff:ff:ff promiscuity
>> 0
>> addrgenmode eui64 numtxqueues 448 numrxqueues 56 gso_max_size 65536
>> gso_max_segs 65535
>> RX: bytes packets errors dropped overrun mcast
>> 99686284170801 173507590134 61 669685 0 100304421
>> TX: bytes packets errors dropped carrier collsns
>> 184435107970545 142383178304 0 0 0 0
>>
>>
>> ./softnet.sh
>> cpu total dropped squeezed collision rps flow_limit
>> 0 3961392822 0 1221478 0 0 0
>> 1 3701952251 0 1258234 0 0 0
>> 2 3879522030 0 1584282 0 0 0
>> 3 3731349789 0 1529029 0 0 0
>> 4 1323956701 0 2176371 0 0 0
>> 5 420528963 0 1880146 0 0 0
>> 6 348720322 0 1830142 0 0 0
>> 7 372736328 0 1820891 0 0 0
>> 8 567888751 0 1414763 0 0 0
>> 9 476075775 0 1868150 0 0 0
>> 10 468946725 0 1841428 0 0 0
>> 11 676591958 0 1900160 0 0 0
>> 12 346803472 0 1834600 0 0 0
>> 13 457960872 0 1874529 0 0 0
>> 14 1990279665 0 4699000 0 0 0
>> 15 1211873601 0 4541281 0 0 0
>> 16 1123871928 0 4544712 0 0 0
>> 17 1014957263 0 4152355 0 0 0
>> 18 2603779724 0 4593869 0 0 0
>> 19 2181924054 0 4930618 0 0 0
>> 20 2273502182 0 4894627 0 0 0
>> 21 2232030947 0 4860048 0 0 0
>> 22 2203555394 0 4603830 0 0 0
>> 23 2194756800 0 4921294 0 0 0
>> 24 2347158294 0 4818354 0 0 0
>> 25 2291097883 0 4744469 0 0 0
>> 26 2206945011 0 4836483 0 0 0
>> 27 2318530217 0 4917617 0 0 0
>> 28 512797543 0 1895200 0 0 0
>> 29 597279474 0 1532134 0 0 0
>> 30 475317503 0 1451523 0 0 0
>> 31 499172796 0 1901207 0 0 0
>> 32 493874745 0 1915382 0 0 0
>> 33 296056288 0 1865535 0 0 0
>> 34 3905097041 0 1580822 0 0 0
>> 35 3905112345 0 1536105 0 0 0
>> 36 3900358950 0 1166319 0 0 0
>> 37 3940978093 0 1600219 0 0 0
>> 38 3878632215 0 1180389 0 0 0
>> 39 3814804736 0 1584925 0 0 0
>> 40 4152934337 0 1663660 0 0 0
>> 41 3855273904 0 1552219 0 0 0
>> 42 2319538182 0 4884480 0 0 0
>> 43 2448606991 0 4387456 0 0 0
>> 44 1436136753 0 4485073 0 0 0
>> 45 1200500141 0 4537284 0 0 0
>> 46 1307799923 0 4534156 0 0 0
>> 47 1586575293 0 4272997 0 0 0
>> 48 3852574 0 4162653 0 0 0
>> 49 391449390 0 3935202 0 0 0
>> 50 791388200 0 4290738 0 0 0
>> 51 127107573 0 3907750 0 0 0
>> 52 115622148 0 4012843 0 0 0
>> 53 71098871 0 4200625 0 0 0
>> 54 305121466 0 4365614 0 0 0
>> 55 10914257 0 4369426 0 0 0
>>
>>
>>
>>
>> PerfTop: 108490 irqs/sec kernel:99.6% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> -------------------------------------------------------------------
>> -------------------------------------------------------------------
>> -------------------------------------------------------------------
>> ------
>>
>> 26.78% [kernel] [k] queued_spin_lock_slowpath
>> 9.09% [kernel] [k] mlx5e_skb_from_cqe_linear
>> 4.94% [kernel] [k] mlx5e_sq_xmit
>> 3.63% [kernel] [k] memcpy_erms
>> 3.30% [kernel] [k] fib_table_lookup
>> 3.26% [kernel] [k] build_skb
>> 2.41% [kernel] [k] mlx5e_poll_tx_cq
>> 2.11% [kernel] [k] get_page_from_freelist
>> 1.51% [kernel] [k] vlan_do_receive
>> 1.51% [kernel] [k] _raw_spin_lock
>> 1.43% [kernel] [k] __dev_queue_xmit
>> 1.41% [kernel] [k] dev_gro_receive
>> 1.34% [kernel] [k] mlx5e_poll_rx_cq
>> 1.26% [kernel] [k] tcp_gro_receive
>> 1.21% [kernel] [k] free_one_page
>> 1.13% [kernel] [k] swiotlb_map_page
>> 1.13% [kernel] [k] mlx5e_post_rx_wqes
>> 1.05% [kernel] [k] pfifo_fast_dequeue
>> 1.05% [kernel] [k] mlx5e_handle_rx_cqe
>> 1.03% [kernel] [k] ip_finish_output2
>> 1.02% [kernel] [k] ipt_do_table
>> 0.96% [kernel] [k] inet_gro_receive
>> 0.91% [kernel] [k] mlx5_eq_int
>> 0.88% [kernel] [k] __slab_free.isra.79
>> 0.86% [kernel] [k] __build_skb
>> 0.84% [kernel] [k] page_frag_free
>> 0.76% [kernel] [k] skb_release_data
>> 0.75% [kernel] [k] __netif_receive_skb_core
>> 0.75% [kernel] [k] irq_entries_start
>> 0.71% [kernel] [k] ip_route_input_rcu
>> 0.65% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.56% [kernel] [k] ip_forward
>> 0.56% [kernel] [k] __memcpy
>> 0.52% [kernel] [k] kmem_cache_alloc
>> 0.52% [kernel] [k] kmem_cache_free_bulk
>> 0.49% [kernel] [k] mlx5e_page_release
>> 0.47% [kernel] [k] netif_skb_features
>> 0.47% [kernel] [k] mlx5e_build_rx_skb
>> 0.47% [kernel] [k] dev_hard_start_xmit
>> 0.43% [kernel] [k] __page_pool_put_page
>> 0.43% [kernel] [k] __netif_schedule
>> 0.43% [kernel] [k] mlx5e_xmit
>> 0.41% [kernel] [k] __qdisc_run
>> 0.41% [kernel] [k] validate_xmit_skb.isra.142
>> 0.41% [kernel] [k] swiotlb_unmap_page
>> 0.40% [kernel] [k] inet_lookup_ifaddr_rcu
>> 0.34% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>> 0.34% [kernel] [k] tcp4_gro_receive
>> 0.29% [kernel] [k] _raw_spin_lock_irqsave
>> 0.29% [kernel] [k] napi_consume_skb
>> 0.29% [kernel] [k] skb_gro_receive
>> 0.29% [kernel] [k] ___slab_alloc.isra.80
>> 0.27% [kernel] [k] eth_type_trans
>> 0.26% [kernel] [k] __free_pages_ok
>> 0.26% [kernel] [k] __get_xps_queue_idx
>> 0.24% [kernel] [k] _raw_spin_trylock
>> 0.23% [kernel] [k] __local_bh_enable_ip
>> 0.22% [kernel] [k] pfifo_fast_enqueue
>> 0.21% [kernel] [k] tasklet_action_common.isra.21
>> 0.21% [kernel] [k] sch_direct_xmit
>> 0.21% [kernel] [k] skb_network_protocol
>> 0.21% [kernel] [k] kmem_cache_free
>> 0.20% [kernel] [k] netdev_pick_tx
>> 0.18% [kernel] [k] napi_gro_complete
>> 0.18% [kernel] [k] __sched_text_start
>> 0.18% [kernel] [k] mlx5e_xdp_handle
>> 0.17% [kernel] [k] ip_finish_output
>> 0.16% [kernel] [k] napi_gro_flush
>> 0.16% [kernel] [k] vlan_passthru_hard_header
>> 0.16% [kernel] [k] skb_segment
>> 0.15% [kernel] [k] __alloc_pages_nodemask
>> 0.15% [kernel] [k] mlx5e_features_check
>> 0.15% [kernel] [k] mlx5e_napi_poll
>> 0.15% [kernel] [k] napi_gro_receive
>> 0.14% [kernel] [k] fib_validate_source
>> 0.14% [kernel] [k] _raw_spin_lock_irq
>> 0.14% [kernel] [k] inet_gro_complete
>> 0.14% [kernel] [k] get_partial_node.isra.78
>> 0.13% [kernel] [k] napi_complete_done
>> 0.13% [kernel] [k] ip_rcv_finish_core.isra.17
>> 0.13% [kernel] [k] cmd_exec
>>
>>
>>
>> ethtool -S enp175s0f1
>> NIC statistics:
>> rx_packets: 173730800927
>> rx_bytes: 99827422751332
>> tx_packets: 142532009512
>> tx_bytes: 184633045911222
>> tx_tso_packets: 25989113891
>> tx_tso_bytes: 132933363384458
>> tx_tso_inner_packets: 0
>> tx_tso_inner_bytes: 0
>> tx_added_vlan_packets: 74630239613
>> tx_nop: 2029817748
>> rx_lro_packets: 0
>> rx_lro_bytes: 0
>> rx_ecn_mark: 0
>> rx_removed_vlan_packets: 173730800927
>> rx_csum_unnecessary: 0
>> rx_csum_none: 434357
>> rx_csum_complete: 173730366570
>> rx_csum_unnecessary_inner: 0
>> rx_xdp_drop: 0
>> rx_xdp_redirect: 0
>> rx_xdp_tx_xmit: 0
>> rx_xdp_tx_full: 0
>> rx_xdp_tx_err: 0
>> rx_xdp_tx_cqe: 0
>> tx_csum_none: 38260960853
>> tx_csum_partial: 36369278774
>> tx_csum_partial_inner: 0
>> tx_queue_stopped: 1
>> tx_queue_dropped: 0
>> tx_xmit_more: 748638099
>> tx_recover: 0
>> tx_cqes: 73881645031
>> tx_queue_wake: 1
>> tx_udp_seg_rem: 0
>> tx_cqe_err: 0
>> tx_xdp_xmit: 0
>> tx_xdp_full: 0
>> tx_xdp_err: 0
>> tx_xdp_cqes: 0
>> rx_wqe_err: 0
>> rx_mpwqe_filler_cqes: 0
>> rx_mpwqe_filler_strides: 0
>> rx_buff_alloc_err: 0
>> rx_cqe_compress_blks: 0
>> rx_cqe_compress_pkts: 0
> If this is a pcie bottleneck it might be useful to enable CQE
> compression (to reduce PCIe completion descriptors transactions)
> you should see the above rx_cqe_compress_pkts increasing when enabled.
>
> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
> $ ethtool --show-priv-flags enp175s0f1
> Private flags for p6p1:
> rx_cqe_moder : on
> cqe_moder : off
> rx_cqe_compress : on
> ...
>
> try this on both interfaces.
Done
ethtool --show-priv-flags enp175s0f1
Private flags for enp175s0f1:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : on
rx_striding_rq : off
rx_no_csum_complete: off
ethtool --show-priv-flags enp175s0f0
Private flags for enp175s0f0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : on
rx_striding_rq : off
rx_no_csum_complete: off
>
>> rx_page_reuse: 0
>> rx_cache_reuse: 14441066823
>> rx_cache_full: 51126004413
>> rx_cache_empty: 21297344082
>> rx_cache_busy: 51127247487
>> rx_cache_waive: 21298322293
>> rx_congst_umr: 0
>> rx_arfs_err: 0
>> ch_events: 24603119858
>> ch_poll: 25180949074
>> ch_arm: 24480437587
>> ch_aff_change: 75
>> ch_eq_rearm: 0
>> rx_out_of_buffer: 669685
> comparing this to rx_vport_unicast_packets, it is a very small
> percentage of dropped packets due to stalled rx cpu, so rx cpu is not a
> bottleneck, at least for the driver rx rings.
>
>> rx_if_down_packets: 61
>> rx_vport_unicast_packets: 173731641945
>> rx_vport_unicast_bytes: 100522745036693
>> tx_vport_unicast_packets: 142531901313
>> tx_vport_unicast_bytes: 185189071776429
>> rx_vport_multicast_packets: 100360886
>> rx_vport_multicast_bytes: 6639236688
>> tx_vport_multicast_packets: 32837
>> tx_vport_multicast_bytes: 2978810
>> rx_vport_broadcast_packets: 44854
>> rx_vport_broadcast_bytes: 6313510
>> tx_vport_broadcast_packets: 72258
>> tx_vport_broadcast_bytes: 4335480
>> rx_vport_rdma_unicast_packets: 0
>> rx_vport_rdma_unicast_bytes: 0
>> tx_vport_rdma_unicast_packets: 0
>> tx_vport_rdma_unicast_bytes: 0
>> rx_vport_rdma_multicast_packets: 0
>> rx_vport_rdma_multicast_bytes: 0
>> tx_vport_rdma_multicast_packets: 0
>> tx_vport_rdma_multicast_bytes: 0
>> tx_packets_phy: 142532004669
>> rx_packets_phy: 173980375752
>> rx_crc_errors_phy: 0
>> tx_bytes_phy: 185759204762903
>> rx_bytes_phy: 101326109361379
>> tx_multicast_phy: 32837
>> tx_broadcast_phy: 72258
>> rx_multicast_phy: 100360885
>> rx_broadcast_phy: 44854
>> rx_in_range_len_errors_phy: 2
>> rx_out_of_range_len_phy: 0
>> rx_oversize_pkts_phy: 59
>> rx_symbol_err_phy: 0
>> tx_mac_control_phy: 0
>> rx_mac_control_phy: 0
>> rx_unsupported_op_phy: 0
>> rx_pause_ctrl_phy: 0
>> tx_pause_ctrl_phy: 0
>> rx_discards_phy: 148328738
>> tx_discards_phy: 0
>> tx_errors_phy: 0
>> rx_undersize_pkts_phy: 0
>> rx_fragments_phy: 0
>> rx_jabbers_phy: 0
>> rx_64_bytes_phy: 36551843112
>> rx_65_to_127_bytes_phy: 65102131735
>> rx_128_to_255_bytes_phy: 5755731137
>> rx_256_to_511_bytes_phy: 2475619839
>> rx_512_to_1023_bytes_phy: 2826971156
>> rx_1024_to_1518_bytes_phy: 42474023107
>> rx_1519_to_2047_bytes_phy: 18794051270
>> rx_2048_to_4095_bytes_phy: 0
>> rx_4096_to_8191_bytes_phy: 0
>> rx_8192_to_10239_bytes_phy: 0
>> link_down_events_phy: 0
>> rx_pcs_symbol_err_phy: 0
>> rx_corrected_bits_phy: 0
>> rx_pci_signal_integrity: 0
>> tx_pci_signal_integrity: 48
>> rx_prio0_bytes: 101316322498995
>> rx_prio0_packets: 173711151686
>> tx_prio0_bytes: 185759176566814
>> tx_prio0_packets: 142531983704
>> rx_prio1_bytes: 47062768
>> rx_prio1_packets: 228932
>> tx_prio1_bytes: 0
>> tx_prio1_packets: 0
>> rx_prio2_bytes: 12434759
>> rx_prio2_packets: 83773
>> tx_prio2_bytes: 0
>> tx_prio2_packets: 0
>> rx_prio3_bytes: 288843134
>> rx_prio3_packets: 982102
>> tx_prio3_bytes: 0
>> tx_prio3_packets: 0
>> rx_prio4_bytes: 699797236
>> rx_prio4_packets: 8109231
>> tx_prio4_bytes: 0
>> tx_prio4_packets: 0
>> rx_prio5_bytes: 1385386738
>> rx_prio5_packets: 9661187
>> tx_prio5_bytes: 0
>> tx_prio5_packets: 0
>> rx_prio6_bytes: 317092102
>> rx_prio6_packets: 1951538
>> tx_prio6_bytes: 0
>> tx_prio6_packets: 0
>> rx_prio7_bytes: 7015734695
>> rx_prio7_packets: 99847456
>> tx_prio7_bytes: 0
>> tx_prio7_packets: 0
>> module_unplug: 0
>> module_bus_stuck: 0
>> module_high_temp: 0
>> module_bad_shorted: 0
>> ch0_events: 936264703
>> ch0_poll: 963766474
>> ch0_arm: 930246079
>> ch0_aff_change: 0
>> ch0_eq_rearm: 0
>> ch1_events: 869408429
>> ch1_poll: 896099392
>> ch1_arm: 864336861
>> ch1_aff_change: 0
>> ch1_eq_rearm: 0
>> ch2_events: 843345698
>> ch2_poll: 869749522
>> ch2_arm: 838186113
>> ch2_aff_change: 2
>> ch2_eq_rearm: 0
>> ch3_events: 850261340
>> ch3_poll: 876721111
>> ch3_arm: 845295235
>> ch3_aff_change: 3
>> ch3_eq_rearm: 0
>> ch4_events: 974985780
>> ch4_poll: 997781915
>> ch4_arm: 969618250
>> ch4_aff_change: 3
>> ch4_eq_rearm: 0
>> ch5_events: 888559089
>> ch5_poll: 912783615
>> ch5_arm: 883826078
>> ch5_aff_change: 2
>> ch5_eq_rearm: 0
>> ch6_events: 873730730
>> ch6_poll: 899635752
>> ch6_arm: 868677574
>> ch6_aff_change: 4
>> ch6_eq_rearm: 0
>> ch7_events: 873478411
>> ch7_poll: 899216716
>> ch7_arm: 868693645
>> ch7_aff_change: 3
>> ch7_eq_rearm: 0
>> ch8_events: 871900967
>> ch8_poll: 898575518
>> ch8_arm: 866763693
>> ch8_aff_change: 3
>> ch8_eq_rearm: 0
>> ch9_events: 880325565
>> ch9_poll: 904983269
>> ch9_arm: 875643922
>> ch9_aff_change: 2
>> ch9_eq_rearm: 0
>> ch10_events: 889919775
>> ch10_poll: 915335809
>> ch10_arm: 885110225
>> ch10_aff_change: 4
>> ch10_eq_rearm: 0
>> ch11_events: 962709175
>> ch11_poll: 983963451
>> ch11_arm: 958117526
>> ch11_aff_change: 2
>> ch11_eq_rearm: 0
>> ch12_events: 941333837
>> ch12_poll: 964625523
>> ch12_arm: 936409706
>> ch12_aff_change: 2
>> ch12_eq_rearm: 0
>> ch13_events: 914996974
>> ch13_poll: 937441049
>> ch13_arm: 910478393
>> ch13_aff_change: 4
>> ch13_eq_rearm: 0
>> ch14_events: 888050001
>> ch14_poll: 911818008
>> ch14_arm: 883465035
>> ch14_aff_change: 4
>> ch14_eq_rearm: 0
>> ch15_events: 947547704
>> ch15_poll: 969073194
>> ch15_arm: 942686515
>> ch15_aff_change: 4
>> ch15_eq_rearm: 0
>> ch16_events: 825804904
>> ch16_poll: 840630747
>> ch16_arm: 822227488
>> ch16_aff_change: 2
>> ch16_eq_rearm: 0
>> ch17_events: 861673823
>> ch17_poll: 874754041
>> ch17_arm: 858520448
>> ch17_aff_change: 2
>> ch17_eq_rearm: 0
>> ch18_events: 879413440
>> ch18_poll: 893962529
>> ch18_arm: 875983204
>> ch18_aff_change: 4
>> ch18_eq_rearm: 0
>> ch19_events: 896073709
>> ch19_poll: 909216857
>> ch19_arm: 893022121
>> ch19_aff_change: 4
>> ch19_eq_rearm: 0
>> ch20_events: 865188535
>> ch20_poll: 880692345
>> ch20_arm: 861440265
>> ch20_aff_change: 3
>> ch20_eq_rearm: 0
>> ch21_events: 862709303
>> ch21_poll: 878104242
>> ch21_arm: 859041767
>> ch21_aff_change: 2
>> ch21_eq_rearm: 0
>> ch22_events: 887720551
>> ch22_poll: 904122074
>> ch22_arm: 883983794
>> ch22_aff_change: 2
>> ch22_eq_rearm: 0
>> ch23_events: 813355027
>> ch23_poll: 828074467
>> ch23_arm: 809912398
>> ch23_aff_change: 4
>> ch23_eq_rearm: 0
>> ch24_events: 822366675
>> ch24_poll: 839917937
>> ch24_arm: 818422754
>> ch24_aff_change: 2
>> ch24_eq_rearm: 0
>> ch25_events: 826642292
>> ch25_poll: 842630121
>> ch25_arm: 822642618
>> ch25_aff_change: 2
>> ch25_eq_rearm: 0
>> ch26_events: 826392584
>> ch26_poll: 843406973
>> ch26_arm: 822455000
>> ch26_aff_change: 3
>> ch26_eq_rearm: 0
>> ch27_events: 828960899
>> ch27_poll: 843866518
>> ch27_arm: 825230937
>> ch27_aff_change: 3
>> ch27_eq_rearm: 0
>> ch28_events: 7
>> ch28_poll: 7
>> ch28_arm: 7
>> ch28_aff_change: 0
>> ch28_eq_rearm: 0
>> ch29_events: 4
>> ch29_poll: 4
>> ch29_arm: 4
>> ch29_aff_change: 0
>> ch29_eq_rearm: 0
>> ch30_events: 4
>> ch30_poll: 4
>> ch30_arm: 4
>> ch30_aff_change: 0
>> ch30_eq_rearm: 0
>> ch31_events: 4
>> ch31_poll: 4
>> ch31_arm: 4
>> ch31_aff_change: 0
>> ch31_eq_rearm: 0
>> ch32_events: 4
>> ch32_poll: 4
>> ch32_arm: 4
>> ch32_aff_change: 0
>> ch32_eq_rearm: 0
>> ch33_events: 4
>> ch33_poll: 4
>> ch33_arm: 4
>> ch33_aff_change: 0
>> ch33_eq_rearm: 0
>> ch34_events: 4
>> ch34_poll: 4
>> ch34_arm: 4
>> ch34_aff_change: 0
>> ch34_eq_rearm: 0
>> ch35_events: 4
>> ch35_poll: 4
>> ch35_arm: 4
>> ch35_aff_change: 0
>> ch35_eq_rearm: 0
>> ch36_events: 4
>> ch36_poll: 4
>> ch36_arm: 4
>> ch36_aff_change: 0
>> ch36_eq_rearm: 0
>> ch37_events: 4
>> ch37_poll: 4
>> ch37_arm: 4
>> ch37_aff_change: 0
>> ch37_eq_rearm: 0
>> ch38_events: 4
>> ch38_poll: 4
>> ch38_arm: 4
>> ch38_aff_change: 0
>> ch38_eq_rearm: 0
>> ch39_events: 4
>> ch39_poll: 4
>> ch39_arm: 4
>> ch39_aff_change: 0
>> ch39_eq_rearm: 0
>> ch40_events: 4
>> ch40_poll: 4
>> ch40_arm: 4
>> ch40_aff_change: 0
>> ch40_eq_rearm: 0
>> ch41_events: 4
>> ch41_poll: 4
>> ch41_arm: 4
>> ch41_aff_change: 0
>> ch41_eq_rearm: 0
>> ch42_events: 4
>> ch42_poll: 4
>> ch42_arm: 4
>> ch42_aff_change: 0
>> ch42_eq_rearm: 0
>> ch43_events: 4
>> ch43_poll: 4
>> ch43_arm: 4
>> ch43_aff_change: 0
>> ch43_eq_rearm: 0
>> ch44_events: 4
>> ch44_poll: 4
>> ch44_arm: 4
>> ch44_aff_change: 0
>> ch44_eq_rearm: 0
>> ch45_events: 4
>> ch45_poll: 4
>> ch45_arm: 4
>> ch45_aff_change: 0
>> ch45_eq_rearm: 0
>> ch46_events: 4
>> ch46_poll: 4
>> ch46_arm: 4
>> ch46_aff_change: 0
>> ch46_eq_rearm: 0
>> ch47_events: 4
>> ch47_poll: 4
>> ch47_arm: 4
>> ch47_aff_change: 0
>> ch47_eq_rearm: 0
>> ch48_events: 4
>> ch48_poll: 4
>> ch48_arm: 4
>> ch48_aff_change: 0
>> ch48_eq_rearm: 0
>> ch49_events: 4
>> ch49_poll: 4
>> ch49_arm: 4
>> ch49_aff_change: 0
>> ch49_eq_rearm: 0
>> ch50_events: 4
>> ch50_poll: 4
>> ch50_arm: 4
>> ch50_aff_change: 0
>> ch50_eq_rearm: 0
>> ch51_events: 4
>> ch51_poll: 4
>> ch51_arm: 4
>> ch51_aff_change: 0
>> ch51_eq_rearm: 0
>> ch52_events: 4
>> ch52_poll: 4
>> ch52_arm: 4
>> ch52_aff_change: 0
>> ch52_eq_rearm: 0
>> ch53_events: 4
>> ch53_poll: 4
>> ch53_arm: 4
>> ch53_aff_change: 0
>> ch53_eq_rearm: 0
>> ch54_events: 4
>> ch54_poll: 4
>> ch54_arm: 4
>> ch54_aff_change: 0
>> ch54_eq_rearm: 0
>> ch55_events: 4
>> ch55_poll: 4
>> ch55_arm: 4
>> ch55_aff_change: 0
>> ch55_eq_rearm: 0
>> rx0_packets: 7284057433
>> rx0_bytes: 4330611281319
>> rx0_csum_complete: 7283623076
>> rx0_csum_unnecessary: 0
>> rx0_csum_unnecessary_inner: 0
>> rx0_csum_none: 434357
>> rx0_xdp_drop: 0
>> rx0_xdp_redirect: 0
>> rx0_lro_packets: 0
>> rx0_lro_bytes: 0
>> rx0_ecn_mark: 0
>> rx0_removed_vlan_packets: 7284057433
>> rx0_wqe_err: 0
>> rx0_mpwqe_filler_cqes: 0
>> rx0_mpwqe_filler_strides: 0
>> rx0_buff_alloc_err: 0
>> rx0_cqe_compress_blks: 0
>> rx0_cqe_compress_pkts: 0
>> rx0_page_reuse: 0
>> rx0_cache_reuse: 1989731589
>> rx0_cache_full: 28213297
>> rx0_cache_empty: 1624089822
>> rx0_cache_busy: 28213961
>> rx0_cache_waive: 1624083610
>> rx0_congst_umr: 0
>> rx0_arfs_err: 0
>> rx0_xdp_tx_xmit: 0
>> rx0_xdp_tx_full: 0
>> rx0_xdp_tx_err: 0
>> rx0_xdp_tx_cqes: 0
>> rx1_packets: 6691319211
>> rx1_bytes: 3799580210608
>> rx1_csum_complete: 6691319211
>> rx1_csum_unnecessary: 0
>> rx1_csum_unnecessary_inner: 0
>> rx1_csum_none: 0
>> rx1_xdp_drop: 0
>> rx1_xdp_redirect: 0
>> rx1_lro_packets: 0
>> rx1_lro_bytes: 0
>> rx1_ecn_mark: 0
>> rx1_removed_vlan_packets: 6691319211
>> rx1_wqe_err: 0
>> rx1_mpwqe_filler_cqes: 0
>> rx1_mpwqe_filler_strides: 0
>> rx1_buff_alloc_err: 0
>> rx1_cqe_compress_blks: 0
>> rx1_cqe_compress_pkts: 0
>> rx1_page_reuse: 0
>> rx1_cache_reuse: 2270019
>> rx1_cache_full: 3343389331
>> rx1_cache_empty: 6656
>> rx1_cache_busy: 3343389585
>> rx1_cache_waive: 0
>> rx1_congst_umr: 0
>> rx1_arfs_err: 0
>> rx1_xdp_tx_xmit: 0
>> rx1_xdp_tx_full: 0
>> rx1_xdp_tx_err: 0
>> rx1_xdp_tx_cqes: 0
>> rx2_packets: 6618370416
>> rx2_bytes: 3762508364015
>> rx2_csum_complete: 6618370416
>> rx2_csum_unnecessary: 0
>> rx2_csum_unnecessary_inner: 0
>> rx2_csum_none: 0
>> rx2_xdp_drop: 0
>> rx2_xdp_redirect: 0
>> rx2_lro_packets: 0
>> rx2_lro_bytes: 0
>> rx2_ecn_mark: 0
>> rx2_removed_vlan_packets: 6618370416
>> rx2_wqe_err: 0
>> rx2_mpwqe_filler_cqes: 0
>> rx2_mpwqe_filler_strides: 0
>> rx2_buff_alloc_err: 0
>> rx2_cqe_compress_blks: 0
>> rx2_cqe_compress_pkts: 0
>> rx2_page_reuse: 0
>> rx2_cache_reuse: 111419328
>> rx2_cache_full: 1807563903
>> rx2_cache_empty: 1390208158
>> rx2_cache_busy: 1807564378
>> rx2_cache_waive: 1390201722
>> rx2_congst_umr: 0
>> rx2_arfs_err: 0
>> rx2_xdp_tx_xmit: 0
>> rx2_xdp_tx_full: 0
>> rx2_xdp_tx_err: 0
>> rx2_xdp_tx_cqes: 0
>> rx3_packets: 6665308976
>> rx3_bytes: 3828546206006
>> rx3_csum_complete: 6665308976
>> rx3_csum_unnecessary: 0
>> rx3_csum_unnecessary_inner: 0
>> rx3_csum_none: 0
>> rx3_xdp_drop: 0
>> rx3_xdp_redirect: 0
>> rx3_lro_packets: 0
>> rx3_lro_bytes: 0
>> rx3_ecn_mark: 0
>> rx3_removed_vlan_packets: 6665308976
>> rx3_wqe_err: 0
>> rx3_mpwqe_filler_cqes: 0
>> rx3_mpwqe_filler_strides: 0
>> rx3_buff_alloc_err: 0
>> rx3_cqe_compress_blks: 0
>> rx3_cqe_compress_pkts: 0
>> rx3_page_reuse: 0
>> rx3_cache_reuse: 215779091
>> rx3_cache_full: 1720040649
>> rx3_cache_empty: 1396840926
>> rx3_cache_busy: 1720041127
>> rx3_cache_waive: 1396834493
>> rx3_congst_umr: 0
>> rx3_arfs_err: 0
>> rx3_xdp_tx_xmit: 0
>> rx3_xdp_tx_full: 0
>> rx3_xdp_tx_err: 0
>> rx3_xdp_tx_cqes: 0
>> rx4_packets: 6764448165
>> rx4_bytes: 3883101339142
>> rx4_csum_complete: 6764448165
>> rx4_csum_unnecessary: 0
>> rx4_csum_unnecessary_inner: 0
>> rx4_csum_none: 0
>> rx4_xdp_drop: 0
>> rx4_xdp_redirect: 0
>> rx4_lro_packets: 0
>> rx4_lro_bytes: 0
>> rx4_ecn_mark: 0
>> rx4_removed_vlan_packets: 6764448165
>> rx4_wqe_err: 0
>> rx4_mpwqe_filler_cqes: 0
>> rx4_mpwqe_filler_strides: 0
>> rx4_buff_alloc_err: 0
>> rx4_cqe_compress_blks: 0
>> rx4_cqe_compress_pkts: 0
>> rx4_page_reuse: 0
>> rx4_cache_reuse: 1930710653
>> rx4_cache_full: 6490815
>> rx4_cache_empty: 1445028605
>> rx4_cache_busy: 6491478
>> rx4_cache_waive: 1445022392
>> rx4_congst_umr: 0
>> rx4_arfs_err: 0
>> rx4_xdp_tx_xmit: 0
>> rx4_xdp_tx_full: 0
>> rx4_xdp_tx_err: 0
>> rx4_xdp_tx_cqes: 0
>> rx5_packets: 6736853264
>> rx5_bytes: 3925186068552
>> rx5_csum_complete: 6736853264
>> rx5_csum_unnecessary: 0
>> rx5_csum_unnecessary_inner: 0
>> rx5_csum_none: 0
>> rx5_xdp_drop: 0
>> rx5_xdp_redirect: 0
>> rx5_lro_packets: 0
>> rx5_lro_bytes: 0
>> rx5_ecn_mark: 0
>> rx5_removed_vlan_packets: 6736853264
>> rx5_wqe_err: 0
>> rx5_mpwqe_filler_cqes: 0
>> rx5_mpwqe_filler_strides: 0
>> rx5_buff_alloc_err: 0
>> rx5_cqe_compress_blks: 0
>> rx5_cqe_compress_pkts: 0
>> rx5_page_reuse: 0
>> rx5_cache_reuse: 7283914
>> rx5_cache_full: 3361142463
>> rx5_cache_empty: 6656
>> rx5_cache_busy: 3361142718
>> rx5_cache_waive: 0
>> rx5_congst_umr: 0
>> rx5_arfs_err: 0
>> rx5_xdp_tx_xmit: 0
>> rx5_xdp_tx_full: 0
>> rx5_xdp_tx_err: 0
>> rx5_xdp_tx_cqes: 0
>> rx6_packets: 6751588828
>> rx6_bytes: 3860537598885
>> rx6_csum_complete: 6751588828
>> rx6_csum_unnecessary: 0
>> rx6_csum_unnecessary_inner: 0
>> rx6_csum_none: 0
>> rx6_xdp_drop: 0
>> rx6_xdp_redirect: 0
>> rx6_lro_packets: 0
>> rx6_lro_bytes: 0
>> rx6_ecn_mark: 0
>> rx6_removed_vlan_packets: 6751588828
>> rx6_wqe_err: 0
>> rx6_mpwqe_filler_cqes: 0
>> rx6_mpwqe_filler_strides: 0
>> rx6_buff_alloc_err: 0
>> rx6_cqe_compress_blks: 0
>> rx6_cqe_compress_pkts: 0
>> rx6_page_reuse: 0
>> rx6_cache_reuse: 96032126
>> rx6_cache_full: 1857890923
>> rx6_cache_empty: 1421877543
>> rx6_cache_busy: 1857891399
>> rx6_cache_waive: 1421871110
>> rx6_congst_umr: 0
>> rx6_arfs_err: 0
>> rx6_xdp_tx_xmit: 0
>> rx6_xdp_tx_full: 0
>> rx6_xdp_tx_err: 0
>> rx6_xdp_tx_cqes: 0
>> rx7_packets: 6935300074
>> rx7_bytes: 4004713524388
>> rx7_csum_complete: 6935300074
>> rx7_csum_unnecessary: 0
>> rx7_csum_unnecessary_inner: 0
>> rx7_csum_none: 0
>> rx7_xdp_drop: 0
>> rx7_xdp_redirect: 0
>> rx7_lro_packets: 0
>> rx7_lro_bytes: 0
>> rx7_ecn_mark: 0
>> rx7_removed_vlan_packets: 6935300074
>> rx7_wqe_err: 0
>> rx7_mpwqe_filler_cqes: 0
>> rx7_mpwqe_filler_strides: 0
>> rx7_buff_alloc_err: 0
>> rx7_cqe_compress_blks: 0
>> rx7_cqe_compress_pkts: 0
>> rx7_page_reuse: 0
>> rx7_cache_reuse: 17555187
>> rx7_cache_full: 3450094595
>> rx7_cache_empty: 6656
>> rx7_cache_busy: 3450094849
>> rx7_cache_waive: 0
>> rx7_congst_umr: 0
>> rx7_arfs_err: 0
>> rx7_xdp_tx_xmit: 0
>> rx7_xdp_tx_full: 0
>> rx7_xdp_tx_err: 0
>> rx7_xdp_tx_cqes: 0
>> rx8_packets: 6678640094
>> rx8_bytes: 3783722686028
>> rx8_csum_complete: 6678640094
>> rx8_csum_unnecessary: 0
>> rx8_csum_unnecessary_inner: 0
>> rx8_csum_none: 0
>> rx8_xdp_drop: 0
>> rx8_xdp_redirect: 0
>> rx8_lro_packets: 0
>> rx8_lro_bytes: 0
>> rx8_ecn_mark: 0
>> rx8_removed_vlan_packets: 6678640094
>> rx8_wqe_err: 0
>> rx8_mpwqe_filler_cqes: 0
>> rx8_mpwqe_filler_strides: 0
>> rx8_buff_alloc_err: 0
>> rx8_cqe_compress_blks: 0
>> rx8_cqe_compress_pkts: 0
>> rx8_page_reuse: 0
>> rx8_cache_reuse: 71006578
>> rx8_cache_full: 1879380649
>> rx8_cache_empty: 1388938999
>> rx8_cache_busy: 1879381123
>> rx8_cache_waive: 1388932565
>> rx8_congst_umr: 0
>> rx8_arfs_err: 0
>> rx8_xdp_tx_xmit: 0
>> rx8_xdp_tx_full: 0
>> rx8_xdp_tx_err: 0
>> rx8_xdp_tx_cqes: 0
>> rx9_packets: 6709855557
>> rx9_bytes: 3849522227880
>> rx9_csum_complete: 6709855557
>> rx9_csum_unnecessary: 0
>> rx9_csum_unnecessary_inner: 0
>> rx9_csum_none: 0
>> rx9_xdp_drop: 0
>> rx9_xdp_redirect: 0
>> rx9_lro_packets: 0
>> rx9_lro_bytes: 0
>> rx9_ecn_mark: 0
>> rx9_removed_vlan_packets: 6709855557
>> rx9_wqe_err: 0
>> rx9_mpwqe_filler_cqes: 0
>> rx9_mpwqe_filler_strides: 0
>> rx9_buff_alloc_err: 0
>> rx9_cqe_compress_blks: 0
>> rx9_cqe_compress_pkts: 0
>> rx9_page_reuse: 0
>> rx9_cache_reuse: 108980215
>> rx9_cache_full: 1822730121
>> rx9_cache_empty: 1423223623
>> rx9_cache_busy: 1822730594
>> rx9_cache_waive: 1423217187
>> rx9_congst_umr: 0
>> rx9_arfs_err: 0
>> rx9_xdp_tx_xmit: 0
>> rx9_xdp_tx_full: 0
>> rx9_xdp_tx_err: 0
>> rx9_xdp_tx_cqes: 0
>> rx10_packets: 6761861066
>> rx10_bytes: 3816266733385
>> rx10_csum_complete: 6761861066
>> rx10_csum_unnecessary: 0
>> rx10_csum_unnecessary_inner: 0
>> rx10_csum_none: 0
>> rx10_xdp_drop: 0
>> rx10_xdp_redirect: 0
>> rx10_lro_packets: 0
>> rx10_lro_bytes: 0
>> rx10_ecn_mark: 0
>> rx10_removed_vlan_packets: 6761861066
>> rx10_wqe_err: 0
>> rx10_mpwqe_filler_cqes: 0
>> rx10_mpwqe_filler_strides: 0
>> rx10_buff_alloc_err: 0
>> rx10_cqe_compress_blks: 0
>> rx10_cqe_compress_pkts: 0
>> rx10_page_reuse: 0
>> rx10_cache_reuse: 3489300
>> rx10_cache_full: 3377440977
>> rx10_cache_empty: 6656
>> rx10_cache_busy: 3377441216
>> rx10_cache_waive: 0
>> rx10_congst_umr: 0
>> rx10_arfs_err: 0
>> rx10_xdp_tx_xmit: 0
>> rx10_xdp_tx_full: 0
>> rx10_xdp_tx_err: 0
>> rx10_xdp_tx_cqes: 0
>> rx11_packets: 6868113938
>> rx11_bytes: 4048196300710
>> rx11_csum_complete: 6868113938
>> rx11_csum_unnecessary: 0
>> rx11_csum_unnecessary_inner: 0
>> rx11_csum_none: 0
>> rx11_xdp_drop: 0
>> rx11_xdp_redirect: 0
>> rx11_lro_packets: 0
>> rx11_lro_bytes: 0
>> rx11_ecn_mark: 0
>> rx11_removed_vlan_packets: 6868113938
>> rx11_wqe_err: 0
>> rx11_mpwqe_filler_cqes: 0
>> rx11_mpwqe_filler_strides: 0
>> rx11_buff_alloc_err: 0
>> rx11_cqe_compress_blks: 0
>> rx11_cqe_compress_pkts: 0
>> rx11_page_reuse: 0
>> rx11_cache_reuse: 1948516819
>> rx11_cache_full: 17132157
>> rx11_cache_empty: 1468413985
>> rx11_cache_busy: 17132820
>> rx11_cache_waive: 1468407772
>> rx11_congst_umr: 0
>> rx11_arfs_err: 0
>> rx11_xdp_tx_xmit: 0
>> rx11_xdp_tx_full: 0
>> rx11_xdp_tx_err: 0
>> rx11_xdp_tx_cqes: 0
>> rx12_packets: 6742955386
>> rx12_bytes: 3865747629271
>> rx12_csum_complete: 6742955386
>> rx12_csum_unnecessary: 0
>> rx12_csum_unnecessary_inner: 0
>> rx12_csum_none: 0
>> rx12_xdp_drop: 0
>> rx12_xdp_redirect: 0
>> rx12_lro_packets: 0
>> rx12_lro_bytes: 0
>> rx12_ecn_mark: 0
>> rx12_removed_vlan_packets: 6742955386
>> rx12_wqe_err: 0
>> rx12_mpwqe_filler_cqes: 0
>> rx12_mpwqe_filler_strides: 0
>> rx12_buff_alloc_err: 0
>> rx12_cqe_compress_blks: 0
>> rx12_cqe_compress_pkts: 0
>> rx12_page_reuse: 0
>> rx12_cache_reuse: 30809331
>> rx12_cache_full: 3340668106
>> rx12_cache_empty: 6656
>> rx12_cache_busy: 3340668333
>> rx12_cache_waive: 0
>> rx12_congst_umr: 0
>> rx12_arfs_err: 0
>> rx12_xdp_tx_xmit: 0
>> rx12_xdp_tx_full: 0
>> rx12_xdp_tx_err: 0
>> rx12_xdp_tx_cqes: 0
>> rx13_packets: 6707028036
>> rx13_bytes: 3813462190623
>> rx13_csum_complete: 6707028036
>> rx13_csum_unnecessary: 0
>> rx13_csum_unnecessary_inner: 0
>> rx13_csum_none: 0
>> rx13_xdp_drop: 0
>> rx13_xdp_redirect: 0
>> rx13_lro_packets: 0
>> rx13_lro_bytes: 0
>> rx13_ecn_mark: 0
>> rx13_removed_vlan_packets: 6707028036
>> rx13_wqe_err: 0
>> rx13_mpwqe_filler_cqes: 0
>> rx13_mpwqe_filler_strides: 0
>> rx13_buff_alloc_err: 0
>> rx13_cqe_compress_blks: 0
>> rx13_cqe_compress_pkts: 0
>> rx13_page_reuse: 0
>> rx13_cache_reuse: 14951053
>> rx13_cache_full: 3338562710
>> rx13_cache_empty: 6656
>> rx13_cache_busy: 3338562963
>> rx13_cache_waive: 0
>> rx13_congst_umr: 0
>> rx13_arfs_err: 0
>> rx13_xdp_tx_xmit: 0
>> rx13_xdp_tx_full: 0
>> rx13_xdp_tx_err: 0
>> rx13_xdp_tx_cqes: 0
>> rx14_packets: 6737074410
>> rx14_bytes: 3868905276119
>> rx14_csum_complete: 6737074410
>> rx14_csum_unnecessary: 0
>> rx14_csum_unnecessary_inner: 0
>> rx14_csum_none: 0
>> rx14_xdp_drop: 0
>> rx14_xdp_redirect: 0
>> rx14_lro_packets: 0
>> rx14_lro_bytes: 0
>> rx14_ecn_mark: 0
>> rx14_removed_vlan_packets: 6737074410
>> rx14_wqe_err: 0
>> rx14_mpwqe_filler_cqes: 0
>> rx14_mpwqe_filler_strides: 0
>> rx14_buff_alloc_err: 0
>> rx14_cqe_compress_blks: 0
>> rx14_cqe_compress_pkts: 0
>> rx14_page_reuse: 0
>> rx14_cache_reuse: 967799432
>> rx14_cache_full: 982704312
>> rx14_cache_empty: 1418039639
>> rx14_cache_busy: 982704789
>> rx14_cache_waive: 1418033206
>> rx14_congst_umr: 0
>> rx14_arfs_err: 0
>> rx14_xdp_tx_xmit: 0
>> rx14_xdp_tx_full: 0
>> rx14_xdp_tx_err: 0
>> rx14_xdp_tx_cqes: 0
>> rx15_packets: 6641887441
>> rx15_bytes: 3742874400402
>> rx15_csum_complete: 6641887441
>> rx15_csum_unnecessary: 0
>> rx15_csum_unnecessary_inner: 0
>> rx15_csum_none: 0
>> rx15_xdp_drop: 0
>> rx15_xdp_redirect: 0
>> rx15_lro_packets: 0
>> rx15_lro_bytes: 0
>> rx15_ecn_mark: 0
>> rx15_removed_vlan_packets: 6641887441
>> rx15_wqe_err: 0
>> rx15_mpwqe_filler_cqes: 0
>> rx15_mpwqe_filler_strides: 0
>> rx15_buff_alloc_err: 0
>> rx15_cqe_compress_blks: 0
>> rx15_cqe_compress_pkts: 0
>> rx15_page_reuse: 0
>> rx15_cache_reuse: 1920227538
>> rx15_cache_full: 19386129
>> rx15_cache_empty: 1381335137
>> rx15_cache_busy: 19387693
>> rx15_cache_waive: 1381329825
>> rx15_congst_umr: 0
>> rx15_arfs_err: 0
>> rx15_xdp_tx_xmit: 0
>> rx15_xdp_tx_full: 0
>> rx15_xdp_tx_err: 0
>> rx15_xdp_tx_cqes: 0
>> rx16_packets: 5420472874
>> rx16_bytes: 3079293332581
>> rx16_csum_complete: 5420472874
>> rx16_csum_unnecessary: 0
>> rx16_csum_unnecessary_inner: 0
>> rx16_csum_none: 0
>> rx16_xdp_drop: 0
>> rx16_xdp_redirect: 0
>> rx16_lro_packets: 0
>> rx16_lro_bytes: 0
>> rx16_ecn_mark: 0
>> rx16_removed_vlan_packets: 5420472874
>> rx16_wqe_err: 0
>> rx16_mpwqe_filler_cqes: 0
>> rx16_mpwqe_filler_strides: 0
>> rx16_buff_alloc_err: 0
>> rx16_cqe_compress_blks: 0
>> rx16_cqe_compress_pkts: 0
>> rx16_page_reuse: 0
>> rx16_cache_reuse: 2361079
>> rx16_cache_full: 2707875103
>> rx16_cache_empty: 6656
>> rx16_cache_busy: 2707875349
>> rx16_cache_waive: 0
>> rx16_congst_umr: 0
>> rx16_arfs_err: 0
>> rx16_xdp_tx_xmit: 0
>> rx16_xdp_tx_full: 0
>> rx16_xdp_tx_err: 0
>> rx16_xdp_tx_cqes: 0
>> rx17_packets: 5428380986
>> rx17_bytes: 3080981893118
>> rx17_csum_complete: 5428380986
>> rx17_csum_unnecessary: 0
>> rx17_csum_unnecessary_inner: 0
>> rx17_csum_none: 0
>> rx17_xdp_drop: 0
>> rx17_xdp_redirect: 0
>> rx17_lro_packets: 0
>> rx17_lro_bytes: 0
>> rx17_ecn_mark: 0
>> rx17_removed_vlan_packets: 5428380986
>> rx17_wqe_err: 0
>> rx17_mpwqe_filler_cqes: 0
>> rx17_mpwqe_filler_strides: 0
>> rx17_buff_alloc_err: 0
>> rx17_cqe_compress_blks: 0
>> rx17_cqe_compress_pkts: 0
>> rx17_page_reuse: 0
>> rx17_cache_reuse: 1552266402
>> rx17_cache_full: 5947505
>> rx17_cache_empty: 1155981856
>> rx17_cache_busy: 5948870
>> rx17_cache_waive: 1155976345
>> rx17_congst_umr: 0
>> rx17_arfs_err: 0
>> rx17_xdp_tx_xmit: 0
>> rx17_xdp_tx_full: 0
>> rx17_xdp_tx_err: 0
>> rx17_xdp_tx_cqes: 0
>> rx18_packets: 5529118410
>> rx18_bytes: 3254749573833
>> rx18_csum_complete: 5529118410
>> rx18_csum_unnecessary: 0
>> rx18_csum_unnecessary_inner: 0
>> rx18_csum_none: 0
>> rx18_xdp_drop: 0
>> rx18_xdp_redirect: 0
>> rx18_lro_packets: 0
>> rx18_lro_bytes: 0
>> rx18_ecn_mark: 0
>> rx18_removed_vlan_packets: 5529118410
>> rx18_wqe_err: 0
>> rx18_mpwqe_filler_cqes: 0
>> rx18_mpwqe_filler_strides: 0
>> rx18_buff_alloc_err: 0
>> rx18_cqe_compress_blks: 0
>> rx18_cqe_compress_pkts: 0
>> rx18_page_reuse: 0
>> rx18_cache_reuse: 67438840
>> rx18_cache_full: 1536718472
>> rx18_cache_empty: 1160408072
>> rx18_cache_busy: 1536718932
>> rx18_cache_waive: 1160401638
>> rx18_congst_umr: 0
>> rx18_arfs_err: 0
>> rx18_xdp_tx_xmit: 0
>> rx18_xdp_tx_full: 0
>> rx18_xdp_tx_err: 0
>> rx18_xdp_tx_cqes: 0
>> rx19_packets: 5449932653
>> rx19_bytes: 3148726579411
>> rx19_csum_complete: 5449932653
>> rx19_csum_unnecessary: 0
>> rx19_csum_unnecessary_inner: 0
>> rx19_csum_none: 0
>> rx19_xdp_drop: 0
>> rx19_xdp_redirect: 0
>> rx19_lro_packets: 0
>> rx19_lro_bytes: 0
>> rx19_ecn_mark: 0
>> rx19_removed_vlan_packets: 5449932653
>> rx19_wqe_err: 0
>> rx19_mpwqe_filler_cqes: 0
>> rx19_mpwqe_filler_strides: 0
>> rx19_buff_alloc_err: 0
>> rx19_cqe_compress_blks: 0
>> rx19_cqe_compress_pkts: 0
>> rx19_page_reuse: 0
>> rx19_cache_reuse: 1537841743
>> rx19_cache_full: 9920960
>> rx19_cache_empty: 1177208938
>> rx19_cache_busy: 9922299
>> rx19_cache_waive: 1177203401
>> rx19_congst_umr: 0
>> rx19_arfs_err: 0
>> rx19_xdp_tx_xmit: 0
>> rx19_xdp_tx_full: 0
>> rx19_xdp_tx_err: 0
>> rx19_xdp_tx_cqes: 0
>> rx20_packets: 5407910071
>> rx20_bytes: 3123560861922
>> rx20_csum_complete: 5407910071
>> rx20_csum_unnecessary: 0
>> rx20_csum_unnecessary_inner: 0
>> rx20_csum_none: 0
>> rx20_xdp_drop: 0
>> rx20_xdp_redirect: 0
>> rx20_lro_packets: 0
>> rx20_lro_bytes: 0
>> rx20_ecn_mark: 0
>> rx20_removed_vlan_packets: 5407910071
>> rx20_wqe_err: 0
>> rx20_mpwqe_filler_cqes: 0
>> rx20_mpwqe_filler_strides: 0
>> rx20_buff_alloc_err: 0
>> rx20_cqe_compress_blks: 0
>> rx20_cqe_compress_pkts: 0
>> rx20_page_reuse: 0
>> rx20_cache_reuse: 10255209
>> rx20_cache_full: 2693699571
>> rx20_cache_empty: 6656
>> rx20_cache_busy: 2693699823
>> rx20_cache_waive: 0
>> rx20_congst_umr: 0
>> rx20_arfs_err: 0
>> rx20_xdp_tx_xmit: 0
>> rx20_xdp_tx_full: 0
>> rx20_xdp_tx_err: 0
>> rx20_xdp_tx_cqes: 0
>> rx21_packets: 5417498508
>> rx21_bytes: 3131335892379
>> rx21_csum_complete: 5417498508
>> rx21_csum_unnecessary: 0
>> rx21_csum_unnecessary_inner: 0
>> rx21_csum_none: 0
>> rx21_xdp_drop: 0
>> rx21_xdp_redirect: 0
>> rx21_lro_packets: 0
>> rx21_lro_bytes: 0
>> rx21_ecn_mark: 0
>> rx21_removed_vlan_packets: 5417498508
>> rx21_wqe_err: 0
>> rx21_mpwqe_filler_cqes: 0
>> rx21_mpwqe_filler_strides: 0
>> rx21_buff_alloc_err: 0
>> rx21_cqe_compress_blks: 0
>> rx21_cqe_compress_pkts: 0
>> rx21_page_reuse: 0
>> rx21_cache_reuse: 192662917
>> rx21_cache_full: 1374120417
>> rx21_cache_empty: 1141972100
>> rx21_cache_busy: 1374120891
>> rx21_cache_waive: 1141965665
>> rx21_congst_umr: 0
>> rx21_arfs_err: 0
>> rx21_xdp_tx_xmit: 0
>> rx21_xdp_tx_full: 0
>> rx21_xdp_tx_err: 0
>> rx21_xdp_tx_cqes: 0
>> rx22_packets: 5613634706
>> rx22_bytes: 3240055099058
>> rx22_csum_complete: 5613634706
>> rx22_csum_unnecessary: 0
>> rx22_csum_unnecessary_inner: 0
>> rx22_csum_none: 0
>> rx22_xdp_drop: 0
>> rx22_xdp_redirect: 0
>> rx22_lro_packets: 0
>> rx22_lro_bytes: 0
>> rx22_ecn_mark: 0
>> rx22_removed_vlan_packets: 5613634706
>> rx22_wqe_err: 0
>> rx22_mpwqe_filler_cqes: 0
>> rx22_mpwqe_filler_strides: 0
>> rx22_buff_alloc_err: 0
>> rx22_cqe_compress_blks: 0
>> rx22_cqe_compress_pkts: 0
>> rx22_page_reuse: 0
>> rx22_cache_reuse: 12161531
>> rx22_cache_full: 2794655567
>> rx22_cache_empty: 6656
>> rx22_cache_busy: 2794655821
>> rx22_cache_waive: 0
>> rx22_congst_umr: 0
>> rx22_arfs_err: 0
>> rx22_xdp_tx_xmit: 0
>> rx22_xdp_tx_full: 0
>> rx22_xdp_tx_err: 0
>> rx22_xdp_tx_cqes: 0
>> rx23_packets: 5389977167
>> rx23_bytes: 3054270771559
>> rx23_csum_complete: 5389977167
>> rx23_csum_unnecessary: 0
>> rx23_csum_unnecessary_inner: 0
>> rx23_csum_none: 0
>> rx23_xdp_drop: 0
>> rx23_xdp_redirect: 0
>> rx23_lro_packets: 0
>> rx23_lro_bytes: 0
>> rx23_ecn_mark: 0
>> rx23_removed_vlan_packets: 5389977167
>> rx23_wqe_err: 0
>> rx23_mpwqe_filler_cqes: 0
>> rx23_mpwqe_filler_strides: 0
>> rx23_buff_alloc_err: 0
>> rx23_cqe_compress_blks: 0
>> rx23_cqe_compress_pkts: 0
>> rx23_page_reuse: 0
>> rx23_cache_reuse: 709328
>> rx23_cache_full: 2694279000
>> rx23_cache_empty: 6656
>> rx23_cache_busy: 2694279252
>> rx23_cache_waive: 0
>> rx23_congst_umr: 0
>> rx23_arfs_err: 0
>> rx23_xdp_tx_xmit: 0
>> rx23_xdp_tx_full: 0
>> rx23_xdp_tx_err: 0
>> rx23_xdp_tx_cqes: 0
>> rx24_packets: 5547561932
>> rx24_bytes: 3166602453443
>> rx24_csum_complete: 5547561932
>> rx24_csum_unnecessary: 0
>> rx24_csum_unnecessary_inner: 0
>> rx24_csum_none: 0
>> rx24_xdp_drop: 0
>> rx24_xdp_redirect: 0
>> rx24_lro_packets: 0
>> rx24_lro_bytes: 0
>> rx24_ecn_mark: 0
>> rx24_removed_vlan_packets: 5547561932
>> rx24_wqe_err: 0
>> rx24_mpwqe_filler_cqes: 0
>> rx24_mpwqe_filler_strides: 0
>> rx24_buff_alloc_err: 0
>> rx24_cqe_compress_blks: 0
>> rx24_cqe_compress_pkts: 0
>> rx24_page_reuse: 0
>> rx24_cache_reuse: 57885119
>> rx24_cache_full: 1529450077
>> rx24_cache_empty: 1186451948
>> rx24_cache_busy: 1529450553
>> rx24_cache_waive: 1186445515
>> rx24_congst_umr: 0
>> rx24_arfs_err: 0
>> rx24_xdp_tx_xmit: 0
>> rx24_xdp_tx_full: 0
>> rx24_xdp_tx_err: 0
>> rx24_xdp_tx_cqes: 0
>> rx25_packets: 5414569326
>> rx25_bytes: 3184757708091
>> rx25_csum_complete: 5414569326
>> rx25_csum_unnecessary: 0
>> rx25_csum_unnecessary_inner: 0
>> rx25_csum_none: 0
>> rx25_xdp_drop: 0
>> rx25_xdp_redirect: 0
>> rx25_lro_packets: 0
>> rx25_lro_bytes: 0
>> rx25_ecn_mark: 0
>> rx25_removed_vlan_packets: 5414569326
>> rx25_wqe_err: 0
>> rx25_mpwqe_filler_cqes: 0
>> rx25_mpwqe_filler_strides: 0
>> rx25_buff_alloc_err: 0
>> rx25_cqe_compress_blks: 0
>> rx25_cqe_compress_pkts: 0
>> rx25_page_reuse: 0
>> rx25_cache_reuse: 5080853
>> rx25_cache_full: 2702203555
>> rx25_cache_empty: 6656
>> rx25_cache_busy: 2702203807
>> rx25_cache_waive: 0
>> rx25_congst_umr: 0
>> rx25_arfs_err: 0
>> rx25_xdp_tx_xmit: 0
>> rx25_xdp_tx_full: 0
>> rx25_xdp_tx_err: 0
>> rx25_xdp_tx_cqes: 0
>> rx26_packets: 5479972151
>> rx26_bytes: 3110642276239
>> rx26_csum_complete: 5479972151
>> rx26_csum_unnecessary: 0
>> rx26_csum_unnecessary_inner: 0
>> rx26_csum_none: 0
>> rx26_xdp_drop: 0
>> rx26_xdp_redirect: 0
>> rx26_lro_packets: 0
>> rx26_lro_bytes: 0
>> rx26_ecn_mark: 0
>> rx26_removed_vlan_packets: 5479972151
>> rx26_wqe_err: 0
>> rx26_mpwqe_filler_cqes: 0
>> rx26_mpwqe_filler_strides: 0
>> rx26_buff_alloc_err: 0
>> rx26_cqe_compress_blks: 0
>> rx26_cqe_compress_pkts: 0
>> rx26_page_reuse: 0
>> rx26_cache_reuse: 26543335
>> rx26_cache_full: 2713442485
>> rx26_cache_empty: 6656
>> rx26_cache_busy: 2713442737
>> rx26_cache_waive: 0
>> rx26_congst_umr: 0
>> rx26_arfs_err: 0
>> rx26_xdp_tx_xmit: 0
>> rx26_xdp_tx_full: 0
>> rx26_xdp_tx_err: 0
>> rx26_xdp_tx_cqes: 0
>> rx27_packets: 5337113900
>> rx27_bytes: 3068966906075
>> rx27_csum_complete: 5337113900
>> rx27_csum_unnecessary: 0
>> rx27_csum_unnecessary_inner: 0
>> rx27_csum_none: 0
>> rx27_xdp_drop: 0
>> rx27_xdp_redirect: 0
>> rx27_lro_packets: 0
>> rx27_lro_bytes: 0
>> rx27_ecn_mark: 0
>> rx27_removed_vlan_packets: 5337113900
>> rx27_wqe_err: 0
>> rx27_mpwqe_filler_cqes: 0
>> rx27_mpwqe_filler_strides: 0
>> rx27_buff_alloc_err: 0
>> rx27_cqe_compress_blks: 0
>> rx27_cqe_compress_pkts: 0
>> rx27_page_reuse: 0
>> rx27_cache_reuse: 1539298962
>> rx27_cache_full: 10861919
>> rx27_cache_empty: 1117173179
>> rx27_cache_busy: 12091463
>> rx27_cache_waive: 1118395847
>> rx27_congst_umr: 0
>> rx27_arfs_err: 0
>> rx27_xdp_tx_xmit: 0
>> rx27_xdp_tx_full: 0
>> rx27_xdp_tx_err: 0
>> rx27_xdp_tx_cqes: 0
>> rx28_packets: 0
>> rx28_bytes: 0
>> rx28_csum_complete: 0
>> rx28_csum_unnecessary: 0
>> rx28_csum_unnecessary_inner: 0
>> rx28_csum_none: 0
>> rx28_xdp_drop: 0
>> rx28_xdp_redirect: 0
>> rx28_lro_packets: 0
>> rx28_lro_bytes: 0
>> rx28_ecn_mark: 0
>> rx28_removed_vlan_packets: 0
>> rx28_wqe_err: 0
>> rx28_mpwqe_filler_cqes: 0
>> rx28_mpwqe_filler_strides: 0
>> rx28_buff_alloc_err: 0
>> rx28_cqe_compress_blks: 0
>> rx28_cqe_compress_pkts: 0
>> rx28_page_reuse: 0
>> rx28_cache_reuse: 0
>> rx28_cache_full: 0
>> rx28_cache_empty: 2560
>> rx28_cache_busy: 0
>> rx28_cache_waive: 0
>> rx28_congst_umr: 0
>> rx28_arfs_err: 0
>> rx28_xdp_tx_xmit: 0
>> rx28_xdp_tx_full: 0
>> rx28_xdp_tx_err: 0
>> rx28_xdp_tx_cqes: 0
>> rx29_packets: 0
>> rx29_bytes: 0
>> rx29_csum_complete: 0
>> rx29_csum_unnecessary: 0
>> rx29_csum_unnecessary_inner: 0
>> rx29_csum_none: 0
>> rx29_xdp_drop: 0
>> rx29_xdp_redirect: 0
>> rx29_lro_packets: 0
>> rx29_lro_bytes: 0
>> rx29_ecn_mark: 0
>> rx29_removed_vlan_packets: 0
>> rx29_wqe_err: 0
>> rx29_mpwqe_filler_cqes: 0
>> rx29_mpwqe_filler_strides: 0
>> rx29_buff_alloc_err: 0
>> rx29_cqe_compress_blks: 0
>> rx29_cqe_compress_pkts: 0
>> rx29_page_reuse: 0
>> rx29_cache_reuse: 0
>> rx29_cache_full: 0
>> rx29_cache_empty: 2560
>> rx29_cache_busy: 0
>> rx29_cache_waive: 0
>> rx29_congst_umr: 0
>> rx29_arfs_err: 0
>> rx29_xdp_tx_xmit: 0
>> rx29_xdp_tx_full: 0
>> rx29_xdp_tx_err: 0
>> rx29_xdp_tx_cqes: 0
>> rx30_packets: 0
>> rx30_bytes: 0
>> rx30_csum_complete: 0
>> rx30_csum_unnecessary: 0
>> rx30_csum_unnecessary_inner: 0
>> rx30_csum_none: 0
>> rx30_xdp_drop: 0
>> rx30_xdp_redirect: 0
>> rx30_lro_packets: 0
>> rx30_lro_bytes: 0
>> rx30_ecn_mark: 0
>> rx30_removed_vlan_packets: 0
>> rx30_wqe_err: 0
>> rx30_mpwqe_filler_cqes: 0
>> rx30_mpwqe_filler_strides: 0
>> rx30_buff_alloc_err: 0
>> rx30_cqe_compress_blks: 0
>> rx30_cqe_compress_pkts: 0
>> rx30_page_reuse: 0
>> rx30_cache_reuse: 0
>> rx30_cache_full: 0
>> rx30_cache_empty: 2560
>> rx30_cache_busy: 0
>> rx30_cache_waive: 0
>> rx30_congst_umr: 0
>> rx30_arfs_err: 0
>> rx30_xdp_tx_xmit: 0
>> rx30_xdp_tx_full: 0
>> rx30_xdp_tx_err: 0
>> rx30_xdp_tx_cqes: 0
>> rx31_packets: 0
>> rx31_bytes: 0
>> rx31_csum_complete: 0
>> rx31_csum_unnecessary: 0
>> rx31_csum_unnecessary_inner: 0
>> rx31_csum_none: 0
>> rx31_xdp_drop: 0
>> rx31_xdp_redirect: 0
>> rx31_lro_packets: 0
>> rx31_lro_bytes: 0
>> rx31_ecn_mark: 0
>> rx31_removed_vlan_packets: 0
>> rx31_wqe_err: 0
>> rx31_mpwqe_filler_cqes: 0
>> rx31_mpwqe_filler_strides: 0
>> rx31_buff_alloc_err: 0
>> rx31_cqe_compress_blks: 0
>> rx31_cqe_compress_pkts: 0
>> rx31_page_reuse: 0
>> rx31_cache_reuse: 0
>> rx31_cache_full: 0
>> rx31_cache_empty: 2560
>> rx31_cache_busy: 0
>> rx31_cache_waive: 0
>> rx31_congst_umr: 0
>> rx31_arfs_err: 0
>> rx31_xdp_tx_xmit: 0
>> rx31_xdp_tx_full: 0
>> rx31_xdp_tx_err: 0
>> rx31_xdp_tx_cqes: 0
>> rx32_packets: 0
>> rx32_bytes: 0
>> rx32_csum_complete: 0
>> rx32_csum_unnecessary: 0
>> rx32_csum_unnecessary_inner: 0
>> rx32_csum_none: 0
>> rx32_xdp_drop: 0
>> rx32_xdp_redirect: 0
>> rx32_lro_packets: 0
>> rx32_lro_bytes: 0
>> rx32_ecn_mark: 0
>> rx32_removed_vlan_packets: 0
>> rx32_wqe_err: 0
>> rx32_mpwqe_filler_cqes: 0
>> rx32_mpwqe_filler_strides: 0
>> rx32_buff_alloc_err: 0
>> rx32_cqe_compress_blks: 0
>> rx32_cqe_compress_pkts: 0
>> rx32_page_reuse: 0
>> rx32_cache_reuse: 0
>> rx32_cache_full: 0
>> rx32_cache_empty: 2560
>> rx32_cache_busy: 0
>> rx32_cache_waive: 0
>> rx32_congst_umr: 0
>> rx32_arfs_err: 0
>> rx32_xdp_tx_xmit: 0
>> rx32_xdp_tx_full: 0
>> rx32_xdp_tx_err: 0
>> rx32_xdp_tx_cqes: 0
>> rx33_packets: 0
>> rx33_bytes: 0
>> rx33_csum_complete: 0
>> rx33_csum_unnecessary: 0
>> rx33_csum_unnecessary_inner: 0
>> rx33_csum_none: 0
>> rx33_xdp_drop: 0
>> rx33_xdp_redirect: 0
>> rx33_lro_packets: 0
>> rx33_lro_bytes: 0
>> rx33_ecn_mark: 0
>> rx33_removed_vlan_packets: 0
>> rx33_wqe_err: 0
>> rx33_mpwqe_filler_cqes: 0
>> rx33_mpwqe_filler_strides: 0
>> rx33_buff_alloc_err: 0
>> rx33_cqe_compress_blks: 0
>> rx33_cqe_compress_pkts: 0
>> rx33_page_reuse: 0
>> rx33_cache_reuse: 0
>> rx33_cache_full: 0
>> rx33_cache_empty: 2560
>> rx33_cache_busy: 0
>> rx33_cache_waive: 0
>> rx33_congst_umr: 0
>> rx33_arfs_err: 0
>> rx33_xdp_tx_xmit: 0
>> rx33_xdp_tx_full: 0
>> rx33_xdp_tx_err: 0
>> rx33_xdp_tx_cqes: 0
>> rx34_packets: 0
>> rx34_bytes: 0
>> rx34_csum_complete: 0
>> rx34_csum_unnecessary: 0
>> rx34_csum_unnecessary_inner: 0
>> rx34_csum_none: 0
>> rx34_xdp_drop: 0
>> rx34_xdp_redirect: 0
>> rx34_lro_packets: 0
>> rx34_lro_bytes: 0
>> rx34_ecn_mark: 0
>> rx34_removed_vlan_packets: 0
>> rx34_wqe_err: 0
>> rx34_mpwqe_filler_cqes: 0
>> rx34_mpwqe_filler_strides: 0
>> rx34_buff_alloc_err: 0
>> rx34_cqe_compress_blks: 0
>> rx34_cqe_compress_pkts: 0
>> rx34_page_reuse: 0
>> rx34_cache_reuse: 0
>> rx34_cache_full: 0
>> rx34_cache_empty: 2560
>> rx34_cache_busy: 0
>> rx34_cache_waive: 0
>> rx34_congst_umr: 0
>> rx34_arfs_err: 0
>> rx34_xdp_tx_xmit: 0
>> rx34_xdp_tx_full: 0
>> rx34_xdp_tx_err: 0
>> rx34_xdp_tx_cqes: 0
>> rx35_packets: 0
>> rx35_bytes: 0
>> rx35_csum_complete: 0
>> rx35_csum_unnecessary: 0
>> rx35_csum_unnecessary_inner: 0
>> rx35_csum_none: 0
>> rx35_xdp_drop: 0
>> rx35_xdp_redirect: 0
>> rx35_lro_packets: 0
>> rx35_lro_bytes: 0
>> rx35_ecn_mark: 0
>> rx35_removed_vlan_packets: 0
>> rx35_wqe_err: 0
>> rx35_mpwqe_filler_cqes: 0
>> rx35_mpwqe_filler_strides: 0
>> rx35_buff_alloc_err: 0
>> rx35_cqe_compress_blks: 0
>> rx35_cqe_compress_pkts: 0
>> rx35_page_reuse: 0
>> rx35_cache_reuse: 0
>> rx35_cache_full: 0
>> rx35_cache_empty: 2560
>> rx35_cache_busy: 0
>> rx35_cache_waive: 0
>> rx35_congst_umr: 0
>> rx35_arfs_err: 0
>> rx35_xdp_tx_xmit: 0
>> rx35_xdp_tx_full: 0
>> rx35_xdp_tx_err: 0
>> rx35_xdp_tx_cqes: 0
>> rx36_packets: 0
>> rx36_bytes: 0
>> rx36_csum_complete: 0
>> rx36_csum_unnecessary: 0
>> rx36_csum_unnecessary_inner: 0
>> rx36_csum_none: 0
>> rx36_xdp_drop: 0
>> rx36_xdp_redirect: 0
>> rx36_lro_packets: 0
>> rx36_lro_bytes: 0
>> rx36_ecn_mark: 0
>> rx36_removed_vlan_packets: 0
>> rx36_wqe_err: 0
>> rx36_mpwqe_filler_cqes: 0
>> rx36_mpwqe_filler_strides: 0
>> rx36_buff_alloc_err: 0
>> rx36_cqe_compress_blks: 0
>> rx36_cqe_compress_pkts: 0
>> rx36_page_reuse: 0
>> rx36_cache_reuse: 0
>> rx36_cache_full: 0
>> rx36_cache_empty: 2560
>> rx36_cache_busy: 0
>> rx36_cache_waive: 0
>> rx36_congst_umr: 0
>> rx36_arfs_err: 0
>> rx36_xdp_tx_xmit: 0
>> rx36_xdp_tx_full: 0
>> rx36_xdp_tx_err: 0
>> rx36_xdp_tx_cqes: 0
>> rx37_packets: 0
>> rx37_bytes: 0
>> rx37_csum_complete: 0
>> rx37_csum_unnecessary: 0
>> rx37_csum_unnecessary_inner: 0
>> rx37_csum_none: 0
>> rx37_xdp_drop: 0
>> rx37_xdp_redirect: 0
>> rx37_lro_packets: 0
>> rx37_lro_bytes: 0
>> rx37_ecn_mark: 0
>> rx37_removed_vlan_packets: 0
>> rx37_wqe_err: 0
>> rx37_mpwqe_filler_cqes: 0
>> rx37_mpwqe_filler_strides: 0
>> rx37_buff_alloc_err: 0
>> rx37_cqe_compress_blks: 0
>> rx37_cqe_compress_pkts: 0
>> rx37_page_reuse: 0
>> rx37_cache_reuse: 0
>> rx37_cache_full: 0
>> rx37_cache_empty: 2560
>> rx37_cache_busy: 0
>> rx37_cache_waive: 0
>> rx37_congst_umr: 0
>> rx37_arfs_err: 0
>> rx37_xdp_tx_xmit: 0
>> rx37_xdp_tx_full: 0
>> rx37_xdp_tx_err: 0
>> rx37_xdp_tx_cqes: 0
>> rx38_packets: 0
>> rx38_bytes: 0
>> rx38_csum_complete: 0
>> rx38_csum_unnecessary: 0
>> rx38_csum_unnecessary_inner: 0
>> rx38_csum_none: 0
>> rx38_xdp_drop: 0
>> rx38_xdp_redirect: 0
>> rx38_lro_packets: 0
>> rx38_lro_bytes: 0
>> rx38_ecn_mark: 0
>> rx38_removed_vlan_packets: 0
>> rx38_wqe_err: 0
>> rx38_mpwqe_filler_cqes: 0
>> rx38_mpwqe_filler_strides: 0
>> rx38_buff_alloc_err: 0
>> rx38_cqe_compress_blks: 0
>> rx38_cqe_compress_pkts: 0
>> rx38_page_reuse: 0
>> rx38_cache_reuse: 0
>> rx38_cache_full: 0
>> rx38_cache_empty: 2560
>> rx38_cache_busy: 0
>> rx38_cache_waive: 0
>> rx38_congst_umr: 0
>> rx38_arfs_err: 0
>> rx38_xdp_tx_xmit: 0
>> rx38_xdp_tx_full: 0
>> rx38_xdp_tx_err: 0
>> rx38_xdp_tx_cqes: 0
>> rx39_packets: 0
>> rx39_bytes: 0
>> rx39_csum_complete: 0
>> rx39_csum_unnecessary: 0
>> rx39_csum_unnecessary_inner: 0
>> rx39_csum_none: 0
>> rx39_xdp_drop: 0
>> rx39_xdp_redirect: 0
>> rx39_lro_packets: 0
>> rx39_lro_bytes: 0
>> rx39_ecn_mark: 0
>> rx39_removed_vlan_packets: 0
>> rx39_wqe_err: 0
>> rx39_mpwqe_filler_cqes: 0
>> rx39_mpwqe_filler_strides: 0
>> rx39_buff_alloc_err: 0
>> rx39_cqe_compress_blks: 0
>> rx39_cqe_compress_pkts: 0
>> rx39_page_reuse: 0
>> rx39_cache_reuse: 0
>> rx39_cache_full: 0
>> rx39_cache_empty: 2560
>> rx39_cache_busy: 0
>> rx39_cache_waive: 0
>> rx39_congst_umr: 0
>> rx39_arfs_err: 0
>> rx39_xdp_tx_xmit: 0
>> rx39_xdp_tx_full: 0
>> rx39_xdp_tx_err: 0
>> rx39_xdp_tx_cqes: 0
>> rx40_packets: 0
>> rx40_bytes: 0
>> rx40_csum_complete: 0
>> rx40_csum_unnecessary: 0
>> rx40_csum_unnecessary_inner: 0
>> rx40_csum_none: 0
>> rx40_xdp_drop: 0
>> rx40_xdp_redirect: 0
>> rx40_lro_packets: 0
>> rx40_lro_bytes: 0
>> rx40_ecn_mark: 0
>> rx40_removed_vlan_packets: 0
>> rx40_wqe_err: 0
>> rx40_mpwqe_filler_cqes: 0
>> rx40_mpwqe_filler_strides: 0
>> rx40_buff_alloc_err: 0
>> rx40_cqe_compress_blks: 0
>> rx40_cqe_compress_pkts: 0
>> rx40_page_reuse: 0
>> rx40_cache_reuse: 0
>> rx40_cache_full: 0
>> rx40_cache_empty: 2560
>> rx40_cache_busy: 0
>> rx40_cache_waive: 0
>> rx40_congst_umr: 0
>> rx40_arfs_err: 0
>> rx40_xdp_tx_xmit: 0
>> rx40_xdp_tx_full: 0
>> rx40_xdp_tx_err: 0
>> rx40_xdp_tx_cqes: 0
>> rx41_packets: 0
>> rx41_bytes: 0
>> rx41_csum_complete: 0
>> rx41_csum_unnecessary: 0
>> rx41_csum_unnecessary_inner: 0
>> rx41_csum_none: 0
>> rx41_xdp_drop: 0
>> rx41_xdp_redirect: 0
>> rx41_lro_packets: 0
>> rx41_lro_bytes: 0
>> rx41_ecn_mark: 0
>> rx41_removed_vlan_packets: 0
>> rx41_wqe_err: 0
>> rx41_mpwqe_filler_cqes: 0
>> rx41_mpwqe_filler_strides: 0
>> rx41_buff_alloc_err: 0
>> rx41_cqe_compress_blks: 0
>> rx41_cqe_compress_pkts: 0
>> rx41_page_reuse: 0
>> rx41_cache_reuse: 0
>> rx41_cache_full: 0
>> rx41_cache_empty: 2560
>> rx41_cache_busy: 0
>> rx41_cache_waive: 0
>> rx41_congst_umr: 0
>> rx41_arfs_err: 0
>> rx41_xdp_tx_xmit: 0
>> rx41_xdp_tx_full: 0
>> rx41_xdp_tx_err: 0
>> rx41_xdp_tx_cqes: 0
>> rx42_packets: 0
>> rx42_bytes: 0
>> rx42_csum_complete: 0
>> rx42_csum_unnecessary: 0
>> rx42_csum_unnecessary_inner: 0
>> rx42_csum_none: 0
>> rx42_xdp_drop: 0
>> rx42_xdp_redirect: 0
>> rx42_lro_packets: 0
>> rx42_lro_bytes: 0
>> rx42_ecn_mark: 0
>> rx42_removed_vlan_packets: 0
>> rx42_wqe_err: 0
>> rx42_mpwqe_filler_cqes: 0
>> rx42_mpwqe_filler_strides: 0
>> rx42_buff_alloc_err: 0
>> rx42_cqe_compress_blks: 0
>> rx42_cqe_compress_pkts: 0
>> rx42_page_reuse: 0
>> rx42_cache_reuse: 0
>> rx42_cache_full: 0
>> rx42_cache_empty: 2560
>> rx42_cache_busy: 0
>> rx42_cache_waive: 0
>> rx42_congst_umr: 0
>> rx42_arfs_err: 0
>> rx42_xdp_tx_xmit: 0
>> rx42_xdp_tx_full: 0
>> rx42_xdp_tx_err: 0
>> rx42_xdp_tx_cqes: 0
>> rx43_packets: 0
>> rx43_bytes: 0
>> rx43_csum_complete: 0
>> rx43_csum_unnecessary: 0
>> rx43_csum_unnecessary_inner: 0
>> rx43_csum_none: 0
>> rx43_xdp_drop: 0
>> rx43_xdp_redirect: 0
>> rx43_lro_packets: 0
>> rx43_lro_bytes: 0
>> rx43_ecn_mark: 0
>> rx43_removed_vlan_packets: 0
>> rx43_wqe_err: 0
>> rx43_mpwqe_filler_cqes: 0
>> rx43_mpwqe_filler_strides: 0
>> rx43_buff_alloc_err: 0
>> rx43_cqe_compress_blks: 0
>> rx43_cqe_compress_pkts: 0
>> rx43_page_reuse: 0
>> rx43_cache_reuse: 0
>> rx43_cache_full: 0
>> rx43_cache_empty: 2560
>> rx43_cache_busy: 0
>> rx43_cache_waive: 0
>> rx43_congst_umr: 0
>> rx43_arfs_err: 0
>> rx43_xdp_tx_xmit: 0
>> rx43_xdp_tx_full: 0
>> rx43_xdp_tx_err: 0
>> rx43_xdp_tx_cqes: 0
>> rx44_packets: 0
>> rx44_bytes: 0
>> rx44_csum_complete: 0
>> rx44_csum_unnecessary: 0
>> rx44_csum_unnecessary_inner: 0
>> rx44_csum_none: 0
>> rx44_xdp_drop: 0
>> rx44_xdp_redirect: 0
>> rx44_lro_packets: 0
>> rx44_lro_bytes: 0
>> rx44_ecn_mark: 0
>> rx44_removed_vlan_packets: 0
>> rx44_wqe_err: 0
>> rx44_mpwqe_filler_cqes: 0
>> rx44_mpwqe_filler_strides: 0
>> rx44_buff_alloc_err: 0
>> rx44_cqe_compress_blks: 0
>> rx44_cqe_compress_pkts: 0
>> rx44_page_reuse: 0
>> rx44_cache_reuse: 0
>> rx44_cache_full: 0
>> rx44_cache_empty: 2560
>> rx44_cache_busy: 0
>> rx44_cache_waive: 0
>> rx44_congst_umr: 0
>> rx44_arfs_err: 0
>> rx44_xdp_tx_xmit: 0
>> rx44_xdp_tx_full: 0
>> rx44_xdp_tx_err: 0
>> rx44_xdp_tx_cqes: 0
>> rx45_packets: 0
>> rx45_bytes: 0
>> rx45_csum_complete: 0
>> rx45_csum_unnecessary: 0
>> rx45_csum_unnecessary_inner: 0
>> rx45_csum_none: 0
>> rx45_xdp_drop: 0
>> rx45_xdp_redirect: 0
>> rx45_lro_packets: 0
>> rx45_lro_bytes: 0
>> rx45_ecn_mark: 0
>> rx45_removed_vlan_packets: 0
>> rx45_wqe_err: 0
>> rx45_mpwqe_filler_cqes: 0
>> rx45_mpwqe_filler_strides: 0
>> rx45_buff_alloc_err: 0
>> rx45_cqe_compress_blks: 0
>> rx45_cqe_compress_pkts: 0
>> rx45_page_reuse: 0
>> rx45_cache_reuse: 0
>> rx45_cache_full: 0
>> rx45_cache_empty: 2560
>> rx45_cache_busy: 0
>> rx45_cache_waive: 0
>> rx45_congst_umr: 0
>> rx45_arfs_err: 0
>> rx45_xdp_tx_xmit: 0
>> rx45_xdp_tx_full: 0
>> rx45_xdp_tx_err: 0
>> rx45_xdp_tx_cqes: 0
>> rx46_packets: 0
>> rx46_bytes: 0
>> rx46_csum_complete: 0
>> rx46_csum_unnecessary: 0
>> rx46_csum_unnecessary_inner: 0
>> rx46_csum_none: 0
>> rx46_xdp_drop: 0
>> rx46_xdp_redirect: 0
>> rx46_lro_packets: 0
>> rx46_lro_bytes: 0
>> rx46_ecn_mark: 0
>> rx46_removed_vlan_packets: 0
>> rx46_wqe_err: 0
>> rx46_mpwqe_filler_cqes: 0
>> rx46_mpwqe_filler_strides: 0
>> rx46_buff_alloc_err: 0
>> rx46_cqe_compress_blks: 0
>> rx46_cqe_compress_pkts: 0
>> rx46_page_reuse: 0
>> rx46_cache_reuse: 0
>> rx46_cache_full: 0
>> rx46_cache_empty: 2560
>> rx46_cache_busy: 0
>> rx46_cache_waive: 0
>> rx46_congst_umr: 0
>> rx46_arfs_err: 0
>> rx46_xdp_tx_xmit: 0
>> rx46_xdp_tx_full: 0
>> rx46_xdp_tx_err: 0
>> rx46_xdp_tx_cqes: 0
>> rx47_packets: 0
>> rx47_bytes: 0
>> rx47_csum_complete: 0
>> rx47_csum_unnecessary: 0
>> rx47_csum_unnecessary_inner: 0
>> rx47_csum_none: 0
>> rx47_xdp_drop: 0
>> rx47_xdp_redirect: 0
>> rx47_lro_packets: 0
>> rx47_lro_bytes: 0
>> rx47_ecn_mark: 0
>> rx47_removed_vlan_packets: 0
>> rx47_wqe_err: 0
>> rx47_mpwqe_filler_cqes: 0
>> rx47_mpwqe_filler_strides: 0
>> rx47_buff_alloc_err: 0
>> rx47_cqe_compress_blks: 0
>> rx47_cqe_compress_pkts: 0
>> rx47_page_reuse: 0
>> rx47_cache_reuse: 0
>> rx47_cache_full: 0
>> rx47_cache_empty: 2560
>> rx47_cache_busy: 0
>> rx47_cache_waive: 0
>> rx47_congst_umr: 0
>> rx47_arfs_err: 0
>> rx47_xdp_tx_xmit: 0
>> rx47_xdp_tx_full: 0
>> rx47_xdp_tx_err: 0
>> rx47_xdp_tx_cqes: 0
>> rx48_packets: 0
>> rx48_bytes: 0
>> rx48_csum_complete: 0
>> rx48_csum_unnecessary: 0
>> rx48_csum_unnecessary_inner: 0
>> rx48_csum_none: 0
>> rx48_xdp_drop: 0
>> rx48_xdp_redirect: 0
>> rx48_lro_packets: 0
>> rx48_lro_bytes: 0
>> rx48_ecn_mark: 0
>> rx48_removed_vlan_packets: 0
>> rx48_wqe_err: 0
>> rx48_mpwqe_filler_cqes: 0
>> rx48_mpwqe_filler_strides: 0
>> rx48_buff_alloc_err: 0
>> rx48_cqe_compress_blks: 0
>> rx48_cqe_compress_pkts: 0
>> rx48_page_reuse: 0
>> rx48_cache_reuse: 0
>> rx48_cache_full: 0
>> rx48_cache_empty: 2560
>> rx48_cache_busy: 0
>> rx48_cache_waive: 0
>> rx48_congst_umr: 0
>> rx48_arfs_err: 0
>> rx48_xdp_tx_xmit: 0
>> rx48_xdp_tx_full: 0
>> rx48_xdp_tx_err: 0
>> rx48_xdp_tx_cqes: 0
>> rx49_packets: 0
>> rx49_bytes: 0
>> rx49_csum_complete: 0
>> rx49_csum_unnecessary: 0
>> rx49_csum_unnecessary_inner: 0
>> rx49_csum_none: 0
>> rx49_xdp_drop: 0
>> rx49_xdp_redirect: 0
>> rx49_lro_packets: 0
>> rx49_lro_bytes: 0
>> rx49_ecn_mark: 0
>> rx49_removed_vlan_packets: 0
>> rx49_wqe_err: 0
>> rx49_mpwqe_filler_cqes: 0
>> rx49_mpwqe_filler_strides: 0
>> rx49_buff_alloc_err: 0
>> rx49_cqe_compress_blks: 0
>> rx49_cqe_compress_pkts: 0
>> rx49_page_reuse: 0
>> rx49_cache_reuse: 0
>> rx49_cache_full: 0
>> rx49_cache_empty: 2560
>> rx49_cache_busy: 0
>> rx49_cache_waive: 0
>> rx49_congst_umr: 0
>> rx49_arfs_err: 0
>> rx49_xdp_tx_xmit: 0
>> rx49_xdp_tx_full: 0
>> rx49_xdp_tx_err: 0
>> rx49_xdp_tx_cqes: 0
>> rx50_packets: 0
>> rx50_bytes: 0
>> rx50_csum_complete: 0
>> rx50_csum_unnecessary: 0
>> rx50_csum_unnecessary_inner: 0
>> rx50_csum_none: 0
>> rx50_xdp_drop: 0
>> rx50_xdp_redirect: 0
>> rx50_lro_packets: 0
>> rx50_lro_bytes: 0
>> rx50_ecn_mark: 0
>> rx50_removed_vlan_packets: 0
>> rx50_wqe_err: 0
>> rx50_mpwqe_filler_cqes: 0
>> rx50_mpwqe_filler_strides: 0
>> rx50_buff_alloc_err: 0
>> rx50_cqe_compress_blks: 0
>> rx50_cqe_compress_pkts: 0
>> rx50_page_reuse: 0
>> rx50_cache_reuse: 0
>> rx50_cache_full: 0
>> rx50_cache_empty: 2560
>> rx50_cache_busy: 0
>> rx50_cache_waive: 0
>> rx50_congst_umr: 0
>> rx50_arfs_err: 0
>> rx50_xdp_tx_xmit: 0
>> rx50_xdp_tx_full: 0
>> rx50_xdp_tx_err: 0
>> rx50_xdp_tx_cqes: 0
>> rx51_packets: 0
>> rx51_bytes: 0
>> rx51_csum_complete: 0
>> rx51_csum_unnecessary: 0
>> rx51_csum_unnecessary_inner: 0
>> rx51_csum_none: 0
>> rx51_xdp_drop: 0
>> rx51_xdp_redirect: 0
>> rx51_lro_packets: 0
>> rx51_lro_bytes: 0
>> rx51_ecn_mark: 0
>> rx51_removed_vlan_packets: 0
>> rx51_wqe_err: 0
>> rx51_mpwqe_filler_cqes: 0
>> rx51_mpwqe_filler_strides: 0
>> rx51_buff_alloc_err: 0
>> rx51_cqe_compress_blks: 0
>> rx51_cqe_compress_pkts: 0
>> rx51_page_reuse: 0
>> rx51_cache_reuse: 0
>> rx51_cache_full: 0
>> rx51_cache_empty: 2560
>> rx51_cache_busy: 0
>> rx51_cache_waive: 0
>> rx51_congst_umr: 0
>> rx51_arfs_err: 0
>> rx51_xdp_tx_xmit: 0
>> rx51_xdp_tx_full: 0
>> rx51_xdp_tx_err: 0
>> rx51_xdp_tx_cqes: 0
>> rx52_packets: 0
>> rx52_bytes: 0
>> rx52_csum_complete: 0
>> rx52_csum_unnecessary: 0
>> rx52_csum_unnecessary_inner: 0
>> rx52_csum_none: 0
>> rx52_xdp_drop: 0
>> rx52_xdp_redirect: 0
>> rx52_lro_packets: 0
>> rx52_lro_bytes: 0
>> rx52_ecn_mark: 0
>> rx52_removed_vlan_packets: 0
>> rx52_wqe_err: 0
>> rx52_mpwqe_filler_cqes: 0
>> rx52_mpwqe_filler_strides: 0
>> rx52_buff_alloc_err: 0
>> rx52_cqe_compress_blks: 0
>> rx52_cqe_compress_pkts: 0
>> rx52_page_reuse: 0
>> rx52_cache_reuse: 0
>> rx52_cache_full: 0
>> rx52_cache_empty: 2560
>> rx52_cache_busy: 0
>> rx52_cache_waive: 0
>> rx52_congst_umr: 0
>> rx52_arfs_err: 0
>> rx52_xdp_tx_xmit: 0
>> rx52_xdp_tx_full: 0
>> rx52_xdp_tx_err: 0
>> rx52_xdp_tx_cqes: 0
>> rx53_packets: 0
>> rx53_bytes: 0
>> rx53_csum_complete: 0
>> rx53_csum_unnecessary: 0
>> rx53_csum_unnecessary_inner: 0
>> rx53_csum_none: 0
>> rx53_xdp_drop: 0
>> rx53_xdp_redirect: 0
>> rx53_lro_packets: 0
>> rx53_lro_bytes: 0
>> rx53_ecn_mark: 0
>> rx53_removed_vlan_packets: 0
>> rx53_wqe_err: 0
>> rx53_mpwqe_filler_cqes: 0
>> rx53_mpwqe_filler_strides: 0
>> rx53_buff_alloc_err: 0
>> rx53_cqe_compress_blks: 0
>> rx53_cqe_compress_pkts: 0
>> rx53_page_reuse: 0
>> rx53_cache_reuse: 0
>> rx53_cache_full: 0
>> rx53_cache_empty: 2560
>> rx53_cache_busy: 0
>> rx53_cache_waive: 0
>> rx53_congst_umr: 0
>> rx53_arfs_err: 0
>> rx53_xdp_tx_xmit: 0
>> rx53_xdp_tx_full: 0
>> rx53_xdp_tx_err: 0
>> rx53_xdp_tx_cqes: 0
>> rx54_packets: 0
>> rx54_bytes: 0
>> rx54_csum_complete: 0
>> rx54_csum_unnecessary: 0
>> rx54_csum_unnecessary_inner: 0
>> rx54_csum_none: 0
>> rx54_xdp_drop: 0
>> rx54_xdp_redirect: 0
>> rx54_lro_packets: 0
>> rx54_lro_bytes: 0
>> rx54_ecn_mark: 0
>> rx54_removed_vlan_packets: 0
>> rx54_wqe_err: 0
>> rx54_mpwqe_filler_cqes: 0
>> rx54_mpwqe_filler_strides: 0
>> rx54_buff_alloc_err: 0
>> rx54_cqe_compress_blks: 0
>> rx54_cqe_compress_pkts: 0
>> rx54_page_reuse: 0
>> rx54_cache_reuse: 0
>> rx54_cache_full: 0
>> rx54_cache_empty: 2560
>> rx54_cache_busy: 0
>> rx54_cache_waive: 0
>> rx54_congst_umr: 0
>> rx54_arfs_err: 0
>> rx54_xdp_tx_xmit: 0
>> rx54_xdp_tx_full: 0
>> rx54_xdp_tx_err: 0
>> rx54_xdp_tx_cqes: 0
>> rx55_packets: 0
>> rx55_bytes: 0
>> rx55_csum_complete: 0
>> rx55_csum_unnecessary: 0
>> rx55_csum_unnecessary_inner: 0
>> rx55_csum_none: 0
>> rx55_xdp_drop: 0
>> rx55_xdp_redirect: 0
>> rx55_lro_packets: 0
>> rx55_lro_bytes: 0
>> rx55_ecn_mark: 0
>> rx55_removed_vlan_packets: 0
>> rx55_wqe_err: 0
>> rx55_mpwqe_filler_cqes: 0
>> rx55_mpwqe_filler_strides: 0
>> rx55_buff_alloc_err: 0
>> rx55_cqe_compress_blks: 0
>> rx55_cqe_compress_pkts: 0
>> rx55_page_reuse: 0
>> rx55_cache_reuse: 0
>> rx55_cache_full: 0
>> rx55_cache_empty: 2560
>> rx55_cache_busy: 0
>> rx55_cache_waive: 0
>> rx55_congst_umr: 0
>> rx55_arfs_err: 0
>> rx55_xdp_tx_xmit: 0
>> rx55_xdp_tx_full: 0
>> rx55_xdp_tx_err: 0
>> rx55_xdp_tx_cqes: 0
>> tx0_packets: 5868971166
>> tx0_bytes: 7384241881537
>> tx0_tso_packets: 1005089669
>> tx0_tso_bytes: 5138882499687
>> tx0_tso_inner_packets: 0
>> tx0_tso_inner_bytes: 0
>> tx0_csum_partial: 1405330470
>> tx0_csum_partial_inner: 0
>> tx0_added_vlan_packets: 3247061022
>> tx0_nop: 83925216
>> tx0_csum_none: 1841730552
>> tx0_stopped: 0
>> tx0_dropped: 0
>> tx0_xmit_more: 29664303
>> tx0_recover: 0
>> tx0_cqes: 3217398842
>> tx0_wake: 0
>> tx0_cqe_err: 0
>> tx1_packets: 5599378674
>> tx1_bytes: 7272236466962
>> tx1_tso_packets: 1024612268
>> tx1_tso_bytes: 5244192050917
>> tx1_tso_inner_packets: 0
>> tx1_tso_inner_bytes: 0
>> tx1_csum_partial: 1438007932
>> tx1_csum_partial_inner: 0
>> tx1_added_vlan_packets: 2919765857
>> tx1_nop: 79661231
>> tx1_csum_none: 1481757925
>> tx1_stopped: 0
>> tx1_dropped: 0
>> tx1_xmit_more: 29485355
>> tx1_recover: 0
>> tx1_cqes: 2890282176
>> tx1_wake: 0
>> tx1_cqe_err: 0
>> tx2_packets: 5413821094
>> tx2_bytes: 7033951631334
>> tx2_tso_packets: 1002868589
>> tx2_tso_bytes: 5089549008985
>> tx2_tso_inner_packets: 0
>> tx2_tso_inner_bytes: 0
>> tx2_csum_partial: 1404186175
>> tx2_csum_partial_inner: 0
>> tx2_added_vlan_packets: 2822670460
>> tx2_nop: 77115408
>> tx2_csum_none: 1418484285
>> tx2_stopped: 0
>> tx2_dropped: 0
>> tx2_xmit_more: 29321129
>> tx2_recover: 0
>> tx2_cqes: 2793351019
>> tx2_wake: 0
>> tx2_cqe_err: 0
>> tx3_packets: 5479609727
>> tx3_bytes: 7116904107659
>> tx3_tso_packets: 1002992639
>> tx3_tso_bytes: 5154225081979
>> tx3_tso_inner_packets: 0
>> tx3_tso_inner_bytes: 0
>> tx3_csum_partial: 1415739849
>> tx3_csum_partial_inner: 0
>> tx3_added_vlan_packets: 2842823811
>> tx3_nop: 78060813
>> tx3_csum_none: 1427083971
>> tx3_stopped: 0
>> tx3_dropped: 0
>> tx3_xmit_more: 28575040
>> tx3_recover: 0
>> tx3_cqes: 2814250785
>> tx3_wake: 0
>> tx3_cqe_err: 0
>> tx4_packets: 5508297397
>> tx4_bytes: 7127659369902
>> tx4_tso_packets: 1007356432
>> tx4_tso_bytes: 5145975736034
>> tx4_tso_inner_packets: 0
>> tx4_tso_inner_bytes: 0
>> tx4_csum_partial: 1411271000
>> tx4_csum_partial_inner: 0
>> tx4_added_vlan_packets: 2882086825
>> tx4_nop: 78433610
>> tx4_csum_none: 1470815825
>> tx4_stopped: 0
>> tx4_dropped: 0
>> tx4_xmit_more: 28632444
>> tx4_recover: 0
>> tx4_cqes: 2853456464
>> tx4_wake: 0
>> tx4_cqe_err: 0
>> tx5_packets: 5513864156
>> tx5_bytes: 7165864145517
>> tx5_tso_packets: 1014046485
>> tx5_tso_bytes: 5192635614477
>> tx5_tso_inner_packets: 0
>> tx5_tso_inner_bytes: 0
>> tx5_csum_partial: 1420810473
>> tx5_csum_partial_inner: 0
>> tx5_added_vlan_packets: 2861370556
>> tx5_nop: 78481355
>> tx5_csum_none: 1440560083
>> tx5_stopped: 0
>> tx5_dropped: 0
>> tx5_xmit_more: 28222467
>> tx5_recover: 0
>> tx5_cqes: 2833149758
>> tx5_wake: 0
>> tx5_cqe_err: 0
>> tx6_packets: 5560724761
>> tx6_bytes: 7210309972086
>> tx6_tso_packets: 994050514
>> tx6_tso_bytes: 5171393741595
>> tx6_tso_inner_packets: 0
>> tx6_tso_inner_bytes: 0
>> tx6_csum_partial: 1414303265
>> tx6_csum_partial_inner: 0
>> tx6_added_vlan_packets: 2905794177
>> tx6_nop: 79353318
>> tx6_csum_none: 1491490912
>> tx6_stopped: 0
>> tx6_dropped: 0
>> tx6_xmit_more: 31246664
>> tx6_recover: 0
>> tx6_cqes: 2874549217
>> tx6_wake: 0
>> tx6_cqe_err: 0
>> tx7_packets: 5557594170
>> tx7_bytes: 7223138778685
>> tx7_tso_packets: 1013475396
>> tx7_tso_bytes: 5241530065484
>> tx7_tso_inner_packets: 0
>> tx7_tso_inner_bytes: 0
>> tx7_csum_partial: 1438604314
>> tx7_csum_partial_inner: 0
>> tx7_added_vlan_packets: 2873917552
>> tx7_nop: 79057059
>> tx7_csum_none: 1435313239
>> tx7_stopped: 0
>> tx7_dropped: 0
>> tx7_xmit_more: 29258761
>> tx7_recover: 0
>> tx7_cqes: 2844660578
>> tx7_wake: 0
>> tx7_cqe_err: 0
>> tx8_packets: 5521254733
>> tx8_bytes: 7208043146297
>> tx8_tso_packets: 1014670801
>> tx8_tso_bytes: 5185842447246
>> tx8_tso_inner_packets: 0
>> tx8_tso_inner_bytes: 0
>> tx8_csum_partial: 1431631562
>> tx8_csum_partial_inner: 0
>> tx8_added_vlan_packets: 2872641129
>> tx8_nop: 78545776
>> tx8_csum_none: 1441009567
>> tx8_stopped: 0
>> tx8_dropped: 0
>> tx8_xmit_more: 29106291
>> tx8_recover: 0
>> tx8_cqes: 2843536748
>> tx8_wake: 0
>> tx8_cqe_err: 0
>> tx9_packets: 5528889957
>> tx9_bytes: 7191793816058
>> tx9_tso_packets: 1015955476
>> tx9_tso_bytes: 5207232047828
>> tx9_tso_inner_packets: 0
>> tx9_tso_inner_bytes: 0
>> tx9_csum_partial: 1421266796
>> tx9_csum_partial_inner: 0
>> tx9_added_vlan_packets: 2869523921
>> tx9_nop: 78586218
>> tx9_csum_none: 1448257125
>> tx9_stopped: 0
>> tx9_dropped: 0
>> tx9_xmit_more: 29483347
>> tx9_recover: 0
>> tx9_cqes: 2840042245
>> tx9_wake: 0
>> tx9_cqe_err: 0
>> tx10_packets: 5556351222
>> tx10_bytes: 7254798330757
>> tx10_tso_packets: 1028554460
>> tx10_tso_bytes: 5246179615774
>> tx10_tso_inner_packets: 0
>> tx10_tso_inner_bytes: 0
>> tx10_csum_partial: 1430459021
>> tx10_csum_partial_inner: 0
>> tx10_added_vlan_packets: 2881683382
>> tx10_nop: 79139584
>> tx10_csum_none: 1451224361
>> tx10_stopped: 0
>> tx10_dropped: 0
>> tx10_xmit_more: 29217190
>> tx10_recover: 0
>> tx10_cqes: 2852467898
>> tx10_wake: 0
>> tx10_cqe_err: 0
>> tx11_packets: 5455631854
>> tx11_bytes: 7061121713772
>> tx11_tso_packets: 992133383
>> tx11_tso_bytes: 5089419722682
>> tx11_tso_inner_packets: 0
>> tx11_tso_inner_bytes: 0
>> tx11_csum_partial: 1395542033
>> tx11_csum_partial_inner: 0
>> tx11_added_vlan_packets: 2852589093
>> tx11_nop: 77799857
>> tx11_csum_none: 1457047060
>> tx11_stopped: 0
>> tx11_dropped: 0
>> tx11_xmit_more: 29559927
>> tx11_recover: 0
>> tx11_cqes: 2823031110
>> tx11_wake: 0
>> tx11_cqe_err: 0
>> tx12_packets: 5488286808
>> tx12_bytes: 7137087569303
>> tx12_tso_packets: 1006435537
>> tx12_tso_bytes: 5163371416750
>> tx12_tso_inner_packets: 0
>> tx12_tso_inner_bytes: 0
>> tx12_csum_partial: 1414799411
>> tx12_csum_partial_inner: 0
>> tx12_added_vlan_packets: 2841679543
>> tx12_nop: 78387039
>> tx12_csum_none: 1426880132
>> tx12_stopped: 0
>> tx12_dropped: 0
>> tx12_xmit_more: 28607526
>> tx12_recover: 0
>> tx12_cqes: 2813073557
>> tx12_wake: 0
>> tx12_cqe_err: 0
>> tx13_packets: 5594132290
>> tx13_bytes: 7251106284829
>> tx13_tso_packets: 1035172061
>> tx13_tso_bytes: 5251200286298
>> tx13_tso_inner_packets: 0
>> tx13_tso_inner_bytes: 0
>> tx13_csum_partial: 1443665981
>> tx13_csum_partial_inner: 0
>> tx13_added_vlan_packets: 2916604799
>> tx13_nop: 79670465
>> tx13_csum_none: 1472938818
>> tx13_stopped: 0
>> tx13_dropped: 0
>> tx13_xmit_more: 27797067
>> tx13_recover: 0
>> tx13_cqes: 2888809352
>> tx13_wake: 0
>> tx13_cqe_err: 0
>> tx14_packets: 5548790952
>> tx14_bytes: 7194211868411
>> tx14_tso_packets: 1021015561
>> tx14_tso_bytes: 5231483708869
>> tx14_tso_inner_packets: 0
>> tx14_tso_inner_bytes: 0
>> tx14_csum_partial: 1427711576
>> tx14_csum_partial_inner: 0
>> tx14_added_vlan_packets: 2875288572
>> tx14_nop: 78900224
>> tx14_csum_none: 1447576996
>> tx14_stopped: 0
>> tx14_dropped: 0
>> tx14_xmit_more: 30003496
>> tx14_recover: 0
>> tx14_cqes: 2845286732
>> tx14_wake: 0
>> tx14_cqe_err: 0
>> tx15_packets: 5609310963
>> tx15_bytes: 7271380831798
>> tx15_tso_packets: 1027830118
>> tx15_tso_bytes: 5229697431506
>> tx15_tso_inner_packets: 0
>> tx15_tso_inner_bytes: 0
>> tx15_csum_partial: 1429209941
>> tx15_csum_partial_inner: 0
>> tx15_added_vlan_packets: 2940315402
>> tx15_nop: 79950883
>> tx15_csum_none: 1511105462
>> tx15_stopped: 0
>> tx15_dropped: 0
>> tx15_xmit_more: 28820740
>> tx15_recover: 0
>> tx15_cqes: 2911496633
>> tx15_wake: 0
>> tx15_cqe_err: 0
>> tx16_packets: 4465363036
>> tx16_bytes: 5769771803704
>> tx16_tso_packets: 817101913
>> tx16_tso_bytes: 4180172833814
>> tx16_tso_inner_packets: 0
>> tx16_tso_inner_bytes: 0
>> tx16_csum_partial: 1136731404
>> tx16_csum_partial_inner: 0
>> tx16_added_vlan_packets: 2332178232
>> tx16_nop: 63458573
>> tx16_csum_none: 1195446828
>> tx16_stopped: 0
>> tx16_dropped: 0
>> tx16_xmit_more: 23756254
>> tx16_recover: 0
>> tx16_cqes: 2308423025
>> tx16_wake: 0
>> tx16_cqe_err: 0
>> tx17_packets: 4380386348
>> tx17_bytes: 5708702994526
>> tx17_tso_packets: 813638023
>> tx17_tso_bytes: 4130806014947
>> tx17_tso_inner_packets: 0
>> tx17_tso_inner_bytes: 0
>> tx17_csum_partial: 1133007164
>> tx17_csum_partial_inner: 0
>> tx17_added_vlan_packets: 2277314787
>> tx17_nop: 62377372
>> tx17_csum_none: 1144307623
>> tx17_stopped: 0
>> tx17_dropped: 0
>> tx17_xmit_more: 23731361
>> tx17_recover: 0
>> tx17_cqes: 2253584638
>> tx17_wake: 0
>> tx17_cqe_err: 0
>> tx18_packets: 4450359743
>> tx18_bytes: 5758968674820
>> tx18_tso_packets: 815791601
>> tx18_tso_bytes: 4179942688909
>> tx18_tso_inner_packets: 0
>> tx18_tso_inner_bytes: 0
>> tx18_csum_partial: 1137649257
>> tx18_csum_partial_inner: 0
>> tx18_added_vlan_packets: 2314556550
>> tx18_nop: 63271085
>> tx18_csum_none: 1176907293
>> tx18_stopped: 0
>> tx18_dropped: 0
>> tx18_xmit_more: 23055770
>> tx18_recover: 0
>> tx18_cqes: 2291501928
>> tx18_wake: 0
>> tx18_cqe_err: 0
>> tx19_packets: 4596064378
>> tx19_bytes: 5916675706535
>> tx19_tso_packets: 825788649
>> tx19_tso_bytes: 4208046929921
>> tx19_tso_inner_packets: 0
>> tx19_tso_inner_bytes: 0
>> tx19_csum_partial: 1150666569
>> tx19_csum_partial_inner: 0
>> tx19_added_vlan_packets: 2450567026
>> tx19_nop: 65468504
>> tx19_csum_none: 1299900457
>> tx19_stopped: 0
>> tx19_dropped: 0
>> tx19_xmit_more: 23846250
>> tx19_recover: 0
>> tx19_cqes: 2426722127
>> tx19_wake: 0
>> tx19_cqe_err: 0
>> tx20_packets: 4424935388
>> tx20_bytes: 5757631205901
>> tx20_tso_packets: 804875006
>> tx20_tso_bytes: 4156262736109
>> tx20_tso_inner_packets: 0
>> tx20_tso_inner_bytes: 0
>> tx20_csum_partial: 1134144916
>> tx20_csum_partial_inner: 0
>> tx20_added_vlan_packets: 2294839665
>> tx20_nop: 63023986
>> tx20_csum_none: 1160694749
>> tx20_stopped: 0
>> tx20_dropped: 0
>> tx20_xmit_more: 23393201
>> tx20_recover: 0
>> tx20_cqes: 2271447623
>> tx20_wake: 0
>> tx20_cqe_err: 0
>> tx21_packets: 4595062285
>> tx21_bytes: 5958671993467
>> tx21_tso_packets: 821936215
>> tx21_tso_bytes: 4187977870684
>> tx21_tso_inner_packets: 0
>> tx21_tso_inner_bytes: 0
>> tx21_csum_partial: 1143339787
>> tx21_csum_partial_inner: 0
>> tx21_added_vlan_packets: 2457167412
>> tx21_nop: 65697763
>> tx21_csum_none: 1313827625
>> tx21_stopped: 0
>> tx21_dropped: 0
>> tx21_xmit_more: 23858345
>> tx21_recover: 0
>> tx21_cqes: 2433310348
>> tx21_wake: 0
>> tx21_cqe_err: 0
>> tx22_packets: 4664446513
>> tx22_bytes: 5931429292082
>> tx22_tso_packets: 814457881
>> tx22_tso_bytes: 4148607956533
>> tx22_tso_inner_packets: 0
>> tx22_tso_inner_bytes: 0
>> tx22_csum_partial: 1127284783
>> tx22_csum_partial_inner: 0
>> tx22_added_vlan_packets: 2548650146
>> tx22_nop: 66299909
>> tx22_csum_none: 1421365363
>> tx22_stopped: 0
>> tx22_dropped: 0
>> tx22_xmit_more: 23800911
>> tx22_recover: 0
>> tx22_cqes: 2524850415
>> tx22_wake: 0
>> tx22_cqe_err: 0
>> tx23_packets: 4416221747
>> tx23_bytes: 5721472587985
>> tx23_tso_packets: 823538520
>> tx23_tso_bytes: 4163520218617
>> tx23_tso_inner_packets: 0
>> tx23_tso_inner_bytes: 0
>> tx23_csum_partial: 1135996006
>> tx23_csum_partial_inner: 0
>> tx23_added_vlan_packets: 2292404120
>> tx23_nop: 62709432
>> tx23_csum_none: 1156408114
>> tx23_stopped: 0
>> tx23_dropped: 0
>> tx23_xmit_more: 22299889
>> tx23_recover: 0
>> tx23_cqes: 2270105487
>> tx23_wake: 0
>> tx23_cqe_err: 0
>> tx24_packets: 4420014824
>> tx24_bytes: 5740767318521
>> tx24_tso_packets: 820838072
>> tx24_tso_bytes: 4183722948422
>> tx24_tso_inner_packets: 0
>> tx24_tso_inner_bytes: 0
>> tx24_csum_partial: 1138070059
>> tx24_csum_partial_inner: 0
>> tx24_added_vlan_packets: 2289043946
>> tx24_nop: 62797341
>> tx24_csum_none: 1150973887
>> tx24_stopped: 0
>> tx24_dropped: 0
>> tx24_xmit_more: 22744690
>> tx24_recover: 0
>> tx24_cqes: 2266300568
>> tx24_wake: 0
>> tx24_cqe_err: 0
>> tx25_packets: 4413225545
>> tx25_bytes: 5716162617155
>> tx25_tso_packets: 808274341
>> tx25_tso_bytes: 4138408857714
>> tx25_tso_inner_packets: 0
>> tx25_tso_inner_bytes: 0
>> tx25_csum_partial: 1134587898
>> tx25_csum_partial_inner: 0
>> tx25_added_vlan_packets: 2297149310
>> tx25_nop: 62958238
>> tx25_csum_none: 1162561412
>> tx25_stopped: 0
>> tx25_dropped: 0
>> tx25_xmit_more: 24463552
>> tx25_recover: 0
>> tx25_cqes: 2272686971
>> tx25_wake: 0
>> tx25_cqe_err: 0
>> tx26_packets: 4524907591
>> tx26_bytes: 5865394280699
>> tx26_tso_packets: 807270022
>> tx26_tso_bytes: 4148754705317
>> tx26_tso_inner_packets: 0
>> tx26_tso_inner_bytes: 0
>> tx26_csum_partial: 1130306933
>> tx26_csum_partial_inner: 0
>> tx26_added_vlan_packets: 2402682460
>> tx26_nop: 64474322
>> tx26_csum_none: 1272375527
>> tx26_stopped: 1
>> tx26_dropped: 0
>> tx26_xmit_more: 23316186
>> tx26_recover: 0
>> tx26_cqes: 2379367502
>> tx26_wake: 1
>> tx26_cqe_err: 0
>> tx27_packets: 4376114969
>> tx27_bytes: 5683551238304
>> tx27_tso_packets: 809344829
>> tx27_tso_bytes: 4124331859270
>> tx27_tso_inner_packets: 0
>> tx27_tso_inner_bytes: 0
>> tx27_csum_partial: 1124954937
>> tx27_csum_partial_inner: 0
>> tx27_added_vlan_packets: 2267871300
>> tx27_nop: 62213214
>> tx27_csum_none: 1142916363
>> tx27_stopped: 0
>> tx27_dropped: 0
>> tx27_xmit_more: 23369974
>> tx27_recover: 0
>> tx27_cqes: 2244502686
>> tx27_wake: 0
>> tx27_cqe_err: 0
>> tx28_packets: 3
>> tx28_bytes: 266
>> tx28_tso_packets: 0
>> tx28_tso_bytes: 0
>> tx28_tso_inner_packets: 0
>> tx28_tso_inner_bytes: 0
>> tx28_csum_partial: 0
>> tx28_csum_partial_inner: 0
>> tx28_added_vlan_packets: 0
>> tx28_nop: 0
>> tx28_csum_none: 3
>> tx28_stopped: 0
>> tx28_dropped: 0
>> tx28_xmit_more: 0
>> tx28_recover: 0
>> tx28_cqes: 3
>> tx28_wake: 0
>> tx28_cqe_err: 0
>> tx29_packets: 0
>> tx29_bytes: 0
>> tx29_tso_packets: 0
>> tx29_tso_bytes: 0
>> tx29_tso_inner_packets: 0
>> tx29_tso_inner_bytes: 0
>> tx29_csum_partial: 0
>> tx29_csum_partial_inner: 0
>> tx29_added_vlan_packets: 0
>> tx29_nop: 0
>> tx29_csum_none: 0
>> tx29_stopped: 0
>> tx29_dropped: 0
>> tx29_xmit_more: 0
>> tx29_recover: 0
>> tx29_cqes: 0
>> tx29_wake: 0
>> tx29_cqe_err: 0
>> tx30_packets: 0
>> tx30_bytes: 0
>> tx30_tso_packets: 0
>> tx30_tso_bytes: 0
>> tx30_tso_inner_packets: 0
>> tx30_tso_inner_bytes: 0
>> tx30_csum_partial: 0
>> tx30_csum_partial_inner: 0
>> tx30_added_vlan_packets: 0
>> tx30_nop: 0
>> tx30_csum_none: 0
>> tx30_stopped: 0
>> tx30_dropped: 0
>> tx30_xmit_more: 0
>> tx30_recover: 0
>> tx30_cqes: 0
>> tx30_wake: 0
>> tx30_cqe_err: 0
>> tx31_packets: 0
>> tx31_bytes: 0
>> tx31_tso_packets: 0
>> tx31_tso_bytes: 0
>> tx31_tso_inner_packets: 0
>> tx31_tso_inner_bytes: 0
>> tx31_csum_partial: 0
>> tx31_csum_partial_inner: 0
>> tx31_added_vlan_packets: 0
>> tx31_nop: 0
>> tx31_csum_none: 0
>> tx31_stopped: 0
>> tx31_dropped: 0
>> tx31_xmit_more: 0
>> tx31_recover: 0
>> tx31_cqes: 0
>> tx31_wake: 0
>> tx31_cqe_err: 0
>> tx32_packets: 0
>> tx32_bytes: 0
>> tx32_tso_packets: 0
>> tx32_tso_bytes: 0
>> tx32_tso_inner_packets: 0
>> tx32_tso_inner_bytes: 0
>> tx32_csum_partial: 0
>> tx32_csum_partial_inner: 0
>> tx32_added_vlan_packets: 0
>> tx32_nop: 0
>> tx32_csum_none: 0
>> tx32_stopped: 0
>> tx32_dropped: 0
>> tx32_xmit_more: 0
>> tx32_recover: 0
>> tx32_cqes: 0
>> tx32_wake: 0
>> tx32_cqe_err: 0
>> tx33_packets: 0
>> tx33_bytes: 0
>> tx33_tso_packets: 0
>> tx33_tso_bytes: 0
>> tx33_tso_inner_packets: 0
>> tx33_tso_inner_bytes: 0
>> tx33_csum_partial: 0
>> tx33_csum_partial_inner: 0
>> tx33_added_vlan_packets: 0
>> tx33_nop: 0
>> tx33_csum_none: 0
>> tx33_stopped: 0
>> tx33_dropped: 0
>> tx33_xmit_more: 0
>> tx33_recover: 0
>> tx33_cqes: 0
>> tx33_wake: 0
>> tx33_cqe_err: 0
>> tx34_packets: 0
>> tx34_bytes: 0
>> tx34_tso_packets: 0
>> tx34_tso_bytes: 0
>> tx34_tso_inner_packets: 0
>> tx34_tso_inner_bytes: 0
>> tx34_csum_partial: 0
>> tx34_csum_partial_inner: 0
>> tx34_added_vlan_packets: 0
>> tx34_nop: 0
>> tx34_csum_none: 0
>> tx34_stopped: 0
>> tx34_dropped: 0
>> tx34_xmit_more: 0
>> tx34_recover: 0
>> tx34_cqes: 0
>> tx34_wake: 0
>> tx34_cqe_err: 0
>> tx35_packets: 0
>> tx35_bytes: 0
>> tx35_tso_packets: 0
>> tx35_tso_bytes: 0
>> tx35_tso_inner_packets: 0
>> tx35_tso_inner_bytes: 0
>> tx35_csum_partial: 0
>> tx35_csum_partial_inner: 0
>> tx35_added_vlan_packets: 0
>> tx35_nop: 0
>> tx35_csum_none: 0
>> tx35_stopped: 0
>> tx35_dropped: 0
>> tx35_xmit_more: 0
>> tx35_recover: 0
>> tx35_cqes: 0
>> tx35_wake: 0
>> tx35_cqe_err: 0
>> tx36_packets: 0
>> tx36_bytes: 0
>> tx36_tso_packets: 0
>> tx36_tso_bytes: 0
>> tx36_tso_inner_packets: 0
>> tx36_tso_inner_bytes: 0
>> tx36_csum_partial: 0
>> tx36_csum_partial_inner: 0
>> tx36_added_vlan_packets: 0
>> tx36_nop: 0
>> tx36_csum_none: 0
>> tx36_stopped: 0
>> tx36_dropped: 0
>> tx36_xmit_more: 0
>> tx36_recover: 0
>> tx36_cqes: 0
>> tx36_wake: 0
>> tx36_cqe_err: 0
>> tx37_packets: 0
>> tx37_bytes: 0
>> tx37_tso_packets: 0
>> tx37_tso_bytes: 0
>> tx37_tso_inner_packets: 0
>> tx37_tso_inner_bytes: 0
>> tx37_csum_partial: 0
>> tx37_csum_partial_inner: 0
>> tx37_added_vlan_packets: 0
>> tx37_nop: 0
>> tx37_csum_none: 0
>> tx37_stopped: 0
>> tx37_dropped: 0
>> tx37_xmit_more: 0
>> tx37_recover: 0
>> tx37_cqes: 0
>> tx37_wake: 0
>> tx37_cqe_err: 0
>> tx38_packets: 0
>> tx38_bytes: 0
>> tx38_tso_packets: 0
>> tx38_tso_bytes: 0
>> tx38_tso_inner_packets: 0
>> tx38_tso_inner_bytes: 0
>> tx38_csum_partial: 0
>> tx38_csum_partial_inner: 0
>> tx38_added_vlan_packets: 0
>> tx38_nop: 0
>> tx38_csum_none: 0
>> tx38_stopped: 0
>> tx38_dropped: 0
>> tx38_xmit_more: 0
>> tx38_recover: 0
>> tx38_cqes: 0
>> tx38_wake: 0
>> tx38_cqe_err: 0
>> tx39_packets: 0
>> tx39_bytes: 0
>> tx39_tso_packets: 0
>> tx39_tso_bytes: 0
>> tx39_tso_inner_packets: 0
>> tx39_tso_inner_bytes: 0
>> tx39_csum_partial: 0
>> tx39_csum_partial_inner: 0
>> tx39_added_vlan_packets: 0
>> tx39_nop: 0
>> tx39_csum_none: 0
>> tx39_stopped: 0
>> tx39_dropped: 0
>> tx39_xmit_more: 0
>> tx39_recover: 0
>> tx39_cqes: 0
>> tx39_wake: 0
>> tx39_cqe_err: 0
>> tx40_packets: 0
>> tx40_bytes: 0
>> tx40_tso_packets: 0
>> tx40_tso_bytes: 0
>> tx40_tso_inner_packets: 0
>> tx40_tso_inner_bytes: 0
>> tx40_csum_partial: 0
>> tx40_csum_partial_inner: 0
>> tx40_added_vlan_packets: 0
>> tx40_nop: 0
>> tx40_csum_none: 0
>> tx40_stopped: 0
>> tx40_dropped: 0
>> tx40_xmit_more: 0
>> tx40_recover: 0
>> tx40_cqes: 0
>> tx40_wake: 0
>> tx40_cqe_err: 0
>> tx41_packets: 0
>> tx41_bytes: 0
>> tx41_tso_packets: 0
>> tx41_tso_bytes: 0
>> tx41_tso_inner_packets: 0
>> tx41_tso_inner_bytes: 0
>> tx41_csum_partial: 0
>> tx41_csum_partial_inner: 0
>> tx41_added_vlan_packets: 0
>> tx41_nop: 0
>> tx41_csum_none: 0
>> tx41_stopped: 0
>> tx41_dropped: 0
>> tx41_xmit_more: 0
>> tx41_recover: 0
>> tx41_cqes: 0
>> tx41_wake: 0
>> tx41_cqe_err: 0
>> tx42_packets: 0
>> tx42_bytes: 0
>> tx42_tso_packets: 0
>> tx42_tso_bytes: 0
>> tx42_tso_inner_packets: 0
>> tx42_tso_inner_bytes: 0
>> tx42_csum_partial: 0
>> tx42_csum_partial_inner: 0
>> tx42_added_vlan_packets: 0
>> tx42_nop: 0
>> tx42_csum_none: 0
>> tx42_stopped: 0
>> tx42_dropped: 0
>> tx42_xmit_more: 0
>> tx42_recover: 0
>> tx42_cqes: 0
>> tx42_wake: 0
>> tx42_cqe_err: 0
>> tx43_packets: 0
>> tx43_bytes: 0
>> tx43_tso_packets: 0
>> tx43_tso_bytes: 0
>> tx43_tso_inner_packets: 0
>> tx43_tso_inner_bytes: 0
>> tx43_csum_partial: 0
>> tx43_csum_partial_inner: 0
>> tx43_added_vlan_packets: 0
>> tx43_nop: 0
>> tx43_csum_none: 0
>> tx43_stopped: 0
>> tx43_dropped: 0
>> tx43_xmit_more: 0
>> tx43_recover: 0
>> tx43_cqes: 0
>> tx43_wake: 0
>> tx43_cqe_err: 0
>> tx44_packets: 0
>> tx44_bytes: 0
>> tx44_tso_packets: 0
>> tx44_tso_bytes: 0
>> tx44_tso_inner_packets: 0
>> tx44_tso_inner_bytes: 0
>> tx44_csum_partial: 0
>> tx44_csum_partial_inner: 0
>> tx44_added_vlan_packets: 0
>> tx44_nop: 0
>> tx44_csum_none: 0
>> tx44_stopped: 0
>> tx44_dropped: 0
>> tx44_xmit_more: 0
>> tx44_recover: 0
>> tx44_cqes: 0
>> tx44_wake: 0
>> tx44_cqe_err: 0
>> tx45_packets: 0
>> tx45_bytes: 0
>> tx45_tso_packets: 0
>> tx45_tso_bytes: 0
>> tx45_tso_inner_packets: 0
>> tx45_tso_inner_bytes: 0
>> tx45_csum_partial: 0
>> tx45_csum_partial_inner: 0
>> tx45_added_vlan_packets: 0
>> tx45_nop: 0
>> tx45_csum_none: 0
>> tx45_stopped: 0
>> tx45_dropped: 0
>> tx45_xmit_more: 0
>> tx45_recover: 0
>> tx45_cqes: 0
>> tx45_wake: 0
>> tx45_cqe_err: 0
>> tx46_packets: 0
>> tx46_bytes: 0
>> tx46_tso_packets: 0
>> tx46_tso_bytes: 0
>> tx46_tso_inner_packets: 0
>> tx46_tso_inner_bytes: 0
>> tx46_csum_partial: 0
>> tx46_csum_partial_inner: 0
>> tx46_added_vlan_packets: 0
>> tx46_nop: 0
>> tx46_csum_none: 0
>> tx46_stopped: 0
>> tx46_dropped: 0
>> tx46_xmit_more: 0
>> tx46_recover: 0
>> tx46_cqes: 0
>> tx46_wake: 0
>> tx46_cqe_err: 0
>> tx47_packets: 0
>> tx47_bytes: 0
>> tx47_tso_packets: 0
>> tx47_tso_bytes: 0
>> tx47_tso_inner_packets: 0
>> tx47_tso_inner_bytes: 0
>> tx47_csum_partial: 0
>> tx47_csum_partial_inner: 0
>> tx47_added_vlan_packets: 0
>> tx47_nop: 0
>> tx47_csum_none: 0
>> tx47_stopped: 0
>> tx47_dropped: 0
>> tx47_xmit_more: 0
>> tx47_recover: 0
>> tx47_cqes: 0
>> tx47_wake: 0
>> tx47_cqe_err: 0
>> tx48_packets: 0
>> tx48_bytes: 0
>> tx48_tso_packets: 0
>> tx48_tso_bytes: 0
>> tx48_tso_inner_packets: 0
>> tx48_tso_inner_bytes: 0
>> tx48_csum_partial: 0
>> tx48_csum_partial_inner: 0
>> tx48_added_vlan_packets: 0
>> tx48_nop: 0
>> tx48_csum_none: 0
>> tx48_stopped: 0
>> tx48_dropped: 0
>> tx48_xmit_more: 0
>> tx48_recover: 0
>> tx48_cqes: 0
>> tx48_wake: 0
>> tx48_cqe_err: 0
>> tx49_packets: 0
>> tx49_bytes: 0
>> tx49_tso_packets: 0
>> tx49_tso_bytes: 0
>> tx49_tso_inner_packets: 0
>> tx49_tso_inner_bytes: 0
>> tx49_csum_partial: 0
>> tx49_csum_partial_inner: 0
>> tx49_added_vlan_packets: 0
>> tx49_nop: 0
>> tx49_csum_none: 0
>> tx49_stopped: 0
>> tx49_dropped: 0
>> tx49_xmit_more: 0
>> tx49_recover: 0
>> tx49_cqes: 0
>> tx49_wake: 0
>> tx49_cqe_err: 0
>> tx50_packets: 0
>> tx50_bytes: 0
>> tx50_tso_packets: 0
>> tx50_tso_bytes: 0
>> tx50_tso_inner_packets: 0
>> tx50_tso_inner_bytes: 0
>> tx50_csum_partial: 0
>> tx50_csum_partial_inner: 0
>> tx50_added_vlan_packets: 0
>> tx50_nop: 0
>> tx50_csum_none: 0
>> tx50_stopped: 0
>> tx50_dropped: 0
>> tx50_xmit_more: 0
>> tx50_recover: 0
>> tx50_cqes: 0
>> tx50_wake: 0
>> tx50_cqe_err: 0
>> tx51_packets: 0
>> tx51_bytes: 0
>> tx51_tso_packets: 0
>> tx51_tso_bytes: 0
>> tx51_tso_inner_packets: 0
>> tx51_tso_inner_bytes: 0
>> tx51_csum_partial: 0
>> tx51_csum_partial_inner: 0
>> tx51_added_vlan_packets: 0
>> tx51_nop: 0
>> tx51_csum_none: 0
>> tx51_stopped: 0
>> tx51_dropped: 0
>> tx51_xmit_more: 0
>> tx51_recover: 0
>> tx51_cqes: 0
>> tx51_wake: 0
>> tx51_cqe_err: 0
>> tx52_packets: 0
>> tx52_bytes: 0
>> tx52_tso_packets: 0
>> tx52_tso_bytes: 0
>> tx52_tso_inner_packets: 0
>> tx52_tso_inner_bytes: 0
>> tx52_csum_partial: 0
>> tx52_csum_partial_inner: 0
>> tx52_added_vlan_packets: 0
>> tx52_nop: 0
>> tx52_csum_none: 0
>> tx52_stopped: 0
>> tx52_dropped: 0
>> tx52_xmit_more: 0
>> tx52_recover: 0
>> tx52_cqes: 0
>> tx52_wake: 0
>> tx52_cqe_err: 0
>> tx53_packets: 0
>> tx53_bytes: 0
>> tx53_tso_packets: 0
>> tx53_tso_bytes: 0
>> tx53_tso_inner_packets: 0
>> tx53_tso_inner_bytes: 0
>> tx53_csum_partial: 0
>> tx53_csum_partial_inner: 0
>> tx53_added_vlan_packets: 0
>> tx53_nop: 0
>> tx53_csum_none: 0
>> tx53_stopped: 0
>> tx53_dropped: 0
>> tx53_xmit_more: 0
>> tx53_recover: 0
>> tx53_cqes: 0
>> tx53_wake: 0
>> tx53_cqe_err: 0
>> tx54_packets: 0
>> tx54_bytes: 0
>> tx54_tso_packets: 0
>> tx54_tso_bytes: 0
>> tx54_tso_inner_packets: 0
>> tx54_tso_inner_bytes: 0
>> tx54_csum_partial: 0
>> tx54_csum_partial_inner: 0
>> tx54_added_vlan_packets: 0
>> tx54_nop: 0
>> tx54_csum_none: 0
>> tx54_stopped: 0
>> tx54_dropped: 0
>> tx54_xmit_more: 0
>> tx54_recover: 0
>> tx54_cqes: 0
>> tx54_wake: 0
>> tx54_cqe_err: 0
>> tx55_packets: 0
>> tx55_bytes: 0
>> tx55_tso_packets: 0
>> tx55_tso_bytes: 0
>> tx55_tso_inner_packets: 0
>> tx55_tso_inner_bytes: 0
>> tx55_csum_partial: 0
>> tx55_csum_partial_inner: 0
>> tx55_added_vlan_packets: 0
>> tx55_nop: 0
>> tx55_csum_none: 0
>> tx55_stopped: 0
>> tx55_dropped: 0
>> tx55_xmit_more: 0
>> tx55_recover: 0
>> tx55_cqes: 0
>> tx55_wake: 0
>> tx55_cqe_err: 0
>> tx0_xdp_xmit: 0
>> tx0_xdp_full: 0
>> tx0_xdp_err: 0
>> tx0_xdp_cqes: 0
>> tx1_xdp_xmit: 0
>> tx1_xdp_full: 0
>> tx1_xdp_err: 0
>> tx1_xdp_cqes: 0
>> tx2_xdp_xmit: 0
>> tx2_xdp_full: 0
>> tx2_xdp_err: 0
>> tx2_xdp_cqes: 0
>> tx3_xdp_xmit: 0
>> tx3_xdp_full: 0
>> tx3_xdp_err: 0
>> tx3_xdp_cqes: 0
>> tx4_xdp_xmit: 0
>> tx4_xdp_full: 0
>> tx4_xdp_err: 0
>> tx4_xdp_cqes: 0
>> tx5_xdp_xmit: 0
>> tx5_xdp_full: 0
>> tx5_xdp_err: 0
>> tx5_xdp_cqes: 0
>> tx6_xdp_xmit: 0
>> tx6_xdp_full: 0
>> tx6_xdp_err: 0
>> tx6_xdp_cqes: 0
>> tx7_xdp_xmit: 0
>> tx7_xdp_full: 0
>> tx7_xdp_err: 0
>> tx7_xdp_cqes: 0
>> tx8_xdp_xmit: 0
>> tx8_xdp_full: 0
>> tx8_xdp_err: 0
>> tx8_xdp_cqes: 0
>> tx9_xdp_xmit: 0
>> tx9_xdp_full: 0
>> tx9_xdp_err: 0
>> tx9_xdp_cqes: 0
>> tx10_xdp_xmit: 0
>> tx10_xdp_full: 0
>> tx10_xdp_err: 0
>> tx10_xdp_cqes: 0
>> tx11_xdp_xmit: 0
>> tx11_xdp_full: 0
>> tx11_xdp_err: 0
>> tx11_xdp_cqes: 0
>> tx12_xdp_xmit: 0
>> tx12_xdp_full: 0
>> tx12_xdp_err: 0
>> tx12_xdp_cqes: 0
>> tx13_xdp_xmit: 0
>> tx13_xdp_full: 0
>> tx13_xdp_err: 0
>> tx13_xdp_cqes: 0
>> tx14_xdp_xmit: 0
>> tx14_xdp_full: 0
>> tx14_xdp_err: 0
>> tx14_xdp_cqes: 0
>> tx15_xdp_xmit: 0
>> tx15_xdp_full: 0
>> tx15_xdp_err: 0
>> tx15_xdp_cqes: 0
>> tx16_xdp_xmit: 0
>> tx16_xdp_full: 0
>> tx16_xdp_err: 0
>> tx16_xdp_cqes: 0
>> tx17_xdp_xmit: 0
>> tx17_xdp_full: 0
>> tx17_xdp_err: 0
>> tx17_xdp_cqes: 0
>> tx18_xdp_xmit: 0
>> tx18_xdp_full: 0
>> tx18_xdp_err: 0
>> tx18_xdp_cqes: 0
>> tx19_xdp_xmit: 0
>> tx19_xdp_full: 0
>> tx19_xdp_err: 0
>> tx19_xdp_cqes: 0
>> tx20_xdp_xmit: 0
>> tx20_xdp_full: 0
>> tx20_xdp_err: 0
>> tx20_xdp_cqes: 0
>> tx21_xdp_xmit: 0
>> tx21_xdp_full: 0
>> tx21_xdp_err: 0
>> tx21_xdp_cqes: 0
>> tx22_xdp_xmit: 0
>> tx22_xdp_full: 0
>> tx22_xdp_err: 0
>> tx22_xdp_cqes: 0
>> tx23_xdp_xmit: 0
>> tx23_xdp_full: 0
>> tx23_xdp_err: 0
>> tx23_xdp_cqes: 0
>> tx24_xdp_xmit: 0
>> tx24_xdp_full: 0
>> tx24_xdp_err: 0
>> tx24_xdp_cqes: 0
>> tx25_xdp_xmit: 0
>> tx25_xdp_full: 0
>> tx25_xdp_err: 0
>> tx25_xdp_cqes: 0
>> tx26_xdp_xmit: 0
>> tx26_xdp_full: 0
>> tx26_xdp_err: 0
>> tx26_xdp_cqes: 0
>> tx27_xdp_xmit: 0
>> tx27_xdp_full: 0
>> tx27_xdp_err: 0
>> tx27_xdp_cqes: 0
>> tx28_xdp_xmit: 0
>> tx28_xdp_full: 0
>> tx28_xdp_err: 0
>> tx28_xdp_cqes: 0
>> tx29_xdp_xmit: 0
>> tx29_xdp_full: 0
>> tx29_xdp_err: 0
>> tx29_xdp_cqes: 0
>> tx30_xdp_xmit: 0
>> tx30_xdp_full: 0
>> tx30_xdp_err: 0
>> tx30_xdp_cqes: 0
>> tx31_xdp_xmit: 0
>> tx31_xdp_full: 0
>> tx31_xdp_err: 0
>> tx31_xdp_cqes: 0
>> tx32_xdp_xmit: 0
>> tx32_xdp_full: 0
>> tx32_xdp_err: 0
>> tx32_xdp_cqes: 0
>> tx33_xdp_xmit: 0
>> tx33_xdp_full: 0
>> tx33_xdp_err: 0
>> tx33_xdp_cqes: 0
>> tx34_xdp_xmit: 0
>> tx34_xdp_full: 0
>> tx34_xdp_err: 0
>> tx34_xdp_cqes: 0
>> tx35_xdp_xmit: 0
>> tx35_xdp_full: 0
>> tx35_xdp_err: 0
>> tx35_xdp_cqes: 0
>> tx36_xdp_xmit: 0
>> tx36_xdp_full: 0
>> tx36_xdp_err: 0
>> tx36_xdp_cqes: 0
>> tx37_xdp_xmit: 0
>> tx37_xdp_full: 0
>> tx37_xdp_err: 0
>> tx37_xdp_cqes: 0
>> tx38_xdp_xmit: 0
>> tx38_xdp_full: 0
>> tx38_xdp_err: 0
>> tx38_xdp_cqes: 0
>> tx39_xdp_xmit: 0
>> tx39_xdp_full: 0
>> tx39_xdp_err: 0
>> tx39_xdp_cqes: 0
>> tx40_xdp_xmit: 0
>> tx40_xdp_full: 0
>> tx40_xdp_err: 0
>> tx40_xdp_cqes: 0
>> tx41_xdp_xmit: 0
>> tx41_xdp_full: 0
>> tx41_xdp_err: 0
>> tx41_xdp_cqes: 0
>> tx42_xdp_xmit: 0
>> tx42_xdp_full: 0
>> tx42_xdp_err: 0
>> tx42_xdp_cqes: 0
>> tx43_xdp_xmit: 0
>> tx43_xdp_full: 0
>> tx43_xdp_err: 0
>> tx43_xdp_cqes: 0
>> tx44_xdp_xmit: 0
>> tx44_xdp_full: 0
>> tx44_xdp_err: 0
>> tx44_xdp_cqes: 0
>> tx45_xdp_xmit: 0
>> tx45_xdp_full: 0
>> tx45_xdp_err: 0
>> tx45_xdp_cqes: 0
>> tx46_xdp_xmit: 0
>> tx46_xdp_full: 0
>> tx46_xdp_err: 0
>> tx46_xdp_cqes: 0
>> tx47_xdp_xmit: 0
>> tx47_xdp_full: 0
>> tx47_xdp_err: 0
>> tx47_xdp_cqes: 0
>> tx48_xdp_xmit: 0
>> tx48_xdp_full: 0
>> tx48_xdp_err: 0
>> tx48_xdp_cqes: 0
>> tx49_xdp_xmit: 0
>> tx49_xdp_full: 0
>> tx49_xdp_err: 0
>> tx49_xdp_cqes: 0
>> tx50_xdp_xmit: 0
>> tx50_xdp_full: 0
>> tx50_xdp_err: 0
>> tx50_xdp_cqes: 0
>> tx51_xdp_xmit: 0
>> tx51_xdp_full: 0
>> tx51_xdp_err: 0
>> tx51_xdp_cqes: 0
>> tx52_xdp_xmit: 0
>> tx52_xdp_full: 0
>> tx52_xdp_err: 0
>> tx52_xdp_cqes: 0
>> tx53_xdp_xmit: 0
>> tx53_xdp_full: 0
>> tx53_xdp_err: 0
>> tx53_xdp_cqes: 0
>> tx54_xdp_xmit: 0
>> tx54_xdp_full: 0
>> tx54_xdp_err: 0
>> tx54_xdp_cqes: 0
>> tx55_xdp_xmit: 0
>> tx55_xdp_full: 0
>> tx55_xdp_err: 0
>> tx55_xdp_cqes: 0
>>
>> ethtool -S enp175s0f0
>> NIC statistics:
>> rx_packets: 141574897253
>> rx_bytes: 184445040406258
>> tx_packets: 172569543894
>> tx_bytes: 99486882076365
>> tx_tso_packets: 9367664195
>> tx_tso_bytes: 56435233992948
>> tx_tso_inner_packets: 0
>> tx_tso_inner_bytes: 0
>> tx_added_vlan_packets: 141297671626
>> tx_nop: 2102916272
>> rx_lro_packets: 0
>> rx_lro_bytes: 0
>> rx_ecn_mark: 0
>> rx_removed_vlan_packets: 141574897252
>> rx_csum_unnecessary: 0
>> rx_csum_none: 23135854
>> rx_csum_complete: 141551761398
>> rx_csum_unnecessary_inner: 0
>> rx_xdp_drop: 0
>> rx_xdp_redirect: 0
>> rx_xdp_tx_xmit: 0
>> rx_xdp_tx_full: 0
>> rx_xdp_tx_err: 0
>> rx_xdp_tx_cqe: 0
>> tx_csum_none: 127934791664
> It is a good idea to look into this, tx is not requesting hw tx
> csumming for a lot of packets, maybe you are wasting a lot of cpu on
> calculating csum, or maybe this is just the rx csum complete..
>
>> tx_csum_partial: 13362879974
>> tx_csum_partial_inner: 0
>> tx_queue_stopped: 232561
> TX queues are stalling, could be an indentation for the pcie
> bottelneck.
>
>> tx_queue_dropped: 0
>> tx_xmit_more: 1266021946
>> tx_recover: 0
>> tx_cqes: 140031716469
>> tx_queue_wake: 232561
>> tx_udp_seg_rem: 0
>> tx_cqe_err: 0
>> tx_xdp_xmit: 0
>> tx_xdp_full: 0
>> tx_xdp_err: 0
>> tx_xdp_cqes: 0
>> rx_wqe_err: 0
>> rx_mpwqe_filler_cqes: 0
>> rx_mpwqe_filler_strides: 0
>> rx_buff_alloc_err: 0
>> rx_cqe_compress_blks: 0
>> rx_cqe_compress_pkts: 0
>> rx_page_reuse: 0
>> rx_cache_reuse: 16625975793
>> rx_cache_full: 54161465914
>> rx_cache_empty: 258048
>> rx_cache_busy: 54161472735
>> rx_cache_waive: 0
>> rx_congst_umr: 0
>> rx_arfs_err: 0
>> ch_events: 40572621887
>> ch_poll: 40885650979
>> ch_arm: 40429276692
>> ch_aff_change: 0
>> ch_eq_rearm: 0
>> rx_out_of_buffer: 2791690
>> rx_if_down_packets: 74
>> rx_vport_unicast_packets: 141843476308
>> rx_vport_unicast_bytes: 185421265403318
>> tx_vport_unicast_packets: 172569484005
>> tx_vport_unicast_bytes: 100019940094298
>> rx_vport_multicast_packets: 85122935
>> rx_vport_multicast_bytes: 5761316431
>> tx_vport_multicast_packets: 6452
>> tx_vport_multicast_bytes: 643540
>> rx_vport_broadcast_packets: 22423624
>> rx_vport_broadcast_bytes: 1390127090
>> tx_vport_broadcast_packets: 22024
>> tx_vport_broadcast_bytes: 1321440
>> rx_vport_rdma_unicast_packets: 0
>> rx_vport_rdma_unicast_bytes: 0
>> tx_vport_rdma_unicast_packets: 0
>> tx_vport_rdma_unicast_bytes: 0
>> rx_vport_rdma_multicast_packets: 0
>> rx_vport_rdma_multicast_bytes: 0
>> tx_vport_rdma_multicast_packets: 0
>> tx_vport_rdma_multicast_bytes: 0
>> tx_packets_phy: 172569501577
>> rx_packets_phy: 142871314588
>> rx_crc_errors_phy: 0
>> tx_bytes_phy: 100710212814151
>> rx_bytes_phy: 187209224289564
>> tx_multicast_phy: 6452
>> tx_broadcast_phy: 22024
>> rx_multicast_phy: 85122933
>> rx_broadcast_phy: 22423623
>> rx_in_range_len_errors_phy: 2
>> rx_out_of_range_len_phy: 0
>> rx_oversize_pkts_phy: 0
>> rx_symbol_err_phy: 0
>> tx_mac_control_phy: 0
>> rx_mac_control_phy: 0
>> rx_unsupported_op_phy: 0
>> rx_pause_ctrl_phy: 0
>> tx_pause_ctrl_phy: 0
>> rx_discards_phy: 920161423
> Ok, this port seem to be suffering more, RX is congested, maybe due to
> the pcie bottleneck.
Yes this side is receiving more traffic - second port is +10G more tx
>> tx_discards_phy: 0
>> tx_errors_phy: 0
>> rx_undersize_pkts_phy: 0
>> rx_fragments_phy: 0
>> rx_jabbers_phy: 0
>> rx_64_bytes_phy: 412006326
>> rx_65_to_127_bytes_phy: 11934371453
>> rx_128_to_255_bytes_phy: 3415281165
>> rx_256_to_511_bytes_phy: 2072955511
>> rx_512_to_1023_bytes_phy: 2415393005
>> rx_1024_to_1518_bytes_phy: 72182391608
>> rx_1519_to_2047_bytes_phy: 50438902587
>> rx_2048_to_4095_bytes_phy: 0
>> rx_4096_to_8191_bytes_phy: 0
>> rx_8192_to_10239_bytes_phy: 0
>> link_down_events_phy: 0
>> rx_pcs_symbol_err_phy: 0
>> rx_corrected_bits_phy: 0
>> rx_pci_signal_integrity: 0
>> tx_pci_signal_integrity: 48
>> rx_prio0_bytes: 186709842592642
>> rx_prio0_packets: 141481966007
>> tx_prio0_bytes: 100710171118138
>> tx_prio0_packets: 172569437949
>> rx_prio1_bytes: 492288152326
>> rx_prio1_packets: 385996045
>> tx_prio1_bytes: 0
>> tx_prio1_packets: 0
>> rx_prio2_bytes: 22119952
>> rx_prio2_packets: 70788
>> tx_prio2_bytes: 0
>> tx_prio2_packets: 0
>> rx_prio3_bytes: 546141102
>> rx_prio3_packets: 681608
>> tx_prio3_bytes: 0
>> tx_prio3_packets: 0
>> rx_prio4_bytes: 14665067
>> rx_prio4_packets: 29486
>> tx_prio4_bytes: 0
>> tx_prio4_packets: 0
>> rx_prio5_bytes: 158862504
>> rx_prio5_packets: 965307
>> tx_prio5_bytes: 0
>> tx_prio5_packets: 0
>> rx_prio6_bytes: 669337783
>> rx_prio6_packets: 1475775
>> tx_prio6_bytes: 0
>> tx_prio6_packets: 0
>> rx_prio7_bytes: 5623481349
>> rx_prio7_packets: 79926412
>> tx_prio7_bytes: 0
>> tx_prio7_packets: 0
>> module_unplug: 0
>> module_bus_stuck: 0
>> module_high_temp: 0
>> module_bad_shorted: 0
>> ch0_events: 1446162630
>> ch0_poll: 1463312972
>> ch0_arm: 1440728278
>> ch0_aff_change: 0
>> ch0_eq_rearm: 0
>> ch1_events: 1384301405
>> ch1_poll: 1399210915
>> ch1_arm: 1378636486
>> ch1_aff_change: 0
>> ch1_eq_rearm: 0
>> ch2_events: 1382788887
>> ch2_poll: 1397231470
>> ch2_arm: 1377058116
>> ch2_aff_change: 0
>> ch2_eq_rearm: 0
>> ch3_events: 1461956995
>> ch3_poll: 1475553146
>> ch3_arm: 1456571625
>> ch3_aff_change: 0
>> ch3_eq_rearm: 0
>> ch4_events: 1497359109
>> ch4_poll: 1511021037
>> ch4_arm: 1491733757
>> ch4_aff_change: 0
>> ch4_eq_rearm: 0
>> ch5_events: 1387736262
>> ch5_poll: 1400964615
>> ch5_arm: 1382382834
>> ch5_aff_change: 0
>> ch5_eq_rearm: 0
>> ch6_events: 1376772405
>> ch6_poll: 1390851449
>> ch6_arm: 1371551764
>> ch6_aff_change: 0
>> ch6_eq_rearm: 0
>> ch7_events: 1431271514
>> ch7_poll: 1445049729
>> ch7_arm: 1425753718
>> ch7_aff_change: 0
>> ch7_eq_rearm: 0
>> ch8_events: 1426976374
>> ch8_poll: 1439938692
>> ch8_arm: 1421392984
>> ch8_aff_change: 0
>> ch8_eq_rearm: 0
>> ch9_events: 1456160031
>> ch9_poll: 1468922870
>> ch9_arm: 1450930446
>> ch9_aff_change: 0
>> ch9_eq_rearm: 0
>> ch10_events: 1443640165
>> ch10_poll: 1456812203
>> ch10_arm: 1438425101
>> ch10_aff_change: 0
>> ch10_eq_rearm: 0
>> ch11_events: 1381104776
>> ch11_poll: 1393811057
>> ch11_arm: 1376059326
>> ch11_aff_change: 0
>> ch11_eq_rearm: 0
>> ch12_events: 1365223276
>> ch12_poll: 1378406059
>> ch12_arm: 1359950494
>> ch12_aff_change: 0
>> ch12_eq_rearm: 0
>> ch13_events: 1421622259
>> ch13_poll: 1434670996
>> ch13_arm: 1416241801
>> ch13_aff_change: 0
>> ch13_eq_rearm: 0
>> ch14_events: 1379084590
>> ch14_poll: 1392425015
>> ch14_arm: 1373675179
>> ch14_aff_change: 0
>> ch14_eq_rearm: 0
>> ch15_events: 1531217338
>> ch15_poll: 1543353833
>> ch15_arm: 1526350453
>> ch15_aff_change: 0
>> ch15_eq_rearm: 0
>> ch16_events: 1460469776
>> ch16_poll: 1467995928
>> ch16_arm: 1456010194
>> ch16_aff_change: 0
>> ch16_eq_rearm: 0
>> ch17_events: 1494067670
>> ch17_poll: 1500856680
>> ch17_arm: 1489232674
>> ch17_aff_change: 0
>> ch17_eq_rearm: 0
>> ch18_events: 1530126866
>> ch18_poll: 1537293620
>> ch18_arm: 1525476123
>> ch18_aff_change: 0
>> ch18_eq_rearm: 0
>> ch19_events: 1499526149
>> ch19_poll: 1506789309
>> ch19_arm: 1495161602
>> ch19_aff_change: 0
>> ch19_eq_rearm: 0
>> ch20_events: 1451479763
>> ch20_poll: 1459767921
>> ch20_arm: 1446360801
>> ch20_aff_change: 0
>> ch20_eq_rearm: 0
>> ch21_events: 1521413613
>> ch21_poll: 1529345146
>> ch21_arm: 1517229314
>> ch21_aff_change: 0
>> ch21_eq_rearm: 0
>> ch22_events: 1471950045
>> ch22_poll: 1479746764
>> ch22_arm: 1467681629
>> ch22_aff_change: 0
>> ch22_eq_rearm: 0
>> ch23_events: 1502968393
>> ch23_poll: 1510419909
>> ch23_arm: 1498168438
>> ch23_aff_change: 0
>> ch23_eq_rearm: 0
>> ch24_events: 1473451639
>> ch24_poll: 1482606899
>> ch24_arm: 1468212489
>> ch24_aff_change: 0
>> ch24_eq_rearm: 0
>> ch25_events: 1440399182
>> ch25_poll: 1448897475
>> ch25_arm: 1435044786
>> ch25_aff_change: 0
>> ch25_eq_rearm: 0
>> ch26_events: 1436831565
>> ch26_poll: 1445485731
>> ch26_arm: 1431827527
>> ch26_aff_change: 0
>> ch26_eq_rearm: 0
>> ch27_events: 1516560621
>> ch27_poll: 1524911010
>> ch27_arm: 1511430164
>> ch27_aff_change: 0
>> ch27_eq_rearm: 0
>> ch28_events: 4
>> ch28_poll: 4
>> ch28_arm: 4
>> ch28_aff_change: 0
>> ch28_eq_rearm: 0
>> ch29_events: 6
>> ch29_poll: 6
>> ch29_arm: 6
>> ch29_aff_change: 0
>> ch29_eq_rearm: 0
>> ch30_events: 4
>> ch30_poll: 4
>> ch30_arm: 4
>> ch30_aff_change: 0
>> ch30_eq_rearm: 0
>> ch31_events: 4
>> ch31_poll: 4
>> ch31_arm: 4
>> ch31_aff_change: 0
>> ch31_eq_rearm: 0
>> ch32_events: 4
>> ch32_poll: 4
>> ch32_arm: 4
>> ch32_aff_change: 0
>> ch32_eq_rearm: 0
>> ch33_events: 4
>> ch33_poll: 4
>> ch33_arm: 4
>> ch33_aff_change: 0
>> ch33_eq_rearm: 0
>> ch34_events: 4
>> ch34_poll: 4
>> ch34_arm: 4
>> ch34_aff_change: 0
>> ch34_eq_rearm: 0
>> ch35_events: 4
>> ch35_poll: 4
>> ch35_arm: 4
>> ch35_aff_change: 0
>> ch35_eq_rearm: 0
>> ch36_events: 4
>> ch36_poll: 4
>> ch36_arm: 4
>> ch36_aff_change: 0
>> ch36_eq_rearm: 0
>> ch37_events: 4
>> ch37_poll: 4
>> ch37_arm: 4
>> ch37_aff_change: 0
>> ch37_eq_rearm: 0
>> ch38_events: 4
>> ch38_poll: 4
>> ch38_arm: 4
>> ch38_aff_change: 0
>> ch38_eq_rearm: 0
>> ch39_events: 4
>> ch39_poll: 4
>> ch39_arm: 4
>> ch39_aff_change: 0
>> ch39_eq_rearm: 0
>> ch40_events: 4
>> ch40_poll: 4
>> ch40_arm: 4
>> ch40_aff_change: 0
>> ch40_eq_rearm: 0
>> ch41_events: 4
>> ch41_poll: 4
>> ch41_arm: 4
>> ch41_aff_change: 0
>> ch41_eq_rearm: 0
>> ch42_events: 4
>> ch42_poll: 4
>> ch42_arm: 4
>> ch42_aff_change: 0
>> ch42_eq_rearm: 0
>> ch43_events: 4
>> ch43_poll: 4
>> ch43_arm: 4
>> ch43_aff_change: 0
>> ch43_eq_rearm: 0
>> ch44_events: 4
>> ch44_poll: 4
>> ch44_arm: 4
>> ch44_aff_change: 0
>> ch44_eq_rearm: 0
>> ch45_events: 4
>> ch45_poll: 4
>> ch45_arm: 4
>> ch45_aff_change: 0
>> ch45_eq_rearm: 0
>> ch46_events: 4
>> ch46_poll: 4
>> ch46_arm: 4
>> ch46_aff_change: 0
>> ch46_eq_rearm: 0
>> ch47_events: 4
>> ch47_poll: 4
>> ch47_arm: 4
>> ch47_aff_change: 0
>> ch47_eq_rearm: 0
>> ch48_events: 4
>> ch48_poll: 4
>> ch48_arm: 4
>> ch48_aff_change: 0
>> ch48_eq_rearm: 0
>> ch49_events: 4
>> ch49_poll: 4
>> ch49_arm: 4
>> ch49_aff_change: 0
>> ch49_eq_rearm: 0
>> ch50_events: 4
>> ch50_poll: 4
>> ch50_arm: 4
>> ch50_aff_change: 0
>> ch50_eq_rearm: 0
>> ch51_events: 4
>> ch51_poll: 4
>> ch51_arm: 4
>> ch51_aff_change: 0
>> ch51_eq_rearm: 0
>> ch52_events: 4
>> ch52_poll: 4
>> ch52_arm: 4
>> ch52_aff_change: 0
>> ch52_eq_rearm: 0
>> ch53_events: 4
>> ch53_poll: 4
>> ch53_arm: 4
>> ch53_aff_change: 0
>> ch53_eq_rearm: 0
>> ch54_events: 4
>> ch54_poll: 4
>> ch54_arm: 4
>> ch54_aff_change: 0
>> ch54_eq_rearm: 0
>> ch55_events: 4
>> ch55_poll: 4
>> ch55_arm: 4
>> ch55_aff_change: 0
>> ch55_eq_rearm: 0
>> rx0_packets: 5861448653
>> rx0_bytes: 7389128595728
>> rx0_csum_complete: 5838312798
>> rx0_csum_unnecessary: 0
>> rx0_csum_unnecessary_inner: 0
>> rx0_csum_none: 23135855
>> rx0_xdp_drop: 0
>> rx0_xdp_redirect: 0
>> rx0_lro_packets: 0
>> rx0_lro_bytes: 0
>> rx0_ecn_mark: 0
>> rx0_removed_vlan_packets: 5861448653
>> rx0_wqe_err: 0
>> rx0_mpwqe_filler_cqes: 0
>> rx0_mpwqe_filler_strides: 0
>> rx0_buff_alloc_err: 0
>> rx0_cqe_compress_blks: 0
>> rx0_cqe_compress_pkts: 0
>> rx0_page_reuse: 0
>> rx0_cache_reuse: 2559
>> rx0_cache_full: 2930721512
>> rx0_cache_empty: 6656
>> rx0_cache_busy: 2930721765
>> rx0_cache_waive: 0
>> rx0_congst_umr: 0
>> rx0_arfs_err: 0
>> rx0_xdp_tx_xmit: 0
>> rx0_xdp_tx_full: 0
>> rx0_xdp_tx_err: 0
>> rx0_xdp_tx_cqes: 0
>> rx1_packets: 5550585106
>> rx1_bytes: 7255635262803
>> rx1_csum_complete: 5550585106
>> rx1_csum_unnecessary: 0
>> rx1_csum_unnecessary_inner: 0
>> rx1_csum_none: 0
>> rx1_xdp_drop: 0
>> rx1_xdp_redirect: 0
>> rx1_lro_packets: 0
>> rx1_lro_bytes: 0
>> rx1_ecn_mark: 0
>> rx1_removed_vlan_packets: 5550585106
>> rx1_wqe_err: 0
>> rx1_mpwqe_filler_cqes: 0
>> rx1_mpwqe_filler_strides: 0
>> rx1_buff_alloc_err: 0
>> rx1_cqe_compress_blks: 0
>> rx1_cqe_compress_pkts: 0
>> rx1_page_reuse: 0
>> rx1_cache_reuse: 2918845
>> rx1_cache_full: 2772373453
>> rx1_cache_empty: 6656
>> rx1_cache_busy: 2772373707
>> rx1_cache_waive: 0
>> rx1_congst_umr: 0
>> rx1_arfs_err: 0
>> rx1_xdp_tx_xmit: 0
>> rx1_xdp_tx_full: 0
>> rx1_xdp_tx_err: 0
>> rx1_xdp_tx_cqes: 0
>> rx2_packets: 5383874739
>> rx2_bytes: 7031545423967
>> rx2_csum_complete: 5383874739
>> rx2_csum_unnecessary: 0
>> rx2_csum_unnecessary_inner: 0
>> rx2_csum_none: 0
>> rx2_xdp_drop: 0
>> rx2_xdp_redirect: 0
>> rx2_lro_packets: 0
>> rx2_lro_bytes: 0
>> rx2_ecn_mark: 0
>> rx2_removed_vlan_packets: 5383874739
>> rx2_wqe_err: 0
>> rx2_mpwqe_filler_cqes: 0
>> rx2_mpwqe_filler_strides: 0
>> rx2_buff_alloc_err: 0
>> rx2_cqe_compress_blks: 0
>> rx2_cqe_compress_pkts: 0
>> rx2_page_reuse: 0
>> rx2_cache_reuse: 2173370
>> rx2_cache_full: 2689763744
>> rx2_cache_empty: 6656
>> rx2_cache_busy: 2689763998
>> rx2_cache_waive: 0
>> rx2_congst_umr: 0
>> rx2_arfs_err: 0
>> rx2_xdp_tx_xmit: 0
>> rx2_xdp_tx_full: 0
>> rx2_xdp_tx_err: 0
>> rx2_xdp_tx_cqes: 0
>> rx3_packets: 5456494012
>> rx3_bytes: 7120241119485
>> rx3_csum_complete: 5456494012
>> rx3_csum_unnecessary: 0
>> rx3_csum_unnecessary_inner: 0
>> rx3_csum_none: 0
>> rx3_xdp_drop: 0
>> rx3_xdp_redirect: 0
>> rx3_lro_packets: 0
>> rx3_lro_bytes: 0
>> rx3_ecn_mark: 0
>> rx3_removed_vlan_packets: 5456494012
>> rx3_wqe_err: 0
>> rx3_mpwqe_filler_cqes: 0
>> rx3_mpwqe_filler_strides: 0
>> rx3_buff_alloc_err: 0
>> rx3_cqe_compress_blks: 0
>> rx3_cqe_compress_pkts: 0
>> rx3_page_reuse: 0
>> rx3_cache_reuse: 2120123
>> rx3_cache_full: 2726126628
>> rx3_cache_empty: 6656
>> rx3_cache_busy: 2726126881
>> rx3_cache_waive: 0
>> rx3_congst_umr: 0
>> rx3_arfs_err: 0
>> rx3_xdp_tx_xmit: 0
>> rx3_xdp_tx_full: 0
>> rx3_xdp_tx_err: 0
>> rx3_xdp_tx_cqes: 0
>> rx4_packets: 5475216251
>> rx4_bytes: 7123129170196
>> rx4_csum_complete: 5475216251
>> rx4_csum_unnecessary: 0
>> rx4_csum_unnecessary_inner: 0
>> rx4_csum_none: 0
>> rx4_xdp_drop: 0
>> rx4_xdp_redirect: 0
>> rx4_lro_packets: 0
>> rx4_lro_bytes: 0
>> rx4_ecn_mark: 0
>> rx4_removed_vlan_packets: 5475216251
>> rx4_wqe_err: 0
>> rx4_mpwqe_filler_cqes: 0
>> rx4_mpwqe_filler_strides: 0
>> rx4_buff_alloc_err: 0
>> rx4_cqe_compress_blks: 0
>> rx4_cqe_compress_pkts: 0
>> rx4_page_reuse: 0
>> rx4_cache_reuse: 2668296355
>> rx4_cache_full: 69311549
>> rx4_cache_empty: 6656
>> rx4_cache_busy: 69311769
>> rx4_cache_waive: 0
>> rx4_congst_umr: 0
>> rx4_arfs_err: 0
>> rx4_xdp_tx_xmit: 0
>> rx4_xdp_tx_full: 0
>> rx4_xdp_tx_err: 0
>> rx4_xdp_tx_cqes: 0
>> rx5_packets: 5474372232
>> rx5_bytes: 7159146801926
>> rx5_csum_complete: 5474372232
>> rx5_csum_unnecessary: 0
>> rx5_csum_unnecessary_inner: 0
>> rx5_csum_none: 0
>> rx5_xdp_drop: 0
>> rx5_xdp_redirect: 0
>> rx5_lro_packets: 0
>> rx5_lro_bytes: 0
>> rx5_ecn_mark: 0
>> rx5_removed_vlan_packets: 5474372232
>> rx5_wqe_err: 0
>> rx5_mpwqe_filler_cqes: 0
>> rx5_mpwqe_filler_strides: 0
>> rx5_buff_alloc_err: 0
>> rx5_cqe_compress_blks: 0
>> rx5_cqe_compress_pkts: 0
>> rx5_page_reuse: 0
>> rx5_cache_reuse: 626187
>> rx5_cache_full: 2736559674
>> rx5_cache_empty: 6656
>> rx5_cache_busy: 2736559929
>> rx5_cache_waive: 0
>> rx5_congst_umr: 0
>> rx5_arfs_err: 0
>> rx5_xdp_tx_xmit: 0
>> rx5_xdp_tx_full: 0
>> rx5_xdp_tx_err: 0
>> rx5_xdp_tx_cqes: 0
>> rx6_packets: 5533622456
>> rx6_bytes: 7207308809081
>> rx6_csum_complete: 5533622456
>> rx6_csum_unnecessary: 0
>> rx6_csum_unnecessary_inner: 0
>> rx6_csum_none: 0
>> rx6_xdp_drop: 0
>> rx6_xdp_redirect: 0
>> rx6_lro_packets: 0
>> rx6_lro_bytes: 0
>> rx6_ecn_mark: 0
>> rx6_removed_vlan_packets: 5533622456
>> rx6_wqe_err: 0
>> rx6_mpwqe_filler_cqes: 0
>> rx6_mpwqe_filler_strides: 0
>> rx6_buff_alloc_err: 0
>> rx6_cqe_compress_blks: 0
>> rx6_cqe_compress_pkts: 0
>> rx6_page_reuse: 0
>> rx6_cache_reuse: 2325217
>> rx6_cache_full: 2764485756
>> rx6_cache_empty: 6656
>> rx6_cache_busy: 2764486011
>> rx6_cache_waive: 0
>> rx6_congst_umr: 0
>> rx6_arfs_err: 0
>> rx6_xdp_tx_xmit: 0
>> rx6_xdp_tx_full: 0
>> rx6_xdp_tx_err: 0
>> rx6_xdp_tx_cqes: 0
>> rx7_packets: 5533901822
>> rx7_bytes: 7227441240536
>> rx7_csum_complete: 5533901822
>> rx7_csum_unnecessary: 0
>> rx7_csum_unnecessary_inner: 0
>> rx7_csum_none: 0
>> rx7_xdp_drop: 0
>> rx7_xdp_redirect: 0
>> rx7_lro_packets: 0
>> rx7_lro_bytes: 0
>> rx7_ecn_mark: 0
>> rx7_removed_vlan_packets: 5533901822
>> rx7_wqe_err: 0
>> rx7_mpwqe_filler_cqes: 0
>> rx7_mpwqe_filler_strides: 0
>> rx7_buff_alloc_err: 0
>> rx7_cqe_compress_blks: 0
>> rx7_cqe_compress_pkts: 0
>> rx7_page_reuse: 0
>> rx7_cache_reuse: 2372505
>> rx7_cache_full: 2764578151
>> rx7_cache_empty: 6656
>> rx7_cache_busy: 2764578403
>> rx7_cache_waive: 0
>> rx7_congst_umr: 0
>> rx7_arfs_err: 0
>> rx7_xdp_tx_xmit: 0
>> rx7_xdp_tx_full: 0
>> rx7_xdp_tx_err: 0
>> rx7_xdp_tx_cqes: 0
>> rx8_packets: 5485670137
>> rx8_bytes: 7203339989013
>> rx8_csum_complete: 5485670137
>> rx8_csum_unnecessary: 0
>> rx8_csum_unnecessary_inner: 0
>> rx8_csum_none: 0
>> rx8_xdp_drop: 0
>> rx8_xdp_redirect: 0
>> rx8_lro_packets: 0
>> rx8_lro_bytes: 0
>> rx8_ecn_mark: 0
>> rx8_removed_vlan_packets: 5485670137
>> rx8_wqe_err: 0
>> rx8_mpwqe_filler_cqes: 0
>> rx8_mpwqe_filler_strides: 0
>> rx8_buff_alloc_err: 0
>> rx8_cqe_compress_blks: 0
>> rx8_cqe_compress_pkts: 0
>> rx8_page_reuse: 0
>> rx8_cache_reuse: 7522232
>> rx8_cache_full: 2735312581
>> rx8_cache_empty: 6656
>> rx8_cache_busy: 2735312836
>> rx8_cache_waive: 0
>> rx8_congst_umr: 0
>> rx8_arfs_err: 0
>> rx8_xdp_tx_xmit: 0
>> rx8_xdp_tx_full: 0
>> rx8_xdp_tx_err: 0
>> rx8_xdp_tx_cqes: 0
>> rx9_packets: 5482212354
>> rx9_bytes: 7169663341718
>> rx9_csum_complete: 5482212354
>> rx9_csum_unnecessary: 0
>> rx9_csum_unnecessary_inner: 0
>> rx9_csum_none: 0
>> rx9_xdp_drop: 0
>> rx9_xdp_redirect: 0
>> rx9_lro_packets: 0
>> rx9_lro_bytes: 0
>> rx9_ecn_mark: 0
>> rx9_removed_vlan_packets: 5482212354
>> rx9_wqe_err: 0
>> rx9_mpwqe_filler_cqes: 0
>> rx9_mpwqe_filler_strides: 0
>> rx9_buff_alloc_err: 0
>> rx9_cqe_compress_blks: 0
>> rx9_cqe_compress_pkts: 0
>> rx9_page_reuse: 0
>> rx9_cache_reuse: 37279961
>> rx9_cache_full: 2703825961
>> rx9_cache_empty: 6656
>> rx9_cache_busy: 2703826215
>> rx9_cache_waive: 0
>> rx9_congst_umr: 0
>> rx9_arfs_err: 0
>> rx9_xdp_tx_xmit: 0
>> rx9_xdp_tx_full: 0
>> rx9_xdp_tx_err: 0
>> rx9_xdp_tx_cqes: 0
>> rx10_packets: 5524679952
>> rx10_bytes: 7248301275181
>> rx10_csum_complete: 5524679952
>> rx10_csum_unnecessary: 0
>> rx10_csum_unnecessary_inner: 0
>> rx10_csum_none: 0
>> rx10_xdp_drop: 0
>> rx10_xdp_redirect: 0
>> rx10_lro_packets: 0
>> rx10_lro_bytes: 0
>> rx10_ecn_mark: 0
>> rx10_removed_vlan_packets: 5524679952
>> rx10_wqe_err: 0
>> rx10_mpwqe_filler_cqes: 0
>> rx10_mpwqe_filler_strides: 0
>> rx10_buff_alloc_err: 0
>> rx10_cqe_compress_blks: 0
>> rx10_cqe_compress_pkts: 0
>> rx10_page_reuse: 0
>> rx10_cache_reuse: 2049666
>> rx10_cache_full: 2760290055
>> rx10_cache_empty: 6656
>> rx10_cache_busy: 2760290310
>> rx10_cache_waive: 0
>> rx10_congst_umr: 0
>> rx10_arfs_err: 0
>> rx10_xdp_tx_xmit: 0
>> rx10_xdp_tx_full: 0
>> rx10_xdp_tx_err: 0
>> rx10_xdp_tx_cqes: 0
>> rx11_packets: 5394633545
>> rx11_bytes: 7033509636092
>> rx11_csum_complete: 5394633545
>> rx11_csum_unnecessary: 0
>> rx11_csum_unnecessary_inner: 0
>> rx11_csum_none: 0
>> rx11_xdp_drop: 0
>> rx11_xdp_redirect: 0
>> rx11_lro_packets: 0
>> rx11_lro_bytes: 0
>> rx11_ecn_mark: 0
>> rx11_removed_vlan_packets: 5394633545
>> rx11_wqe_err: 0
>> rx11_mpwqe_filler_cqes: 0
>> rx11_mpwqe_filler_strides: 0
>> rx11_buff_alloc_err: 0
>> rx11_cqe_compress_blks: 0
>> rx11_cqe_compress_pkts: 0
>> rx11_page_reuse: 0
>> rx11_cache_reuse: 2617466268
>> rx11_cache_full: 79850284
>> rx11_cache_empty: 6656
>> rx11_cache_busy: 79850504
>> rx11_cache_waive: 0
>> rx11_congst_umr: 0
>> rx11_arfs_err: 0
>> rx11_xdp_tx_xmit: 0
>> rx11_xdp_tx_full: 0
>> rx11_xdp_tx_err: 0
>> rx11_xdp_tx_cqes: 0
>> rx12_packets: 5458907385
>> rx12_bytes: 7134867867515
>> rx12_csum_complete: 5458907385
>> rx12_csum_unnecessary: 0
>> rx12_csum_unnecessary_inner: 0
>> rx12_csum_none: 0
>> rx12_xdp_drop: 0
>> rx12_xdp_redirect: 0
>> rx12_lro_packets: 0
>> rx12_lro_bytes: 0
>> rx12_ecn_mark: 0
>> rx12_removed_vlan_packets: 5458907385
>> rx12_wqe_err: 0
>> rx12_mpwqe_filler_cqes: 0
>> rx12_mpwqe_filler_strides: 0
>> rx12_buff_alloc_err: 0
>> rx12_cqe_compress_blks: 0
>> rx12_cqe_compress_pkts: 0
>> rx12_page_reuse: 0
>> rx12_cache_reuse: 2650214169
>> rx12_cache_full: 79239303
>> rx12_cache_empty: 6656
>> rx12_cache_busy: 79239523
>> rx12_cache_waive: 0
>> rx12_congst_umr: 0
>> rx12_arfs_err: 0
>> rx12_xdp_tx_xmit: 0
>> rx12_xdp_tx_full: 0
>> rx12_xdp_tx_err: 0
>> rx12_xdp_tx_cqes: 0
>> rx13_packets: 5549932912
>> rx13_bytes: 7232548705586
>> rx13_csum_complete: 5549932912
>> rx13_csum_unnecessary: 0
>> rx13_csum_unnecessary_inner: 0
>> rx13_csum_none: 0
>> rx13_xdp_drop: 0
>> rx13_xdp_redirect: 0
>> rx13_lro_packets: 0
>> rx13_lro_bytes: 0
>> rx13_ecn_mark: 0
>> rx13_removed_vlan_packets: 5549932912
>> rx13_wqe_err: 0
>> rx13_mpwqe_filler_cqes: 0
>> rx13_mpwqe_filler_strides: 0
>> rx13_buff_alloc_err: 0
>> rx13_cqe_compress_blks: 0
>> rx13_cqe_compress_pkts: 0
>> rx13_page_reuse: 0
>> rx13_cache_reuse: 2417696
>> rx13_cache_full: 2772548505
>> rx13_cache_empty: 6656
>> rx13_cache_busy: 2772548760
>> rx13_cache_waive: 0
>> rx13_congst_umr: 0
>> rx13_arfs_err: 0
>> rx13_xdp_tx_xmit: 0
>> rx13_xdp_tx_full: 0
>> rx13_xdp_tx_err: 0
>> rx13_xdp_tx_cqes: 0
>> rx14_packets: 5517712329
>> rx14_bytes: 7192111965227
>> rx14_csum_complete: 5517712329
>> rx14_csum_unnecessary: 0
>> rx14_csum_unnecessary_inner: 0
>> rx14_csum_none: 0
>> rx14_xdp_drop: 0
>> rx14_xdp_redirect: 0
>> rx14_lro_packets: 0
>> rx14_lro_bytes: 0
>> rx14_ecn_mark: 0
>> rx14_removed_vlan_packets: 5517712329
>> rx14_wqe_err: 0
>> rx14_mpwqe_filler_cqes: 0
>> rx14_mpwqe_filler_strides: 0
>> rx14_buff_alloc_err: 0
>> rx14_cqe_compress_blks: 0
>> rx14_cqe_compress_pkts: 0
>> rx14_page_reuse: 0
>> rx14_cache_reuse: 1830206
>> rx14_cache_full: 2757025703
>> rx14_cache_empty: 6656
>> rx14_cache_busy: 2757025958
>> rx14_cache_waive: 0
>> rx14_congst_umr: 0
>> rx14_arfs_err: 0
>> rx14_xdp_tx_xmit: 0
>> rx14_xdp_tx_full: 0
>> rx14_xdp_tx_err: 0
>> rx14_xdp_tx_cqes: 0
>> rx15_packets: 5578343373
>> rx15_bytes: 7268484501219
>> rx15_csum_complete: 5578343373
>> rx15_csum_unnecessary: 0
>> rx15_csum_unnecessary_inner: 0
>> rx15_csum_none: 0
>> rx15_xdp_drop: 0
>> rx15_xdp_redirect: 0
>> rx15_lro_packets: 0
>> rx15_lro_bytes: 0
>> rx15_ecn_mark: 0
>> rx15_removed_vlan_packets: 5578343373
>> rx15_wqe_err: 0
>> rx15_mpwqe_filler_cqes: 0
>> rx15_mpwqe_filler_strides: 0
>> rx15_buff_alloc_err: 0
>> rx15_cqe_compress_blks: 0
>> rx15_cqe_compress_pkts: 0
>> rx15_page_reuse: 0
>> rx15_cache_reuse: 2317165
>> rx15_cache_full: 2786854266
>> rx15_cache_empty: 6656
>> rx15_cache_busy: 2786854519
>> rx15_cache_waive: 0
>> rx15_congst_umr: 0
>> rx15_arfs_err: 0
>> rx15_xdp_tx_xmit: 0
>> rx15_xdp_tx_full: 0
>> rx15_xdp_tx_err: 0
>> rx15_xdp_tx_cqes: 0
>> rx16_packets: 4435773951
>> rx16_bytes: 5766665272007
>> rx16_csum_complete: 4435773951
>> rx16_csum_unnecessary: 0
>> rx16_csum_unnecessary_inner: 0
>> rx16_csum_none: 0
>> rx16_xdp_drop: 0
>> rx16_xdp_redirect: 0
>> rx16_lro_packets: 0
>> rx16_lro_bytes: 0
>> rx16_ecn_mark: 0
>> rx16_removed_vlan_packets: 4435773951
>> rx16_wqe_err: 0
>> rx16_mpwqe_filler_cqes: 0
>> rx16_mpwqe_filler_strides: 0
>> rx16_buff_alloc_err: 0
>> rx16_cqe_compress_blks: 0
>> rx16_cqe_compress_pkts: 0
>> rx16_page_reuse: 0
>> rx16_cache_reuse: 2033793
>> rx16_cache_full: 2215852927
>> rx16_cache_empty: 6656
>> rx16_cache_busy: 2215853179
>> rx16_cache_waive: 0
>> rx16_congst_umr: 0
>> rx16_arfs_err: 0
>> rx16_xdp_tx_xmit: 0
>> rx16_xdp_tx_full: 0
>> rx16_xdp_tx_err: 0
>> rx16_xdp_tx_cqes: 0
>> rx17_packets: 4344087587
>> rx17_bytes: 5695006496323
>> rx17_csum_complete: 4344087587
>> rx17_csum_unnecessary: 0
>> rx17_csum_unnecessary_inner: 0
>> rx17_csum_none: 0
>> rx17_xdp_drop: 0
>> rx17_xdp_redirect: 0
>> rx17_lro_packets: 0
>> rx17_lro_bytes: 0
>> rx17_ecn_mark: 0
>> rx17_removed_vlan_packets: 4344087587
>> rx17_wqe_err: 0
>> rx17_mpwqe_filler_cqes: 0
>> rx17_mpwqe_filler_strides: 0
>> rx17_buff_alloc_err: 0
>> rx17_cqe_compress_blks: 0
>> rx17_cqe_compress_pkts: 0
>> rx17_page_reuse: 0
>> rx17_cache_reuse: 2652127
>> rx17_cache_full: 2169391411
>> rx17_cache_empty: 6656
>> rx17_cache_busy: 2169391665
>> rx17_cache_waive: 0
>> rx17_congst_umr: 0
>> rx17_arfs_err: 0
>> rx17_xdp_tx_xmit: 0
>> rx17_xdp_tx_full: 0
>> rx17_xdp_tx_err: 0
>> rx17_xdp_tx_cqes: 0
>> rx18_packets: 4407422804
>> rx18_bytes: 5741134634177
>> rx18_csum_complete: 4407422804
>> rx18_csum_unnecessary: 0
>> rx18_csum_unnecessary_inner: 0
>> rx18_csum_none: 0
>> rx18_xdp_drop: 0
>> rx18_xdp_redirect: 0
>> rx18_lro_packets: 0
>> rx18_lro_bytes: 0
>> rx18_ecn_mark: 0
>> rx18_removed_vlan_packets: 4407422804
>> rx18_wqe_err: 0
>> rx18_mpwqe_filler_cqes: 0
>> rx18_mpwqe_filler_strides: 0
>> rx18_buff_alloc_err: 0
>> rx18_cqe_compress_blks: 0
>> rx18_cqe_compress_pkts: 0
>> rx18_page_reuse: 0
>> rx18_cache_reuse: 2156080239
>> rx18_cache_full: 47630941
>> rx18_cache_empty: 6656
>> rx18_cache_busy: 47631161
>> rx18_cache_waive: 0
>> rx18_congst_umr: 0
>> rx18_arfs_err: 0
>> rx18_xdp_tx_xmit: 0
>> rx18_xdp_tx_full: 0
>> rx18_xdp_tx_err: 0
>> rx18_xdp_tx_cqes: 0
>> rx19_packets: 4545554180
>> rx19_bytes: 5905277503466
>> rx19_csum_complete: 4545554180
>> rx19_csum_unnecessary: 0
>> rx19_csum_unnecessary_inner: 0
>> rx19_csum_none: 0
>> rx19_xdp_drop: 0
>> rx19_xdp_redirect: 0
>> rx19_lro_packets: 0
>> rx19_lro_bytes: 0
>> rx19_ecn_mark: 0
>> rx19_removed_vlan_packets: 4545554180
>> rx19_wqe_err: 0
>> rx19_mpwqe_filler_cqes: 0
>> rx19_mpwqe_filler_strides: 0
>> rx19_buff_alloc_err: 0
>> rx19_cqe_compress_blks: 0
>> rx19_cqe_compress_pkts: 0
>> rx19_page_reuse: 0
>> rx19_cache_reuse: 11112455
>> rx19_cache_full: 2261664379
>> rx19_cache_empty: 6656
>> rx19_cache_busy: 2261664601
>> rx19_cache_waive: 0
>> rx19_congst_umr: 0
>> rx19_arfs_err: 0
>> rx19_xdp_tx_xmit: 0
>> rx19_xdp_tx_full: 0
>> rx19_xdp_tx_err: 0
>> rx19_xdp_tx_cqes: 0
>> rx20_packets: 4397428553
>> rx20_bytes: 5757329184301
>> rx20_csum_complete: 4397428553
>> rx20_csum_unnecessary: 0
>> rx20_csum_unnecessary_inner: 0
>> rx20_csum_none: 0
>> rx20_xdp_drop: 0
>> rx20_xdp_redirect: 0
>> rx20_lro_packets: 0
>> rx20_lro_bytes: 0
>> rx20_ecn_mark: 0
>> rx20_removed_vlan_packets: 4397428553
>> rx20_wqe_err: 0
>> rx20_mpwqe_filler_cqes: 0
>> rx20_mpwqe_filler_strides: 0
>> rx20_buff_alloc_err: 0
>> rx20_cqe_compress_blks: 0
>> rx20_cqe_compress_pkts: 0
>> rx20_page_reuse: 0
>> rx20_cache_reuse: 2168116995
>> rx20_cache_full: 30597061
>> rx20_cache_empty: 6656
>> rx20_cache_busy: 30597281
>> rx20_cache_waive: 0
>> rx20_congst_umr: 0
>> rx20_arfs_err: 0
>> rx20_xdp_tx_xmit: 0
>> rx20_xdp_tx_full: 0
>> rx20_xdp_tx_err: 0
>> rx20_xdp_tx_cqes: 0
>> rx21_packets: 4552564821
>> rx21_bytes: 5944840329249
>> rx21_csum_complete: 4552564821
>> rx21_csum_unnecessary: 0
>> rx21_csum_unnecessary_inner: 0
>> rx21_csum_none: 0
>> rx21_xdp_drop: 0
>> rx21_xdp_redirect: 0
>> rx21_lro_packets: 0
>> rx21_lro_bytes: 0
>> rx21_ecn_mark: 0
>> rx21_removed_vlan_packets: 4552564821
>> rx21_wqe_err: 0
>> rx21_mpwqe_filler_cqes: 0
>> rx21_mpwqe_filler_strides: 0
>> rx21_buff_alloc_err: 0
>> rx21_cqe_compress_blks: 0
>> rx21_cqe_compress_pkts: 0
>> rx21_page_reuse: 0
>> rx21_cache_reuse: 2295681
>> rx21_cache_full: 2273986474
>> rx21_cache_empty: 6656
>> rx21_cache_busy: 2273986727
>> rx21_cache_waive: 0
>> rx21_congst_umr: 0
>> rx21_arfs_err: 0
>> rx21_xdp_tx_xmit: 0
>> rx21_xdp_tx_full: 0
>> rx21_xdp_tx_err: 0
>> rx21_xdp_tx_cqes: 0
>> rx22_packets: 4629499740
>> rx22_bytes: 5924206566499
>> rx22_csum_complete: 4629499740
>> rx22_csum_unnecessary: 0
>> rx22_csum_unnecessary_inner: 0
>> rx22_csum_none: 0
>> rx22_xdp_drop: 0
>> rx22_xdp_redirect: 0
>> rx22_lro_packets: 0
>> rx22_lro_bytes: 0
>> rx22_ecn_mark: 0
>> rx22_removed_vlan_packets: 4629499740
>> rx22_wqe_err: 0
>> rx22_mpwqe_filler_cqes: 0
>> rx22_mpwqe_filler_strides: 0
>> rx22_buff_alloc_err: 0
>> rx22_cqe_compress_blks: 0
>> rx22_cqe_compress_pkts: 0
>> rx22_page_reuse: 0
>> rx22_cache_reuse: 1407527
>> rx22_cache_full: 2313342088
>> rx22_cache_empty: 6656
>> rx22_cache_busy: 2313342341
>> rx22_cache_waive: 0
>> rx22_congst_umr: 0
>> rx22_arfs_err: 0
>> rx22_xdp_tx_xmit: 0
>> rx22_xdp_tx_full: 0
>> rx22_xdp_tx_err: 0
>> rx22_xdp_tx_cqes: 0
>> rx23_packets: 4387124505
>> rx23_bytes: 5718118678470
>> rx23_csum_complete: 4387124505
>> rx23_csum_unnecessary: 0
>> rx23_csum_unnecessary_inner: 0
>> rx23_csum_none: 0
>> rx23_xdp_drop: 0
>> rx23_xdp_redirect: 0
>> rx23_lro_packets: 0
>> rx23_lro_bytes: 0
>> rx23_ecn_mark: 0
>> rx23_removed_vlan_packets: 4387124505
>> rx23_wqe_err: 0
>> rx23_mpwqe_filler_cqes: 0
>> rx23_mpwqe_filler_strides: 0
>> rx23_buff_alloc_err: 0
>> rx23_cqe_compress_blks: 0
>> rx23_cqe_compress_pkts: 0
>> rx23_page_reuse: 0
>> rx23_cache_reuse: 2013280
>> rx23_cache_full: 2191548717
>> rx23_cache_empty: 6656
>> rx23_cache_busy: 2191548972
>> rx23_cache_waive: 0
>> rx23_congst_umr: 0
>> rx23_arfs_err: 0
>> rx23_xdp_tx_xmit: 0
>> rx23_xdp_tx_full: 0
>> rx23_xdp_tx_err: 0
>> rx23_xdp_tx_cqes: 0
>> rx24_packets: 4398791634
>> rx24_bytes: 5744875564632
>> rx24_csum_complete: 4398791634
>> rx24_csum_unnecessary: 0
>> rx24_csum_unnecessary_inner: 0
>> rx24_csum_none: 0
>> rx24_xdp_drop: 0
>> rx24_xdp_redirect: 0
>> rx24_lro_packets: 0
>> rx24_lro_bytes: 0
>> rx24_ecn_mark: 0
>> rx24_removed_vlan_packets: 4398791634
>> rx24_wqe_err: 0
>> rx24_mpwqe_filler_cqes: 0
>> rx24_mpwqe_filler_strides: 0
>> rx24_buff_alloc_err: 0
>> rx24_cqe_compress_blks: 0
>> rx24_cqe_compress_pkts: 0
>> rx24_page_reuse: 0
>> rx24_cache_reuse: 2143926100
>> rx24_cache_full: 55469496
>> rx24_cache_empty: 6656
>> rx24_cache_busy: 55469716
>> rx24_cache_waive: 0
>> rx24_congst_umr: 0
>> rx24_arfs_err: 0
>> rx24_xdp_tx_xmit: 0
>> rx24_xdp_tx_full: 0
>> rx24_xdp_tx_err: 0
>> rx24_xdp_tx_cqes: 0
>> rx25_packets: 4377204935
>> rx25_bytes: 5710369124105
>> rx25_csum_complete: 4377204935
>> rx25_csum_unnecessary: 0
>> rx25_csum_unnecessary_inner: 0
>> rx25_csum_none: 0
>> rx25_xdp_drop: 0
>> rx25_xdp_redirect: 0
>> rx25_lro_packets: 0
>> rx25_lro_bytes: 0
>> rx25_ecn_mark: 0
>> rx25_removed_vlan_packets: 4377204935
>> rx25_wqe_err: 0
>> rx25_mpwqe_filler_cqes: 0
>> rx25_mpwqe_filler_strides: 0
>> rx25_buff_alloc_err: 0
>> rx25_cqe_compress_blks: 0
>> rx25_cqe_compress_pkts: 0
>> rx25_page_reuse: 0
>> rx25_cache_reuse: 2132658660
>> rx25_cache_full: 55943584
>> rx25_cache_empty: 6656
>> rx25_cache_busy: 55943804
>> rx25_cache_waive: 0
>> rx25_congst_umr: 0
>> rx25_arfs_err: 0
>> rx25_xdp_tx_xmit: 0
>> rx25_xdp_tx_full: 0
>> rx25_xdp_tx_err: 0
>> rx25_xdp_tx_cqes: 0
>> rx26_packets: 4496003688
>> rx26_bytes: 5862180715503
>> rx26_csum_complete: 4496003688
>> rx26_csum_unnecessary: 0
>> rx26_csum_unnecessary_inner: 0
>> rx26_csum_none: 0
>> rx26_xdp_drop: 0
>> rx26_xdp_redirect: 0
>> rx26_lro_packets: 0
>> rx26_lro_bytes: 0
>> rx26_ecn_mark: 0
>> rx26_removed_vlan_packets: 4496003688
>> rx26_wqe_err: 0
>> rx26_mpwqe_filler_cqes: 0
>> rx26_mpwqe_filler_strides: 0
>> rx26_buff_alloc_err: 0
>> rx26_cqe_compress_blks: 0
>> rx26_cqe_compress_pkts: 0
>> rx26_page_reuse: 0
>> rx26_cache_reuse: 8
>> rx26_cache_full: 2248001581
>> rx26_cache_empty: 6656
>> rx26_cache_busy: 2248001836
>> rx26_cache_waive: 0
>> rx26_congst_umr: 0
>> rx26_arfs_err: 0
>> rx26_xdp_tx_xmit: 0
>> rx26_xdp_tx_full: 0
>> rx26_xdp_tx_err: 0
>> rx26_xdp_tx_cqes: 0
>> rx27_packets: 4341849333
>> rx27_bytes: 5678653545018
>> rx27_csum_complete: 4341849333
>> rx27_csum_unnecessary: 0
>> rx27_csum_unnecessary_inner: 0
>> rx27_csum_none: 0
>> rx27_xdp_drop: 0
>> rx27_xdp_redirect: 0
>> rx27_lro_packets: 0
>> rx27_lro_bytes: 0
>> rx27_ecn_mark: 0
>> rx27_removed_vlan_packets: 4341849333
>> rx27_wqe_err: 0
>> rx27_mpwqe_filler_cqes: 0
>> rx27_mpwqe_filler_strides: 0
>> rx27_buff_alloc_err: 0
>> rx27_cqe_compress_blks: 0
>> rx27_cqe_compress_pkts: 0
>> rx27_page_reuse: 0
>> rx27_cache_reuse: 1748188
>> rx27_cache_full: 2169176223
>> rx27_cache_empty: 6656
>> rx27_cache_busy: 2169176476
>> rx27_cache_waive: 0
>> rx27_congst_umr: 0
>> rx27_arfs_err: 0
>> rx27_xdp_tx_xmit: 0
>> rx27_xdp_tx_full: 0
>> rx27_xdp_tx_err: 0
>> rx27_xdp_tx_cqes: 0
>> rx28_packets: 0
>> rx28_bytes: 0
>> rx28_csum_complete: 0
>> rx28_csum_unnecessary: 0
>> rx28_csum_unnecessary_inner: 0
>> rx28_csum_none: 0
>> rx28_xdp_drop: 0
>> rx28_xdp_redirect: 0
>> rx28_lro_packets: 0
>> rx28_lro_bytes: 0
>> rx28_ecn_mark: 0
>> rx28_removed_vlan_packets: 0
>> rx28_wqe_err: 0
>> rx28_mpwqe_filler_cqes: 0
>> rx28_mpwqe_filler_strides: 0
>> rx28_buff_alloc_err: 0
>> rx28_cqe_compress_blks: 0
>> rx28_cqe_compress_pkts: 0
>> rx28_page_reuse: 0
>> rx28_cache_reuse: 0
>> rx28_cache_full: 0
>> rx28_cache_empty: 2560
>> rx28_cache_busy: 0
>> rx28_cache_waive: 0
>> rx28_congst_umr: 0
>> rx28_arfs_err: 0
>> rx28_xdp_tx_xmit: 0
>> rx28_xdp_tx_full: 0
>> rx28_xdp_tx_err: 0
>> rx28_xdp_tx_cqes: 0
>> rx29_packets: 0
>> rx29_bytes: 0
>> rx29_csum_complete: 0
>> rx29_csum_unnecessary: 0
>> rx29_csum_unnecessary_inner: 0
>> rx29_csum_none: 0
>> rx29_xdp_drop: 0
>> rx29_xdp_redirect: 0
>> rx29_lro_packets: 0
>> rx29_lro_bytes: 0
>> rx29_ecn_mark: 0
>> rx29_removed_vlan_packets: 0
>> rx29_wqe_err: 0
>> rx29_mpwqe_filler_cqes: 0
>> rx29_mpwqe_filler_strides: 0
>> rx29_buff_alloc_err: 0
>> rx29_cqe_compress_blks: 0
>> rx29_cqe_compress_pkts: 0
>> rx29_page_reuse: 0
>> rx29_cache_reuse: 0
>> rx29_cache_full: 0
>> rx29_cache_empty: 2560
>> rx29_cache_busy: 0
>> rx29_cache_waive: 0
>> rx29_congst_umr: 0
>> rx29_arfs_err: 0
>> rx29_xdp_tx_xmit: 0
>> rx29_xdp_tx_full: 0
>> rx29_xdp_tx_err: 0
>> rx29_xdp_tx_cqes: 0
>> rx30_packets: 0
>> rx30_bytes: 0
>> rx30_csum_complete: 0
>> rx30_csum_unnecessary: 0
>> rx30_csum_unnecessary_inner: 0
>> rx30_csum_none: 0
>> rx30_xdp_drop: 0
>> rx30_xdp_redirect: 0
>> rx30_lro_packets: 0
>> rx30_lro_bytes: 0
>> rx30_ecn_mark: 0
>> rx30_removed_vlan_packets: 0
>> rx30_wqe_err: 0
>> rx30_mpwqe_filler_cqes: 0
>> rx30_mpwqe_filler_strides: 0
>> rx30_buff_alloc_err: 0
>> rx30_cqe_compress_blks: 0
>> rx30_cqe_compress_pkts: 0
>> rx30_page_reuse: 0
>> rx30_cache_reuse: 0
>> rx30_cache_full: 0
>> rx30_cache_empty: 2560
>> rx30_cache_busy: 0
>> rx30_cache_waive: 0
>> rx30_congst_umr: 0
>> rx30_arfs_err: 0
>> rx30_xdp_tx_xmit: 0
>> rx30_xdp_tx_full: 0
>> rx30_xdp_tx_err: 0
>> rx30_xdp_tx_cqes: 0
>> rx31_packets: 0
>> rx31_bytes: 0
>> rx31_csum_complete: 0
>> rx31_csum_unnecessary: 0
>> rx31_csum_unnecessary_inner: 0
>> rx31_csum_none: 0
>> rx31_xdp_drop: 0
>> rx31_xdp_redirect: 0
>> rx31_lro_packets: 0
>> rx31_lro_bytes: 0
>> rx31_ecn_mark: 0
>> rx31_removed_vlan_packets: 0
>> rx31_wqe_err: 0
>> rx31_mpwqe_filler_cqes: 0
>> rx31_mpwqe_filler_strides: 0
>> rx31_buff_alloc_err: 0
>> rx31_cqe_compress_blks: 0
>> rx31_cqe_compress_pkts: 0
>> rx31_page_reuse: 0
>> rx31_cache_reuse: 0
>> rx31_cache_full: 0
>> rx31_cache_empty: 2560
>> rx31_cache_busy: 0
>> rx31_cache_waive: 0
>> rx31_congst_umr: 0
>> rx31_arfs_err: 0
>> rx31_xdp_tx_xmit: 0
>> rx31_xdp_tx_full: 0
>> rx31_xdp_tx_err: 0
>> rx31_xdp_tx_cqes: 0
>> rx32_packets: 0
>> rx32_bytes: 0
>> rx32_csum_complete: 0
>> rx32_csum_unnecessary: 0
>> rx32_csum_unnecessary_inner: 0
>> rx32_csum_none: 0
>> rx32_xdp_drop: 0
>> rx32_xdp_redirect: 0
>> rx32_lro_packets: 0
>> rx32_lro_bytes: 0
>> rx32_ecn_mark: 0
>> rx32_removed_vlan_packets: 0
>> rx32_wqe_err: 0
>> rx32_mpwqe_filler_cqes: 0
>> rx32_mpwqe_filler_strides: 0
>> rx32_buff_alloc_err: 0
>> rx32_cqe_compress_blks: 0
>> rx32_cqe_compress_pkts: 0
>> rx32_page_reuse: 0
>> rx32_cache_reuse: 0
>> rx32_cache_full: 0
>> rx32_cache_empty: 2560
>> rx32_cache_busy: 0
>> rx32_cache_waive: 0
>> rx32_congst_umr: 0
>> rx32_arfs_err: 0
>> rx32_xdp_tx_xmit: 0
>> rx32_xdp_tx_full: 0
>> rx32_xdp_tx_err: 0
>> rx32_xdp_tx_cqes: 0
>> rx33_packets: 0
>> rx33_bytes: 0
>> rx33_csum_complete: 0
>> rx33_csum_unnecessary: 0
>> rx33_csum_unnecessary_inner: 0
>> rx33_csum_none: 0
>> rx33_xdp_drop: 0
>> rx33_xdp_redirect: 0
>> rx33_lro_packets: 0
>> rx33_lro_bytes: 0
>> rx33_ecn_mark: 0
>> rx33_removed_vlan_packets: 0
>> rx33_wqe_err: 0
>> rx33_mpwqe_filler_cqes: 0
>> rx33_mpwqe_filler_strides: 0
>> rx33_buff_alloc_err: 0
>> rx33_cqe_compress_blks: 0
>> rx33_cqe_compress_pkts: 0
>> rx33_page_reuse: 0
>> rx33_cache_reuse: 0
>> rx33_cache_full: 0
>> rx33_cache_empty: 2560
>> rx33_cache_busy: 0
>> rx33_cache_waive: 0
>> rx33_congst_umr: 0
>> rx33_arfs_err: 0
>> rx33_xdp_tx_xmit: 0
>> rx33_xdp_tx_full: 0
>> rx33_xdp_tx_err: 0
>> rx33_xdp_tx_cqes: 0
>> rx34_packets: 0
>> rx34_bytes: 0
>> rx34_csum_complete: 0
>> rx34_csum_unnecessary: 0
>> rx34_csum_unnecessary_inner: 0
>> rx34_csum_none: 0
>> rx34_xdp_drop: 0
>> rx34_xdp_redirect: 0
>> rx34_lro_packets: 0
>> rx34_lro_bytes: 0
>> rx34_ecn_mark: 0
>> rx34_removed_vlan_packets: 0
>> rx34_wqe_err: 0
>> rx34_mpwqe_filler_cqes: 0
>> rx34_mpwqe_filler_strides: 0
>> rx34_buff_alloc_err: 0
>> rx34_cqe_compress_blks: 0
>> rx34_cqe_compress_pkts: 0
>> rx34_page_reuse: 0
>> rx34_cache_reuse: 0
>> rx34_cache_full: 0
>> rx34_cache_empty: 2560
>> rx34_cache_busy: 0
>> rx34_cache_waive: 0
>> rx34_congst_umr: 0
>> rx34_arfs_err: 0
>> rx34_xdp_tx_xmit: 0
>> rx34_xdp_tx_full: 0
>> rx34_xdp_tx_err: 0
>> rx34_xdp_tx_cqes: 0
>> rx35_packets: 0
>> rx35_bytes: 0
>> rx35_csum_complete: 0
>> rx35_csum_unnecessary: 0
>> rx35_csum_unnecessary_inner: 0
>> rx35_csum_none: 0
>> rx35_xdp_drop: 0
>> rx35_xdp_redirect: 0
>> rx35_lro_packets: 0
>> rx35_lro_bytes: 0
>> rx35_ecn_mark: 0
>> rx35_removed_vlan_packets: 0
>> rx35_wqe_err: 0
>> rx35_mpwqe_filler_cqes: 0
>> rx35_mpwqe_filler_strides: 0
>> rx35_buff_alloc_err: 0
>> rx35_cqe_compress_blks: 0
>> rx35_cqe_compress_pkts: 0
>> rx35_page_reuse: 0
>> rx35_cache_reuse: 0
>> rx35_cache_full: 0
>> rx35_cache_empty: 2560
>> rx35_cache_busy: 0
>> rx35_cache_waive: 0
>> rx35_congst_umr: 0
>> rx35_arfs_err: 0
>> rx35_xdp_tx_xmit: 0
>> rx35_xdp_tx_full: 0
>> rx35_xdp_tx_err: 0
>> rx35_xdp_tx_cqes: 0
>> rx36_packets: 0
>> rx36_bytes: 0
>> rx36_csum_complete: 0
>> rx36_csum_unnecessary: 0
>> rx36_csum_unnecessary_inner: 0
>> rx36_csum_none: 0
>> rx36_xdp_drop: 0
>> rx36_xdp_redirect: 0
>> rx36_lro_packets: 0
>> rx36_lro_bytes: 0
>> rx36_ecn_mark: 0
>> rx36_removed_vlan_packets: 0
>> rx36_wqe_err: 0
>> rx36_mpwqe_filler_cqes: 0
>> rx36_mpwqe_filler_strides: 0
>> rx36_buff_alloc_err: 0
>> rx36_cqe_compress_blks: 0
>> rx36_cqe_compress_pkts: 0
>> rx36_page_reuse: 0
>> rx36_cache_reuse: 0
>> rx36_cache_full: 0
>> rx36_cache_empty: 2560
>> rx36_cache_busy: 0
>> rx36_cache_waive: 0
>> rx36_congst_umr: 0
>> rx36_arfs_err: 0
>> rx36_xdp_tx_xmit: 0
>> rx36_xdp_tx_full: 0
>> rx36_xdp_tx_err: 0
>> rx36_xdp_tx_cqes: 0
>> rx37_packets: 0
>> rx37_bytes: 0
>> rx37_csum_complete: 0
>> rx37_csum_unnecessary: 0
>> rx37_csum_unnecessary_inner: 0
>> rx37_csum_none: 0
>> rx37_xdp_drop: 0
>> rx37_xdp_redirect: 0
>> rx37_lro_packets: 0
>> rx37_lro_bytes: 0
>> rx37_ecn_mark: 0
>> rx37_removed_vlan_packets: 0
>> rx37_wqe_err: 0
>> rx37_mpwqe_filler_cqes: 0
>> rx37_mpwqe_filler_strides: 0
>> rx37_buff_alloc_err: 0
>> rx37_cqe_compress_blks: 0
>> rx37_cqe_compress_pkts: 0
>> rx37_page_reuse: 0
>> rx37_cache_reuse: 0
>> rx37_cache_full: 0
>> rx37_cache_empty: 2560
>> rx37_cache_busy: 0
>> rx37_cache_waive: 0
>> rx37_congst_umr: 0
>> rx37_arfs_err: 0
>> rx37_xdp_tx_xmit: 0
>> rx37_xdp_tx_full: 0
>> rx37_xdp_tx_err: 0
>> rx37_xdp_tx_cqes: 0
>> rx38_packets: 0
>> rx38_bytes: 0
>> rx38_csum_complete: 0
>> rx38_csum_unnecessary: 0
>> rx38_csum_unnecessary_inner: 0
>> rx38_csum_none: 0
>> rx38_xdp_drop: 0
>> rx38_xdp_redirect: 0
>> rx38_lro_packets: 0
>> rx38_lro_bytes: 0
>> rx38_ecn_mark: 0
>> rx38_removed_vlan_packets: 0
>> rx38_wqe_err: 0
>> rx38_mpwqe_filler_cqes: 0
>> rx38_mpwqe_filler_strides: 0
>> rx38_buff_alloc_err: 0
>> rx38_cqe_compress_blks: 0
>> rx38_cqe_compress_pkts: 0
>> rx38_page_reuse: 0
>> rx38_cache_reuse: 0
>> rx38_cache_full: 0
>> rx38_cache_empty: 2560
>> rx38_cache_busy: 0
>> rx38_cache_waive: 0
>> rx38_congst_umr: 0
>> rx38_arfs_err: 0
>> rx38_xdp_tx_xmit: 0
>> rx38_xdp_tx_full: 0
>> rx38_xdp_tx_err: 0
>> rx38_xdp_tx_cqes: 0
>> rx39_packets: 0
>> rx39_bytes: 0
>> rx39_csum_complete: 0
>> rx39_csum_unnecessary: 0
>> rx39_csum_unnecessary_inner: 0
>> rx39_csum_none: 0
>> rx39_xdp_drop: 0
>> rx39_xdp_redirect: 0
>> rx39_lro_packets: 0
>> rx39_lro_bytes: 0
>> rx39_ecn_mark: 0
>> rx39_removed_vlan_packets: 0
>> rx39_wqe_err: 0
>> rx39_mpwqe_filler_cqes: 0
>> rx39_mpwqe_filler_strides: 0
>> rx39_buff_alloc_err: 0
>> rx39_cqe_compress_blks: 0
>> rx39_cqe_compress_pkts: 0
>> rx39_page_reuse: 0
>> rx39_cache_reuse: 0
>> rx39_cache_full: 0
>> rx39_cache_empty: 2560
>> rx39_cache_busy: 0
>> rx39_cache_waive: 0
>> rx39_congst_umr: 0
>> rx39_arfs_err: 0
>> rx39_xdp_tx_xmit: 0
>> rx39_xdp_tx_full: 0
>> rx39_xdp_tx_err: 0
>> rx39_xdp_tx_cqes: 0
>> rx40_packets: 0
>> rx40_bytes: 0
>> rx40_csum_complete: 0
>> rx40_csum_unnecessary: 0
>> rx40_csum_unnecessary_inner: 0
>> rx40_csum_none: 0
>> rx40_xdp_drop: 0
>> rx40_xdp_redirect: 0
>> rx40_lro_packets: 0
>> rx40_lro_bytes: 0
>> rx40_ecn_mark: 0
>> rx40_removed_vlan_packets: 0
>> rx40_wqe_err: 0
>> rx40_mpwqe_filler_cqes: 0
>> rx40_mpwqe_filler_strides: 0
>> rx40_buff_alloc_err: 0
>> rx40_cqe_compress_blks: 0
>> rx40_cqe_compress_pkts: 0
>> rx40_page_reuse: 0
>> rx40_cache_reuse: 0
>> rx40_cache_full: 0
>> rx40_cache_empty: 2560
>> rx40_cache_busy: 0
>> rx40_cache_waive: 0
>> rx40_congst_umr: 0
>> rx40_arfs_err: 0
>> rx40_xdp_tx_xmit: 0
>> rx40_xdp_tx_full: 0
>> rx40_xdp_tx_err: 0
>> rx40_xdp_tx_cqes: 0
>> rx41_packets: 0
>> rx41_bytes: 0
>> rx41_csum_complete: 0
>> rx41_csum_unnecessary: 0
>> rx41_csum_unnecessary_inner: 0
>> rx41_csum_none: 0
>> rx41_xdp_drop: 0
>> rx41_xdp_redirect: 0
>> rx41_lro_packets: 0
>> rx41_lro_bytes: 0
>> rx41_ecn_mark: 0
>> rx41_removed_vlan_packets: 0
>> rx41_wqe_err: 0
>> rx41_mpwqe_filler_cqes: 0
>> rx41_mpwqe_filler_strides: 0
>> rx41_buff_alloc_err: 0
>> rx41_cqe_compress_blks: 0
>> rx41_cqe_compress_pkts: 0
>> rx41_page_reuse: 0
>> rx41_cache_reuse: 0
>> rx41_cache_full: 0
>> rx41_cache_empty: 2560
>> rx41_cache_busy: 0
>> rx41_cache_waive: 0
>> rx41_congst_umr: 0
>> rx41_arfs_err: 0
>> rx41_xdp_tx_xmit: 0
>> rx41_xdp_tx_full: 0
>> rx41_xdp_tx_err: 0
>> rx41_xdp_tx_cqes: 0
>> rx42_packets: 0
>> rx42_bytes: 0
>> rx42_csum_complete: 0
>> rx42_csum_unnecessary: 0
>> rx42_csum_unnecessary_inner: 0
>> rx42_csum_none: 0
>> rx42_xdp_drop: 0
>> rx42_xdp_redirect: 0
>> rx42_lro_packets: 0
>> rx42_lro_bytes: 0
>> rx42_ecn_mark: 0
>> rx42_removed_vlan_packets: 0
>> rx42_wqe_err: 0
>> rx42_mpwqe_filler_cqes: 0
>> rx42_mpwqe_filler_strides: 0
>> rx42_buff_alloc_err: 0
>> rx42_cqe_compress_blks: 0
>> rx42_cqe_compress_pkts: 0
>> rx42_page_reuse: 0
>> rx42_cache_reuse: 0
>> rx42_cache_full: 0
>> rx42_cache_empty: 2560
>> rx42_cache_busy: 0
>> rx42_cache_waive: 0
>> rx42_congst_umr: 0
>> rx42_arfs_err: 0
>> rx42_xdp_tx_xmit: 0
>> rx42_xdp_tx_full: 0
>> rx42_xdp_tx_err: 0
>> rx42_xdp_tx_cqes: 0
>> rx43_packets: 0
>> rx43_bytes: 0
>> rx43_csum_complete: 0
>> rx43_csum_unnecessary: 0
>> rx43_csum_unnecessary_inner: 0
>> rx43_csum_none: 0
>> rx43_xdp_drop: 0
>> rx43_xdp_redirect: 0
>> rx43_lro_packets: 0
>> rx43_lro_bytes: 0
>> rx43_ecn_mark: 0
>> rx43_removed_vlan_packets: 0
>> rx43_wqe_err: 0
>> rx43_mpwqe_filler_cqes: 0
>> rx43_mpwqe_filler_strides: 0
>> rx43_buff_alloc_err: 0
>> rx43_cqe_compress_blks: 0
>> rx43_cqe_compress_pkts: 0
>> rx43_page_reuse: 0
>> rx43_cache_reuse: 0
>> rx43_cache_full: 0
>> rx43_cache_empty: 2560
>> rx43_cache_busy: 0
>> rx43_cache_waive: 0
>> rx43_congst_umr: 0
>> rx43_arfs_err: 0
>> rx43_xdp_tx_xmit: 0
>> rx43_xdp_tx_full: 0
>> rx43_xdp_tx_err: 0
>> rx43_xdp_tx_cqes: 0
>> rx44_packets: 0
>> rx44_bytes: 0
>> rx44_csum_complete: 0
>> rx44_csum_unnecessary: 0
>> rx44_csum_unnecessary_inner: 0
>> rx44_csum_none: 0
>> rx44_xdp_drop: 0
>> rx44_xdp_redirect: 0
>> rx44_lro_packets: 0
>> rx44_lro_bytes: 0
>> rx44_ecn_mark: 0
>> rx44_removed_vlan_packets: 0
>> rx44_wqe_err: 0
>> rx44_mpwqe_filler_cqes: 0
>> rx44_mpwqe_filler_strides: 0
>> rx44_buff_alloc_err: 0
>> rx44_cqe_compress_blks: 0
>> rx44_cqe_compress_pkts: 0
>> rx44_page_reuse: 0
>> rx44_cache_reuse: 0
>> rx44_cache_full: 0
>> rx44_cache_empty: 2560
>> rx44_cache_busy: 0
>> rx44_cache_waive: 0
>> rx44_congst_umr: 0
>> rx44_arfs_err: 0
>> rx44_xdp_tx_xmit: 0
>> rx44_xdp_tx_full: 0
>> rx44_xdp_tx_err: 0
>> rx44_xdp_tx_cqes: 0
>> rx45_packets: 0
>> rx45_bytes: 0
>> rx45_csum_complete: 0
>> rx45_csum_unnecessary: 0
>> rx45_csum_unnecessary_inner: 0
>> rx45_csum_none: 0
>> rx45_xdp_drop: 0
>> rx45_xdp_redirect: 0
>> rx45_lro_packets: 0
>> rx45_lro_bytes: 0
>> rx45_ecn_mark: 0
>> rx45_removed_vlan_packets: 0
>> rx45_wqe_err: 0
>> rx45_mpwqe_filler_cqes: 0
>> rx45_mpwqe_filler_strides: 0
>> rx45_buff_alloc_err: 0
>> rx45_cqe_compress_blks: 0
>> rx45_cqe_compress_pkts: 0
>> rx45_page_reuse: 0
>> rx45_cache_reuse: 0
>> rx45_cache_full: 0
>> rx45_cache_empty: 2560
>> rx45_cache_busy: 0
>> rx45_cache_waive: 0
>> rx45_congst_umr: 0
>> rx45_arfs_err: 0
>> rx45_xdp_tx_xmit: 0
>> rx45_xdp_tx_full: 0
>> rx45_xdp_tx_err: 0
>> rx45_xdp_tx_cqes: 0
>> rx46_packets: 0
>> rx46_bytes: 0
>> rx46_csum_complete: 0
>> rx46_csum_unnecessary: 0
>> rx46_csum_unnecessary_inner: 0
>> rx46_csum_none: 0
>> rx46_xdp_drop: 0
>> rx46_xdp_redirect: 0
>> rx46_lro_packets: 0
>> rx46_lro_bytes: 0
>> rx46_ecn_mark: 0
>> rx46_removed_vlan_packets: 0
>> rx46_wqe_err: 0
>> rx46_mpwqe_filler_cqes: 0
>> rx46_mpwqe_filler_strides: 0
>> rx46_buff_alloc_err: 0
>> rx46_cqe_compress_blks: 0
>> rx46_cqe_compress_pkts: 0
>> rx46_page_reuse: 0
>> rx46_cache_reuse: 0
>> rx46_cache_full: 0
>> rx46_cache_empty: 2560
>> rx46_cache_busy: 0
>> rx46_cache_waive: 0
>> rx46_congst_umr: 0
>> rx46_arfs_err: 0
>> rx46_xdp_tx_xmit: 0
>> rx46_xdp_tx_full: 0
>> rx46_xdp_tx_err: 0
>> rx46_xdp_tx_cqes: 0
>> rx47_packets: 0
>> rx47_bytes: 0
>> rx47_csum_complete: 0
>> rx47_csum_unnecessary: 0
>> rx47_csum_unnecessary_inner: 0
>> rx47_csum_none: 0
>> rx47_xdp_drop: 0
>> rx47_xdp_redirect: 0
>> rx47_lro_packets: 0
>> rx47_lro_bytes: 0
>> rx47_ecn_mark: 0
>> rx47_removed_vlan_packets: 0
>> rx47_wqe_err: 0
>> rx47_mpwqe_filler_cqes: 0
>> rx47_mpwqe_filler_strides: 0
>> rx47_buff_alloc_err: 0
>> rx47_cqe_compress_blks: 0
>> rx47_cqe_compress_pkts: 0
>> rx47_page_reuse: 0
>> rx47_cache_reuse: 0
>> rx47_cache_full: 0
>> rx47_cache_empty: 2560
>> rx47_cache_busy: 0
>> rx47_cache_waive: 0
>> rx47_congst_umr: 0
>> rx47_arfs_err: 0
>> rx47_xdp_tx_xmit: 0
>> rx47_xdp_tx_full: 0
>> rx47_xdp_tx_err: 0
>> rx47_xdp_tx_cqes: 0
>> rx48_packets: 0
>> rx48_bytes: 0
>> rx48_csum_complete: 0
>> rx48_csum_unnecessary: 0
>> rx48_csum_unnecessary_inner: 0
>> rx48_csum_none: 0
>> rx48_xdp_drop: 0
>> rx48_xdp_redirect: 0
>> rx48_lro_packets: 0
>> rx48_lro_bytes: 0
>> rx48_ecn_mark: 0
>> rx48_removed_vlan_packets: 0
>> rx48_wqe_err: 0
>> rx48_mpwqe_filler_cqes: 0
>> rx48_mpwqe_filler_strides: 0
>> rx48_buff_alloc_err: 0
>> rx48_cqe_compress_blks: 0
>> rx48_cqe_compress_pkts: 0
>> rx48_page_reuse: 0
>> rx48_cache_reuse: 0
>> rx48_cache_full: 0
>> rx48_cache_empty: 2560
>> rx48_cache_busy: 0
>> rx48_cache_waive: 0
>> rx48_congst_umr: 0
>> rx48_arfs_err: 0
>> rx48_xdp_tx_xmit: 0
>> rx48_xdp_tx_full: 0
>> rx48_xdp_tx_err: 0
>> rx48_xdp_tx_cqes: 0
>> rx49_packets: 0
>> rx49_bytes: 0
>> rx49_csum_complete: 0
>> rx49_csum_unnecessary: 0
>> rx49_csum_unnecessary_inner: 0
>> rx49_csum_none: 0
>> rx49_xdp_drop: 0
>> rx49_xdp_redirect: 0
>> rx49_lro_packets: 0
>> rx49_lro_bytes: 0
>> rx49_ecn_mark: 0
>> rx49_removed_vlan_packets: 0
>> rx49_wqe_err: 0
>> rx49_mpwqe_filler_cqes: 0
>> rx49_mpwqe_filler_strides: 0
>> rx49_buff_alloc_err: 0
>> rx49_cqe_compress_blks: 0
>> rx49_cqe_compress_pkts: 0
>> rx49_page_reuse: 0
>> rx49_cache_reuse: 0
>> rx49_cache_full: 0
>> rx49_cache_empty: 2560
>> rx49_cache_busy: 0
>> rx49_cache_waive: 0
>> rx49_congst_umr: 0
>> rx49_arfs_err: 0
>> rx49_xdp_tx_xmit: 0
>> rx49_xdp_tx_full: 0
>> rx49_xdp_tx_err: 0
>> rx49_xdp_tx_cqes: 0
>> rx50_packets: 0
>> rx50_bytes: 0
>> rx50_csum_complete: 0
>> rx50_csum_unnecessary: 0
>> rx50_csum_unnecessary_inner: 0
>> rx50_csum_none: 0
>> rx50_xdp_drop: 0
>> rx50_xdp_redirect: 0
>> rx50_lro_packets: 0
>> rx50_lro_bytes: 0
>> rx50_ecn_mark: 0
>> rx50_removed_vlan_packets: 0
>> rx50_wqe_err: 0
>> rx50_mpwqe_filler_cqes: 0
>> rx50_mpwqe_filler_strides: 0
>> rx50_buff_alloc_err: 0
>> rx50_cqe_compress_blks: 0
>> rx50_cqe_compress_pkts: 0
>> rx50_page_reuse: 0
>> rx50_cache_reuse: 0
>> rx50_cache_full: 0
>> rx50_cache_empty: 2560
>> rx50_cache_busy: 0
>> rx50_cache_waive: 0
>> rx50_congst_umr: 0
>> rx50_arfs_err: 0
>> rx50_xdp_tx_xmit: 0
>> rx50_xdp_tx_full: 0
>> rx50_xdp_tx_err: 0
>> rx50_xdp_tx_cqes: 0
>> rx51_packets: 0
>> rx51_bytes: 0
>> rx51_csum_complete: 0
>> rx51_csum_unnecessary: 0
>> rx51_csum_unnecessary_inner: 0
>> rx51_csum_none: 0
>> rx51_xdp_drop: 0
>> rx51_xdp_redirect: 0
>> rx51_lro_packets: 0
>> rx51_lro_bytes: 0
>> rx51_ecn_mark: 0
>> rx51_removed_vlan_packets: 0
>> rx51_wqe_err: 0
>> rx51_mpwqe_filler_cqes: 0
>> rx51_mpwqe_filler_strides: 0
>> rx51_buff_alloc_err: 0
>> rx51_cqe_compress_blks: 0
>> rx51_cqe_compress_pkts: 0
>> rx51_page_reuse: 0
>> rx51_cache_reuse: 0
>> rx51_cache_full: 0
>> rx51_cache_empty: 2560
>> rx51_cache_busy: 0
>> rx51_cache_waive: 0
>> rx51_congst_umr: 0
>> rx51_arfs_err: 0
>> rx51_xdp_tx_xmit: 0
>> rx51_xdp_tx_full: 0
>> rx51_xdp_tx_err: 0
>> rx51_xdp_tx_cqes: 0
>> rx52_packets: 0
>> rx52_bytes: 0
>> rx52_csum_complete: 0
>> rx52_csum_unnecessary: 0
>> rx52_csum_unnecessary_inner: 0
>> rx52_csum_none: 0
>> rx52_xdp_drop: 0
>> rx52_xdp_redirect: 0
>> rx52_lro_packets: 0
>> rx52_lro_bytes: 0
>> rx52_ecn_mark: 0
>> rx52_removed_vlan_packets: 0
>> rx52_wqe_err: 0
>> rx52_mpwqe_filler_cqes: 0
>> rx52_mpwqe_filler_strides: 0
>> rx52_buff_alloc_err: 0
>> rx52_cqe_compress_blks: 0
>> rx52_cqe_compress_pkts: 0
>> rx52_page_reuse: 0
>> rx52_cache_reuse: 0
>> rx52_cache_full: 0
>> rx52_cache_empty: 2560
>> rx52_cache_busy: 0
>> rx52_cache_waive: 0
>> rx52_congst_umr: 0
>> rx52_arfs_err: 0
>> rx52_xdp_tx_xmit: 0
>> rx52_xdp_tx_full: 0
>> rx52_xdp_tx_err: 0
>> rx52_xdp_tx_cqes: 0
>> rx53_packets: 0
>> rx53_bytes: 0
>> rx53_csum_complete: 0
>> rx53_csum_unnecessary: 0
>> rx53_csum_unnecessary_inner: 0
>> rx53_csum_none: 0
>> rx53_xdp_drop: 0
>> rx53_xdp_redirect: 0
>> rx53_lro_packets: 0
>> rx53_lro_bytes: 0
>> rx53_ecn_mark: 0
>> rx53_removed_vlan_packets: 0
>> rx53_wqe_err: 0
>> rx53_mpwqe_filler_cqes: 0
>> rx53_mpwqe_filler_strides: 0
>> rx53_buff_alloc_err: 0
>> rx53_cqe_compress_blks: 0
>> rx53_cqe_compress_pkts: 0
>> rx53_page_reuse: 0
>> rx53_cache_reuse: 0
>> rx53_cache_full: 0
>> rx53_cache_empty: 2560
>> rx53_cache_busy: 0
>> rx53_cache_waive: 0
>> rx53_congst_umr: 0
>> rx53_arfs_err: 0
>> rx53_xdp_tx_xmit: 0
>> rx53_xdp_tx_full: 0
>> rx53_xdp_tx_err: 0
>> rx53_xdp_tx_cqes: 0
>> rx54_packets: 0
>> rx54_bytes: 0
>> rx54_csum_complete: 0
>> rx54_csum_unnecessary: 0
>> rx54_csum_unnecessary_inner: 0
>> rx54_csum_none: 0
>> rx54_xdp_drop: 0
>> rx54_xdp_redirect: 0
>> rx54_lro_packets: 0
>> rx54_lro_bytes: 0
>> rx54_ecn_mark: 0
>> rx54_removed_vlan_packets: 0
>> rx54_wqe_err: 0
>> rx54_mpwqe_filler_cqes: 0
>> rx54_mpwqe_filler_strides: 0
>> rx54_buff_alloc_err: 0
>> rx54_cqe_compress_blks: 0
>> rx54_cqe_compress_pkts: 0
>> rx54_page_reuse: 0
>> rx54_cache_reuse: 0
>> rx54_cache_full: 0
>> rx54_cache_empty: 2560
>> rx54_cache_busy: 0
>> rx54_cache_waive: 0
>> rx54_congst_umr: 0
>> rx54_arfs_err: 0
>> rx54_xdp_tx_xmit: 0
>> rx54_xdp_tx_full: 0
>> rx54_xdp_tx_err: 0
>> rx54_xdp_tx_cqes: 0
>> rx55_packets: 0
>> rx55_bytes: 0
>> rx55_csum_complete: 0
>> rx55_csum_unnecessary: 0
>> rx55_csum_unnecessary_inner: 0
>> rx55_csum_none: 0
>> rx55_xdp_drop: 0
>> rx55_xdp_redirect: 0
>> rx55_lro_packets: 0
>> rx55_lro_bytes: 0
>> rx55_ecn_mark: 0
>> rx55_removed_vlan_packets: 0
>> rx55_wqe_err: 0
>> rx55_mpwqe_filler_cqes: 0
>> rx55_mpwqe_filler_strides: 0
>> rx55_buff_alloc_err: 0
>> rx55_cqe_compress_blks: 0
>> rx55_cqe_compress_pkts: 0
>> rx55_page_reuse: 0
>> rx55_cache_reuse: 0
>> rx55_cache_full: 0
>> rx55_cache_empty: 2560
>> rx55_cache_busy: 0
>> rx55_cache_waive: 0
>> rx55_congst_umr: 0
>> rx55_arfs_err: 0
>> rx55_xdp_tx_xmit: 0
>> rx55_xdp_tx_full: 0
>> rx55_xdp_tx_err: 0
>> rx55_xdp_tx_cqes: 0
>> tx0_packets: 6019477917
>> tx0_bytes: 3445238940825
>> tx0_tso_packets: 311304622
>> tx0_tso_bytes: 1897094773213
>> tx0_tso_inner_packets: 0
>> tx0_tso_inner_bytes: 0
>> tx0_csum_partial: 457981794
>> tx0_csum_partial_inner: 0
>> tx0_added_vlan_packets: 4965567654
>> tx0_nop: 72290329
>> tx0_csum_none: 4507585860
>> tx0_stopped: 9118
>> tx0_dropped: 0
>> tx0_xmit_more: 51651593
>> tx0_recover: 0
>> tx0_cqes: 4913918402
>> tx0_wake: 9118
>> tx0_cqe_err: 0
>> tx1_packets: 5700413414
>> tx1_bytes: 3340870662350
>> tx1_tso_packets: 318201557
>> tx1_tso_bytes: 1915233462303
>> tx1_tso_inner_packets: 0
>> tx1_tso_inner_bytes: 0
>> tx1_csum_partial: 461736722
>> tx1_csum_partial_inner: 0
>> tx1_added_vlan_packets: 4638708749
>> tx1_nop: 70061796
>> tx1_csum_none: 4176972027
>> tx1_stopped: 9248
>> tx1_dropped: 0
>> tx1_xmit_more: 39531959
>> tx1_recover: 0
>> tx1_cqes: 4599179178
>> tx1_wake: 9248
>> tx1_cqe_err: 0
>> tx2_packets: 5795960848
>> tx2_bytes: 3394876820271
>> tx2_tso_packets: 322935065
>> tx2_tso_bytes: 1910825901109
>> tx2_tso_inner_packets: 0
>> tx2_tso_inner_bytes: 0
>> tx2_csum_partial: 460747092
>> tx2_csum_partial_inner: 0
>> tx2_added_vlan_packets: 4743705654
>> tx2_nop: 72722430
>> tx2_csum_none: 4282958562
>> tx2_stopped: 8938
>> tx2_dropped: 0
>> tx2_xmit_more: 44084718
>> tx2_recover: 0
>> tx2_cqes: 4699623410
>> tx2_wake: 8938
>> tx2_cqe_err: 0
>> tx3_packets: 5580215878
>> tx3_bytes: 3191677257787
>> tx3_tso_packets: 305771141
>> tx3_tso_bytes: 1823265793476
>> tx3_tso_inner_packets: 0
>> tx3_tso_inner_bytes: 0
>> tx3_csum_partial: 434976070
>> tx3_csum_partial_inner: 0
>> tx3_added_vlan_packets: 4569899956
>> tx3_nop: 68184348
>> tx3_csum_none: 4134923886
>> tx3_stopped: 8383
>> tx3_dropped: 0
>> tx3_xmit_more: 41940375
>> tx3_recover: 0
>> tx3_cqes: 4527961924
>> tx3_wake: 8383
>> tx3_cqe_err: 0
>> tx4_packets: 6795007068
>> tx4_bytes: 3963890025270
>> tx4_tso_packets: 358437617
>> tx4_tso_bytes: 2154747995355
>> tx4_tso_inner_packets: 0
>> tx4_tso_inner_bytes: 0
>> tx4_csum_partial: 504764524
>> tx4_csum_partial_inner: 0
>> tx4_added_vlan_packets: 5602510191
>> tx4_nop: 81345604
>> tx4_csum_none: 5097745667
>> tx4_stopped: 10248
>> tx4_dropped: 0
>> tx4_xmit_more: 49068571
>> tx4_recover: 0
>> tx4_cqes: 5553444276
>> tx4_wake: 10248
>> tx4_cqe_err: 0
>> tx5_packets: 6408089261
>> tx5_bytes: 3676275848279
>> tx5_tso_packets: 345129329
>> tx5_tso_bytes: 2108447877473
>> tx5_tso_inner_packets: 0
>> tx5_tso_inner_bytes: 0
>> tx5_csum_partial: 494705894
>> tx5_csum_partial_inner: 0
>> tx5_added_vlan_packets: 5235998343
>> tx5_nop: 77694627
>> tx5_csum_none: 4741292449
>> tx5_stopped: 46
>> tx5_dropped: 0
>> tx5_xmit_more: 46675831
>> tx5_recover: 0
>> tx5_cqes: 5189323550
>> tx5_wake: 46
>> tx5_cqe_err: 0
>> tx6_packets: 6382289663
>> tx6_bytes: 3670991418150
>> tx6_tso_packets: 342927826
>> tx6_tso_bytes: 2075049679904
>> tx6_tso_inner_packets: 0
>> tx6_tso_inner_bytes: 0
>> tx6_csum_partial: 490369221
>> tx6_csum_partial_inner: 0
>> tx6_added_vlan_packets: 5232144528
>> tx6_nop: 77391246
>> tx6_csum_none: 4741775307
>> tx6_stopped: 10823
>> tx6_dropped: 0
>> tx6_xmit_more: 44487607
>> tx6_recover: 0
>> tx6_cqes: 5187659877
>> tx6_wake: 10823
>> tx6_cqe_err: 0
>> tx7_packets: 6456378284
>> tx7_bytes: 3758013320518
>> tx7_tso_packets: 350958294
>> tx7_tso_bytes: 2126833408524
>> tx7_tso_inner_packets: 0
>> tx7_tso_inner_bytes: 0
>> tx7_csum_partial: 501804109
>> tx7_csum_partial_inner: 0
>> tx7_added_vlan_packets: 5275635204
>> tx7_nop: 79010883
>> tx7_csum_none: 4773831096
>> tx7_stopped: 14684
>> tx7_dropped: 0
>> tx7_xmit_more: 44447469
>> tx7_recover: 0
>> tx7_cqes: 5231191770
>> tx7_wake: 14684
>> tx7_cqe_err: 0
>> tx8_packets: 6401799768
>> tx8_bytes: 3681210808766
>> tx8_tso_packets: 342878228
>> tx8_tso_bytes: 2089688012191
>> tx8_tso_inner_packets: 0
>> tx8_tso_inner_bytes: 0
>> tx8_csum_partial: 494865145
>> tx8_csum_partial_inner: 0
>> tx8_added_vlan_packets: 5242288908
>> tx8_nop: 77250910
>> tx8_csum_none: 4747423763
>> tx8_stopped: 2
>> tx8_dropped: 0
>> tx8_xmit_more: 44191737
>> tx8_recover: 0
>> tx8_cqes: 5198098454
>> tx8_wake: 2
>> tx8_cqe_err: 0
>> tx9_packets: 6632882888
>> tx9_bytes: 3820110338309
>> tx9_tso_packets: 354189056
>> tx9_tso_bytes: 2187883597128
>> tx9_tso_inner_packets: 0
>> tx9_tso_inner_bytes: 0
>> tx9_csum_partial: 511108218
>> tx9_csum_partial_inner: 0
>> tx9_added_vlan_packets: 5413836353
>> tx9_nop: 80560668
>> tx9_csum_none: 4902728135
>> tx9_stopped: 9091
>> tx9_dropped: 0
>> tx9_xmit_more: 54501293
>> tx9_recover: 0
>> tx9_cqes: 5359337638
>> tx9_wake: 9091
>> tx9_cqe_err: 0
>> tx10_packets: 6421786406
>> tx10_bytes: 3692798413429
>> tx10_tso_packets: 346878943
>> tx10_tso_bytes: 2111921062110
>> tx10_tso_inner_packets: 0
>> tx10_tso_inner_bytes: 0
>> tx10_csum_partial: 494356645
>> tx10_csum_partial_inner: 0
>> tx10_added_vlan_packets: 5248274374
>> tx10_nop: 77922624
>> tx10_csum_none: 4753917730
>> tx10_stopped: 9617
>> tx10_dropped: 0
>> tx10_xmit_more: 44473939
>> tx10_recover: 0
>> tx10_cqes: 5203802547
>> tx10_wake: 9617
>> tx10_cqe_err: 0
>> tx11_packets: 6406750938
>> tx11_bytes: 3660343565126
>> tx11_tso_packets: 355917271
>> tx11_tso_bytes: 2130812246956
>> tx11_tso_inner_packets: 0
>> tx11_tso_inner_bytes: 0
>> tx11_csum_partial: 500336369
>> tx11_csum_partial_inner: 0
>> tx11_added_vlan_packets: 5228267547
>> tx11_nop: 78906315
>> tx11_csum_none: 4727931178
>> tx11_stopped: 9607
>> tx11_dropped: 0
>> tx11_xmit_more: 40041492
>> tx11_recover: 0
>> tx11_cqes: 5188228290
>> tx11_wake: 9607
>> tx11_cqe_err: 0
>> tx12_packets: 6422347846
>> tx12_bytes: 3718772753227
>> tx12_tso_packets: 355397223
>> tx12_tso_bytes: 2162614059758
>> tx12_tso_inner_packets: 0
>> tx12_tso_inner_bytes: 0
>> tx12_csum_partial: 511437844
>> tx12_csum_partial_inner: 0
>> tx12_added_vlan_packets: 5221373746
>> tx12_nop: 78866779
>> tx12_csum_none: 4709935902
>> tx12_stopped: 10280
>> tx12_dropped: 0
>> tx12_xmit_more: 42189399
>> tx12_recover: 0
>> tx12_cqes: 5179187154
>> tx12_wake: 10280
>> tx12_cqe_err: 0
>> tx13_packets: 6429383816
>> tx13_bytes: 3725679445046
>> tx13_tso_packets: 360934759
>> tx13_tso_bytes: 2148016411436
>> tx13_tso_inner_packets: 0
>> tx13_tso_inner_bytes: 0
>> tx13_csum_partial: 505245849
>> tx13_csum_partial_inner: 0
>> tx13_added_vlan_packets: 5240267441
>> tx13_nop: 80295637
>> tx13_csum_none: 4735021592
>> tx13_stopped: 84
>> tx13_dropped: 0
>> tx13_xmit_more: 43118045
>> tx13_recover: 0
>> tx13_cqes: 5197150348
>> tx13_wake: 84
>> tx13_cqe_err: 0
>> tx14_packets: 6375279148
>> tx14_bytes: 3624267203336
>> tx14_tso_packets: 344388148
>> tx14_tso_bytes: 2094966273548
>> tx14_tso_inner_packets: 0
>> tx14_tso_inner_bytes: 0
>> tx14_csum_partial: 494129407
>> tx14_csum_partial_inner: 0
>> tx14_added_vlan_packets: 5210749337
>> tx14_nop: 77280615
>> tx14_csum_none: 4716619930
>> tx14_stopped: 13057
>> tx14_dropped: 0
>> tx14_xmit_more: 40849682
>> tx14_recover: 0
>> tx14_cqes: 5169902694
>> tx14_wake: 13057
>> tx14_cqe_err: 0
>> tx15_packets: 6489306520
>> tx15_bytes: 3775716194795
>> tx15_tso_packets: 368716406
>> tx15_tso_bytes: 2165876423354
>> tx15_tso_inner_packets: 0
>> tx15_tso_inner_bytes: 0
>> tx15_csum_partial: 509887864
>> tx15_csum_partial_inner: 0
>> tx15_added_vlan_packets: 5296767390
>> tx15_nop: 80803468
>> tx15_csum_none: 4786879529
>> tx15_stopped: 1
>> tx15_dropped: 0
>> tx15_xmit_more: 46979676
>> tx15_recover: 0
>> tx15_cqes: 5249789328
>> tx15_wake: 1
>> tx15_cqe_err: 0
>> tx16_packets: 6559857761
>> tx16_bytes: 3724080573905
>> tx16_tso_packets: 350864176
>> tx16_tso_bytes: 2099634006033
>> tx16_tso_inner_packets: 0
>> tx16_tso_inner_bytes: 0
>> tx16_csum_partial: 489397232
>> tx16_csum_partial_inner: 0
>> tx16_added_vlan_packets: 5398869334
>> tx16_nop: 79046075
>> tx16_csum_none: 4909472106
>> tx16_stopped: 4480
>> tx16_dropped: 0
>> tx16_xmit_more: 47273286
>> tx16_recover: 0
>> tx16_cqes: 5351598315
>> tx16_wake: 4480
>> tx16_cqe_err: 0
>> tx17_packets: 6358711533
>> tx17_bytes: 3650180865573
>> tx17_tso_packets: 350723136
>> tx17_tso_bytes: 2109426587128
>> tx17_tso_inner_packets: 0
>> tx17_tso_inner_bytes: 0
>> tx17_csum_partial: 494719487
>> tx17_csum_partial_inner: 0
>> tx17_added_vlan_packets: 5190068796
>> tx17_nop: 77285612
>> tx17_csum_none: 4695349309
>> tx17_stopped: 10443
>> tx17_dropped: 0
>> tx17_xmit_more: 45582108
>> tx17_recover: 0
>> tx17_cqes: 5144489363
>> tx17_wake: 10443
>> tx17_cqe_err: 0
>> tx18_packets: 6655328437
>> tx18_bytes: 3801768461807
>> tx18_tso_packets: 356516373
>> tx18_tso_bytes: 2164829247550
>> tx18_tso_inner_packets: 0
>> tx18_tso_inner_bytes: 0
>> tx18_csum_partial: 500508446
>> tx18_csum_partial_inner: 0
>> tx18_added_vlan_packets: 5454166840
>> tx18_nop: 80423007
>> tx18_csum_none: 4953658394
>> tx18_stopped: 14760
>> tx18_dropped: 0
>> tx18_xmit_more: 50837465
>> tx18_recover: 0
>> tx18_cqes: 5403332553
>> tx18_wake: 14760
>> tx18_cqe_err: 0
>> tx19_packets: 6408680611
>> tx19_bytes: 3644119934372
>> tx19_tso_packets: 350727530
>> tx19_tso_bytes: 2089896715365
>> tx19_tso_inner_packets: 0
>> tx19_tso_inner_bytes: 0
>> tx19_csum_partial: 486536490
>> tx19_csum_partial_inner: 0
>> tx19_added_vlan_packets: 5255839020
>> tx19_nop: 78525198
>> tx19_csum_none: 4769302530
>> tx19_stopped: 8614
>> tx19_dropped: 0
>> tx19_xmit_more: 43605232
>> tx19_recover: 0
>> tx19_cqes: 5212236833
>> tx19_wake: 8614
>> tx19_cqe_err: 0
>> tx20_packets: 5609275141
>> tx20_bytes: 3187279031581
>> tx20_tso_packets: 298609303
>> tx20_tso_bytes: 1794382229379
>> tx20_tso_inner_packets: 0
>> tx20_tso_inner_bytes: 0
>> tx20_csum_partial: 430691178
>> tx20_csum_partial_inner: 0
>> tx20_added_vlan_packets: 4616844286
>> tx20_nop: 67450040
>> tx20_csum_none: 4186153108
>> tx20_stopped: 9099
>> tx20_dropped: 0
>> tx20_xmit_more: 42040991
>> tx20_recover: 0
>> tx20_cqes: 4574805846
>> tx20_wake: 9099
>> tx20_cqe_err: 0
>> tx21_packets: 5641621183
>> tx21_bytes: 3279282331124
>> tx21_tso_packets: 311297057
>> tx21_tso_bytes: 1875735401012
>> tx21_tso_inner_packets: 0
>> tx21_tso_inner_bytes: 0
>> tx21_csum_partial: 444333894
>> tx21_csum_partial_inner: 0
>> tx21_added_vlan_packets: 4603527701
>> tx21_nop: 68857983
>> tx21_csum_none: 4159193807
>> tx21_stopped: 10082
>> tx21_dropped: 0
>> tx21_xmit_more: 43988081
>> tx21_recover: 0
>> tx21_cqes: 4559542410
>> tx21_wake: 10082
>> tx21_cqe_err: 0
>> tx22_packets: 5822168288
>> tx22_bytes: 3452026726862
>> tx22_tso_packets: 308230791
>> tx22_tso_bytes: 1859686450671
>> tx22_tso_inner_packets: 0
>> tx22_tso_inner_bytes: 0
>> tx22_csum_partial: 442751518
>> tx22_csum_partial_inner: 0
>> tx22_added_vlan_packets: 4792100335
>> tx22_nop: 70631706
>> tx22_csum_none: 4349348817
>> tx22_stopped: 9355
>> tx22_dropped: 0
>> tx22_xmit_more: 45165994
>> tx22_recover: 0
>> tx22_cqes: 4746936601
>> tx22_wake: 9355
>> tx22_cqe_err: 0
>> tx23_packets: 5664896066
>> tx23_bytes: 3207724186946
>> tx23_tso_packets: 300418757
>> tx23_tso_bytes: 1794180478679
>> tx23_tso_inner_packets: 0
>> tx23_tso_inner_bytes: 0
>> tx23_csum_partial: 429898848
>> tx23_csum_partial_inner: 0
>> tx23_added_vlan_packets: 4674317320
>> tx23_nop: 67899896
>> tx23_csum_none: 4244418472
>> tx23_stopped: 11684
>> tx23_dropped: 0
>> tx23_xmit_more: 43351132
>> tx23_recover: 0
>> tx23_cqes: 4630969028
>> tx23_wake: 11684
>> tx23_cqe_err: 0
>> tx24_packets: 5663326601
>> tx24_bytes: 3250127095110
>> tx24_tso_packets: 301327422
>> tx24_tso_bytes: 1831260534157
>> tx24_tso_inner_packets: 0
>> tx24_tso_inner_bytes: 0
>> tx24_csum_partial: 438757312
>> tx24_csum_partial_inner: 0
>> tx24_added_vlan_packets: 4646014986
>> tx24_nop: 68431153
>> tx24_csum_none: 4207257674
>> tx24_stopped: 9240
>> tx24_dropped: 0
>> tx24_xmit_more: 47699542
>> tx24_recover: 0
>> tx24_cqes: 4598317913
>> tx24_wake: 9240
>> tx24_cqe_err: 0
>> tx25_packets: 5703883962
>> tx25_bytes: 3291856915695
>> tx25_tso_packets: 308900318
>> tx25_tso_bytes: 1855516128386
>> tx25_tso_inner_packets: 0
>> tx25_tso_inner_bytes: 0
>> tx25_csum_partial: 444753744
>> tx25_csum_partial_inner: 0
>> tx25_added_vlan_packets: 4676528924
>> tx25_nop: 69230967
>> tx25_csum_none: 4231775180
>> tx25_stopped: 1140
>> tx25_dropped: 0
>> tx25_xmit_more: 40819195
>> tx25_recover: 0
>> tx25_cqes: 4635710966
>> tx25_wake: 1140
>> tx25_cqe_err: 0
>> tx26_packets: 5803495984
>> tx26_bytes: 3413564272139
>> tx26_tso_packets: 319986230
>> tx26_tso_bytes: 1929042839677
>> tx26_tso_inner_packets: 0
>> tx26_tso_inner_bytes: 0
>> tx26_csum_partial: 464771163
>> tx26_csum_partial_inner: 0
>> tx26_added_vlan_packets: 4734767280
>> tx26_nop: 71345080
>> tx26_csum_none: 4269996117
>> tx26_stopped: 10972
>> tx26_dropped: 0
>> tx26_xmit_more: 43793424
>> tx26_recover: 0
>> tx26_cqes: 4690976400
>> tx26_wake: 10972
>> tx26_cqe_err: 0
>> tx27_packets: 5960955343
>> tx27_bytes: 3444156164526
>> tx27_tso_packets: 325099639
>> tx27_tso_bytes: 1928378678784
>> tx27_tso_inner_packets: 0
>> tx27_tso_inner_bytes: 0
>> tx27_csum_partial: 467310289
>> tx27_csum_partial_inner: 0
>> tx27_added_vlan_packets: 4888651368
>> tx27_nop: 73201664
>> tx27_csum_none: 4421341079
>> tx27_stopped: 9465
>> tx27_dropped: 0
>> tx27_xmit_more: 53632121
>> tx27_recover: 0
>> tx27_cqes: 4835021398
>> tx27_wake: 9465
>> tx27_cqe_err: 0
>> tx28_packets: 0
>> tx28_bytes: 0
>> tx28_tso_packets: 0
>> tx28_tso_bytes: 0
>> tx28_tso_inner_packets: 0
>> tx28_tso_inner_bytes: 0
>> tx28_csum_partial: 0
>> tx28_csum_partial_inner: 0
>> tx28_added_vlan_packets: 0
>> tx28_nop: 0
>> tx28_csum_none: 0
>> tx28_stopped: 0
>> tx28_dropped: 0
>> tx28_xmit_more: 0
>> tx28_recover: 0
>> tx28_cqes: 0
>> tx28_wake: 0
>> tx28_cqe_err: 0
>> tx29_packets: 3
>> tx29_bytes: 266
>> tx29_tso_packets: 0
>> tx29_tso_bytes: 0
>> tx29_tso_inner_packets: 0
>> tx29_tso_inner_bytes: 0
>> tx29_csum_partial: 0
>> tx29_csum_partial_inner: 0
>> tx29_added_vlan_packets: 0
>> tx29_nop: 0
>> tx29_csum_none: 3
>> tx29_stopped: 0
>> tx29_dropped: 0
>> tx29_xmit_more: 1
>> tx29_recover: 0
>> tx29_cqes: 2
>> tx29_wake: 0
>> tx29_cqe_err: 0
>> tx30_packets: 0
>> tx30_bytes: 0
>> tx30_tso_packets: 0
>> tx30_tso_bytes: 0
>> tx30_tso_inner_packets: 0
>> tx30_tso_inner_bytes: 0
>> tx30_csum_partial: 0
>> tx30_csum_partial_inner: 0
>> tx30_added_vlan_packets: 0
>> tx30_nop: 0
>> tx30_csum_none: 0
>> tx30_stopped: 0
>> tx30_dropped: 0
>> tx30_xmit_more: 0
>> tx30_recover: 0
>> tx30_cqes: 0
>> tx30_wake: 0
>> tx30_cqe_err: 0
>> tx31_packets: 0
>> tx31_bytes: 0
>> tx31_tso_packets: 0
>> tx31_tso_bytes: 0
>> tx31_tso_inner_packets: 0
>> tx31_tso_inner_bytes: 0
>> tx31_csum_partial: 0
>> tx31_csum_partial_inner: 0
>> tx31_added_vlan_packets: 0
>> tx31_nop: 0
>> tx31_csum_none: 0
>> tx31_stopped: 0
>> tx31_dropped: 0
>> tx31_xmit_more: 0
>> tx31_recover: 0
>> tx31_cqes: 0
>> tx31_wake: 0
>> tx31_cqe_err: 0
>> tx32_packets: 0
>> tx32_bytes: 0
>> tx32_tso_packets: 0
>> tx32_tso_bytes: 0
>> tx32_tso_inner_packets: 0
>> tx32_tso_inner_bytes: 0
>> tx32_csum_partial: 0
>> tx32_csum_partial_inner: 0
>> tx32_added_vlan_packets: 0
>> tx32_nop: 0
>> tx32_csum_none: 0
>> tx32_stopped: 0
>> tx32_dropped: 0
>> tx32_xmit_more: 0
>> tx32_recover: 0
>> tx32_cqes: 0
>> tx32_wake: 0
>> tx32_cqe_err: 0
>> tx33_packets: 0
>> tx33_bytes: 0
>> tx33_tso_packets: 0
>> tx33_tso_bytes: 0
>> tx33_tso_inner_packets: 0
>> tx33_tso_inner_bytes: 0
>> tx33_csum_partial: 0
>> tx33_csum_partial_inner: 0
>> tx33_added_vlan_packets: 0
>> tx33_nop: 0
>> tx33_csum_none: 0
>> tx33_stopped: 0
>> tx33_dropped: 0
>> tx33_xmit_more: 0
>> tx33_recover: 0
>> tx33_cqes: 0
>> tx33_wake: 0
>> tx33_cqe_err: 0
>> tx34_packets: 0
>> tx34_bytes: 0
>> tx34_tso_packets: 0
>> tx34_tso_bytes: 0
>> tx34_tso_inner_packets: 0
>> tx34_tso_inner_bytes: 0
>> tx34_csum_partial: 0
>> tx34_csum_partial_inner: 0
>> tx34_added_vlan_packets: 0
>> tx34_nop: 0
>> tx34_csum_none: 0
>> tx34_stopped: 0
>> tx34_dropped: 0
>> tx34_xmit_more: 0
>> tx34_recover: 0
>> tx34_cqes: 0
>> tx34_wake: 0
>> tx34_cqe_err: 0
>> tx35_packets: 0
>> tx35_bytes: 0
>> tx35_tso_packets: 0
>> tx35_tso_bytes: 0
>> tx35_tso_inner_packets: 0
>> tx35_tso_inner_bytes: 0
>> tx35_csum_partial: 0
>> tx35_csum_partial_inner: 0
>> tx35_added_vlan_packets: 0
>> tx35_nop: 0
>> tx35_csum_none: 0
>> tx35_stopped: 0
>> tx35_dropped: 0
>> tx35_xmit_more: 0
>> tx35_recover: 0
>> tx35_cqes: 0
>> tx35_wake: 0
>> tx35_cqe_err: 0
>> tx36_packets: 0
>> tx36_bytes: 0
>> tx36_tso_packets: 0
>> tx36_tso_bytes: 0
>> tx36_tso_inner_packets: 0
>> tx36_tso_inner_bytes: 0
>> tx36_csum_partial: 0
>> tx36_csum_partial_inner: 0
>> tx36_added_vlan_packets: 0
>> tx36_nop: 0
>> tx36_csum_none: 0
>> tx36_stopped: 0
>> tx36_dropped: 0
>> tx36_xmit_more: 0
>> tx36_recover: 0
>> tx36_cqes: 0
>> tx36_wake: 0
>> tx36_cqe_err: 0
>> tx37_packets: 0
>> tx37_bytes: 0
>> tx37_tso_packets: 0
>> tx37_tso_bytes: 0
>> tx37_tso_inner_packets: 0
>> tx37_tso_inner_bytes: 0
>> tx37_csum_partial: 0
>> tx37_csum_partial_inner: 0
>> tx37_added_vlan_packets: 0
>> tx37_nop: 0
>> tx37_csum_none: 0
>> tx37_stopped: 0
>> tx37_dropped: 0
>> tx37_xmit_more: 0
>> tx37_recover: 0
>> tx37_cqes: 0
>> tx37_wake: 0
>> tx37_cqe_err: 0
>> tx38_packets: 0
>> tx38_bytes: 0
>> tx38_tso_packets: 0
>> tx38_tso_bytes: 0
>> tx38_tso_inner_packets: 0
>> tx38_tso_inner_bytes: 0
>> tx38_csum_partial: 0
>> tx38_csum_partial_inner: 0
>> tx38_added_vlan_packets: 0
>> tx38_nop: 0
>> tx38_csum_none: 0
>> tx38_stopped: 0
>> tx38_dropped: 0
>> tx38_xmit_more: 0
>> tx38_recover: 0
>> tx38_cqes: 0
>> tx38_wake: 0
>> tx38_cqe_err: 0
>> tx39_packets: 0
>> tx39_bytes: 0
>> tx39_tso_packets: 0
>> tx39_tso_bytes: 0
>> tx39_tso_inner_packets: 0
>> tx39_tso_inner_bytes: 0
>> tx39_csum_partial: 0
>> tx39_csum_partial_inner: 0
>> tx39_added_vlan_packets: 0
>> tx39_nop: 0
>> tx39_csum_none: 0
>> tx39_stopped: 0
>> tx39_dropped: 0
>> tx39_xmit_more: 0
>> tx39_recover: 0
>> tx39_cqes: 0
>> tx39_wake: 0
>> tx39_cqe_err: 0
>> tx40_packets: 0
>> tx40_bytes: 0
>> tx40_tso_packets: 0
>> tx40_tso_bytes: 0
>> tx40_tso_inner_packets: 0
>> tx40_tso_inner_bytes: 0
>> tx40_csum_partial: 0
>> tx40_csum_partial_inner: 0
>> tx40_added_vlan_packets: 0
>> tx40_nop: 0
>> tx40_csum_none: 0
>> tx40_stopped: 0
>> tx40_dropped: 0
>> tx40_xmit_more: 0
>> tx40_recover: 0
>> tx40_cqes: 0
>> tx40_wake: 0
>> tx40_cqe_err: 0
>> tx41_packets: 0
>> tx41_bytes: 0
>> tx41_tso_packets: 0
>> tx41_tso_bytes: 0
>> tx41_tso_inner_packets: 0
>> tx41_tso_inner_bytes: 0
>> tx41_csum_partial: 0
>> tx41_csum_partial_inner: 0
>> tx41_added_vlan_packets: 0
>> tx41_nop: 0
>> tx41_csum_none: 0
>> tx41_stopped: 0
>> tx41_dropped: 0
>> tx41_xmit_more: 0
>> tx41_recover: 0
>> tx41_cqes: 0
>> tx41_wake: 0
>> tx41_cqe_err: 0
>> tx42_packets: 0
>> tx42_bytes: 0
>> tx42_tso_packets: 0
>> tx42_tso_bytes: 0
>> tx42_tso_inner_packets: 0
>> tx42_tso_inner_bytes: 0
>> tx42_csum_partial: 0
>> tx42_csum_partial_inner: 0
>> tx42_added_vlan_packets: 0
>> tx42_nop: 0
>> tx42_csum_none: 0
>> tx42_stopped: 0
>> tx42_dropped: 0
>> tx42_xmit_more: 0
>> tx42_recover: 0
>> tx42_cqes: 0
>> tx42_wake: 0
>> tx42_cqe_err: 0
>> tx43_packets: 0
>> tx43_bytes: 0
>> tx43_tso_packets: 0
>> tx43_tso_bytes: 0
>> tx43_tso_inner_packets: 0
>> tx43_tso_inner_bytes: 0
>> tx43_csum_partial: 0
>> tx43_csum_partial_inner: 0
>> tx43_added_vlan_packets: 0
>> tx43_nop: 0
>> tx43_csum_none: 0
>> tx43_stopped: 0
>> tx43_dropped: 0
>> tx43_xmit_more: 0
>> tx43_recover: 0
>> tx43_cqes: 0
>> tx43_wake: 0
>> tx43_cqe_err: 0
>> tx44_packets: 0
>> tx44_bytes: 0
>> tx44_tso_packets: 0
>> tx44_tso_bytes: 0
>> tx44_tso_inner_packets: 0
>> tx44_tso_inner_bytes: 0
>> tx44_csum_partial: 0
>> tx44_csum_partial_inner: 0
>> tx44_added_vlan_packets: 0
>> tx44_nop: 0
>> tx44_csum_none: 0
>> tx44_stopped: 0
>> tx44_dropped: 0
>> tx44_xmit_more: 0
>> tx44_recover: 0
>> tx44_cqes: 0
>> tx44_wake: 0
>> tx44_cqe_err: 0
>> tx45_packets: 0
>> tx45_bytes: 0
>> tx45_tso_packets: 0
>> tx45_tso_bytes: 0
>> tx45_tso_inner_packets: 0
>> tx45_tso_inner_bytes: 0
>> tx45_csum_partial: 0
>> tx45_csum_partial_inner: 0
>> tx45_added_vlan_packets: 0
>> tx45_nop: 0
>> tx45_csum_none: 0
>> tx45_stopped: 0
>> tx45_dropped: 0
>> tx45_xmit_more: 0
>> tx45_recover: 0
>> tx45_cqes: 0
>> tx45_wake: 0
>> tx45_cqe_err: 0
>> tx46_packets: 0
>> tx46_bytes: 0
>> tx46_tso_packets: 0
>> tx46_tso_bytes: 0
>> tx46_tso_inner_packets: 0
>> tx46_tso_inner_bytes: 0
>> tx46_csum_partial: 0
>> tx46_csum_partial_inner: 0
>> tx46_added_vlan_packets: 0
>> tx46_nop: 0
>> tx46_csum_none: 0
>> tx46_stopped: 0
>> tx46_dropped: 0
>> tx46_xmit_more: 0
>> tx46_recover: 0
>> tx46_cqes: 0
>> tx46_wake: 0
>> tx46_cqe_err: 0
>> tx47_packets: 0
>> tx47_bytes: 0
>> tx47_tso_packets: 0
>> tx47_tso_bytes: 0
>> tx47_tso_inner_packets: 0
>> tx47_tso_inner_bytes: 0
>> tx47_csum_partial: 0
>> tx47_csum_partial_inner: 0
>> tx47_added_vlan_packets: 0
>> tx47_nop: 0
>> tx47_csum_none: 0
>> tx47_stopped: 0
>> tx47_dropped: 0
>> tx47_xmit_more: 0
>> tx47_recover: 0
>> tx47_cqes: 0
>> tx47_wake: 0
>> tx47_cqe_err: 0
>> tx48_packets: 0
>> tx48_bytes: 0
>> tx48_tso_packets: 0
>> tx48_tso_bytes: 0
>> tx48_tso_inner_packets: 0
>> tx48_tso_inner_bytes: 0
>> tx48_csum_partial: 0
>> tx48_csum_partial_inner: 0
>> tx48_added_vlan_packets: 0
>> tx48_nop: 0
>> tx48_csum_none: 0
>> tx48_stopped: 0
>> tx48_dropped: 0
>> tx48_xmit_more: 0
>> tx48_recover: 0
>> tx48_cqes: 0
>> tx48_wake: 0
>> tx48_cqe_err: 0
>> tx49_packets: 0
>> tx49_bytes: 0
>> tx49_tso_packets: 0
>> tx49_tso_bytes: 0
>> tx49_tso_inner_packets: 0
>> tx49_tso_inner_bytes: 0
>> tx49_csum_partial: 0
>> tx49_csum_partial_inner: 0
>> tx49_added_vlan_packets: 0
>> tx49_nop: 0
>> tx49_csum_none: 0
>> tx49_stopped: 0
>> tx49_dropped: 0
>> tx49_xmit_more: 0
>> tx49_recover: 0
>> tx49_cqes: 0
>> tx49_wake: 0
>> tx49_cqe_err: 0
>> tx50_packets: 0
>> tx50_bytes: 0
>> tx50_tso_packets: 0
>> tx50_tso_bytes: 0
>> tx50_tso_inner_packets: 0
>> tx50_tso_inner_bytes: 0
>> tx50_csum_partial: 0
>> tx50_csum_partial_inner: 0
>> tx50_added_vlan_packets: 0
>> tx50_nop: 0
>> tx50_csum_none: 0
>> tx50_stopped: 0
>> tx50_dropped: 0
>> tx50_xmit_more: 0
>> tx50_recover: 0
>> tx50_cqes: 0
>> tx50_wake: 0
>> tx50_cqe_err: 0
>> tx51_packets: 0
>> tx51_bytes: 0
>> tx51_tso_packets: 0
>> tx51_tso_bytes: 0
>> tx51_tso_inner_packets: 0
>> tx51_tso_inner_bytes: 0
>> tx51_csum_partial: 0
>> tx51_csum_partial_inner: 0
>> tx51_added_vlan_packets: 0
>> tx51_nop: 0
>> tx51_csum_none: 0
>> tx51_stopped: 0
>> tx51_dropped: 0
>> tx51_xmit_more: 0
>> tx51_recover: 0
>> tx51_cqes: 0
>> tx51_wake: 0
>> tx51_cqe_err: 0
>> tx52_packets: 0
>> tx52_bytes: 0
>> tx52_tso_packets: 0
>> tx52_tso_bytes: 0
>> tx52_tso_inner_packets: 0
>> tx52_tso_inner_bytes: 0
>> tx52_csum_partial: 0
>> tx52_csum_partial_inner: 0
>> tx52_added_vlan_packets: 0
>> tx52_nop: 0
>> tx52_csum_none: 0
>> tx52_stopped: 0
>> tx52_dropped: 0
>> tx52_xmit_more: 0
>> tx52_recover: 0
>> tx52_cqes: 0
>> tx52_wake: 0
>> tx52_cqe_err: 0
>> tx53_packets: 0
>> tx53_bytes: 0
>> tx53_tso_packets: 0
>> tx53_tso_bytes: 0
>> tx53_tso_inner_packets: 0
>> tx53_tso_inner_bytes: 0
>> tx53_csum_partial: 0
>> tx53_csum_partial_inner: 0
>> tx53_added_vlan_packets: 0
>> tx53_nop: 0
>> tx53_csum_none: 0
>> tx53_stopped: 0
>> tx53_dropped: 0
>> tx53_xmit_more: 0
>> tx53_recover: 0
>> tx53_cqes: 0
>> tx53_wake: 0
>> tx53_cqe_err: 0
>> tx54_packets: 0
>> tx54_bytes: 0
>> tx54_tso_packets: 0
>> tx54_tso_bytes: 0
>> tx54_tso_inner_packets: 0
>> tx54_tso_inner_bytes: 0
>> tx54_csum_partial: 0
>> tx54_csum_partial_inner: 0
>> tx54_added_vlan_packets: 0
>> tx54_nop: 0
>> tx54_csum_none: 0
>> tx54_stopped: 0
>> tx54_dropped: 0
>> tx54_xmit_more: 0
>> tx54_recover: 0
>> tx54_cqes: 0
>> tx54_wake: 0
>> tx54_cqe_err: 0
>> tx55_packets: 0
>> tx55_bytes: 0
>> tx55_tso_packets: 0
>> tx55_tso_bytes: 0
>> tx55_tso_inner_packets: 0
>> tx55_tso_inner_bytes: 0
>> tx55_csum_partial: 0
>> tx55_csum_partial_inner: 0
>> tx55_added_vlan_packets: 0
>> tx55_nop: 0
>> tx55_csum_none: 0
>> tx55_stopped: 0
>> tx55_dropped: 0
>> tx55_xmit_more: 0
>> tx55_recover: 0
>> tx55_cqes: 0
>> tx55_wake: 0
>> tx55_cqe_err: 0
>> tx0_xdp_xmit: 0
>> tx0_xdp_full: 0
>> tx0_xdp_err: 0
>> tx0_xdp_cqes: 0
>> tx1_xdp_xmit: 0
>> tx1_xdp_full: 0
>> tx1_xdp_err: 0
>> tx1_xdp_cqes: 0
>> tx2_xdp_xmit: 0
>> tx2_xdp_full: 0
>> tx2_xdp_err: 0
>> tx2_xdp_cqes: 0
>> tx3_xdp_xmit: 0
>> tx3_xdp_full: 0
>> tx3_xdp_err: 0
>> tx3_xdp_cqes: 0
>> tx4_xdp_xmit: 0
>> tx4_xdp_full: 0
>> tx4_xdp_err: 0
>> tx4_xdp_cqes: 0
>> tx5_xdp_xmit: 0
>> tx5_xdp_full: 0
>> tx5_xdp_err: 0
>> tx5_xdp_cqes: 0
>> tx6_xdp_xmit: 0
>> tx6_xdp_full: 0
>> tx6_xdp_err: 0
>> tx6_xdp_cqes: 0
>> tx7_xdp_xmit: 0
>> tx7_xdp_full: 0
>> tx7_xdp_err: 0
>> tx7_xdp_cqes: 0
>> tx8_xdp_xmit: 0
>> tx8_xdp_full: 0
>> tx8_xdp_err: 0
>> tx8_xdp_cqes: 0
>> tx9_xdp_xmit: 0
>> tx9_xdp_full: 0
>> tx9_xdp_err: 0
>> tx9_xdp_cqes: 0
>> tx10_xdp_xmit: 0
>> tx10_xdp_full: 0
>> tx10_xdp_err: 0
>> tx10_xdp_cqes: 0
>> tx11_xdp_xmit: 0
>> tx11_xdp_full: 0
>> tx11_xdp_err: 0
>> tx11_xdp_cqes: 0
>> tx12_xdp_xmit: 0
>> tx12_xdp_full: 0
>> tx12_xdp_err: 0
>> tx12_xdp_cqes: 0
>> tx13_xdp_xmit: 0
>> tx13_xdp_full: 0
>> tx13_xdp_err: 0
>> tx13_xdp_cqes: 0
>> tx14_xdp_xmit: 0
>> tx14_xdp_full: 0
>> tx14_xdp_err: 0
>> tx14_xdp_cqes: 0
>> tx15_xdp_xmit: 0
>> tx15_xdp_full: 0
>> tx15_xdp_err: 0
>> tx15_xdp_cqes: 0
>> tx16_xdp_xmit: 0
>> tx16_xdp_full: 0
>> tx16_xdp_err: 0
>> tx16_xdp_cqes: 0
>> tx17_xdp_xmit: 0
>> tx17_xdp_full: 0
>> tx17_xdp_err: 0
>> tx17_xdp_cqes: 0
>> tx18_xdp_xmit: 0
>> tx18_xdp_full: 0
>> tx18_xdp_err: 0
>> tx18_xdp_cqes: 0
>> tx19_xdp_xmit: 0
>> tx19_xdp_full: 0
>> tx19_xdp_err: 0
>> tx19_xdp_cqes: 0
>> tx20_xdp_xmit: 0
>> tx20_xdp_full: 0
>> tx20_xdp_err: 0
>> tx20_xdp_cqes: 0
>> tx21_xdp_xmit: 0
>> tx21_xdp_full: 0
>> tx21_xdp_err: 0
>> tx21_xdp_cqes: 0
>> tx22_xdp_xmit: 0
>> tx22_xdp_full: 0
>> tx22_xdp_err: 0
>> tx22_xdp_cqes: 0
>> tx23_xdp_xmit: 0
>> tx23_xdp_full: 0
>> tx23_xdp_err: 0
>> tx23_xdp_cqes: 0
>> tx24_xdp_xmit: 0
>> tx24_xdp_full: 0
>> tx24_xdp_err: 0
>> tx24_xdp_cqes: 0
>> tx25_xdp_xmit: 0
>> tx25_xdp_full: 0
>> tx25_xdp_err: 0
>> tx25_xdp_cqes: 0
>> tx26_xdp_xmit: 0
>> tx26_xdp_full: 0
>> tx26_xdp_err: 0
>> tx26_xdp_cqes: 0
>> tx27_xdp_xmit: 0
>> tx27_xdp_full: 0
>> tx27_xdp_err: 0
>> tx27_xdp_cqes: 0
>> tx28_xdp_xmit: 0
>> tx28_xdp_full: 0
>> tx28_xdp_err: 0
>> tx28_xdp_cqes: 0
>> tx29_xdp_xmit: 0
>> tx29_xdp_full: 0
>> tx29_xdp_err: 0
>> tx29_xdp_cqes: 0
>> tx30_xdp_xmit: 0
>> tx30_xdp_full: 0
>> tx30_xdp_err: 0
>> tx30_xdp_cqes: 0
>> tx31_xdp_xmit: 0
>> tx31_xdp_full: 0
>> tx31_xdp_err: 0
>> tx31_xdp_cqes: 0
>> tx32_xdp_xmit: 0
>> tx32_xdp_full: 0
>> tx32_xdp_err: 0
>> tx32_xdp_cqes: 0
>> tx33_xdp_xmit: 0
>> tx33_xdp_full: 0
>> tx33_xdp_err: 0
>> tx33_xdp_cqes: 0
>> tx34_xdp_xmit: 0
>> tx34_xdp_full: 0
>> tx34_xdp_err: 0
>> tx34_xdp_cqes: 0
>> tx35_xdp_xmit: 0
>> tx35_xdp_full: 0
>> tx35_xdp_err: 0
>> tx35_xdp_cqes: 0
>> tx36_xdp_xmit: 0
>> tx36_xdp_full: 0
>> tx36_xdp_err: 0
>> tx36_xdp_cqes: 0
>> tx37_xdp_xmit: 0
>> tx37_xdp_full: 0
>> tx37_xdp_err: 0
>> tx37_xdp_cqes: 0
>> tx38_xdp_xmit: 0
>> tx38_xdp_full: 0
>> tx38_xdp_err: 0
>> tx38_xdp_cqes: 0
>> tx39_xdp_xmit: 0
>> tx39_xdp_full: 0
>> tx39_xdp_err: 0
>> tx39_xdp_cqes: 0
>> tx40_xdp_xmit: 0
>> tx40_xdp_full: 0
>> tx40_xdp_err: 0
>> tx40_xdp_cqes: 0
>> tx41_xdp_xmit: 0
>> tx41_xdp_full: 0
>> tx41_xdp_err: 0
>> tx41_xdp_cqes: 0
>> tx42_xdp_xmit: 0
>> tx42_xdp_full: 0
>> tx42_xdp_err: 0
>> tx42_xdp_cqes: 0
>> tx43_xdp_xmit: 0
>> tx43_xdp_full: 0
>> tx43_xdp_err: 0
>> tx43_xdp_cqes: 0
>> tx44_xdp_xmit: 0
>> tx44_xdp_full: 0
>> tx44_xdp_err: 0
>> tx44_xdp_cqes: 0
>> tx45_xdp_xmit: 0
>> tx45_xdp_full: 0
>> tx45_xdp_err: 0
>> tx45_xdp_cqes: 0
>> tx46_xdp_xmit: 0
>> tx46_xdp_full: 0
>> tx46_xdp_err: 0
>> tx46_xdp_cqes: 0
>> tx47_xdp_xmit: 0
>> tx47_xdp_full: 0
>> tx47_xdp_err: 0
>> tx47_xdp_cqes: 0
>> tx48_xdp_xmit: 0
>> tx48_xdp_full: 0
>> tx48_xdp_err: 0
>> tx48_xdp_cqes: 0
>> tx49_xdp_xmit: 0
>> tx49_xdp_full: 0
>> tx49_xdp_err: 0
>> tx49_xdp_cqes: 0
>> tx50_xdp_xmit: 0
>> tx50_xdp_full: 0
>> tx50_xdp_err: 0
>> tx50_xdp_cqes: 0
>> tx51_xdp_xmit: 0
>> tx51_xdp_full: 0
>> tx51_xdp_err: 0
>> tx51_xdp_cqes: 0
>> tx52_xdp_xmit: 0
>> tx52_xdp_full: 0
>> tx52_xdp_err: 0
>> tx52_xdp_cqes: 0
>> tx53_xdp_xmit: 0
>> tx53_xdp_full: 0
>> tx53_xdp_err: 0
>> tx53_xdp_cqes: 0
>> tx54_xdp_xmit: 0
>> tx54_xdp_full: 0
>> tx54_xdp_err: 0
>> tx54_xdp_cqes: 0
>> tx55_xdp_xmit: 0
>> tx55_xdp_full: 0
>> tx55_xdp_err: 0
>> tx55_xdp_cqes: 0
>>
>>
>> mpstat -P ALL 1 10
>> Average: CPU %usr %nice %sys %iowait %irq %soft
>> %steal
>> %guest %gnice %idle
>> Average: all 0.04 0.00 6.94 0.02 0.00 32.00
>> 0.00 0.00 0.00 61.00
>> Average: 0 0.00 0.00 1.20 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 98.80
>> Average: 1 0.00 0.00 2.30 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 97.70
>> Average: 2 0.10 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.90
>> Average: 3 0.10 0.00 1.50 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 98.40
>> Average: 4 0.50 0.00 2.50 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 97.00
>> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 6 0.90 0.00 10.20 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 88.90
>> Average: 7 0.00 0.00 0.00 1.40 0.00 0.00
>> 0.00
>> 0.00 0.00 98.60
>> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 11 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 14 0.00 0.00 12.99 0.00 0.00 62.64
>> 0.00 0.00 0.00 24.38
>> Average: 15 0.00 0.00 12.70 0.00 0.00 63.40
>> 0.00 0.00 0.00 23.90
>> Average: 16 0.00 0.00 11.20 0.00 0.00 66.40
>> 0.00 0.00 0.00 22.40
>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>> 0.00 0.00 0.00 31.30
>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>> 0.00 0.00 0.00 24.90
>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>> 0.00 0.00 0.00 19.68
>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>> 0.00 0.00 0.00 18.00
>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>> 0.00 0.00 0.00 17.40
>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>> 0.00 0.00 0.00 26.03
>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>> 0.00 0.00 0.00 17.52
>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>> 0.00 0.00 0.00 18.20
>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>> 0.00 0.00 0.00 17.68
>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>> 0.00 0.00 0.00 18.20
>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>> 0.00 0.00 0.00 20.42
> The numa cores are not at 100% util, you have around 20% of idle on
> each one.
Yes - no 100% cpu - but the difference between 80% and 100% is like push
aditional 1-2Gbit/s
>
>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 96.10
>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.70
>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.50
>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 100.00
>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 97.40
>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.10
>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>> 0.00
>> 0.00 0.00 99.40
>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>> 0.00 0.00 0.00 19.42
>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>> 0.00 0.00 0.00 26.60
>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>> 0.00 0.00 0.00 21.60
>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>> 0.00 0.00 0.00 20.50
>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>> 0.00 0.00 0.00 21.60
>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>> 0.00 0.00 0.00 24.58
>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>> 0.00 0.00 0.00 24.68
>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>> 0.00 0.00 0.00 28.34
>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>> 0.00 0.00 0.00 25.83
>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>> 0.00 0.00 0.00 27.20
>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>> 0.00 0.00 0.00 28.03
>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>> 0.00 0.00 0.00 24.70
>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>> 0.00 0.00 0.00 25.30
>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>> 0.00 0.00 0.00 26.77
>>
>>
>> ethtool -k enp175s0f0
>> Features for enp175s0f0:
>> rx-checksumming: on
>> tx-checksumming: on
>> tx-checksum-ipv4: on
>> tx-checksum-ip-generic: off [fixed]
>> tx-checksum-ipv6: on
>> tx-checksum-fcoe-crc: off [fixed]
>> tx-checksum-sctp: off [fixed]
>> scatter-gather: on
>> tx-scatter-gather: on
>> tx-scatter-gather-fraglist: off [fixed]
>> tcp-segmentation-offload: on
>> tx-tcp-segmentation: on
>> tx-tcp-ecn-segmentation: off [fixed]
>> tx-tcp-mangleid-segmentation: off
>> tx-tcp6-segmentation: on
>> udp-fragmentation-offload: off
>> generic-segmentation-offload: on
>> generic-receive-offload: on
>> large-receive-offload: off [fixed]
>> rx-vlan-offload: on
>> tx-vlan-offload: on
>> ntuple-filters: off
>> receive-hashing: on
>> highdma: on [fixed]
>> rx-vlan-filter: on
>> vlan-challenged: off [fixed]
>> tx-lockless: off [fixed]
>> netns-local: off [fixed]
>> tx-gso-robust: off [fixed]
>> tx-fcoe-segmentation: off [fixed]
>> tx-gre-segmentation: on
>> tx-gre-csum-segmentation: on
>> tx-ipxip4-segmentation: off [fixed]
>> tx-ipxip6-segmentation: off [fixed]
>> tx-udp_tnl-segmentation: on
>> tx-udp_tnl-csum-segmentation: on
>> tx-gso-partial: on
>> tx-sctp-segmentation: off [fixed]
>> tx-esp-segmentation: off [fixed]
>> tx-udp-segmentation: on
>> fcoe-mtu: off [fixed]
>> tx-nocache-copy: off
>> loopback: off [fixed]
>> rx-fcs: off
>> rx-all: off
>> tx-vlan-stag-hw-insert: on
>> rx-vlan-stag-hw-parse: off [fixed]
>> rx-vlan-stag-filter: on [fixed]
>> l2-fwd-offload: off [fixed]
>> hw-tc-offload: off
>> esp-hw-offload: off [fixed]
>> esp-tx-csum-hw-offload: off [fixed]
>> rx-udp_tunnel-port-offload: on
>> tls-hw-tx-offload: off [fixed]
>> tls-hw-rx-offload: off [fixed]
>> rx-gro-hw: off [fixed]
>> tls-hw-record: off [fixed]
>>
>> ethtool -c enp175s0f0
>> Coalesce parameters for enp175s0f0:
>> Adaptive RX: off TX: on
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>> dmac: 32703
>>
>> rx-usecs: 256
>> rx-frames: 128
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 8
>> tx-frames: 128
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
>> rx-usecs-low: 0
>> rx-frame-low: 0
>> tx-usecs-low: 0
>> tx-frame-low: 0
>>
>> rx-usecs-high: 0
>> rx-frame-high: 0
>> tx-usecs-high: 0
>> tx-frame-high: 0
>>
>> ethtool -g enp175s0f0
>> Ring parameters for enp175s0f0:
>> Pre-set maximums:
>> RX: 8192
>> RX Mini: 0
>> RX Jumbo: 0
>> TX: 8192
>> Current hardware settings:
>> RX: 4096
>> RX Mini: 0
>> RX Jumbo: 0
>> TX: 4096
>>
>>
>>
>>
>>
>>
Also changed a little coalesce params - and best for this config are:
ethtool -c enp175s0f0
Coalesce parameters for enp175s0f0:
Adaptive RX: off TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0
dmac: 32573
rx-usecs: 40
rx-frames: 128
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 8
tx-frames: 8
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
Less drops on RX side - and more pps in overall forwarded.
^ permalink raw reply
* Re: [RFC PATCH] lib: Introduce generic __cmpxchg_u64() and use it where needed
From: Paul E. McKenney @ 2018-11-01 20:29 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Eric Dumazet, Trond Myklebust, mark.rutland@arm.com,
linux-kernel@vger.kernel.org, ralf@linux-mips.org,
jlayton@kernel.org, linuxppc-dev@lists.ozlabs.org,
bfields@fieldses.org, linux-mips@linux-mips.org,
linux@roeck-us.net, linux-nfs@vger.kernel.org,
akpm@linux-foundation.org, will.deacon@arm.com,
boqun.feng@gmail.com, paul.burton@mips.com,
"anna.schumaker@netapp.com" <an
In-Reply-To: <20181101172739.GA3196@hirez.programming.kicks-ass.net>
On Thu, Nov 01, 2018 at 06:27:39PM +0100, Peter Zijlstra wrote:
> On Thu, Nov 01, 2018 at 06:14:32PM +0100, Peter Zijlstra wrote:
> > > This reminds me of this sooooo silly patch :/
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=adb03115f4590baa280ddc440a8eff08a6be0cb7
>
> You'd probably want to write it like so; +- some ordering stuff, that
> code didn't look like it really needs the memory barriers implied by
> these, but I didn't look too hard.
The atomic_fetch_add() API would need to be propagated out to the other
architectures, correct?
Thanx, Paul
> diff --git a/net/ipv4/route.c b/net/ipv4/route.c
> index c0a9d26c06ce..11deb1d7e96b 100644
> --- a/net/ipv4/route.c
> +++ b/net/ipv4/route.c
> @@ -485,16 +485,10 @@ u32 ip_idents_reserve(u32 hash, int segs)
> u32 now = (u32)jiffies;
> u32 new, delta = 0;
>
> - if (old != now && cmpxchg(p_tstamp, old, now) == old)
> + if (old != now && try_cmpxchg(p_tstamp, &old, now))
> delta = prandom_u32_max(now - old);
>
> - /* Do not use atomic_add_return() as it makes UBSAN unhappy */
> - do {
> - old = (u32)atomic_read(p_id);
> - new = old + delta + segs;
> - } while (atomic_cmpxchg(p_id, old, new) != old);
> -
> - return new - segs;
> + return atomic_fetch_add(segs + delta, p_id) + delta;
> }
> EXPORT_SYMBOL(ip_idents_reserve);
>
>
^ permalink raw reply
* [PATCH net] net: hns3: Fix for out-of-bounds access when setting pfc back pressure
From: Yunsheng Lin @ 2018-11-01 12:12 UTC (permalink / raw)
To: davem
Cc: huangdaode, xuwei5, liguozhu, Yisen.Zhuang, john.garry, linuxarm,
yisen.zhuang, salil.mehta, lipeng321, netdev, linux-kernel
The vport should be initialized to hdev->vport for each bp group,
otherwise it will cause out-of-bounds access and bp setting not
correct problem.
[ 35.254124] BUG: KASAN: slab-out-of-bounds in hclge_pause_setup_hw+0x2a0/0x3f8 [hclge]
[ 35.254126] Read of size 2 at addr ffff803b6651581a by task kworker/0:1/14
[ 35.254132] CPU: 0 PID: 14 Comm: kworker/0:1 Not tainted 4.19.0-rc7-hulk+ #85
[ 35.254133] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B052 (V0.52) 09/14/2018
[ 35.254141] Workqueue: events work_for_cpu_fn
[ 35.254144] Call trace:
[ 35.254147] dump_backtrace+0x0/0x2f0
[ 35.254149] show_stack+0x24/0x30
[ 35.254154] dump_stack+0x110/0x184
[ 35.254157] print_address_description+0x168/0x2b0
[ 35.254160] kasan_report+0x184/0x310
[ 35.254162] __asan_load2+0x7c/0xa0
[ 35.254170] hclge_pause_setup_hw+0x2a0/0x3f8 [hclge]
[ 35.254177] hclge_tm_init_hw+0x794/0x9f0 [hclge]
[ 35.254184] hclge_tm_schd_init+0x48/0x58 [hclge]
[ 35.254191] hclge_init_ae_dev+0x778/0x1168 [hclge]
[ 35.254196] hnae3_register_ae_dev+0x14c/0x298 [hnae3]
[ 35.254206] hns3_probe+0x88/0xa8 [hns3]
[ 35.254210] local_pci_probe+0x7c/0xf0
[ 35.254212] work_for_cpu_fn+0x34/0x50
[ 35.254214] process_one_work+0x4d4/0xa38
[ 35.254216] worker_thread+0x55c/0x8d8
[ 35.254219] kthread+0x1b0/0x1b8
[ 35.254222] ret_from_fork+0x10/0x1c
[ 35.254224] The buggy address belongs to the page:
[ 35.254228] page:ffff7e00ed994400 count:1 mapcount:0 mapping:0000000000000000 index:0x0 compound_mapcount: 0
[ 35.273835] flags: 0xfffff8000008000(head)
[ 35.282007] raw: 0fffff8000008000 dead000000000100 dead000000000200 0000000000000000
[ 35.282010] raw: 0000000000000000 0000000000000000 00000001ffffffff 0000000000000000
[ 35.282012] page dumped because: kasan: bad access detected
[ 35.282014] Memory state around the buggy address:
[ 35.282017] ffff803b66515700: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
[ 35.282019] ffff803b66515780: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
[ 35.282021] >ffff803b66515800: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
[ 35.282022] ^
[ 35.282024] ffff803b66515880: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
[ 35.282026] ffff803b66515900: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
[ 35.282028] ==================================================================
[ 35.282029] Disabling lock debugging due to kernel taint
[ 35.282747] hclge driver initialization finished.
Fixes: 67bf2541f4b9 ("net: hns3: Fixes the back pressure setting when sriov is enabled")
Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
---
drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c
index aa5cb98..0c9c6ae 100644
--- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c
+++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_tm.c
@@ -1168,11 +1168,12 @@ static int hclge_pfc_setup_hw(struct hclge_dev *hdev)
*/
static int hclge_bp_setup_hw(struct hclge_dev *hdev, u8 tc)
{
- struct hclge_vport *vport = hdev->vport;
u32 i, k, qs_bitmap;
int ret;
for (i = 0; i < HCLGE_BP_GRP_NUM; i++) {
+ struct hclge_vport *vport = hdev->vport;
+
qs_bitmap = 0;
for (k = 0; k < hdev->num_alloc_vport; k++) {
--
2.8.1
^ permalink raw reply related
* Re: [PATCH v3 1/2] kretprobe: produce sane stack traces
From: Aleksa Sarai @ 2018-11-01 21:13 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Naveen N. Rao, Anil S Keshavamurthy, David S. Miller,
Jonathan Corbet, Peter Zijlstra, Ingo Molnar,
Arnaldo Carvalho de Melo, Alexander Shishkin, Jiri Olsa,
Namhyung Kim, Steven Rostedt, Shuah Khan, Alexei Starovoitov,
Daniel Borkmann, Brendan Gregg, Christian Brauner, Aleksa Sarai,
netdev, linux-doc, linux-kernel
In-Reply-To: <20181102002039.8f22c10fa47cae75fa709165@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]
On 2018-11-02, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> Please split the test case as an independent patch.
Will do. Should the Documentation/ change also be a separate patch?
> > new file mode 100644
> > index 000000000000..03146c6a1a3c
> > --- /dev/null
> > +++ b/tools/testing/selftests/ftrace/test.d/kprobe/kretprobe_stacktrace.tc
> > @@ -0,0 +1,25 @@
> > +#!/bin/sh
> > +# SPDX-License-Identifier: GPL-2.0+
> > +# description: Kretprobe dynamic event with a stacktrace
> > +
> > +[ -f kprobe_events ] || exit_unsupported # this is configurable
> > +
> > +echo 0 > events/enable
> > +echo 1 > options/stacktrace
> > +
> > +echo 'r:teststackprobe sched_fork $retval' > kprobe_events
> > +grep teststackprobe kprobe_events
> > +test -d events/kprobes/teststackprobe
>
> Hmm, what happen if we have 2 or more kretprobes on same stack?
> It seems you just save stack in pre_handler, but that stack can already
> includes another kretprobe's trampline address...
Yeah, you're quite right...
My first instinct was to do something awful like iterate over the set of
"kretprobe_instance"s with ->task == current, and then correct
kretprobe_trampoline entries using ->ret_addr. (I think this would be
correct because each task can only be in one context at once, and in
order to get to a particular kretprobe all of your caller kretprobes
were inserted before you and any sibling or later kretprobe_instances
will have been removed. But I might be insanely wrong on this front.)
However (as I noted in the other thread), there is a problem where
kretprobe_trampoline actually stops the unwinder in its tracks and thus
you only get the first kretprobe_trampoline. This is something I'm going
to look into some more (despite not having made progress on it last
time) since now it's something that actually needs to be fixed (and
as I mentioned in the other thread, show_stack() actually works on x86
in this context unlike the other stack_trace users).
--
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* [PULL] vhost: cleanups and fixes
From: Michael S. Tsirkin @ 2018-11-01 21:19 UTC (permalink / raw)
To: Linus Torvalds
Cc: lenaic, mhocko, bijan.mottahedeh, kvm, mst, netdev, liang.z.li,
linux-kernel, virtualization, stefanha, joe, akpm, mhocko,
torvalds
The following changes since commit 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d:
Linux 4.19 (2018-10-22 07:37:37 +0100)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git tags/for_linus
for you to fetch changes up to 79f800b2e76923cd8ce0aa659cb5c019d9643bc9:
MAINTAINERS: remove reference to bogus vsock file (2018-10-24 21:16:14 -0400)
----------------------------------------------------------------
virtio, vhost: fixes, tweaks
virtio balloon page hinting support
vhost scsi control queue
misc fixes.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
----------------------------------------------------------------
Bijan Mottahedeh (3):
vhost/scsi: Respond to control queue operations
vhost/scsi: Extract common handling code from control queue handler
vhost/scsi: Use common handling code in request queue handler
Greg Edwards (1):
vhost/scsi: truncate T10 PI iov_iter to prot_bytes
Lénaïc Huard (1):
kvm_config: add CONFIG_VIRTIO_MENU
Stefan Hajnoczi (1):
MAINTAINERS: remove reference to bogus vsock file
Wei Wang (3):
virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT
mm/page_poison: expose page_poisoning_enabled to kernel modules
virtio-balloon: VIRTIO_BALLOON_F_PAGE_POISON
MAINTAINERS | 1 -
drivers/vhost/scsi.c | 426 ++++++++++++++++++++++++++++--------
drivers/virtio/virtio_balloon.c | 380 +++++++++++++++++++++++++++++---
include/uapi/linux/virtio_balloon.h | 8 +
kernel/configs/kvm_guest.config | 1 +
mm/page_poison.c | 6 +
6 files changed, 688 insertions(+), 134 deletions(-)
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH v1 net] net: dsa: microchip: initialize mutex before use
From: Andrew Lunn @ 2018-11-01 12:17 UTC (permalink / raw)
To: Tristram.Ha
Cc: David S. Miller, Florian Fainelli, Pavel Machek, UNGLinuxDriver,
netdev
In-Reply-To: <1541040548-13465-1-git-send-email-Tristram.Ha@microchip.com>
On Wed, Oct 31, 2018 at 07:49:08PM -0700, Tristram.Ha@microchip.com wrote:
> From: Tristram Ha <Tristram.Ha@microchip.com>
>
> Initialize mutex before use. Avoid kernel complaint when
> CONFIG_DEBUG_LOCK_ALLOC is enabled.
>
> Fixes: b987e98e50ab90e5 ("dsa: add DSA switch driver for Microchip KSZ9477")
> Signed-off-by: Tristram Ha <Tristram.Ha@microchip.com>
Reviewed-by: Andrew Lunn <andrew@lunn.ch>
Andrew
^ permalink raw reply
* [PATCH net] bonding/802.3ad: fix link_failure_count tracking
From: Jarod Wilson @ 2018-11-01 21:22 UTC (permalink / raw)
To: linux-kernel
Cc: Jarod Wilson, Mahesh Bandewar, David S . Miller, netdev, stable
Commit 4d2c0cda07448ea6980f00102dc3964eb25e241c set slave->link to
BOND_LINK_DOWN for 802.3ad bonds whenever invalid speed/duplex values
were read, to fix a problem with slaves getting into weird states, but
in the process, broke tracking of link failures, as going straight to
BOND_LINK_DOWN when a link is indeed down (cable pulled, switch rebooted)
means we broke out of bond_miimon_inspect()'s BOND_LINK_DOWN case because
!link_state was already true, we never incremented commit, and never got
a chance to call bond_miimon_commit(), where slave->link_failure_count
would be incremented. I believe the simple fix here is to mark the slave
as BOND_LINK_FAIL, and let bond_miimon_inspect() transition the link from
_FAIL to either _UP or _DOWN, and in the latter case, we now get proper
incrementing of link_failure_count again.
Fixes: 4d2c0cda07448ea6980f00102dc3964eb25e241c
CC: Mahesh Bandewar <maheshb@google.com>
CC: David S. Miller <davem@davemloft.net>
CC: netdev@vger.kernel.org
CC: stable@vger.kernel.org
Signed-off-by: Jarod Wilson <jarod@redhat.com>
---
drivers/net/bonding/bond_main.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index ffa37adb7681..333387f1f1fe 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -3112,13 +3112,13 @@ static int bond_slave_netdev_event(unsigned long event,
case NETDEV_CHANGE:
/* For 802.3ad mode only:
* Getting invalid Speed/Duplex values here will put slave
- * in weird state. So mark it as link-down for the time
+ * in weird state. So mark it as link-fail for the time
* being and let link-monitoring (miimon) set it right when
* correct speeds/duplex are available.
*/
if (bond_update_speed_duplex(slave) &&
BOND_MODE(bond) == BOND_MODE_8023AD)
- slave->link = BOND_LINK_DOWN;
+ slave->link = BOND_LINK_FAIL;
if (BOND_MODE(bond) == BOND_MODE_8023AD)
bond_3ad_adapter_speed_duplex_changed(slave);
--
2.16.1
^ permalink raw reply related
* Re: [RFC PATCH] lib: Introduce generic __cmpxchg_u64() and use it where needed
From: Peter Zijlstra @ 2018-11-01 21:38 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Eric Dumazet, Trond Myklebust, mark.rutland@arm.com,
linux-kernel@vger.kernel.org, ralf@linux-mips.org,
jlayton@kernel.org, linuxppc-dev@lists.ozlabs.org,
bfields@fieldses.org, linux-mips@linux-mips.org,
linux@roeck-us.net, linux-nfs@vger.kernel.org,
akpm@linux-foundation.org, will.deacon@arm.com,
boqun.feng@gmail.com, paul.burton@mips.com,
"anna.schumaker@netapp.com" <an
In-Reply-To: <20181101202910.GB4170@linux.ibm.com>
On Thu, Nov 01, 2018 at 01:29:10PM -0700, Paul E. McKenney wrote:
> On Thu, Nov 01, 2018 at 06:27:39PM +0100, Peter Zijlstra wrote:
> > On Thu, Nov 01, 2018 at 06:14:32PM +0100, Peter Zijlstra wrote:
> > > > This reminds me of this sooooo silly patch :/
> > > >
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=adb03115f4590baa280ddc440a8eff08a6be0cb7
> >
> > You'd probably want to write it like so; +- some ordering stuff, that
> > code didn't look like it really needs the memory barriers implied by
> > these, but I didn't look too hard.
>
> The atomic_fetch_add() API would need to be propagated out to the other
> architectures, correct?
Like these commits I did like 2 years ago ? :-)
$ git log --oneline 6dc25876cdb1...1f51dee7ca74
6dc25876cdb1 locking/atomic, arch/xtensa: Implement atomic_fetch_{add,sub,and,or,xor}()
a8bcccaba162 locking/atomic, arch/x86: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
1af5de9af138 locking/atomic, arch/tile: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
3a1adb23a52c locking/atomic, arch/sparc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
7d9794e75237 locking/atomic, arch/sh: Implement atomic_fetch_{add,sub,and,or,xor}()
56fefbbc3f13 locking/atomic, arch/s390: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
a28cc7bbe8e3 locking/atomic, arch/powerpc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}{,_relaxed,_acquire,_release}()
e5857a6ed600 locking/atomic, arch/parisc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
f8d638e28d7c locking/atomic, arch/mn10300: Implement atomic_fetch_{add,sub,and,or,xor}()
4edac529eb62 locking/atomic, arch/mips: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
e898eb27ffd8 locking/atomic, arch/metag: Implement atomic_fetch_{add,sub,and,or,xor}()
e39d88ea3ce4 locking/atomic, arch/m68k: Implement atomic_fetch_{add,sub,and,or,xor}()
f64937052303 locking/atomic, arch/m32r: Implement atomic_fetch_{add,sub,and,or,xor}()
cc102507fac7 locking/atomic, arch/ia64: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
4be7dd393515 locking/atomic, arch/hexagon: Implement atomic_fetch_{add,sub,and,or,xor}()
0c074cbc3309 locking/atomic, arch/h8300: Implement atomic_fetch_{add,sub,and,or,xor}()
d9c730281617 locking/atomic, arch/frv: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
e87fc0ec0705 locking/atomic, arch/blackfin: Implement atomic_fetch_{add,sub,and,or,xor}()
1a6eafacd481 locking/atomic, arch/avr32: Implement atomic_fetch_{add,sub,and,or,xor}()
2efe95fe6952 locking/atomic, arch/arm64: Implement atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() for LSE instructions
6822a84dd4e3 locking/atomic, arch/arm64: Generate LSE non-return cases using common macros
e490f9b1d3b4 locking/atomic, arch/arm64: Implement atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}()
6da068c1beba locking/atomic, arch/arm: Implement atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}()
fbffe892e525 locking/atomic, arch/arc: Implement atomic_fetch_{add,sub,and,andnot,or,xor}()
^ permalink raw reply
* Re: [RFC PATCH] lib: Introduce generic __cmpxchg_u64() and use it where needed
From: Peter Zijlstra @ 2018-11-01 21:45 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Paul E. McKenney, Trond Myklebust, mark.rutland@arm.com,
linux-kernel@vger.kernel.org, ralf@linux-mips.org,
jlayton@kernel.org, linuxppc-dev@lists.ozlabs.org,
bfields@fieldses.org, linux-mips@linux-mips.org,
linux@roeck-us.net, linux-nfs@vger.kernel.org,
akpm@linux-foundation.org, will.deacon@arm.com,
boqun.feng@gmail.com, paul.burton@mips.com,
"anna.schumaker@netapp.com
In-Reply-To: <CACT4Y+aC45BtS88DXarn3A+LV2RRRsPQoSs_3_DnKjU4O3AMHQ@mail.gmail.com>
On Thu, Nov 01, 2018 at 06:46:50PM +0100, Dmitry Vyukov wrote:
> If there is a warning that we don't want to see at all, then we can
> disable it. It supposed to be a useful tool, rather than a thing in
> itself that lives own life. We already I think removed 1 particularly
> noisy warning and made another optional via a config.
> But the thing with overflows is that, even if it's defined, it's not
> necessary the intended behavior. For example, take allocation size
> calculation done via unsigned size_t. If it overflows it does not help
> if C defines result or not, it still gives a user controlled write
> primitive. We've seen similar cases with timeout/deadline calculation
> in kernel, we really don't want it to just wrap modulo-2, right. Some
> user-space projects even test with unsigned overflow warnings or
> implicit truncation warnings, which are formally legal, but frequently
> bugs.
Sure; but then don't call it UB.
If we want to have an additional integer over/underflow checker (ideally
with a gcc plugin that has explicit annotations like __wrap to make it
go away) that is fine; and it can be done on unsigned and signed.
^ permalink raw reply
* Re: [PATCH net] rtnetlink: invoke 'cb->done' destructor before 'cb->args' reset
From: Alexey Kodanev @ 2018-11-01 13:11 UTC (permalink / raw)
To: David Ahern, netdev; +Cc: David Miller
In-Reply-To: <50d88abb-bfba-6c70-af68-7bac60a0d4b1@gmail.com>
On 10/31/2018 08:35 PM, David Ahern wrote:
> On 10/31/18 10:55 AM, David Ahern wrote:
>> I think the simplest fix for 4.20 is to break the loop if ret is non-0 -
>> restore the previous behavior.
>
> that is the only recourse. It has to bail if ret is non-0. Do you want
> to send a patch with that fix?
>
I see, and inet6_dump_fib() cleanups fib6_walker if ret is zero. Will send the fix.
^ permalink raw reply
* RE: [PATCH iproute2-next] rdma: Refresh help section of resource information
From: Steve Wise @ 2018-11-01 13:22 UTC (permalink / raw)
To: 'Leon Romanovsky', 'David Ahern'
Cc: 'Leon Romanovsky', 'netdev',
'RDMA mailing list', 'Stephen Hemminger'
In-Reply-To: <20181101083505.25152-1-leon@kernel.org>
> -----Original Message-----
> From: Leon Romanovsky <leon@kernel.org>
> Sent: Thursday, November 1, 2018 3:35 AM
> To: David Ahern <dsahern@gmail.com>
> Cc: Leon Romanovsky <leonro@mellanox.com>; netdev
> <netdev@vger.kernel.org>; RDMA mailing list <linux-rdma@vger.kernel.org>;
> Stephen Hemminger <stephen@networkplumber.org>; Steve Wise
> <swise@opengridcomputing.com>
> Subject: [PATCH iproute2-next] rdma: Refresh help section of resource
> information
>
> From: Leon Romanovsky <leonro@mellanox.com>
>
> After commit 4060e4c0d257 ("rdma: Add PD resource tracking
> information"), the resource information shows PDs and MRs,
> but help pages didn't fully reflect it.
>
> Signed-off-by: Leon Romanovsky <leonro@mellanox.com>
Oops. Thanks. Looks fine.
Reviewed-by: Steve Wise <swise@opengridcomputing.com>
^ permalink raw reply
* Re: [PATCH net] rtnetlink: invoke 'cb->done' destructor before 'cb->args' reset
From: Alexey Kodanev @ 2018-11-01 13:42 UTC (permalink / raw)
To: David Ahern, netdev; +Cc: David Miller
In-Reply-To: <dbd48fde-3a7f-2800-4317-4eab2ecfd41d@oracle.com>
On 11/01/2018 04:11 PM, Alexey Kodanev wrote:
> On 10/31/2018 08:35 PM, David Ahern wrote:
>> On 10/31/18 10:55 AM, David Ahern wrote:
>>> I think the simplest fix for 4.20 is to break the loop if ret is non-0 -
>>> restore the previous behavior.
>>
>> that is the only recourse. It has to bail if ret is non-0. Do you want
>> to send a patch with that fix?
>>
>
> I see, and inet6_dump_fib() cleanups fib6_walker if ret is zero. Will send the fix.
Can it happen that inet6_dump_fib() returns skb->len (0) in the below cases?
* if (arg.filter.flags & RTM_F_CLONED)
return skb->len;
...
w = (void *)cb->args[2];
if (!w) {
...
w = kzalloc(...)
...
* if (arg.filter.table_id) {
...
if (!tb) {
if (arg.filter.dump_all_families)
return skb->len;
Would it be safer to add "res = skb->len; goto out;" instead of "return skb->len;"
so that it can call fib6_dump_end() for "res <= 0"? Or use cb->data instead of
cb->args?
^ permalink raw reply
* Re: [PATCH v1 net] net: dsa: microchip: initialize mutex before use
From: Pavel Machek @ 2018-11-01 13:47 UTC (permalink / raw)
To: Andrew Lunn
Cc: Tristram.Ha, David S. Miller, Florian Fainelli, UNGLinuxDriver,
netdev
In-Reply-To: <20181101121719.GA7652@lunn.ch>
[-- Attachment #1: Type: text/plain, Size: 634 bytes --]
On Thu 2018-11-01 13:17:19, Andrew Lunn wrote:
> On Wed, Oct 31, 2018 at 07:49:08PM -0700, Tristram.Ha@microchip.com wrote:
> > From: Tristram Ha <Tristram.Ha@microchip.com>
> >
> > Initialize mutex before use. Avoid kernel complaint when
> > CONFIG_DEBUG_LOCK_ALLOC is enabled.
> >
> > Fixes: b987e98e50ab90e5 ("dsa: add DSA switch driver for Microchip KSZ9477")
> > Signed-off-by: Tristram Ha <Tristram.Ha@microchip.com>
Reviewed-by: Pavel Machek <pavel@ucw.cz>
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: [PATCH net-next 5/6] net/ncsi: Reset channel state in ncsi_start_dev()
From: Samuel Mendoza-Jonas @ 2018-11-01 22:53 UTC (permalink / raw)
To: Justin.Lee1, netdev; +Cc: davem, linux-kernel, openbmc
In-Reply-To: <96158616f0774e3d9fae0e99bb16e605@AUSX13MPS306.AMER.DELL.COM>
On Tue, 2018-10-30 at 21:26 +0000, Justin.Lee1@Dell.com wrote:
> > +int ncsi_reset_dev(struct ncsi_dev *nd)
> > +{
> > + struct ncsi_dev_priv *ndp = TO_NCSI_DEV_PRIV(nd);
> > + struct ncsi_channel *nc, *active;
> > + struct ncsi_package *np;
> > + unsigned long flags;
> > + bool enabled;
> > + int state;
> > +
> > + active = NULL;
> > + NCSI_FOR_EACH_PACKAGE(ndp, np) {
> > + NCSI_FOR_EACH_CHANNEL(np, nc) {
> > + spin_lock_irqsave(&nc->lock, flags);
> > + enabled = nc->monitor.enabled;
> > + state = nc->state;
> > + spin_unlock_irqrestore(&nc->lock, flags);
> > +
> > + if (enabled)
> > + ncsi_stop_channel_monitor(nc);
> > + if (state == NCSI_CHANNEL_ACTIVE) {
> > + active = nc;
> > + break;
>
> Is the original intention to process the channel one by one?
> If it is the case, there are two loops and we might need to use
> "goto found" instead.
Yes we'll need to break out of the package loop here as well.
>
> > + }
> > + }
> > + }
> > +
>
> found: ?
>
> > + if (!active) {
> > + /* Done */
> > + spin_lock_irqsave(&ndp->lock, flags);
> > + ndp->flags &= ~NCSI_DEV_RESET;
> > + spin_unlock_irqrestore(&ndp->lock, flags);
> > + return ncsi_choose_active_channel(ndp);
> > + }
> > +
> > + spin_lock_irqsave(&ndp->lock, flags);
> > + ndp->flags |= NCSI_DEV_RESET;
> > + ndp->active_channel = active;
> > + ndp->active_package = active->package;
> > + spin_unlock_irqrestore(&ndp->lock, flags);
> > +
> > + nd->state = ncsi_dev_state_suspend;
> > + schedule_work(&ndp->work);
> > + return 0;
> > +}
>
> Also similar issue in ncsi_choose_active_channel() function below.
>
> > @@ -916,32 +1045,49 @@ static int ncsi_choose_active_channel(struct ncsi_dev_priv *ndp)
> >
> > ncm = &nc->modes[NCSI_MODE_LINK];
> > if (ncm->data[2] & 0x1) {
> > - spin_unlock_irqrestore(&nc->lock, flags);
> > found = nc;
> > - goto out;
> > + with_link = true;
> > }
> >
> > - spin_unlock_irqrestore(&nc->lock, flags);
> > + /* If multi_channel is enabled configure all valid
> > + * channels whether or not they currently have link
> > + * so they will have AENs enabled.
> > + */
> > + if (with_link || np->multi_channel) {
>
> I notice that there is a case that we will misconfigure the interface.
> For example below, multi-channel is not enable for package 1.
> But we enable the channel for ncsi2 below (package 1 channel 0) as that interface is the first
> channel for that package with link.
I don't think I see the issue here; multi-channel is not set on package
1, but both channels are in the channel whitelist. Channel 0 is
configured since it's the first found on package 1, and channel 1 is not
since channel 0 is already found. Are you expecting something different?
>
> cat /sys/kernel/debug/ncsi_protocol/ncsi_device_
> IFIDX IFNAME NAME PID CID RX TX MP MC WP WC PC CS PS LS RU CR NQ HA
> =====================================================================
> 2 eth2 ncsi0 000 000 1 1 1 1 1 1 0 2 1 1 1 1 0 1
> 2 eth2 ncsi1 000 001 1 0 1 1 1 1 0 2 1 1 1 1 0 1
> 2 eth2 ncsi2 001 000 1 0 1 0 1 1 0 2 1 1 1 1 0 1
> 2 eth2 ncsi3 001 001 0 0 1 0 1 1 0 1 0 1 1 1 0 1
> =====================================================================
> MP: Multi-mode Package WP: Whitelist Package
> MC: Multi-mode Channel WC: Whitelist Channel
> PC: Primary Channel CS: Channel State IA/A/IV 1/2/3
> PS: Poll Status LS: Link Status
> RU: Running CR: Carrier OK
> NQ: Queue Stopped HA: Hardware Arbitration
>
> I temporally change to the following to avoid that.
> if ((with_link &&
> !np->multi_channel &&
> list_empty(&ndp->channel_queue)) || np->multi_channel) {
>
> > + spin_lock_irqsave(&ndp->lock, flags);
> > + list_add_tail_rcu(&nc->link,
> > + &ndp->channel_queue);
> > + spin_unlock_irqrestore(&ndp->lock, flags);
> > +
> > + netdev_dbg(ndp->ndev.dev,
> > + "NCSI: Channel %u added to queue (link %s)\n",
> > + nc->id,
> > + ncm->data[2] & 0x1 ? "up" : "down");
> > + }
> > +
> > + spin_unlock_irqrestore(&nc->lock, cflags);
> > +
> > + if (with_link && !np->multi_channel)
> > + break;
>
> Similar issue here. As we are using break, so each package will configure one active TX.
>
I believe this is handled properly in ncsi_channel_is_tx() in the most
recent revision.
> > }
> > + if (with_link && !ndp->multi_package)
> > + break;
> > }
> >
> > - if (!found) {
> > + if (list_empty(&ndp->channel_queue) && found) {
> > + netdev_info(ndp->ndev.dev,
> > + "NCSI: No channel with link found, configuring channel %u\n",
> > + found->id);
> > + spin_lock_irqsave(&ndp->lock, flags);
> > + list_add_tail_rcu(&found->link, &ndp->channel_queue);
> > + spin_unlock_irqrestore(&ndp->lock, flags);
> > + } else if (!found) {
> > netdev_warn(ndp->ndev.dev,
> > - "NCSI: No channel found with link\n");
> > + "NCSI: No channel found to configure!\n");
> > ncsi_report_link(ndp, true);
> > return -ENODEV;
> > }
>
> Also, for deselect package handler function, do we want to set to inactive here?
> If we just change the state, the cached data still keeps the old value. If the new
> ncsi_reset_dev() function is handling one by one, can we skip this part?
Technically yes we could skip the state change here since
ncsi_reset_dev() will have already done it. However if we send a DP
command via some other means then it is probably best to ensure we treat
all channels on that package as inactive.
>
> static int ncsi_rsp_handler_dp(struct ncsi_request *nr)
> {
> struct ncsi_rsp_pkt *rsp;
> struct ncsi_dev_priv *ndp = nr->ndp;
> struct ncsi_package *np;
> struct ncsi_channel *nc;
> unsigned long flags;
>
> /* Find the package */
> rsp = (struct ncsi_rsp_pkt *)skb_network_header(nr->rsp);
> ncsi_find_package_and_channel(ndp, rsp->rsp.common.channel,
> &np, NULL);
> if (!np)
> return -ENODEV;
>
> /* Change state of all channels attached to the package */
> NCSI_FOR_EACH_CHANNEL(np, nc) {
> spin_lock_irqsave(&nc->lock, flags);
> nc->state = NCSI_CHANNEL_INACTIVE;
>
> spin_unlock_irqrestore(&nc->lock, flags);
> }
>
> return 0;
> }
>
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox