* Re: [PULL] vhost: cleanups and fixes
From: Linus Torvalds @ 2018-11-01 21:44 UTC (permalink / raw)
To: mst
Cc: kvm, virtualization, netdev, Linux Kernel Mailing List,
Andrew Morton, bijan.mottahedeh, gedwards, joe, lenaic,
liang.z.li, mhocko, mhocko, stefanha, wei.w.wang
In-Reply-To: <20181101171938-mutt-send-email-mst@kernel.org>
On Thu, Nov 1, 2018 at 2:19 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> virtio, vhost: fixes, tweaks
Pulled.
Linus
^ permalink raw reply
* Re: [PATCH iproute2 net-next 0/3] ss: Allow selection of columns to be displayed
From: Stephen Hemminger @ 2018-11-01 21:38 UTC (permalink / raw)
To: David Ahern; +Cc: Jakub Kicinski, Stefano Brivio, Yoann P., netdev
In-Reply-To: <b3a7235f-3272-309e-6b8c-38dd0b7a0556@gmail.com>
On Thu, 1 Nov 2018 15:18:03 -0600
David Ahern <dsahern@gmail.com> wrote:
> On 11/1/18 3:06 PM, Jakub Kicinski wrote:
> > On Wed, 31 Oct 2018 20:48:05 -0600, David Ahern wrote:
> >>> spacing with a special character in the format string, that is:
> >>>
> >>> "%S.%Qr.%Qs %Al:%Pl %Ar:%Pr %p\n"
> >>>
> >>> would mean "align everything to the right, distribute remaining
> >>> whitespace between %S, %Qr and %Qs". But it looks rather complicated
> >>> at a glance.
> >>>
> >>
> >> My concern here is that once this goes in for 1 command, the others in
> >> iproute2 need to follow suit - meaning same syntax style for all
> >> commands. Given that I'd prefer we get a reasonable consensus on syntax
> >> that will work across commands -- ss, ip, tc. If it is as simple as
> >> column names with a fixed order, that is fine but just give proper
> >> consideration given the impact.
> >
> > FWIW I just started piping iproute2 commands to jq. Example:
> >
> > tc -s -j qdisc show dev em1 | \
> > jq -r '.[] | [.kind,.parent,.handle,.offloaded,.bytes,.packets,.drops,.overlimits,.requeues,.backlog,.qlen,.marked] | @tsv'
> >
> > JSONification would probably be quite an undertaking for ss :(
> >
>
> Right, that is used in some of the scripts under
> tools/testing/selftests. I would put that in the 'heavyweight solution'
> category.
>
> A number of key commands offer the capability to control the output via
> command line argument (e.g., ps, perf script). Given the amount of data
> iproute2 commands throw at a user by default, it would be a good
> usability feature to allow a user to customize the output without having
> to pipe it into other commands.
I would rather see ss grow json support than having to make the output
formatting of every iproute2 command grow a new format management.
The jq tool looks cool, and I can see how someone could easily have
a bunch of mini-scripts to do what they want.
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-01 21:34 UTC (permalink / raw)
To: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <41525a83-9e8a-52e1-d536-d383399c795c@itcare.pl>
W dniu 01.11.2018 o 22:24, Paweł Staszewski pisze:
>
>
> W dniu 01.11.2018 o 22:18, Paweł Staszewski pisze:
>>
>>
>> W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
>>> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>>>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>>>> Hi
>>>>>>
>>>>>> So maybee someone will be interested how linux kernel handles
>>>>>> normal
>>>>>> traffic (not pktgen :) )
>>>>>>
>>>>>>
>>>>>> Server HW configuration:
>>>>>>
>>>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>>>
>>>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>>>
>>>>>>
>>>>>> Server software:
>>>>>>
>>>>>> FRR - as routing daemon
>>>>>>
>>>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>>>> local
>>>>>> numa
>>>>>> node)
>>>>>>
>>>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>>>> numa
>>>>>> node)
>>>>>>
>>>>>>
>>>>>> Maximum traffic that server can handle:
>>>>>>
>>>>>> Bandwidth
>>>>>>
>>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>> input: /proc/net/dev type: rate
>>>>>> \ iface Rx Tx Total
>>>>>> =================================================================
>>>>>> ====
>>>>>> =========
>>>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>>>> Gb/s
>>>>>> 65.74 Gb/s
>>>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>>>> Gb/s
>>>>>> 66.51 Gb/s
>>>>>> ---------------------------------------------------------------
>>>>>> ----
>>>>>> -----------
>>>>>> total: 66.58 Gb/s 65.67
>>>>>> Gb/s
>>>>>> 132.25 Gb/s
>>>>>>
>>>>>>
>>>>>> Packets per second:
>>>>>>
>>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>> input: /proc/net/dev type: rate
>>>>>> - iface Rx Tx Total
>>>>>> =================================================================
>>>>>> ====
>>>>>> =========
>>>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>>>> 8735207.00 P/s
>>>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>>>> 8790460.00 P/s
>>>>>> ---------------------------------------------------------------
>>>>>> ----
>>>>>> -----------
>>>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>>>> 17525668.00 P/s
>>>>>>
>>>>>>
>>>>>> After reaching that limits nics on the upstream side (more RX
>>>>>> traffic)
>>>>>> start to drop packets
>>>>>>
>>>>>>
>>>>>> I just dont understand that server can't handle more bandwidth
>>>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>>>> RX
>>>>>> side are increasing.
>>>>>>
>>>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>>>> NIC (
>>>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>>>> 132.25
>>>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>>>> hmm yes that was my concern also - cause cant find anywhere
>>>> informations
>>>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>>>> 8GT
>>>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>>>> bw
>>>> on both ports
>>> i think it is bidir
>>>
>>>> This can explain maybee also why cpuload is rising rapidly from
>>>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>>>> so
>>>> there can be some error in reading them when offloading (gro/gso/tso)
>>>> on
>>>> nic's is enabled that is why
>>>>
>>>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>>>> 8GT
>>>>>> is
>>>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>>>> and
>>>>>> pps
>>>>>> (like 4x more comparing to normal internet traffic)
>>>>>>
>>>>> Are you forwarding when using pktgen as well or you just testing
>>>>> the RX
>>>>> side pps ?
>>>> Yes pktgen was tested on single port RX
>>>> Can check also forwarding to eliminate pciex limits
>>>>
>>> So this explains why you have more RX pps, since tx is idle and pcie
>>> will be free to do only rx.
>>>
>>> [...]
>>>
>>>
>>>>>> ethtool -S enp175s0f1
>>>>>> NIC statistics:
>>>>>> rx_packets: 173730800927
>>>>>> rx_bytes: 99827422751332
>>>>>> tx_packets: 142532009512
>>>>>> tx_bytes: 184633045911222
>>>>>> tx_tso_packets: 25989113891
>>>>>> tx_tso_bytes: 132933363384458
>>>>>> tx_tso_inner_packets: 0
>>>>>> tx_tso_inner_bytes: 0
>>>>>> tx_added_vlan_packets: 74630239613
>>>>>> tx_nop: 2029817748
>>>>>> rx_lro_packets: 0
>>>>>> rx_lro_bytes: 0
>>>>>> rx_ecn_mark: 0
>>>>>> rx_removed_vlan_packets: 173730800927
>>>>>> rx_csum_unnecessary: 0
>>>>>> rx_csum_none: 434357
>>>>>> rx_csum_complete: 173730366570
>>>>>> rx_csum_unnecessary_inner: 0
>>>>>> rx_xdp_drop: 0
>>>>>> rx_xdp_redirect: 0
>>>>>> rx_xdp_tx_xmit: 0
>>>>>> rx_xdp_tx_full: 0
>>>>>> rx_xdp_tx_err: 0
>>>>>> rx_xdp_tx_cqe: 0
>>>>>> tx_csum_none: 38260960853
>>>>>> tx_csum_partial: 36369278774
>>>>>> tx_csum_partial_inner: 0
>>>>>> tx_queue_stopped: 1
>>>>>> tx_queue_dropped: 0
>>>>>> tx_xmit_more: 748638099
>>>>>> tx_recover: 0
>>>>>> tx_cqes: 73881645031
>>>>>> tx_queue_wake: 1
>>>>>> tx_udp_seg_rem: 0
>>>>>> tx_cqe_err: 0
>>>>>> tx_xdp_xmit: 0
>>>>>> tx_xdp_full: 0
>>>>>> tx_xdp_err: 0
>>>>>> tx_xdp_cqes: 0
>>>>>> rx_wqe_err: 0
>>>>>> rx_mpwqe_filler_cqes: 0
>>>>>> rx_mpwqe_filler_strides: 0
>>>>>> rx_buff_alloc_err: 0
>>>>>> rx_cqe_compress_blks: 0
>>>>>> rx_cqe_compress_pkts: 0
>>>>> If this is a pcie bottleneck it might be useful to enable CQE
>>>>> compression (to reduce PCIe completion descriptors transactions)
>>>>> you should see the above rx_cqe_compress_pkts increasing when
>>>>> enabled.
>>>>>
>>>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>>>> $ ethtool --show-priv-flags enp175s0f1
>>>>> Private flags for p6p1:
>>>>> rx_cqe_moder : on
>>>>> cqe_moder : off
>>>>> rx_cqe_compress : on
>>>>> ...
>>>>>
>>>>> try this on both interfaces.
>>>> Done
>>>> ethtool --show-priv-flags enp175s0f1
>>>> Private flags for enp175s0f1:
>>>> rx_cqe_moder : on
>>>> tx_cqe_moder : off
>>>> rx_cqe_compress : on
>>>> rx_striding_rq : off
>>>> rx_no_csum_complete: off
>>>>
>>>> ethtool --show-priv-flags enp175s0f0
>>>> Private flags for enp175s0f0:
>>>> rx_cqe_moder : on
>>>> tx_cqe_moder : off
>>>> rx_cqe_compress : on
>>>> rx_striding_rq : off
>>>> rx_no_csum_complete: off
>>>>
>>> did it help reduce the load on the pcie ? do you see more pps ?
>>> what is the ratio between rx_cqe_compress_pkts and over all rx packets
>>> ?
>> So - a little more pps
>> Before change top - graph / after bottom -> image with graph stats
>> from proc/net/dev
> Attached link to graph
> https://uploadfiles.io/5vgbh
>
>
Enabling tx_cqe_moder for both ports
helps alot with cpu load at same traffic lvl - the change is about -20%
on all 28 cores
>> cqe_compress enabled at 11:55
>>
>> Sorry - but for real life traffic it is hard to do any counter
>> differences - cause traffic just rising alone from minute to minute :)
>> But for that time the change is visible on graph - cause was almost
>> same for past 20minutes before change.
>>
>>
>> full ethtool below:
>> NIC statistics:
>> rx_packets: 516522465438
>> rx_bytes: 680052911258729
>> tx_packets: 677697545586
>> tx_bytes: 413647643141709
>> tx_tso_packets: 42530913279
>> tx_tso_bytes: 235655668554142
>> tx_tso_inner_packets: 0
>> tx_tso_inner_bytes: 0
>> tx_added_vlan_packets: 551156530885
>> tx_nop: 8536823558
>> rx_lro_packets: 0
>> rx_lro_bytes: 0
>> rx_ecn_mark: 0
>> rx_removed_vlan_packets: 516522465438
>> rx_csum_unnecessary: 0
>> rx_csum_none: 50382868
>> rx_csum_complete: 516472082570
>> rx_csum_unnecessary_inner: 0
>> rx_xdp_drop: 0
>> rx_xdp_redirect: 0
>> rx_xdp_tx_xmit: 0
>> rx_xdp_tx_full: 0
>> rx_xdp_tx_err: 0
>> rx_xdp_tx_cqe: 0
>> tx_csum_none: 494075047017
>> tx_csum_partial: 57081483898
>> tx_csum_partial_inner: 0
>> tx_queue_stopped: 518624
>> tx_queue_dropped: 0
>> tx_xmit_more: 1717880628
>> tx_recover: 0
>> tx_cqes: 549438869029
>> tx_queue_wake: 518627
>> tx_udp_seg_rem: 0
>> tx_cqe_err: 0
>> tx_xdp_xmit: 0
>> tx_xdp_full: 0
>> tx_xdp_err: 0
>> tx_xdp_cqes: 0
>> rx_wqe_err: 0
>> rx_mpwqe_filler_cqes: 0
>> rx_mpwqe_filler_strides: 0
>> rx_buff_alloc_err: 0
>> rx_cqe_compress_blks: 11483228712
>> rx_cqe_compress_pkts: 25794213324
>> rx_page_reuse: 0
>> rx_cache_reuse: 63610249810
>> rx_cache_full: 194650916511
>> rx_cache_empty: 1118208
>> rx_cache_busy: 194650982430
>> rx_cache_waive: 0
>> rx_congst_umr: 0
>> rx_arfs_err: 0
>> ch_events: 119556002196
>> ch_poll: 121107424977
>> ch_arm: 115856746008
>> ch_aff_change: 31
>> ch_eq_rearm: 0
>> rx_out_of_buffer: 6880325
>> rx_if_down_packets: 2062529
>> rx_vport_unicast_packets: 517433716795
>> rx_vport_unicast_bytes: 683464347301443
>> tx_vport_unicast_packets: 677697453738
>> tx_vport_unicast_bytes: 415788589663315
>> rx_vport_multicast_packets: 208258309
>> rx_vport_multicast_bytes: 14224046052
>> tx_vport_multicast_packets: 21689
>> tx_vport_multicast_bytes: 2158334
>> rx_vport_broadcast_packets: 75838646
>> rx_vport_broadcast_bytes: 4697944695
>> tx_vport_broadcast_packets: 68730
>> tx_vport_broadcast_bytes: 4123800
>> rx_vport_rdma_unicast_packets: 0
>> rx_vport_rdma_unicast_bytes: 0
>> tx_vport_rdma_unicast_packets: 0
>> tx_vport_rdma_unicast_bytes: 0
>> rx_vport_rdma_multicast_packets: 0
>> rx_vport_rdma_multicast_bytes: 0
>> tx_vport_rdma_multicast_packets: 0
>> tx_vport_rdma_multicast_bytes: 0
>> tx_packets_phy: 677697543252
>> rx_packets_phy: 521319491878
>> rx_crc_errors_phy: 0
>> tx_bytes_phy: 418499385791411
>> rx_bytes_phy: 690322537017274
>> tx_multicast_phy: 21689
>> tx_broadcast_phy: 68730
>> rx_multicast_phy: 208258305
>> rx_broadcast_phy: 75838646
>> rx_in_range_len_errors_phy: 4
>> rx_out_of_range_len_phy: 0
>> rx_oversize_pkts_phy: 0
>> rx_symbol_err_phy: 0
>> tx_mac_control_phy: 0
>> rx_mac_control_phy: 0
>> rx_unsupported_op_phy: 0
>> rx_pause_ctrl_phy: 0
>> tx_pause_ctrl_phy: 0
>> rx_discards_phy: 3601449265
>> tx_discards_phy: 0
>> tx_errors_phy: 0
>> rx_undersize_pkts_phy: 0
>> rx_fragments_phy: 0
>> rx_jabbers_phy: 0
>> rx_64_bytes_phy: 1416456771
>> rx_65_to_127_bytes_phy: 40750434737
>> rx_128_to_255_bytes_phy: 11518110310
>> rx_256_to_511_bytes_phy: 7055850637
>> rx_512_to_1023_bytes_phy: 7811550424
>> rx_1024_to_1518_bytes_phy: 265547564845
>> rx_1519_to_2047_bytes_phy: 187219522899
>> rx_2048_to_4095_bytes_phy: 0
>> rx_4096_to_8191_bytes_phy: 0
>> rx_8192_to_10239_bytes_phy: 0
>> link_down_events_phy: 0
>> rx_pcs_symbol_err_phy: 0
>> rx_corrected_bits_phy: 0
>> rx_pci_signal_integrity: 0
>> tx_pci_signal_integrity: 48
>> rx_prio0_bytes: 688807632117485
>> rx_prio0_packets: 516310309931
>> tx_prio0_bytes: 418499382756025
>> tx_prio0_packets: 677697534982
>> rx_prio1_bytes: 1497701612877
>> rx_prio1_packets: 1206768094
>> tx_prio1_bytes: 0
>> tx_prio1_packets: 0
>> rx_prio2_bytes: 112271227
>> rx_prio2_packets: 337295
>> tx_prio2_bytes: 0
>> tx_prio2_packets: 0
>> rx_prio3_bytes: 1165455555
>> rx_prio3_packets: 1544310
>> tx_prio3_bytes: 0
>> tx_prio3_packets: 0
>> rx_prio4_bytes: 161857240
>> rx_prio4_packets: 341392
>> tx_prio4_bytes: 0
>> tx_prio4_packets: 0
>> rx_prio5_bytes: 455031612
>> rx_prio5_packets: 2861469
>> tx_prio5_bytes: 0
>> tx_prio5_packets: 0
>> rx_prio6_bytes: 1873928697
>> rx_prio6_packets: 5146981
>> tx_prio6_bytes: 0
>> tx_prio6_packets: 0
>> rx_prio7_bytes: 13423452430
>> rx_prio7_packets: 190724796
>> tx_prio7_bytes: 0
>> tx_prio7_packets: 0
>> module_unplug: 0
>> module_bus_stuck: 0
>> module_high_temp: 0
>> module_bad_shorted: 0
>> ch0_events: 4252266777
>> ch0_poll: 4330804273
>> ch0_arm: 4120233182
>> ch0_aff_change: 2
>> ch0_eq_rearm: 0
>> ch1_events: 3938415938
>> ch1_poll: 4012621322
>> ch1_arm: 3810131188
>> ch1_aff_change: 2
>> ch1_eq_rearm: 0
>> ch2_events: 3897428860
>> ch2_poll: 3973886848
>> ch2_arm: 3773019397
>> ch2_aff_change: 1
>> ch2_eq_rearm: 0
>> ch3_events: 4108000541
>> ch3_poll: 4180139872
>> ch3_arm: 3982093366
>> ch3_aff_change: 1
>> ch3_eq_rearm: 0
>> ch4_events: 4652570079
>> ch4_poll: 4720541090
>> ch4_arm: 4524475054
>> ch4_aff_change: 2
>> ch4_eq_rearm: 0
>> ch5_events: 3899177385
>> ch5_poll: 3974274186
>> ch5_arm: 3772299186
>> ch5_aff_change: 2
>> ch5_eq_rearm: 0
>> ch6_events: 3915161350
>> ch6_poll: 3992338199
>> ch6_arm: 3794710989
>> ch6_aff_change: 0
>> ch6_eq_rearm: 0
>> ch7_events: 4008175631
>> ch7_poll: 4081321248
>> ch7_arm: 3882263723
>> ch7_aff_change: 0
>> ch7_eq_rearm: 0
>> ch8_events: 4207422352
>> ch8_poll: 4276465449
>> ch8_arm: 4077650366
>> ch8_aff_change: 0
>> ch8_eq_rearm: 0
>> ch9_events: 4036491879
>> ch9_poll: 4108975987
>> ch9_arm: 3914493694
>> ch9_aff_change: 0
>> ch9_eq_rearm: 0
>> ch10_events: 4066261595
>> ch10_poll: 4134419606
>> ch10_arm: 3936637711
>> ch10_aff_change: 1
>> ch10_eq_rearm: 0
>> ch11_events: 4440494043
>> ch11_poll: 4507578730
>> ch11_arm: 4318629438
>> ch11_aff_change: 0
>> ch11_eq_rearm: 0
>> ch12_events: 4066958252
>> ch12_poll: 4130191506
>> ch12_arm: 3934337782
>> ch12_aff_change: 0
>> ch12_eq_rearm: 0
>> ch13_events: 4051309159
>> ch13_poll: 4118864120
>> ch13_arm: 3921011919
>> ch13_aff_change: 0
>> ch13_eq_rearm: 0
>> ch14_events: 4321664800
>> ch14_poll: 4382433680
>> ch14_arm: 4186130552
>> ch14_aff_change: 0
>> ch14_eq_rearm: 0
>> ch15_events: 4701102075
>> ch15_poll: 4760373932
>> ch15_arm: 4570151468
>> ch15_aff_change: 0
>> ch15_eq_rearm: 0
>> ch16_events: 4311052687
>> ch16_poll: 4345937129
>> ch16_arm: 4170883819
>> ch16_aff_change: 0
>> ch16_eq_rearm: 0
>> ch17_events: 4647570931
>> ch17_poll: 4680218533
>> ch17_arm: 4509426288
>> ch17_aff_change: 0
>> ch17_eq_rearm: 0
>> ch18_events: 4598195702
>> ch18_poll: 4631314898
>> ch18_arm: 4457267084
>> ch18_aff_change: 0
>> ch18_eq_rearm: 0
>> ch19_events: 4808094560
>> ch19_poll: 4841368340
>> ch19_arm: 4670604358
>> ch19_aff_change: 0
>> ch19_eq_rearm: 0
>> ch20_events: 4240910605
>> ch20_poll: 4276531502
>> ch20_arm: 4101767278
>> ch20_aff_change: 1
>> ch20_eq_rearm: 0
>> ch21_events: 4389371472
>> ch21_poll: 4426870311
>> ch21_arm: 4249339045
>> ch21_aff_change: 2
>> ch21_eq_rearm: 0
>> ch22_events: 4282958754
>> ch22_poll: 4319228073
>> ch22_arm: 4145102991
>> ch22_aff_change: 2
>> ch22_eq_rearm: 0
>> ch23_events: 4440196528
>> ch23_poll: 4474090188
>> ch23_arm: 4300837147
>> ch23_aff_change: 2
>> ch23_eq_rearm: 0
>> ch24_events: 4326875785
>> ch24_poll: 4364971263
>> ch24_arm: 4186404526
>> ch24_aff_change: 2
>> ch24_eq_rearm: 0
>> ch25_events: 4286528453
>> ch25_poll: 4324089445
>> ch25_arm: 4147222616
>> ch25_aff_change: 3
>> ch25_eq_rearm: 0
>> ch26_events: 4098043104
>> ch26_poll: 4138133745
>> ch26_arm: 3967438971
>> ch26_aff_change: 4
>> ch26_eq_rearm: 0
>> ch27_events: 4563302840
>> ch27_poll: 4599441446
>> ch27_arm: 4432182806
>> ch27_aff_change: 4
>> ch27_eq_rearm: 0
>> ch28_events: 4
>> ch28_poll: 4
>> ch28_arm: 4
>> ch28_aff_change: 0
>> ch28_eq_rearm: 0
>> ch29_events: 6
>> ch29_poll: 6
>> ch29_arm: 6
>> ch29_aff_change: 0
>> ch29_eq_rearm: 0
>> ch30_events: 4
>> ch30_poll: 4
>> ch30_arm: 4
>> ch30_aff_change: 0
>> ch30_eq_rearm: 0
>> ch31_events: 4
>> ch31_poll: 4
>> ch31_arm: 4
>> ch31_aff_change: 0
>> ch31_eq_rearm: 0
>> ch32_events: 4
>> ch32_poll: 4
>> ch32_arm: 4
>> ch32_aff_change: 0
>> ch32_eq_rearm: 0
>> ch33_events: 4
>> ch33_poll: 4
>> ch33_arm: 4
>> ch33_aff_change: 0
>> ch33_eq_rearm: 0
>> ch34_events: 4
>> ch34_poll: 4
>> ch34_arm: 4
>> ch34_aff_change: 0
>> ch34_eq_rearm: 0
>> ch35_events: 4
>> ch35_poll: 4
>> ch35_arm: 4
>> ch35_aff_change: 0
>> ch35_eq_rearm: 0
>> ch36_events: 4
>> ch36_poll: 4
>> ch36_arm: 4
>> ch36_aff_change: 0
>> ch36_eq_rearm: 0
>> ch37_events: 4
>> ch37_poll: 4
>> ch37_arm: 4
>> ch37_aff_change: 0
>> ch37_eq_rearm: 0
>> ch38_events: 4
>> ch38_poll: 4
>> ch38_arm: 4
>> ch38_aff_change: 0
>> ch38_eq_rearm: 0
>> ch39_events: 4
>> ch39_poll: 4
>> ch39_arm: 4
>> ch39_aff_change: 0
>> ch39_eq_rearm: 0
>> ch40_events: 4
>> ch40_poll: 4
>> ch40_arm: 4
>> ch40_aff_change: 0
>> ch40_eq_rearm: 0
>> ch41_events: 4
>> ch41_poll: 4
>> ch41_arm: 4
>> ch41_aff_change: 0
>> ch41_eq_rearm: 0
>> ch42_events: 4
>> ch42_poll: 4
>> ch42_arm: 4
>> ch42_aff_change: 0
>> ch42_eq_rearm: 0
>> ch43_events: 4
>> ch43_poll: 4
>> ch43_arm: 4
>> ch43_aff_change: 0
>> ch43_eq_rearm: 0
>> ch44_events: 4
>> ch44_poll: 4
>> ch44_arm: 4
>> ch44_aff_change: 0
>> ch44_eq_rearm: 0
>> ch45_events: 4
>> ch45_poll: 4
>> ch45_arm: 4
>> ch45_aff_change: 0
>> ch45_eq_rearm: 0
>> ch46_events: 4
>> ch46_poll: 4
>> ch46_arm: 4
>> ch46_aff_change: 0
>> ch46_eq_rearm: 0
>> ch47_events: 4
>> ch47_poll: 4
>> ch47_arm: 4
>> ch47_aff_change: 0
>> ch47_eq_rearm: 0
>> ch48_events: 4
>> ch48_poll: 4
>> ch48_arm: 4
>> ch48_aff_change: 0
>> ch48_eq_rearm: 0
>> ch49_events: 4
>> ch49_poll: 4
>> ch49_arm: 4
>> ch49_aff_change: 0
>> ch49_eq_rearm: 0
>> ch50_events: 4
>> ch50_poll: 4
>> ch50_arm: 4
>> ch50_aff_change: 0
>> ch50_eq_rearm: 0
>> ch51_events: 4
>> ch51_poll: 4
>> ch51_arm: 4
>> ch51_aff_change: 0
>> ch51_eq_rearm: 0
>> ch52_events: 4
>> ch52_poll: 4
>> ch52_arm: 4
>> ch52_aff_change: 0
>> ch52_eq_rearm: 0
>> ch53_events: 4
>> ch53_poll: 4
>> ch53_arm: 4
>> ch53_aff_change: 0
>> ch53_eq_rearm: 0
>> ch54_events: 4
>> ch54_poll: 4
>> ch54_arm: 4
>> ch54_aff_change: 0
>> ch54_eq_rearm: 0
>> ch55_events: 4
>> ch55_poll: 4
>> ch55_arm: 4
>> ch55_aff_change: 0
>> ch55_eq_rearm: 0
>> rx0_packets: 21390033774
>> rx0_bytes: 27326856299122
>> rx0_csum_complete: 21339650906
>> rx0_csum_unnecessary: 0
>> rx0_csum_unnecessary_inner: 0
>> rx0_csum_none: 50382868
>> rx0_xdp_drop: 0
>> rx0_xdp_redirect: 0
>> rx0_lro_packets: 0
>> rx0_lro_bytes: 0
>> rx0_ecn_mark: 0
>> rx0_removed_vlan_packets: 21390033774
>> rx0_wqe_err: 0
>> rx0_mpwqe_filler_cqes: 0
>> rx0_mpwqe_filler_strides: 0
>> rx0_buff_alloc_err: 0
>> rx0_cqe_compress_blks: 481077641
>> rx0_cqe_compress_pkts: 1085647489
>> rx0_page_reuse: 0
>> rx0_cache_reuse: 19050049
>> rx0_cache_full: 10675964285
>> rx0_cache_empty: 37376
>> rx0_cache_busy: 10675966819
>> rx0_cache_waive: 0
>> rx0_congst_umr: 0
>> rx0_arfs_err: 0
>> rx0_xdp_tx_xmit: 0
>> rx0_xdp_tx_full: 0
>> rx0_xdp_tx_err: 0
>> rx0_xdp_tx_cqes: 0
>> rx1_packets: 19868919527
>> rx1_bytes: 26149716991561
>> rx1_csum_complete: 19868919527
>> rx1_csum_unnecessary: 0
>> rx1_csum_unnecessary_inner: 0
>> rx1_csum_none: 0
>> rx1_xdp_drop: 0
>> rx1_xdp_redirect: 0
>> rx1_lro_packets: 0
>> rx1_lro_bytes: 0
>> rx1_ecn_mark: 0
>> rx1_removed_vlan_packets: 19868919527
>> rx1_wqe_err: 0
>> rx1_mpwqe_filler_cqes: 0
>> rx1_mpwqe_filler_strides: 0
>> rx1_buff_alloc_err: 0
>> rx1_cqe_compress_blks: 420210560
>> rx1_cqe_compress_pkts: 941233388
>> rx1_page_reuse: 0
>> rx1_cache_reuse: 46200002
>> rx1_cache_full: 9888257242
>> rx1_cache_empty: 37376
>> rx1_cache_busy: 9888259746
>> rx1_cache_waive: 0
>> rx1_congst_umr: 0
>> rx1_arfs_err: 0
>> rx1_xdp_tx_xmit: 0
>> rx1_xdp_tx_full: 0
>> rx1_xdp_tx_err: 0
>> rx1_xdp_tx_cqes: 0
>> rx2_packets: 19575013662
>> rx2_bytes: 25759818417945
>> rx2_csum_complete: 19575013662
>> rx2_csum_unnecessary: 0
>> rx2_csum_unnecessary_inner: 0
>> rx2_csum_none: 0
>> rx2_xdp_drop: 0
>> rx2_xdp_redirect: 0
>> rx2_lro_packets: 0
>> rx2_lro_bytes: 0
>> rx2_ecn_mark: 0
>> rx2_removed_vlan_packets: 19575013662
>> rx2_wqe_err: 0
>> rx2_mpwqe_filler_cqes: 0
>> rx2_mpwqe_filler_strides: 0
>> rx2_buff_alloc_err: 0
>> rx2_cqe_compress_blks: 412345511
>> rx2_cqe_compress_pkts: 923376167
>> rx2_page_reuse: 0
>> rx2_cache_reuse: 38837731
>> rx2_cache_full: 9748666548
>> rx2_cache_empty: 37376
>> rx2_cache_busy: 9748669093
>> rx2_cache_waive: 0
>> rx2_congst_umr: 0
>> rx2_arfs_err: 0
>> rx2_xdp_tx_xmit: 0
>> rx2_xdp_tx_full: 0
>> rx2_xdp_tx_err: 0
>> rx2_xdp_tx_cqes: 0
>> rx3_packets: 19795911749
>> rx3_bytes: 25969475566905
>> rx3_csum_complete: 19795911749
>> rx3_csum_unnecessary: 0
>> rx3_csum_unnecessary_inner: 0
>> rx3_csum_none: 0
>> rx3_xdp_drop: 0
>> rx3_xdp_redirect: 0
>> rx3_lro_packets: 0
>> rx3_lro_bytes: 0
>> rx3_ecn_mark: 0
>> rx3_removed_vlan_packets: 19795911749
>> rx3_wqe_err: 0
>> rx3_mpwqe_filler_cqes: 0
>> rx3_mpwqe_filler_strides: 0
>> rx3_buff_alloc_err: 0
>> rx3_cqe_compress_blks: 416658765
>> rx3_cqe_compress_pkts: 934986266
>> rx3_page_reuse: 0
>> rx3_cache_reuse: 34542124
>> rx3_cache_full: 9863411232
>> rx3_cache_empty: 37376
>> rx3_cache_busy: 9863413732
>> rx3_cache_waive: 0
>> rx3_congst_umr: 0
>> rx3_arfs_err: 0
>> rx3_xdp_tx_xmit: 0
>> rx3_xdp_tx_full: 0
>> rx3_xdp_tx_err: 0
>> rx3_xdp_tx_cqes: 0
>> rx4_packets: 20445652378
>> rx4_bytes: 26949065110265
>> rx4_csum_complete: 20445652378
>> rx4_csum_unnecessary: 0
>> rx4_csum_unnecessary_inner: 0
>> rx4_csum_none: 0
>> rx4_xdp_drop: 0
>> rx4_xdp_redirect: 0
>> rx4_lro_packets: 0
>> rx4_lro_bytes: 0
>> rx4_ecn_mark: 0
>> rx4_removed_vlan_packets: 20445652378
>> rx4_wqe_err: 0
>> rx4_mpwqe_filler_cqes: 0
>> rx4_mpwqe_filler_strides: 0
>> rx4_buff_alloc_err: 0
>> rx4_cqe_compress_blks: 506085858
>> rx4_cqe_compress_pkts: 1147860328
>> rx4_page_reuse: 0
>> rx4_cache_reuse: 10122542864
>> rx4_cache_full: 100281206
>> rx4_cache_empty: 37376
>> rx4_cache_busy: 100283304
>> rx4_cache_waive: 0
>> rx4_congst_umr: 0
>> rx4_arfs_err: 0
>> rx4_xdp_tx_xmit: 0
>> rx4_xdp_tx_full: 0
>> rx4_xdp_tx_err: 0
>> rx4_xdp_tx_cqes: 0
>> rx5_packets: 19622362246
>> rx5_bytes: 25843450982982
>> rx5_csum_complete: 19622362246
>> rx5_csum_unnecessary: 0
>> rx5_csum_unnecessary_inner: 0
>> rx5_csum_none: 0
>> rx5_xdp_drop: 0
>> rx5_xdp_redirect: 0
>> rx5_lro_packets: 0
>> rx5_lro_bytes: 0
>> rx5_ecn_mark: 0
>> rx5_removed_vlan_packets: 19622362246
>> rx5_wqe_err: 0
>> rx5_mpwqe_filler_cqes: 0
>> rx5_mpwqe_filler_strides: 0
>> rx5_buff_alloc_err: 0
>> rx5_cqe_compress_blks: 422840924
>> rx5_cqe_compress_pkts: 948005878
>> rx5_page_reuse: 0
>> rx5_cache_reuse: 31285453
>> rx5_cache_full: 9779893117
>> rx5_cache_empty: 37376
>> rx5_cache_busy: 9779895647
>> rx5_cache_waive: 0
>> rx5_congst_umr: 0
>> rx5_arfs_err: 0
>> rx5_xdp_tx_xmit: 0
>> rx5_xdp_tx_full: 0
>> rx5_xdp_tx_err: 0
>> rx5_xdp_tx_cqes: 0
>> rx6_packets: 19788231278
>> rx6_bytes: 25985783006486
>> rx6_csum_complete: 19788231278
>> rx6_csum_unnecessary: 0
>> rx6_csum_unnecessary_inner: 0
>> rx6_csum_none: 0
>> rx6_xdp_drop: 0
>> rx6_xdp_redirect: 0
>> rx6_lro_packets: 0
>> rx6_lro_bytes: 0
>> rx6_ecn_mark: 0
>> rx6_removed_vlan_packets: 19788231278
>> rx6_wqe_err: 0
>> rx6_mpwqe_filler_cqes: 0
>> rx6_mpwqe_filler_strides: 0
>> rx6_buff_alloc_err: 0
>> rx6_cqe_compress_blks: 418799056
>> rx6_cqe_compress_pkts: 938282685
>> rx6_page_reuse: 0
>> rx6_cache_reuse: 18114793
>> rx6_cache_full: 9875998295
>> rx6_cache_empty: 37376
>> rx6_cache_busy: 9876000831
>> rx6_cache_waive: 0
>> rx6_congst_umr: 0
>> rx6_arfs_err: 0
>> rx6_xdp_tx_xmit: 0
>> rx6_xdp_tx_full: 0
>> rx6_xdp_tx_err: 0
>> rx6_xdp_tx_cqes: 0
>> rx7_packets: 19795759168
>> rx7_bytes: 26085056586860
>> rx7_csum_complete: 19795759168
>> rx7_csum_unnecessary: 0
>> rx7_csum_unnecessary_inner: 0
>> rx7_csum_none: 0
>> rx7_xdp_drop: 0
>> rx7_xdp_redirect: 0
>> rx7_lro_packets: 0
>> rx7_lro_bytes: 0
>> rx7_ecn_mark: 0
>> rx7_removed_vlan_packets: 19795759168
>> rx7_wqe_err: 0
>> rx7_mpwqe_filler_cqes: 0
>> rx7_mpwqe_filler_strides: 0
>> rx7_buff_alloc_err: 0
>> rx7_cqe_compress_blks: 413959224
>> rx7_cqe_compress_pkts: 927675936
>> rx7_page_reuse: 0
>> rx7_cache_reuse: 23902990
>> rx7_cache_full: 9873974042
>> rx7_cache_empty: 37376
>> rx7_cache_busy: 9873976574
>> rx7_cache_waive: 0
>> rx7_congst_umr: 0
>> rx7_arfs_err: 0
>> rx7_xdp_tx_xmit: 0
>> rx7_xdp_tx_full: 0
>> rx7_xdp_tx_err: 0
>> rx7_xdp_tx_cqes: 0
>> rx8_packets: 19963477439
>> rx8_bytes: 26384640501789
>> rx8_csum_complete: 19963477439
>> rx8_csum_unnecessary: 0
>> rx8_csum_unnecessary_inner: 0
>> rx8_csum_none: 0
>> rx8_xdp_drop: 0
>> rx8_xdp_redirect: 0
>> rx8_lro_packets: 0
>> rx8_lro_bytes: 0
>> rx8_ecn_mark: 0
>> rx8_removed_vlan_packets: 19963477439
>> rx8_wqe_err: 0
>> rx8_mpwqe_filler_cqes: 0
>> rx8_mpwqe_filler_strides: 0
>> rx8_buff_alloc_err: 0
>> rx8_cqe_compress_blks: 420422857
>> rx8_cqe_compress_pkts: 942720292
>> rx8_page_reuse: 0
>> rx8_cache_reuse: 88181713
>> rx8_cache_full: 9893554525
>> rx8_cache_empty: 37376
>> rx8_cache_busy: 9893556983
>> rx8_cache_waive: 0
>> rx8_congst_umr: 0
>> rx8_arfs_err: 0
>> rx8_xdp_tx_xmit: 0
>> rx8_xdp_tx_full: 0
>> rx8_xdp_tx_err: 0
>> rx8_xdp_tx_cqes: 0
>> rx9_packets: 19726642138
>> rx9_bytes: 26063924286499
>> rx9_csum_complete: 19726642138
>> rx9_csum_unnecessary: 0
>> rx9_csum_unnecessary_inner: 0
>> rx9_csum_none: 0
>> rx9_xdp_drop: 0
>> rx9_xdp_redirect: 0
>> rx9_lro_packets: 0
>> rx9_lro_bytes: 0
>> rx9_ecn_mark: 0
>> rx9_removed_vlan_packets: 19726642138
>> rx9_wqe_err: 0
>> rx9_mpwqe_filler_cqes: 0
>> rx9_mpwqe_filler_strides: 0
>> rx9_buff_alloc_err: 0
>> rx9_cqe_compress_blks: 424227411
>> rx9_cqe_compress_pkts: 951534873
>> rx9_page_reuse: 0
>> rx9_cache_reuse: 482901440
>> rx9_cache_full: 9380417487
>> rx9_cache_empty: 37376
>> rx9_cache_busy: 9380419608
>> rx9_cache_waive: 0
>> rx9_congst_umr: 0
>> rx9_arfs_err: 0
>> rx9_xdp_tx_xmit: 0
>> rx9_xdp_tx_full: 0
>> rx9_xdp_tx_err: 0
>> rx9_xdp_tx_cqes: 0
>> rx10_packets: 19901229170
>> rx10_bytes: 26300854495044
>> rx10_csum_complete: 19901229170
>> rx10_csum_unnecessary: 0
>> rx10_csum_unnecessary_inner: 0
>> rx10_csum_none: 0
>> rx10_xdp_drop: 0
>> rx10_xdp_redirect: 0
>> rx10_lro_packets: 0
>> rx10_lro_bytes: 0
>> rx10_ecn_mark: 0
>> rx10_removed_vlan_packets: 19901229170
>> rx10_wqe_err: 0
>> rx10_mpwqe_filler_cqes: 0
>> rx10_mpwqe_filler_strides: 0
>> rx10_buff_alloc_err: 0
>> rx10_cqe_compress_blks: 419082938
>> rx10_cqe_compress_pkts: 940791347
>> rx10_page_reuse: 0
>> rx10_cache_reuse: 14896055
>> rx10_cache_full: 9935715977
>> rx10_cache_empty: 37376
>> rx10_cache_busy: 9935718513
>> rx10_cache_waive: 0
>> rx10_congst_umr: 0
>> rx10_arfs_err: 0
>> rx10_xdp_tx_xmit: 0
>> rx10_xdp_tx_full: 0
>> rx10_xdp_tx_err: 0
>> rx10_xdp_tx_cqes: 0
>> rx11_packets: 20352190494
>> rx11_bytes: 26851034425372
>> rx11_csum_complete: 20352190494
>> rx11_csum_unnecessary: 0
>> rx11_csum_unnecessary_inner: 0
>> rx11_csum_none: 0
>> rx11_xdp_drop: 0
>> rx11_xdp_redirect: 0
>> rx11_lro_packets: 0
>> rx11_lro_bytes: 0
>> rx11_ecn_mark: 0
>> rx11_removed_vlan_packets: 20352190494
>> rx11_wqe_err: 0
>> rx11_mpwqe_filler_cqes: 0
>> rx11_mpwqe_filler_strides: 0
>> rx11_buff_alloc_err: 0
>> rx11_cqe_compress_blks: 501992147
>> rx11_cqe_compress_pkts: 1140398610
>> rx11_page_reuse: 0
>> rx11_cache_reuse: 10071721531
>> rx11_cache_full: 104371621
>> rx11_cache_empty: 37376
>> rx11_cache_busy: 104373697
>> rx11_cache_waive: 0
>> rx11_congst_umr: 0
>> rx11_arfs_err: 0
>> rx11_xdp_tx_xmit: 0
>> rx11_xdp_tx_full: 0
>> rx11_xdp_tx_err: 0
>> rx11_xdp_tx_cqes: 0
>> rx12_packets: 19934747149
>> rx12_bytes: 26296478787829
>> rx12_csum_complete: 19934747149
>> rx12_csum_unnecessary: 0
>> rx12_csum_unnecessary_inner: 0
>> rx12_csum_none: 0
>> rx12_xdp_drop: 0
>> rx12_xdp_redirect: 0
>> rx12_lro_packets: 0
>> rx12_lro_bytes: 0
>> rx12_ecn_mark: 0
>> rx12_removed_vlan_packets: 19934747149
>> rx12_wqe_err: 0
>> rx12_mpwqe_filler_cqes: 0
>> rx12_mpwqe_filler_strides: 0
>> rx12_buff_alloc_err: 0
>> rx12_cqe_compress_blks: 443350570
>> rx12_cqe_compress_pkts: 995997220
>> rx12_page_reuse: 0
>> rx12_cache_reuse: 9864934174
>> rx12_cache_full: 102437428
>> rx12_cache_empty: 37376
>> rx12_cache_busy: 102439382
>> rx12_cache_waive: 0
>> rx12_congst_umr: 0
>> rx12_arfs_err: 0
>> rx12_xdp_tx_xmit: 0
>> rx12_xdp_tx_full: 0
>> rx12_xdp_tx_err: 0
>> rx12_xdp_tx_cqes: 0
>> rx13_packets: 19866908096
>> rx13_bytes: 26160931936186
>> rx13_csum_complete: 19866908096
>> rx13_csum_unnecessary: 0
>> rx13_csum_unnecessary_inner: 0
>> rx13_csum_none: 0
>> rx13_xdp_drop: 0
>> rx13_xdp_redirect: 0
>> rx13_lro_packets: 0
>> rx13_lro_bytes: 0
>> rx13_ecn_mark: 0
>> rx13_removed_vlan_packets: 19866908096
>> rx13_wqe_err: 0
>> rx13_mpwqe_filler_cqes: 0
>> rx13_mpwqe_filler_strides: 0
>> rx13_buff_alloc_err: 0
>> rx13_cqe_compress_blks: 413640141
>> rx13_cqe_compress_pkts: 926175066
>> rx13_page_reuse: 0
>> rx13_cache_reuse: 36358610
>> rx13_cache_full: 9897092921
>> rx13_cache_empty: 37376
>> rx13_cache_busy: 9897095422
>> rx13_cache_waive: 0
>> rx13_congst_umr: 0
>> rx13_arfs_err: 0
>> rx13_xdp_tx_xmit: 0
>> rx13_xdp_tx_full: 0
>> rx13_xdp_tx_err: 0
>> rx13_xdp_tx_cqes: 0
>> rx14_packets: 20229035746
>> rx14_bytes: 26655092809172
>> rx14_csum_complete: 20229035746
>> rx14_csum_unnecessary: 0
>> rx14_csum_unnecessary_inner: 0
>> rx14_csum_none: 0
>> rx14_xdp_drop: 0
>> rx14_xdp_redirect: 0
>> rx14_lro_packets: 0
>> rx14_lro_bytes: 0
>> rx14_ecn_mark: 0
>> rx14_removed_vlan_packets: 20229035746
>> rx14_wqe_err: 0
>> rx14_mpwqe_filler_cqes: 0
>> rx14_mpwqe_filler_strides: 0
>> rx14_buff_alloc_err: 0
>> rx14_cqe_compress_blks: 460990337
>> rx14_cqe_compress_pkts: 1041287948
>> rx14_page_reuse: 0
>> rx14_cache_reuse: 25649275
>> rx14_cache_full: 10088866045
>> rx14_cache_empty: 37376
>> rx14_cache_busy: 10088868574
>> rx14_cache_waive: 0
>> rx14_congst_umr: 0
>> rx14_arfs_err: 0
>> rx14_xdp_tx_xmit: 0
>> rx14_xdp_tx_full: 0
>> rx14_xdp_tx_err: 0
>> rx14_xdp_tx_cqes: 0
>> rx15_packets: 20528177154
>> rx15_bytes: 27029263893264
>> rx15_csum_complete: 20528177154
>> rx15_csum_unnecessary: 0
>> rx15_csum_unnecessary_inner: 0
>> rx15_csum_none: 0
>> rx15_xdp_drop: 0
>> rx15_xdp_redirect: 0
>> rx15_lro_packets: 0
>> rx15_lro_bytes: 0
>> rx15_ecn_mark: 0
>> rx15_removed_vlan_packets: 20528177154
>> rx15_wqe_err: 0
>> rx15_mpwqe_filler_cqes: 0
>> rx15_mpwqe_filler_strides: 0
>> rx15_buff_alloc_err: 0
>> rx15_cqe_compress_blks: 476776176
>> rx15_cqe_compress_pkts: 1076153263
>> rx15_page_reuse: 0
>> rx15_cache_reuse: 48426735
>> rx15_cache_full: 10215659289
>> rx15_cache_empty: 37376
>> rx15_cache_busy: 10215661817
>> rx15_cache_waive: 0
>> rx15_congst_umr: 0
>> rx15_arfs_err: 0
>> rx15_xdp_tx_xmit: 0
>> rx15_xdp_tx_full: 0
>> rx15_xdp_tx_err: 0
>> rx15_xdp_tx_cqes: 0
>> rx16_packets: 16104078098
>> rx16_bytes: 21256361789679
>> rx16_csum_complete: 16104078098
>> rx16_csum_unnecessary: 0
>> rx16_csum_unnecessary_inner: 0
>> rx16_csum_none: 0
>> rx16_xdp_drop: 0
>> rx16_xdp_redirect: 0
>> rx16_lro_packets: 0
>> rx16_lro_bytes: 0
>> rx16_ecn_mark: 0
>> rx16_removed_vlan_packets: 16104078098
>> rx16_wqe_err: 0
>> rx16_mpwqe_filler_cqes: 0
>> rx16_mpwqe_filler_strides: 0
>> rx16_buff_alloc_err: 0
>> rx16_cqe_compress_blks: 352082054
>> rx16_cqe_compress_pkts: 787161670
>> rx16_page_reuse: 0
>> rx16_cache_reuse: 25912567
>> rx16_cache_full: 8026124051
>> rx16_cache_empty: 37376
>> rx16_cache_busy: 8026126465
>> rx16_cache_waive: 0
>> rx16_congst_umr: 0
>> rx16_arfs_err: 0
>> rx16_xdp_tx_xmit: 0
>> rx16_xdp_tx_full: 0
>> rx16_xdp_tx_err: 0
>> rx16_xdp_tx_cqes: 0
>> rx17_packets: 16314055017
>> rx17_bytes: 21589139030173
>> rx17_csum_complete: 16314055017
>> rx17_csum_unnecessary: 0
>> rx17_csum_unnecessary_inner: 0
>> rx17_csum_none: 0
>> rx17_xdp_drop: 0
>> rx17_xdp_redirect: 0
>> rx17_lro_packets: 0
>> rx17_lro_bytes: 0
>> rx17_ecn_mark: 0
>> rx17_removed_vlan_packets: 16314055017
>> rx17_wqe_err: 0
>> rx17_mpwqe_filler_cqes: 0
>> rx17_mpwqe_filler_strides: 0
>> rx17_buff_alloc_err: 0
>> rx17_cqe_compress_blks: 387834541
>> rx17_cqe_compress_pkts: 871851081
>> rx17_page_reuse: 0
>> rx17_cache_reuse: 24021313
>> rx17_cache_full: 8133003829
>> rx17_cache_empty: 37376
>> rx17_cache_busy: 8133006175
>> rx17_cache_waive: 0
>> rx17_congst_umr: 0
>> rx17_arfs_err: 0
>> rx17_xdp_tx_xmit: 0
>> rx17_xdp_tx_full: 0
>> rx17_xdp_tx_err: 0
>> rx17_xdp_tx_cqes: 0
>> rx18_packets: 16439016814
>> rx18_bytes: 21648651917475
>> rx18_csum_complete: 16439016814
>> rx18_csum_unnecessary: 0
>> rx18_csum_unnecessary_inner: 0
>> rx18_csum_none: 0
>> rx18_xdp_drop: 0
>> rx18_xdp_redirect: 0
>> rx18_lro_packets: 0
>> rx18_lro_bytes: 0
>> rx18_ecn_mark: 0
>> rx18_removed_vlan_packets: 16439016814
>> rx18_wqe_err: 0
>> rx18_mpwqe_filler_cqes: 0
>> rx18_mpwqe_filler_strides: 0
>> rx18_buff_alloc_err: 0
>> rx18_cqe_compress_blks: 375066666
>> rx18_cqe_compress_pkts: 843563974
>> rx18_page_reuse: 0
>> rx18_cache_reuse: 8151064266
>> rx18_cache_full: 68442025
>> rx18_cache_empty: 37376
>> rx18_cache_busy: 68444122
>> rx18_cache_waive: 0
>> rx18_congst_umr: 0
>> rx18_arfs_err: 0
>> rx18_xdp_tx_xmit: 0
>> rx18_xdp_tx_full: 0
>> rx18_xdp_tx_err: 0
>> rx18_xdp_tx_cqes: 0
>> rx19_packets: 16641223506
>> rx19_bytes: 21964749940935
>> rx19_csum_complete: 16641223506
>> rx19_csum_unnecessary: 0
>> rx19_csum_unnecessary_inner: 0
>> rx19_csum_none: 0
>> rx19_xdp_drop: 0
>> rx19_xdp_redirect: 0
>> rx19_lro_packets: 0
>> rx19_lro_bytes: 0
>> rx19_ecn_mark: 0
>> rx19_removed_vlan_packets: 16641223506
>> rx19_wqe_err: 0
>> rx19_mpwqe_filler_cqes: 0
>> rx19_mpwqe_filler_strides: 0
>> rx19_buff_alloc_err: 0
>> rx19_cqe_compress_blks: 387825932
>> rx19_cqe_compress_pkts: 872266355
>> rx19_page_reuse: 0
>> rx19_cache_reuse: 116433620
>> rx19_cache_full: 8204175954
>> rx19_cache_empty: 37376
>> rx19_cache_busy: 8204178120
>> rx19_cache_waive: 0
>> rx19_congst_umr: 0
>> rx19_arfs_err: 0
>> rx19_xdp_tx_xmit: 0
>> rx19_xdp_tx_full: 0
>> rx19_xdp_tx_err: 0
>> rx19_xdp_tx_cqes: 0
>> rx20_packets: 16206927741
>> rx20_bytes: 21387447038430
>> rx20_csum_complete: 16206927741
>> rx20_csum_unnecessary: 0
>> rx20_csum_unnecessary_inner: 0
>> rx20_csum_none: 0
>> rx20_xdp_drop: 0
>> rx20_xdp_redirect: 0
>> rx20_lro_packets: 0
>> rx20_lro_bytes: 0
>> rx20_ecn_mark: 0
>> rx20_removed_vlan_packets: 16206927741
>> rx20_wqe_err: 0
>> rx20_mpwqe_filler_cqes: 0
>> rx20_mpwqe_filler_strides: 0
>> rx20_buff_alloc_err: 0
>> rx20_cqe_compress_blks: 370144620
>> rx20_cqe_compress_pkts: 829122671
>> rx20_page_reuse: 0
>> rx20_cache_reuse: 8053733744
>> rx20_cache_full: 49728026
>> rx20_cache_empty: 37376
>> rx20_cache_busy: 49730116
>> rx20_cache_waive: 0
>> rx20_congst_umr: 0
>> rx20_arfs_err: 0
>> rx20_xdp_tx_xmit: 0
>> rx20_xdp_tx_full: 0
>> rx20_xdp_tx_err: 0
>> rx20_xdp_tx_cqes: 0
>> rx21_packets: 16562361314
>> rx21_bytes: 21856653284356
>> rx21_csum_complete: 16562361314
>> rx21_csum_unnecessary: 0
>> rx21_csum_unnecessary_inner: 0
>> rx21_csum_none: 0
>> rx21_xdp_drop: 0
>> rx21_xdp_redirect: 0
>> rx21_lro_packets: 0
>> rx21_lro_bytes: 0
>> rx21_ecn_mark: 0
>> rx21_removed_vlan_packets: 16562361314
>> rx21_wqe_err: 0
>> rx21_mpwqe_filler_cqes: 0
>> rx21_mpwqe_filler_strides: 0
>> rx21_buff_alloc_err: 0
>> rx21_cqe_compress_blks: 350790425
>> rx21_cqe_compress_pkts: 783850729
>> rx21_page_reuse: 0
>> rx21_cache_reuse: 28077493
>> rx21_cache_full: 8253100706
>> rx21_cache_empty: 37376
>> rx21_cache_busy: 8253103147
>> rx21_cache_waive: 0
>> rx21_congst_umr: 0
>> rx21_arfs_err: 0
>> rx21_xdp_tx_xmit: 0
>> rx21_xdp_tx_full: 0
>> rx21_xdp_tx_err: 0
>> rx21_xdp_tx_cqes: 0
>> rx22_packets: 16350307571
>> rx22_bytes: 21408575325592
>> rx22_csum_complete: 16350307571
>> rx22_csum_unnecessary: 0
>> rx22_csum_unnecessary_inner: 0
>> rx22_csum_none: 0
>> rx22_xdp_drop: 0
>> rx22_xdp_redirect: 0
>> rx22_lro_packets: 0
>> rx22_lro_bytes: 0
>> rx22_ecn_mark: 0
>> rx22_removed_vlan_packets: 16350307571
>> rx22_wqe_err: 0
>> rx22_mpwqe_filler_cqes: 0
>> rx22_mpwqe_filler_strides: 0
>> rx22_buff_alloc_err: 0
>> rx22_cqe_compress_blks: 353531065
>> rx22_cqe_compress_pkts: 790814415
>> rx22_page_reuse: 0
>> rx22_cache_reuse: 16934343
>> rx22_cache_full: 8158216889
>> rx22_cache_empty: 37376
>> rx22_cache_busy: 8158219417
>> rx22_cache_waive: 0
>> rx22_congst_umr: 0
>> rx22_arfs_err: 0
>> rx22_xdp_tx_xmit: 0
>> rx22_xdp_tx_full: 0
>> rx22_xdp_tx_err: 0
>> rx22_xdp_tx_cqes: 0
>> rx23_packets: 16019811764
>> rx23_bytes: 21137182570985
>> rx23_csum_complete: 16019811764
>> rx23_csum_unnecessary: 0
>> rx23_csum_unnecessary_inner: 0
>> rx23_csum_none: 0
>> rx23_xdp_drop: 0
>> rx23_xdp_redirect: 0
>> rx23_lro_packets: 0
>> rx23_lro_bytes: 0
>> rx23_ecn_mark: 0
>> rx23_removed_vlan_packets: 16019811764
>> rx23_wqe_err: 0
>> rx23_mpwqe_filler_cqes: 0
>> rx23_mpwqe_filler_strides: 0
>> rx23_buff_alloc_err: 0
>> rx23_cqe_compress_blks: 349733033
>> rx23_cqe_compress_pkts: 781248862
>> rx23_page_reuse: 0
>> rx23_cache_reuse: 33422343
>> rx23_cache_full: 7976481152
>> rx23_cache_empty: 37376
>> rx23_cache_busy: 7976483525
>> rx23_cache_waive: 0
>> rx23_congst_umr: 0
>> rx23_arfs_err: 0
>> rx23_xdp_tx_xmit: 0
>> rx23_xdp_tx_full: 0
>> rx23_xdp_tx_err: 0
>> rx23_xdp_tx_cqes: 0
>> rx24_packets: 16212040646
>> rx24_bytes: 21393399325700
>> rx24_csum_complete: 16212040646
>> rx24_csum_unnecessary: 0
>> rx24_csum_unnecessary_inner: 0
>> rx24_csum_none: 0
>> rx24_xdp_drop: 0
>> rx24_xdp_redirect: 0
>> rx24_lro_packets: 0
>> rx24_lro_bytes: 0
>> rx24_ecn_mark: 0
>> rx24_removed_vlan_packets: 16212040646
>> rx24_wqe_err: 0
>> rx24_mpwqe_filler_cqes: 0
>> rx24_mpwqe_filler_strides: 0
>> rx24_buff_alloc_err: 0
>> rx24_cqe_compress_blks: 379833752
>> rx24_cqe_compress_pkts: 852020179
>> rx24_page_reuse: 0
>> rx24_cache_reuse: 8033552512
>> rx24_cache_full: 72465843
>> rx24_cache_empty: 37376
>> rx24_cache_busy: 72467789
>> rx24_cache_waive: 0
>> rx24_congst_umr: 0
>> rx24_arfs_err: 0
>> rx24_xdp_tx_xmit: 0
>> rx24_xdp_tx_full: 0
>> rx24_xdp_tx_err: 0
>> rx24_xdp_tx_cqes: 0
>> rx25_packets: 16412186257
>> rx25_bytes: 21651198388407
>> rx25_csum_complete: 16412186257
>> rx25_csum_unnecessary: 0
>> rx25_csum_unnecessary_inner: 0
>> rx25_csum_none: 0
>> rx25_xdp_drop: 0
>> rx25_xdp_redirect: 0
>> rx25_lro_packets: 0
>> rx25_lro_bytes: 0
>> rx25_ecn_mark: 0
>> rx25_removed_vlan_packets: 16412186257
>> rx25_wqe_err: 0
>> rx25_mpwqe_filler_cqes: 0
>> rx25_mpwqe_filler_strides: 0
>> rx25_buff_alloc_err: 0
>> rx25_cqe_compress_blks: 383979685
>> rx25_cqe_compress_pkts: 861985772
>> rx25_page_reuse: 0
>> rx25_cache_reuse: 8129807841
>> rx25_cache_full: 76283342
>> rx25_cache_empty: 37376
>> rx25_cache_busy: 76285271
>> rx25_cache_waive: 0
>> rx25_congst_umr: 0
>> rx25_arfs_err: 0
>> rx25_xdp_tx_xmit: 0
>> rx25_xdp_tx_full: 0
>> rx25_xdp_tx_err: 0
>> rx25_xdp_tx_cqes: 0
>> rx26_packets: 16304310003
>> rx26_bytes: 21571217538721
>> rx26_csum_complete: 16304310003
>> rx26_csum_unnecessary: 0
>> rx26_csum_unnecessary_inner: 0
>> rx26_csum_none: 0
>> rx26_xdp_drop: 0
>> rx26_xdp_redirect: 0
>> rx26_lro_packets: 0
>> rx26_lro_bytes: 0
>> rx26_ecn_mark: 0
>> rx26_removed_vlan_packets: 16304310003
>> rx26_wqe_err: 0
>> rx26_mpwqe_filler_cqes: 0
>> rx26_mpwqe_filler_strides: 0
>> rx26_buff_alloc_err: 0
>> rx26_cqe_compress_blks: 353314041
>> rx26_cqe_compress_pkts: 788838424
>> rx26_page_reuse: 0
>> rx26_cache_reuse: 19673790
>> rx26_cache_full: 8132478659
>> rx26_cache_empty: 37376
>> rx26_cache_busy: 8132481198
>> rx26_cache_waive: 0
>> rx26_congst_umr: 0
>> rx26_arfs_err: 0
>> rx26_xdp_tx_xmit: 0
>> rx26_xdp_tx_full: 0
>> rx26_xdp_tx_err: 0
>> rx26_xdp_tx_cqes: 0
>> rx27_packets: 16171856079
>> rx27_bytes: 21376891736540
>> rx27_csum_complete: 16171856079
>> rx27_csum_unnecessary: 0
>> rx27_csum_unnecessary_inner: 0
>> rx27_csum_none: 0
>> rx27_xdp_drop: 0
>> rx27_xdp_redirect: 0
>> rx27_lro_packets: 0
>> rx27_lro_bytes: 0
>> rx27_ecn_mark: 0
>> rx27_removed_vlan_packets: 16171856079
>> rx27_wqe_err: 0
>> rx27_mpwqe_filler_cqes: 0
>> rx27_mpwqe_filler_strides: 0
>> rx27_buff_alloc_err: 0
>> rx27_cqe_compress_blks: 386632845
>> rx27_cqe_compress_pkts: 869362576
>> rx27_page_reuse: 0
>> rx27_cache_reuse: 10070560
>> rx27_cache_full: 8075854928
>> rx27_cache_empty: 37376
>> rx27_cache_busy: 8075857468
>> rx27_cache_waive: 0
>> rx27_congst_umr: 0
>> rx27_arfs_err: 0
>> rx27_xdp_tx_xmit: 0
>> rx27_xdp_tx_full: 0
>> rx27_xdp_tx_err: 0
>> rx27_xdp_tx_cqes: 0
>> rx28_packets: 0
>> rx28_bytes: 0
>> rx28_csum_complete: 0
>> rx28_csum_unnecessary: 0
>> rx28_csum_unnecessary_inner: 0
>> rx28_csum_none: 0
>> rx28_xdp_drop: 0
>> rx28_xdp_redirect: 0
>> rx28_lro_packets: 0
>> rx28_lro_bytes: 0
>> rx28_ecn_mark: 0
>> rx28_removed_vlan_packets: 0
>> rx28_wqe_err: 0
>> rx28_mpwqe_filler_cqes: 0
>> rx28_mpwqe_filler_strides: 0
>> rx28_buff_alloc_err: 0
>> rx28_cqe_compress_blks: 0
>> rx28_cqe_compress_pkts: 0
>> rx28_page_reuse: 0
>> rx28_cache_reuse: 0
>> rx28_cache_full: 0
>> rx28_cache_empty: 2560
>> rx28_cache_busy: 0
>> rx28_cache_waive: 0
>> rx28_congst_umr: 0
>> rx28_arfs_err: 0
>> rx28_xdp_tx_xmit: 0
>> rx28_xdp_tx_full: 0
>> rx28_xdp_tx_err: 0
>> rx28_xdp_tx_cqes: 0
>> rx29_packets: 0
>> rx29_bytes: 0
>> rx29_csum_complete: 0
>> rx29_csum_unnecessary: 0
>> rx29_csum_unnecessary_inner: 0
>> rx29_csum_none: 0
>> rx29_xdp_drop: 0
>> rx29_xdp_redirect: 0
>> rx29_lro_packets: 0
>> rx29_lro_bytes: 0
>> rx29_ecn_mark: 0
>> rx29_removed_vlan_packets: 0
>> rx29_wqe_err: 0
>> rx29_mpwqe_filler_cqes: 0
>> rx29_mpwqe_filler_strides: 0
>> rx29_buff_alloc_err: 0
>> rx29_cqe_compress_blks: 0
>> rx29_cqe_compress_pkts: 0
>> rx29_page_reuse: 0
>> rx29_cache_reuse: 0
>> rx29_cache_full: 0
>> rx29_cache_empty: 2560
>> rx29_cache_busy: 0
>> rx29_cache_waive: 0
>> rx29_congst_umr: 0
>> rx29_arfs_err: 0
>> rx29_xdp_tx_xmit: 0
>> rx29_xdp_tx_full: 0
>> rx29_xdp_tx_err: 0
>> rx29_xdp_tx_cqes: 0
>> rx30_packets: 0
>> rx30_bytes: 0
>> rx30_csum_complete: 0
>> rx30_csum_unnecessary: 0
>> rx30_csum_unnecessary_inner: 0
>> rx30_csum_none: 0
>> rx30_xdp_drop: 0
>> rx30_xdp_redirect: 0
>> rx30_lro_packets: 0
>> rx30_lro_bytes: 0
>> rx30_ecn_mark: 0
>> rx30_removed_vlan_packets: 0
>> rx30_wqe_err: 0
>> rx30_mpwqe_filler_cqes: 0
>> rx30_mpwqe_filler_strides: 0
>> rx30_buff_alloc_err: 0
>> rx30_cqe_compress_blks: 0
>> rx30_cqe_compress_pkts: 0
>> rx30_page_reuse: 0
>> rx30_cache_reuse: 0
>> rx30_cache_full: 0
>> rx30_cache_empty: 2560
>> rx30_cache_busy: 0
>> rx30_cache_waive: 0
>> rx30_congst_umr: 0
>> rx30_arfs_err: 0
>> rx30_xdp_tx_xmit: 0
>> rx30_xdp_tx_full: 0
>> rx30_xdp_tx_err: 0
>> rx30_xdp_tx_cqes: 0
>> rx31_packets: 0
>> rx31_bytes: 0
>> rx31_csum_complete: 0
>> rx31_csum_unnecessary: 0
>> rx31_csum_unnecessary_inner: 0
>> rx31_csum_none: 0
>> rx31_xdp_drop: 0
>> rx31_xdp_redirect: 0
>> rx31_lro_packets: 0
>> rx31_lro_bytes: 0
>> rx31_ecn_mark: 0
>> rx31_removed_vlan_packets: 0
>> rx31_wqe_err: 0
>> rx31_mpwqe_filler_cqes: 0
>> rx31_mpwqe_filler_strides: 0
>> rx31_buff_alloc_err: 0
>> rx31_cqe_compress_blks: 0
>> rx31_cqe_compress_pkts: 0
>> rx31_page_reuse: 0
>> rx31_cache_reuse: 0
>> rx31_cache_full: 0
>> rx31_cache_empty: 2560
>> rx31_cache_busy: 0
>> rx31_cache_waive: 0
>> rx31_congst_umr: 0
>> rx31_arfs_err: 0
>> rx31_xdp_tx_xmit: 0
>> rx31_xdp_tx_full: 0
>> rx31_xdp_tx_err: 0
>> rx31_xdp_tx_cqes: 0
>> rx32_packets: 0
>> rx32_bytes: 0
>> rx32_csum_complete: 0
>> rx32_csum_unnecessary: 0
>> rx32_csum_unnecessary_inner: 0
>> rx32_csum_none: 0
>> rx32_xdp_drop: 0
>> rx32_xdp_redirect: 0
>> rx32_lro_packets: 0
>> rx32_lro_bytes: 0
>> rx32_ecn_mark: 0
>> rx32_removed_vlan_packets: 0
>> rx32_wqe_err: 0
>> rx32_mpwqe_filler_cqes: 0
>> rx32_mpwqe_filler_strides: 0
>> rx32_buff_alloc_err: 0
>> rx32_cqe_compress_blks: 0
>> rx32_cqe_compress_pkts: 0
>> rx32_page_reuse: 0
>> rx32_cache_reuse: 0
>> rx32_cache_full: 0
>> rx32_cache_empty: 2560
>> rx32_cache_busy: 0
>> rx32_cache_waive: 0
>> rx32_congst_umr: 0
>> rx32_arfs_err: 0
>> rx32_xdp_tx_xmit: 0
>> rx32_xdp_tx_full: 0
>> rx32_xdp_tx_err: 0
>> rx32_xdp_tx_cqes: 0
>> rx33_packets: 0
>> rx33_bytes: 0
>> rx33_csum_complete: 0
>> rx33_csum_unnecessary: 0
>> rx33_csum_unnecessary_inner: 0
>> rx33_csum_none: 0
>> rx33_xdp_drop: 0
>> rx33_xdp_redirect: 0
>> rx33_lro_packets: 0
>> rx33_lro_bytes: 0
>> rx33_ecn_mark: 0
>> rx33_removed_vlan_packets: 0
>> rx33_wqe_err: 0
>> rx33_mpwqe_filler_cqes: 0
>> rx33_mpwqe_filler_strides: 0
>> rx33_buff_alloc_err: 0
>> rx33_cqe_compress_blks: 0
>> rx33_cqe_compress_pkts: 0
>> rx33_page_reuse: 0
>> rx33_cache_reuse: 0
>> rx33_cache_full: 0
>> rx33_cache_empty: 2560
>> rx33_cache_busy: 0
>> rx33_cache_waive: 0
>> rx33_congst_umr: 0
>> rx33_arfs_err: 0
>> rx33_xdp_tx_xmit: 0
>> rx33_xdp_tx_full: 0
>> rx33_xdp_tx_err: 0
>> rx33_xdp_tx_cqes: 0
>> rx34_packets: 0
>> rx34_bytes: 0
>> rx34_csum_complete: 0
>> rx34_csum_unnecessary: 0
>> rx34_csum_unnecessary_inner: 0
>> rx34_csum_none: 0
>> rx34_xdp_drop: 0
>> rx34_xdp_redirect: 0
>> rx34_lro_packets: 0
>> rx34_lro_bytes: 0
>> rx34_ecn_mark: 0
>> rx34_removed_vlan_packets: 0
>> rx34_wqe_err: 0
>> rx34_mpwqe_filler_cqes: 0
>> rx34_mpwqe_filler_strides: 0
>> rx34_buff_alloc_err: 0
>> rx34_cqe_compress_blks: 0
>> rx34_cqe_compress_pkts: 0
>> rx34_page_reuse: 0
>> rx34_cache_reuse: 0
>> rx34_cache_full: 0
>> rx34_cache_empty: 2560
>> rx34_cache_busy: 0
>> rx34_cache_waive: 0
>> rx34_congst_umr: 0
>> rx34_arfs_err: 0
>> rx34_xdp_tx_xmit: 0
>> rx34_xdp_tx_full: 0
>> rx34_xdp_tx_err: 0
>> rx34_xdp_tx_cqes: 0
>> rx35_packets: 0
>> rx35_bytes: 0
>> rx35_csum_complete: 0
>> rx35_csum_unnecessary: 0
>> rx35_csum_unnecessary_inner: 0
>> rx35_csum_none: 0
>> rx35_xdp_drop: 0
>> rx35_xdp_redirect: 0
>> rx35_lro_packets: 0
>> rx35_lro_bytes: 0
>> rx35_ecn_mark: 0
>> rx35_removed_vlan_packets: 0
>> rx35_wqe_err: 0
>> rx35_mpwqe_filler_cqes: 0
>> rx35_mpwqe_filler_strides: 0
>> rx35_buff_alloc_err: 0
>> rx35_cqe_compress_blks: 0
>> rx35_cqe_compress_pkts: 0
>> rx35_page_reuse: 0
>> rx35_cache_reuse: 0
>> rx35_cache_full: 0
>> rx35_cache_empty: 2560
>> rx35_cache_busy: 0
>> rx35_cache_waive: 0
>> rx35_congst_umr: 0
>> rx35_arfs_err: 0
>> rx35_xdp_tx_xmit: 0
>> rx35_xdp_tx_full: 0
>> rx35_xdp_tx_err: 0
>> rx35_xdp_tx_cqes: 0
>> rx36_packets: 0
>> rx36_bytes: 0
>> rx36_csum_complete: 0
>> rx36_csum_unnecessary: 0
>> rx36_csum_unnecessary_inner: 0
>> rx36_csum_none: 0
>> rx36_xdp_drop: 0
>> rx36_xdp_redirect: 0
>> rx36_lro_packets: 0
>> rx36_lro_bytes: 0
>> rx36_ecn_mark: 0
>> rx36_removed_vlan_packets: 0
>> rx36_wqe_err: 0
>> rx36_mpwqe_filler_cqes: 0
>> rx36_mpwqe_filler_strides: 0
>> rx36_buff_alloc_err: 0
>> rx36_cqe_compress_blks: 0
>> rx36_cqe_compress_pkts: 0
>> rx36_page_reuse: 0
>> rx36_cache_reuse: 0
>> rx36_cache_full: 0
>> rx36_cache_empty: 2560
>> rx36_cache_busy: 0
>> rx36_cache_waive: 0
>> rx36_congst_umr: 0
>> rx36_arfs_err: 0
>> rx36_xdp_tx_xmit: 0
>> rx36_xdp_tx_full: 0
>> rx36_xdp_tx_err: 0
>> rx36_xdp_tx_cqes: 0
>> rx37_packets: 0
>> rx37_bytes: 0
>> rx37_csum_complete: 0
>> rx37_csum_unnecessary: 0
>> rx37_csum_unnecessary_inner: 0
>> rx37_csum_none: 0
>> rx37_xdp_drop: 0
>> rx37_xdp_redirect: 0
>> rx37_lro_packets: 0
>> rx37_lro_bytes: 0
>> rx37_ecn_mark: 0
>> rx37_removed_vlan_packets: 0
>> rx37_wqe_err: 0
>> rx37_mpwqe_filler_cqes: 0
>> rx37_mpwqe_filler_strides: 0
>> rx37_buff_alloc_err: 0
>> rx37_cqe_compress_blks: 0
>> rx37_cqe_compress_pkts: 0
>> rx37_page_reuse: 0
>> rx37_cache_reuse: 0
>> rx37_cache_full: 0
>> rx37_cache_empty: 2560
>> rx37_cache_busy: 0
>> rx37_cache_waive: 0
>> rx37_congst_umr: 0
>> rx37_arfs_err: 0
>> rx37_xdp_tx_xmit: 0
>> rx37_xdp_tx_full: 0
>> rx37_xdp_tx_err: 0
>> rx37_xdp_tx_cqes: 0
>> rx38_packets: 0
>> rx38_bytes: 0
>> rx38_csum_complete: 0
>> rx38_csum_unnecessary: 0
>> rx38_csum_unnecessary_inner: 0
>> rx38_csum_none: 0
>> rx38_xdp_drop: 0
>> rx38_xdp_redirect: 0
>> rx38_lro_packets: 0
>> rx38_lro_bytes: 0
>> rx38_ecn_mark: 0
>> rx38_removed_vlan_packets: 0
>> rx38_wqe_err: 0
>> rx38_mpwqe_filler_cqes: 0
>> rx38_mpwqe_filler_strides: 0
>> rx38_buff_alloc_err: 0
>> rx38_cqe_compress_blks: 0
>> rx38_cqe_compress_pkts: 0
>> rx38_page_reuse: 0
>> rx38_cache_reuse: 0
>> rx38_cache_full: 0
>> rx38_cache_empty: 2560
>> rx38_cache_busy: 0
>> rx38_cache_waive: 0
>> rx38_congst_umr: 0
>> rx38_arfs_err: 0
>> rx38_xdp_tx_xmit: 0
>> rx38_xdp_tx_full: 0
>> rx38_xdp_tx_err: 0
>> rx38_xdp_tx_cqes: 0
>> rx39_packets: 0
>> rx39_bytes: 0
>> rx39_csum_complete: 0
>> rx39_csum_unnecessary: 0
>> rx39_csum_unnecessary_inner: 0
>> rx39_csum_none: 0
>> rx39_xdp_drop: 0
>> rx39_xdp_redirect: 0
>> rx39_lro_packets: 0
>> rx39_lro_bytes: 0
>> rx39_ecn_mark: 0
>> rx39_removed_vlan_packets: 0
>> rx39_wqe_err: 0
>> rx39_mpwqe_filler_cqes: 0
>> rx39_mpwqe_filler_strides: 0
>> rx39_buff_alloc_err: 0
>> rx39_cqe_compress_blks: 0
>> rx39_cqe_compress_pkts: 0
>> rx39_page_reuse: 0
>> rx39_cache_reuse: 0
>> rx39_cache_full: 0
>> rx39_cache_empty: 2560
>> rx39_cache_busy: 0
>> rx39_cache_waive: 0
>> rx39_congst_umr: 0
>> rx39_arfs_err: 0
>> rx39_xdp_tx_xmit: 0
>> rx39_xdp_tx_full: 0
>> rx39_xdp_tx_err: 0
>> rx39_xdp_tx_cqes: 0
>> rx40_packets: 0
>> rx40_bytes: 0
>> rx40_csum_complete: 0
>> rx40_csum_unnecessary: 0
>> rx40_csum_unnecessary_inner: 0
>> rx40_csum_none: 0
>> rx40_xdp_drop: 0
>> rx40_xdp_redirect: 0
>> rx40_lro_packets: 0
>> rx40_lro_bytes: 0
>> rx40_ecn_mark: 0
>> rx40_removed_vlan_packets: 0
>> rx40_wqe_err: 0
>> rx40_mpwqe_filler_cqes: 0
>> rx40_mpwqe_filler_strides: 0
>> rx40_buff_alloc_err: 0
>> rx40_cqe_compress_blks: 0
>> rx40_cqe_compress_pkts: 0
>> rx40_page_reuse: 0
>> rx40_cache_reuse: 0
>> rx40_cache_full: 0
>> rx40_cache_empty: 2560
>> rx40_cache_busy: 0
>> rx40_cache_waive: 0
>> rx40_congst_umr: 0
>> rx40_arfs_err: 0
>> rx40_xdp_tx_xmit: 0
>> rx40_xdp_tx_full: 0
>> rx40_xdp_tx_err: 0
>> rx40_xdp_tx_cqes: 0
>> rx41_packets: 0
>> rx41_bytes: 0
>> rx41_csum_complete: 0
>> rx41_csum_unnecessary: 0
>> rx41_csum_unnecessary_inner: 0
>> rx41_csum_none: 0
>> rx41_xdp_drop: 0
>> rx41_xdp_redirect: 0
>> rx41_lro_packets: 0
>> rx41_lro_bytes: 0
>> rx41_ecn_mark: 0
>> rx41_removed_vlan_packets: 0
>> rx41_wqe_err: 0
>> rx41_mpwqe_filler_cqes: 0
>> rx41_mpwqe_filler_strides: 0
>> rx41_buff_alloc_err: 0
>> rx41_cqe_compress_blks: 0
>> rx41_cqe_compress_pkts: 0
>> rx41_page_reuse: 0
>> rx41_cache_reuse: 0
>> rx41_cache_full: 0
>> rx41_cache_empty: 2560
>> rx41_cache_busy: 0
>> rx41_cache_waive: 0
>> rx41_congst_umr: 0
>> rx41_arfs_err: 0
>> rx41_xdp_tx_xmit: 0
>> rx41_xdp_tx_full: 0
>> rx41_xdp_tx_err: 0
>> rx41_xdp_tx_cqes: 0
>> rx42_packets: 0
>> rx42_bytes: 0
>> rx42_csum_complete: 0
>> rx42_csum_unnecessary: 0
>> rx42_csum_unnecessary_inner: 0
>> rx42_csum_none: 0
>> rx42_xdp_drop: 0
>> rx42_xdp_redirect: 0
>> rx42_lro_packets: 0
>> rx42_lro_bytes: 0
>> rx42_ecn_mark: 0
>> rx42_removed_vlan_packets: 0
>> rx42_wqe_err: 0
>> rx42_mpwqe_filler_cqes: 0
>> rx42_mpwqe_filler_strides: 0
>> rx42_buff_alloc_err: 0
>> rx42_cqe_compress_blks: 0
>> rx42_cqe_compress_pkts: 0
>> rx42_page_reuse: 0
>> rx42_cache_reuse: 0
>> rx42_cache_full: 0
>> rx42_cache_empty: 2560
>> rx42_cache_busy: 0
>> rx42_cache_waive: 0
>> rx42_congst_umr: 0
>> rx42_arfs_err: 0
>> rx42_xdp_tx_xmit: 0
>> rx42_xdp_tx_full: 0
>> rx42_xdp_tx_err: 0
>> rx42_xdp_tx_cqes: 0
>> rx43_packets: 0
>> rx43_bytes: 0
>> rx43_csum_complete: 0
>> rx43_csum_unnecessary: 0
>> rx43_csum_unnecessary_inner: 0
>> rx43_csum_none: 0
>> rx43_xdp_drop: 0
>> rx43_xdp_redirect: 0
>> rx43_lro_packets: 0
>> rx43_lro_bytes: 0
>> rx43_ecn_mark: 0
>> rx43_removed_vlan_packets: 0
>> rx43_wqe_err: 0
>> rx43_mpwqe_filler_cqes: 0
>> rx43_mpwqe_filler_strides: 0
>> rx43_buff_alloc_err: 0
>> rx43_cqe_compress_blks: 0
>> rx43_cqe_compress_pkts: 0
>> rx43_page_reuse: 0
>> rx43_cache_reuse: 0
>> rx43_cache_full: 0
>> rx43_cache_empty: 2560
>> rx43_cache_busy: 0
>> rx43_cache_waive: 0
>> rx43_congst_umr: 0
>> rx43_arfs_err: 0
>> rx43_xdp_tx_xmit: 0
>> rx43_xdp_tx_full: 0
>> rx43_xdp_tx_err: 0
>> rx43_xdp_tx_cqes: 0
>> rx44_packets: 0
>> rx44_bytes: 0
>> rx44_csum_complete: 0
>> rx44_csum_unnecessary: 0
>> rx44_csum_unnecessary_inner: 0
>> rx44_csum_none: 0
>> rx44_xdp_drop: 0
>> rx44_xdp_redirect: 0
>> rx44_lro_packets: 0
>> rx44_lro_bytes: 0
>> rx44_ecn_mark: 0
>> rx44_removed_vlan_packets: 0
>> rx44_wqe_err: 0
>> rx44_mpwqe_filler_cqes: 0
>> rx44_mpwqe_filler_strides: 0
>> rx44_buff_alloc_err: 0
>> rx44_cqe_compress_blks: 0
>> rx44_cqe_compress_pkts: 0
>> rx44_page_reuse: 0
>> rx44_cache_reuse: 0
>> rx44_cache_full: 0
>> rx44_cache_empty: 2560
>> rx44_cache_busy: 0
>> rx44_cache_waive: 0
>> rx44_congst_umr: 0
>> rx44_arfs_err: 0
>> rx44_xdp_tx_xmit: 0
>> rx44_xdp_tx_full: 0
>> rx44_xdp_tx_err: 0
>> rx44_xdp_tx_cqes: 0
>> rx45_packets: 0
>> rx45_bytes: 0
>> rx45_csum_complete: 0
>> rx45_csum_unnecessary: 0
>> rx45_csum_unnecessary_inner: 0
>> rx45_csum_none: 0
>> rx45_xdp_drop: 0
>> rx45_xdp_redirect: 0
>> rx45_lro_packets: 0
>> rx45_lro_bytes: 0
>> rx45_ecn_mark: 0
>> rx45_removed_vlan_packets: 0
>> rx45_wqe_err: 0
>> rx45_mpwqe_filler_cqes: 0
>> rx45_mpwqe_filler_strides: 0
>> rx45_buff_alloc_err: 0
>> rx45_cqe_compress_blks: 0
>> rx45_cqe_compress_pkts: 0
>> rx45_page_reuse: 0
>> rx45_cache_reuse: 0
>> rx45_cache_full: 0
>> rx45_cache_empty: 2560
>> rx45_cache_busy: 0
>> rx45_cache_waive: 0
>> rx45_congst_umr: 0
>> rx45_arfs_err: 0
>> rx45_xdp_tx_xmit: 0
>> rx45_xdp_tx_full: 0
>> rx45_xdp_tx_err: 0
>> rx45_xdp_tx_cqes: 0
>> rx46_packets: 0
>> rx46_bytes: 0
>> rx46_csum_complete: 0
>> rx46_csum_unnecessary: 0
>> rx46_csum_unnecessary_inner: 0
>> rx46_csum_none: 0
>> rx46_xdp_drop: 0
>> rx46_xdp_redirect: 0
>> rx46_lro_packets: 0
>> rx46_lro_bytes: 0
>> rx46_ecn_mark: 0
>> rx46_removed_vlan_packets: 0
>> rx46_wqe_err: 0
>> rx46_mpwqe_filler_cqes: 0
>> rx46_mpwqe_filler_strides: 0
>> rx46_buff_alloc_err: 0
>> rx46_cqe_compress_blks: 0
>> rx46_cqe_compress_pkts: 0
>> rx46_page_reuse: 0
>> rx46_cache_reuse: 0
>> rx46_cache_full: 0
>> rx46_cache_empty: 2560
>> rx46_cache_busy: 0
>> rx46_cache_waive: 0
>> rx46_congst_umr: 0
>> rx46_arfs_err: 0
>> rx46_xdp_tx_xmit: 0
>> rx46_xdp_tx_full: 0
>> rx46_xdp_tx_err: 0
>> rx46_xdp_tx_cqes: 0
>> rx47_packets: 0
>> rx47_bytes: 0
>> rx47_csum_complete: 0
>> rx47_csum_unnecessary: 0
>> rx47_csum_unnecessary_inner: 0
>> rx47_csum_none: 0
>> rx47_xdp_drop: 0
>> rx47_xdp_redirect: 0
>> rx47_lro_packets: 0
>> rx47_lro_bytes: 0
>> rx47_ecn_mark: 0
>> rx47_removed_vlan_packets: 0
>> rx47_wqe_err: 0
>> rx47_mpwqe_filler_cqes: 0
>> rx47_mpwqe_filler_strides: 0
>> rx47_buff_alloc_err: 0
>> rx47_cqe_compress_blks: 0
>> rx47_cqe_compress_pkts: 0
>> rx47_page_reuse: 0
>> rx47_cache_reuse: 0
>> rx47_cache_full: 0
>> rx47_cache_empty: 2560
>> rx47_cache_busy: 0
>> rx47_cache_waive: 0
>> rx47_congst_umr: 0
>> rx47_arfs_err: 0
>> rx47_xdp_tx_xmit: 0
>> rx47_xdp_tx_full: 0
>> rx47_xdp_tx_err: 0
>> rx47_xdp_tx_cqes: 0
>> rx48_packets: 0
>> rx48_bytes: 0
>> rx48_csum_complete: 0
>> rx48_csum_unnecessary: 0
>> rx48_csum_unnecessary_inner: 0
>> rx48_csum_none: 0
>> rx48_xdp_drop: 0
>> rx48_xdp_redirect: 0
>> rx48_lro_packets: 0
>> rx48_lro_bytes: 0
>> rx48_ecn_mark: 0
>> rx48_removed_vlan_packets: 0
>> rx48_wqe_err: 0
>> rx48_mpwqe_filler_cqes: 0
>> rx48_mpwqe_filler_strides: 0
>> rx48_buff_alloc_err: 0
>> rx48_cqe_compress_blks: 0
>> rx48_cqe_compress_pkts: 0
>> rx48_page_reuse: 0
>> rx48_cache_reuse: 0
>> rx48_cache_full: 0
>> rx48_cache_empty: 2560
>> rx48_cache_busy: 0
>> rx48_cache_waive: 0
>> rx48_congst_umr: 0
>> rx48_arfs_err: 0
>> rx48_xdp_tx_xmit: 0
>> rx48_xdp_tx_full: 0
>> rx48_xdp_tx_err: 0
>> rx48_xdp_tx_cqes: 0
>> rx49_packets: 0
>> rx49_bytes: 0
>> rx49_csum_complete: 0
>> rx49_csum_unnecessary: 0
>> rx49_csum_unnecessary_inner: 0
>> rx49_csum_none: 0
>> rx49_xdp_drop: 0
>> rx49_xdp_redirect: 0
>> rx49_lro_packets: 0
>> rx49_lro_bytes: 0
>> rx49_ecn_mark: 0
>> rx49_removed_vlan_packets: 0
>> rx49_wqe_err: 0
>> rx49_mpwqe_filler_cqes: 0
>> rx49_mpwqe_filler_strides: 0
>> rx49_buff_alloc_err: 0
>> rx49_cqe_compress_blks: 0
>> rx49_cqe_compress_pkts: 0
>> rx49_page_reuse: 0
>> rx49_cache_reuse: 0
>> rx49_cache_full: 0
>> rx49_cache_empty: 2560
>> rx49_cache_busy: 0
>> rx49_cache_waive: 0
>> rx49_congst_umr: 0
>> rx49_arfs_err: 0
>> rx49_xdp_tx_xmit: 0
>> rx49_xdp_tx_full: 0
>> rx49_xdp_tx_err: 0
>> rx49_xdp_tx_cqes: 0
>> rx50_packets: 0
>> rx50_bytes: 0
>> rx50_csum_complete: 0
>> rx50_csum_unnecessary: 0
>> rx50_csum_unnecessary_inner: 0
>> rx50_csum_none: 0
>> rx50_xdp_drop: 0
>> rx50_xdp_redirect: 0
>> rx50_lro_packets: 0
>> rx50_lro_bytes: 0
>> rx50_ecn_mark: 0
>> rx50_removed_vlan_packets: 0
>> rx50_wqe_err: 0
>> rx50_mpwqe_filler_cqes: 0
>> rx50_mpwqe_filler_strides: 0
>> rx50_buff_alloc_err: 0
>> rx50_cqe_compress_blks: 0
>> rx50_cqe_compress_pkts: 0
>> rx50_page_reuse: 0
>> rx50_cache_reuse: 0
>> rx50_cache_full: 0
>> rx50_cache_empty: 2560
>> rx50_cache_busy: 0
>> rx50_cache_waive: 0
>> rx50_congst_umr: 0
>> rx50_arfs_err: 0
>> rx50_xdp_tx_xmit: 0
>> rx50_xdp_tx_full: 0
>> rx50_xdp_tx_err: 0
>> rx50_xdp_tx_cqes: 0
>> rx51_packets: 0
>> rx51_bytes: 0
>> rx51_csum_complete: 0
>> rx51_csum_unnecessary: 0
>> rx51_csum_unnecessary_inner: 0
>> rx51_csum_none: 0
>> rx51_xdp_drop: 0
>> rx51_xdp_redirect: 0
>> rx51_lro_packets: 0
>> rx51_lro_bytes: 0
>> rx51_ecn_mark: 0
>> rx51_removed_vlan_packets: 0
>> rx51_wqe_err: 0
>> rx51_mpwqe_filler_cqes: 0
>> rx51_mpwqe_filler_strides: 0
>> rx51_buff_alloc_err: 0
>> rx51_cqe_compress_blks: 0
>> rx51_cqe_compress_pkts: 0
>> rx51_page_reuse: 0
>> rx51_cache_reuse: 0
>> rx51_cache_full: 0
>> rx51_cache_empty: 2560
>> rx51_cache_busy: 0
>> rx51_cache_waive: 0
>> rx51_congst_umr: 0
>> rx51_arfs_err: 0
>> rx51_xdp_tx_xmit: 0
>> rx51_xdp_tx_full: 0
>> rx51_xdp_tx_err: 0
>> rx51_xdp_tx_cqes: 0
>> rx52_packets: 0
>> rx52_bytes: 0
>> rx52_csum_complete: 0
>> rx52_csum_unnecessary: 0
>> rx52_csum_unnecessary_inner: 0
>> rx52_csum_none: 0
>> rx52_xdp_drop: 0
>> rx52_xdp_redirect: 0
>> rx52_lro_packets: 0
>> rx52_lro_bytes: 0
>> rx52_ecn_mark: 0
>> rx52_removed_vlan_packets: 0
>> rx52_wqe_err: 0
>> rx52_mpwqe_filler_cqes: 0
>> rx52_mpwqe_filler_strides: 0
>> rx52_buff_alloc_err: 0
>> rx52_cqe_compress_blks: 0
>> rx52_cqe_compress_pkts: 0
>> rx52_page_reuse: 0
>> rx52_cache_reuse: 0
>> rx52_cache_full: 0
>> rx52_cache_empty: 2560
>> rx52_cache_busy: 0
>> rx52_cache_waive: 0
>> rx52_congst_umr: 0
>> rx52_arfs_err: 0
>> rx52_xdp_tx_xmit: 0
>> rx52_xdp_tx_full: 0
>> rx52_xdp_tx_err: 0
>> rx52_xdp_tx_cqes: 0
>> rx53_packets: 0
>> rx53_bytes: 0
>> rx53_csum_complete: 0
>> rx53_csum_unnecessary: 0
>> rx53_csum_unnecessary_inner: 0
>> rx53_csum_none: 0
>> rx53_xdp_drop: 0
>> rx53_xdp_redirect: 0
>> rx53_lro_packets: 0
>> rx53_lro_bytes: 0
>> rx53_ecn_mark: 0
>> rx53_removed_vlan_packets: 0
>> rx53_wqe_err: 0
>> rx53_mpwqe_filler_cqes: 0
>> rx53_mpwqe_filler_strides: 0
>> rx53_buff_alloc_err: 0
>> rx53_cqe_compress_blks: 0
>> rx53_cqe_compress_pkts: 0
>> rx53_page_reuse: 0
>> rx53_cache_reuse: 0
>> rx53_cache_full: 0
>> rx53_cache_empty: 2560
>> rx53_cache_busy: 0
>> rx53_cache_waive: 0
>> rx53_congst_umr: 0
>> rx53_arfs_err: 0
>> rx53_xdp_tx_xmit: 0
>> rx53_xdp_tx_full: 0
>> rx53_xdp_tx_err: 0
>> rx53_xdp_tx_cqes: 0
>> rx54_packets: 0
>> rx54_bytes: 0
>> rx54_csum_complete: 0
>> rx54_csum_unnecessary: 0
>> rx54_csum_unnecessary_inner: 0
>> rx54_csum_none: 0
>> rx54_xdp_drop: 0
>> rx54_xdp_redirect: 0
>> rx54_lro_packets: 0
>> rx54_lro_bytes: 0
>> rx54_ecn_mark: 0
>> rx54_removed_vlan_packets: 0
>> rx54_wqe_err: 0
>> rx54_mpwqe_filler_cqes: 0
>> rx54_mpwqe_filler_strides: 0
>> rx54_buff_alloc_err: 0
>> rx54_cqe_compress_blks: 0
>> rx54_cqe_compress_pkts: 0
>> rx54_page_reuse: 0
>> rx54_cache_reuse: 0
>> rx54_cache_full: 0
>> rx54_cache_empty: 2560
>> rx54_cache_busy: 0
>> rx54_cache_waive: 0
>> rx54_congst_umr: 0
>> rx54_arfs_err: 0
>> rx54_xdp_tx_xmit: 0
>> rx54_xdp_tx_full: 0
>> rx54_xdp_tx_err: 0
>> rx54_xdp_tx_cqes: 0
>> rx55_packets: 0
>> rx55_bytes: 0
>> rx55_csum_complete: 0
>> rx55_csum_unnecessary: 0
>> rx55_csum_unnecessary_inner: 0
>> rx55_csum_none: 0
>> rx55_xdp_drop: 0
>> rx55_xdp_redirect: 0
>> rx55_lro_packets: 0
>> rx55_lro_bytes: 0
>> rx55_ecn_mark: 0
>> rx55_removed_vlan_packets: 0
>> rx55_wqe_err: 0
>> rx55_mpwqe_filler_cqes: 0
>> rx55_mpwqe_filler_strides: 0
>> rx55_buff_alloc_err: 0
>> rx55_cqe_compress_blks: 0
>> rx55_cqe_compress_pkts: 0
>> rx55_page_reuse: 0
>> rx55_cache_reuse: 0
>> rx55_cache_full: 0
>> rx55_cache_empty: 2560
>> rx55_cache_busy: 0
>> rx55_cache_waive: 0
>> rx55_congst_umr: 0
>> rx55_arfs_err: 0
>> rx55_xdp_tx_xmit: 0
>> rx55_xdp_tx_full: 0
>> rx55_xdp_tx_err: 0
>> rx55_xdp_tx_cqes: 0
>> tx0_packets: 24512439668
>> tx0_bytes: 15287569052791
>> tx0_tso_packets: 1536157106
>> tx0_tso_bytes: 8571753637944
>> tx0_tso_inner_packets: 0
>> tx0_tso_inner_bytes: 0
>> tx0_csum_partial: 2132156117
>> tx0_csum_partial_inner: 0
>> tx0_added_vlan_packets: 19906601448
>> tx0_nop: 308098536
>> tx0_csum_none: 17774445331
>> tx0_stopped: 19625
>> tx0_dropped: 0
>> tx0_xmit_more: 67864870
>> tx0_recover: 0
>> tx0_cqes: 19838744246
>> tx0_wake: 19624
>> tx0_cqe_err: 0
>> tx1_packets: 22598557053
>> tx1_bytes: 13568850145010
>> tx1_tso_packets: 1369529475
>> tx1_tso_bytes: 7661777265382
>> tx1_tso_inner_packets: 0
>> tx1_tso_inner_bytes: 0
>> tx1_csum_partial: 1884639496
>> tx1_csum_partial_inner: 0
>> tx1_added_vlan_packets: 18468333696
>> tx1_nop: 281301783
>> tx1_csum_none: 16583694200
>> tx1_stopped: 19457
>> tx1_dropped: 0
>> tx1_xmit_more: 55170875
>> tx1_recover: 0
>> tx1_cqes: 18413169824
>> tx1_wake: 19455
>> tx1_cqe_err: 0
>> tx2_packets: 22821611433
>> tx2_bytes: 13752535163683
>> tx2_tso_packets: 1396978825
>> tx2_tso_bytes: 7774704508463
>> tx2_tso_inner_packets: 0
>> tx2_tso_inner_bytes: 0
>> tx2_csum_partial: 1897834558
>> tx2_csum_partial_inner: 0
>> tx2_added_vlan_packets: 18641958085
>> tx2_nop: 286934891
>> tx2_csum_none: 16744123527
>> tx2_stopped: 13214
>> tx2_dropped: 0
>> tx2_xmit_more: 61749446
>> tx2_recover: 0
>> tx2_cqes: 18580215654
>> tx2_wake: 13214
>> tx2_cqe_err: 0
>> tx3_packets: 22580809948
>> tx3_bytes: 13730542936609
>> tx3_tso_packets: 1370434579
>> tx3_tso_bytes: 7605636711455
>> tx3_tso_inner_packets: 0
>> tx3_tso_inner_bytes: 0
>> tx3_csum_partial: 1865573748
>> tx3_csum_partial_inner: 0
>> tx3_added_vlan_packets: 18491873644
>> tx3_nop: 281195875
>> tx3_csum_none: 16626299896
>> tx3_stopped: 12542
>> tx3_dropped: 0
>> tx3_xmit_more: 57681647
>> tx3_recover: 0
>> tx3_cqes: 18434198757
>> tx3_wake: 12540
>> tx3_cqe_err: 0
>> tx4_packets: 27801801208
>> tx4_bytes: 17058453171137
>> tx4_tso_packets: 1740500105
>> tx4_tso_bytes: 9474905622036
>> tx4_tso_inner_packets: 0
>> tx4_tso_inner_bytes: 0
>> tx4_csum_partial: 2279225376
>> tx4_csum_partial_inner: 0
>> tx4_added_vlan_packets: 22744081633
>> tx4_nop: 349753979
>> tx4_csum_none: 20464856257
>> tx4_stopped: 14816
>> tx4_dropped: 0
>> tx4_xmit_more: 65469322
>> tx4_recover: 0
>> tx4_cqes: 22678618972
>> tx4_wake: 14816
>> tx4_cqe_err: 0
>> tx5_packets: 25099783024
>> tx5_bytes: 14917740698381
>> tx5_tso_packets: 1512988013
>> tx5_tso_bytes: 8571208921023
>> tx5_tso_inner_packets: 0
>> tx5_tso_inner_bytes: 0
>> tx5_csum_partial: 2078498561
>> tx5_csum_partial_inner: 0
>> tx5_added_vlan_packets: 20465533760
>> tx5_nop: 312614719
>> tx5_csum_none: 18387035199
>> tx5_stopped: 4605
>> tx5_dropped: 0
>> tx5_xmit_more: 64188936
>> tx5_recover: 0
>> tx5_cqes: 20401350718
>> tx5_wake: 4604
>> tx5_cqe_err: 0
>> tx6_packets: 25025504896
>> tx6_bytes: 14908021946070
>> tx6_tso_packets: 1515718977
>> tx6_tso_bytes: 8511442522461
>> tx6_tso_inner_packets: 0
>> tx6_tso_inner_bytes: 0
>> tx6_csum_partial: 2056378610
>> tx6_csum_partial_inner: 0
>> tx6_added_vlan_packets: 20434066400
>> tx6_nop: 310594020
>> tx6_csum_none: 18377687790
>> tx6_stopped: 15234
>> tx6_dropped: 0
>> tx6_xmit_more: 61130422
>> tx6_recover: 0
>> tx6_cqes: 20372943611
>> tx6_wake: 15234
>> tx6_cqe_err: 0
>> tx7_packets: 25457096169
>> tx7_bytes: 15456289446172
>> tx7_tso_packets: 1553342799
>> tx7_tso_bytes: 8764550988105
>> tx7_tso_inner_packets: 0
>> tx7_tso_inner_bytes: 0
>> tx7_csum_partial: 2105765233
>> tx7_csum_partial_inner: 0
>> tx7_added_vlan_packets: 20720382377
>> tx7_nop: 319044853
>> tx7_csum_none: 18614617145
>> tx7_stopped: 18745
>> tx7_dropped: 0
>> tx7_xmit_more: 57050107
>> tx7_recover: 0
>> tx7_cqes: 20663340775
>> tx7_wake: 18746
>> tx7_cqe_err: 0
>> tx8_packets: 25389771649
>> tx8_bytes: 15225503883962
>> tx8_tso_packets: 1563367648
>> tx8_tso_bytes: 8710384514258
>> tx8_tso_inner_packets: 0
>> tx8_tso_inner_bytes: 0
>> tx8_csum_partial: 2106586634
>> tx8_csum_partial_inner: 0
>> tx8_added_vlan_packets: 20704676274
>> tx8_nop: 318149261
>> tx8_csum_none: 18598089640
>> tx8_stopped: 4733
>> tx8_dropped: 0
>> tx8_xmit_more: 61014317
>> tx8_recover: 0
>> tx8_cqes: 20643667301
>> tx8_wake: 4735
>> tx8_cqe_err: 0
>> tx9_packets: 25521500166
>> tx9_bytes: 15302293145755
>> tx9_tso_packets: 1546316697
>> tx9_tso_bytes: 8770688145926
>> tx9_tso_inner_packets: 0
>> tx9_tso_inner_bytes: 0
>> tx9_csum_partial: 2097652880
>> tx9_csum_partial_inner: 0
>> tx9_added_vlan_packets: 20778408432
>> tx9_nop: 318538543
>> tx9_csum_none: 18680755556
>> tx9_stopped: 16118
>> tx9_dropped: 0
>> tx9_xmit_more: 68509728
>> tx9_recover: 0
>> tx9_cqes: 20709906498
>> tx9_wake: 16118
>> tx9_cqe_err: 0
>> tx10_packets: 25451605829
>> tx10_bytes: 15386896170792
>> tx10_tso_packets: 1576473520
>> tx10_tso_bytes: 8880888676383
>> tx10_tso_inner_packets: 0
>> tx10_tso_inner_bytes: 0
>> tx10_csum_partial: 2129796141
>> tx10_csum_partial_inner: 0
>> tx10_added_vlan_packets: 20659622590
>> tx10_nop: 319117433
>> tx10_csum_none: 18529826450
>> tx10_stopped: 20187
>> tx10_dropped: 0
>> tx10_xmit_more: 58892184
>> tx10_recover: 0
>> tx10_cqes: 20600737739
>> tx10_wake: 20188
>> tx10_cqe_err: 0
>> tx11_packets: 27008919793
>> tx11_bytes: 16587719213058
>> tx11_tso_packets: 1734884654
>> tx11_tso_bytes: 9475681471870
>> tx11_tso_inner_packets: 0
>> tx11_tso_inner_bytes: 0
>> tx11_csum_partial: 2296162292
>> tx11_csum_partial_inner: 0
>> tx11_added_vlan_packets: 21943096263
>> tx11_nop: 344188182
>> tx11_csum_none: 19646933971
>> tx11_stopped: 9703
>> tx11_dropped: 0
>> tx11_xmit_more: 66530718
>> tx11_recover: 0
>> tx11_cqes: 21876571667
>> tx11_wake: 9704
>> tx11_cqe_err: 0
>> tx12_packets: 25969493269
>> tx12_bytes: 15980767963416
>> tx12_tso_packets: 1671396456
>> tx12_tso_bytes: 9268973672821
>> tx12_tso_inner_packets: 0
>> tx12_tso_inner_bytes: 0
>> tx12_csum_partial: 2243809182
>> tx12_csum_partial_inner: 0
>> tx12_added_vlan_packets: 20980642456
>> tx12_nop: 330241007
>> tx12_csum_none: 18736833276
>> tx12_stopped: 10341
>> tx12_dropped: 0
>> tx12_xmit_more: 57834100
>> tx12_recover: 0
>> tx12_cqes: 20922815079
>> tx12_wake: 10342
>> tx12_cqe_err: 0
>> tx13_packets: 25332762261
>> tx13_bytes: 15353213283280
>> tx13_tso_packets: 1577433599
>> tx13_tso_bytes: 8785240284281
>> tx13_tso_inner_packets: 0
>> tx13_tso_inner_bytes: 0
>> tx13_csum_partial: 2110640515
>> tx13_csum_partial_inner: 0
>> tx13_added_vlan_packets: 20605670910
>> tx13_nop: 319805741
>> tx13_csum_none: 18495030395
>> tx13_stopped: 7006
>> tx13_dropped: 0
>> tx13_xmit_more: 58314402
>> tx13_recover: 0
>> tx13_cqes: 20547362770
>> tx13_wake: 7008
>> tx13_cqe_err: 0
>> tx14_packets: 26333743548
>> tx14_bytes: 16070719060573
>> tx14_tso_packets: 1677922970
>> tx14_tso_bytes: 9240299765487
>> tx14_tso_inner_packets: 0
>> tx14_tso_inner_bytes: 0
>> tx14_csum_partial: 2215668906
>> tx14_csum_partial_inner: 0
>> tx14_added_vlan_packets: 21384410786
>> tx14_nop: 332734939
>> tx14_csum_none: 19168741880
>> tx14_stopped: 13160
>> tx14_dropped: 0
>> tx14_xmit_more: 57650391
>> tx14_recover: 0
>> tx14_cqes: 21326767783
>> tx14_wake: 13161
>> tx14_cqe_err: 0
>> tx15_packets: 26824968971
>> tx15_bytes: 16687994233452
>> tx15_tso_packets: 1755745052
>> tx15_tso_bytes: 9533814012441
>> tx15_tso_inner_packets: 0
>> tx15_tso_inner_bytes: 0
>> tx15_csum_partial: 2304778064
>> tx15_csum_partial_inner: 0
>> tx15_added_vlan_packets: 21740906107
>> tx15_nop: 344143287
>> tx15_csum_none: 19436128058
>> tx15_stopped: 75
>> tx15_dropped: 0
>> tx15_xmit_more: 63325832
>> tx15_recover: 0
>> tx15_cqes: 21677585345
>> tx15_wake: 74
>> tx15_cqe_err: 0
>> tx16_packets: 24488158946
>> tx16_bytes: 15027415004570
>> tx16_tso_packets: 1559127391
>> tx16_tso_bytes: 8658691917845
>> tx16_tso_inner_packets: 0
>> tx16_tso_inner_bytes: 0
>> tx16_csum_partial: 2075856395
>> tx16_csum_partial_inner: 0
>> tx16_added_vlan_packets: 19835695731
>> tx16_nop: 308464189
>> tx16_csum_none: 17759839340
>> tx16_stopped: 4567
>> tx16_dropped: 0
>> tx16_xmit_more: 62631422
>> tx16_recover: 0
>> tx16_cqes: 19773070012
>> tx16_wake: 4568
>> tx16_cqe_err: 0
>> tx17_packets: 24700413784
>> tx17_bytes: 15216529713715
>> tx17_tso_packets: 1597555108
>> tx17_tso_bytes: 8773728661243
>> tx17_tso_inner_packets: 0
>> tx17_tso_inner_bytes: 0
>> tx17_csum_partial: 2127177297
>> tx17_csum_partial_inner: 0
>> tx17_added_vlan_packets: 20003144561
>> tx17_nop: 313356918
>> tx17_csum_none: 17875967264
>> tx17_stopped: 12572
>> tx17_dropped: 0
>> tx17_xmit_more: 62742980
>> tx17_recover: 0
>> tx17_cqes: 19940407615
>> tx17_wake: 12573
>> tx17_cqe_err: 0
>> tx18_packets: 24887710046
>> tx18_bytes: 15245034034664
>> tx18_tso_packets: 1582550520
>> tx18_tso_bytes: 8782692335483
>> tx18_tso_inner_packets: 0
>> tx18_tso_inner_bytes: 0
>> tx18_csum_partial: 2084514331
>> tx18_csum_partial_inner: 0
>> tx18_added_vlan_packets: 20173879181
>> tx18_nop: 314818702
>> tx18_csum_none: 18089364850
>> tx18_stopped: 21366
>> tx18_dropped: 0
>> tx18_xmit_more: 62485819
>> tx18_recover: 0
>> tx18_cqes: 20111400935
>> tx18_wake: 21366
>> tx18_cqe_err: 0
>> tx19_packets: 24831057648
>> tx19_bytes: 15164663890576
>> tx19_tso_packets: 1599135489
>> tx19_tso_bytes: 8756045449746
>> tx19_tso_inner_packets: 0
>> tx19_tso_inner_bytes: 0
>> tx19_csum_partial: 2119746608
>> tx19_csum_partial_inner: 0
>> tx19_added_vlan_packets: 20143573903
>> tx19_nop: 316966450
>> tx19_csum_none: 18023827295
>> tx19_stopped: 11431
>> tx19_dropped: 0
>> tx19_xmit_more: 57535904
>> tx19_recover: 0
>> tx19_cqes: 20086045325
>> tx19_wake: 11431
>> tx19_cqe_err: 0
>> tx20_packets: 21943735263
>> tx20_bytes: 13528749492187
>> tx20_tso_packets: 1390048103
>> tx20_tso_bytes: 7629058809637
>> tx20_tso_inner_packets: 0
>> tx20_tso_inner_bytes: 0
>> tx20_csum_partial: 1848533941
>> tx20_csum_partial_inner: 0
>> tx20_added_vlan_packets: 17861417651
>> tx20_nop: 276840365
>> tx20_csum_none: 16012883710
>> tx20_stopped: 38457
>> tx20_dropped: 0
>> tx20_xmit_more: 57042753
>> tx20_recover: 0
>> tx20_cqes: 17804384839
>> tx20_wake: 38457
>> tx20_cqe_err: 0
>> tx21_packets: 21476926958
>> tx21_bytes: 13096410597896
>> tx21_tso_packets: 1367724090
>> tx21_tso_bytes: 7568364585127
>> tx21_tso_inner_packets: 0
>> tx21_tso_inner_bytes: 0
>> tx21_csum_partial: 1830570727
>> tx21_csum_partial_inner: 0
>> tx21_added_vlan_packets: 17421087814
>> tx21_nop: 270611519
>> tx21_csum_none: 15590517087
>> tx21_stopped: 31213
>> tx21_dropped: 0
>> tx21_xmit_more: 60305389
>> tx21_recover: 0
>> tx21_cqes: 17360791205
>> tx21_wake: 31213
>> tx21_cqe_err: 0
>> tx22_packets: 21819106444
>> tx22_bytes: 13492871887100
>> tx22_tso_packets: 1387002018
>> tx22_tso_bytes: 7617705727669
>> tx22_tso_inner_packets: 0
>> tx22_tso_inner_bytes: 0
>> tx22_csum_partial: 1853632107
>> tx22_csum_partial_inner: 0
>> tx22_added_vlan_packets: 17743255447
>> tx22_nop: 274820992
>> tx22_csum_none: 15889623340
>> tx22_stopped: 24814
>> tx22_dropped: 0
>> tx22_xmit_more: 60811304
>> tx22_recover: 0
>> tx22_cqes: 17682451111
>> tx22_wake: 24815
>> tx22_cqe_err: 0
>> tx23_packets: 21830455800
>> tx23_bytes: 13427551902532
>> tx23_tso_packets: 1388556038
>> tx23_tso_bytes: 7604040587125
>> tx23_tso_inner_packets: 0
>> tx23_tso_inner_bytes: 0
>> tx23_csum_partial: 1850819694
>> tx23_csum_partial_inner: 0
>> tx23_added_vlan_packets: 17761271122
>> tx23_nop: 275142775
>> tx23_csum_none: 15910451428
>> tx23_stopped: 29899
>> tx23_dropped: 0
>> tx23_xmit_more: 58924909
>> tx23_recover: 0
>> tx23_cqes: 17702355187
>> tx23_wake: 29898
>> tx23_cqe_err: 0
>> tx24_packets: 21961484213
>> tx24_bytes: 13531373062497
>> tx24_tso_packets: 1394697504
>> tx24_tso_bytes: 7663866609308
>> tx24_tso_inner_packets: 0
>> tx24_tso_inner_bytes: 0
>> tx24_csum_partial: 1857072074
>> tx24_csum_partial_inner: 0
>> tx24_added_vlan_packets: 17856887568
>> tx24_nop: 276352855
>> tx24_csum_none: 15999815494
>> tx24_stopped: 33924
>> tx24_dropped: 0
>> tx24_xmit_more: 63992426
>> tx24_recover: 0
>> tx24_cqes: 17792905243
>> tx24_wake: 33923
>> tx24_cqe_err: 0
>> tx25_packets: 21853593838
>> tx25_bytes: 13357487830519
>> tx25_tso_packets: 1398822411
>> tx25_tso_bytes: 7691191518838
>> tx25_tso_inner_packets: 0
>> tx25_tso_inner_bytes: 0
>> tx25_csum_partial: 1869483109
>> tx25_csum_partial_inner: 0
>> tx25_added_vlan_packets: 17734634614
>> tx25_nop: 276327643
>> tx25_csum_none: 15865151505
>> tx25_stopped: 38651
>> tx25_dropped: 0
>> tx25_xmit_more: 56410535
>> tx25_recover: 0
>> tx25_cqes: 17678234537
>> tx25_wake: 38650
>> tx25_cqe_err: 0
>> tx26_packets: 21480261205
>> tx26_bytes: 13148973015935
>> tx26_tso_packets: 1348132284
>> tx26_tso_bytes: 7523489481775
>> tx26_tso_inner_packets: 0
>> tx26_tso_inner_bytes: 0
>> tx26_csum_partial: 1839740745
>> tx26_csum_partial_inner: 0
>> tx26_added_vlan_packets: 17430592911
>> tx26_nop: 270367836
>> tx26_csum_none: 15590852166
>> tx26_stopped: 34044
>> tx26_dropped: 0
>> tx26_xmit_more: 59870114
>> tx26_recover: 0
>> tx26_cqes: 17370736612
>> tx26_wake: 34043
>> tx26_cqe_err: 0
>> tx27_packets: 22694273108
>> tx27_bytes: 14135473431004
>> tx27_tso_packets: 1418371875
>> tx27_tso_bytes: 7784842263038
>> tx27_tso_inner_packets: 0
>> tx27_tso_inner_bytes: 0
>> tx27_csum_partial: 1919170584
>> tx27_csum_partial_inner: 0
>> tx27_added_vlan_packets: 18520826023
>> tx27_nop: 286296272
>> tx27_csum_none: 16601655439
>> tx27_stopped: 38125
>> tx27_dropped: 0
>> tx27_xmit_more: 72749775
>> tx27_recover: 0
>> tx27_cqes: 18448090270
>> tx27_wake: 38127
>> tx27_cqe_err: 0
>> tx28_packets: 0
>> tx28_bytes: 0
>> tx28_tso_packets: 0
>> tx28_tso_bytes: 0
>> tx28_tso_inner_packets: 0
>> tx28_tso_inner_bytes: 0
>> tx28_csum_partial: 0
>> tx28_csum_partial_inner: 0
>> tx28_added_vlan_packets: 0
>> tx28_nop: 0
>> tx28_csum_none: 0
>> tx28_stopped: 0
>> tx28_dropped: 0
>> tx28_xmit_more: 0
>> tx28_recover: 0
>> tx28_cqes: 0
>> tx28_wake: 0
>> tx28_cqe_err: 0
>> tx29_packets: 3
>> tx29_bytes: 266
>> tx29_tso_packets: 0
>> tx29_tso_bytes: 0
>> tx29_tso_inner_packets: 0
>> tx29_tso_inner_bytes: 0
>> tx29_csum_partial: 0
>> tx29_csum_partial_inner: 0
>> tx29_added_vlan_packets: 0
>> tx29_nop: 0
>> tx29_csum_none: 3
>> tx29_stopped: 0
>> tx29_dropped: 0
>> tx29_xmit_more: 1
>> tx29_recover: 0
>> tx29_cqes: 2
>> tx29_wake: 0
>> tx29_cqe_err: 0
>> tx30_packets: 0
>> tx30_bytes: 0
>> tx30_tso_packets: 0
>> tx30_tso_bytes: 0
>> tx30_tso_inner_packets: 0
>> tx30_tso_inner_bytes: 0
>> tx30_csum_partial: 0
>> tx30_csum_partial_inner: 0
>> tx30_added_vlan_packets: 0
>> tx30_nop: 0
>> tx30_csum_none: 0
>> tx30_stopped: 0
>> tx30_dropped: 0
>> tx30_xmit_more: 0
>> tx30_recover: 0
>> tx30_cqes: 0
>> tx30_wake: 0
>> tx30_cqe_err: 0
>> tx31_packets: 0
>> tx31_bytes: 0
>> tx31_tso_packets: 0
>> tx31_tso_bytes: 0
>> tx31_tso_inner_packets: 0
>> tx31_tso_inner_bytes: 0
>> tx31_csum_partial: 0
>> tx31_csum_partial_inner: 0
>> tx31_added_vlan_packets: 0
>> tx31_nop: 0
>> tx31_csum_none: 0
>> tx31_stopped: 0
>> tx31_dropped: 0
>> tx31_xmit_more: 0
>> tx31_recover: 0
>> tx31_cqes: 0
>> tx31_wake: 0
>> tx31_cqe_err: 0
>> tx32_packets: 0
>> tx32_bytes: 0
>> tx32_tso_packets: 0
>> tx32_tso_bytes: 0
>> tx32_tso_inner_packets: 0
>> tx32_tso_inner_bytes: 0
>> tx32_csum_partial: 0
>> tx32_csum_partial_inner: 0
>> tx32_added_vlan_packets: 0
>> tx32_nop: 0
>> tx32_csum_none: 0
>> tx32_stopped: 0
>> tx32_dropped: 0
>> tx32_xmit_more: 0
>> tx32_recover: 0
>> tx32_cqes: 0
>> tx32_wake: 0
>> tx32_cqe_err: 0
>> tx33_packets: 0
>> tx33_bytes: 0
>> tx33_tso_packets: 0
>> tx33_tso_bytes: 0
>> tx33_tso_inner_packets: 0
>> tx33_tso_inner_bytes: 0
>> tx33_csum_partial: 0
>> tx33_csum_partial_inner: 0
>> tx33_added_vlan_packets: 0
>> tx33_nop: 0
>> tx33_csum_none: 0
>> tx33_stopped: 0
>> tx33_dropped: 0
>> tx33_xmit_more: 0
>> tx33_recover: 0
>> tx33_cqes: 0
>> tx33_wake: 0
>> tx33_cqe_err: 0
>> tx34_packets: 0
>> tx34_bytes: 0
>> tx34_tso_packets: 0
>> tx34_tso_bytes: 0
>> tx34_tso_inner_packets: 0
>> tx34_tso_inner_bytes: 0
>> tx34_csum_partial: 0
>> tx34_csum_partial_inner: 0
>> tx34_added_vlan_packets: 0
>> tx34_nop: 0
>> tx34_csum_none: 0
>> tx34_stopped: 0
>> tx34_dropped: 0
>> tx34_xmit_more: 0
>> tx34_recover: 0
>> tx34_cqes: 0
>> tx34_wake: 0
>> tx34_cqe_err: 0
>> tx35_packets: 0
>> tx35_bytes: 0
>> tx35_tso_packets: 0
>> tx35_tso_bytes: 0
>> tx35_tso_inner_packets: 0
>> tx35_tso_inner_bytes: 0
>> tx35_csum_partial: 0
>> tx35_csum_partial_inner: 0
>> tx35_added_vlan_packets: 0
>> tx35_nop: 0
>> tx35_csum_none: 0
>> tx35_stopped: 0
>> tx35_dropped: 0
>> tx35_xmit_more: 0
>> tx35_recover: 0
>> tx35_cqes: 0
>> tx35_wake: 0
>> tx35_cqe_err: 0
>> tx36_packets: 0
>> tx36_bytes: 0
>> tx36_tso_packets: 0
>> tx36_tso_bytes: 0
>> tx36_tso_inner_packets: 0
>> tx36_tso_inner_bytes: 0
>> tx36_csum_partial: 0
>> tx36_csum_partial_inner: 0
>> tx36_added_vlan_packets: 0
>> tx36_nop: 0
>> tx36_csum_none: 0
>> tx36_stopped: 0
>> tx36_dropped: 0
>> tx36_xmit_more: 0
>> tx36_recover: 0
>> tx36_cqes: 0
>> tx36_wake: 0
>> tx36_cqe_err: 0
>> tx37_packets: 0
>> tx37_bytes: 0
>> tx37_tso_packets: 0
>> tx37_tso_bytes: 0
>> tx37_tso_inner_packets: 0
>> tx37_tso_inner_bytes: 0
>> tx37_csum_partial: 0
>> tx37_csum_partial_inner: 0
>> tx37_added_vlan_packets: 0
>> tx37_nop: 0
>> tx37_csum_none: 0
>> tx37_stopped: 0
>> tx37_dropped: 0
>> tx37_xmit_more: 0
>> tx37_recover: 0
>> tx37_cqes: 0
>> tx37_wake: 0
>> tx37_cqe_err: 0
>> tx38_packets: 0
>> tx38_bytes: 0
>> tx38_tso_packets: 0
>> tx38_tso_bytes: 0
>> tx38_tso_inner_packets: 0
>> tx38_tso_inner_bytes: 0
>> tx38_csum_partial: 0
>> tx38_csum_partial_inner: 0
>> tx38_added_vlan_packets: 0
>> tx38_nop: 0
>> tx38_csum_none: 0
>> tx38_stopped: 0
>> tx38_dropped: 0
>> tx38_xmit_more: 0
>> tx38_recover: 0
>> tx38_cqes: 0
>> tx38_wake: 0
>> tx38_cqe_err: 0
>> tx39_packets: 0
>> tx39_bytes: 0
>> tx39_tso_packets: 0
>> tx39_tso_bytes: 0
>> tx39_tso_inner_packets: 0
>> tx39_tso_inner_bytes: 0
>> tx39_csum_partial: 0
>> tx39_csum_partial_inner: 0
>> tx39_added_vlan_packets: 0
>> tx39_nop: 0
>> tx39_csum_none: 0
>> tx39_stopped: 0
>> tx39_dropped: 0
>> tx39_xmit_more: 0
>> tx39_recover: 0
>> tx39_cqes: 0
>> tx39_wake: 0
>> tx39_cqe_err: 0
>> tx40_packets: 0
>> tx40_bytes: 0
>> tx40_tso_packets: 0
>> tx40_tso_bytes: 0
>> tx40_tso_inner_packets: 0
>> tx40_tso_inner_bytes: 0
>> tx40_csum_partial: 0
>> tx40_csum_partial_inner: 0
>> tx40_added_vlan_packets: 0
>> tx40_nop: 0
>> tx40_csum_none: 0
>> tx40_stopped: 0
>> tx40_dropped: 0
>> tx40_xmit_more: 0
>> tx40_recover: 0
>> tx40_cqes: 0
>> tx40_wake: 0
>> tx40_cqe_err: 0
>> tx41_packets: 0
>> tx41_bytes: 0
>> tx41_tso_packets: 0
>> tx41_tso_bytes: 0
>> tx41_tso_inner_packets: 0
>> tx41_tso_inner_bytes: 0
>> tx41_csum_partial: 0
>> tx41_csum_partial_inner: 0
>> tx41_added_vlan_packets: 0
>> tx41_nop: 0
>> tx41_csum_none: 0
>> tx41_stopped: 0
>> tx41_dropped: 0
>> tx41_xmit_more: 0
>> tx41_recover: 0
>> tx41_cqes: 0
>> tx41_wake: 0
>> tx41_cqe_err: 0
>> tx42_packets: 0
>> tx42_bytes: 0
>> tx42_tso_packets: 0
>> tx42_tso_bytes: 0
>> tx42_tso_inner_packets: 0
>> tx42_tso_inner_bytes: 0
>> tx42_csum_partial: 0
>> tx42_csum_partial_inner: 0
>> tx42_added_vlan_packets: 0
>> tx42_nop: 0
>> tx42_csum_none: 0
>> tx42_stopped: 0
>> tx42_dropped: 0
>> tx42_xmit_more: 0
>> tx42_recover: 0
>> tx42_cqes: 0
>> tx42_wake: 0
>> tx42_cqe_err: 0
>> tx43_packets: 0
>> tx43_bytes: 0
>> tx43_tso_packets: 0
>> tx43_tso_bytes: 0
>> tx43_tso_inner_packets: 0
>> tx43_tso_inner_bytes: 0
>> tx43_csum_partial: 0
>> tx43_csum_partial_inner: 0
>> tx43_added_vlan_packets: 0
>> tx43_nop: 0
>> tx43_csum_none: 0
>> tx43_stopped: 0
>> tx43_dropped: 0
>> tx43_xmit_more: 0
>> tx43_recover: 0
>> tx43_cqes: 0
>> tx43_wake: 0
>> tx43_cqe_err: 0
>> tx44_packets: 0
>> tx44_bytes: 0
>> tx44_tso_packets: 0
>> tx44_tso_bytes: 0
>> tx44_tso_inner_packets: 0
>> tx44_tso_inner_bytes: 0
>> tx44_csum_partial: 0
>> tx44_csum_partial_inner: 0
>> tx44_added_vlan_packets: 0
>> tx44_nop: 0
>> tx44_csum_none: 0
>> tx44_stopped: 0
>> tx44_dropped: 0
>> tx44_xmit_more: 0
>> tx44_recover: 0
>> tx44_cqes: 0
>> tx44_wake: 0
>> tx44_cqe_err: 0
>> tx45_packets: 0
>> tx45_bytes: 0
>> tx45_tso_packets: 0
>> tx45_tso_bytes: 0
>> tx45_tso_inner_packets: 0
>> tx45_tso_inner_bytes: 0
>> tx45_csum_partial: 0
>> tx45_csum_partial_inner: 0
>> tx45_added_vlan_packets: 0
>> tx45_nop: 0
>> tx45_csum_none: 0
>> tx45_stopped: 0
>> tx45_dropped: 0
>> tx45_xmit_more: 0
>> tx45_recover: 0
>> tx45_cqes: 0
>> tx45_wake: 0
>> tx45_cqe_err: 0
>> tx46_packets: 0
>> tx46_bytes: 0
>> tx46_tso_packets: 0
>> tx46_tso_bytes: 0
>> tx46_tso_inner_packets: 0
>> tx46_tso_inner_bytes: 0
>> tx46_csum_partial: 0
>> tx46_csum_partial_inner: 0
>> tx46_added_vlan_packets: 0
>> tx46_nop: 0
>> tx46_csum_none: 0
>> tx46_stopped: 0
>> tx46_dropped: 0
>> tx46_xmit_more: 0
>> tx46_recover: 0
>> tx46_cqes: 0
>> tx46_wake: 0
>> tx46_cqe_err: 0
>> tx47_packets: 0
>> tx47_bytes: 0
>> tx47_tso_packets: 0
>> tx47_tso_bytes: 0
>> tx47_tso_inner_packets: 0
>> tx47_tso_inner_bytes: 0
>> tx47_csum_partial: 0
>> tx47_csum_partial_inner: 0
>> tx47_added_vlan_packets: 0
>> tx47_nop: 0
>> tx47_csum_none: 0
>> tx47_stopped: 0
>> tx47_dropped: 0
>> tx47_xmit_more: 0
>> tx47_recover: 0
>> tx47_cqes: 0
>> tx47_wake: 0
>> tx47_cqe_err: 0
>> tx48_packets: 0
>> tx48_bytes: 0
>> tx48_tso_packets: 0
>> tx48_tso_bytes: 0
>> tx48_tso_inner_packets: 0
>> tx48_tso_inner_bytes: 0
>> tx48_csum_partial: 0
>> tx48_csum_partial_inner: 0
>> tx48_added_vlan_packets: 0
>> tx48_nop: 0
>> tx48_csum_none: 0
>> tx48_stopped: 0
>> tx48_dropped: 0
>> tx48_xmit_more: 0
>> tx48_recover: 0
>> tx48_cqes: 0
>> tx48_wake: 0
>> tx48_cqe_err: 0
>> tx49_packets: 0
>> tx49_bytes: 0
>> tx49_tso_packets: 0
>> tx49_tso_bytes: 0
>> tx49_tso_inner_packets: 0
>> tx49_tso_inner_bytes: 0
>> tx49_csum_partial: 0
>> tx49_csum_partial_inner: 0
>> tx49_added_vlan_packets: 0
>> tx49_nop: 0
>> tx49_csum_none: 0
>> tx49_stopped: 0
>> tx49_dropped: 0
>> tx49_xmit_more: 0
>> tx49_recover: 0
>> tx49_cqes: 0
>> tx49_wake: 0
>> tx49_cqe_err: 0
>> tx50_packets: 0
>> tx50_bytes: 0
>> tx50_tso_packets: 0
>> tx50_tso_bytes: 0
>> tx50_tso_inner_packets: 0
>> tx50_tso_inner_bytes: 0
>> tx50_csum_partial: 0
>> tx50_csum_partial_inner: 0
>> tx50_added_vlan_packets: 0
>> tx50_nop: 0
>> tx50_csum_none: 0
>> tx50_stopped: 0
>> tx50_dropped: 0
>> tx50_xmit_more: 0
>> tx50_recover: 0
>> tx50_cqes: 0
>> tx50_wake: 0
>> tx50_cqe_err: 0
>> tx51_packets: 0
>> tx51_bytes: 0
>> tx51_tso_packets: 0
>> tx51_tso_bytes: 0
>> tx51_tso_inner_packets: 0
>> tx51_tso_inner_bytes: 0
>> tx51_csum_partial: 0
>> tx51_csum_partial_inner: 0
>> tx51_added_vlan_packets: 0
>> tx51_nop: 0
>> tx51_csum_none: 0
>> tx51_stopped: 0
>> tx51_dropped: 0
>> tx51_xmit_more: 0
>> tx51_recover: 0
>> tx51_cqes: 0
>> tx51_wake: 0
>> tx51_cqe_err: 0
>> tx52_packets: 0
>> tx52_bytes: 0
>> tx52_tso_packets: 0
>> tx52_tso_bytes: 0
>> tx52_tso_inner_packets: 0
>> tx52_tso_inner_bytes: 0
>> tx52_csum_partial: 0
>> tx52_csum_partial_inner: 0
>> tx52_added_vlan_packets: 0
>> tx52_nop: 0
>> tx52_csum_none: 0
>> tx52_stopped: 0
>> tx52_dropped: 0
>> tx52_xmit_more: 0
>> tx52_recover: 0
>> tx52_cqes: 0
>> tx52_wake: 0
>> tx52_cqe_err: 0
>> tx53_packets: 0
>> tx53_bytes: 0
>> tx53_tso_packets: 0
>> tx53_tso_bytes: 0
>> tx53_tso_inner_packets: 0
>> tx53_tso_inner_bytes: 0
>> tx53_csum_partial: 0
>> tx53_csum_partial_inner: 0
>> tx53_added_vlan_packets: 0
>> tx53_nop: 0
>> tx53_csum_none: 0
>> tx53_stopped: 0
>> tx53_dropped: 0
>> tx53_xmit_more: 0
>> tx53_recover: 0
>> tx53_cqes: 0
>> tx53_wake: 0
>> tx53_cqe_err: 0
>> tx54_packets: 0
>> tx54_bytes: 0
>> tx54_tso_packets: 0
>> tx54_tso_bytes: 0
>> tx54_tso_inner_packets: 0
>> tx54_tso_inner_bytes: 0
>> tx54_csum_partial: 0
>> tx54_csum_partial_inner: 0
>> tx54_added_vlan_packets: 0
>> tx54_nop: 0
>> tx54_csum_none: 0
>> tx54_stopped: 0
>> tx54_dropped: 0
>> tx54_xmit_more: 0
>> tx54_recover: 0
>> tx54_cqes: 0
>> tx54_wake: 0
>> tx54_cqe_err: 0
>> tx55_packets: 0
>> tx55_bytes: 0
>> tx55_tso_packets: 0
>> tx55_tso_bytes: 0
>> tx55_tso_inner_packets: 0
>> tx55_tso_inner_bytes: 0
>> tx55_csum_partial: 0
>> tx55_csum_partial_inner: 0
>> tx55_added_vlan_packets: 0
>> tx55_nop: 0
>> tx55_csum_none: 0
>> tx55_stopped: 0
>> tx55_dropped: 0
>> tx55_xmit_more: 0
>> tx55_recover: 0
>> tx55_cqes: 0
>> tx55_wake: 0
>> tx55_cqe_err: 0
>> tx0_xdp_xmit: 0
>> tx0_xdp_full: 0
>> tx0_xdp_err: 0
>> tx0_xdp_cqes: 0
>> tx1_xdp_xmit: 0
>> tx1_xdp_full: 0
>> tx1_xdp_err: 0
>> tx1_xdp_cqes: 0
>> tx2_xdp_xmit: 0
>> tx2_xdp_full: 0
>> tx2_xdp_err: 0
>> tx2_xdp_cqes: 0
>> tx3_xdp_xmit: 0
>> tx3_xdp_full: 0
>> tx3_xdp_err: 0
>> tx3_xdp_cqes: 0
>> tx4_xdp_xmit: 0
>> tx4_xdp_full: 0
>> tx4_xdp_err: 0
>> tx4_xdp_cqes: 0
>> tx5_xdp_xmit: 0
>> tx5_xdp_full: 0
>> tx5_xdp_err: 0
>> tx5_xdp_cqes: 0
>> tx6_xdp_xmit: 0
>> tx6_xdp_full: 0
>> tx6_xdp_err: 0
>> tx6_xdp_cqes: 0
>> tx7_xdp_xmit: 0
>> tx7_xdp_full: 0
>> tx7_xdp_err: 0
>> tx7_xdp_cqes: 0
>> tx8_xdp_xmit: 0
>> tx8_xdp_full: 0
>> tx8_xdp_err: 0
>> tx8_xdp_cqes: 0
>> tx9_xdp_xmit: 0
>> tx9_xdp_full: 0
>> tx9_xdp_err: 0
>> tx9_xdp_cqes: 0
>> tx10_xdp_xmit: 0
>> tx10_xdp_full: 0
>> tx10_xdp_err: 0
>> tx10_xdp_cqes: 0
>> tx11_xdp_xmit: 0
>> tx11_xdp_full: 0
>> tx11_xdp_err: 0
>> tx11_xdp_cqes: 0
>> tx12_xdp_xmit: 0
>> tx12_xdp_full: 0
>> tx12_xdp_err: 0
>> tx12_xdp_cqes: 0
>> tx13_xdp_xmit: 0
>> tx13_xdp_full: 0
>> tx13_xdp_err: 0
>> tx13_xdp_cqes: 0
>> tx14_xdp_xmit: 0
>> tx14_xdp_full: 0
>> tx14_xdp_err: 0
>> tx14_xdp_cqes: 0
>> tx15_xdp_xmit: 0
>> tx15_xdp_full: 0
>> tx15_xdp_err: 0
>> tx15_xdp_cqes: 0
>> tx16_xdp_xmit: 0
>> tx16_xdp_full: 0
>> tx16_xdp_err: 0
>> tx16_xdp_cqes: 0
>> tx17_xdp_xmit: 0
>> tx17_xdp_full: 0
>> tx17_xdp_err: 0
>> tx17_xdp_cqes: 0
>> tx18_xdp_xmit: 0
>> tx18_xdp_full: 0
>> tx18_xdp_err: 0
>> tx18_xdp_cqes: 0
>> tx19_xdp_xmit: 0
>> tx19_xdp_full: 0
>> tx19_xdp_err: 0
>> tx19_xdp_cqes: 0
>> tx20_xdp_xmit: 0
>> tx20_xdp_full: 0
>> tx20_xdp_err: 0
>> tx20_xdp_cqes: 0
>> tx21_xdp_xmit: 0
>> tx21_xdp_full: 0
>> tx21_xdp_err: 0
>> tx21_xdp_cqes: 0
>> tx22_xdp_xmit: 0
>> tx22_xdp_full: 0
>> tx22_xdp_err: 0
>> tx22_xdp_cqes: 0
>> tx23_xdp_xmit: 0
>> tx23_xdp_full: 0
>> tx23_xdp_err: 0
>> tx23_xdp_cqes: 0
>> tx24_xdp_xmit: 0
>> tx24_xdp_full: 0
>> tx24_xdp_err: 0
>> tx24_xdp_cqes: 0
>> tx25_xdp_xmit: 0
>> tx25_xdp_full: 0
>> tx25_xdp_err: 0
>> tx25_xdp_cqes: 0
>> tx26_xdp_xmit: 0
>> tx26_xdp_full: 0
>> tx26_xdp_err: 0
>> tx26_xdp_cqes: 0
>> tx27_xdp_xmit: 0
>> tx27_xdp_full: 0
>> tx27_xdp_err: 0
>> tx27_xdp_cqes: 0
>> tx28_xdp_xmit: 0
>> tx28_xdp_full: 0
>> tx28_xdp_err: 0
>> tx28_xdp_cqes: 0
>> tx29_xdp_xmit: 0
>> tx29_xdp_full: 0
>> tx29_xdp_err: 0
>> tx29_xdp_cqes: 0
>> tx30_xdp_xmit: 0
>> tx30_xdp_full: 0
>> tx30_xdp_err: 0
>> tx30_xdp_cqes: 0
>> tx31_xdp_xmit: 0
>> tx31_xdp_full: 0
>> tx31_xdp_err: 0
>> tx31_xdp_cqes: 0
>> tx32_xdp_xmit: 0
>> tx32_xdp_full: 0
>> tx32_xdp_err: 0
>> tx32_xdp_cqes: 0
>> tx33_xdp_xmit: 0
>> tx33_xdp_full: 0
>> tx33_xdp_err: 0
>> tx33_xdp_cqes: 0
>> tx34_xdp_xmit: 0
>> tx34_xdp_full: 0
>> tx34_xdp_err: 0
>> tx34_xdp_cqes: 0
>> tx35_xdp_xmit: 0
>> tx35_xdp_full: 0
>> tx35_xdp_err: 0
>> tx35_xdp_cqes: 0
>> tx36_xdp_xmit: 0
>> tx36_xdp_full: 0
>> tx36_xdp_err: 0
>> tx36_xdp_cqes: 0
>> tx37_xdp_xmit: 0
>> tx37_xdp_full: 0
>> tx37_xdp_err: 0
>> tx37_xdp_cqes: 0
>> tx38_xdp_xmit: 0
>> tx38_xdp_full: 0
>> tx38_xdp_err: 0
>> tx38_xdp_cqes: 0
>> tx39_xdp_xmit: 0
>> tx39_xdp_full: 0
>> tx39_xdp_err: 0
>> tx39_xdp_cqes: 0
>> tx40_xdp_xmit: 0
>> tx40_xdp_full: 0
>> tx40_xdp_err: 0
>> tx40_xdp_cqes: 0
>> tx41_xdp_xmit: 0
>> tx41_xdp_full: 0
>> tx41_xdp_err: 0
>> tx41_xdp_cqes: 0
>> tx42_xdp_xmit: 0
>> tx42_xdp_full: 0
>> tx42_xdp_err: 0
>> tx42_xdp_cqes: 0
>> tx43_xdp_xmit: 0
>> tx43_xdp_full: 0
>> tx43_xdp_err: 0
>> tx43_xdp_cqes: 0
>> tx44_xdp_xmit: 0
>> tx44_xdp_full: 0
>> tx44_xdp_err: 0
>> tx44_xdp_cqes: 0
>> tx45_xdp_xmit: 0
>> tx45_xdp_full: 0
>> tx45_xdp_err: 0
>> tx45_xdp_cqes: 0
>> tx46_xdp_xmit: 0
>> tx46_xdp_full: 0
>> tx46_xdp_err: 0
>> tx46_xdp_cqes: 0
>> tx47_xdp_xmit: 0
>> tx47_xdp_full: 0
>> tx47_xdp_err: 0
>> tx47_xdp_cqes: 0
>> tx48_xdp_xmit: 0
>> tx48_xdp_full: 0
>> tx48_xdp_err: 0
>> tx48_xdp_cqes: 0
>> tx49_xdp_xmit: 0
>> tx49_xdp_full: 0
>> tx49_xdp_err: 0
>> tx49_xdp_cqes: 0
>> tx50_xdp_xmit: 0
>> tx50_xdp_full: 0
>> tx50_xdp_err: 0
>> tx50_xdp_cqes: 0
>> tx51_xdp_xmit: 0
>> tx51_xdp_full: 0
>> tx51_xdp_err: 0
>> tx51_xdp_cqes: 0
>> tx52_xdp_xmit: 0
>> tx52_xdp_full: 0
>> tx52_xdp_err: 0
>> tx52_xdp_cqes: 0
>> tx53_xdp_xmit: 0
>> tx53_xdp_full: 0
>> tx53_xdp_err: 0
>> tx53_xdp_cqes: 0
>> tx54_xdp_xmit: 0
>> tx54_xdp_full: 0
>> tx54_xdp_err: 0
>> tx54_xdp_cqes: 0
>> tx55_xdp_xmit: 0
>> tx55_xdp_full: 0
>> tx55_xdp_err: 0
>> tx55_xdp_cqes: 0
>>
>>
>>> [...]
>>>
>>>>>> ethtool -S enp175s0f0
>>>>>> NIC statistics:
>>>>>> rx_packets: 141574897253
>>>>>> rx_bytes: 184445040406258
>>>>>> tx_packets: 172569543894
>>>>>> tx_bytes: 99486882076365
>>>>>> tx_tso_packets: 9367664195
>>>>>> tx_tso_bytes: 56435233992948
>>>>>> tx_tso_inner_packets: 0
>>>>>> tx_tso_inner_bytes: 0
>>>>>> tx_added_vlan_packets: 141297671626
>>>>>> tx_nop: 2102916272
>>>>>> rx_lro_packets: 0
>>>>>> rx_lro_bytes: 0
>>>>>> rx_ecn_mark: 0
>>>>>> rx_removed_vlan_packets: 141574897252
>>>>>> rx_csum_unnecessary: 0
>>>>>> rx_csum_none: 23135854
>>>>>> rx_csum_complete: 141551761398
>>>>>> rx_csum_unnecessary_inner: 0
>>>>>> rx_xdp_drop: 0
>>>>>> rx_xdp_redirect: 0
>>>>>> rx_xdp_tx_xmit: 0
>>>>>> rx_xdp_tx_full: 0
>>>>>> rx_xdp_tx_err: 0
>>>>>> rx_xdp_tx_cqe: 0
>>>>>> tx_csum_none: 127934791664
>>>>> It is a good idea to look into this, tx is not requesting hw tx
>>>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>>>> on
>>>>> calculating csum, or maybe this is just the rx csum complete..
>>>>>
>>>>>> tx_csum_partial: 13362879974
>>>>>> tx_csum_partial_inner: 0
>>>>>> tx_queue_stopped: 232561
>>>>> TX queues are stalling, could be an indentation for the pcie
>>>>> bottelneck.
>>>>>
>>>>>> tx_queue_dropped: 0
>>>>>> tx_xmit_more: 1266021946
>>>>>> tx_recover: 0
>>>>>> tx_cqes: 140031716469
>>>>>> tx_queue_wake: 232561
>>>>>> tx_udp_seg_rem: 0
>>>>>> tx_cqe_err: 0
>>>>>> tx_xdp_xmit: 0
>>>>>> tx_xdp_full: 0
>>>>>> tx_xdp_err: 0
>>>>>> tx_xdp_cqes: 0
>>>>>> rx_wqe_err: 0
>>>>>> rx_mpwqe_filler_cqes: 0
>>>>>> rx_mpwqe_filler_strides: 0
>>>>>> rx_buff_alloc_err: 0
>>>>>> rx_cqe_compress_blks: 0
>>>>>> rx_cqe_compress_pkts: 0
>>>>>> rx_page_reuse: 0
>>>>>> rx_cache_reuse: 16625975793
>>>>>> rx_cache_full: 54161465914
>>>>>> rx_cache_empty: 258048
>>>>>> rx_cache_busy: 54161472735
>>>>>> rx_cache_waive: 0
>>>>>> rx_congst_umr: 0
>>>>>> rx_arfs_err: 0
>>>>>> ch_events: 40572621887
>>>>>> ch_poll: 40885650979
>>>>>> ch_arm: 40429276692
>>>>>> ch_aff_change: 0
>>>>>> ch_eq_rearm: 0
>>>>>> rx_out_of_buffer: 2791690
>>>>>> rx_if_down_packets: 74
>>>>>> rx_vport_unicast_packets: 141843476308
>>>>>> rx_vport_unicast_bytes: 185421265403318
>>>>>> tx_vport_unicast_packets: 172569484005
>>>>>> tx_vport_unicast_bytes: 100019940094298
>>>>>> rx_vport_multicast_packets: 85122935
>>>>>> rx_vport_multicast_bytes: 5761316431
>>>>>> tx_vport_multicast_packets: 6452
>>>>>> tx_vport_multicast_bytes: 643540
>>>>>> rx_vport_broadcast_packets: 22423624
>>>>>> rx_vport_broadcast_bytes: 1390127090
>>>>>> tx_vport_broadcast_packets: 22024
>>>>>> tx_vport_broadcast_bytes: 1321440
>>>>>> rx_vport_rdma_unicast_packets: 0
>>>>>> rx_vport_rdma_unicast_bytes: 0
>>>>>> tx_vport_rdma_unicast_packets: 0
>>>>>> tx_vport_rdma_unicast_bytes: 0
>>>>>> rx_vport_rdma_multicast_packets: 0
>>>>>> rx_vport_rdma_multicast_bytes: 0
>>>>>> tx_vport_rdma_multicast_packets: 0
>>>>>> tx_vport_rdma_multicast_bytes: 0
>>>>>> tx_packets_phy: 172569501577
>>>>>> rx_packets_phy: 142871314588
>>>>>> rx_crc_errors_phy: 0
>>>>>> tx_bytes_phy: 100710212814151
>>>>>> rx_bytes_phy: 187209224289564
>>>>>> tx_multicast_phy: 6452
>>>>>> tx_broadcast_phy: 22024
>>>>>> rx_multicast_phy: 85122933
>>>>>> rx_broadcast_phy: 22423623
>>>>>> rx_in_range_len_errors_phy: 2
>>>>>> rx_out_of_range_len_phy: 0
>>>>>> rx_oversize_pkts_phy: 0
>>>>>> rx_symbol_err_phy: 0
>>>>>> tx_mac_control_phy: 0
>>>>>> rx_mac_control_phy: 0
>>>>>> rx_unsupported_op_phy: 0
>>>>>> rx_pause_ctrl_phy: 0
>>>>>> tx_pause_ctrl_phy: 0
>>>>>> rx_discards_phy: 920161423
>>>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>>>> to
>>>>> the pcie bottleneck.
>>>> Yes this side is receiving more traffic - second port is +10G more tx
>>>>
>>> [...]
>>>
>>>
>>>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>>>> 0.00 0.00 0.00 31.30
>>>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>>>> 0.00 0.00 0.00 24.90
>>>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>>>> 0.00 0.00 0.00 19.68
>>>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>>>> 0.00 0.00 0.00 18.00
>>>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>>>> 0.00 0.00 0.00 17.40
>>>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>>>> 0.00 0.00 0.00 26.03
>>>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>>>> 0.00 0.00 0.00 17.52
>>>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>>>> 0.00 0.00 0.00 18.20
>>>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>>>> 0.00 0.00 0.00 17.68
>>>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>>>> 0.00 0.00 0.00 18.20
>>>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>>>> 0.00 0.00 0.00 20.42
>>>>> The numa cores are not at 100% util, you have around 20% of idle on
>>>>> each one.
>>>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>>>> push
>>>> aditional 1-2Gbit/s
>>>>
>>> yes but, it doens't look like the bottleneck is the cpu, although it is
>>> close to be :)..
>>>
>>>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 96.10
>>>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.70
>>>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.50
>>>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 100.00
>>>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 97.40
>>>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.10
>>>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>>>> 0.00
>>>>>> 0.00 0.00 99.40
>>>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>>>> 0.00 0.00 0.00 19.42
>>>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>>>> 0.00 0.00 0.00 26.60
>>>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>>>> 0.00 0.00 0.00 21.60
>>>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>>>> 0.00 0.00 0.00 20.50
>>>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>>>> 0.00 0.00 0.00 21.60
>>>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>>>> 0.00 0.00 0.00 24.58
>>>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>>>> 0.00 0.00 0.00 24.68
>>>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>>>> 0.00 0.00 0.00 28.34
>>>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>>>> 0.00 0.00 0.00 25.83
>>>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>>>> 0.00 0.00 0.00 27.20
>>>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>>>> 0.00 0.00 0.00 28.03
>>>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>>>> 0.00 0.00 0.00 24.70
>>>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>>>> 0.00 0.00 0.00 25.30
>>>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>>>> 0.00 0.00 0.00 26.77
>>>>>>
>>>>>>
>>>>>> ethtool -k enp175s0f0
>>>>>> Features for enp175s0f0:
>>>>>> rx-checksumming: on
>>>>>> tx-checksumming: on
>>>>>> tx-checksum-ipv4: on
>>>>>> tx-checksum-ip-generic: off [fixed]
>>>>>> tx-checksum-ipv6: on
>>>>>> tx-checksum-fcoe-crc: off [fixed]
>>>>>> tx-checksum-sctp: off [fixed]
>>>>>> scatter-gather: on
>>>>>> tx-scatter-gather: on
>>>>>> tx-scatter-gather-fraglist: off [fixed]
>>>>>> tcp-segmentation-offload: on
>>>>>> tx-tcp-segmentation: on
>>>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>>>> tx-tcp-mangleid-segmentation: off
>>>>>> tx-tcp6-segmentation: on
>>>>>> udp-fragmentation-offload: off
>>>>>> generic-segmentation-offload: on
>>>>>> generic-receive-offload: on
>>>>>> large-receive-offload: off [fixed]
>>>>>> rx-vlan-offload: on
>>>>>> tx-vlan-offload: on
>>>>>> ntuple-filters: off
>>>>>> receive-hashing: on
>>>>>> highdma: on [fixed]
>>>>>> rx-vlan-filter: on
>>>>>> vlan-challenged: off [fixed]
>>>>>> tx-lockless: off [fixed]
>>>>>> netns-local: off [fixed]
>>>>>> tx-gso-robust: off [fixed]
>>>>>> tx-fcoe-segmentation: off [fixed]
>>>>>> tx-gre-segmentation: on
>>>>>> tx-gre-csum-segmentation: on
>>>>>> tx-ipxip4-segmentation: off [fixed]
>>>>>> tx-ipxip6-segmentation: off [fixed]
>>>>>> tx-udp_tnl-segmentation: on
>>>>>> tx-udp_tnl-csum-segmentation: on
>>>>>> tx-gso-partial: on
>>>>>> tx-sctp-segmentation: off [fixed]
>>>>>> tx-esp-segmentation: off [fixed]
>>>>>> tx-udp-segmentation: on
>>>>>> fcoe-mtu: off [fixed]
>>>>>> tx-nocache-copy: off
>>>>>> loopback: off [fixed]
>>>>>> rx-fcs: off
>>>>>> rx-all: off
>>>>>> tx-vlan-stag-hw-insert: on
>>>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>>>> rx-vlan-stag-filter: on [fixed]
>>>>>> l2-fwd-offload: off [fixed]
>>>>>> hw-tc-offload: off
>>>>>> esp-hw-offload: off [fixed]
>>>>>> esp-tx-csum-hw-offload: off [fixed]
>>>>>> rx-udp_tunnel-port-offload: on
>>>>>> tls-hw-tx-offload: off [fixed]
>>>>>> tls-hw-rx-offload: off [fixed]
>>>>>> rx-gro-hw: off [fixed]
>>>>>> tls-hw-record: off [fixed]
>>>>>>
>>>>>> ethtool -c enp175s0f0
>>>>>> Coalesce parameters for enp175s0f0:
>>>>>> Adaptive RX: off TX: on
>>>>>> stats-block-usecs: 0
>>>>>> sample-interval: 0
>>>>>> pkt-rate-low: 0
>>>>>> pkt-rate-high: 0
>>>>>> dmac: 32703
>>>>>>
>>>>>> rx-usecs: 256
>>>>>> rx-frames: 128
>>>>>> rx-usecs-irq: 0
>>>>>> rx-frames-irq: 0
>>>>>>
>>>>>> tx-usecs: 8
>>>>>> tx-frames: 128
>>>>>> tx-usecs-irq: 0
>>>>>> tx-frames-irq: 0
>>>>>>
>>>>>> rx-usecs-low: 0
>>>>>> rx-frame-low: 0
>>>>>> tx-usecs-low: 0
>>>>>> tx-frame-low: 0
>>>>>>
>>>>>> rx-usecs-high: 0
>>>>>> rx-frame-high: 0
>>>>>> tx-usecs-high: 0
>>>>>> tx-frame-high: 0
>>>>>>
>>>>>> ethtool -g enp175s0f0
>>>>>> Ring parameters for enp175s0f0:
>>>>>> Pre-set maximums:
>>>>>> RX: 8192
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: 8192
>>>>>> Current hardware settings:
>>>>>> RX: 4096
>>>>>> RX Mini: 0
>>>>>> RX Jumbo: 0
>>>>>> TX: 4096
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> Also changed a little coalesce params - and best for this config are:
>>>> ethtool -c enp175s0f0
>>>> Coalesce parameters for enp175s0f0:
>>>> Adaptive RX: off TX: off
>>>> stats-block-usecs: 0
>>>> sample-interval: 0
>>>> pkt-rate-low: 0
>>>> pkt-rate-high: 0
>>>> dmac: 32573
>>>>
>>>> rx-usecs: 40
>>>> rx-frames: 128
>>>> rx-usecs-irq: 0
>>>> rx-frames-irq: 0
>>>>
>>>> tx-usecs: 8
>>>> tx-frames: 8
>>>> tx-usecs-irq: 0
>>>> tx-frames-irq: 0
>>>>
>>>> rx-usecs-low: 0
>>>> rx-frame-low: 0
>>>> tx-usecs-low: 0
>>>> tx-frame-low: 0
>>>>
>>>> rx-usecs-high: 0
>>>> rx-frame-high: 0
>>>> tx-usecs-high: 0
>>>> tx-frame-high: 0
>>>>
>>>>
>>>> Less drops on RX side - and more pps in overall forwarded.
>>>>
>>> how much improvement ? maybe we can improve our adaptive rx coal to be
>>> efficient for this work load.
>>>
>>>
>> I can prepare more stats with ethtool maybee to compare - but
>> normally tested with simple icmp forwarded from interface to interface
>> - before change coalescence params:
>> adaptive-rx off rx-usecs 384 rx-frames 128
>> 3% loss for icmp
>> - after change to:
>> adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8
>> tx-frames 8
>> 2% loss for icmp
>>
>> But yes - to know better will need to compare rx/tx counters from
>> ethtool + /proc/net/dev
>>
>>
>> Was trying to turn on adaptative-tx+rx - but 100% saturation at
>> 43Gbit/s RX / 43Gbit/s TX
>>
>>
>>
>>
>>
>>
>>
>
>
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-01 21:24 UTC (permalink / raw)
To: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <5c55fc87-4508-bb67-8f89-f95708e33e0f@itcare.pl>
W dniu 01.11.2018 o 22:18, Paweł Staszewski pisze:
>
>
> W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
>> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>>> Hi
>>>>>
>>>>> So maybee someone will be interested how linux kernel handles
>>>>> normal
>>>>> traffic (not pktgen :) )
>>>>>
>>>>>
>>>>> Server HW configuration:
>>>>>
>>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>>
>>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>>
>>>>>
>>>>> Server software:
>>>>>
>>>>> FRR - as routing daemon
>>>>>
>>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>>> local
>>>>> numa
>>>>> node)
>>>>>
>>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>>> numa
>>>>> node)
>>>>>
>>>>>
>>>>> Maximum traffic that server can handle:
>>>>>
>>>>> Bandwidth
>>>>>
>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>> input: /proc/net/dev type: rate
>>>>> \ iface Rx Tx Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>>> Gb/s
>>>>> 65.74 Gb/s
>>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>>> Gb/s
>>>>> 66.51 Gb/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>> total: 66.58 Gb/s 65.67
>>>>> Gb/s
>>>>> 132.25 Gb/s
>>>>>
>>>>>
>>>>> Packets per second:
>>>>>
>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>> input: /proc/net/dev type: rate
>>>>> - iface Rx Tx Total
>>>>> =================================================================
>>>>> ====
>>>>> =========
>>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>>> 8735207.00 P/s
>>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>>> 8790460.00 P/s
>>>>> ---------------------------------------------------------------
>>>>> ----
>>>>> -----------
>>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>>> 17525668.00 P/s
>>>>>
>>>>>
>>>>> After reaching that limits nics on the upstream side (more RX
>>>>> traffic)
>>>>> start to drop packets
>>>>>
>>>>>
>>>>> I just dont understand that server can't handle more bandwidth
>>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>>> RX
>>>>> side are increasing.
>>>>>
>>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>>> NIC (
>>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>>> 132.25
>>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>>> hmm yes that was my concern also - cause cant find anywhere
>>> informations
>>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>>> 8GT
>>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>>> bw
>>> on both ports
>> i think it is bidir
>>
>>> This can explain maybee also why cpuload is rising rapidly from
>>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>>> so
>>> there can be some error in reading them when offloading (gro/gso/tso)
>>> on
>>> nic's is enabled that is why
>>>
>>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>>> 8GT
>>>>> is
>>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>>> and
>>>>> pps
>>>>> (like 4x more comparing to normal internet traffic)
>>>>>
>>>> Are you forwarding when using pktgen as well or you just testing
>>>> the RX
>>>> side pps ?
>>> Yes pktgen was tested on single port RX
>>> Can check also forwarding to eliminate pciex limits
>>>
>> So this explains why you have more RX pps, since tx is idle and pcie
>> will be free to do only rx.
>>
>> [...]
>>
>>
>>>>> ethtool -S enp175s0f1
>>>>> NIC statistics:
>>>>> rx_packets: 173730800927
>>>>> rx_bytes: 99827422751332
>>>>> tx_packets: 142532009512
>>>>> tx_bytes: 184633045911222
>>>>> tx_tso_packets: 25989113891
>>>>> tx_tso_bytes: 132933363384458
>>>>> tx_tso_inner_packets: 0
>>>>> tx_tso_inner_bytes: 0
>>>>> tx_added_vlan_packets: 74630239613
>>>>> tx_nop: 2029817748
>>>>> rx_lro_packets: 0
>>>>> rx_lro_bytes: 0
>>>>> rx_ecn_mark: 0
>>>>> rx_removed_vlan_packets: 173730800927
>>>>> rx_csum_unnecessary: 0
>>>>> rx_csum_none: 434357
>>>>> rx_csum_complete: 173730366570
>>>>> rx_csum_unnecessary_inner: 0
>>>>> rx_xdp_drop: 0
>>>>> rx_xdp_redirect: 0
>>>>> rx_xdp_tx_xmit: 0
>>>>> rx_xdp_tx_full: 0
>>>>> rx_xdp_tx_err: 0
>>>>> rx_xdp_tx_cqe: 0
>>>>> tx_csum_none: 38260960853
>>>>> tx_csum_partial: 36369278774
>>>>> tx_csum_partial_inner: 0
>>>>> tx_queue_stopped: 1
>>>>> tx_queue_dropped: 0
>>>>> tx_xmit_more: 748638099
>>>>> tx_recover: 0
>>>>> tx_cqes: 73881645031
>>>>> tx_queue_wake: 1
>>>>> tx_udp_seg_rem: 0
>>>>> tx_cqe_err: 0
>>>>> tx_xdp_xmit: 0
>>>>> tx_xdp_full: 0
>>>>> tx_xdp_err: 0
>>>>> tx_xdp_cqes: 0
>>>>> rx_wqe_err: 0
>>>>> rx_mpwqe_filler_cqes: 0
>>>>> rx_mpwqe_filler_strides: 0
>>>>> rx_buff_alloc_err: 0
>>>>> rx_cqe_compress_blks: 0
>>>>> rx_cqe_compress_pkts: 0
>>>> If this is a pcie bottleneck it might be useful to enable CQE
>>>> compression (to reduce PCIe completion descriptors transactions)
>>>> you should see the above rx_cqe_compress_pkts increasing when
>>>> enabled.
>>>>
>>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>>> $ ethtool --show-priv-flags enp175s0f1
>>>> Private flags for p6p1:
>>>> rx_cqe_moder : on
>>>> cqe_moder : off
>>>> rx_cqe_compress : on
>>>> ...
>>>>
>>>> try this on both interfaces.
>>> Done
>>> ethtool --show-priv-flags enp175s0f1
>>> Private flags for enp175s0f1:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : on
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>>> ethtool --show-priv-flags enp175s0f0
>>> Private flags for enp175s0f0:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : on
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>> did it help reduce the load on the pcie ? do you see more pps ?
>> what is the ratio between rx_cqe_compress_pkts and over all rx packets
>> ?
> So - a little more pps
> Before change top - graph / after bottom -> image with graph stats
> from proc/net/dev
Attached link to graph
https://uploadfiles.io/5vgbh
> cqe_compress enabled at 11:55
>
> Sorry - but for real life traffic it is hard to do any counter
> differences - cause traffic just rising alone from minute to minute :)
> But for that time the change is visible on graph - cause was almost
> same for past 20minutes before change.
>
>
> full ethtool below:
> NIC statistics:
> rx_packets: 516522465438
> rx_bytes: 680052911258729
> tx_packets: 677697545586
> tx_bytes: 413647643141709
> tx_tso_packets: 42530913279
> tx_tso_bytes: 235655668554142
> tx_tso_inner_packets: 0
> tx_tso_inner_bytes: 0
> tx_added_vlan_packets: 551156530885
> tx_nop: 8536823558
> rx_lro_packets: 0
> rx_lro_bytes: 0
> rx_ecn_mark: 0
> rx_removed_vlan_packets: 516522465438
> rx_csum_unnecessary: 0
> rx_csum_none: 50382868
> rx_csum_complete: 516472082570
> rx_csum_unnecessary_inner: 0
> rx_xdp_drop: 0
> rx_xdp_redirect: 0
> rx_xdp_tx_xmit: 0
> rx_xdp_tx_full: 0
> rx_xdp_tx_err: 0
> rx_xdp_tx_cqe: 0
> tx_csum_none: 494075047017
> tx_csum_partial: 57081483898
> tx_csum_partial_inner: 0
> tx_queue_stopped: 518624
> tx_queue_dropped: 0
> tx_xmit_more: 1717880628
> tx_recover: 0
> tx_cqes: 549438869029
> tx_queue_wake: 518627
> tx_udp_seg_rem: 0
> tx_cqe_err: 0
> tx_xdp_xmit: 0
> tx_xdp_full: 0
> tx_xdp_err: 0
> tx_xdp_cqes: 0
> rx_wqe_err: 0
> rx_mpwqe_filler_cqes: 0
> rx_mpwqe_filler_strides: 0
> rx_buff_alloc_err: 0
> rx_cqe_compress_blks: 11483228712
> rx_cqe_compress_pkts: 25794213324
> rx_page_reuse: 0
> rx_cache_reuse: 63610249810
> rx_cache_full: 194650916511
> rx_cache_empty: 1118208
> rx_cache_busy: 194650982430
> rx_cache_waive: 0
> rx_congst_umr: 0
> rx_arfs_err: 0
> ch_events: 119556002196
> ch_poll: 121107424977
> ch_arm: 115856746008
> ch_aff_change: 31
> ch_eq_rearm: 0
> rx_out_of_buffer: 6880325
> rx_if_down_packets: 2062529
> rx_vport_unicast_packets: 517433716795
> rx_vport_unicast_bytes: 683464347301443
> tx_vport_unicast_packets: 677697453738
> tx_vport_unicast_bytes: 415788589663315
> rx_vport_multicast_packets: 208258309
> rx_vport_multicast_bytes: 14224046052
> tx_vport_multicast_packets: 21689
> tx_vport_multicast_bytes: 2158334
> rx_vport_broadcast_packets: 75838646
> rx_vport_broadcast_bytes: 4697944695
> tx_vport_broadcast_packets: 68730
> tx_vport_broadcast_bytes: 4123800
> rx_vport_rdma_unicast_packets: 0
> rx_vport_rdma_unicast_bytes: 0
> tx_vport_rdma_unicast_packets: 0
> tx_vport_rdma_unicast_bytes: 0
> rx_vport_rdma_multicast_packets: 0
> rx_vport_rdma_multicast_bytes: 0
> tx_vport_rdma_multicast_packets: 0
> tx_vport_rdma_multicast_bytes: 0
> tx_packets_phy: 677697543252
> rx_packets_phy: 521319491878
> rx_crc_errors_phy: 0
> tx_bytes_phy: 418499385791411
> rx_bytes_phy: 690322537017274
> tx_multicast_phy: 21689
> tx_broadcast_phy: 68730
> rx_multicast_phy: 208258305
> rx_broadcast_phy: 75838646
> rx_in_range_len_errors_phy: 4
> rx_out_of_range_len_phy: 0
> rx_oversize_pkts_phy: 0
> rx_symbol_err_phy: 0
> tx_mac_control_phy: 0
> rx_mac_control_phy: 0
> rx_unsupported_op_phy: 0
> rx_pause_ctrl_phy: 0
> tx_pause_ctrl_phy: 0
> rx_discards_phy: 3601449265
> tx_discards_phy: 0
> tx_errors_phy: 0
> rx_undersize_pkts_phy: 0
> rx_fragments_phy: 0
> rx_jabbers_phy: 0
> rx_64_bytes_phy: 1416456771
> rx_65_to_127_bytes_phy: 40750434737
> rx_128_to_255_bytes_phy: 11518110310
> rx_256_to_511_bytes_phy: 7055850637
> rx_512_to_1023_bytes_phy: 7811550424
> rx_1024_to_1518_bytes_phy: 265547564845
> rx_1519_to_2047_bytes_phy: 187219522899
> rx_2048_to_4095_bytes_phy: 0
> rx_4096_to_8191_bytes_phy: 0
> rx_8192_to_10239_bytes_phy: 0
> link_down_events_phy: 0
> rx_pcs_symbol_err_phy: 0
> rx_corrected_bits_phy: 0
> rx_pci_signal_integrity: 0
> tx_pci_signal_integrity: 48
> rx_prio0_bytes: 688807632117485
> rx_prio0_packets: 516310309931
> tx_prio0_bytes: 418499382756025
> tx_prio0_packets: 677697534982
> rx_prio1_bytes: 1497701612877
> rx_prio1_packets: 1206768094
> tx_prio1_bytes: 0
> tx_prio1_packets: 0
> rx_prio2_bytes: 112271227
> rx_prio2_packets: 337295
> tx_prio2_bytes: 0
> tx_prio2_packets: 0
> rx_prio3_bytes: 1165455555
> rx_prio3_packets: 1544310
> tx_prio3_bytes: 0
> tx_prio3_packets: 0
> rx_prio4_bytes: 161857240
> rx_prio4_packets: 341392
> tx_prio4_bytes: 0
> tx_prio4_packets: 0
> rx_prio5_bytes: 455031612
> rx_prio5_packets: 2861469
> tx_prio5_bytes: 0
> tx_prio5_packets: 0
> rx_prio6_bytes: 1873928697
> rx_prio6_packets: 5146981
> tx_prio6_bytes: 0
> tx_prio6_packets: 0
> rx_prio7_bytes: 13423452430
> rx_prio7_packets: 190724796
> tx_prio7_bytes: 0
> tx_prio7_packets: 0
> module_unplug: 0
> module_bus_stuck: 0
> module_high_temp: 0
> module_bad_shorted: 0
> ch0_events: 4252266777
> ch0_poll: 4330804273
> ch0_arm: 4120233182
> ch0_aff_change: 2
> ch0_eq_rearm: 0
> ch1_events: 3938415938
> ch1_poll: 4012621322
> ch1_arm: 3810131188
> ch1_aff_change: 2
> ch1_eq_rearm: 0
> ch2_events: 3897428860
> ch2_poll: 3973886848
> ch2_arm: 3773019397
> ch2_aff_change: 1
> ch2_eq_rearm: 0
> ch3_events: 4108000541
> ch3_poll: 4180139872
> ch3_arm: 3982093366
> ch3_aff_change: 1
> ch3_eq_rearm: 0
> ch4_events: 4652570079
> ch4_poll: 4720541090
> ch4_arm: 4524475054
> ch4_aff_change: 2
> ch4_eq_rearm: 0
> ch5_events: 3899177385
> ch5_poll: 3974274186
> ch5_arm: 3772299186
> ch5_aff_change: 2
> ch5_eq_rearm: 0
> ch6_events: 3915161350
> ch6_poll: 3992338199
> ch6_arm: 3794710989
> ch6_aff_change: 0
> ch6_eq_rearm: 0
> ch7_events: 4008175631
> ch7_poll: 4081321248
> ch7_arm: 3882263723
> ch7_aff_change: 0
> ch7_eq_rearm: 0
> ch8_events: 4207422352
> ch8_poll: 4276465449
> ch8_arm: 4077650366
> ch8_aff_change: 0
> ch8_eq_rearm: 0
> ch9_events: 4036491879
> ch9_poll: 4108975987
> ch9_arm: 3914493694
> ch9_aff_change: 0
> ch9_eq_rearm: 0
> ch10_events: 4066261595
> ch10_poll: 4134419606
> ch10_arm: 3936637711
> ch10_aff_change: 1
> ch10_eq_rearm: 0
> ch11_events: 4440494043
> ch11_poll: 4507578730
> ch11_arm: 4318629438
> ch11_aff_change: 0
> ch11_eq_rearm: 0
> ch12_events: 4066958252
> ch12_poll: 4130191506
> ch12_arm: 3934337782
> ch12_aff_change: 0
> ch12_eq_rearm: 0
> ch13_events: 4051309159
> ch13_poll: 4118864120
> ch13_arm: 3921011919
> ch13_aff_change: 0
> ch13_eq_rearm: 0
> ch14_events: 4321664800
> ch14_poll: 4382433680
> ch14_arm: 4186130552
> ch14_aff_change: 0
> ch14_eq_rearm: 0
> ch15_events: 4701102075
> ch15_poll: 4760373932
> ch15_arm: 4570151468
> ch15_aff_change: 0
> ch15_eq_rearm: 0
> ch16_events: 4311052687
> ch16_poll: 4345937129
> ch16_arm: 4170883819
> ch16_aff_change: 0
> ch16_eq_rearm: 0
> ch17_events: 4647570931
> ch17_poll: 4680218533
> ch17_arm: 4509426288
> ch17_aff_change: 0
> ch17_eq_rearm: 0
> ch18_events: 4598195702
> ch18_poll: 4631314898
> ch18_arm: 4457267084
> ch18_aff_change: 0
> ch18_eq_rearm: 0
> ch19_events: 4808094560
> ch19_poll: 4841368340
> ch19_arm: 4670604358
> ch19_aff_change: 0
> ch19_eq_rearm: 0
> ch20_events: 4240910605
> ch20_poll: 4276531502
> ch20_arm: 4101767278
> ch20_aff_change: 1
> ch20_eq_rearm: 0
> ch21_events: 4389371472
> ch21_poll: 4426870311
> ch21_arm: 4249339045
> ch21_aff_change: 2
> ch21_eq_rearm: 0
> ch22_events: 4282958754
> ch22_poll: 4319228073
> ch22_arm: 4145102991
> ch22_aff_change: 2
> ch22_eq_rearm: 0
> ch23_events: 4440196528
> ch23_poll: 4474090188
> ch23_arm: 4300837147
> ch23_aff_change: 2
> ch23_eq_rearm: 0
> ch24_events: 4326875785
> ch24_poll: 4364971263
> ch24_arm: 4186404526
> ch24_aff_change: 2
> ch24_eq_rearm: 0
> ch25_events: 4286528453
> ch25_poll: 4324089445
> ch25_arm: 4147222616
> ch25_aff_change: 3
> ch25_eq_rearm: 0
> ch26_events: 4098043104
> ch26_poll: 4138133745
> ch26_arm: 3967438971
> ch26_aff_change: 4
> ch26_eq_rearm: 0
> ch27_events: 4563302840
> ch27_poll: 4599441446
> ch27_arm: 4432182806
> ch27_aff_change: 4
> ch27_eq_rearm: 0
> ch28_events: 4
> ch28_poll: 4
> ch28_arm: 4
> ch28_aff_change: 0
> ch28_eq_rearm: 0
> ch29_events: 6
> ch29_poll: 6
> ch29_arm: 6
> ch29_aff_change: 0
> ch29_eq_rearm: 0
> ch30_events: 4
> ch30_poll: 4
> ch30_arm: 4
> ch30_aff_change: 0
> ch30_eq_rearm: 0
> ch31_events: 4
> ch31_poll: 4
> ch31_arm: 4
> ch31_aff_change: 0
> ch31_eq_rearm: 0
> ch32_events: 4
> ch32_poll: 4
> ch32_arm: 4
> ch32_aff_change: 0
> ch32_eq_rearm: 0
> ch33_events: 4
> ch33_poll: 4
> ch33_arm: 4
> ch33_aff_change: 0
> ch33_eq_rearm: 0
> ch34_events: 4
> ch34_poll: 4
> ch34_arm: 4
> ch34_aff_change: 0
> ch34_eq_rearm: 0
> ch35_events: 4
> ch35_poll: 4
> ch35_arm: 4
> ch35_aff_change: 0
> ch35_eq_rearm: 0
> ch36_events: 4
> ch36_poll: 4
> ch36_arm: 4
> ch36_aff_change: 0
> ch36_eq_rearm: 0
> ch37_events: 4
> ch37_poll: 4
> ch37_arm: 4
> ch37_aff_change: 0
> ch37_eq_rearm: 0
> ch38_events: 4
> ch38_poll: 4
> ch38_arm: 4
> ch38_aff_change: 0
> ch38_eq_rearm: 0
> ch39_events: 4
> ch39_poll: 4
> ch39_arm: 4
> ch39_aff_change: 0
> ch39_eq_rearm: 0
> ch40_events: 4
> ch40_poll: 4
> ch40_arm: 4
> ch40_aff_change: 0
> ch40_eq_rearm: 0
> ch41_events: 4
> ch41_poll: 4
> ch41_arm: 4
> ch41_aff_change: 0
> ch41_eq_rearm: 0
> ch42_events: 4
> ch42_poll: 4
> ch42_arm: 4
> ch42_aff_change: 0
> ch42_eq_rearm: 0
> ch43_events: 4
> ch43_poll: 4
> ch43_arm: 4
> ch43_aff_change: 0
> ch43_eq_rearm: 0
> ch44_events: 4
> ch44_poll: 4
> ch44_arm: 4
> ch44_aff_change: 0
> ch44_eq_rearm: 0
> ch45_events: 4
> ch45_poll: 4
> ch45_arm: 4
> ch45_aff_change: 0
> ch45_eq_rearm: 0
> ch46_events: 4
> ch46_poll: 4
> ch46_arm: 4
> ch46_aff_change: 0
> ch46_eq_rearm: 0
> ch47_events: 4
> ch47_poll: 4
> ch47_arm: 4
> ch47_aff_change: 0
> ch47_eq_rearm: 0
> ch48_events: 4
> ch48_poll: 4
> ch48_arm: 4
> ch48_aff_change: 0
> ch48_eq_rearm: 0
> ch49_events: 4
> ch49_poll: 4
> ch49_arm: 4
> ch49_aff_change: 0
> ch49_eq_rearm: 0
> ch50_events: 4
> ch50_poll: 4
> ch50_arm: 4
> ch50_aff_change: 0
> ch50_eq_rearm: 0
> ch51_events: 4
> ch51_poll: 4
> ch51_arm: 4
> ch51_aff_change: 0
> ch51_eq_rearm: 0
> ch52_events: 4
> ch52_poll: 4
> ch52_arm: 4
> ch52_aff_change: 0
> ch52_eq_rearm: 0
> ch53_events: 4
> ch53_poll: 4
> ch53_arm: 4
> ch53_aff_change: 0
> ch53_eq_rearm: 0
> ch54_events: 4
> ch54_poll: 4
> ch54_arm: 4
> ch54_aff_change: 0
> ch54_eq_rearm: 0
> ch55_events: 4
> ch55_poll: 4
> ch55_arm: 4
> ch55_aff_change: 0
> ch55_eq_rearm: 0
> rx0_packets: 21390033774
> rx0_bytes: 27326856299122
> rx0_csum_complete: 21339650906
> rx0_csum_unnecessary: 0
> rx0_csum_unnecessary_inner: 0
> rx0_csum_none: 50382868
> rx0_xdp_drop: 0
> rx0_xdp_redirect: 0
> rx0_lro_packets: 0
> rx0_lro_bytes: 0
> rx0_ecn_mark: 0
> rx0_removed_vlan_packets: 21390033774
> rx0_wqe_err: 0
> rx0_mpwqe_filler_cqes: 0
> rx0_mpwqe_filler_strides: 0
> rx0_buff_alloc_err: 0
> rx0_cqe_compress_blks: 481077641
> rx0_cqe_compress_pkts: 1085647489
> rx0_page_reuse: 0
> rx0_cache_reuse: 19050049
> rx0_cache_full: 10675964285
> rx0_cache_empty: 37376
> rx0_cache_busy: 10675966819
> rx0_cache_waive: 0
> rx0_congst_umr: 0
> rx0_arfs_err: 0
> rx0_xdp_tx_xmit: 0
> rx0_xdp_tx_full: 0
> rx0_xdp_tx_err: 0
> rx0_xdp_tx_cqes: 0
> rx1_packets: 19868919527
> rx1_bytes: 26149716991561
> rx1_csum_complete: 19868919527
> rx1_csum_unnecessary: 0
> rx1_csum_unnecessary_inner: 0
> rx1_csum_none: 0
> rx1_xdp_drop: 0
> rx1_xdp_redirect: 0
> rx1_lro_packets: 0
> rx1_lro_bytes: 0
> rx1_ecn_mark: 0
> rx1_removed_vlan_packets: 19868919527
> rx1_wqe_err: 0
> rx1_mpwqe_filler_cqes: 0
> rx1_mpwqe_filler_strides: 0
> rx1_buff_alloc_err: 0
> rx1_cqe_compress_blks: 420210560
> rx1_cqe_compress_pkts: 941233388
> rx1_page_reuse: 0
> rx1_cache_reuse: 46200002
> rx1_cache_full: 9888257242
> rx1_cache_empty: 37376
> rx1_cache_busy: 9888259746
> rx1_cache_waive: 0
> rx1_congst_umr: 0
> rx1_arfs_err: 0
> rx1_xdp_tx_xmit: 0
> rx1_xdp_tx_full: 0
> rx1_xdp_tx_err: 0
> rx1_xdp_tx_cqes: 0
> rx2_packets: 19575013662
> rx2_bytes: 25759818417945
> rx2_csum_complete: 19575013662
> rx2_csum_unnecessary: 0
> rx2_csum_unnecessary_inner: 0
> rx2_csum_none: 0
> rx2_xdp_drop: 0
> rx2_xdp_redirect: 0
> rx2_lro_packets: 0
> rx2_lro_bytes: 0
> rx2_ecn_mark: 0
> rx2_removed_vlan_packets: 19575013662
> rx2_wqe_err: 0
> rx2_mpwqe_filler_cqes: 0
> rx2_mpwqe_filler_strides: 0
> rx2_buff_alloc_err: 0
> rx2_cqe_compress_blks: 412345511
> rx2_cqe_compress_pkts: 923376167
> rx2_page_reuse: 0
> rx2_cache_reuse: 38837731
> rx2_cache_full: 9748666548
> rx2_cache_empty: 37376
> rx2_cache_busy: 9748669093
> rx2_cache_waive: 0
> rx2_congst_umr: 0
> rx2_arfs_err: 0
> rx2_xdp_tx_xmit: 0
> rx2_xdp_tx_full: 0
> rx2_xdp_tx_err: 0
> rx2_xdp_tx_cqes: 0
> rx3_packets: 19795911749
> rx3_bytes: 25969475566905
> rx3_csum_complete: 19795911749
> rx3_csum_unnecessary: 0
> rx3_csum_unnecessary_inner: 0
> rx3_csum_none: 0
> rx3_xdp_drop: 0
> rx3_xdp_redirect: 0
> rx3_lro_packets: 0
> rx3_lro_bytes: 0
> rx3_ecn_mark: 0
> rx3_removed_vlan_packets: 19795911749
> rx3_wqe_err: 0
> rx3_mpwqe_filler_cqes: 0
> rx3_mpwqe_filler_strides: 0
> rx3_buff_alloc_err: 0
> rx3_cqe_compress_blks: 416658765
> rx3_cqe_compress_pkts: 934986266
> rx3_page_reuse: 0
> rx3_cache_reuse: 34542124
> rx3_cache_full: 9863411232
> rx3_cache_empty: 37376
> rx3_cache_busy: 9863413732
> rx3_cache_waive: 0
> rx3_congst_umr: 0
> rx3_arfs_err: 0
> rx3_xdp_tx_xmit: 0
> rx3_xdp_tx_full: 0
> rx3_xdp_tx_err: 0
> rx3_xdp_tx_cqes: 0
> rx4_packets: 20445652378
> rx4_bytes: 26949065110265
> rx4_csum_complete: 20445652378
> rx4_csum_unnecessary: 0
> rx4_csum_unnecessary_inner: 0
> rx4_csum_none: 0
> rx4_xdp_drop: 0
> rx4_xdp_redirect: 0
> rx4_lro_packets: 0
> rx4_lro_bytes: 0
> rx4_ecn_mark: 0
> rx4_removed_vlan_packets: 20445652378
> rx4_wqe_err: 0
> rx4_mpwqe_filler_cqes: 0
> rx4_mpwqe_filler_strides: 0
> rx4_buff_alloc_err: 0
> rx4_cqe_compress_blks: 506085858
> rx4_cqe_compress_pkts: 1147860328
> rx4_page_reuse: 0
> rx4_cache_reuse: 10122542864
> rx4_cache_full: 100281206
> rx4_cache_empty: 37376
> rx4_cache_busy: 100283304
> rx4_cache_waive: 0
> rx4_congst_umr: 0
> rx4_arfs_err: 0
> rx4_xdp_tx_xmit: 0
> rx4_xdp_tx_full: 0
> rx4_xdp_tx_err: 0
> rx4_xdp_tx_cqes: 0
> rx5_packets: 19622362246
> rx5_bytes: 25843450982982
> rx5_csum_complete: 19622362246
> rx5_csum_unnecessary: 0
> rx5_csum_unnecessary_inner: 0
> rx5_csum_none: 0
> rx5_xdp_drop: 0
> rx5_xdp_redirect: 0
> rx5_lro_packets: 0
> rx5_lro_bytes: 0
> rx5_ecn_mark: 0
> rx5_removed_vlan_packets: 19622362246
> rx5_wqe_err: 0
> rx5_mpwqe_filler_cqes: 0
> rx5_mpwqe_filler_strides: 0
> rx5_buff_alloc_err: 0
> rx5_cqe_compress_blks: 422840924
> rx5_cqe_compress_pkts: 948005878
> rx5_page_reuse: 0
> rx5_cache_reuse: 31285453
> rx5_cache_full: 9779893117
> rx5_cache_empty: 37376
> rx5_cache_busy: 9779895647
> rx5_cache_waive: 0
> rx5_congst_umr: 0
> rx5_arfs_err: 0
> rx5_xdp_tx_xmit: 0
> rx5_xdp_tx_full: 0
> rx5_xdp_tx_err: 0
> rx5_xdp_tx_cqes: 0
> rx6_packets: 19788231278
> rx6_bytes: 25985783006486
> rx6_csum_complete: 19788231278
> rx6_csum_unnecessary: 0
> rx6_csum_unnecessary_inner: 0
> rx6_csum_none: 0
> rx6_xdp_drop: 0
> rx6_xdp_redirect: 0
> rx6_lro_packets: 0
> rx6_lro_bytes: 0
> rx6_ecn_mark: 0
> rx6_removed_vlan_packets: 19788231278
> rx6_wqe_err: 0
> rx6_mpwqe_filler_cqes: 0
> rx6_mpwqe_filler_strides: 0
> rx6_buff_alloc_err: 0
> rx6_cqe_compress_blks: 418799056
> rx6_cqe_compress_pkts: 938282685
> rx6_page_reuse: 0
> rx6_cache_reuse: 18114793
> rx6_cache_full: 9875998295
> rx6_cache_empty: 37376
> rx6_cache_busy: 9876000831
> rx6_cache_waive: 0
> rx6_congst_umr: 0
> rx6_arfs_err: 0
> rx6_xdp_tx_xmit: 0
> rx6_xdp_tx_full: 0
> rx6_xdp_tx_err: 0
> rx6_xdp_tx_cqes: 0
> rx7_packets: 19795759168
> rx7_bytes: 26085056586860
> rx7_csum_complete: 19795759168
> rx7_csum_unnecessary: 0
> rx7_csum_unnecessary_inner: 0
> rx7_csum_none: 0
> rx7_xdp_drop: 0
> rx7_xdp_redirect: 0
> rx7_lro_packets: 0
> rx7_lro_bytes: 0
> rx7_ecn_mark: 0
> rx7_removed_vlan_packets: 19795759168
> rx7_wqe_err: 0
> rx7_mpwqe_filler_cqes: 0
> rx7_mpwqe_filler_strides: 0
> rx7_buff_alloc_err: 0
> rx7_cqe_compress_blks: 413959224
> rx7_cqe_compress_pkts: 927675936
> rx7_page_reuse: 0
> rx7_cache_reuse: 23902990
> rx7_cache_full: 9873974042
> rx7_cache_empty: 37376
> rx7_cache_busy: 9873976574
> rx7_cache_waive: 0
> rx7_congst_umr: 0
> rx7_arfs_err: 0
> rx7_xdp_tx_xmit: 0
> rx7_xdp_tx_full: 0
> rx7_xdp_tx_err: 0
> rx7_xdp_tx_cqes: 0
> rx8_packets: 19963477439
> rx8_bytes: 26384640501789
> rx8_csum_complete: 19963477439
> rx8_csum_unnecessary: 0
> rx8_csum_unnecessary_inner: 0
> rx8_csum_none: 0
> rx8_xdp_drop: 0
> rx8_xdp_redirect: 0
> rx8_lro_packets: 0
> rx8_lro_bytes: 0
> rx8_ecn_mark: 0
> rx8_removed_vlan_packets: 19963477439
> rx8_wqe_err: 0
> rx8_mpwqe_filler_cqes: 0
> rx8_mpwqe_filler_strides: 0
> rx8_buff_alloc_err: 0
> rx8_cqe_compress_blks: 420422857
> rx8_cqe_compress_pkts: 942720292
> rx8_page_reuse: 0
> rx8_cache_reuse: 88181713
> rx8_cache_full: 9893554525
> rx8_cache_empty: 37376
> rx8_cache_busy: 9893556983
> rx8_cache_waive: 0
> rx8_congst_umr: 0
> rx8_arfs_err: 0
> rx8_xdp_tx_xmit: 0
> rx8_xdp_tx_full: 0
> rx8_xdp_tx_err: 0
> rx8_xdp_tx_cqes: 0
> rx9_packets: 19726642138
> rx9_bytes: 26063924286499
> rx9_csum_complete: 19726642138
> rx9_csum_unnecessary: 0
> rx9_csum_unnecessary_inner: 0
> rx9_csum_none: 0
> rx9_xdp_drop: 0
> rx9_xdp_redirect: 0
> rx9_lro_packets: 0
> rx9_lro_bytes: 0
> rx9_ecn_mark: 0
> rx9_removed_vlan_packets: 19726642138
> rx9_wqe_err: 0
> rx9_mpwqe_filler_cqes: 0
> rx9_mpwqe_filler_strides: 0
> rx9_buff_alloc_err: 0
> rx9_cqe_compress_blks: 424227411
> rx9_cqe_compress_pkts: 951534873
> rx9_page_reuse: 0
> rx9_cache_reuse: 482901440
> rx9_cache_full: 9380417487
> rx9_cache_empty: 37376
> rx9_cache_busy: 9380419608
> rx9_cache_waive: 0
> rx9_congst_umr: 0
> rx9_arfs_err: 0
> rx9_xdp_tx_xmit: 0
> rx9_xdp_tx_full: 0
> rx9_xdp_tx_err: 0
> rx9_xdp_tx_cqes: 0
> rx10_packets: 19901229170
> rx10_bytes: 26300854495044
> rx10_csum_complete: 19901229170
> rx10_csum_unnecessary: 0
> rx10_csum_unnecessary_inner: 0
> rx10_csum_none: 0
> rx10_xdp_drop: 0
> rx10_xdp_redirect: 0
> rx10_lro_packets: 0
> rx10_lro_bytes: 0
> rx10_ecn_mark: 0
> rx10_removed_vlan_packets: 19901229170
> rx10_wqe_err: 0
> rx10_mpwqe_filler_cqes: 0
> rx10_mpwqe_filler_strides: 0
> rx10_buff_alloc_err: 0
> rx10_cqe_compress_blks: 419082938
> rx10_cqe_compress_pkts: 940791347
> rx10_page_reuse: 0
> rx10_cache_reuse: 14896055
> rx10_cache_full: 9935715977
> rx10_cache_empty: 37376
> rx10_cache_busy: 9935718513
> rx10_cache_waive: 0
> rx10_congst_umr: 0
> rx10_arfs_err: 0
> rx10_xdp_tx_xmit: 0
> rx10_xdp_tx_full: 0
> rx10_xdp_tx_err: 0
> rx10_xdp_tx_cqes: 0
> rx11_packets: 20352190494
> rx11_bytes: 26851034425372
> rx11_csum_complete: 20352190494
> rx11_csum_unnecessary: 0
> rx11_csum_unnecessary_inner: 0
> rx11_csum_none: 0
> rx11_xdp_drop: 0
> rx11_xdp_redirect: 0
> rx11_lro_packets: 0
> rx11_lro_bytes: 0
> rx11_ecn_mark: 0
> rx11_removed_vlan_packets: 20352190494
> rx11_wqe_err: 0
> rx11_mpwqe_filler_cqes: 0
> rx11_mpwqe_filler_strides: 0
> rx11_buff_alloc_err: 0
> rx11_cqe_compress_blks: 501992147
> rx11_cqe_compress_pkts: 1140398610
> rx11_page_reuse: 0
> rx11_cache_reuse: 10071721531
> rx11_cache_full: 104371621
> rx11_cache_empty: 37376
> rx11_cache_busy: 104373697
> rx11_cache_waive: 0
> rx11_congst_umr: 0
> rx11_arfs_err: 0
> rx11_xdp_tx_xmit: 0
> rx11_xdp_tx_full: 0
> rx11_xdp_tx_err: 0
> rx11_xdp_tx_cqes: 0
> rx12_packets: 19934747149
> rx12_bytes: 26296478787829
> rx12_csum_complete: 19934747149
> rx12_csum_unnecessary: 0
> rx12_csum_unnecessary_inner: 0
> rx12_csum_none: 0
> rx12_xdp_drop: 0
> rx12_xdp_redirect: 0
> rx12_lro_packets: 0
> rx12_lro_bytes: 0
> rx12_ecn_mark: 0
> rx12_removed_vlan_packets: 19934747149
> rx12_wqe_err: 0
> rx12_mpwqe_filler_cqes: 0
> rx12_mpwqe_filler_strides: 0
> rx12_buff_alloc_err: 0
> rx12_cqe_compress_blks: 443350570
> rx12_cqe_compress_pkts: 995997220
> rx12_page_reuse: 0
> rx12_cache_reuse: 9864934174
> rx12_cache_full: 102437428
> rx12_cache_empty: 37376
> rx12_cache_busy: 102439382
> rx12_cache_waive: 0
> rx12_congst_umr: 0
> rx12_arfs_err: 0
> rx12_xdp_tx_xmit: 0
> rx12_xdp_tx_full: 0
> rx12_xdp_tx_err: 0
> rx12_xdp_tx_cqes: 0
> rx13_packets: 19866908096
> rx13_bytes: 26160931936186
> rx13_csum_complete: 19866908096
> rx13_csum_unnecessary: 0
> rx13_csum_unnecessary_inner: 0
> rx13_csum_none: 0
> rx13_xdp_drop: 0
> rx13_xdp_redirect: 0
> rx13_lro_packets: 0
> rx13_lro_bytes: 0
> rx13_ecn_mark: 0
> rx13_removed_vlan_packets: 19866908096
> rx13_wqe_err: 0
> rx13_mpwqe_filler_cqes: 0
> rx13_mpwqe_filler_strides: 0
> rx13_buff_alloc_err: 0
> rx13_cqe_compress_blks: 413640141
> rx13_cqe_compress_pkts: 926175066
> rx13_page_reuse: 0
> rx13_cache_reuse: 36358610
> rx13_cache_full: 9897092921
> rx13_cache_empty: 37376
> rx13_cache_busy: 9897095422
> rx13_cache_waive: 0
> rx13_congst_umr: 0
> rx13_arfs_err: 0
> rx13_xdp_tx_xmit: 0
> rx13_xdp_tx_full: 0
> rx13_xdp_tx_err: 0
> rx13_xdp_tx_cqes: 0
> rx14_packets: 20229035746
> rx14_bytes: 26655092809172
> rx14_csum_complete: 20229035746
> rx14_csum_unnecessary: 0
> rx14_csum_unnecessary_inner: 0
> rx14_csum_none: 0
> rx14_xdp_drop: 0
> rx14_xdp_redirect: 0
> rx14_lro_packets: 0
> rx14_lro_bytes: 0
> rx14_ecn_mark: 0
> rx14_removed_vlan_packets: 20229035746
> rx14_wqe_err: 0
> rx14_mpwqe_filler_cqes: 0
> rx14_mpwqe_filler_strides: 0
> rx14_buff_alloc_err: 0
> rx14_cqe_compress_blks: 460990337
> rx14_cqe_compress_pkts: 1041287948
> rx14_page_reuse: 0
> rx14_cache_reuse: 25649275
> rx14_cache_full: 10088866045
> rx14_cache_empty: 37376
> rx14_cache_busy: 10088868574
> rx14_cache_waive: 0
> rx14_congst_umr: 0
> rx14_arfs_err: 0
> rx14_xdp_tx_xmit: 0
> rx14_xdp_tx_full: 0
> rx14_xdp_tx_err: 0
> rx14_xdp_tx_cqes: 0
> rx15_packets: 20528177154
> rx15_bytes: 27029263893264
> rx15_csum_complete: 20528177154
> rx15_csum_unnecessary: 0
> rx15_csum_unnecessary_inner: 0
> rx15_csum_none: 0
> rx15_xdp_drop: 0
> rx15_xdp_redirect: 0
> rx15_lro_packets: 0
> rx15_lro_bytes: 0
> rx15_ecn_mark: 0
> rx15_removed_vlan_packets: 20528177154
> rx15_wqe_err: 0
> rx15_mpwqe_filler_cqes: 0
> rx15_mpwqe_filler_strides: 0
> rx15_buff_alloc_err: 0
> rx15_cqe_compress_blks: 476776176
> rx15_cqe_compress_pkts: 1076153263
> rx15_page_reuse: 0
> rx15_cache_reuse: 48426735
> rx15_cache_full: 10215659289
> rx15_cache_empty: 37376
> rx15_cache_busy: 10215661817
> rx15_cache_waive: 0
> rx15_congst_umr: 0
> rx15_arfs_err: 0
> rx15_xdp_tx_xmit: 0
> rx15_xdp_tx_full: 0
> rx15_xdp_tx_err: 0
> rx15_xdp_tx_cqes: 0
> rx16_packets: 16104078098
> rx16_bytes: 21256361789679
> rx16_csum_complete: 16104078098
> rx16_csum_unnecessary: 0
> rx16_csum_unnecessary_inner: 0
> rx16_csum_none: 0
> rx16_xdp_drop: 0
> rx16_xdp_redirect: 0
> rx16_lro_packets: 0
> rx16_lro_bytes: 0
> rx16_ecn_mark: 0
> rx16_removed_vlan_packets: 16104078098
> rx16_wqe_err: 0
> rx16_mpwqe_filler_cqes: 0
> rx16_mpwqe_filler_strides: 0
> rx16_buff_alloc_err: 0
> rx16_cqe_compress_blks: 352082054
> rx16_cqe_compress_pkts: 787161670
> rx16_page_reuse: 0
> rx16_cache_reuse: 25912567
> rx16_cache_full: 8026124051
> rx16_cache_empty: 37376
> rx16_cache_busy: 8026126465
> rx16_cache_waive: 0
> rx16_congst_umr: 0
> rx16_arfs_err: 0
> rx16_xdp_tx_xmit: 0
> rx16_xdp_tx_full: 0
> rx16_xdp_tx_err: 0
> rx16_xdp_tx_cqes: 0
> rx17_packets: 16314055017
> rx17_bytes: 21589139030173
> rx17_csum_complete: 16314055017
> rx17_csum_unnecessary: 0
> rx17_csum_unnecessary_inner: 0
> rx17_csum_none: 0
> rx17_xdp_drop: 0
> rx17_xdp_redirect: 0
> rx17_lro_packets: 0
> rx17_lro_bytes: 0
> rx17_ecn_mark: 0
> rx17_removed_vlan_packets: 16314055017
> rx17_wqe_err: 0
> rx17_mpwqe_filler_cqes: 0
> rx17_mpwqe_filler_strides: 0
> rx17_buff_alloc_err: 0
> rx17_cqe_compress_blks: 387834541
> rx17_cqe_compress_pkts: 871851081
> rx17_page_reuse: 0
> rx17_cache_reuse: 24021313
> rx17_cache_full: 8133003829
> rx17_cache_empty: 37376
> rx17_cache_busy: 8133006175
> rx17_cache_waive: 0
> rx17_congst_umr: 0
> rx17_arfs_err: 0
> rx17_xdp_tx_xmit: 0
> rx17_xdp_tx_full: 0
> rx17_xdp_tx_err: 0
> rx17_xdp_tx_cqes: 0
> rx18_packets: 16439016814
> rx18_bytes: 21648651917475
> rx18_csum_complete: 16439016814
> rx18_csum_unnecessary: 0
> rx18_csum_unnecessary_inner: 0
> rx18_csum_none: 0
> rx18_xdp_drop: 0
> rx18_xdp_redirect: 0
> rx18_lro_packets: 0
> rx18_lro_bytes: 0
> rx18_ecn_mark: 0
> rx18_removed_vlan_packets: 16439016814
> rx18_wqe_err: 0
> rx18_mpwqe_filler_cqes: 0
> rx18_mpwqe_filler_strides: 0
> rx18_buff_alloc_err: 0
> rx18_cqe_compress_blks: 375066666
> rx18_cqe_compress_pkts: 843563974
> rx18_page_reuse: 0
> rx18_cache_reuse: 8151064266
> rx18_cache_full: 68442025
> rx18_cache_empty: 37376
> rx18_cache_busy: 68444122
> rx18_cache_waive: 0
> rx18_congst_umr: 0
> rx18_arfs_err: 0
> rx18_xdp_tx_xmit: 0
> rx18_xdp_tx_full: 0
> rx18_xdp_tx_err: 0
> rx18_xdp_tx_cqes: 0
> rx19_packets: 16641223506
> rx19_bytes: 21964749940935
> rx19_csum_complete: 16641223506
> rx19_csum_unnecessary: 0
> rx19_csum_unnecessary_inner: 0
> rx19_csum_none: 0
> rx19_xdp_drop: 0
> rx19_xdp_redirect: 0
> rx19_lro_packets: 0
> rx19_lro_bytes: 0
> rx19_ecn_mark: 0
> rx19_removed_vlan_packets: 16641223506
> rx19_wqe_err: 0
> rx19_mpwqe_filler_cqes: 0
> rx19_mpwqe_filler_strides: 0
> rx19_buff_alloc_err: 0
> rx19_cqe_compress_blks: 387825932
> rx19_cqe_compress_pkts: 872266355
> rx19_page_reuse: 0
> rx19_cache_reuse: 116433620
> rx19_cache_full: 8204175954
> rx19_cache_empty: 37376
> rx19_cache_busy: 8204178120
> rx19_cache_waive: 0
> rx19_congst_umr: 0
> rx19_arfs_err: 0
> rx19_xdp_tx_xmit: 0
> rx19_xdp_tx_full: 0
> rx19_xdp_tx_err: 0
> rx19_xdp_tx_cqes: 0
> rx20_packets: 16206927741
> rx20_bytes: 21387447038430
> rx20_csum_complete: 16206927741
> rx20_csum_unnecessary: 0
> rx20_csum_unnecessary_inner: 0
> rx20_csum_none: 0
> rx20_xdp_drop: 0
> rx20_xdp_redirect: 0
> rx20_lro_packets: 0
> rx20_lro_bytes: 0
> rx20_ecn_mark: 0
> rx20_removed_vlan_packets: 16206927741
> rx20_wqe_err: 0
> rx20_mpwqe_filler_cqes: 0
> rx20_mpwqe_filler_strides: 0
> rx20_buff_alloc_err: 0
> rx20_cqe_compress_blks: 370144620
> rx20_cqe_compress_pkts: 829122671
> rx20_page_reuse: 0
> rx20_cache_reuse: 8053733744
> rx20_cache_full: 49728026
> rx20_cache_empty: 37376
> rx20_cache_busy: 49730116
> rx20_cache_waive: 0
> rx20_congst_umr: 0
> rx20_arfs_err: 0
> rx20_xdp_tx_xmit: 0
> rx20_xdp_tx_full: 0
> rx20_xdp_tx_err: 0
> rx20_xdp_tx_cqes: 0
> rx21_packets: 16562361314
> rx21_bytes: 21856653284356
> rx21_csum_complete: 16562361314
> rx21_csum_unnecessary: 0
> rx21_csum_unnecessary_inner: 0
> rx21_csum_none: 0
> rx21_xdp_drop: 0
> rx21_xdp_redirect: 0
> rx21_lro_packets: 0
> rx21_lro_bytes: 0
> rx21_ecn_mark: 0
> rx21_removed_vlan_packets: 16562361314
> rx21_wqe_err: 0
> rx21_mpwqe_filler_cqes: 0
> rx21_mpwqe_filler_strides: 0
> rx21_buff_alloc_err: 0
> rx21_cqe_compress_blks: 350790425
> rx21_cqe_compress_pkts: 783850729
> rx21_page_reuse: 0
> rx21_cache_reuse: 28077493
> rx21_cache_full: 8253100706
> rx21_cache_empty: 37376
> rx21_cache_busy: 8253103147
> rx21_cache_waive: 0
> rx21_congst_umr: 0
> rx21_arfs_err: 0
> rx21_xdp_tx_xmit: 0
> rx21_xdp_tx_full: 0
> rx21_xdp_tx_err: 0
> rx21_xdp_tx_cqes: 0
> rx22_packets: 16350307571
> rx22_bytes: 21408575325592
> rx22_csum_complete: 16350307571
> rx22_csum_unnecessary: 0
> rx22_csum_unnecessary_inner: 0
> rx22_csum_none: 0
> rx22_xdp_drop: 0
> rx22_xdp_redirect: 0
> rx22_lro_packets: 0
> rx22_lro_bytes: 0
> rx22_ecn_mark: 0
> rx22_removed_vlan_packets: 16350307571
> rx22_wqe_err: 0
> rx22_mpwqe_filler_cqes: 0
> rx22_mpwqe_filler_strides: 0
> rx22_buff_alloc_err: 0
> rx22_cqe_compress_blks: 353531065
> rx22_cqe_compress_pkts: 790814415
> rx22_page_reuse: 0
> rx22_cache_reuse: 16934343
> rx22_cache_full: 8158216889
> rx22_cache_empty: 37376
> rx22_cache_busy: 8158219417
> rx22_cache_waive: 0
> rx22_congst_umr: 0
> rx22_arfs_err: 0
> rx22_xdp_tx_xmit: 0
> rx22_xdp_tx_full: 0
> rx22_xdp_tx_err: 0
> rx22_xdp_tx_cqes: 0
> rx23_packets: 16019811764
> rx23_bytes: 21137182570985
> rx23_csum_complete: 16019811764
> rx23_csum_unnecessary: 0
> rx23_csum_unnecessary_inner: 0
> rx23_csum_none: 0
> rx23_xdp_drop: 0
> rx23_xdp_redirect: 0
> rx23_lro_packets: 0
> rx23_lro_bytes: 0
> rx23_ecn_mark: 0
> rx23_removed_vlan_packets: 16019811764
> rx23_wqe_err: 0
> rx23_mpwqe_filler_cqes: 0
> rx23_mpwqe_filler_strides: 0
> rx23_buff_alloc_err: 0
> rx23_cqe_compress_blks: 349733033
> rx23_cqe_compress_pkts: 781248862
> rx23_page_reuse: 0
> rx23_cache_reuse: 33422343
> rx23_cache_full: 7976481152
> rx23_cache_empty: 37376
> rx23_cache_busy: 7976483525
> rx23_cache_waive: 0
> rx23_congst_umr: 0
> rx23_arfs_err: 0
> rx23_xdp_tx_xmit: 0
> rx23_xdp_tx_full: 0
> rx23_xdp_tx_err: 0
> rx23_xdp_tx_cqes: 0
> rx24_packets: 16212040646
> rx24_bytes: 21393399325700
> rx24_csum_complete: 16212040646
> rx24_csum_unnecessary: 0
> rx24_csum_unnecessary_inner: 0
> rx24_csum_none: 0
> rx24_xdp_drop: 0
> rx24_xdp_redirect: 0
> rx24_lro_packets: 0
> rx24_lro_bytes: 0
> rx24_ecn_mark: 0
> rx24_removed_vlan_packets: 16212040646
> rx24_wqe_err: 0
> rx24_mpwqe_filler_cqes: 0
> rx24_mpwqe_filler_strides: 0
> rx24_buff_alloc_err: 0
> rx24_cqe_compress_blks: 379833752
> rx24_cqe_compress_pkts: 852020179
> rx24_page_reuse: 0
> rx24_cache_reuse: 8033552512
> rx24_cache_full: 72465843
> rx24_cache_empty: 37376
> rx24_cache_busy: 72467789
> rx24_cache_waive: 0
> rx24_congst_umr: 0
> rx24_arfs_err: 0
> rx24_xdp_tx_xmit: 0
> rx24_xdp_tx_full: 0
> rx24_xdp_tx_err: 0
> rx24_xdp_tx_cqes: 0
> rx25_packets: 16412186257
> rx25_bytes: 21651198388407
> rx25_csum_complete: 16412186257
> rx25_csum_unnecessary: 0
> rx25_csum_unnecessary_inner: 0
> rx25_csum_none: 0
> rx25_xdp_drop: 0
> rx25_xdp_redirect: 0
> rx25_lro_packets: 0
> rx25_lro_bytes: 0
> rx25_ecn_mark: 0
> rx25_removed_vlan_packets: 16412186257
> rx25_wqe_err: 0
> rx25_mpwqe_filler_cqes: 0
> rx25_mpwqe_filler_strides: 0
> rx25_buff_alloc_err: 0
> rx25_cqe_compress_blks: 383979685
> rx25_cqe_compress_pkts: 861985772
> rx25_page_reuse: 0
> rx25_cache_reuse: 8129807841
> rx25_cache_full: 76283342
> rx25_cache_empty: 37376
> rx25_cache_busy: 76285271
> rx25_cache_waive: 0
> rx25_congst_umr: 0
> rx25_arfs_err: 0
> rx25_xdp_tx_xmit: 0
> rx25_xdp_tx_full: 0
> rx25_xdp_tx_err: 0
> rx25_xdp_tx_cqes: 0
> rx26_packets: 16304310003
> rx26_bytes: 21571217538721
> rx26_csum_complete: 16304310003
> rx26_csum_unnecessary: 0
> rx26_csum_unnecessary_inner: 0
> rx26_csum_none: 0
> rx26_xdp_drop: 0
> rx26_xdp_redirect: 0
> rx26_lro_packets: 0
> rx26_lro_bytes: 0
> rx26_ecn_mark: 0
> rx26_removed_vlan_packets: 16304310003
> rx26_wqe_err: 0
> rx26_mpwqe_filler_cqes: 0
> rx26_mpwqe_filler_strides: 0
> rx26_buff_alloc_err: 0
> rx26_cqe_compress_blks: 353314041
> rx26_cqe_compress_pkts: 788838424
> rx26_page_reuse: 0
> rx26_cache_reuse: 19673790
> rx26_cache_full: 8132478659
> rx26_cache_empty: 37376
> rx26_cache_busy: 8132481198
> rx26_cache_waive: 0
> rx26_congst_umr: 0
> rx26_arfs_err: 0
> rx26_xdp_tx_xmit: 0
> rx26_xdp_tx_full: 0
> rx26_xdp_tx_err: 0
> rx26_xdp_tx_cqes: 0
> rx27_packets: 16171856079
> rx27_bytes: 21376891736540
> rx27_csum_complete: 16171856079
> rx27_csum_unnecessary: 0
> rx27_csum_unnecessary_inner: 0
> rx27_csum_none: 0
> rx27_xdp_drop: 0
> rx27_xdp_redirect: 0
> rx27_lro_packets: 0
> rx27_lro_bytes: 0
> rx27_ecn_mark: 0
> rx27_removed_vlan_packets: 16171856079
> rx27_wqe_err: 0
> rx27_mpwqe_filler_cqes: 0
> rx27_mpwqe_filler_strides: 0
> rx27_buff_alloc_err: 0
> rx27_cqe_compress_blks: 386632845
> rx27_cqe_compress_pkts: 869362576
> rx27_page_reuse: 0
> rx27_cache_reuse: 10070560
> rx27_cache_full: 8075854928
> rx27_cache_empty: 37376
> rx27_cache_busy: 8075857468
> rx27_cache_waive: 0
> rx27_congst_umr: 0
> rx27_arfs_err: 0
> rx27_xdp_tx_xmit: 0
> rx27_xdp_tx_full: 0
> rx27_xdp_tx_err: 0
> rx27_xdp_tx_cqes: 0
> rx28_packets: 0
> rx28_bytes: 0
> rx28_csum_complete: 0
> rx28_csum_unnecessary: 0
> rx28_csum_unnecessary_inner: 0
> rx28_csum_none: 0
> rx28_xdp_drop: 0
> rx28_xdp_redirect: 0
> rx28_lro_packets: 0
> rx28_lro_bytes: 0
> rx28_ecn_mark: 0
> rx28_removed_vlan_packets: 0
> rx28_wqe_err: 0
> rx28_mpwqe_filler_cqes: 0
> rx28_mpwqe_filler_strides: 0
> rx28_buff_alloc_err: 0
> rx28_cqe_compress_blks: 0
> rx28_cqe_compress_pkts: 0
> rx28_page_reuse: 0
> rx28_cache_reuse: 0
> rx28_cache_full: 0
> rx28_cache_empty: 2560
> rx28_cache_busy: 0
> rx28_cache_waive: 0
> rx28_congst_umr: 0
> rx28_arfs_err: 0
> rx28_xdp_tx_xmit: 0
> rx28_xdp_tx_full: 0
> rx28_xdp_tx_err: 0
> rx28_xdp_tx_cqes: 0
> rx29_packets: 0
> rx29_bytes: 0
> rx29_csum_complete: 0
> rx29_csum_unnecessary: 0
> rx29_csum_unnecessary_inner: 0
> rx29_csum_none: 0
> rx29_xdp_drop: 0
> rx29_xdp_redirect: 0
> rx29_lro_packets: 0
> rx29_lro_bytes: 0
> rx29_ecn_mark: 0
> rx29_removed_vlan_packets: 0
> rx29_wqe_err: 0
> rx29_mpwqe_filler_cqes: 0
> rx29_mpwqe_filler_strides: 0
> rx29_buff_alloc_err: 0
> rx29_cqe_compress_blks: 0
> rx29_cqe_compress_pkts: 0
> rx29_page_reuse: 0
> rx29_cache_reuse: 0
> rx29_cache_full: 0
> rx29_cache_empty: 2560
> rx29_cache_busy: 0
> rx29_cache_waive: 0
> rx29_congst_umr: 0
> rx29_arfs_err: 0
> rx29_xdp_tx_xmit: 0
> rx29_xdp_tx_full: 0
> rx29_xdp_tx_err: 0
> rx29_xdp_tx_cqes: 0
> rx30_packets: 0
> rx30_bytes: 0
> rx30_csum_complete: 0
> rx30_csum_unnecessary: 0
> rx30_csum_unnecessary_inner: 0
> rx30_csum_none: 0
> rx30_xdp_drop: 0
> rx30_xdp_redirect: 0
> rx30_lro_packets: 0
> rx30_lro_bytes: 0
> rx30_ecn_mark: 0
> rx30_removed_vlan_packets: 0
> rx30_wqe_err: 0
> rx30_mpwqe_filler_cqes: 0
> rx30_mpwqe_filler_strides: 0
> rx30_buff_alloc_err: 0
> rx30_cqe_compress_blks: 0
> rx30_cqe_compress_pkts: 0
> rx30_page_reuse: 0
> rx30_cache_reuse: 0
> rx30_cache_full: 0
> rx30_cache_empty: 2560
> rx30_cache_busy: 0
> rx30_cache_waive: 0
> rx30_congst_umr: 0
> rx30_arfs_err: 0
> rx30_xdp_tx_xmit: 0
> rx30_xdp_tx_full: 0
> rx30_xdp_tx_err: 0
> rx30_xdp_tx_cqes: 0
> rx31_packets: 0
> rx31_bytes: 0
> rx31_csum_complete: 0
> rx31_csum_unnecessary: 0
> rx31_csum_unnecessary_inner: 0
> rx31_csum_none: 0
> rx31_xdp_drop: 0
> rx31_xdp_redirect: 0
> rx31_lro_packets: 0
> rx31_lro_bytes: 0
> rx31_ecn_mark: 0
> rx31_removed_vlan_packets: 0
> rx31_wqe_err: 0
> rx31_mpwqe_filler_cqes: 0
> rx31_mpwqe_filler_strides: 0
> rx31_buff_alloc_err: 0
> rx31_cqe_compress_blks: 0
> rx31_cqe_compress_pkts: 0
> rx31_page_reuse: 0
> rx31_cache_reuse: 0
> rx31_cache_full: 0
> rx31_cache_empty: 2560
> rx31_cache_busy: 0
> rx31_cache_waive: 0
> rx31_congst_umr: 0
> rx31_arfs_err: 0
> rx31_xdp_tx_xmit: 0
> rx31_xdp_tx_full: 0
> rx31_xdp_tx_err: 0
> rx31_xdp_tx_cqes: 0
> rx32_packets: 0
> rx32_bytes: 0
> rx32_csum_complete: 0
> rx32_csum_unnecessary: 0
> rx32_csum_unnecessary_inner: 0
> rx32_csum_none: 0
> rx32_xdp_drop: 0
> rx32_xdp_redirect: 0
> rx32_lro_packets: 0
> rx32_lro_bytes: 0
> rx32_ecn_mark: 0
> rx32_removed_vlan_packets: 0
> rx32_wqe_err: 0
> rx32_mpwqe_filler_cqes: 0
> rx32_mpwqe_filler_strides: 0
> rx32_buff_alloc_err: 0
> rx32_cqe_compress_blks: 0
> rx32_cqe_compress_pkts: 0
> rx32_page_reuse: 0
> rx32_cache_reuse: 0
> rx32_cache_full: 0
> rx32_cache_empty: 2560
> rx32_cache_busy: 0
> rx32_cache_waive: 0
> rx32_congst_umr: 0
> rx32_arfs_err: 0
> rx32_xdp_tx_xmit: 0
> rx32_xdp_tx_full: 0
> rx32_xdp_tx_err: 0
> rx32_xdp_tx_cqes: 0
> rx33_packets: 0
> rx33_bytes: 0
> rx33_csum_complete: 0
> rx33_csum_unnecessary: 0
> rx33_csum_unnecessary_inner: 0
> rx33_csum_none: 0
> rx33_xdp_drop: 0
> rx33_xdp_redirect: 0
> rx33_lro_packets: 0
> rx33_lro_bytes: 0
> rx33_ecn_mark: 0
> rx33_removed_vlan_packets: 0
> rx33_wqe_err: 0
> rx33_mpwqe_filler_cqes: 0
> rx33_mpwqe_filler_strides: 0
> rx33_buff_alloc_err: 0
> rx33_cqe_compress_blks: 0
> rx33_cqe_compress_pkts: 0
> rx33_page_reuse: 0
> rx33_cache_reuse: 0
> rx33_cache_full: 0
> rx33_cache_empty: 2560
> rx33_cache_busy: 0
> rx33_cache_waive: 0
> rx33_congst_umr: 0
> rx33_arfs_err: 0
> rx33_xdp_tx_xmit: 0
> rx33_xdp_tx_full: 0
> rx33_xdp_tx_err: 0
> rx33_xdp_tx_cqes: 0
> rx34_packets: 0
> rx34_bytes: 0
> rx34_csum_complete: 0
> rx34_csum_unnecessary: 0
> rx34_csum_unnecessary_inner: 0
> rx34_csum_none: 0
> rx34_xdp_drop: 0
> rx34_xdp_redirect: 0
> rx34_lro_packets: 0
> rx34_lro_bytes: 0
> rx34_ecn_mark: 0
> rx34_removed_vlan_packets: 0
> rx34_wqe_err: 0
> rx34_mpwqe_filler_cqes: 0
> rx34_mpwqe_filler_strides: 0
> rx34_buff_alloc_err: 0
> rx34_cqe_compress_blks: 0
> rx34_cqe_compress_pkts: 0
> rx34_page_reuse: 0
> rx34_cache_reuse: 0
> rx34_cache_full: 0
> rx34_cache_empty: 2560
> rx34_cache_busy: 0
> rx34_cache_waive: 0
> rx34_congst_umr: 0
> rx34_arfs_err: 0
> rx34_xdp_tx_xmit: 0
> rx34_xdp_tx_full: 0
> rx34_xdp_tx_err: 0
> rx34_xdp_tx_cqes: 0
> rx35_packets: 0
> rx35_bytes: 0
> rx35_csum_complete: 0
> rx35_csum_unnecessary: 0
> rx35_csum_unnecessary_inner: 0
> rx35_csum_none: 0
> rx35_xdp_drop: 0
> rx35_xdp_redirect: 0
> rx35_lro_packets: 0
> rx35_lro_bytes: 0
> rx35_ecn_mark: 0
> rx35_removed_vlan_packets: 0
> rx35_wqe_err: 0
> rx35_mpwqe_filler_cqes: 0
> rx35_mpwqe_filler_strides: 0
> rx35_buff_alloc_err: 0
> rx35_cqe_compress_blks: 0
> rx35_cqe_compress_pkts: 0
> rx35_page_reuse: 0
> rx35_cache_reuse: 0
> rx35_cache_full: 0
> rx35_cache_empty: 2560
> rx35_cache_busy: 0
> rx35_cache_waive: 0
> rx35_congst_umr: 0
> rx35_arfs_err: 0
> rx35_xdp_tx_xmit: 0
> rx35_xdp_tx_full: 0
> rx35_xdp_tx_err: 0
> rx35_xdp_tx_cqes: 0
> rx36_packets: 0
> rx36_bytes: 0
> rx36_csum_complete: 0
> rx36_csum_unnecessary: 0
> rx36_csum_unnecessary_inner: 0
> rx36_csum_none: 0
> rx36_xdp_drop: 0
> rx36_xdp_redirect: 0
> rx36_lro_packets: 0
> rx36_lro_bytes: 0
> rx36_ecn_mark: 0
> rx36_removed_vlan_packets: 0
> rx36_wqe_err: 0
> rx36_mpwqe_filler_cqes: 0
> rx36_mpwqe_filler_strides: 0
> rx36_buff_alloc_err: 0
> rx36_cqe_compress_blks: 0
> rx36_cqe_compress_pkts: 0
> rx36_page_reuse: 0
> rx36_cache_reuse: 0
> rx36_cache_full: 0
> rx36_cache_empty: 2560
> rx36_cache_busy: 0
> rx36_cache_waive: 0
> rx36_congst_umr: 0
> rx36_arfs_err: 0
> rx36_xdp_tx_xmit: 0
> rx36_xdp_tx_full: 0
> rx36_xdp_tx_err: 0
> rx36_xdp_tx_cqes: 0
> rx37_packets: 0
> rx37_bytes: 0
> rx37_csum_complete: 0
> rx37_csum_unnecessary: 0
> rx37_csum_unnecessary_inner: 0
> rx37_csum_none: 0
> rx37_xdp_drop: 0
> rx37_xdp_redirect: 0
> rx37_lro_packets: 0
> rx37_lro_bytes: 0
> rx37_ecn_mark: 0
> rx37_removed_vlan_packets: 0
> rx37_wqe_err: 0
> rx37_mpwqe_filler_cqes: 0
> rx37_mpwqe_filler_strides: 0
> rx37_buff_alloc_err: 0
> rx37_cqe_compress_blks: 0
> rx37_cqe_compress_pkts: 0
> rx37_page_reuse: 0
> rx37_cache_reuse: 0
> rx37_cache_full: 0
> rx37_cache_empty: 2560
> rx37_cache_busy: 0
> rx37_cache_waive: 0
> rx37_congst_umr: 0
> rx37_arfs_err: 0
> rx37_xdp_tx_xmit: 0
> rx37_xdp_tx_full: 0
> rx37_xdp_tx_err: 0
> rx37_xdp_tx_cqes: 0
> rx38_packets: 0
> rx38_bytes: 0
> rx38_csum_complete: 0
> rx38_csum_unnecessary: 0
> rx38_csum_unnecessary_inner: 0
> rx38_csum_none: 0
> rx38_xdp_drop: 0
> rx38_xdp_redirect: 0
> rx38_lro_packets: 0
> rx38_lro_bytes: 0
> rx38_ecn_mark: 0
> rx38_removed_vlan_packets: 0
> rx38_wqe_err: 0
> rx38_mpwqe_filler_cqes: 0
> rx38_mpwqe_filler_strides: 0
> rx38_buff_alloc_err: 0
> rx38_cqe_compress_blks: 0
> rx38_cqe_compress_pkts: 0
> rx38_page_reuse: 0
> rx38_cache_reuse: 0
> rx38_cache_full: 0
> rx38_cache_empty: 2560
> rx38_cache_busy: 0
> rx38_cache_waive: 0
> rx38_congst_umr: 0
> rx38_arfs_err: 0
> rx38_xdp_tx_xmit: 0
> rx38_xdp_tx_full: 0
> rx38_xdp_tx_err: 0
> rx38_xdp_tx_cqes: 0
> rx39_packets: 0
> rx39_bytes: 0
> rx39_csum_complete: 0
> rx39_csum_unnecessary: 0
> rx39_csum_unnecessary_inner: 0
> rx39_csum_none: 0
> rx39_xdp_drop: 0
> rx39_xdp_redirect: 0
> rx39_lro_packets: 0
> rx39_lro_bytes: 0
> rx39_ecn_mark: 0
> rx39_removed_vlan_packets: 0
> rx39_wqe_err: 0
> rx39_mpwqe_filler_cqes: 0
> rx39_mpwqe_filler_strides: 0
> rx39_buff_alloc_err: 0
> rx39_cqe_compress_blks: 0
> rx39_cqe_compress_pkts: 0
> rx39_page_reuse: 0
> rx39_cache_reuse: 0
> rx39_cache_full: 0
> rx39_cache_empty: 2560
> rx39_cache_busy: 0
> rx39_cache_waive: 0
> rx39_congst_umr: 0
> rx39_arfs_err: 0
> rx39_xdp_tx_xmit: 0
> rx39_xdp_tx_full: 0
> rx39_xdp_tx_err: 0
> rx39_xdp_tx_cqes: 0
> rx40_packets: 0
> rx40_bytes: 0
> rx40_csum_complete: 0
> rx40_csum_unnecessary: 0
> rx40_csum_unnecessary_inner: 0
> rx40_csum_none: 0
> rx40_xdp_drop: 0
> rx40_xdp_redirect: 0
> rx40_lro_packets: 0
> rx40_lro_bytes: 0
> rx40_ecn_mark: 0
> rx40_removed_vlan_packets: 0
> rx40_wqe_err: 0
> rx40_mpwqe_filler_cqes: 0
> rx40_mpwqe_filler_strides: 0
> rx40_buff_alloc_err: 0
> rx40_cqe_compress_blks: 0
> rx40_cqe_compress_pkts: 0
> rx40_page_reuse: 0
> rx40_cache_reuse: 0
> rx40_cache_full: 0
> rx40_cache_empty: 2560
> rx40_cache_busy: 0
> rx40_cache_waive: 0
> rx40_congst_umr: 0
> rx40_arfs_err: 0
> rx40_xdp_tx_xmit: 0
> rx40_xdp_tx_full: 0
> rx40_xdp_tx_err: 0
> rx40_xdp_tx_cqes: 0
> rx41_packets: 0
> rx41_bytes: 0
> rx41_csum_complete: 0
> rx41_csum_unnecessary: 0
> rx41_csum_unnecessary_inner: 0
> rx41_csum_none: 0
> rx41_xdp_drop: 0
> rx41_xdp_redirect: 0
> rx41_lro_packets: 0
> rx41_lro_bytes: 0
> rx41_ecn_mark: 0
> rx41_removed_vlan_packets: 0
> rx41_wqe_err: 0
> rx41_mpwqe_filler_cqes: 0
> rx41_mpwqe_filler_strides: 0
> rx41_buff_alloc_err: 0
> rx41_cqe_compress_blks: 0
> rx41_cqe_compress_pkts: 0
> rx41_page_reuse: 0
> rx41_cache_reuse: 0
> rx41_cache_full: 0
> rx41_cache_empty: 2560
> rx41_cache_busy: 0
> rx41_cache_waive: 0
> rx41_congst_umr: 0
> rx41_arfs_err: 0
> rx41_xdp_tx_xmit: 0
> rx41_xdp_tx_full: 0
> rx41_xdp_tx_err: 0
> rx41_xdp_tx_cqes: 0
> rx42_packets: 0
> rx42_bytes: 0
> rx42_csum_complete: 0
> rx42_csum_unnecessary: 0
> rx42_csum_unnecessary_inner: 0
> rx42_csum_none: 0
> rx42_xdp_drop: 0
> rx42_xdp_redirect: 0
> rx42_lro_packets: 0
> rx42_lro_bytes: 0
> rx42_ecn_mark: 0
> rx42_removed_vlan_packets: 0
> rx42_wqe_err: 0
> rx42_mpwqe_filler_cqes: 0
> rx42_mpwqe_filler_strides: 0
> rx42_buff_alloc_err: 0
> rx42_cqe_compress_blks: 0
> rx42_cqe_compress_pkts: 0
> rx42_page_reuse: 0
> rx42_cache_reuse: 0
> rx42_cache_full: 0
> rx42_cache_empty: 2560
> rx42_cache_busy: 0
> rx42_cache_waive: 0
> rx42_congst_umr: 0
> rx42_arfs_err: 0
> rx42_xdp_tx_xmit: 0
> rx42_xdp_tx_full: 0
> rx42_xdp_tx_err: 0
> rx42_xdp_tx_cqes: 0
> rx43_packets: 0
> rx43_bytes: 0
> rx43_csum_complete: 0
> rx43_csum_unnecessary: 0
> rx43_csum_unnecessary_inner: 0
> rx43_csum_none: 0
> rx43_xdp_drop: 0
> rx43_xdp_redirect: 0
> rx43_lro_packets: 0
> rx43_lro_bytes: 0
> rx43_ecn_mark: 0
> rx43_removed_vlan_packets: 0
> rx43_wqe_err: 0
> rx43_mpwqe_filler_cqes: 0
> rx43_mpwqe_filler_strides: 0
> rx43_buff_alloc_err: 0
> rx43_cqe_compress_blks: 0
> rx43_cqe_compress_pkts: 0
> rx43_page_reuse: 0
> rx43_cache_reuse: 0
> rx43_cache_full: 0
> rx43_cache_empty: 2560
> rx43_cache_busy: 0
> rx43_cache_waive: 0
> rx43_congst_umr: 0
> rx43_arfs_err: 0
> rx43_xdp_tx_xmit: 0
> rx43_xdp_tx_full: 0
> rx43_xdp_tx_err: 0
> rx43_xdp_tx_cqes: 0
> rx44_packets: 0
> rx44_bytes: 0
> rx44_csum_complete: 0
> rx44_csum_unnecessary: 0
> rx44_csum_unnecessary_inner: 0
> rx44_csum_none: 0
> rx44_xdp_drop: 0
> rx44_xdp_redirect: 0
> rx44_lro_packets: 0
> rx44_lro_bytes: 0
> rx44_ecn_mark: 0
> rx44_removed_vlan_packets: 0
> rx44_wqe_err: 0
> rx44_mpwqe_filler_cqes: 0
> rx44_mpwqe_filler_strides: 0
> rx44_buff_alloc_err: 0
> rx44_cqe_compress_blks: 0
> rx44_cqe_compress_pkts: 0
> rx44_page_reuse: 0
> rx44_cache_reuse: 0
> rx44_cache_full: 0
> rx44_cache_empty: 2560
> rx44_cache_busy: 0
> rx44_cache_waive: 0
> rx44_congst_umr: 0
> rx44_arfs_err: 0
> rx44_xdp_tx_xmit: 0
> rx44_xdp_tx_full: 0
> rx44_xdp_tx_err: 0
> rx44_xdp_tx_cqes: 0
> rx45_packets: 0
> rx45_bytes: 0
> rx45_csum_complete: 0
> rx45_csum_unnecessary: 0
> rx45_csum_unnecessary_inner: 0
> rx45_csum_none: 0
> rx45_xdp_drop: 0
> rx45_xdp_redirect: 0
> rx45_lro_packets: 0
> rx45_lro_bytes: 0
> rx45_ecn_mark: 0
> rx45_removed_vlan_packets: 0
> rx45_wqe_err: 0
> rx45_mpwqe_filler_cqes: 0
> rx45_mpwqe_filler_strides: 0
> rx45_buff_alloc_err: 0
> rx45_cqe_compress_blks: 0
> rx45_cqe_compress_pkts: 0
> rx45_page_reuse: 0
> rx45_cache_reuse: 0
> rx45_cache_full: 0
> rx45_cache_empty: 2560
> rx45_cache_busy: 0
> rx45_cache_waive: 0
> rx45_congst_umr: 0
> rx45_arfs_err: 0
> rx45_xdp_tx_xmit: 0
> rx45_xdp_tx_full: 0
> rx45_xdp_tx_err: 0
> rx45_xdp_tx_cqes: 0
> rx46_packets: 0
> rx46_bytes: 0
> rx46_csum_complete: 0
> rx46_csum_unnecessary: 0
> rx46_csum_unnecessary_inner: 0
> rx46_csum_none: 0
> rx46_xdp_drop: 0
> rx46_xdp_redirect: 0
> rx46_lro_packets: 0
> rx46_lro_bytes: 0
> rx46_ecn_mark: 0
> rx46_removed_vlan_packets: 0
> rx46_wqe_err: 0
> rx46_mpwqe_filler_cqes: 0
> rx46_mpwqe_filler_strides: 0
> rx46_buff_alloc_err: 0
> rx46_cqe_compress_blks: 0
> rx46_cqe_compress_pkts: 0
> rx46_page_reuse: 0
> rx46_cache_reuse: 0
> rx46_cache_full: 0
> rx46_cache_empty: 2560
> rx46_cache_busy: 0
> rx46_cache_waive: 0
> rx46_congst_umr: 0
> rx46_arfs_err: 0
> rx46_xdp_tx_xmit: 0
> rx46_xdp_tx_full: 0
> rx46_xdp_tx_err: 0
> rx46_xdp_tx_cqes: 0
> rx47_packets: 0
> rx47_bytes: 0
> rx47_csum_complete: 0
> rx47_csum_unnecessary: 0
> rx47_csum_unnecessary_inner: 0
> rx47_csum_none: 0
> rx47_xdp_drop: 0
> rx47_xdp_redirect: 0
> rx47_lro_packets: 0
> rx47_lro_bytes: 0
> rx47_ecn_mark: 0
> rx47_removed_vlan_packets: 0
> rx47_wqe_err: 0
> rx47_mpwqe_filler_cqes: 0
> rx47_mpwqe_filler_strides: 0
> rx47_buff_alloc_err: 0
> rx47_cqe_compress_blks: 0
> rx47_cqe_compress_pkts: 0
> rx47_page_reuse: 0
> rx47_cache_reuse: 0
> rx47_cache_full: 0
> rx47_cache_empty: 2560
> rx47_cache_busy: 0
> rx47_cache_waive: 0
> rx47_congst_umr: 0
> rx47_arfs_err: 0
> rx47_xdp_tx_xmit: 0
> rx47_xdp_tx_full: 0
> rx47_xdp_tx_err: 0
> rx47_xdp_tx_cqes: 0
> rx48_packets: 0
> rx48_bytes: 0
> rx48_csum_complete: 0
> rx48_csum_unnecessary: 0
> rx48_csum_unnecessary_inner: 0
> rx48_csum_none: 0
> rx48_xdp_drop: 0
> rx48_xdp_redirect: 0
> rx48_lro_packets: 0
> rx48_lro_bytes: 0
> rx48_ecn_mark: 0
> rx48_removed_vlan_packets: 0
> rx48_wqe_err: 0
> rx48_mpwqe_filler_cqes: 0
> rx48_mpwqe_filler_strides: 0
> rx48_buff_alloc_err: 0
> rx48_cqe_compress_blks: 0
> rx48_cqe_compress_pkts: 0
> rx48_page_reuse: 0
> rx48_cache_reuse: 0
> rx48_cache_full: 0
> rx48_cache_empty: 2560
> rx48_cache_busy: 0
> rx48_cache_waive: 0
> rx48_congst_umr: 0
> rx48_arfs_err: 0
> rx48_xdp_tx_xmit: 0
> rx48_xdp_tx_full: 0
> rx48_xdp_tx_err: 0
> rx48_xdp_tx_cqes: 0
> rx49_packets: 0
> rx49_bytes: 0
> rx49_csum_complete: 0
> rx49_csum_unnecessary: 0
> rx49_csum_unnecessary_inner: 0
> rx49_csum_none: 0
> rx49_xdp_drop: 0
> rx49_xdp_redirect: 0
> rx49_lro_packets: 0
> rx49_lro_bytes: 0
> rx49_ecn_mark: 0
> rx49_removed_vlan_packets: 0
> rx49_wqe_err: 0
> rx49_mpwqe_filler_cqes: 0
> rx49_mpwqe_filler_strides: 0
> rx49_buff_alloc_err: 0
> rx49_cqe_compress_blks: 0
> rx49_cqe_compress_pkts: 0
> rx49_page_reuse: 0
> rx49_cache_reuse: 0
> rx49_cache_full: 0
> rx49_cache_empty: 2560
> rx49_cache_busy: 0
> rx49_cache_waive: 0
> rx49_congst_umr: 0
> rx49_arfs_err: 0
> rx49_xdp_tx_xmit: 0
> rx49_xdp_tx_full: 0
> rx49_xdp_tx_err: 0
> rx49_xdp_tx_cqes: 0
> rx50_packets: 0
> rx50_bytes: 0
> rx50_csum_complete: 0
> rx50_csum_unnecessary: 0
> rx50_csum_unnecessary_inner: 0
> rx50_csum_none: 0
> rx50_xdp_drop: 0
> rx50_xdp_redirect: 0
> rx50_lro_packets: 0
> rx50_lro_bytes: 0
> rx50_ecn_mark: 0
> rx50_removed_vlan_packets: 0
> rx50_wqe_err: 0
> rx50_mpwqe_filler_cqes: 0
> rx50_mpwqe_filler_strides: 0
> rx50_buff_alloc_err: 0
> rx50_cqe_compress_blks: 0
> rx50_cqe_compress_pkts: 0
> rx50_page_reuse: 0
> rx50_cache_reuse: 0
> rx50_cache_full: 0
> rx50_cache_empty: 2560
> rx50_cache_busy: 0
> rx50_cache_waive: 0
> rx50_congst_umr: 0
> rx50_arfs_err: 0
> rx50_xdp_tx_xmit: 0
> rx50_xdp_tx_full: 0
> rx50_xdp_tx_err: 0
> rx50_xdp_tx_cqes: 0
> rx51_packets: 0
> rx51_bytes: 0
> rx51_csum_complete: 0
> rx51_csum_unnecessary: 0
> rx51_csum_unnecessary_inner: 0
> rx51_csum_none: 0
> rx51_xdp_drop: 0
> rx51_xdp_redirect: 0
> rx51_lro_packets: 0
> rx51_lro_bytes: 0
> rx51_ecn_mark: 0
> rx51_removed_vlan_packets: 0
> rx51_wqe_err: 0
> rx51_mpwqe_filler_cqes: 0
> rx51_mpwqe_filler_strides: 0
> rx51_buff_alloc_err: 0
> rx51_cqe_compress_blks: 0
> rx51_cqe_compress_pkts: 0
> rx51_page_reuse: 0
> rx51_cache_reuse: 0
> rx51_cache_full: 0
> rx51_cache_empty: 2560
> rx51_cache_busy: 0
> rx51_cache_waive: 0
> rx51_congst_umr: 0
> rx51_arfs_err: 0
> rx51_xdp_tx_xmit: 0
> rx51_xdp_tx_full: 0
> rx51_xdp_tx_err: 0
> rx51_xdp_tx_cqes: 0
> rx52_packets: 0
> rx52_bytes: 0
> rx52_csum_complete: 0
> rx52_csum_unnecessary: 0
> rx52_csum_unnecessary_inner: 0
> rx52_csum_none: 0
> rx52_xdp_drop: 0
> rx52_xdp_redirect: 0
> rx52_lro_packets: 0
> rx52_lro_bytes: 0
> rx52_ecn_mark: 0
> rx52_removed_vlan_packets: 0
> rx52_wqe_err: 0
> rx52_mpwqe_filler_cqes: 0
> rx52_mpwqe_filler_strides: 0
> rx52_buff_alloc_err: 0
> rx52_cqe_compress_blks: 0
> rx52_cqe_compress_pkts: 0
> rx52_page_reuse: 0
> rx52_cache_reuse: 0
> rx52_cache_full: 0
> rx52_cache_empty: 2560
> rx52_cache_busy: 0
> rx52_cache_waive: 0
> rx52_congst_umr: 0
> rx52_arfs_err: 0
> rx52_xdp_tx_xmit: 0
> rx52_xdp_tx_full: 0
> rx52_xdp_tx_err: 0
> rx52_xdp_tx_cqes: 0
> rx53_packets: 0
> rx53_bytes: 0
> rx53_csum_complete: 0
> rx53_csum_unnecessary: 0
> rx53_csum_unnecessary_inner: 0
> rx53_csum_none: 0
> rx53_xdp_drop: 0
> rx53_xdp_redirect: 0
> rx53_lro_packets: 0
> rx53_lro_bytes: 0
> rx53_ecn_mark: 0
> rx53_removed_vlan_packets: 0
> rx53_wqe_err: 0
> rx53_mpwqe_filler_cqes: 0
> rx53_mpwqe_filler_strides: 0
> rx53_buff_alloc_err: 0
> rx53_cqe_compress_blks: 0
> rx53_cqe_compress_pkts: 0
> rx53_page_reuse: 0
> rx53_cache_reuse: 0
> rx53_cache_full: 0
> rx53_cache_empty: 2560
> rx53_cache_busy: 0
> rx53_cache_waive: 0
> rx53_congst_umr: 0
> rx53_arfs_err: 0
> rx53_xdp_tx_xmit: 0
> rx53_xdp_tx_full: 0
> rx53_xdp_tx_err: 0
> rx53_xdp_tx_cqes: 0
> rx54_packets: 0
> rx54_bytes: 0
> rx54_csum_complete: 0
> rx54_csum_unnecessary: 0
> rx54_csum_unnecessary_inner: 0
> rx54_csum_none: 0
> rx54_xdp_drop: 0
> rx54_xdp_redirect: 0
> rx54_lro_packets: 0
> rx54_lro_bytes: 0
> rx54_ecn_mark: 0
> rx54_removed_vlan_packets: 0
> rx54_wqe_err: 0
> rx54_mpwqe_filler_cqes: 0
> rx54_mpwqe_filler_strides: 0
> rx54_buff_alloc_err: 0
> rx54_cqe_compress_blks: 0
> rx54_cqe_compress_pkts: 0
> rx54_page_reuse: 0
> rx54_cache_reuse: 0
> rx54_cache_full: 0
> rx54_cache_empty: 2560
> rx54_cache_busy: 0
> rx54_cache_waive: 0
> rx54_congst_umr: 0
> rx54_arfs_err: 0
> rx54_xdp_tx_xmit: 0
> rx54_xdp_tx_full: 0
> rx54_xdp_tx_err: 0
> rx54_xdp_tx_cqes: 0
> rx55_packets: 0
> rx55_bytes: 0
> rx55_csum_complete: 0
> rx55_csum_unnecessary: 0
> rx55_csum_unnecessary_inner: 0
> rx55_csum_none: 0
> rx55_xdp_drop: 0
> rx55_xdp_redirect: 0
> rx55_lro_packets: 0
> rx55_lro_bytes: 0
> rx55_ecn_mark: 0
> rx55_removed_vlan_packets: 0
> rx55_wqe_err: 0
> rx55_mpwqe_filler_cqes: 0
> rx55_mpwqe_filler_strides: 0
> rx55_buff_alloc_err: 0
> rx55_cqe_compress_blks: 0
> rx55_cqe_compress_pkts: 0
> rx55_page_reuse: 0
> rx55_cache_reuse: 0
> rx55_cache_full: 0
> rx55_cache_empty: 2560
> rx55_cache_busy: 0
> rx55_cache_waive: 0
> rx55_congst_umr: 0
> rx55_arfs_err: 0
> rx55_xdp_tx_xmit: 0
> rx55_xdp_tx_full: 0
> rx55_xdp_tx_err: 0
> rx55_xdp_tx_cqes: 0
> tx0_packets: 24512439668
> tx0_bytes: 15287569052791
> tx0_tso_packets: 1536157106
> tx0_tso_bytes: 8571753637944
> tx0_tso_inner_packets: 0
> tx0_tso_inner_bytes: 0
> tx0_csum_partial: 2132156117
> tx0_csum_partial_inner: 0
> tx0_added_vlan_packets: 19906601448
> tx0_nop: 308098536
> tx0_csum_none: 17774445331
> tx0_stopped: 19625
> tx0_dropped: 0
> tx0_xmit_more: 67864870
> tx0_recover: 0
> tx0_cqes: 19838744246
> tx0_wake: 19624
> tx0_cqe_err: 0
> tx1_packets: 22598557053
> tx1_bytes: 13568850145010
> tx1_tso_packets: 1369529475
> tx1_tso_bytes: 7661777265382
> tx1_tso_inner_packets: 0
> tx1_tso_inner_bytes: 0
> tx1_csum_partial: 1884639496
> tx1_csum_partial_inner: 0
> tx1_added_vlan_packets: 18468333696
> tx1_nop: 281301783
> tx1_csum_none: 16583694200
> tx1_stopped: 19457
> tx1_dropped: 0
> tx1_xmit_more: 55170875
> tx1_recover: 0
> tx1_cqes: 18413169824
> tx1_wake: 19455
> tx1_cqe_err: 0
> tx2_packets: 22821611433
> tx2_bytes: 13752535163683
> tx2_tso_packets: 1396978825
> tx2_tso_bytes: 7774704508463
> tx2_tso_inner_packets: 0
> tx2_tso_inner_bytes: 0
> tx2_csum_partial: 1897834558
> tx2_csum_partial_inner: 0
> tx2_added_vlan_packets: 18641958085
> tx2_nop: 286934891
> tx2_csum_none: 16744123527
> tx2_stopped: 13214
> tx2_dropped: 0
> tx2_xmit_more: 61749446
> tx2_recover: 0
> tx2_cqes: 18580215654
> tx2_wake: 13214
> tx2_cqe_err: 0
> tx3_packets: 22580809948
> tx3_bytes: 13730542936609
> tx3_tso_packets: 1370434579
> tx3_tso_bytes: 7605636711455
> tx3_tso_inner_packets: 0
> tx3_tso_inner_bytes: 0
> tx3_csum_partial: 1865573748
> tx3_csum_partial_inner: 0
> tx3_added_vlan_packets: 18491873644
> tx3_nop: 281195875
> tx3_csum_none: 16626299896
> tx3_stopped: 12542
> tx3_dropped: 0
> tx3_xmit_more: 57681647
> tx3_recover: 0
> tx3_cqes: 18434198757
> tx3_wake: 12540
> tx3_cqe_err: 0
> tx4_packets: 27801801208
> tx4_bytes: 17058453171137
> tx4_tso_packets: 1740500105
> tx4_tso_bytes: 9474905622036
> tx4_tso_inner_packets: 0
> tx4_tso_inner_bytes: 0
> tx4_csum_partial: 2279225376
> tx4_csum_partial_inner: 0
> tx4_added_vlan_packets: 22744081633
> tx4_nop: 349753979
> tx4_csum_none: 20464856257
> tx4_stopped: 14816
> tx4_dropped: 0
> tx4_xmit_more: 65469322
> tx4_recover: 0
> tx4_cqes: 22678618972
> tx4_wake: 14816
> tx4_cqe_err: 0
> tx5_packets: 25099783024
> tx5_bytes: 14917740698381
> tx5_tso_packets: 1512988013
> tx5_tso_bytes: 8571208921023
> tx5_tso_inner_packets: 0
> tx5_tso_inner_bytes: 0
> tx5_csum_partial: 2078498561
> tx5_csum_partial_inner: 0
> tx5_added_vlan_packets: 20465533760
> tx5_nop: 312614719
> tx5_csum_none: 18387035199
> tx5_stopped: 4605
> tx5_dropped: 0
> tx5_xmit_more: 64188936
> tx5_recover: 0
> tx5_cqes: 20401350718
> tx5_wake: 4604
> tx5_cqe_err: 0
> tx6_packets: 25025504896
> tx6_bytes: 14908021946070
> tx6_tso_packets: 1515718977
> tx6_tso_bytes: 8511442522461
> tx6_tso_inner_packets: 0
> tx6_tso_inner_bytes: 0
> tx6_csum_partial: 2056378610
> tx6_csum_partial_inner: 0
> tx6_added_vlan_packets: 20434066400
> tx6_nop: 310594020
> tx6_csum_none: 18377687790
> tx6_stopped: 15234
> tx6_dropped: 0
> tx6_xmit_more: 61130422
> tx6_recover: 0
> tx6_cqes: 20372943611
> tx6_wake: 15234
> tx6_cqe_err: 0
> tx7_packets: 25457096169
> tx7_bytes: 15456289446172
> tx7_tso_packets: 1553342799
> tx7_tso_bytes: 8764550988105
> tx7_tso_inner_packets: 0
> tx7_tso_inner_bytes: 0
> tx7_csum_partial: 2105765233
> tx7_csum_partial_inner: 0
> tx7_added_vlan_packets: 20720382377
> tx7_nop: 319044853
> tx7_csum_none: 18614617145
> tx7_stopped: 18745
> tx7_dropped: 0
> tx7_xmit_more: 57050107
> tx7_recover: 0
> tx7_cqes: 20663340775
> tx7_wake: 18746
> tx7_cqe_err: 0
> tx8_packets: 25389771649
> tx8_bytes: 15225503883962
> tx8_tso_packets: 1563367648
> tx8_tso_bytes: 8710384514258
> tx8_tso_inner_packets: 0
> tx8_tso_inner_bytes: 0
> tx8_csum_partial: 2106586634
> tx8_csum_partial_inner: 0
> tx8_added_vlan_packets: 20704676274
> tx8_nop: 318149261
> tx8_csum_none: 18598089640
> tx8_stopped: 4733
> tx8_dropped: 0
> tx8_xmit_more: 61014317
> tx8_recover: 0
> tx8_cqes: 20643667301
> tx8_wake: 4735
> tx8_cqe_err: 0
> tx9_packets: 25521500166
> tx9_bytes: 15302293145755
> tx9_tso_packets: 1546316697
> tx9_tso_bytes: 8770688145926
> tx9_tso_inner_packets: 0
> tx9_tso_inner_bytes: 0
> tx9_csum_partial: 2097652880
> tx9_csum_partial_inner: 0
> tx9_added_vlan_packets: 20778408432
> tx9_nop: 318538543
> tx9_csum_none: 18680755556
> tx9_stopped: 16118
> tx9_dropped: 0
> tx9_xmit_more: 68509728
> tx9_recover: 0
> tx9_cqes: 20709906498
> tx9_wake: 16118
> tx9_cqe_err: 0
> tx10_packets: 25451605829
> tx10_bytes: 15386896170792
> tx10_tso_packets: 1576473520
> tx10_tso_bytes: 8880888676383
> tx10_tso_inner_packets: 0
> tx10_tso_inner_bytes: 0
> tx10_csum_partial: 2129796141
> tx10_csum_partial_inner: 0
> tx10_added_vlan_packets: 20659622590
> tx10_nop: 319117433
> tx10_csum_none: 18529826450
> tx10_stopped: 20187
> tx10_dropped: 0
> tx10_xmit_more: 58892184
> tx10_recover: 0
> tx10_cqes: 20600737739
> tx10_wake: 20188
> tx10_cqe_err: 0
> tx11_packets: 27008919793
> tx11_bytes: 16587719213058
> tx11_tso_packets: 1734884654
> tx11_tso_bytes: 9475681471870
> tx11_tso_inner_packets: 0
> tx11_tso_inner_bytes: 0
> tx11_csum_partial: 2296162292
> tx11_csum_partial_inner: 0
> tx11_added_vlan_packets: 21943096263
> tx11_nop: 344188182
> tx11_csum_none: 19646933971
> tx11_stopped: 9703
> tx11_dropped: 0
> tx11_xmit_more: 66530718
> tx11_recover: 0
> tx11_cqes: 21876571667
> tx11_wake: 9704
> tx11_cqe_err: 0
> tx12_packets: 25969493269
> tx12_bytes: 15980767963416
> tx12_tso_packets: 1671396456
> tx12_tso_bytes: 9268973672821
> tx12_tso_inner_packets: 0
> tx12_tso_inner_bytes: 0
> tx12_csum_partial: 2243809182
> tx12_csum_partial_inner: 0
> tx12_added_vlan_packets: 20980642456
> tx12_nop: 330241007
> tx12_csum_none: 18736833276
> tx12_stopped: 10341
> tx12_dropped: 0
> tx12_xmit_more: 57834100
> tx12_recover: 0
> tx12_cqes: 20922815079
> tx12_wake: 10342
> tx12_cqe_err: 0
> tx13_packets: 25332762261
> tx13_bytes: 15353213283280
> tx13_tso_packets: 1577433599
> tx13_tso_bytes: 8785240284281
> tx13_tso_inner_packets: 0
> tx13_tso_inner_bytes: 0
> tx13_csum_partial: 2110640515
> tx13_csum_partial_inner: 0
> tx13_added_vlan_packets: 20605670910
> tx13_nop: 319805741
> tx13_csum_none: 18495030395
> tx13_stopped: 7006
> tx13_dropped: 0
> tx13_xmit_more: 58314402
> tx13_recover: 0
> tx13_cqes: 20547362770
> tx13_wake: 7008
> tx13_cqe_err: 0
> tx14_packets: 26333743548
> tx14_bytes: 16070719060573
> tx14_tso_packets: 1677922970
> tx14_tso_bytes: 9240299765487
> tx14_tso_inner_packets: 0
> tx14_tso_inner_bytes: 0
> tx14_csum_partial: 2215668906
> tx14_csum_partial_inner: 0
> tx14_added_vlan_packets: 21384410786
> tx14_nop: 332734939
> tx14_csum_none: 19168741880
> tx14_stopped: 13160
> tx14_dropped: 0
> tx14_xmit_more: 57650391
> tx14_recover: 0
> tx14_cqes: 21326767783
> tx14_wake: 13161
> tx14_cqe_err: 0
> tx15_packets: 26824968971
> tx15_bytes: 16687994233452
> tx15_tso_packets: 1755745052
> tx15_tso_bytes: 9533814012441
> tx15_tso_inner_packets: 0
> tx15_tso_inner_bytes: 0
> tx15_csum_partial: 2304778064
> tx15_csum_partial_inner: 0
> tx15_added_vlan_packets: 21740906107
> tx15_nop: 344143287
> tx15_csum_none: 19436128058
> tx15_stopped: 75
> tx15_dropped: 0
> tx15_xmit_more: 63325832
> tx15_recover: 0
> tx15_cqes: 21677585345
> tx15_wake: 74
> tx15_cqe_err: 0
> tx16_packets: 24488158946
> tx16_bytes: 15027415004570
> tx16_tso_packets: 1559127391
> tx16_tso_bytes: 8658691917845
> tx16_tso_inner_packets: 0
> tx16_tso_inner_bytes: 0
> tx16_csum_partial: 2075856395
> tx16_csum_partial_inner: 0
> tx16_added_vlan_packets: 19835695731
> tx16_nop: 308464189
> tx16_csum_none: 17759839340
> tx16_stopped: 4567
> tx16_dropped: 0
> tx16_xmit_more: 62631422
> tx16_recover: 0
> tx16_cqes: 19773070012
> tx16_wake: 4568
> tx16_cqe_err: 0
> tx17_packets: 24700413784
> tx17_bytes: 15216529713715
> tx17_tso_packets: 1597555108
> tx17_tso_bytes: 8773728661243
> tx17_tso_inner_packets: 0
> tx17_tso_inner_bytes: 0
> tx17_csum_partial: 2127177297
> tx17_csum_partial_inner: 0
> tx17_added_vlan_packets: 20003144561
> tx17_nop: 313356918
> tx17_csum_none: 17875967264
> tx17_stopped: 12572
> tx17_dropped: 0
> tx17_xmit_more: 62742980
> tx17_recover: 0
> tx17_cqes: 19940407615
> tx17_wake: 12573
> tx17_cqe_err: 0
> tx18_packets: 24887710046
> tx18_bytes: 15245034034664
> tx18_tso_packets: 1582550520
> tx18_tso_bytes: 8782692335483
> tx18_tso_inner_packets: 0
> tx18_tso_inner_bytes: 0
> tx18_csum_partial: 2084514331
> tx18_csum_partial_inner: 0
> tx18_added_vlan_packets: 20173879181
> tx18_nop: 314818702
> tx18_csum_none: 18089364850
> tx18_stopped: 21366
> tx18_dropped: 0
> tx18_xmit_more: 62485819
> tx18_recover: 0
> tx18_cqes: 20111400935
> tx18_wake: 21366
> tx18_cqe_err: 0
> tx19_packets: 24831057648
> tx19_bytes: 15164663890576
> tx19_tso_packets: 1599135489
> tx19_tso_bytes: 8756045449746
> tx19_tso_inner_packets: 0
> tx19_tso_inner_bytes: 0
> tx19_csum_partial: 2119746608
> tx19_csum_partial_inner: 0
> tx19_added_vlan_packets: 20143573903
> tx19_nop: 316966450
> tx19_csum_none: 18023827295
> tx19_stopped: 11431
> tx19_dropped: 0
> tx19_xmit_more: 57535904
> tx19_recover: 0
> tx19_cqes: 20086045325
> tx19_wake: 11431
> tx19_cqe_err: 0
> tx20_packets: 21943735263
> tx20_bytes: 13528749492187
> tx20_tso_packets: 1390048103
> tx20_tso_bytes: 7629058809637
> tx20_tso_inner_packets: 0
> tx20_tso_inner_bytes: 0
> tx20_csum_partial: 1848533941
> tx20_csum_partial_inner: 0
> tx20_added_vlan_packets: 17861417651
> tx20_nop: 276840365
> tx20_csum_none: 16012883710
> tx20_stopped: 38457
> tx20_dropped: 0
> tx20_xmit_more: 57042753
> tx20_recover: 0
> tx20_cqes: 17804384839
> tx20_wake: 38457
> tx20_cqe_err: 0
> tx21_packets: 21476926958
> tx21_bytes: 13096410597896
> tx21_tso_packets: 1367724090
> tx21_tso_bytes: 7568364585127
> tx21_tso_inner_packets: 0
> tx21_tso_inner_bytes: 0
> tx21_csum_partial: 1830570727
> tx21_csum_partial_inner: 0
> tx21_added_vlan_packets: 17421087814
> tx21_nop: 270611519
> tx21_csum_none: 15590517087
> tx21_stopped: 31213
> tx21_dropped: 0
> tx21_xmit_more: 60305389
> tx21_recover: 0
> tx21_cqes: 17360791205
> tx21_wake: 31213
> tx21_cqe_err: 0
> tx22_packets: 21819106444
> tx22_bytes: 13492871887100
> tx22_tso_packets: 1387002018
> tx22_tso_bytes: 7617705727669
> tx22_tso_inner_packets: 0
> tx22_tso_inner_bytes: 0
> tx22_csum_partial: 1853632107
> tx22_csum_partial_inner: 0
> tx22_added_vlan_packets: 17743255447
> tx22_nop: 274820992
> tx22_csum_none: 15889623340
> tx22_stopped: 24814
> tx22_dropped: 0
> tx22_xmit_more: 60811304
> tx22_recover: 0
> tx22_cqes: 17682451111
> tx22_wake: 24815
> tx22_cqe_err: 0
> tx23_packets: 21830455800
> tx23_bytes: 13427551902532
> tx23_tso_packets: 1388556038
> tx23_tso_bytes: 7604040587125
> tx23_tso_inner_packets: 0
> tx23_tso_inner_bytes: 0
> tx23_csum_partial: 1850819694
> tx23_csum_partial_inner: 0
> tx23_added_vlan_packets: 17761271122
> tx23_nop: 275142775
> tx23_csum_none: 15910451428
> tx23_stopped: 29899
> tx23_dropped: 0
> tx23_xmit_more: 58924909
> tx23_recover: 0
> tx23_cqes: 17702355187
> tx23_wake: 29898
> tx23_cqe_err: 0
> tx24_packets: 21961484213
> tx24_bytes: 13531373062497
> tx24_tso_packets: 1394697504
> tx24_tso_bytes: 7663866609308
> tx24_tso_inner_packets: 0
> tx24_tso_inner_bytes: 0
> tx24_csum_partial: 1857072074
> tx24_csum_partial_inner: 0
> tx24_added_vlan_packets: 17856887568
> tx24_nop: 276352855
> tx24_csum_none: 15999815494
> tx24_stopped: 33924
> tx24_dropped: 0
> tx24_xmit_more: 63992426
> tx24_recover: 0
> tx24_cqes: 17792905243
> tx24_wake: 33923
> tx24_cqe_err: 0
> tx25_packets: 21853593838
> tx25_bytes: 13357487830519
> tx25_tso_packets: 1398822411
> tx25_tso_bytes: 7691191518838
> tx25_tso_inner_packets: 0
> tx25_tso_inner_bytes: 0
> tx25_csum_partial: 1869483109
> tx25_csum_partial_inner: 0
> tx25_added_vlan_packets: 17734634614
> tx25_nop: 276327643
> tx25_csum_none: 15865151505
> tx25_stopped: 38651
> tx25_dropped: 0
> tx25_xmit_more: 56410535
> tx25_recover: 0
> tx25_cqes: 17678234537
> tx25_wake: 38650
> tx25_cqe_err: 0
> tx26_packets: 21480261205
> tx26_bytes: 13148973015935
> tx26_tso_packets: 1348132284
> tx26_tso_bytes: 7523489481775
> tx26_tso_inner_packets: 0
> tx26_tso_inner_bytes: 0
> tx26_csum_partial: 1839740745
> tx26_csum_partial_inner: 0
> tx26_added_vlan_packets: 17430592911
> tx26_nop: 270367836
> tx26_csum_none: 15590852166
> tx26_stopped: 34044
> tx26_dropped: 0
> tx26_xmit_more: 59870114
> tx26_recover: 0
> tx26_cqes: 17370736612
> tx26_wake: 34043
> tx26_cqe_err: 0
> tx27_packets: 22694273108
> tx27_bytes: 14135473431004
> tx27_tso_packets: 1418371875
> tx27_tso_bytes: 7784842263038
> tx27_tso_inner_packets: 0
> tx27_tso_inner_bytes: 0
> tx27_csum_partial: 1919170584
> tx27_csum_partial_inner: 0
> tx27_added_vlan_packets: 18520826023
> tx27_nop: 286296272
> tx27_csum_none: 16601655439
> tx27_stopped: 38125
> tx27_dropped: 0
> tx27_xmit_more: 72749775
> tx27_recover: 0
> tx27_cqes: 18448090270
> tx27_wake: 38127
> tx27_cqe_err: 0
> tx28_packets: 0
> tx28_bytes: 0
> tx28_tso_packets: 0
> tx28_tso_bytes: 0
> tx28_tso_inner_packets: 0
> tx28_tso_inner_bytes: 0
> tx28_csum_partial: 0
> tx28_csum_partial_inner: 0
> tx28_added_vlan_packets: 0
> tx28_nop: 0
> tx28_csum_none: 0
> tx28_stopped: 0
> tx28_dropped: 0
> tx28_xmit_more: 0
> tx28_recover: 0
> tx28_cqes: 0
> tx28_wake: 0
> tx28_cqe_err: 0
> tx29_packets: 3
> tx29_bytes: 266
> tx29_tso_packets: 0
> tx29_tso_bytes: 0
> tx29_tso_inner_packets: 0
> tx29_tso_inner_bytes: 0
> tx29_csum_partial: 0
> tx29_csum_partial_inner: 0
> tx29_added_vlan_packets: 0
> tx29_nop: 0
> tx29_csum_none: 3
> tx29_stopped: 0
> tx29_dropped: 0
> tx29_xmit_more: 1
> tx29_recover: 0
> tx29_cqes: 2
> tx29_wake: 0
> tx29_cqe_err: 0
> tx30_packets: 0
> tx30_bytes: 0
> tx30_tso_packets: 0
> tx30_tso_bytes: 0
> tx30_tso_inner_packets: 0
> tx30_tso_inner_bytes: 0
> tx30_csum_partial: 0
> tx30_csum_partial_inner: 0
> tx30_added_vlan_packets: 0
> tx30_nop: 0
> tx30_csum_none: 0
> tx30_stopped: 0
> tx30_dropped: 0
> tx30_xmit_more: 0
> tx30_recover: 0
> tx30_cqes: 0
> tx30_wake: 0
> tx30_cqe_err: 0
> tx31_packets: 0
> tx31_bytes: 0
> tx31_tso_packets: 0
> tx31_tso_bytes: 0
> tx31_tso_inner_packets: 0
> tx31_tso_inner_bytes: 0
> tx31_csum_partial: 0
> tx31_csum_partial_inner: 0
> tx31_added_vlan_packets: 0
> tx31_nop: 0
> tx31_csum_none: 0
> tx31_stopped: 0
> tx31_dropped: 0
> tx31_xmit_more: 0
> tx31_recover: 0
> tx31_cqes: 0
> tx31_wake: 0
> tx31_cqe_err: 0
> tx32_packets: 0
> tx32_bytes: 0
> tx32_tso_packets: 0
> tx32_tso_bytes: 0
> tx32_tso_inner_packets: 0
> tx32_tso_inner_bytes: 0
> tx32_csum_partial: 0
> tx32_csum_partial_inner: 0
> tx32_added_vlan_packets: 0
> tx32_nop: 0
> tx32_csum_none: 0
> tx32_stopped: 0
> tx32_dropped: 0
> tx32_xmit_more: 0
> tx32_recover: 0
> tx32_cqes: 0
> tx32_wake: 0
> tx32_cqe_err: 0
> tx33_packets: 0
> tx33_bytes: 0
> tx33_tso_packets: 0
> tx33_tso_bytes: 0
> tx33_tso_inner_packets: 0
> tx33_tso_inner_bytes: 0
> tx33_csum_partial: 0
> tx33_csum_partial_inner: 0
> tx33_added_vlan_packets: 0
> tx33_nop: 0
> tx33_csum_none: 0
> tx33_stopped: 0
> tx33_dropped: 0
> tx33_xmit_more: 0
> tx33_recover: 0
> tx33_cqes: 0
> tx33_wake: 0
> tx33_cqe_err: 0
> tx34_packets: 0
> tx34_bytes: 0
> tx34_tso_packets: 0
> tx34_tso_bytes: 0
> tx34_tso_inner_packets: 0
> tx34_tso_inner_bytes: 0
> tx34_csum_partial: 0
> tx34_csum_partial_inner: 0
> tx34_added_vlan_packets: 0
> tx34_nop: 0
> tx34_csum_none: 0
> tx34_stopped: 0
> tx34_dropped: 0
> tx34_xmit_more: 0
> tx34_recover: 0
> tx34_cqes: 0
> tx34_wake: 0
> tx34_cqe_err: 0
> tx35_packets: 0
> tx35_bytes: 0
> tx35_tso_packets: 0
> tx35_tso_bytes: 0
> tx35_tso_inner_packets: 0
> tx35_tso_inner_bytes: 0
> tx35_csum_partial: 0
> tx35_csum_partial_inner: 0
> tx35_added_vlan_packets: 0
> tx35_nop: 0
> tx35_csum_none: 0
> tx35_stopped: 0
> tx35_dropped: 0
> tx35_xmit_more: 0
> tx35_recover: 0
> tx35_cqes: 0
> tx35_wake: 0
> tx35_cqe_err: 0
> tx36_packets: 0
> tx36_bytes: 0
> tx36_tso_packets: 0
> tx36_tso_bytes: 0
> tx36_tso_inner_packets: 0
> tx36_tso_inner_bytes: 0
> tx36_csum_partial: 0
> tx36_csum_partial_inner: 0
> tx36_added_vlan_packets: 0
> tx36_nop: 0
> tx36_csum_none: 0
> tx36_stopped: 0
> tx36_dropped: 0
> tx36_xmit_more: 0
> tx36_recover: 0
> tx36_cqes: 0
> tx36_wake: 0
> tx36_cqe_err: 0
> tx37_packets: 0
> tx37_bytes: 0
> tx37_tso_packets: 0
> tx37_tso_bytes: 0
> tx37_tso_inner_packets: 0
> tx37_tso_inner_bytes: 0
> tx37_csum_partial: 0
> tx37_csum_partial_inner: 0
> tx37_added_vlan_packets: 0
> tx37_nop: 0
> tx37_csum_none: 0
> tx37_stopped: 0
> tx37_dropped: 0
> tx37_xmit_more: 0
> tx37_recover: 0
> tx37_cqes: 0
> tx37_wake: 0
> tx37_cqe_err: 0
> tx38_packets: 0
> tx38_bytes: 0
> tx38_tso_packets: 0
> tx38_tso_bytes: 0
> tx38_tso_inner_packets: 0
> tx38_tso_inner_bytes: 0
> tx38_csum_partial: 0
> tx38_csum_partial_inner: 0
> tx38_added_vlan_packets: 0
> tx38_nop: 0
> tx38_csum_none: 0
> tx38_stopped: 0
> tx38_dropped: 0
> tx38_xmit_more: 0
> tx38_recover: 0
> tx38_cqes: 0
> tx38_wake: 0
> tx38_cqe_err: 0
> tx39_packets: 0
> tx39_bytes: 0
> tx39_tso_packets: 0
> tx39_tso_bytes: 0
> tx39_tso_inner_packets: 0
> tx39_tso_inner_bytes: 0
> tx39_csum_partial: 0
> tx39_csum_partial_inner: 0
> tx39_added_vlan_packets: 0
> tx39_nop: 0
> tx39_csum_none: 0
> tx39_stopped: 0
> tx39_dropped: 0
> tx39_xmit_more: 0
> tx39_recover: 0
> tx39_cqes: 0
> tx39_wake: 0
> tx39_cqe_err: 0
> tx40_packets: 0
> tx40_bytes: 0
> tx40_tso_packets: 0
> tx40_tso_bytes: 0
> tx40_tso_inner_packets: 0
> tx40_tso_inner_bytes: 0
> tx40_csum_partial: 0
> tx40_csum_partial_inner: 0
> tx40_added_vlan_packets: 0
> tx40_nop: 0
> tx40_csum_none: 0
> tx40_stopped: 0
> tx40_dropped: 0
> tx40_xmit_more: 0
> tx40_recover: 0
> tx40_cqes: 0
> tx40_wake: 0
> tx40_cqe_err: 0
> tx41_packets: 0
> tx41_bytes: 0
> tx41_tso_packets: 0
> tx41_tso_bytes: 0
> tx41_tso_inner_packets: 0
> tx41_tso_inner_bytes: 0
> tx41_csum_partial: 0
> tx41_csum_partial_inner: 0
> tx41_added_vlan_packets: 0
> tx41_nop: 0
> tx41_csum_none: 0
> tx41_stopped: 0
> tx41_dropped: 0
> tx41_xmit_more: 0
> tx41_recover: 0
> tx41_cqes: 0
> tx41_wake: 0
> tx41_cqe_err: 0
> tx42_packets: 0
> tx42_bytes: 0
> tx42_tso_packets: 0
> tx42_tso_bytes: 0
> tx42_tso_inner_packets: 0
> tx42_tso_inner_bytes: 0
> tx42_csum_partial: 0
> tx42_csum_partial_inner: 0
> tx42_added_vlan_packets: 0
> tx42_nop: 0
> tx42_csum_none: 0
> tx42_stopped: 0
> tx42_dropped: 0
> tx42_xmit_more: 0
> tx42_recover: 0
> tx42_cqes: 0
> tx42_wake: 0
> tx42_cqe_err: 0
> tx43_packets: 0
> tx43_bytes: 0
> tx43_tso_packets: 0
> tx43_tso_bytes: 0
> tx43_tso_inner_packets: 0
> tx43_tso_inner_bytes: 0
> tx43_csum_partial: 0
> tx43_csum_partial_inner: 0
> tx43_added_vlan_packets: 0
> tx43_nop: 0
> tx43_csum_none: 0
> tx43_stopped: 0
> tx43_dropped: 0
> tx43_xmit_more: 0
> tx43_recover: 0
> tx43_cqes: 0
> tx43_wake: 0
> tx43_cqe_err: 0
> tx44_packets: 0
> tx44_bytes: 0
> tx44_tso_packets: 0
> tx44_tso_bytes: 0
> tx44_tso_inner_packets: 0
> tx44_tso_inner_bytes: 0
> tx44_csum_partial: 0
> tx44_csum_partial_inner: 0
> tx44_added_vlan_packets: 0
> tx44_nop: 0
> tx44_csum_none: 0
> tx44_stopped: 0
> tx44_dropped: 0
> tx44_xmit_more: 0
> tx44_recover: 0
> tx44_cqes: 0
> tx44_wake: 0
> tx44_cqe_err: 0
> tx45_packets: 0
> tx45_bytes: 0
> tx45_tso_packets: 0
> tx45_tso_bytes: 0
> tx45_tso_inner_packets: 0
> tx45_tso_inner_bytes: 0
> tx45_csum_partial: 0
> tx45_csum_partial_inner: 0
> tx45_added_vlan_packets: 0
> tx45_nop: 0
> tx45_csum_none: 0
> tx45_stopped: 0
> tx45_dropped: 0
> tx45_xmit_more: 0
> tx45_recover: 0
> tx45_cqes: 0
> tx45_wake: 0
> tx45_cqe_err: 0
> tx46_packets: 0
> tx46_bytes: 0
> tx46_tso_packets: 0
> tx46_tso_bytes: 0
> tx46_tso_inner_packets: 0
> tx46_tso_inner_bytes: 0
> tx46_csum_partial: 0
> tx46_csum_partial_inner: 0
> tx46_added_vlan_packets: 0
> tx46_nop: 0
> tx46_csum_none: 0
> tx46_stopped: 0
> tx46_dropped: 0
> tx46_xmit_more: 0
> tx46_recover: 0
> tx46_cqes: 0
> tx46_wake: 0
> tx46_cqe_err: 0
> tx47_packets: 0
> tx47_bytes: 0
> tx47_tso_packets: 0
> tx47_tso_bytes: 0
> tx47_tso_inner_packets: 0
> tx47_tso_inner_bytes: 0
> tx47_csum_partial: 0
> tx47_csum_partial_inner: 0
> tx47_added_vlan_packets: 0
> tx47_nop: 0
> tx47_csum_none: 0
> tx47_stopped: 0
> tx47_dropped: 0
> tx47_xmit_more: 0
> tx47_recover: 0
> tx47_cqes: 0
> tx47_wake: 0
> tx47_cqe_err: 0
> tx48_packets: 0
> tx48_bytes: 0
> tx48_tso_packets: 0
> tx48_tso_bytes: 0
> tx48_tso_inner_packets: 0
> tx48_tso_inner_bytes: 0
> tx48_csum_partial: 0
> tx48_csum_partial_inner: 0
> tx48_added_vlan_packets: 0
> tx48_nop: 0
> tx48_csum_none: 0
> tx48_stopped: 0
> tx48_dropped: 0
> tx48_xmit_more: 0
> tx48_recover: 0
> tx48_cqes: 0
> tx48_wake: 0
> tx48_cqe_err: 0
> tx49_packets: 0
> tx49_bytes: 0
> tx49_tso_packets: 0
> tx49_tso_bytes: 0
> tx49_tso_inner_packets: 0
> tx49_tso_inner_bytes: 0
> tx49_csum_partial: 0
> tx49_csum_partial_inner: 0
> tx49_added_vlan_packets: 0
> tx49_nop: 0
> tx49_csum_none: 0
> tx49_stopped: 0
> tx49_dropped: 0
> tx49_xmit_more: 0
> tx49_recover: 0
> tx49_cqes: 0
> tx49_wake: 0
> tx49_cqe_err: 0
> tx50_packets: 0
> tx50_bytes: 0
> tx50_tso_packets: 0
> tx50_tso_bytes: 0
> tx50_tso_inner_packets: 0
> tx50_tso_inner_bytes: 0
> tx50_csum_partial: 0
> tx50_csum_partial_inner: 0
> tx50_added_vlan_packets: 0
> tx50_nop: 0
> tx50_csum_none: 0
> tx50_stopped: 0
> tx50_dropped: 0
> tx50_xmit_more: 0
> tx50_recover: 0
> tx50_cqes: 0
> tx50_wake: 0
> tx50_cqe_err: 0
> tx51_packets: 0
> tx51_bytes: 0
> tx51_tso_packets: 0
> tx51_tso_bytes: 0
> tx51_tso_inner_packets: 0
> tx51_tso_inner_bytes: 0
> tx51_csum_partial: 0
> tx51_csum_partial_inner: 0
> tx51_added_vlan_packets: 0
> tx51_nop: 0
> tx51_csum_none: 0
> tx51_stopped: 0
> tx51_dropped: 0
> tx51_xmit_more: 0
> tx51_recover: 0
> tx51_cqes: 0
> tx51_wake: 0
> tx51_cqe_err: 0
> tx52_packets: 0
> tx52_bytes: 0
> tx52_tso_packets: 0
> tx52_tso_bytes: 0
> tx52_tso_inner_packets: 0
> tx52_tso_inner_bytes: 0
> tx52_csum_partial: 0
> tx52_csum_partial_inner: 0
> tx52_added_vlan_packets: 0
> tx52_nop: 0
> tx52_csum_none: 0
> tx52_stopped: 0
> tx52_dropped: 0
> tx52_xmit_more: 0
> tx52_recover: 0
> tx52_cqes: 0
> tx52_wake: 0
> tx52_cqe_err: 0
> tx53_packets: 0
> tx53_bytes: 0
> tx53_tso_packets: 0
> tx53_tso_bytes: 0
> tx53_tso_inner_packets: 0
> tx53_tso_inner_bytes: 0
> tx53_csum_partial: 0
> tx53_csum_partial_inner: 0
> tx53_added_vlan_packets: 0
> tx53_nop: 0
> tx53_csum_none: 0
> tx53_stopped: 0
> tx53_dropped: 0
> tx53_xmit_more: 0
> tx53_recover: 0
> tx53_cqes: 0
> tx53_wake: 0
> tx53_cqe_err: 0
> tx54_packets: 0
> tx54_bytes: 0
> tx54_tso_packets: 0
> tx54_tso_bytes: 0
> tx54_tso_inner_packets: 0
> tx54_tso_inner_bytes: 0
> tx54_csum_partial: 0
> tx54_csum_partial_inner: 0
> tx54_added_vlan_packets: 0
> tx54_nop: 0
> tx54_csum_none: 0
> tx54_stopped: 0
> tx54_dropped: 0
> tx54_xmit_more: 0
> tx54_recover: 0
> tx54_cqes: 0
> tx54_wake: 0
> tx54_cqe_err: 0
> tx55_packets: 0
> tx55_bytes: 0
> tx55_tso_packets: 0
> tx55_tso_bytes: 0
> tx55_tso_inner_packets: 0
> tx55_tso_inner_bytes: 0
> tx55_csum_partial: 0
> tx55_csum_partial_inner: 0
> tx55_added_vlan_packets: 0
> tx55_nop: 0
> tx55_csum_none: 0
> tx55_stopped: 0
> tx55_dropped: 0
> tx55_xmit_more: 0
> tx55_recover: 0
> tx55_cqes: 0
> tx55_wake: 0
> tx55_cqe_err: 0
> tx0_xdp_xmit: 0
> tx0_xdp_full: 0
> tx0_xdp_err: 0
> tx0_xdp_cqes: 0
> tx1_xdp_xmit: 0
> tx1_xdp_full: 0
> tx1_xdp_err: 0
> tx1_xdp_cqes: 0
> tx2_xdp_xmit: 0
> tx2_xdp_full: 0
> tx2_xdp_err: 0
> tx2_xdp_cqes: 0
> tx3_xdp_xmit: 0
> tx3_xdp_full: 0
> tx3_xdp_err: 0
> tx3_xdp_cqes: 0
> tx4_xdp_xmit: 0
> tx4_xdp_full: 0
> tx4_xdp_err: 0
> tx4_xdp_cqes: 0
> tx5_xdp_xmit: 0
> tx5_xdp_full: 0
> tx5_xdp_err: 0
> tx5_xdp_cqes: 0
> tx6_xdp_xmit: 0
> tx6_xdp_full: 0
> tx6_xdp_err: 0
> tx6_xdp_cqes: 0
> tx7_xdp_xmit: 0
> tx7_xdp_full: 0
> tx7_xdp_err: 0
> tx7_xdp_cqes: 0
> tx8_xdp_xmit: 0
> tx8_xdp_full: 0
> tx8_xdp_err: 0
> tx8_xdp_cqes: 0
> tx9_xdp_xmit: 0
> tx9_xdp_full: 0
> tx9_xdp_err: 0
> tx9_xdp_cqes: 0
> tx10_xdp_xmit: 0
> tx10_xdp_full: 0
> tx10_xdp_err: 0
> tx10_xdp_cqes: 0
> tx11_xdp_xmit: 0
> tx11_xdp_full: 0
> tx11_xdp_err: 0
> tx11_xdp_cqes: 0
> tx12_xdp_xmit: 0
> tx12_xdp_full: 0
> tx12_xdp_err: 0
> tx12_xdp_cqes: 0
> tx13_xdp_xmit: 0
> tx13_xdp_full: 0
> tx13_xdp_err: 0
> tx13_xdp_cqes: 0
> tx14_xdp_xmit: 0
> tx14_xdp_full: 0
> tx14_xdp_err: 0
> tx14_xdp_cqes: 0
> tx15_xdp_xmit: 0
> tx15_xdp_full: 0
> tx15_xdp_err: 0
> tx15_xdp_cqes: 0
> tx16_xdp_xmit: 0
> tx16_xdp_full: 0
> tx16_xdp_err: 0
> tx16_xdp_cqes: 0
> tx17_xdp_xmit: 0
> tx17_xdp_full: 0
> tx17_xdp_err: 0
> tx17_xdp_cqes: 0
> tx18_xdp_xmit: 0
> tx18_xdp_full: 0
> tx18_xdp_err: 0
> tx18_xdp_cqes: 0
> tx19_xdp_xmit: 0
> tx19_xdp_full: 0
> tx19_xdp_err: 0
> tx19_xdp_cqes: 0
> tx20_xdp_xmit: 0
> tx20_xdp_full: 0
> tx20_xdp_err: 0
> tx20_xdp_cqes: 0
> tx21_xdp_xmit: 0
> tx21_xdp_full: 0
> tx21_xdp_err: 0
> tx21_xdp_cqes: 0
> tx22_xdp_xmit: 0
> tx22_xdp_full: 0
> tx22_xdp_err: 0
> tx22_xdp_cqes: 0
> tx23_xdp_xmit: 0
> tx23_xdp_full: 0
> tx23_xdp_err: 0
> tx23_xdp_cqes: 0
> tx24_xdp_xmit: 0
> tx24_xdp_full: 0
> tx24_xdp_err: 0
> tx24_xdp_cqes: 0
> tx25_xdp_xmit: 0
> tx25_xdp_full: 0
> tx25_xdp_err: 0
> tx25_xdp_cqes: 0
> tx26_xdp_xmit: 0
> tx26_xdp_full: 0
> tx26_xdp_err: 0
> tx26_xdp_cqes: 0
> tx27_xdp_xmit: 0
> tx27_xdp_full: 0
> tx27_xdp_err: 0
> tx27_xdp_cqes: 0
> tx28_xdp_xmit: 0
> tx28_xdp_full: 0
> tx28_xdp_err: 0
> tx28_xdp_cqes: 0
> tx29_xdp_xmit: 0
> tx29_xdp_full: 0
> tx29_xdp_err: 0
> tx29_xdp_cqes: 0
> tx30_xdp_xmit: 0
> tx30_xdp_full: 0
> tx30_xdp_err: 0
> tx30_xdp_cqes: 0
> tx31_xdp_xmit: 0
> tx31_xdp_full: 0
> tx31_xdp_err: 0
> tx31_xdp_cqes: 0
> tx32_xdp_xmit: 0
> tx32_xdp_full: 0
> tx32_xdp_err: 0
> tx32_xdp_cqes: 0
> tx33_xdp_xmit: 0
> tx33_xdp_full: 0
> tx33_xdp_err: 0
> tx33_xdp_cqes: 0
> tx34_xdp_xmit: 0
> tx34_xdp_full: 0
> tx34_xdp_err: 0
> tx34_xdp_cqes: 0
> tx35_xdp_xmit: 0
> tx35_xdp_full: 0
> tx35_xdp_err: 0
> tx35_xdp_cqes: 0
> tx36_xdp_xmit: 0
> tx36_xdp_full: 0
> tx36_xdp_err: 0
> tx36_xdp_cqes: 0
> tx37_xdp_xmit: 0
> tx37_xdp_full: 0
> tx37_xdp_err: 0
> tx37_xdp_cqes: 0
> tx38_xdp_xmit: 0
> tx38_xdp_full: 0
> tx38_xdp_err: 0
> tx38_xdp_cqes: 0
> tx39_xdp_xmit: 0
> tx39_xdp_full: 0
> tx39_xdp_err: 0
> tx39_xdp_cqes: 0
> tx40_xdp_xmit: 0
> tx40_xdp_full: 0
> tx40_xdp_err: 0
> tx40_xdp_cqes: 0
> tx41_xdp_xmit: 0
> tx41_xdp_full: 0
> tx41_xdp_err: 0
> tx41_xdp_cqes: 0
> tx42_xdp_xmit: 0
> tx42_xdp_full: 0
> tx42_xdp_err: 0
> tx42_xdp_cqes: 0
> tx43_xdp_xmit: 0
> tx43_xdp_full: 0
> tx43_xdp_err: 0
> tx43_xdp_cqes: 0
> tx44_xdp_xmit: 0
> tx44_xdp_full: 0
> tx44_xdp_err: 0
> tx44_xdp_cqes: 0
> tx45_xdp_xmit: 0
> tx45_xdp_full: 0
> tx45_xdp_err: 0
> tx45_xdp_cqes: 0
> tx46_xdp_xmit: 0
> tx46_xdp_full: 0
> tx46_xdp_err: 0
> tx46_xdp_cqes: 0
> tx47_xdp_xmit: 0
> tx47_xdp_full: 0
> tx47_xdp_err: 0
> tx47_xdp_cqes: 0
> tx48_xdp_xmit: 0
> tx48_xdp_full: 0
> tx48_xdp_err: 0
> tx48_xdp_cqes: 0
> tx49_xdp_xmit: 0
> tx49_xdp_full: 0
> tx49_xdp_err: 0
> tx49_xdp_cqes: 0
> tx50_xdp_xmit: 0
> tx50_xdp_full: 0
> tx50_xdp_err: 0
> tx50_xdp_cqes: 0
> tx51_xdp_xmit: 0
> tx51_xdp_full: 0
> tx51_xdp_err: 0
> tx51_xdp_cqes: 0
> tx52_xdp_xmit: 0
> tx52_xdp_full: 0
> tx52_xdp_err: 0
> tx52_xdp_cqes: 0
> tx53_xdp_xmit: 0
> tx53_xdp_full: 0
> tx53_xdp_err: 0
> tx53_xdp_cqes: 0
> tx54_xdp_xmit: 0
> tx54_xdp_full: 0
> tx54_xdp_err: 0
> tx54_xdp_cqes: 0
> tx55_xdp_xmit: 0
> tx55_xdp_full: 0
> tx55_xdp_err: 0
> tx55_xdp_cqes: 0
>
>
>> [...]
>>
>>>>> ethtool -S enp175s0f0
>>>>> NIC statistics:
>>>>> rx_packets: 141574897253
>>>>> rx_bytes: 184445040406258
>>>>> tx_packets: 172569543894
>>>>> tx_bytes: 99486882076365
>>>>> tx_tso_packets: 9367664195
>>>>> tx_tso_bytes: 56435233992948
>>>>> tx_tso_inner_packets: 0
>>>>> tx_tso_inner_bytes: 0
>>>>> tx_added_vlan_packets: 141297671626
>>>>> tx_nop: 2102916272
>>>>> rx_lro_packets: 0
>>>>> rx_lro_bytes: 0
>>>>> rx_ecn_mark: 0
>>>>> rx_removed_vlan_packets: 141574897252
>>>>> rx_csum_unnecessary: 0
>>>>> rx_csum_none: 23135854
>>>>> rx_csum_complete: 141551761398
>>>>> rx_csum_unnecessary_inner: 0
>>>>> rx_xdp_drop: 0
>>>>> rx_xdp_redirect: 0
>>>>> rx_xdp_tx_xmit: 0
>>>>> rx_xdp_tx_full: 0
>>>>> rx_xdp_tx_err: 0
>>>>> rx_xdp_tx_cqe: 0
>>>>> tx_csum_none: 127934791664
>>>> It is a good idea to look into this, tx is not requesting hw tx
>>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>>> on
>>>> calculating csum, or maybe this is just the rx csum complete..
>>>>
>>>>> tx_csum_partial: 13362879974
>>>>> tx_csum_partial_inner: 0
>>>>> tx_queue_stopped: 232561
>>>> TX queues are stalling, could be an indentation for the pcie
>>>> bottelneck.
>>>>
>>>>> tx_queue_dropped: 0
>>>>> tx_xmit_more: 1266021946
>>>>> tx_recover: 0
>>>>> tx_cqes: 140031716469
>>>>> tx_queue_wake: 232561
>>>>> tx_udp_seg_rem: 0
>>>>> tx_cqe_err: 0
>>>>> tx_xdp_xmit: 0
>>>>> tx_xdp_full: 0
>>>>> tx_xdp_err: 0
>>>>> tx_xdp_cqes: 0
>>>>> rx_wqe_err: 0
>>>>> rx_mpwqe_filler_cqes: 0
>>>>> rx_mpwqe_filler_strides: 0
>>>>> rx_buff_alloc_err: 0
>>>>> rx_cqe_compress_blks: 0
>>>>> rx_cqe_compress_pkts: 0
>>>>> rx_page_reuse: 0
>>>>> rx_cache_reuse: 16625975793
>>>>> rx_cache_full: 54161465914
>>>>> rx_cache_empty: 258048
>>>>> rx_cache_busy: 54161472735
>>>>> rx_cache_waive: 0
>>>>> rx_congst_umr: 0
>>>>> rx_arfs_err: 0
>>>>> ch_events: 40572621887
>>>>> ch_poll: 40885650979
>>>>> ch_arm: 40429276692
>>>>> ch_aff_change: 0
>>>>> ch_eq_rearm: 0
>>>>> rx_out_of_buffer: 2791690
>>>>> rx_if_down_packets: 74
>>>>> rx_vport_unicast_packets: 141843476308
>>>>> rx_vport_unicast_bytes: 185421265403318
>>>>> tx_vport_unicast_packets: 172569484005
>>>>> tx_vport_unicast_bytes: 100019940094298
>>>>> rx_vport_multicast_packets: 85122935
>>>>> rx_vport_multicast_bytes: 5761316431
>>>>> tx_vport_multicast_packets: 6452
>>>>> tx_vport_multicast_bytes: 643540
>>>>> rx_vport_broadcast_packets: 22423624
>>>>> rx_vport_broadcast_bytes: 1390127090
>>>>> tx_vport_broadcast_packets: 22024
>>>>> tx_vport_broadcast_bytes: 1321440
>>>>> rx_vport_rdma_unicast_packets: 0
>>>>> rx_vport_rdma_unicast_bytes: 0
>>>>> tx_vport_rdma_unicast_packets: 0
>>>>> tx_vport_rdma_unicast_bytes: 0
>>>>> rx_vport_rdma_multicast_packets: 0
>>>>> rx_vport_rdma_multicast_bytes: 0
>>>>> tx_vport_rdma_multicast_packets: 0
>>>>> tx_vport_rdma_multicast_bytes: 0
>>>>> tx_packets_phy: 172569501577
>>>>> rx_packets_phy: 142871314588
>>>>> rx_crc_errors_phy: 0
>>>>> tx_bytes_phy: 100710212814151
>>>>> rx_bytes_phy: 187209224289564
>>>>> tx_multicast_phy: 6452
>>>>> tx_broadcast_phy: 22024
>>>>> rx_multicast_phy: 85122933
>>>>> rx_broadcast_phy: 22423623
>>>>> rx_in_range_len_errors_phy: 2
>>>>> rx_out_of_range_len_phy: 0
>>>>> rx_oversize_pkts_phy: 0
>>>>> rx_symbol_err_phy: 0
>>>>> tx_mac_control_phy: 0
>>>>> rx_mac_control_phy: 0
>>>>> rx_unsupported_op_phy: 0
>>>>> rx_pause_ctrl_phy: 0
>>>>> tx_pause_ctrl_phy: 0
>>>>> rx_discards_phy: 920161423
>>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>>> to
>>>> the pcie bottleneck.
>>> Yes this side is receiving more traffic - second port is +10G more tx
>>>
>> [...]
>>
>>
>>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>>> 0.00 0.00 0.00 31.30
>>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>>> 0.00 0.00 0.00 24.90
>>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>>> 0.00 0.00 0.00 19.68
>>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>>> 0.00 0.00 0.00 18.00
>>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>>> 0.00 0.00 0.00 17.40
>>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>>> 0.00 0.00 0.00 26.03
>>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>>> 0.00 0.00 0.00 17.52
>>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>>> 0.00 0.00 0.00 18.20
>>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>>> 0.00 0.00 0.00 17.68
>>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>>> 0.00 0.00 0.00 18.20
>>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>>> 0.00 0.00 0.00 20.42
>>>> The numa cores are not at 100% util, you have around 20% of idle on
>>>> each one.
>>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>>> push
>>> aditional 1-2Gbit/s
>>>
>> yes but, it doens't look like the bottleneck is the cpu, although it is
>> close to be :)..
>>
>>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 96.10
>>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.70
>>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.50
>>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 100.00
>>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 97.40
>>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.10
>>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>>> 0.00
>>>>> 0.00 0.00 99.40
>>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>>> 0.00 0.00 0.00 19.42
>>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>>> 0.00 0.00 0.00 26.60
>>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>>> 0.00 0.00 0.00 21.60
>>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>>> 0.00 0.00 0.00 20.50
>>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>>> 0.00 0.00 0.00 21.60
>>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>>> 0.00 0.00 0.00 24.58
>>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>>> 0.00 0.00 0.00 24.68
>>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>>> 0.00 0.00 0.00 28.34
>>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>>> 0.00 0.00 0.00 25.83
>>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>>> 0.00 0.00 0.00 27.20
>>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>>> 0.00 0.00 0.00 28.03
>>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>>> 0.00 0.00 0.00 24.70
>>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>>> 0.00 0.00 0.00 25.30
>>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>>> 0.00 0.00 0.00 26.77
>>>>>
>>>>>
>>>>> ethtool -k enp175s0f0
>>>>> Features for enp175s0f0:
>>>>> rx-checksumming: on
>>>>> tx-checksumming: on
>>>>> tx-checksum-ipv4: on
>>>>> tx-checksum-ip-generic: off [fixed]
>>>>> tx-checksum-ipv6: on
>>>>> tx-checksum-fcoe-crc: off [fixed]
>>>>> tx-checksum-sctp: off [fixed]
>>>>> scatter-gather: on
>>>>> tx-scatter-gather: on
>>>>> tx-scatter-gather-fraglist: off [fixed]
>>>>> tcp-segmentation-offload: on
>>>>> tx-tcp-segmentation: on
>>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>>> tx-tcp-mangleid-segmentation: off
>>>>> tx-tcp6-segmentation: on
>>>>> udp-fragmentation-offload: off
>>>>> generic-segmentation-offload: on
>>>>> generic-receive-offload: on
>>>>> large-receive-offload: off [fixed]
>>>>> rx-vlan-offload: on
>>>>> tx-vlan-offload: on
>>>>> ntuple-filters: off
>>>>> receive-hashing: on
>>>>> highdma: on [fixed]
>>>>> rx-vlan-filter: on
>>>>> vlan-challenged: off [fixed]
>>>>> tx-lockless: off [fixed]
>>>>> netns-local: off [fixed]
>>>>> tx-gso-robust: off [fixed]
>>>>> tx-fcoe-segmentation: off [fixed]
>>>>> tx-gre-segmentation: on
>>>>> tx-gre-csum-segmentation: on
>>>>> tx-ipxip4-segmentation: off [fixed]
>>>>> tx-ipxip6-segmentation: off [fixed]
>>>>> tx-udp_tnl-segmentation: on
>>>>> tx-udp_tnl-csum-segmentation: on
>>>>> tx-gso-partial: on
>>>>> tx-sctp-segmentation: off [fixed]
>>>>> tx-esp-segmentation: off [fixed]
>>>>> tx-udp-segmentation: on
>>>>> fcoe-mtu: off [fixed]
>>>>> tx-nocache-copy: off
>>>>> loopback: off [fixed]
>>>>> rx-fcs: off
>>>>> rx-all: off
>>>>> tx-vlan-stag-hw-insert: on
>>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>>> rx-vlan-stag-filter: on [fixed]
>>>>> l2-fwd-offload: off [fixed]
>>>>> hw-tc-offload: off
>>>>> esp-hw-offload: off [fixed]
>>>>> esp-tx-csum-hw-offload: off [fixed]
>>>>> rx-udp_tunnel-port-offload: on
>>>>> tls-hw-tx-offload: off [fixed]
>>>>> tls-hw-rx-offload: off [fixed]
>>>>> rx-gro-hw: off [fixed]
>>>>> tls-hw-record: off [fixed]
>>>>>
>>>>> ethtool -c enp175s0f0
>>>>> Coalesce parameters for enp175s0f0:
>>>>> Adaptive RX: off TX: on
>>>>> stats-block-usecs: 0
>>>>> sample-interval: 0
>>>>> pkt-rate-low: 0
>>>>> pkt-rate-high: 0
>>>>> dmac: 32703
>>>>>
>>>>> rx-usecs: 256
>>>>> rx-frames: 128
>>>>> rx-usecs-irq: 0
>>>>> rx-frames-irq: 0
>>>>>
>>>>> tx-usecs: 8
>>>>> tx-frames: 128
>>>>> tx-usecs-irq: 0
>>>>> tx-frames-irq: 0
>>>>>
>>>>> rx-usecs-low: 0
>>>>> rx-frame-low: 0
>>>>> tx-usecs-low: 0
>>>>> tx-frame-low: 0
>>>>>
>>>>> rx-usecs-high: 0
>>>>> rx-frame-high: 0
>>>>> tx-usecs-high: 0
>>>>> tx-frame-high: 0
>>>>>
>>>>> ethtool -g enp175s0f0
>>>>> Ring parameters for enp175s0f0:
>>>>> Pre-set maximums:
>>>>> RX: 8192
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 8192
>>>>> Current hardware settings:
>>>>> RX: 4096
>>>>> RX Mini: 0
>>>>> RX Jumbo: 0
>>>>> TX: 4096
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>> Also changed a little coalesce params - and best for this config are:
>>> ethtool -c enp175s0f0
>>> Coalesce parameters for enp175s0f0:
>>> Adaptive RX: off TX: off
>>> stats-block-usecs: 0
>>> sample-interval: 0
>>> pkt-rate-low: 0
>>> pkt-rate-high: 0
>>> dmac: 32573
>>>
>>> rx-usecs: 40
>>> rx-frames: 128
>>> rx-usecs-irq: 0
>>> rx-frames-irq: 0
>>>
>>> tx-usecs: 8
>>> tx-frames: 8
>>> tx-usecs-irq: 0
>>> tx-frames-irq: 0
>>>
>>> rx-usecs-low: 0
>>> rx-frame-low: 0
>>> tx-usecs-low: 0
>>> tx-frame-low: 0
>>>
>>> rx-usecs-high: 0
>>> rx-frame-high: 0
>>> tx-usecs-high: 0
>>> tx-frame-high: 0
>>>
>>>
>>> Less drops on RX side - and more pps in overall forwarded.
>>>
>> how much improvement ? maybe we can improve our adaptive rx coal to be
>> efficient for this work load.
>>
>>
> I can prepare more stats with ethtool maybee to compare - but normally
> tested with simple icmp forwarded from interface to interface
> - before change coalescence params:
> adaptive-rx off rx-usecs 384 rx-frames 128
> 3% loss for icmp
> - after change to:
> adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8
> tx-frames 8
> 2% loss for icmp
>
> But yes - to know better will need to compare rx/tx counters from
> ethtool + /proc/net/dev
>
>
> Was trying to turn on adaptative-tx+rx - but 100% saturation at
> 43Gbit/s RX / 43Gbit/s TX
>
>
>
>
>
>
>
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-01 21:18 UTC (permalink / raw)
To: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <d8d4b059c4d7dff8f05839c5042051b41fdd1a7a.camel@mellanox.com>
W dniu 01.11.2018 o 21:37, Saeed Mahameed pisze:
> On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
>>> On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
>>>> Hi
>>>>
>>>> So maybee someone will be interested how linux kernel handles
>>>> normal
>>>> traffic (not pktgen :) )
>>>>
>>>>
>>>> Server HW configuration:
>>>>
>>>> CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>>>
>>>> NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
>>>>
>>>>
>>>> Server software:
>>>>
>>>> FRR - as routing daemon
>>>>
>>>> enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
>>>> local
>>>> numa
>>>> node)
>>>>
>>>> enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
>>>> numa
>>>> node)
>>>>
>>>>
>>>> Maximum traffic that server can handle:
>>>>
>>>> Bandwidth
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> \ iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 28.51 Gb/s 37.24
>>>> Gb/s
>>>> 65.74 Gb/s
>>>> enp175s0f0: 38.07 Gb/s 28.44
>>>> Gb/s
>>>> 66.51 Gb/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 66.58 Gb/s 65.67
>>>> Gb/s
>>>> 132.25 Gb/s
>>>>
>>>>
>>>> Packets per second:
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> =================================================================
>>>> ====
>>>> =========
>>>> enp175s0f1: 5248589.00 P/s 3486617.75 P/s
>>>> 8735207.00 P/s
>>>> enp175s0f0: 3557944.25 P/s 5232516.00 P/s
>>>> 8790460.00 P/s
>>>> ---------------------------------------------------------------
>>>> ----
>>>> -----------
>>>> total: 8806533.00 P/s 8719134.00 P/s
>>>> 17525668.00 P/s
>>>>
>>>>
>>>> After reaching that limits nics on the upstream side (more RX
>>>> traffic)
>>>> start to drop packets
>>>>
>>>>
>>>> I just dont understand that server can't handle more bandwidth
>>>> (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
>>>> RX
>>>> side are increasing.
>>>>
>>> Where do you see 40 Gb/s ? you showed that both ports on the same
>>> NIC (
>>> same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
>>> 132.25
>>> Gb/s which aligns with your pcie link limit, what am i missing ?
>> hmm yes that was my concern also - cause cant find anywhere
>> informations
>> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
>> 8GT
>> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
>> bw
>> on both ports
> i think it is bidir
>
>> This can explain maybee also why cpuload is rising rapidly from
>> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
>> so
>> there can be some error in reading them when offloading (gro/gso/tso)
>> on
>> nic's is enabled that is why
>>
>>>> Was thinking that maybee reached some pcie x16 limit - but x16
>>>> 8GT
>>>> is
>>>> 126Gbit - and also when testing with pktgen i can reach more bw
>>>> and
>>>> pps
>>>> (like 4x more comparing to normal internet traffic)
>>>>
>>> Are you forwarding when using pktgen as well or you just testing
>>> the RX
>>> side pps ?
>> Yes pktgen was tested on single port RX
>> Can check also forwarding to eliminate pciex limits
>>
> So this explains why you have more RX pps, since tx is idle and pcie
> will be free to do only rx.
>
> [...]
>
>
>>>> ethtool -S enp175s0f1
>>>> NIC statistics:
>>>> rx_packets: 173730800927
>>>> rx_bytes: 99827422751332
>>>> tx_packets: 142532009512
>>>> tx_bytes: 184633045911222
>>>> tx_tso_packets: 25989113891
>>>> tx_tso_bytes: 132933363384458
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 74630239613
>>>> tx_nop: 2029817748
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 173730800927
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 434357
>>>> rx_csum_complete: 173730366570
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 38260960853
>>>> tx_csum_partial: 36369278774
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 1
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 748638099
>>>> tx_recover: 0
>>>> tx_cqes: 73881645031
>>>> tx_queue_wake: 1
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>> If this is a pcie bottleneck it might be useful to enable CQE
>>> compression (to reduce PCIe completion descriptors transactions)
>>> you should see the above rx_cqe_compress_pkts increasing when
>>> enabled.
>>>
>>> $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
>>> $ ethtool --show-priv-flags enp175s0f1
>>> Private flags for p6p1:
>>> rx_cqe_moder : on
>>> cqe_moder : off
>>> rx_cqe_compress : on
>>> ...
>>>
>>> try this on both interfaces.
>> Done
>> ethtool --show-priv-flags enp175s0f1
>> Private flags for enp175s0f1:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
>> ethtool --show-priv-flags enp175s0f0
>> Private flags for enp175s0f0:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : on
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
> did it help reduce the load on the pcie ? do you see more pps ?
> what is the ratio between rx_cqe_compress_pkts and over all rx packets
> ?
So - a little more pps
Before change top - graph / after bottom -> image with graph stats from
proc/net/dev
cqe_compress enabled at 11:55
Sorry - but for real life traffic it is hard to do any counter
differences - cause traffic just rising alone from minute to minute :)
But for that time the change is visible on graph - cause was almost same
for past 20minutes before change.
full ethtool below:
NIC statistics:
rx_packets: 516522465438
rx_bytes: 680052911258729
tx_packets: 677697545586
tx_bytes: 413647643141709
tx_tso_packets: 42530913279
tx_tso_bytes: 235655668554142
tx_tso_inner_packets: 0
tx_tso_inner_bytes: 0
tx_added_vlan_packets: 551156530885
tx_nop: 8536823558
rx_lro_packets: 0
rx_lro_bytes: 0
rx_ecn_mark: 0
rx_removed_vlan_packets: 516522465438
rx_csum_unnecessary: 0
rx_csum_none: 50382868
rx_csum_complete: 516472082570
rx_csum_unnecessary_inner: 0
rx_xdp_drop: 0
rx_xdp_redirect: 0
rx_xdp_tx_xmit: 0
rx_xdp_tx_full: 0
rx_xdp_tx_err: 0
rx_xdp_tx_cqe: 0
tx_csum_none: 494075047017
tx_csum_partial: 57081483898
tx_csum_partial_inner: 0
tx_queue_stopped: 518624
tx_queue_dropped: 0
tx_xmit_more: 1717880628
tx_recover: 0
tx_cqes: 549438869029
tx_queue_wake: 518627
tx_udp_seg_rem: 0
tx_cqe_err: 0
tx_xdp_xmit: 0
tx_xdp_full: 0
tx_xdp_err: 0
tx_xdp_cqes: 0
rx_wqe_err: 0
rx_mpwqe_filler_cqes: 0
rx_mpwqe_filler_strides: 0
rx_buff_alloc_err: 0
rx_cqe_compress_blks: 11483228712
rx_cqe_compress_pkts: 25794213324
rx_page_reuse: 0
rx_cache_reuse: 63610249810
rx_cache_full: 194650916511
rx_cache_empty: 1118208
rx_cache_busy: 194650982430
rx_cache_waive: 0
rx_congst_umr: 0
rx_arfs_err: 0
ch_events: 119556002196
ch_poll: 121107424977
ch_arm: 115856746008
ch_aff_change: 31
ch_eq_rearm: 0
rx_out_of_buffer: 6880325
rx_if_down_packets: 2062529
rx_vport_unicast_packets: 517433716795
rx_vport_unicast_bytes: 683464347301443
tx_vport_unicast_packets: 677697453738
tx_vport_unicast_bytes: 415788589663315
rx_vport_multicast_packets: 208258309
rx_vport_multicast_bytes: 14224046052
tx_vport_multicast_packets: 21689
tx_vport_multicast_bytes: 2158334
rx_vport_broadcast_packets: 75838646
rx_vport_broadcast_bytes: 4697944695
tx_vport_broadcast_packets: 68730
tx_vport_broadcast_bytes: 4123800
rx_vport_rdma_unicast_packets: 0
rx_vport_rdma_unicast_bytes: 0
tx_vport_rdma_unicast_packets: 0
tx_vport_rdma_unicast_bytes: 0
rx_vport_rdma_multicast_packets: 0
rx_vport_rdma_multicast_bytes: 0
tx_vport_rdma_multicast_packets: 0
tx_vport_rdma_multicast_bytes: 0
tx_packets_phy: 677697543252
rx_packets_phy: 521319491878
rx_crc_errors_phy: 0
tx_bytes_phy: 418499385791411
rx_bytes_phy: 690322537017274
tx_multicast_phy: 21689
tx_broadcast_phy: 68730
rx_multicast_phy: 208258305
rx_broadcast_phy: 75838646
rx_in_range_len_errors_phy: 4
rx_out_of_range_len_phy: 0
rx_oversize_pkts_phy: 0
rx_symbol_err_phy: 0
tx_mac_control_phy: 0
rx_mac_control_phy: 0
rx_unsupported_op_phy: 0
rx_pause_ctrl_phy: 0
tx_pause_ctrl_phy: 0
rx_discards_phy: 3601449265
tx_discards_phy: 0
tx_errors_phy: 0
rx_undersize_pkts_phy: 0
rx_fragments_phy: 0
rx_jabbers_phy: 0
rx_64_bytes_phy: 1416456771
rx_65_to_127_bytes_phy: 40750434737
rx_128_to_255_bytes_phy: 11518110310
rx_256_to_511_bytes_phy: 7055850637
rx_512_to_1023_bytes_phy: 7811550424
rx_1024_to_1518_bytes_phy: 265547564845
rx_1519_to_2047_bytes_phy: 187219522899
rx_2048_to_4095_bytes_phy: 0
rx_4096_to_8191_bytes_phy: 0
rx_8192_to_10239_bytes_phy: 0
link_down_events_phy: 0
rx_pcs_symbol_err_phy: 0
rx_corrected_bits_phy: 0
rx_pci_signal_integrity: 0
tx_pci_signal_integrity: 48
rx_prio0_bytes: 688807632117485
rx_prio0_packets: 516310309931
tx_prio0_bytes: 418499382756025
tx_prio0_packets: 677697534982
rx_prio1_bytes: 1497701612877
rx_prio1_packets: 1206768094
tx_prio1_bytes: 0
tx_prio1_packets: 0
rx_prio2_bytes: 112271227
rx_prio2_packets: 337295
tx_prio2_bytes: 0
tx_prio2_packets: 0
rx_prio3_bytes: 1165455555
rx_prio3_packets: 1544310
tx_prio3_bytes: 0
tx_prio3_packets: 0
rx_prio4_bytes: 161857240
rx_prio4_packets: 341392
tx_prio4_bytes: 0
tx_prio4_packets: 0
rx_prio5_bytes: 455031612
rx_prio5_packets: 2861469
tx_prio5_bytes: 0
tx_prio5_packets: 0
rx_prio6_bytes: 1873928697
rx_prio6_packets: 5146981
tx_prio6_bytes: 0
tx_prio6_packets: 0
rx_prio7_bytes: 13423452430
rx_prio7_packets: 190724796
tx_prio7_bytes: 0
tx_prio7_packets: 0
module_unplug: 0
module_bus_stuck: 0
module_high_temp: 0
module_bad_shorted: 0
ch0_events: 4252266777
ch0_poll: 4330804273
ch0_arm: 4120233182
ch0_aff_change: 2
ch0_eq_rearm: 0
ch1_events: 3938415938
ch1_poll: 4012621322
ch1_arm: 3810131188
ch1_aff_change: 2
ch1_eq_rearm: 0
ch2_events: 3897428860
ch2_poll: 3973886848
ch2_arm: 3773019397
ch2_aff_change: 1
ch2_eq_rearm: 0
ch3_events: 4108000541
ch3_poll: 4180139872
ch3_arm: 3982093366
ch3_aff_change: 1
ch3_eq_rearm: 0
ch4_events: 4652570079
ch4_poll: 4720541090
ch4_arm: 4524475054
ch4_aff_change: 2
ch4_eq_rearm: 0
ch5_events: 3899177385
ch5_poll: 3974274186
ch5_arm: 3772299186
ch5_aff_change: 2
ch5_eq_rearm: 0
ch6_events: 3915161350
ch6_poll: 3992338199
ch6_arm: 3794710989
ch6_aff_change: 0
ch6_eq_rearm: 0
ch7_events: 4008175631
ch7_poll: 4081321248
ch7_arm: 3882263723
ch7_aff_change: 0
ch7_eq_rearm: 0
ch8_events: 4207422352
ch8_poll: 4276465449
ch8_arm: 4077650366
ch8_aff_change: 0
ch8_eq_rearm: 0
ch9_events: 4036491879
ch9_poll: 4108975987
ch9_arm: 3914493694
ch9_aff_change: 0
ch9_eq_rearm: 0
ch10_events: 4066261595
ch10_poll: 4134419606
ch10_arm: 3936637711
ch10_aff_change: 1
ch10_eq_rearm: 0
ch11_events: 4440494043
ch11_poll: 4507578730
ch11_arm: 4318629438
ch11_aff_change: 0
ch11_eq_rearm: 0
ch12_events: 4066958252
ch12_poll: 4130191506
ch12_arm: 3934337782
ch12_aff_change: 0
ch12_eq_rearm: 0
ch13_events: 4051309159
ch13_poll: 4118864120
ch13_arm: 3921011919
ch13_aff_change: 0
ch13_eq_rearm: 0
ch14_events: 4321664800
ch14_poll: 4382433680
ch14_arm: 4186130552
ch14_aff_change: 0
ch14_eq_rearm: 0
ch15_events: 4701102075
ch15_poll: 4760373932
ch15_arm: 4570151468
ch15_aff_change: 0
ch15_eq_rearm: 0
ch16_events: 4311052687
ch16_poll: 4345937129
ch16_arm: 4170883819
ch16_aff_change: 0
ch16_eq_rearm: 0
ch17_events: 4647570931
ch17_poll: 4680218533
ch17_arm: 4509426288
ch17_aff_change: 0
ch17_eq_rearm: 0
ch18_events: 4598195702
ch18_poll: 4631314898
ch18_arm: 4457267084
ch18_aff_change: 0
ch18_eq_rearm: 0
ch19_events: 4808094560
ch19_poll: 4841368340
ch19_arm: 4670604358
ch19_aff_change: 0
ch19_eq_rearm: 0
ch20_events: 4240910605
ch20_poll: 4276531502
ch20_arm: 4101767278
ch20_aff_change: 1
ch20_eq_rearm: 0
ch21_events: 4389371472
ch21_poll: 4426870311
ch21_arm: 4249339045
ch21_aff_change: 2
ch21_eq_rearm: 0
ch22_events: 4282958754
ch22_poll: 4319228073
ch22_arm: 4145102991
ch22_aff_change: 2
ch22_eq_rearm: 0
ch23_events: 4440196528
ch23_poll: 4474090188
ch23_arm: 4300837147
ch23_aff_change: 2
ch23_eq_rearm: 0
ch24_events: 4326875785
ch24_poll: 4364971263
ch24_arm: 4186404526
ch24_aff_change: 2
ch24_eq_rearm: 0
ch25_events: 4286528453
ch25_poll: 4324089445
ch25_arm: 4147222616
ch25_aff_change: 3
ch25_eq_rearm: 0
ch26_events: 4098043104
ch26_poll: 4138133745
ch26_arm: 3967438971
ch26_aff_change: 4
ch26_eq_rearm: 0
ch27_events: 4563302840
ch27_poll: 4599441446
ch27_arm: 4432182806
ch27_aff_change: 4
ch27_eq_rearm: 0
ch28_events: 4
ch28_poll: 4
ch28_arm: 4
ch28_aff_change: 0
ch28_eq_rearm: 0
ch29_events: 6
ch29_poll: 6
ch29_arm: 6
ch29_aff_change: 0
ch29_eq_rearm: 0
ch30_events: 4
ch30_poll: 4
ch30_arm: 4
ch30_aff_change: 0
ch30_eq_rearm: 0
ch31_events: 4
ch31_poll: 4
ch31_arm: 4
ch31_aff_change: 0
ch31_eq_rearm: 0
ch32_events: 4
ch32_poll: 4
ch32_arm: 4
ch32_aff_change: 0
ch32_eq_rearm: 0
ch33_events: 4
ch33_poll: 4
ch33_arm: 4
ch33_aff_change: 0
ch33_eq_rearm: 0
ch34_events: 4
ch34_poll: 4
ch34_arm: 4
ch34_aff_change: 0
ch34_eq_rearm: 0
ch35_events: 4
ch35_poll: 4
ch35_arm: 4
ch35_aff_change: 0
ch35_eq_rearm: 0
ch36_events: 4
ch36_poll: 4
ch36_arm: 4
ch36_aff_change: 0
ch36_eq_rearm: 0
ch37_events: 4
ch37_poll: 4
ch37_arm: 4
ch37_aff_change: 0
ch37_eq_rearm: 0
ch38_events: 4
ch38_poll: 4
ch38_arm: 4
ch38_aff_change: 0
ch38_eq_rearm: 0
ch39_events: 4
ch39_poll: 4
ch39_arm: 4
ch39_aff_change: 0
ch39_eq_rearm: 0
ch40_events: 4
ch40_poll: 4
ch40_arm: 4
ch40_aff_change: 0
ch40_eq_rearm: 0
ch41_events: 4
ch41_poll: 4
ch41_arm: 4
ch41_aff_change: 0
ch41_eq_rearm: 0
ch42_events: 4
ch42_poll: 4
ch42_arm: 4
ch42_aff_change: 0
ch42_eq_rearm: 0
ch43_events: 4
ch43_poll: 4
ch43_arm: 4
ch43_aff_change: 0
ch43_eq_rearm: 0
ch44_events: 4
ch44_poll: 4
ch44_arm: 4
ch44_aff_change: 0
ch44_eq_rearm: 0
ch45_events: 4
ch45_poll: 4
ch45_arm: 4
ch45_aff_change: 0
ch45_eq_rearm: 0
ch46_events: 4
ch46_poll: 4
ch46_arm: 4
ch46_aff_change: 0
ch46_eq_rearm: 0
ch47_events: 4
ch47_poll: 4
ch47_arm: 4
ch47_aff_change: 0
ch47_eq_rearm: 0
ch48_events: 4
ch48_poll: 4
ch48_arm: 4
ch48_aff_change: 0
ch48_eq_rearm: 0
ch49_events: 4
ch49_poll: 4
ch49_arm: 4
ch49_aff_change: 0
ch49_eq_rearm: 0
ch50_events: 4
ch50_poll: 4
ch50_arm: 4
ch50_aff_change: 0
ch50_eq_rearm: 0
ch51_events: 4
ch51_poll: 4
ch51_arm: 4
ch51_aff_change: 0
ch51_eq_rearm: 0
ch52_events: 4
ch52_poll: 4
ch52_arm: 4
ch52_aff_change: 0
ch52_eq_rearm: 0
ch53_events: 4
ch53_poll: 4
ch53_arm: 4
ch53_aff_change: 0
ch53_eq_rearm: 0
ch54_events: 4
ch54_poll: 4
ch54_arm: 4
ch54_aff_change: 0
ch54_eq_rearm: 0
ch55_events: 4
ch55_poll: 4
ch55_arm: 4
ch55_aff_change: 0
ch55_eq_rearm: 0
rx0_packets: 21390033774
rx0_bytes: 27326856299122
rx0_csum_complete: 21339650906
rx0_csum_unnecessary: 0
rx0_csum_unnecessary_inner: 0
rx0_csum_none: 50382868
rx0_xdp_drop: 0
rx0_xdp_redirect: 0
rx0_lro_packets: 0
rx0_lro_bytes: 0
rx0_ecn_mark: 0
rx0_removed_vlan_packets: 21390033774
rx0_wqe_err: 0
rx0_mpwqe_filler_cqes: 0
rx0_mpwqe_filler_strides: 0
rx0_buff_alloc_err: 0
rx0_cqe_compress_blks: 481077641
rx0_cqe_compress_pkts: 1085647489
rx0_page_reuse: 0
rx0_cache_reuse: 19050049
rx0_cache_full: 10675964285
rx0_cache_empty: 37376
rx0_cache_busy: 10675966819
rx0_cache_waive: 0
rx0_congst_umr: 0
rx0_arfs_err: 0
rx0_xdp_tx_xmit: 0
rx0_xdp_tx_full: 0
rx0_xdp_tx_err: 0
rx0_xdp_tx_cqes: 0
rx1_packets: 19868919527
rx1_bytes: 26149716991561
rx1_csum_complete: 19868919527
rx1_csum_unnecessary: 0
rx1_csum_unnecessary_inner: 0
rx1_csum_none: 0
rx1_xdp_drop: 0
rx1_xdp_redirect: 0
rx1_lro_packets: 0
rx1_lro_bytes: 0
rx1_ecn_mark: 0
rx1_removed_vlan_packets: 19868919527
rx1_wqe_err: 0
rx1_mpwqe_filler_cqes: 0
rx1_mpwqe_filler_strides: 0
rx1_buff_alloc_err: 0
rx1_cqe_compress_blks: 420210560
rx1_cqe_compress_pkts: 941233388
rx1_page_reuse: 0
rx1_cache_reuse: 46200002
rx1_cache_full: 9888257242
rx1_cache_empty: 37376
rx1_cache_busy: 9888259746
rx1_cache_waive: 0
rx1_congst_umr: 0
rx1_arfs_err: 0
rx1_xdp_tx_xmit: 0
rx1_xdp_tx_full: 0
rx1_xdp_tx_err: 0
rx1_xdp_tx_cqes: 0
rx2_packets: 19575013662
rx2_bytes: 25759818417945
rx2_csum_complete: 19575013662
rx2_csum_unnecessary: 0
rx2_csum_unnecessary_inner: 0
rx2_csum_none: 0
rx2_xdp_drop: 0
rx2_xdp_redirect: 0
rx2_lro_packets: 0
rx2_lro_bytes: 0
rx2_ecn_mark: 0
rx2_removed_vlan_packets: 19575013662
rx2_wqe_err: 0
rx2_mpwqe_filler_cqes: 0
rx2_mpwqe_filler_strides: 0
rx2_buff_alloc_err: 0
rx2_cqe_compress_blks: 412345511
rx2_cqe_compress_pkts: 923376167
rx2_page_reuse: 0
rx2_cache_reuse: 38837731
rx2_cache_full: 9748666548
rx2_cache_empty: 37376
rx2_cache_busy: 9748669093
rx2_cache_waive: 0
rx2_congst_umr: 0
rx2_arfs_err: 0
rx2_xdp_tx_xmit: 0
rx2_xdp_tx_full: 0
rx2_xdp_tx_err: 0
rx2_xdp_tx_cqes: 0
rx3_packets: 19795911749
rx3_bytes: 25969475566905
rx3_csum_complete: 19795911749
rx3_csum_unnecessary: 0
rx3_csum_unnecessary_inner: 0
rx3_csum_none: 0
rx3_xdp_drop: 0
rx3_xdp_redirect: 0
rx3_lro_packets: 0
rx3_lro_bytes: 0
rx3_ecn_mark: 0
rx3_removed_vlan_packets: 19795911749
rx3_wqe_err: 0
rx3_mpwqe_filler_cqes: 0
rx3_mpwqe_filler_strides: 0
rx3_buff_alloc_err: 0
rx3_cqe_compress_blks: 416658765
rx3_cqe_compress_pkts: 934986266
rx3_page_reuse: 0
rx3_cache_reuse: 34542124
rx3_cache_full: 9863411232
rx3_cache_empty: 37376
rx3_cache_busy: 9863413732
rx3_cache_waive: 0
rx3_congst_umr: 0
rx3_arfs_err: 0
rx3_xdp_tx_xmit: 0
rx3_xdp_tx_full: 0
rx3_xdp_tx_err: 0
rx3_xdp_tx_cqes: 0
rx4_packets: 20445652378
rx4_bytes: 26949065110265
rx4_csum_complete: 20445652378
rx4_csum_unnecessary: 0
rx4_csum_unnecessary_inner: 0
rx4_csum_none: 0
rx4_xdp_drop: 0
rx4_xdp_redirect: 0
rx4_lro_packets: 0
rx4_lro_bytes: 0
rx4_ecn_mark: 0
rx4_removed_vlan_packets: 20445652378
rx4_wqe_err: 0
rx4_mpwqe_filler_cqes: 0
rx4_mpwqe_filler_strides: 0
rx4_buff_alloc_err: 0
rx4_cqe_compress_blks: 506085858
rx4_cqe_compress_pkts: 1147860328
rx4_page_reuse: 0
rx4_cache_reuse: 10122542864
rx4_cache_full: 100281206
rx4_cache_empty: 37376
rx4_cache_busy: 100283304
rx4_cache_waive: 0
rx4_congst_umr: 0
rx4_arfs_err: 0
rx4_xdp_tx_xmit: 0
rx4_xdp_tx_full: 0
rx4_xdp_tx_err: 0
rx4_xdp_tx_cqes: 0
rx5_packets: 19622362246
rx5_bytes: 25843450982982
rx5_csum_complete: 19622362246
rx5_csum_unnecessary: 0
rx5_csum_unnecessary_inner: 0
rx5_csum_none: 0
rx5_xdp_drop: 0
rx5_xdp_redirect: 0
rx5_lro_packets: 0
rx5_lro_bytes: 0
rx5_ecn_mark: 0
rx5_removed_vlan_packets: 19622362246
rx5_wqe_err: 0
rx5_mpwqe_filler_cqes: 0
rx5_mpwqe_filler_strides: 0
rx5_buff_alloc_err: 0
rx5_cqe_compress_blks: 422840924
rx5_cqe_compress_pkts: 948005878
rx5_page_reuse: 0
rx5_cache_reuse: 31285453
rx5_cache_full: 9779893117
rx5_cache_empty: 37376
rx5_cache_busy: 9779895647
rx5_cache_waive: 0
rx5_congst_umr: 0
rx5_arfs_err: 0
rx5_xdp_tx_xmit: 0
rx5_xdp_tx_full: 0
rx5_xdp_tx_err: 0
rx5_xdp_tx_cqes: 0
rx6_packets: 19788231278
rx6_bytes: 25985783006486
rx6_csum_complete: 19788231278
rx6_csum_unnecessary: 0
rx6_csum_unnecessary_inner: 0
rx6_csum_none: 0
rx6_xdp_drop: 0
rx6_xdp_redirect: 0
rx6_lro_packets: 0
rx6_lro_bytes: 0
rx6_ecn_mark: 0
rx6_removed_vlan_packets: 19788231278
rx6_wqe_err: 0
rx6_mpwqe_filler_cqes: 0
rx6_mpwqe_filler_strides: 0
rx6_buff_alloc_err: 0
rx6_cqe_compress_blks: 418799056
rx6_cqe_compress_pkts: 938282685
rx6_page_reuse: 0
rx6_cache_reuse: 18114793
rx6_cache_full: 9875998295
rx6_cache_empty: 37376
rx6_cache_busy: 9876000831
rx6_cache_waive: 0
rx6_congst_umr: 0
rx6_arfs_err: 0
rx6_xdp_tx_xmit: 0
rx6_xdp_tx_full: 0
rx6_xdp_tx_err: 0
rx6_xdp_tx_cqes: 0
rx7_packets: 19795759168
rx7_bytes: 26085056586860
rx7_csum_complete: 19795759168
rx7_csum_unnecessary: 0
rx7_csum_unnecessary_inner: 0
rx7_csum_none: 0
rx7_xdp_drop: 0
rx7_xdp_redirect: 0
rx7_lro_packets: 0
rx7_lro_bytes: 0
rx7_ecn_mark: 0
rx7_removed_vlan_packets: 19795759168
rx7_wqe_err: 0
rx7_mpwqe_filler_cqes: 0
rx7_mpwqe_filler_strides: 0
rx7_buff_alloc_err: 0
rx7_cqe_compress_blks: 413959224
rx7_cqe_compress_pkts: 927675936
rx7_page_reuse: 0
rx7_cache_reuse: 23902990
rx7_cache_full: 9873974042
rx7_cache_empty: 37376
rx7_cache_busy: 9873976574
rx7_cache_waive: 0
rx7_congst_umr: 0
rx7_arfs_err: 0
rx7_xdp_tx_xmit: 0
rx7_xdp_tx_full: 0
rx7_xdp_tx_err: 0
rx7_xdp_tx_cqes: 0
rx8_packets: 19963477439
rx8_bytes: 26384640501789
rx8_csum_complete: 19963477439
rx8_csum_unnecessary: 0
rx8_csum_unnecessary_inner: 0
rx8_csum_none: 0
rx8_xdp_drop: 0
rx8_xdp_redirect: 0
rx8_lro_packets: 0
rx8_lro_bytes: 0
rx8_ecn_mark: 0
rx8_removed_vlan_packets: 19963477439
rx8_wqe_err: 0
rx8_mpwqe_filler_cqes: 0
rx8_mpwqe_filler_strides: 0
rx8_buff_alloc_err: 0
rx8_cqe_compress_blks: 420422857
rx8_cqe_compress_pkts: 942720292
rx8_page_reuse: 0
rx8_cache_reuse: 88181713
rx8_cache_full: 9893554525
rx8_cache_empty: 37376
rx8_cache_busy: 9893556983
rx8_cache_waive: 0
rx8_congst_umr: 0
rx8_arfs_err: 0
rx8_xdp_tx_xmit: 0
rx8_xdp_tx_full: 0
rx8_xdp_tx_err: 0
rx8_xdp_tx_cqes: 0
rx9_packets: 19726642138
rx9_bytes: 26063924286499
rx9_csum_complete: 19726642138
rx9_csum_unnecessary: 0
rx9_csum_unnecessary_inner: 0
rx9_csum_none: 0
rx9_xdp_drop: 0
rx9_xdp_redirect: 0
rx9_lro_packets: 0
rx9_lro_bytes: 0
rx9_ecn_mark: 0
rx9_removed_vlan_packets: 19726642138
rx9_wqe_err: 0
rx9_mpwqe_filler_cqes: 0
rx9_mpwqe_filler_strides: 0
rx9_buff_alloc_err: 0
rx9_cqe_compress_blks: 424227411
rx9_cqe_compress_pkts: 951534873
rx9_page_reuse: 0
rx9_cache_reuse: 482901440
rx9_cache_full: 9380417487
rx9_cache_empty: 37376
rx9_cache_busy: 9380419608
rx9_cache_waive: 0
rx9_congst_umr: 0
rx9_arfs_err: 0
rx9_xdp_tx_xmit: 0
rx9_xdp_tx_full: 0
rx9_xdp_tx_err: 0
rx9_xdp_tx_cqes: 0
rx10_packets: 19901229170
rx10_bytes: 26300854495044
rx10_csum_complete: 19901229170
rx10_csum_unnecessary: 0
rx10_csum_unnecessary_inner: 0
rx10_csum_none: 0
rx10_xdp_drop: 0
rx10_xdp_redirect: 0
rx10_lro_packets: 0
rx10_lro_bytes: 0
rx10_ecn_mark: 0
rx10_removed_vlan_packets: 19901229170
rx10_wqe_err: 0
rx10_mpwqe_filler_cqes: 0
rx10_mpwqe_filler_strides: 0
rx10_buff_alloc_err: 0
rx10_cqe_compress_blks: 419082938
rx10_cqe_compress_pkts: 940791347
rx10_page_reuse: 0
rx10_cache_reuse: 14896055
rx10_cache_full: 9935715977
rx10_cache_empty: 37376
rx10_cache_busy: 9935718513
rx10_cache_waive: 0
rx10_congst_umr: 0
rx10_arfs_err: 0
rx10_xdp_tx_xmit: 0
rx10_xdp_tx_full: 0
rx10_xdp_tx_err: 0
rx10_xdp_tx_cqes: 0
rx11_packets: 20352190494
rx11_bytes: 26851034425372
rx11_csum_complete: 20352190494
rx11_csum_unnecessary: 0
rx11_csum_unnecessary_inner: 0
rx11_csum_none: 0
rx11_xdp_drop: 0
rx11_xdp_redirect: 0
rx11_lro_packets: 0
rx11_lro_bytes: 0
rx11_ecn_mark: 0
rx11_removed_vlan_packets: 20352190494
rx11_wqe_err: 0
rx11_mpwqe_filler_cqes: 0
rx11_mpwqe_filler_strides: 0
rx11_buff_alloc_err: 0
rx11_cqe_compress_blks: 501992147
rx11_cqe_compress_pkts: 1140398610
rx11_page_reuse: 0
rx11_cache_reuse: 10071721531
rx11_cache_full: 104371621
rx11_cache_empty: 37376
rx11_cache_busy: 104373697
rx11_cache_waive: 0
rx11_congst_umr: 0
rx11_arfs_err: 0
rx11_xdp_tx_xmit: 0
rx11_xdp_tx_full: 0
rx11_xdp_tx_err: 0
rx11_xdp_tx_cqes: 0
rx12_packets: 19934747149
rx12_bytes: 26296478787829
rx12_csum_complete: 19934747149
rx12_csum_unnecessary: 0
rx12_csum_unnecessary_inner: 0
rx12_csum_none: 0
rx12_xdp_drop: 0
rx12_xdp_redirect: 0
rx12_lro_packets: 0
rx12_lro_bytes: 0
rx12_ecn_mark: 0
rx12_removed_vlan_packets: 19934747149
rx12_wqe_err: 0
rx12_mpwqe_filler_cqes: 0
rx12_mpwqe_filler_strides: 0
rx12_buff_alloc_err: 0
rx12_cqe_compress_blks: 443350570
rx12_cqe_compress_pkts: 995997220
rx12_page_reuse: 0
rx12_cache_reuse: 9864934174
rx12_cache_full: 102437428
rx12_cache_empty: 37376
rx12_cache_busy: 102439382
rx12_cache_waive: 0
rx12_congst_umr: 0
rx12_arfs_err: 0
rx12_xdp_tx_xmit: 0
rx12_xdp_tx_full: 0
rx12_xdp_tx_err: 0
rx12_xdp_tx_cqes: 0
rx13_packets: 19866908096
rx13_bytes: 26160931936186
rx13_csum_complete: 19866908096
rx13_csum_unnecessary: 0
rx13_csum_unnecessary_inner: 0
rx13_csum_none: 0
rx13_xdp_drop: 0
rx13_xdp_redirect: 0
rx13_lro_packets: 0
rx13_lro_bytes: 0
rx13_ecn_mark: 0
rx13_removed_vlan_packets: 19866908096
rx13_wqe_err: 0
rx13_mpwqe_filler_cqes: 0
rx13_mpwqe_filler_strides: 0
rx13_buff_alloc_err: 0
rx13_cqe_compress_blks: 413640141
rx13_cqe_compress_pkts: 926175066
rx13_page_reuse: 0
rx13_cache_reuse: 36358610
rx13_cache_full: 9897092921
rx13_cache_empty: 37376
rx13_cache_busy: 9897095422
rx13_cache_waive: 0
rx13_congst_umr: 0
rx13_arfs_err: 0
rx13_xdp_tx_xmit: 0
rx13_xdp_tx_full: 0
rx13_xdp_tx_err: 0
rx13_xdp_tx_cqes: 0
rx14_packets: 20229035746
rx14_bytes: 26655092809172
rx14_csum_complete: 20229035746
rx14_csum_unnecessary: 0
rx14_csum_unnecessary_inner: 0
rx14_csum_none: 0
rx14_xdp_drop: 0
rx14_xdp_redirect: 0
rx14_lro_packets: 0
rx14_lro_bytes: 0
rx14_ecn_mark: 0
rx14_removed_vlan_packets: 20229035746
rx14_wqe_err: 0
rx14_mpwqe_filler_cqes: 0
rx14_mpwqe_filler_strides: 0
rx14_buff_alloc_err: 0
rx14_cqe_compress_blks: 460990337
rx14_cqe_compress_pkts: 1041287948
rx14_page_reuse: 0
rx14_cache_reuse: 25649275
rx14_cache_full: 10088866045
rx14_cache_empty: 37376
rx14_cache_busy: 10088868574
rx14_cache_waive: 0
rx14_congst_umr: 0
rx14_arfs_err: 0
rx14_xdp_tx_xmit: 0
rx14_xdp_tx_full: 0
rx14_xdp_tx_err: 0
rx14_xdp_tx_cqes: 0
rx15_packets: 20528177154
rx15_bytes: 27029263893264
rx15_csum_complete: 20528177154
rx15_csum_unnecessary: 0
rx15_csum_unnecessary_inner: 0
rx15_csum_none: 0
rx15_xdp_drop: 0
rx15_xdp_redirect: 0
rx15_lro_packets: 0
rx15_lro_bytes: 0
rx15_ecn_mark: 0
rx15_removed_vlan_packets: 20528177154
rx15_wqe_err: 0
rx15_mpwqe_filler_cqes: 0
rx15_mpwqe_filler_strides: 0
rx15_buff_alloc_err: 0
rx15_cqe_compress_blks: 476776176
rx15_cqe_compress_pkts: 1076153263
rx15_page_reuse: 0
rx15_cache_reuse: 48426735
rx15_cache_full: 10215659289
rx15_cache_empty: 37376
rx15_cache_busy: 10215661817
rx15_cache_waive: 0
rx15_congst_umr: 0
rx15_arfs_err: 0
rx15_xdp_tx_xmit: 0
rx15_xdp_tx_full: 0
rx15_xdp_tx_err: 0
rx15_xdp_tx_cqes: 0
rx16_packets: 16104078098
rx16_bytes: 21256361789679
rx16_csum_complete: 16104078098
rx16_csum_unnecessary: 0
rx16_csum_unnecessary_inner: 0
rx16_csum_none: 0
rx16_xdp_drop: 0
rx16_xdp_redirect: 0
rx16_lro_packets: 0
rx16_lro_bytes: 0
rx16_ecn_mark: 0
rx16_removed_vlan_packets: 16104078098
rx16_wqe_err: 0
rx16_mpwqe_filler_cqes: 0
rx16_mpwqe_filler_strides: 0
rx16_buff_alloc_err: 0
rx16_cqe_compress_blks: 352082054
rx16_cqe_compress_pkts: 787161670
rx16_page_reuse: 0
rx16_cache_reuse: 25912567
rx16_cache_full: 8026124051
rx16_cache_empty: 37376
rx16_cache_busy: 8026126465
rx16_cache_waive: 0
rx16_congst_umr: 0
rx16_arfs_err: 0
rx16_xdp_tx_xmit: 0
rx16_xdp_tx_full: 0
rx16_xdp_tx_err: 0
rx16_xdp_tx_cqes: 0
rx17_packets: 16314055017
rx17_bytes: 21589139030173
rx17_csum_complete: 16314055017
rx17_csum_unnecessary: 0
rx17_csum_unnecessary_inner: 0
rx17_csum_none: 0
rx17_xdp_drop: 0
rx17_xdp_redirect: 0
rx17_lro_packets: 0
rx17_lro_bytes: 0
rx17_ecn_mark: 0
rx17_removed_vlan_packets: 16314055017
rx17_wqe_err: 0
rx17_mpwqe_filler_cqes: 0
rx17_mpwqe_filler_strides: 0
rx17_buff_alloc_err: 0
rx17_cqe_compress_blks: 387834541
rx17_cqe_compress_pkts: 871851081
rx17_page_reuse: 0
rx17_cache_reuse: 24021313
rx17_cache_full: 8133003829
rx17_cache_empty: 37376
rx17_cache_busy: 8133006175
rx17_cache_waive: 0
rx17_congst_umr: 0
rx17_arfs_err: 0
rx17_xdp_tx_xmit: 0
rx17_xdp_tx_full: 0
rx17_xdp_tx_err: 0
rx17_xdp_tx_cqes: 0
rx18_packets: 16439016814
rx18_bytes: 21648651917475
rx18_csum_complete: 16439016814
rx18_csum_unnecessary: 0
rx18_csum_unnecessary_inner: 0
rx18_csum_none: 0
rx18_xdp_drop: 0
rx18_xdp_redirect: 0
rx18_lro_packets: 0
rx18_lro_bytes: 0
rx18_ecn_mark: 0
rx18_removed_vlan_packets: 16439016814
rx18_wqe_err: 0
rx18_mpwqe_filler_cqes: 0
rx18_mpwqe_filler_strides: 0
rx18_buff_alloc_err: 0
rx18_cqe_compress_blks: 375066666
rx18_cqe_compress_pkts: 843563974
rx18_page_reuse: 0
rx18_cache_reuse: 8151064266
rx18_cache_full: 68442025
rx18_cache_empty: 37376
rx18_cache_busy: 68444122
rx18_cache_waive: 0
rx18_congst_umr: 0
rx18_arfs_err: 0
rx18_xdp_tx_xmit: 0
rx18_xdp_tx_full: 0
rx18_xdp_tx_err: 0
rx18_xdp_tx_cqes: 0
rx19_packets: 16641223506
rx19_bytes: 21964749940935
rx19_csum_complete: 16641223506
rx19_csum_unnecessary: 0
rx19_csum_unnecessary_inner: 0
rx19_csum_none: 0
rx19_xdp_drop: 0
rx19_xdp_redirect: 0
rx19_lro_packets: 0
rx19_lro_bytes: 0
rx19_ecn_mark: 0
rx19_removed_vlan_packets: 16641223506
rx19_wqe_err: 0
rx19_mpwqe_filler_cqes: 0
rx19_mpwqe_filler_strides: 0
rx19_buff_alloc_err: 0
rx19_cqe_compress_blks: 387825932
rx19_cqe_compress_pkts: 872266355
rx19_page_reuse: 0
rx19_cache_reuse: 116433620
rx19_cache_full: 8204175954
rx19_cache_empty: 37376
rx19_cache_busy: 8204178120
rx19_cache_waive: 0
rx19_congst_umr: 0
rx19_arfs_err: 0
rx19_xdp_tx_xmit: 0
rx19_xdp_tx_full: 0
rx19_xdp_tx_err: 0
rx19_xdp_tx_cqes: 0
rx20_packets: 16206927741
rx20_bytes: 21387447038430
rx20_csum_complete: 16206927741
rx20_csum_unnecessary: 0
rx20_csum_unnecessary_inner: 0
rx20_csum_none: 0
rx20_xdp_drop: 0
rx20_xdp_redirect: 0
rx20_lro_packets: 0
rx20_lro_bytes: 0
rx20_ecn_mark: 0
rx20_removed_vlan_packets: 16206927741
rx20_wqe_err: 0
rx20_mpwqe_filler_cqes: 0
rx20_mpwqe_filler_strides: 0
rx20_buff_alloc_err: 0
rx20_cqe_compress_blks: 370144620
rx20_cqe_compress_pkts: 829122671
rx20_page_reuse: 0
rx20_cache_reuse: 8053733744
rx20_cache_full: 49728026
rx20_cache_empty: 37376
rx20_cache_busy: 49730116
rx20_cache_waive: 0
rx20_congst_umr: 0
rx20_arfs_err: 0
rx20_xdp_tx_xmit: 0
rx20_xdp_tx_full: 0
rx20_xdp_tx_err: 0
rx20_xdp_tx_cqes: 0
rx21_packets: 16562361314
rx21_bytes: 21856653284356
rx21_csum_complete: 16562361314
rx21_csum_unnecessary: 0
rx21_csum_unnecessary_inner: 0
rx21_csum_none: 0
rx21_xdp_drop: 0
rx21_xdp_redirect: 0
rx21_lro_packets: 0
rx21_lro_bytes: 0
rx21_ecn_mark: 0
rx21_removed_vlan_packets: 16562361314
rx21_wqe_err: 0
rx21_mpwqe_filler_cqes: 0
rx21_mpwqe_filler_strides: 0
rx21_buff_alloc_err: 0
rx21_cqe_compress_blks: 350790425
rx21_cqe_compress_pkts: 783850729
rx21_page_reuse: 0
rx21_cache_reuse: 28077493
rx21_cache_full: 8253100706
rx21_cache_empty: 37376
rx21_cache_busy: 8253103147
rx21_cache_waive: 0
rx21_congst_umr: 0
rx21_arfs_err: 0
rx21_xdp_tx_xmit: 0
rx21_xdp_tx_full: 0
rx21_xdp_tx_err: 0
rx21_xdp_tx_cqes: 0
rx22_packets: 16350307571
rx22_bytes: 21408575325592
rx22_csum_complete: 16350307571
rx22_csum_unnecessary: 0
rx22_csum_unnecessary_inner: 0
rx22_csum_none: 0
rx22_xdp_drop: 0
rx22_xdp_redirect: 0
rx22_lro_packets: 0
rx22_lro_bytes: 0
rx22_ecn_mark: 0
rx22_removed_vlan_packets: 16350307571
rx22_wqe_err: 0
rx22_mpwqe_filler_cqes: 0
rx22_mpwqe_filler_strides: 0
rx22_buff_alloc_err: 0
rx22_cqe_compress_blks: 353531065
rx22_cqe_compress_pkts: 790814415
rx22_page_reuse: 0
rx22_cache_reuse: 16934343
rx22_cache_full: 8158216889
rx22_cache_empty: 37376
rx22_cache_busy: 8158219417
rx22_cache_waive: 0
rx22_congst_umr: 0
rx22_arfs_err: 0
rx22_xdp_tx_xmit: 0
rx22_xdp_tx_full: 0
rx22_xdp_tx_err: 0
rx22_xdp_tx_cqes: 0
rx23_packets: 16019811764
rx23_bytes: 21137182570985
rx23_csum_complete: 16019811764
rx23_csum_unnecessary: 0
rx23_csum_unnecessary_inner: 0
rx23_csum_none: 0
rx23_xdp_drop: 0
rx23_xdp_redirect: 0
rx23_lro_packets: 0
rx23_lro_bytes: 0
rx23_ecn_mark: 0
rx23_removed_vlan_packets: 16019811764
rx23_wqe_err: 0
rx23_mpwqe_filler_cqes: 0
rx23_mpwqe_filler_strides: 0
rx23_buff_alloc_err: 0
rx23_cqe_compress_blks: 349733033
rx23_cqe_compress_pkts: 781248862
rx23_page_reuse: 0
rx23_cache_reuse: 33422343
rx23_cache_full: 7976481152
rx23_cache_empty: 37376
rx23_cache_busy: 7976483525
rx23_cache_waive: 0
rx23_congst_umr: 0
rx23_arfs_err: 0
rx23_xdp_tx_xmit: 0
rx23_xdp_tx_full: 0
rx23_xdp_tx_err: 0
rx23_xdp_tx_cqes: 0
rx24_packets: 16212040646
rx24_bytes: 21393399325700
rx24_csum_complete: 16212040646
rx24_csum_unnecessary: 0
rx24_csum_unnecessary_inner: 0
rx24_csum_none: 0
rx24_xdp_drop: 0
rx24_xdp_redirect: 0
rx24_lro_packets: 0
rx24_lro_bytes: 0
rx24_ecn_mark: 0
rx24_removed_vlan_packets: 16212040646
rx24_wqe_err: 0
rx24_mpwqe_filler_cqes: 0
rx24_mpwqe_filler_strides: 0
rx24_buff_alloc_err: 0
rx24_cqe_compress_blks: 379833752
rx24_cqe_compress_pkts: 852020179
rx24_page_reuse: 0
rx24_cache_reuse: 8033552512
rx24_cache_full: 72465843
rx24_cache_empty: 37376
rx24_cache_busy: 72467789
rx24_cache_waive: 0
rx24_congst_umr: 0
rx24_arfs_err: 0
rx24_xdp_tx_xmit: 0
rx24_xdp_tx_full: 0
rx24_xdp_tx_err: 0
rx24_xdp_tx_cqes: 0
rx25_packets: 16412186257
rx25_bytes: 21651198388407
rx25_csum_complete: 16412186257
rx25_csum_unnecessary: 0
rx25_csum_unnecessary_inner: 0
rx25_csum_none: 0
rx25_xdp_drop: 0
rx25_xdp_redirect: 0
rx25_lro_packets: 0
rx25_lro_bytes: 0
rx25_ecn_mark: 0
rx25_removed_vlan_packets: 16412186257
rx25_wqe_err: 0
rx25_mpwqe_filler_cqes: 0
rx25_mpwqe_filler_strides: 0
rx25_buff_alloc_err: 0
rx25_cqe_compress_blks: 383979685
rx25_cqe_compress_pkts: 861985772
rx25_page_reuse: 0
rx25_cache_reuse: 8129807841
rx25_cache_full: 76283342
rx25_cache_empty: 37376
rx25_cache_busy: 76285271
rx25_cache_waive: 0
rx25_congst_umr: 0
rx25_arfs_err: 0
rx25_xdp_tx_xmit: 0
rx25_xdp_tx_full: 0
rx25_xdp_tx_err: 0
rx25_xdp_tx_cqes: 0
rx26_packets: 16304310003
rx26_bytes: 21571217538721
rx26_csum_complete: 16304310003
rx26_csum_unnecessary: 0
rx26_csum_unnecessary_inner: 0
rx26_csum_none: 0
rx26_xdp_drop: 0
rx26_xdp_redirect: 0
rx26_lro_packets: 0
rx26_lro_bytes: 0
rx26_ecn_mark: 0
rx26_removed_vlan_packets: 16304310003
rx26_wqe_err: 0
rx26_mpwqe_filler_cqes: 0
rx26_mpwqe_filler_strides: 0
rx26_buff_alloc_err: 0
rx26_cqe_compress_blks: 353314041
rx26_cqe_compress_pkts: 788838424
rx26_page_reuse: 0
rx26_cache_reuse: 19673790
rx26_cache_full: 8132478659
rx26_cache_empty: 37376
rx26_cache_busy: 8132481198
rx26_cache_waive: 0
rx26_congst_umr: 0
rx26_arfs_err: 0
rx26_xdp_tx_xmit: 0
rx26_xdp_tx_full: 0
rx26_xdp_tx_err: 0
rx26_xdp_tx_cqes: 0
rx27_packets: 16171856079
rx27_bytes: 21376891736540
rx27_csum_complete: 16171856079
rx27_csum_unnecessary: 0
rx27_csum_unnecessary_inner: 0
rx27_csum_none: 0
rx27_xdp_drop: 0
rx27_xdp_redirect: 0
rx27_lro_packets: 0
rx27_lro_bytes: 0
rx27_ecn_mark: 0
rx27_removed_vlan_packets: 16171856079
rx27_wqe_err: 0
rx27_mpwqe_filler_cqes: 0
rx27_mpwqe_filler_strides: 0
rx27_buff_alloc_err: 0
rx27_cqe_compress_blks: 386632845
rx27_cqe_compress_pkts: 869362576
rx27_page_reuse: 0
rx27_cache_reuse: 10070560
rx27_cache_full: 8075854928
rx27_cache_empty: 37376
rx27_cache_busy: 8075857468
rx27_cache_waive: 0
rx27_congst_umr: 0
rx27_arfs_err: 0
rx27_xdp_tx_xmit: 0
rx27_xdp_tx_full: 0
rx27_xdp_tx_err: 0
rx27_xdp_tx_cqes: 0
rx28_packets: 0
rx28_bytes: 0
rx28_csum_complete: 0
rx28_csum_unnecessary: 0
rx28_csum_unnecessary_inner: 0
rx28_csum_none: 0
rx28_xdp_drop: 0
rx28_xdp_redirect: 0
rx28_lro_packets: 0
rx28_lro_bytes: 0
rx28_ecn_mark: 0
rx28_removed_vlan_packets: 0
rx28_wqe_err: 0
rx28_mpwqe_filler_cqes: 0
rx28_mpwqe_filler_strides: 0
rx28_buff_alloc_err: 0
rx28_cqe_compress_blks: 0
rx28_cqe_compress_pkts: 0
rx28_page_reuse: 0
rx28_cache_reuse: 0
rx28_cache_full: 0
rx28_cache_empty: 2560
rx28_cache_busy: 0
rx28_cache_waive: 0
rx28_congst_umr: 0
rx28_arfs_err: 0
rx28_xdp_tx_xmit: 0
rx28_xdp_tx_full: 0
rx28_xdp_tx_err: 0
rx28_xdp_tx_cqes: 0
rx29_packets: 0
rx29_bytes: 0
rx29_csum_complete: 0
rx29_csum_unnecessary: 0
rx29_csum_unnecessary_inner: 0
rx29_csum_none: 0
rx29_xdp_drop: 0
rx29_xdp_redirect: 0
rx29_lro_packets: 0
rx29_lro_bytes: 0
rx29_ecn_mark: 0
rx29_removed_vlan_packets: 0
rx29_wqe_err: 0
rx29_mpwqe_filler_cqes: 0
rx29_mpwqe_filler_strides: 0
rx29_buff_alloc_err: 0
rx29_cqe_compress_blks: 0
rx29_cqe_compress_pkts: 0
rx29_page_reuse: 0
rx29_cache_reuse: 0
rx29_cache_full: 0
rx29_cache_empty: 2560
rx29_cache_busy: 0
rx29_cache_waive: 0
rx29_congst_umr: 0
rx29_arfs_err: 0
rx29_xdp_tx_xmit: 0
rx29_xdp_tx_full: 0
rx29_xdp_tx_err: 0
rx29_xdp_tx_cqes: 0
rx30_packets: 0
rx30_bytes: 0
rx30_csum_complete: 0
rx30_csum_unnecessary: 0
rx30_csum_unnecessary_inner: 0
rx30_csum_none: 0
rx30_xdp_drop: 0
rx30_xdp_redirect: 0
rx30_lro_packets: 0
rx30_lro_bytes: 0
rx30_ecn_mark: 0
rx30_removed_vlan_packets: 0
rx30_wqe_err: 0
rx30_mpwqe_filler_cqes: 0
rx30_mpwqe_filler_strides: 0
rx30_buff_alloc_err: 0
rx30_cqe_compress_blks: 0
rx30_cqe_compress_pkts: 0
rx30_page_reuse: 0
rx30_cache_reuse: 0
rx30_cache_full: 0
rx30_cache_empty: 2560
rx30_cache_busy: 0
rx30_cache_waive: 0
rx30_congst_umr: 0
rx30_arfs_err: 0
rx30_xdp_tx_xmit: 0
rx30_xdp_tx_full: 0
rx30_xdp_tx_err: 0
rx30_xdp_tx_cqes: 0
rx31_packets: 0
rx31_bytes: 0
rx31_csum_complete: 0
rx31_csum_unnecessary: 0
rx31_csum_unnecessary_inner: 0
rx31_csum_none: 0
rx31_xdp_drop: 0
rx31_xdp_redirect: 0
rx31_lro_packets: 0
rx31_lro_bytes: 0
rx31_ecn_mark: 0
rx31_removed_vlan_packets: 0
rx31_wqe_err: 0
rx31_mpwqe_filler_cqes: 0
rx31_mpwqe_filler_strides: 0
rx31_buff_alloc_err: 0
rx31_cqe_compress_blks: 0
rx31_cqe_compress_pkts: 0
rx31_page_reuse: 0
rx31_cache_reuse: 0
rx31_cache_full: 0
rx31_cache_empty: 2560
rx31_cache_busy: 0
rx31_cache_waive: 0
rx31_congst_umr: 0
rx31_arfs_err: 0
rx31_xdp_tx_xmit: 0
rx31_xdp_tx_full: 0
rx31_xdp_tx_err: 0
rx31_xdp_tx_cqes: 0
rx32_packets: 0
rx32_bytes: 0
rx32_csum_complete: 0
rx32_csum_unnecessary: 0
rx32_csum_unnecessary_inner: 0
rx32_csum_none: 0
rx32_xdp_drop: 0
rx32_xdp_redirect: 0
rx32_lro_packets: 0
rx32_lro_bytes: 0
rx32_ecn_mark: 0
rx32_removed_vlan_packets: 0
rx32_wqe_err: 0
rx32_mpwqe_filler_cqes: 0
rx32_mpwqe_filler_strides: 0
rx32_buff_alloc_err: 0
rx32_cqe_compress_blks: 0
rx32_cqe_compress_pkts: 0
rx32_page_reuse: 0
rx32_cache_reuse: 0
rx32_cache_full: 0
rx32_cache_empty: 2560
rx32_cache_busy: 0
rx32_cache_waive: 0
rx32_congst_umr: 0
rx32_arfs_err: 0
rx32_xdp_tx_xmit: 0
rx32_xdp_tx_full: 0
rx32_xdp_tx_err: 0
rx32_xdp_tx_cqes: 0
rx33_packets: 0
rx33_bytes: 0
rx33_csum_complete: 0
rx33_csum_unnecessary: 0
rx33_csum_unnecessary_inner: 0
rx33_csum_none: 0
rx33_xdp_drop: 0
rx33_xdp_redirect: 0
rx33_lro_packets: 0
rx33_lro_bytes: 0
rx33_ecn_mark: 0
rx33_removed_vlan_packets: 0
rx33_wqe_err: 0
rx33_mpwqe_filler_cqes: 0
rx33_mpwqe_filler_strides: 0
rx33_buff_alloc_err: 0
rx33_cqe_compress_blks: 0
rx33_cqe_compress_pkts: 0
rx33_page_reuse: 0
rx33_cache_reuse: 0
rx33_cache_full: 0
rx33_cache_empty: 2560
rx33_cache_busy: 0
rx33_cache_waive: 0
rx33_congst_umr: 0
rx33_arfs_err: 0
rx33_xdp_tx_xmit: 0
rx33_xdp_tx_full: 0
rx33_xdp_tx_err: 0
rx33_xdp_tx_cqes: 0
rx34_packets: 0
rx34_bytes: 0
rx34_csum_complete: 0
rx34_csum_unnecessary: 0
rx34_csum_unnecessary_inner: 0
rx34_csum_none: 0
rx34_xdp_drop: 0
rx34_xdp_redirect: 0
rx34_lro_packets: 0
rx34_lro_bytes: 0
rx34_ecn_mark: 0
rx34_removed_vlan_packets: 0
rx34_wqe_err: 0
rx34_mpwqe_filler_cqes: 0
rx34_mpwqe_filler_strides: 0
rx34_buff_alloc_err: 0
rx34_cqe_compress_blks: 0
rx34_cqe_compress_pkts: 0
rx34_page_reuse: 0
rx34_cache_reuse: 0
rx34_cache_full: 0
rx34_cache_empty: 2560
rx34_cache_busy: 0
rx34_cache_waive: 0
rx34_congst_umr: 0
rx34_arfs_err: 0
rx34_xdp_tx_xmit: 0
rx34_xdp_tx_full: 0
rx34_xdp_tx_err: 0
rx34_xdp_tx_cqes: 0
rx35_packets: 0
rx35_bytes: 0
rx35_csum_complete: 0
rx35_csum_unnecessary: 0
rx35_csum_unnecessary_inner: 0
rx35_csum_none: 0
rx35_xdp_drop: 0
rx35_xdp_redirect: 0
rx35_lro_packets: 0
rx35_lro_bytes: 0
rx35_ecn_mark: 0
rx35_removed_vlan_packets: 0
rx35_wqe_err: 0
rx35_mpwqe_filler_cqes: 0
rx35_mpwqe_filler_strides: 0
rx35_buff_alloc_err: 0
rx35_cqe_compress_blks: 0
rx35_cqe_compress_pkts: 0
rx35_page_reuse: 0
rx35_cache_reuse: 0
rx35_cache_full: 0
rx35_cache_empty: 2560
rx35_cache_busy: 0
rx35_cache_waive: 0
rx35_congst_umr: 0
rx35_arfs_err: 0
rx35_xdp_tx_xmit: 0
rx35_xdp_tx_full: 0
rx35_xdp_tx_err: 0
rx35_xdp_tx_cqes: 0
rx36_packets: 0
rx36_bytes: 0
rx36_csum_complete: 0
rx36_csum_unnecessary: 0
rx36_csum_unnecessary_inner: 0
rx36_csum_none: 0
rx36_xdp_drop: 0
rx36_xdp_redirect: 0
rx36_lro_packets: 0
rx36_lro_bytes: 0
rx36_ecn_mark: 0
rx36_removed_vlan_packets: 0
rx36_wqe_err: 0
rx36_mpwqe_filler_cqes: 0
rx36_mpwqe_filler_strides: 0
rx36_buff_alloc_err: 0
rx36_cqe_compress_blks: 0
rx36_cqe_compress_pkts: 0
rx36_page_reuse: 0
rx36_cache_reuse: 0
rx36_cache_full: 0
rx36_cache_empty: 2560
rx36_cache_busy: 0
rx36_cache_waive: 0
rx36_congst_umr: 0
rx36_arfs_err: 0
rx36_xdp_tx_xmit: 0
rx36_xdp_tx_full: 0
rx36_xdp_tx_err: 0
rx36_xdp_tx_cqes: 0
rx37_packets: 0
rx37_bytes: 0
rx37_csum_complete: 0
rx37_csum_unnecessary: 0
rx37_csum_unnecessary_inner: 0
rx37_csum_none: 0
rx37_xdp_drop: 0
rx37_xdp_redirect: 0
rx37_lro_packets: 0
rx37_lro_bytes: 0
rx37_ecn_mark: 0
rx37_removed_vlan_packets: 0
rx37_wqe_err: 0
rx37_mpwqe_filler_cqes: 0
rx37_mpwqe_filler_strides: 0
rx37_buff_alloc_err: 0
rx37_cqe_compress_blks: 0
rx37_cqe_compress_pkts: 0
rx37_page_reuse: 0
rx37_cache_reuse: 0
rx37_cache_full: 0
rx37_cache_empty: 2560
rx37_cache_busy: 0
rx37_cache_waive: 0
rx37_congst_umr: 0
rx37_arfs_err: 0
rx37_xdp_tx_xmit: 0
rx37_xdp_tx_full: 0
rx37_xdp_tx_err: 0
rx37_xdp_tx_cqes: 0
rx38_packets: 0
rx38_bytes: 0
rx38_csum_complete: 0
rx38_csum_unnecessary: 0
rx38_csum_unnecessary_inner: 0
rx38_csum_none: 0
rx38_xdp_drop: 0
rx38_xdp_redirect: 0
rx38_lro_packets: 0
rx38_lro_bytes: 0
rx38_ecn_mark: 0
rx38_removed_vlan_packets: 0
rx38_wqe_err: 0
rx38_mpwqe_filler_cqes: 0
rx38_mpwqe_filler_strides: 0
rx38_buff_alloc_err: 0
rx38_cqe_compress_blks: 0
rx38_cqe_compress_pkts: 0
rx38_page_reuse: 0
rx38_cache_reuse: 0
rx38_cache_full: 0
rx38_cache_empty: 2560
rx38_cache_busy: 0
rx38_cache_waive: 0
rx38_congst_umr: 0
rx38_arfs_err: 0
rx38_xdp_tx_xmit: 0
rx38_xdp_tx_full: 0
rx38_xdp_tx_err: 0
rx38_xdp_tx_cqes: 0
rx39_packets: 0
rx39_bytes: 0
rx39_csum_complete: 0
rx39_csum_unnecessary: 0
rx39_csum_unnecessary_inner: 0
rx39_csum_none: 0
rx39_xdp_drop: 0
rx39_xdp_redirect: 0
rx39_lro_packets: 0
rx39_lro_bytes: 0
rx39_ecn_mark: 0
rx39_removed_vlan_packets: 0
rx39_wqe_err: 0
rx39_mpwqe_filler_cqes: 0
rx39_mpwqe_filler_strides: 0
rx39_buff_alloc_err: 0
rx39_cqe_compress_blks: 0
rx39_cqe_compress_pkts: 0
rx39_page_reuse: 0
rx39_cache_reuse: 0
rx39_cache_full: 0
rx39_cache_empty: 2560
rx39_cache_busy: 0
rx39_cache_waive: 0
rx39_congst_umr: 0
rx39_arfs_err: 0
rx39_xdp_tx_xmit: 0
rx39_xdp_tx_full: 0
rx39_xdp_tx_err: 0
rx39_xdp_tx_cqes: 0
rx40_packets: 0
rx40_bytes: 0
rx40_csum_complete: 0
rx40_csum_unnecessary: 0
rx40_csum_unnecessary_inner: 0
rx40_csum_none: 0
rx40_xdp_drop: 0
rx40_xdp_redirect: 0
rx40_lro_packets: 0
rx40_lro_bytes: 0
rx40_ecn_mark: 0
rx40_removed_vlan_packets: 0
rx40_wqe_err: 0
rx40_mpwqe_filler_cqes: 0
rx40_mpwqe_filler_strides: 0
rx40_buff_alloc_err: 0
rx40_cqe_compress_blks: 0
rx40_cqe_compress_pkts: 0
rx40_page_reuse: 0
rx40_cache_reuse: 0
rx40_cache_full: 0
rx40_cache_empty: 2560
rx40_cache_busy: 0
rx40_cache_waive: 0
rx40_congst_umr: 0
rx40_arfs_err: 0
rx40_xdp_tx_xmit: 0
rx40_xdp_tx_full: 0
rx40_xdp_tx_err: 0
rx40_xdp_tx_cqes: 0
rx41_packets: 0
rx41_bytes: 0
rx41_csum_complete: 0
rx41_csum_unnecessary: 0
rx41_csum_unnecessary_inner: 0
rx41_csum_none: 0
rx41_xdp_drop: 0
rx41_xdp_redirect: 0
rx41_lro_packets: 0
rx41_lro_bytes: 0
rx41_ecn_mark: 0
rx41_removed_vlan_packets: 0
rx41_wqe_err: 0
rx41_mpwqe_filler_cqes: 0
rx41_mpwqe_filler_strides: 0
rx41_buff_alloc_err: 0
rx41_cqe_compress_blks: 0
rx41_cqe_compress_pkts: 0
rx41_page_reuse: 0
rx41_cache_reuse: 0
rx41_cache_full: 0
rx41_cache_empty: 2560
rx41_cache_busy: 0
rx41_cache_waive: 0
rx41_congst_umr: 0
rx41_arfs_err: 0
rx41_xdp_tx_xmit: 0
rx41_xdp_tx_full: 0
rx41_xdp_tx_err: 0
rx41_xdp_tx_cqes: 0
rx42_packets: 0
rx42_bytes: 0
rx42_csum_complete: 0
rx42_csum_unnecessary: 0
rx42_csum_unnecessary_inner: 0
rx42_csum_none: 0
rx42_xdp_drop: 0
rx42_xdp_redirect: 0
rx42_lro_packets: 0
rx42_lro_bytes: 0
rx42_ecn_mark: 0
rx42_removed_vlan_packets: 0
rx42_wqe_err: 0
rx42_mpwqe_filler_cqes: 0
rx42_mpwqe_filler_strides: 0
rx42_buff_alloc_err: 0
rx42_cqe_compress_blks: 0
rx42_cqe_compress_pkts: 0
rx42_page_reuse: 0
rx42_cache_reuse: 0
rx42_cache_full: 0
rx42_cache_empty: 2560
rx42_cache_busy: 0
rx42_cache_waive: 0
rx42_congst_umr: 0
rx42_arfs_err: 0
rx42_xdp_tx_xmit: 0
rx42_xdp_tx_full: 0
rx42_xdp_tx_err: 0
rx42_xdp_tx_cqes: 0
rx43_packets: 0
rx43_bytes: 0
rx43_csum_complete: 0
rx43_csum_unnecessary: 0
rx43_csum_unnecessary_inner: 0
rx43_csum_none: 0
rx43_xdp_drop: 0
rx43_xdp_redirect: 0
rx43_lro_packets: 0
rx43_lro_bytes: 0
rx43_ecn_mark: 0
rx43_removed_vlan_packets: 0
rx43_wqe_err: 0
rx43_mpwqe_filler_cqes: 0
rx43_mpwqe_filler_strides: 0
rx43_buff_alloc_err: 0
rx43_cqe_compress_blks: 0
rx43_cqe_compress_pkts: 0
rx43_page_reuse: 0
rx43_cache_reuse: 0
rx43_cache_full: 0
rx43_cache_empty: 2560
rx43_cache_busy: 0
rx43_cache_waive: 0
rx43_congst_umr: 0
rx43_arfs_err: 0
rx43_xdp_tx_xmit: 0
rx43_xdp_tx_full: 0
rx43_xdp_tx_err: 0
rx43_xdp_tx_cqes: 0
rx44_packets: 0
rx44_bytes: 0
rx44_csum_complete: 0
rx44_csum_unnecessary: 0
rx44_csum_unnecessary_inner: 0
rx44_csum_none: 0
rx44_xdp_drop: 0
rx44_xdp_redirect: 0
rx44_lro_packets: 0
rx44_lro_bytes: 0
rx44_ecn_mark: 0
rx44_removed_vlan_packets: 0
rx44_wqe_err: 0
rx44_mpwqe_filler_cqes: 0
rx44_mpwqe_filler_strides: 0
rx44_buff_alloc_err: 0
rx44_cqe_compress_blks: 0
rx44_cqe_compress_pkts: 0
rx44_page_reuse: 0
rx44_cache_reuse: 0
rx44_cache_full: 0
rx44_cache_empty: 2560
rx44_cache_busy: 0
rx44_cache_waive: 0
rx44_congst_umr: 0
rx44_arfs_err: 0
rx44_xdp_tx_xmit: 0
rx44_xdp_tx_full: 0
rx44_xdp_tx_err: 0
rx44_xdp_tx_cqes: 0
rx45_packets: 0
rx45_bytes: 0
rx45_csum_complete: 0
rx45_csum_unnecessary: 0
rx45_csum_unnecessary_inner: 0
rx45_csum_none: 0
rx45_xdp_drop: 0
rx45_xdp_redirect: 0
rx45_lro_packets: 0
rx45_lro_bytes: 0
rx45_ecn_mark: 0
rx45_removed_vlan_packets: 0
rx45_wqe_err: 0
rx45_mpwqe_filler_cqes: 0
rx45_mpwqe_filler_strides: 0
rx45_buff_alloc_err: 0
rx45_cqe_compress_blks: 0
rx45_cqe_compress_pkts: 0
rx45_page_reuse: 0
rx45_cache_reuse: 0
rx45_cache_full: 0
rx45_cache_empty: 2560
rx45_cache_busy: 0
rx45_cache_waive: 0
rx45_congst_umr: 0
rx45_arfs_err: 0
rx45_xdp_tx_xmit: 0
rx45_xdp_tx_full: 0
rx45_xdp_tx_err: 0
rx45_xdp_tx_cqes: 0
rx46_packets: 0
rx46_bytes: 0
rx46_csum_complete: 0
rx46_csum_unnecessary: 0
rx46_csum_unnecessary_inner: 0
rx46_csum_none: 0
rx46_xdp_drop: 0
rx46_xdp_redirect: 0
rx46_lro_packets: 0
rx46_lro_bytes: 0
rx46_ecn_mark: 0
rx46_removed_vlan_packets: 0
rx46_wqe_err: 0
rx46_mpwqe_filler_cqes: 0
rx46_mpwqe_filler_strides: 0
rx46_buff_alloc_err: 0
rx46_cqe_compress_blks: 0
rx46_cqe_compress_pkts: 0
rx46_page_reuse: 0
rx46_cache_reuse: 0
rx46_cache_full: 0
rx46_cache_empty: 2560
rx46_cache_busy: 0
rx46_cache_waive: 0
rx46_congst_umr: 0
rx46_arfs_err: 0
rx46_xdp_tx_xmit: 0
rx46_xdp_tx_full: 0
rx46_xdp_tx_err: 0
rx46_xdp_tx_cqes: 0
rx47_packets: 0
rx47_bytes: 0
rx47_csum_complete: 0
rx47_csum_unnecessary: 0
rx47_csum_unnecessary_inner: 0
rx47_csum_none: 0
rx47_xdp_drop: 0
rx47_xdp_redirect: 0
rx47_lro_packets: 0
rx47_lro_bytes: 0
rx47_ecn_mark: 0
rx47_removed_vlan_packets: 0
rx47_wqe_err: 0
rx47_mpwqe_filler_cqes: 0
rx47_mpwqe_filler_strides: 0
rx47_buff_alloc_err: 0
rx47_cqe_compress_blks: 0
rx47_cqe_compress_pkts: 0
rx47_page_reuse: 0
rx47_cache_reuse: 0
rx47_cache_full: 0
rx47_cache_empty: 2560
rx47_cache_busy: 0
rx47_cache_waive: 0
rx47_congst_umr: 0
rx47_arfs_err: 0
rx47_xdp_tx_xmit: 0
rx47_xdp_tx_full: 0
rx47_xdp_tx_err: 0
rx47_xdp_tx_cqes: 0
rx48_packets: 0
rx48_bytes: 0
rx48_csum_complete: 0
rx48_csum_unnecessary: 0
rx48_csum_unnecessary_inner: 0
rx48_csum_none: 0
rx48_xdp_drop: 0
rx48_xdp_redirect: 0
rx48_lro_packets: 0
rx48_lro_bytes: 0
rx48_ecn_mark: 0
rx48_removed_vlan_packets: 0
rx48_wqe_err: 0
rx48_mpwqe_filler_cqes: 0
rx48_mpwqe_filler_strides: 0
rx48_buff_alloc_err: 0
rx48_cqe_compress_blks: 0
rx48_cqe_compress_pkts: 0
rx48_page_reuse: 0
rx48_cache_reuse: 0
rx48_cache_full: 0
rx48_cache_empty: 2560
rx48_cache_busy: 0
rx48_cache_waive: 0
rx48_congst_umr: 0
rx48_arfs_err: 0
rx48_xdp_tx_xmit: 0
rx48_xdp_tx_full: 0
rx48_xdp_tx_err: 0
rx48_xdp_tx_cqes: 0
rx49_packets: 0
rx49_bytes: 0
rx49_csum_complete: 0
rx49_csum_unnecessary: 0
rx49_csum_unnecessary_inner: 0
rx49_csum_none: 0
rx49_xdp_drop: 0
rx49_xdp_redirect: 0
rx49_lro_packets: 0
rx49_lro_bytes: 0
rx49_ecn_mark: 0
rx49_removed_vlan_packets: 0
rx49_wqe_err: 0
rx49_mpwqe_filler_cqes: 0
rx49_mpwqe_filler_strides: 0
rx49_buff_alloc_err: 0
rx49_cqe_compress_blks: 0
rx49_cqe_compress_pkts: 0
rx49_page_reuse: 0
rx49_cache_reuse: 0
rx49_cache_full: 0
rx49_cache_empty: 2560
rx49_cache_busy: 0
rx49_cache_waive: 0
rx49_congst_umr: 0
rx49_arfs_err: 0
rx49_xdp_tx_xmit: 0
rx49_xdp_tx_full: 0
rx49_xdp_tx_err: 0
rx49_xdp_tx_cqes: 0
rx50_packets: 0
rx50_bytes: 0
rx50_csum_complete: 0
rx50_csum_unnecessary: 0
rx50_csum_unnecessary_inner: 0
rx50_csum_none: 0
rx50_xdp_drop: 0
rx50_xdp_redirect: 0
rx50_lro_packets: 0
rx50_lro_bytes: 0
rx50_ecn_mark: 0
rx50_removed_vlan_packets: 0
rx50_wqe_err: 0
rx50_mpwqe_filler_cqes: 0
rx50_mpwqe_filler_strides: 0
rx50_buff_alloc_err: 0
rx50_cqe_compress_blks: 0
rx50_cqe_compress_pkts: 0
rx50_page_reuse: 0
rx50_cache_reuse: 0
rx50_cache_full: 0
rx50_cache_empty: 2560
rx50_cache_busy: 0
rx50_cache_waive: 0
rx50_congst_umr: 0
rx50_arfs_err: 0
rx50_xdp_tx_xmit: 0
rx50_xdp_tx_full: 0
rx50_xdp_tx_err: 0
rx50_xdp_tx_cqes: 0
rx51_packets: 0
rx51_bytes: 0
rx51_csum_complete: 0
rx51_csum_unnecessary: 0
rx51_csum_unnecessary_inner: 0
rx51_csum_none: 0
rx51_xdp_drop: 0
rx51_xdp_redirect: 0
rx51_lro_packets: 0
rx51_lro_bytes: 0
rx51_ecn_mark: 0
rx51_removed_vlan_packets: 0
rx51_wqe_err: 0
rx51_mpwqe_filler_cqes: 0
rx51_mpwqe_filler_strides: 0
rx51_buff_alloc_err: 0
rx51_cqe_compress_blks: 0
rx51_cqe_compress_pkts: 0
rx51_page_reuse: 0
rx51_cache_reuse: 0
rx51_cache_full: 0
rx51_cache_empty: 2560
rx51_cache_busy: 0
rx51_cache_waive: 0
rx51_congst_umr: 0
rx51_arfs_err: 0
rx51_xdp_tx_xmit: 0
rx51_xdp_tx_full: 0
rx51_xdp_tx_err: 0
rx51_xdp_tx_cqes: 0
rx52_packets: 0
rx52_bytes: 0
rx52_csum_complete: 0
rx52_csum_unnecessary: 0
rx52_csum_unnecessary_inner: 0
rx52_csum_none: 0
rx52_xdp_drop: 0
rx52_xdp_redirect: 0
rx52_lro_packets: 0
rx52_lro_bytes: 0
rx52_ecn_mark: 0
rx52_removed_vlan_packets: 0
rx52_wqe_err: 0
rx52_mpwqe_filler_cqes: 0
rx52_mpwqe_filler_strides: 0
rx52_buff_alloc_err: 0
rx52_cqe_compress_blks: 0
rx52_cqe_compress_pkts: 0
rx52_page_reuse: 0
rx52_cache_reuse: 0
rx52_cache_full: 0
rx52_cache_empty: 2560
rx52_cache_busy: 0
rx52_cache_waive: 0
rx52_congst_umr: 0
rx52_arfs_err: 0
rx52_xdp_tx_xmit: 0
rx52_xdp_tx_full: 0
rx52_xdp_tx_err: 0
rx52_xdp_tx_cqes: 0
rx53_packets: 0
rx53_bytes: 0
rx53_csum_complete: 0
rx53_csum_unnecessary: 0
rx53_csum_unnecessary_inner: 0
rx53_csum_none: 0
rx53_xdp_drop: 0
rx53_xdp_redirect: 0
rx53_lro_packets: 0
rx53_lro_bytes: 0
rx53_ecn_mark: 0
rx53_removed_vlan_packets: 0
rx53_wqe_err: 0
rx53_mpwqe_filler_cqes: 0
rx53_mpwqe_filler_strides: 0
rx53_buff_alloc_err: 0
rx53_cqe_compress_blks: 0
rx53_cqe_compress_pkts: 0
rx53_page_reuse: 0
rx53_cache_reuse: 0
rx53_cache_full: 0
rx53_cache_empty: 2560
rx53_cache_busy: 0
rx53_cache_waive: 0
rx53_congst_umr: 0
rx53_arfs_err: 0
rx53_xdp_tx_xmit: 0
rx53_xdp_tx_full: 0
rx53_xdp_tx_err: 0
rx53_xdp_tx_cqes: 0
rx54_packets: 0
rx54_bytes: 0
rx54_csum_complete: 0
rx54_csum_unnecessary: 0
rx54_csum_unnecessary_inner: 0
rx54_csum_none: 0
rx54_xdp_drop: 0
rx54_xdp_redirect: 0
rx54_lro_packets: 0
rx54_lro_bytes: 0
rx54_ecn_mark: 0
rx54_removed_vlan_packets: 0
rx54_wqe_err: 0
rx54_mpwqe_filler_cqes: 0
rx54_mpwqe_filler_strides: 0
rx54_buff_alloc_err: 0
rx54_cqe_compress_blks: 0
rx54_cqe_compress_pkts: 0
rx54_page_reuse: 0
rx54_cache_reuse: 0
rx54_cache_full: 0
rx54_cache_empty: 2560
rx54_cache_busy: 0
rx54_cache_waive: 0
rx54_congst_umr: 0
rx54_arfs_err: 0
rx54_xdp_tx_xmit: 0
rx54_xdp_tx_full: 0
rx54_xdp_tx_err: 0
rx54_xdp_tx_cqes: 0
rx55_packets: 0
rx55_bytes: 0
rx55_csum_complete: 0
rx55_csum_unnecessary: 0
rx55_csum_unnecessary_inner: 0
rx55_csum_none: 0
rx55_xdp_drop: 0
rx55_xdp_redirect: 0
rx55_lro_packets: 0
rx55_lro_bytes: 0
rx55_ecn_mark: 0
rx55_removed_vlan_packets: 0
rx55_wqe_err: 0
rx55_mpwqe_filler_cqes: 0
rx55_mpwqe_filler_strides: 0
rx55_buff_alloc_err: 0
rx55_cqe_compress_blks: 0
rx55_cqe_compress_pkts: 0
rx55_page_reuse: 0
rx55_cache_reuse: 0
rx55_cache_full: 0
rx55_cache_empty: 2560
rx55_cache_busy: 0
rx55_cache_waive: 0
rx55_congst_umr: 0
rx55_arfs_err: 0
rx55_xdp_tx_xmit: 0
rx55_xdp_tx_full: 0
rx55_xdp_tx_err: 0
rx55_xdp_tx_cqes: 0
tx0_packets: 24512439668
tx0_bytes: 15287569052791
tx0_tso_packets: 1536157106
tx0_tso_bytes: 8571753637944
tx0_tso_inner_packets: 0
tx0_tso_inner_bytes: 0
tx0_csum_partial: 2132156117
tx0_csum_partial_inner: 0
tx0_added_vlan_packets: 19906601448
tx0_nop: 308098536
tx0_csum_none: 17774445331
tx0_stopped: 19625
tx0_dropped: 0
tx0_xmit_more: 67864870
tx0_recover: 0
tx0_cqes: 19838744246
tx0_wake: 19624
tx0_cqe_err: 0
tx1_packets: 22598557053
tx1_bytes: 13568850145010
tx1_tso_packets: 1369529475
tx1_tso_bytes: 7661777265382
tx1_tso_inner_packets: 0
tx1_tso_inner_bytes: 0
tx1_csum_partial: 1884639496
tx1_csum_partial_inner: 0
tx1_added_vlan_packets: 18468333696
tx1_nop: 281301783
tx1_csum_none: 16583694200
tx1_stopped: 19457
tx1_dropped: 0
tx1_xmit_more: 55170875
tx1_recover: 0
tx1_cqes: 18413169824
tx1_wake: 19455
tx1_cqe_err: 0
tx2_packets: 22821611433
tx2_bytes: 13752535163683
tx2_tso_packets: 1396978825
tx2_tso_bytes: 7774704508463
tx2_tso_inner_packets: 0
tx2_tso_inner_bytes: 0
tx2_csum_partial: 1897834558
tx2_csum_partial_inner: 0
tx2_added_vlan_packets: 18641958085
tx2_nop: 286934891
tx2_csum_none: 16744123527
tx2_stopped: 13214
tx2_dropped: 0
tx2_xmit_more: 61749446
tx2_recover: 0
tx2_cqes: 18580215654
tx2_wake: 13214
tx2_cqe_err: 0
tx3_packets: 22580809948
tx3_bytes: 13730542936609
tx3_tso_packets: 1370434579
tx3_tso_bytes: 7605636711455
tx3_tso_inner_packets: 0
tx3_tso_inner_bytes: 0
tx3_csum_partial: 1865573748
tx3_csum_partial_inner: 0
tx3_added_vlan_packets: 18491873644
tx3_nop: 281195875
tx3_csum_none: 16626299896
tx3_stopped: 12542
tx3_dropped: 0
tx3_xmit_more: 57681647
tx3_recover: 0
tx3_cqes: 18434198757
tx3_wake: 12540
tx3_cqe_err: 0
tx4_packets: 27801801208
tx4_bytes: 17058453171137
tx4_tso_packets: 1740500105
tx4_tso_bytes: 9474905622036
tx4_tso_inner_packets: 0
tx4_tso_inner_bytes: 0
tx4_csum_partial: 2279225376
tx4_csum_partial_inner: 0
tx4_added_vlan_packets: 22744081633
tx4_nop: 349753979
tx4_csum_none: 20464856257
tx4_stopped: 14816
tx4_dropped: 0
tx4_xmit_more: 65469322
tx4_recover: 0
tx4_cqes: 22678618972
tx4_wake: 14816
tx4_cqe_err: 0
tx5_packets: 25099783024
tx5_bytes: 14917740698381
tx5_tso_packets: 1512988013
tx5_tso_bytes: 8571208921023
tx5_tso_inner_packets: 0
tx5_tso_inner_bytes: 0
tx5_csum_partial: 2078498561
tx5_csum_partial_inner: 0
tx5_added_vlan_packets: 20465533760
tx5_nop: 312614719
tx5_csum_none: 18387035199
tx5_stopped: 4605
tx5_dropped: 0
tx5_xmit_more: 64188936
tx5_recover: 0
tx5_cqes: 20401350718
tx5_wake: 4604
tx5_cqe_err: 0
tx6_packets: 25025504896
tx6_bytes: 14908021946070
tx6_tso_packets: 1515718977
tx6_tso_bytes: 8511442522461
tx6_tso_inner_packets: 0
tx6_tso_inner_bytes: 0
tx6_csum_partial: 2056378610
tx6_csum_partial_inner: 0
tx6_added_vlan_packets: 20434066400
tx6_nop: 310594020
tx6_csum_none: 18377687790
tx6_stopped: 15234
tx6_dropped: 0
tx6_xmit_more: 61130422
tx6_recover: 0
tx6_cqes: 20372943611
tx6_wake: 15234
tx6_cqe_err: 0
tx7_packets: 25457096169
tx7_bytes: 15456289446172
tx7_tso_packets: 1553342799
tx7_tso_bytes: 8764550988105
tx7_tso_inner_packets: 0
tx7_tso_inner_bytes: 0
tx7_csum_partial: 2105765233
tx7_csum_partial_inner: 0
tx7_added_vlan_packets: 20720382377
tx7_nop: 319044853
tx7_csum_none: 18614617145
tx7_stopped: 18745
tx7_dropped: 0
tx7_xmit_more: 57050107
tx7_recover: 0
tx7_cqes: 20663340775
tx7_wake: 18746
tx7_cqe_err: 0
tx8_packets: 25389771649
tx8_bytes: 15225503883962
tx8_tso_packets: 1563367648
tx8_tso_bytes: 8710384514258
tx8_tso_inner_packets: 0
tx8_tso_inner_bytes: 0
tx8_csum_partial: 2106586634
tx8_csum_partial_inner: 0
tx8_added_vlan_packets: 20704676274
tx8_nop: 318149261
tx8_csum_none: 18598089640
tx8_stopped: 4733
tx8_dropped: 0
tx8_xmit_more: 61014317
tx8_recover: 0
tx8_cqes: 20643667301
tx8_wake: 4735
tx8_cqe_err: 0
tx9_packets: 25521500166
tx9_bytes: 15302293145755
tx9_tso_packets: 1546316697
tx9_tso_bytes: 8770688145926
tx9_tso_inner_packets: 0
tx9_tso_inner_bytes: 0
tx9_csum_partial: 2097652880
tx9_csum_partial_inner: 0
tx9_added_vlan_packets: 20778408432
tx9_nop: 318538543
tx9_csum_none: 18680755556
tx9_stopped: 16118
tx9_dropped: 0
tx9_xmit_more: 68509728
tx9_recover: 0
tx9_cqes: 20709906498
tx9_wake: 16118
tx9_cqe_err: 0
tx10_packets: 25451605829
tx10_bytes: 15386896170792
tx10_tso_packets: 1576473520
tx10_tso_bytes: 8880888676383
tx10_tso_inner_packets: 0
tx10_tso_inner_bytes: 0
tx10_csum_partial: 2129796141
tx10_csum_partial_inner: 0
tx10_added_vlan_packets: 20659622590
tx10_nop: 319117433
tx10_csum_none: 18529826450
tx10_stopped: 20187
tx10_dropped: 0
tx10_xmit_more: 58892184
tx10_recover: 0
tx10_cqes: 20600737739
tx10_wake: 20188
tx10_cqe_err: 0
tx11_packets: 27008919793
tx11_bytes: 16587719213058
tx11_tso_packets: 1734884654
tx11_tso_bytes: 9475681471870
tx11_tso_inner_packets: 0
tx11_tso_inner_bytes: 0
tx11_csum_partial: 2296162292
tx11_csum_partial_inner: 0
tx11_added_vlan_packets: 21943096263
tx11_nop: 344188182
tx11_csum_none: 19646933971
tx11_stopped: 9703
tx11_dropped: 0
tx11_xmit_more: 66530718
tx11_recover: 0
tx11_cqes: 21876571667
tx11_wake: 9704
tx11_cqe_err: 0
tx12_packets: 25969493269
tx12_bytes: 15980767963416
tx12_tso_packets: 1671396456
tx12_tso_bytes: 9268973672821
tx12_tso_inner_packets: 0
tx12_tso_inner_bytes: 0
tx12_csum_partial: 2243809182
tx12_csum_partial_inner: 0
tx12_added_vlan_packets: 20980642456
tx12_nop: 330241007
tx12_csum_none: 18736833276
tx12_stopped: 10341
tx12_dropped: 0
tx12_xmit_more: 57834100
tx12_recover: 0
tx12_cqes: 20922815079
tx12_wake: 10342
tx12_cqe_err: 0
tx13_packets: 25332762261
tx13_bytes: 15353213283280
tx13_tso_packets: 1577433599
tx13_tso_bytes: 8785240284281
tx13_tso_inner_packets: 0
tx13_tso_inner_bytes: 0
tx13_csum_partial: 2110640515
tx13_csum_partial_inner: 0
tx13_added_vlan_packets: 20605670910
tx13_nop: 319805741
tx13_csum_none: 18495030395
tx13_stopped: 7006
tx13_dropped: 0
tx13_xmit_more: 58314402
tx13_recover: 0
tx13_cqes: 20547362770
tx13_wake: 7008
tx13_cqe_err: 0
tx14_packets: 26333743548
tx14_bytes: 16070719060573
tx14_tso_packets: 1677922970
tx14_tso_bytes: 9240299765487
tx14_tso_inner_packets: 0
tx14_tso_inner_bytes: 0
tx14_csum_partial: 2215668906
tx14_csum_partial_inner: 0
tx14_added_vlan_packets: 21384410786
tx14_nop: 332734939
tx14_csum_none: 19168741880
tx14_stopped: 13160
tx14_dropped: 0
tx14_xmit_more: 57650391
tx14_recover: 0
tx14_cqes: 21326767783
tx14_wake: 13161
tx14_cqe_err: 0
tx15_packets: 26824968971
tx15_bytes: 16687994233452
tx15_tso_packets: 1755745052
tx15_tso_bytes: 9533814012441
tx15_tso_inner_packets: 0
tx15_tso_inner_bytes: 0
tx15_csum_partial: 2304778064
tx15_csum_partial_inner: 0
tx15_added_vlan_packets: 21740906107
tx15_nop: 344143287
tx15_csum_none: 19436128058
tx15_stopped: 75
tx15_dropped: 0
tx15_xmit_more: 63325832
tx15_recover: 0
tx15_cqes: 21677585345
tx15_wake: 74
tx15_cqe_err: 0
tx16_packets: 24488158946
tx16_bytes: 15027415004570
tx16_tso_packets: 1559127391
tx16_tso_bytes: 8658691917845
tx16_tso_inner_packets: 0
tx16_tso_inner_bytes: 0
tx16_csum_partial: 2075856395
tx16_csum_partial_inner: 0
tx16_added_vlan_packets: 19835695731
tx16_nop: 308464189
tx16_csum_none: 17759839340
tx16_stopped: 4567
tx16_dropped: 0
tx16_xmit_more: 62631422
tx16_recover: 0
tx16_cqes: 19773070012
tx16_wake: 4568
tx16_cqe_err: 0
tx17_packets: 24700413784
tx17_bytes: 15216529713715
tx17_tso_packets: 1597555108
tx17_tso_bytes: 8773728661243
tx17_tso_inner_packets: 0
tx17_tso_inner_bytes: 0
tx17_csum_partial: 2127177297
tx17_csum_partial_inner: 0
tx17_added_vlan_packets: 20003144561
tx17_nop: 313356918
tx17_csum_none: 17875967264
tx17_stopped: 12572
tx17_dropped: 0
tx17_xmit_more: 62742980
tx17_recover: 0
tx17_cqes: 19940407615
tx17_wake: 12573
tx17_cqe_err: 0
tx18_packets: 24887710046
tx18_bytes: 15245034034664
tx18_tso_packets: 1582550520
tx18_tso_bytes: 8782692335483
tx18_tso_inner_packets: 0
tx18_tso_inner_bytes: 0
tx18_csum_partial: 2084514331
tx18_csum_partial_inner: 0
tx18_added_vlan_packets: 20173879181
tx18_nop: 314818702
tx18_csum_none: 18089364850
tx18_stopped: 21366
tx18_dropped: 0
tx18_xmit_more: 62485819
tx18_recover: 0
tx18_cqes: 20111400935
tx18_wake: 21366
tx18_cqe_err: 0
tx19_packets: 24831057648
tx19_bytes: 15164663890576
tx19_tso_packets: 1599135489
tx19_tso_bytes: 8756045449746
tx19_tso_inner_packets: 0
tx19_tso_inner_bytes: 0
tx19_csum_partial: 2119746608
tx19_csum_partial_inner: 0
tx19_added_vlan_packets: 20143573903
tx19_nop: 316966450
tx19_csum_none: 18023827295
tx19_stopped: 11431
tx19_dropped: 0
tx19_xmit_more: 57535904
tx19_recover: 0
tx19_cqes: 20086045325
tx19_wake: 11431
tx19_cqe_err: 0
tx20_packets: 21943735263
tx20_bytes: 13528749492187
tx20_tso_packets: 1390048103
tx20_tso_bytes: 7629058809637
tx20_tso_inner_packets: 0
tx20_tso_inner_bytes: 0
tx20_csum_partial: 1848533941
tx20_csum_partial_inner: 0
tx20_added_vlan_packets: 17861417651
tx20_nop: 276840365
tx20_csum_none: 16012883710
tx20_stopped: 38457
tx20_dropped: 0
tx20_xmit_more: 57042753
tx20_recover: 0
tx20_cqes: 17804384839
tx20_wake: 38457
tx20_cqe_err: 0
tx21_packets: 21476926958
tx21_bytes: 13096410597896
tx21_tso_packets: 1367724090
tx21_tso_bytes: 7568364585127
tx21_tso_inner_packets: 0
tx21_tso_inner_bytes: 0
tx21_csum_partial: 1830570727
tx21_csum_partial_inner: 0
tx21_added_vlan_packets: 17421087814
tx21_nop: 270611519
tx21_csum_none: 15590517087
tx21_stopped: 31213
tx21_dropped: 0
tx21_xmit_more: 60305389
tx21_recover: 0
tx21_cqes: 17360791205
tx21_wake: 31213
tx21_cqe_err: 0
tx22_packets: 21819106444
tx22_bytes: 13492871887100
tx22_tso_packets: 1387002018
tx22_tso_bytes: 7617705727669
tx22_tso_inner_packets: 0
tx22_tso_inner_bytes: 0
tx22_csum_partial: 1853632107
tx22_csum_partial_inner: 0
tx22_added_vlan_packets: 17743255447
tx22_nop: 274820992
tx22_csum_none: 15889623340
tx22_stopped: 24814
tx22_dropped: 0
tx22_xmit_more: 60811304
tx22_recover: 0
tx22_cqes: 17682451111
tx22_wake: 24815
tx22_cqe_err: 0
tx23_packets: 21830455800
tx23_bytes: 13427551902532
tx23_tso_packets: 1388556038
tx23_tso_bytes: 7604040587125
tx23_tso_inner_packets: 0
tx23_tso_inner_bytes: 0
tx23_csum_partial: 1850819694
tx23_csum_partial_inner: 0
tx23_added_vlan_packets: 17761271122
tx23_nop: 275142775
tx23_csum_none: 15910451428
tx23_stopped: 29899
tx23_dropped: 0
tx23_xmit_more: 58924909
tx23_recover: 0
tx23_cqes: 17702355187
tx23_wake: 29898
tx23_cqe_err: 0
tx24_packets: 21961484213
tx24_bytes: 13531373062497
tx24_tso_packets: 1394697504
tx24_tso_bytes: 7663866609308
tx24_tso_inner_packets: 0
tx24_tso_inner_bytes: 0
tx24_csum_partial: 1857072074
tx24_csum_partial_inner: 0
tx24_added_vlan_packets: 17856887568
tx24_nop: 276352855
tx24_csum_none: 15999815494
tx24_stopped: 33924
tx24_dropped: 0
tx24_xmit_more: 63992426
tx24_recover: 0
tx24_cqes: 17792905243
tx24_wake: 33923
tx24_cqe_err: 0
tx25_packets: 21853593838
tx25_bytes: 13357487830519
tx25_tso_packets: 1398822411
tx25_tso_bytes: 7691191518838
tx25_tso_inner_packets: 0
tx25_tso_inner_bytes: 0
tx25_csum_partial: 1869483109
tx25_csum_partial_inner: 0
tx25_added_vlan_packets: 17734634614
tx25_nop: 276327643
tx25_csum_none: 15865151505
tx25_stopped: 38651
tx25_dropped: 0
tx25_xmit_more: 56410535
tx25_recover: 0
tx25_cqes: 17678234537
tx25_wake: 38650
tx25_cqe_err: 0
tx26_packets: 21480261205
tx26_bytes: 13148973015935
tx26_tso_packets: 1348132284
tx26_tso_bytes: 7523489481775
tx26_tso_inner_packets: 0
tx26_tso_inner_bytes: 0
tx26_csum_partial: 1839740745
tx26_csum_partial_inner: 0
tx26_added_vlan_packets: 17430592911
tx26_nop: 270367836
tx26_csum_none: 15590852166
tx26_stopped: 34044
tx26_dropped: 0
tx26_xmit_more: 59870114
tx26_recover: 0
tx26_cqes: 17370736612
tx26_wake: 34043
tx26_cqe_err: 0
tx27_packets: 22694273108
tx27_bytes: 14135473431004
tx27_tso_packets: 1418371875
tx27_tso_bytes: 7784842263038
tx27_tso_inner_packets: 0
tx27_tso_inner_bytes: 0
tx27_csum_partial: 1919170584
tx27_csum_partial_inner: 0
tx27_added_vlan_packets: 18520826023
tx27_nop: 286296272
tx27_csum_none: 16601655439
tx27_stopped: 38125
tx27_dropped: 0
tx27_xmit_more: 72749775
tx27_recover: 0
tx27_cqes: 18448090270
tx27_wake: 38127
tx27_cqe_err: 0
tx28_packets: 0
tx28_bytes: 0
tx28_tso_packets: 0
tx28_tso_bytes: 0
tx28_tso_inner_packets: 0
tx28_tso_inner_bytes: 0
tx28_csum_partial: 0
tx28_csum_partial_inner: 0
tx28_added_vlan_packets: 0
tx28_nop: 0
tx28_csum_none: 0
tx28_stopped: 0
tx28_dropped: 0
tx28_xmit_more: 0
tx28_recover: 0
tx28_cqes: 0
tx28_wake: 0
tx28_cqe_err: 0
tx29_packets: 3
tx29_bytes: 266
tx29_tso_packets: 0
tx29_tso_bytes: 0
tx29_tso_inner_packets: 0
tx29_tso_inner_bytes: 0
tx29_csum_partial: 0
tx29_csum_partial_inner: 0
tx29_added_vlan_packets: 0
tx29_nop: 0
tx29_csum_none: 3
tx29_stopped: 0
tx29_dropped: 0
tx29_xmit_more: 1
tx29_recover: 0
tx29_cqes: 2
tx29_wake: 0
tx29_cqe_err: 0
tx30_packets: 0
tx30_bytes: 0
tx30_tso_packets: 0
tx30_tso_bytes: 0
tx30_tso_inner_packets: 0
tx30_tso_inner_bytes: 0
tx30_csum_partial: 0
tx30_csum_partial_inner: 0
tx30_added_vlan_packets: 0
tx30_nop: 0
tx30_csum_none: 0
tx30_stopped: 0
tx30_dropped: 0
tx30_xmit_more: 0
tx30_recover: 0
tx30_cqes: 0
tx30_wake: 0
tx30_cqe_err: 0
tx31_packets: 0
tx31_bytes: 0
tx31_tso_packets: 0
tx31_tso_bytes: 0
tx31_tso_inner_packets: 0
tx31_tso_inner_bytes: 0
tx31_csum_partial: 0
tx31_csum_partial_inner: 0
tx31_added_vlan_packets: 0
tx31_nop: 0
tx31_csum_none: 0
tx31_stopped: 0
tx31_dropped: 0
tx31_xmit_more: 0
tx31_recover: 0
tx31_cqes: 0
tx31_wake: 0
tx31_cqe_err: 0
tx32_packets: 0
tx32_bytes: 0
tx32_tso_packets: 0
tx32_tso_bytes: 0
tx32_tso_inner_packets: 0
tx32_tso_inner_bytes: 0
tx32_csum_partial: 0
tx32_csum_partial_inner: 0
tx32_added_vlan_packets: 0
tx32_nop: 0
tx32_csum_none: 0
tx32_stopped: 0
tx32_dropped: 0
tx32_xmit_more: 0
tx32_recover: 0
tx32_cqes: 0
tx32_wake: 0
tx32_cqe_err: 0
tx33_packets: 0
tx33_bytes: 0
tx33_tso_packets: 0
tx33_tso_bytes: 0
tx33_tso_inner_packets: 0
tx33_tso_inner_bytes: 0
tx33_csum_partial: 0
tx33_csum_partial_inner: 0
tx33_added_vlan_packets: 0
tx33_nop: 0
tx33_csum_none: 0
tx33_stopped: 0
tx33_dropped: 0
tx33_xmit_more: 0
tx33_recover: 0
tx33_cqes: 0
tx33_wake: 0
tx33_cqe_err: 0
tx34_packets: 0
tx34_bytes: 0
tx34_tso_packets: 0
tx34_tso_bytes: 0
tx34_tso_inner_packets: 0
tx34_tso_inner_bytes: 0
tx34_csum_partial: 0
tx34_csum_partial_inner: 0
tx34_added_vlan_packets: 0
tx34_nop: 0
tx34_csum_none: 0
tx34_stopped: 0
tx34_dropped: 0
tx34_xmit_more: 0
tx34_recover: 0
tx34_cqes: 0
tx34_wake: 0
tx34_cqe_err: 0
tx35_packets: 0
tx35_bytes: 0
tx35_tso_packets: 0
tx35_tso_bytes: 0
tx35_tso_inner_packets: 0
tx35_tso_inner_bytes: 0
tx35_csum_partial: 0
tx35_csum_partial_inner: 0
tx35_added_vlan_packets: 0
tx35_nop: 0
tx35_csum_none: 0
tx35_stopped: 0
tx35_dropped: 0
tx35_xmit_more: 0
tx35_recover: 0
tx35_cqes: 0
tx35_wake: 0
tx35_cqe_err: 0
tx36_packets: 0
tx36_bytes: 0
tx36_tso_packets: 0
tx36_tso_bytes: 0
tx36_tso_inner_packets: 0
tx36_tso_inner_bytes: 0
tx36_csum_partial: 0
tx36_csum_partial_inner: 0
tx36_added_vlan_packets: 0
tx36_nop: 0
tx36_csum_none: 0
tx36_stopped: 0
tx36_dropped: 0
tx36_xmit_more: 0
tx36_recover: 0
tx36_cqes: 0
tx36_wake: 0
tx36_cqe_err: 0
tx37_packets: 0
tx37_bytes: 0
tx37_tso_packets: 0
tx37_tso_bytes: 0
tx37_tso_inner_packets: 0
tx37_tso_inner_bytes: 0
tx37_csum_partial: 0
tx37_csum_partial_inner: 0
tx37_added_vlan_packets: 0
tx37_nop: 0
tx37_csum_none: 0
tx37_stopped: 0
tx37_dropped: 0
tx37_xmit_more: 0
tx37_recover: 0
tx37_cqes: 0
tx37_wake: 0
tx37_cqe_err: 0
tx38_packets: 0
tx38_bytes: 0
tx38_tso_packets: 0
tx38_tso_bytes: 0
tx38_tso_inner_packets: 0
tx38_tso_inner_bytes: 0
tx38_csum_partial: 0
tx38_csum_partial_inner: 0
tx38_added_vlan_packets: 0
tx38_nop: 0
tx38_csum_none: 0
tx38_stopped: 0
tx38_dropped: 0
tx38_xmit_more: 0
tx38_recover: 0
tx38_cqes: 0
tx38_wake: 0
tx38_cqe_err: 0
tx39_packets: 0
tx39_bytes: 0
tx39_tso_packets: 0
tx39_tso_bytes: 0
tx39_tso_inner_packets: 0
tx39_tso_inner_bytes: 0
tx39_csum_partial: 0
tx39_csum_partial_inner: 0
tx39_added_vlan_packets: 0
tx39_nop: 0
tx39_csum_none: 0
tx39_stopped: 0
tx39_dropped: 0
tx39_xmit_more: 0
tx39_recover: 0
tx39_cqes: 0
tx39_wake: 0
tx39_cqe_err: 0
tx40_packets: 0
tx40_bytes: 0
tx40_tso_packets: 0
tx40_tso_bytes: 0
tx40_tso_inner_packets: 0
tx40_tso_inner_bytes: 0
tx40_csum_partial: 0
tx40_csum_partial_inner: 0
tx40_added_vlan_packets: 0
tx40_nop: 0
tx40_csum_none: 0
tx40_stopped: 0
tx40_dropped: 0
tx40_xmit_more: 0
tx40_recover: 0
tx40_cqes: 0
tx40_wake: 0
tx40_cqe_err: 0
tx41_packets: 0
tx41_bytes: 0
tx41_tso_packets: 0
tx41_tso_bytes: 0
tx41_tso_inner_packets: 0
tx41_tso_inner_bytes: 0
tx41_csum_partial: 0
tx41_csum_partial_inner: 0
tx41_added_vlan_packets: 0
tx41_nop: 0
tx41_csum_none: 0
tx41_stopped: 0
tx41_dropped: 0
tx41_xmit_more: 0
tx41_recover: 0
tx41_cqes: 0
tx41_wake: 0
tx41_cqe_err: 0
tx42_packets: 0
tx42_bytes: 0
tx42_tso_packets: 0
tx42_tso_bytes: 0
tx42_tso_inner_packets: 0
tx42_tso_inner_bytes: 0
tx42_csum_partial: 0
tx42_csum_partial_inner: 0
tx42_added_vlan_packets: 0
tx42_nop: 0
tx42_csum_none: 0
tx42_stopped: 0
tx42_dropped: 0
tx42_xmit_more: 0
tx42_recover: 0
tx42_cqes: 0
tx42_wake: 0
tx42_cqe_err: 0
tx43_packets: 0
tx43_bytes: 0
tx43_tso_packets: 0
tx43_tso_bytes: 0
tx43_tso_inner_packets: 0
tx43_tso_inner_bytes: 0
tx43_csum_partial: 0
tx43_csum_partial_inner: 0
tx43_added_vlan_packets: 0
tx43_nop: 0
tx43_csum_none: 0
tx43_stopped: 0
tx43_dropped: 0
tx43_xmit_more: 0
tx43_recover: 0
tx43_cqes: 0
tx43_wake: 0
tx43_cqe_err: 0
tx44_packets: 0
tx44_bytes: 0
tx44_tso_packets: 0
tx44_tso_bytes: 0
tx44_tso_inner_packets: 0
tx44_tso_inner_bytes: 0
tx44_csum_partial: 0
tx44_csum_partial_inner: 0
tx44_added_vlan_packets: 0
tx44_nop: 0
tx44_csum_none: 0
tx44_stopped: 0
tx44_dropped: 0
tx44_xmit_more: 0
tx44_recover: 0
tx44_cqes: 0
tx44_wake: 0
tx44_cqe_err: 0
tx45_packets: 0
tx45_bytes: 0
tx45_tso_packets: 0
tx45_tso_bytes: 0
tx45_tso_inner_packets: 0
tx45_tso_inner_bytes: 0
tx45_csum_partial: 0
tx45_csum_partial_inner: 0
tx45_added_vlan_packets: 0
tx45_nop: 0
tx45_csum_none: 0
tx45_stopped: 0
tx45_dropped: 0
tx45_xmit_more: 0
tx45_recover: 0
tx45_cqes: 0
tx45_wake: 0
tx45_cqe_err: 0
tx46_packets: 0
tx46_bytes: 0
tx46_tso_packets: 0
tx46_tso_bytes: 0
tx46_tso_inner_packets: 0
tx46_tso_inner_bytes: 0
tx46_csum_partial: 0
tx46_csum_partial_inner: 0
tx46_added_vlan_packets: 0
tx46_nop: 0
tx46_csum_none: 0
tx46_stopped: 0
tx46_dropped: 0
tx46_xmit_more: 0
tx46_recover: 0
tx46_cqes: 0
tx46_wake: 0
tx46_cqe_err: 0
tx47_packets: 0
tx47_bytes: 0
tx47_tso_packets: 0
tx47_tso_bytes: 0
tx47_tso_inner_packets: 0
tx47_tso_inner_bytes: 0
tx47_csum_partial: 0
tx47_csum_partial_inner: 0
tx47_added_vlan_packets: 0
tx47_nop: 0
tx47_csum_none: 0
tx47_stopped: 0
tx47_dropped: 0
tx47_xmit_more: 0
tx47_recover: 0
tx47_cqes: 0
tx47_wake: 0
tx47_cqe_err: 0
tx48_packets: 0
tx48_bytes: 0
tx48_tso_packets: 0
tx48_tso_bytes: 0
tx48_tso_inner_packets: 0
tx48_tso_inner_bytes: 0
tx48_csum_partial: 0
tx48_csum_partial_inner: 0
tx48_added_vlan_packets: 0
tx48_nop: 0
tx48_csum_none: 0
tx48_stopped: 0
tx48_dropped: 0
tx48_xmit_more: 0
tx48_recover: 0
tx48_cqes: 0
tx48_wake: 0
tx48_cqe_err: 0
tx49_packets: 0
tx49_bytes: 0
tx49_tso_packets: 0
tx49_tso_bytes: 0
tx49_tso_inner_packets: 0
tx49_tso_inner_bytes: 0
tx49_csum_partial: 0
tx49_csum_partial_inner: 0
tx49_added_vlan_packets: 0
tx49_nop: 0
tx49_csum_none: 0
tx49_stopped: 0
tx49_dropped: 0
tx49_xmit_more: 0
tx49_recover: 0
tx49_cqes: 0
tx49_wake: 0
tx49_cqe_err: 0
tx50_packets: 0
tx50_bytes: 0
tx50_tso_packets: 0
tx50_tso_bytes: 0
tx50_tso_inner_packets: 0
tx50_tso_inner_bytes: 0
tx50_csum_partial: 0
tx50_csum_partial_inner: 0
tx50_added_vlan_packets: 0
tx50_nop: 0
tx50_csum_none: 0
tx50_stopped: 0
tx50_dropped: 0
tx50_xmit_more: 0
tx50_recover: 0
tx50_cqes: 0
tx50_wake: 0
tx50_cqe_err: 0
tx51_packets: 0
tx51_bytes: 0
tx51_tso_packets: 0
tx51_tso_bytes: 0
tx51_tso_inner_packets: 0
tx51_tso_inner_bytes: 0
tx51_csum_partial: 0
tx51_csum_partial_inner: 0
tx51_added_vlan_packets: 0
tx51_nop: 0
tx51_csum_none: 0
tx51_stopped: 0
tx51_dropped: 0
tx51_xmit_more: 0
tx51_recover: 0
tx51_cqes: 0
tx51_wake: 0
tx51_cqe_err: 0
tx52_packets: 0
tx52_bytes: 0
tx52_tso_packets: 0
tx52_tso_bytes: 0
tx52_tso_inner_packets: 0
tx52_tso_inner_bytes: 0
tx52_csum_partial: 0
tx52_csum_partial_inner: 0
tx52_added_vlan_packets: 0
tx52_nop: 0
tx52_csum_none: 0
tx52_stopped: 0
tx52_dropped: 0
tx52_xmit_more: 0
tx52_recover: 0
tx52_cqes: 0
tx52_wake: 0
tx52_cqe_err: 0
tx53_packets: 0
tx53_bytes: 0
tx53_tso_packets: 0
tx53_tso_bytes: 0
tx53_tso_inner_packets: 0
tx53_tso_inner_bytes: 0
tx53_csum_partial: 0
tx53_csum_partial_inner: 0
tx53_added_vlan_packets: 0
tx53_nop: 0
tx53_csum_none: 0
tx53_stopped: 0
tx53_dropped: 0
tx53_xmit_more: 0
tx53_recover: 0
tx53_cqes: 0
tx53_wake: 0
tx53_cqe_err: 0
tx54_packets: 0
tx54_bytes: 0
tx54_tso_packets: 0
tx54_tso_bytes: 0
tx54_tso_inner_packets: 0
tx54_tso_inner_bytes: 0
tx54_csum_partial: 0
tx54_csum_partial_inner: 0
tx54_added_vlan_packets: 0
tx54_nop: 0
tx54_csum_none: 0
tx54_stopped: 0
tx54_dropped: 0
tx54_xmit_more: 0
tx54_recover: 0
tx54_cqes: 0
tx54_wake: 0
tx54_cqe_err: 0
tx55_packets: 0
tx55_bytes: 0
tx55_tso_packets: 0
tx55_tso_bytes: 0
tx55_tso_inner_packets: 0
tx55_tso_inner_bytes: 0
tx55_csum_partial: 0
tx55_csum_partial_inner: 0
tx55_added_vlan_packets: 0
tx55_nop: 0
tx55_csum_none: 0
tx55_stopped: 0
tx55_dropped: 0
tx55_xmit_more: 0
tx55_recover: 0
tx55_cqes: 0
tx55_wake: 0
tx55_cqe_err: 0
tx0_xdp_xmit: 0
tx0_xdp_full: 0
tx0_xdp_err: 0
tx0_xdp_cqes: 0
tx1_xdp_xmit: 0
tx1_xdp_full: 0
tx1_xdp_err: 0
tx1_xdp_cqes: 0
tx2_xdp_xmit: 0
tx2_xdp_full: 0
tx2_xdp_err: 0
tx2_xdp_cqes: 0
tx3_xdp_xmit: 0
tx3_xdp_full: 0
tx3_xdp_err: 0
tx3_xdp_cqes: 0
tx4_xdp_xmit: 0
tx4_xdp_full: 0
tx4_xdp_err: 0
tx4_xdp_cqes: 0
tx5_xdp_xmit: 0
tx5_xdp_full: 0
tx5_xdp_err: 0
tx5_xdp_cqes: 0
tx6_xdp_xmit: 0
tx6_xdp_full: 0
tx6_xdp_err: 0
tx6_xdp_cqes: 0
tx7_xdp_xmit: 0
tx7_xdp_full: 0
tx7_xdp_err: 0
tx7_xdp_cqes: 0
tx8_xdp_xmit: 0
tx8_xdp_full: 0
tx8_xdp_err: 0
tx8_xdp_cqes: 0
tx9_xdp_xmit: 0
tx9_xdp_full: 0
tx9_xdp_err: 0
tx9_xdp_cqes: 0
tx10_xdp_xmit: 0
tx10_xdp_full: 0
tx10_xdp_err: 0
tx10_xdp_cqes: 0
tx11_xdp_xmit: 0
tx11_xdp_full: 0
tx11_xdp_err: 0
tx11_xdp_cqes: 0
tx12_xdp_xmit: 0
tx12_xdp_full: 0
tx12_xdp_err: 0
tx12_xdp_cqes: 0
tx13_xdp_xmit: 0
tx13_xdp_full: 0
tx13_xdp_err: 0
tx13_xdp_cqes: 0
tx14_xdp_xmit: 0
tx14_xdp_full: 0
tx14_xdp_err: 0
tx14_xdp_cqes: 0
tx15_xdp_xmit: 0
tx15_xdp_full: 0
tx15_xdp_err: 0
tx15_xdp_cqes: 0
tx16_xdp_xmit: 0
tx16_xdp_full: 0
tx16_xdp_err: 0
tx16_xdp_cqes: 0
tx17_xdp_xmit: 0
tx17_xdp_full: 0
tx17_xdp_err: 0
tx17_xdp_cqes: 0
tx18_xdp_xmit: 0
tx18_xdp_full: 0
tx18_xdp_err: 0
tx18_xdp_cqes: 0
tx19_xdp_xmit: 0
tx19_xdp_full: 0
tx19_xdp_err: 0
tx19_xdp_cqes: 0
tx20_xdp_xmit: 0
tx20_xdp_full: 0
tx20_xdp_err: 0
tx20_xdp_cqes: 0
tx21_xdp_xmit: 0
tx21_xdp_full: 0
tx21_xdp_err: 0
tx21_xdp_cqes: 0
tx22_xdp_xmit: 0
tx22_xdp_full: 0
tx22_xdp_err: 0
tx22_xdp_cqes: 0
tx23_xdp_xmit: 0
tx23_xdp_full: 0
tx23_xdp_err: 0
tx23_xdp_cqes: 0
tx24_xdp_xmit: 0
tx24_xdp_full: 0
tx24_xdp_err: 0
tx24_xdp_cqes: 0
tx25_xdp_xmit: 0
tx25_xdp_full: 0
tx25_xdp_err: 0
tx25_xdp_cqes: 0
tx26_xdp_xmit: 0
tx26_xdp_full: 0
tx26_xdp_err: 0
tx26_xdp_cqes: 0
tx27_xdp_xmit: 0
tx27_xdp_full: 0
tx27_xdp_err: 0
tx27_xdp_cqes: 0
tx28_xdp_xmit: 0
tx28_xdp_full: 0
tx28_xdp_err: 0
tx28_xdp_cqes: 0
tx29_xdp_xmit: 0
tx29_xdp_full: 0
tx29_xdp_err: 0
tx29_xdp_cqes: 0
tx30_xdp_xmit: 0
tx30_xdp_full: 0
tx30_xdp_err: 0
tx30_xdp_cqes: 0
tx31_xdp_xmit: 0
tx31_xdp_full: 0
tx31_xdp_err: 0
tx31_xdp_cqes: 0
tx32_xdp_xmit: 0
tx32_xdp_full: 0
tx32_xdp_err: 0
tx32_xdp_cqes: 0
tx33_xdp_xmit: 0
tx33_xdp_full: 0
tx33_xdp_err: 0
tx33_xdp_cqes: 0
tx34_xdp_xmit: 0
tx34_xdp_full: 0
tx34_xdp_err: 0
tx34_xdp_cqes: 0
tx35_xdp_xmit: 0
tx35_xdp_full: 0
tx35_xdp_err: 0
tx35_xdp_cqes: 0
tx36_xdp_xmit: 0
tx36_xdp_full: 0
tx36_xdp_err: 0
tx36_xdp_cqes: 0
tx37_xdp_xmit: 0
tx37_xdp_full: 0
tx37_xdp_err: 0
tx37_xdp_cqes: 0
tx38_xdp_xmit: 0
tx38_xdp_full: 0
tx38_xdp_err: 0
tx38_xdp_cqes: 0
tx39_xdp_xmit: 0
tx39_xdp_full: 0
tx39_xdp_err: 0
tx39_xdp_cqes: 0
tx40_xdp_xmit: 0
tx40_xdp_full: 0
tx40_xdp_err: 0
tx40_xdp_cqes: 0
tx41_xdp_xmit: 0
tx41_xdp_full: 0
tx41_xdp_err: 0
tx41_xdp_cqes: 0
tx42_xdp_xmit: 0
tx42_xdp_full: 0
tx42_xdp_err: 0
tx42_xdp_cqes: 0
tx43_xdp_xmit: 0
tx43_xdp_full: 0
tx43_xdp_err: 0
tx43_xdp_cqes: 0
tx44_xdp_xmit: 0
tx44_xdp_full: 0
tx44_xdp_err: 0
tx44_xdp_cqes: 0
tx45_xdp_xmit: 0
tx45_xdp_full: 0
tx45_xdp_err: 0
tx45_xdp_cqes: 0
tx46_xdp_xmit: 0
tx46_xdp_full: 0
tx46_xdp_err: 0
tx46_xdp_cqes: 0
tx47_xdp_xmit: 0
tx47_xdp_full: 0
tx47_xdp_err: 0
tx47_xdp_cqes: 0
tx48_xdp_xmit: 0
tx48_xdp_full: 0
tx48_xdp_err: 0
tx48_xdp_cqes: 0
tx49_xdp_xmit: 0
tx49_xdp_full: 0
tx49_xdp_err: 0
tx49_xdp_cqes: 0
tx50_xdp_xmit: 0
tx50_xdp_full: 0
tx50_xdp_err: 0
tx50_xdp_cqes: 0
tx51_xdp_xmit: 0
tx51_xdp_full: 0
tx51_xdp_err: 0
tx51_xdp_cqes: 0
tx52_xdp_xmit: 0
tx52_xdp_full: 0
tx52_xdp_err: 0
tx52_xdp_cqes: 0
tx53_xdp_xmit: 0
tx53_xdp_full: 0
tx53_xdp_err: 0
tx53_xdp_cqes: 0
tx54_xdp_xmit: 0
tx54_xdp_full: 0
tx54_xdp_err: 0
tx54_xdp_cqes: 0
tx55_xdp_xmit: 0
tx55_xdp_full: 0
tx55_xdp_err: 0
tx55_xdp_cqes: 0
> [...]
>
>>>> ethtool -S enp175s0f0
>>>> NIC statistics:
>>>> rx_packets: 141574897253
>>>> rx_bytes: 184445040406258
>>>> tx_packets: 172569543894
>>>> tx_bytes: 99486882076365
>>>> tx_tso_packets: 9367664195
>>>> tx_tso_bytes: 56435233992948
>>>> tx_tso_inner_packets: 0
>>>> tx_tso_inner_bytes: 0
>>>> tx_added_vlan_packets: 141297671626
>>>> tx_nop: 2102916272
>>>> rx_lro_packets: 0
>>>> rx_lro_bytes: 0
>>>> rx_ecn_mark: 0
>>>> rx_removed_vlan_packets: 141574897252
>>>> rx_csum_unnecessary: 0
>>>> rx_csum_none: 23135854
>>>> rx_csum_complete: 141551761398
>>>> rx_csum_unnecessary_inner: 0
>>>> rx_xdp_drop: 0
>>>> rx_xdp_redirect: 0
>>>> rx_xdp_tx_xmit: 0
>>>> rx_xdp_tx_full: 0
>>>> rx_xdp_tx_err: 0
>>>> rx_xdp_tx_cqe: 0
>>>> tx_csum_none: 127934791664
>>> It is a good idea to look into this, tx is not requesting hw tx
>>> csumming for a lot of packets, maybe you are wasting a lot of cpu
>>> on
>>> calculating csum, or maybe this is just the rx csum complete..
>>>
>>>> tx_csum_partial: 13362879974
>>>> tx_csum_partial_inner: 0
>>>> tx_queue_stopped: 232561
>>> TX queues are stalling, could be an indentation for the pcie
>>> bottelneck.
>>>
>>>> tx_queue_dropped: 0
>>>> tx_xmit_more: 1266021946
>>>> tx_recover: 0
>>>> tx_cqes: 140031716469
>>>> tx_queue_wake: 232561
>>>> tx_udp_seg_rem: 0
>>>> tx_cqe_err: 0
>>>> tx_xdp_xmit: 0
>>>> tx_xdp_full: 0
>>>> tx_xdp_err: 0
>>>> tx_xdp_cqes: 0
>>>> rx_wqe_err: 0
>>>> rx_mpwqe_filler_cqes: 0
>>>> rx_mpwqe_filler_strides: 0
>>>> rx_buff_alloc_err: 0
>>>> rx_cqe_compress_blks: 0
>>>> rx_cqe_compress_pkts: 0
>>>> rx_page_reuse: 0
>>>> rx_cache_reuse: 16625975793
>>>> rx_cache_full: 54161465914
>>>> rx_cache_empty: 258048
>>>> rx_cache_busy: 54161472735
>>>> rx_cache_waive: 0
>>>> rx_congst_umr: 0
>>>> rx_arfs_err: 0
>>>> ch_events: 40572621887
>>>> ch_poll: 40885650979
>>>> ch_arm: 40429276692
>>>> ch_aff_change: 0
>>>> ch_eq_rearm: 0
>>>> rx_out_of_buffer: 2791690
>>>> rx_if_down_packets: 74
>>>> rx_vport_unicast_packets: 141843476308
>>>> rx_vport_unicast_bytes: 185421265403318
>>>> tx_vport_unicast_packets: 172569484005
>>>> tx_vport_unicast_bytes: 100019940094298
>>>> rx_vport_multicast_packets: 85122935
>>>> rx_vport_multicast_bytes: 5761316431
>>>> tx_vport_multicast_packets: 6452
>>>> tx_vport_multicast_bytes: 643540
>>>> rx_vport_broadcast_packets: 22423624
>>>> rx_vport_broadcast_bytes: 1390127090
>>>> tx_vport_broadcast_packets: 22024
>>>> tx_vport_broadcast_bytes: 1321440
>>>> rx_vport_rdma_unicast_packets: 0
>>>> rx_vport_rdma_unicast_bytes: 0
>>>> tx_vport_rdma_unicast_packets: 0
>>>> tx_vport_rdma_unicast_bytes: 0
>>>> rx_vport_rdma_multicast_packets: 0
>>>> rx_vport_rdma_multicast_bytes: 0
>>>> tx_vport_rdma_multicast_packets: 0
>>>> tx_vport_rdma_multicast_bytes: 0
>>>> tx_packets_phy: 172569501577
>>>> rx_packets_phy: 142871314588
>>>> rx_crc_errors_phy: 0
>>>> tx_bytes_phy: 100710212814151
>>>> rx_bytes_phy: 187209224289564
>>>> tx_multicast_phy: 6452
>>>> tx_broadcast_phy: 22024
>>>> rx_multicast_phy: 85122933
>>>> rx_broadcast_phy: 22423623
>>>> rx_in_range_len_errors_phy: 2
>>>> rx_out_of_range_len_phy: 0
>>>> rx_oversize_pkts_phy: 0
>>>> rx_symbol_err_phy: 0
>>>> tx_mac_control_phy: 0
>>>> rx_mac_control_phy: 0
>>>> rx_unsupported_op_phy: 0
>>>> rx_pause_ctrl_phy: 0
>>>> tx_pause_ctrl_phy: 0
>>>> rx_discards_phy: 920161423
>>> Ok, this port seem to be suffering more, RX is congested, maybe due
>>> to
>>> the pcie bottleneck.
>> Yes this side is receiving more traffic - second port is +10G more tx
>>
> [...]
>
>
>>>> Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
>>>> 0.00 0.00 0.00 31.30
>>>> Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
>>>> 0.00 0.00 0.00 24.90
>>>> Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
>>>> 0.00 0.00 0.00 19.68
>>>> Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
>>>> 0.00 0.00 0.00 18.00
>>>> Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
>>>> 0.00 0.00 0.00 17.40
>>>> Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
>>>> 0.00 0.00 0.00 26.03
>>>> Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
>>>> 0.00 0.00 0.00 17.52
>>>> Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
>>>> 0.00 0.00 0.00 17.68
>>>> Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
>>>> 0.00 0.00 0.00 18.20
>>>> Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
>>>> 0.00 0.00 0.00 20.42
>>> The numa cores are not at 100% util, you have around 20% of idle on
>>> each one.
>> Yes - no 100% cpu - but the difference between 80% and 100% is like
>> push
>> aditional 1-2Gbit/s
>>
> yes but, it doens't look like the bottleneck is the cpu, although it is
> close to be :)..
>
>>>> Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 96.10
>>>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.70
>>>> Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.50
>>>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 100.00
>>>> Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 97.40
>>>> Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.10
>>>> Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
>>>> 0.00
>>>> 0.00 0.00 99.40
>>>> Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
>>>> 0.00 0.00 0.00 19.42
>>>> Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
>>>> 0.00 0.00 0.00 26.60
>>>> Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
>>>> 0.00 0.00 0.00 20.50
>>>> Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
>>>> 0.00 0.00 0.00 21.60
>>>> Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
>>>> 0.00 0.00 0.00 24.58
>>>> Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
>>>> 0.00 0.00 0.00 24.68
>>>> Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
>>>> 0.00 0.00 0.00 28.34
>>>> Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
>>>> 0.00 0.00 0.00 25.83
>>>> Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
>>>> 0.00 0.00 0.00 27.20
>>>> Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
>>>> 0.00 0.00 0.00 28.03
>>>> Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
>>>> 0.00 0.00 0.00 24.70
>>>> Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
>>>> 0.00 0.00 0.00 25.30
>>>> Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
>>>> 0.00 0.00 0.00 26.77
>>>>
>>>>
>>>> ethtool -k enp175s0f0
>>>> Features for enp175s0f0:
>>>> rx-checksumming: on
>>>> tx-checksumming: on
>>>> tx-checksum-ipv4: on
>>>> tx-checksum-ip-generic: off [fixed]
>>>> tx-checksum-ipv6: on
>>>> tx-checksum-fcoe-crc: off [fixed]
>>>> tx-checksum-sctp: off [fixed]
>>>> scatter-gather: on
>>>> tx-scatter-gather: on
>>>> tx-scatter-gather-fraglist: off [fixed]
>>>> tcp-segmentation-offload: on
>>>> tx-tcp-segmentation: on
>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>> tx-tcp-mangleid-segmentation: off
>>>> tx-tcp6-segmentation: on
>>>> udp-fragmentation-offload: off
>>>> generic-segmentation-offload: on
>>>> generic-receive-offload: on
>>>> large-receive-offload: off [fixed]
>>>> rx-vlan-offload: on
>>>> tx-vlan-offload: on
>>>> ntuple-filters: off
>>>> receive-hashing: on
>>>> highdma: on [fixed]
>>>> rx-vlan-filter: on
>>>> vlan-challenged: off [fixed]
>>>> tx-lockless: off [fixed]
>>>> netns-local: off [fixed]
>>>> tx-gso-robust: off [fixed]
>>>> tx-fcoe-segmentation: off [fixed]
>>>> tx-gre-segmentation: on
>>>> tx-gre-csum-segmentation: on
>>>> tx-ipxip4-segmentation: off [fixed]
>>>> tx-ipxip6-segmentation: off [fixed]
>>>> tx-udp_tnl-segmentation: on
>>>> tx-udp_tnl-csum-segmentation: on
>>>> tx-gso-partial: on
>>>> tx-sctp-segmentation: off [fixed]
>>>> tx-esp-segmentation: off [fixed]
>>>> tx-udp-segmentation: on
>>>> fcoe-mtu: off [fixed]
>>>> tx-nocache-copy: off
>>>> loopback: off [fixed]
>>>> rx-fcs: off
>>>> rx-all: off
>>>> tx-vlan-stag-hw-insert: on
>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>> rx-vlan-stag-filter: on [fixed]
>>>> l2-fwd-offload: off [fixed]
>>>> hw-tc-offload: off
>>>> esp-hw-offload: off [fixed]
>>>> esp-tx-csum-hw-offload: off [fixed]
>>>> rx-udp_tunnel-port-offload: on
>>>> tls-hw-tx-offload: off [fixed]
>>>> tls-hw-rx-offload: off [fixed]
>>>> rx-gro-hw: off [fixed]
>>>> tls-hw-record: off [fixed]
>>>>
>>>> ethtool -c enp175s0f0
>>>> Coalesce parameters for enp175s0f0:
>>>> Adaptive RX: off TX: on
>>>> stats-block-usecs: 0
>>>> sample-interval: 0
>>>> pkt-rate-low: 0
>>>> pkt-rate-high: 0
>>>> dmac: 32703
>>>>
>>>> rx-usecs: 256
>>>> rx-frames: 128
>>>> rx-usecs-irq: 0
>>>> rx-frames-irq: 0
>>>>
>>>> tx-usecs: 8
>>>> tx-frames: 128
>>>> tx-usecs-irq: 0
>>>> tx-frames-irq: 0
>>>>
>>>> rx-usecs-low: 0
>>>> rx-frame-low: 0
>>>> tx-usecs-low: 0
>>>> tx-frame-low: 0
>>>>
>>>> rx-usecs-high: 0
>>>> rx-frame-high: 0
>>>> tx-usecs-high: 0
>>>> tx-frame-high: 0
>>>>
>>>> ethtool -g enp175s0f0
>>>> Ring parameters for enp175s0f0:
>>>> Pre-set maximums:
>>>> RX: 8192
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 8192
>>>> Current hardware settings:
>>>> RX: 4096
>>>> RX Mini: 0
>>>> RX Jumbo: 0
>>>> TX: 4096
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>> Also changed a little coalesce params - and best for this config are:
>> ethtool -c enp175s0f0
>> Coalesce parameters for enp175s0f0:
>> Adaptive RX: off TX: off
>> stats-block-usecs: 0
>> sample-interval: 0
>> pkt-rate-low: 0
>> pkt-rate-high: 0
>> dmac: 32573
>>
>> rx-usecs: 40
>> rx-frames: 128
>> rx-usecs-irq: 0
>> rx-frames-irq: 0
>>
>> tx-usecs: 8
>> tx-frames: 8
>> tx-usecs-irq: 0
>> tx-frames-irq: 0
>>
>> rx-usecs-low: 0
>> rx-frame-low: 0
>> tx-usecs-low: 0
>> tx-frame-low: 0
>>
>> rx-usecs-high: 0
>> rx-frame-high: 0
>> tx-usecs-high: 0
>> tx-frame-high: 0
>>
>>
>> Less drops on RX side - and more pps in overall forwarded.
>>
> how much improvement ? maybe we can improve our adaptive rx coal to be
> efficient for this work load.
>
>
I can prepare more stats with ethtool maybee to compare - but normally
tested with simple icmp forwarded from interface to interface
- before change coalescence params:
adaptive-rx off rx-usecs 384 rx-frames 128
3% loss for icmp
- after change to:
adaptive-rx off rx-usecs 40 rx-frames 128 adaptive-tx off tx-usecs 8
tx-frames 8
2% loss for icmp
But yes - to know better will need to compare rx/tx counters from
ethtool + /proc/net/dev
Was trying to turn on adaptative-tx+rx - but 100% saturation at 43Gbit/s
RX / 43Gbit/s TX
^ permalink raw reply
* Re: [PATCH iproute2 net-next 0/3] ss: Allow selection of columns to be displayed
From: David Ahern @ 2018-11-01 21:18 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Stefano Brivio, Yoann P., Stephen Hemminger, netdev
In-Reply-To: <20181101140623.4d6211a0@cakuba.netronome.com>
On 11/1/18 3:06 PM, Jakub Kicinski wrote:
> On Wed, 31 Oct 2018 20:48:05 -0600, David Ahern wrote:
>>> spacing with a special character in the format string, that is:
>>>
>>> "%S.%Qr.%Qs %Al:%Pl %Ar:%Pr %p\n"
>>>
>>> would mean "align everything to the right, distribute remaining
>>> whitespace between %S, %Qr and %Qs". But it looks rather complicated
>>> at a glance.
>>>
>>
>> My concern here is that once this goes in for 1 command, the others in
>> iproute2 need to follow suit - meaning same syntax style for all
>> commands. Given that I'd prefer we get a reasonable consensus on syntax
>> that will work across commands -- ss, ip, tc. If it is as simple as
>> column names with a fixed order, that is fine but just give proper
>> consideration given the impact.
>
> FWIW I just started piping iproute2 commands to jq. Example:
>
> tc -s -j qdisc show dev em1 | \
> jq -r '.[] | [.kind,.parent,.handle,.offloaded,.bytes,.packets,.drops,.overlimits,.requeues,.backlog,.qlen,.marked] | @tsv'
>
> JSONification would probably be quite an undertaking for ss :(
>
Right, that is used in some of the scripts under
tools/testing/selftests. I would put that in the 'heavyweight solution'
category.
A number of key commands offer the capability to control the output via
command line argument (e.g., ps, perf script). Given the amount of data
iproute2 commands throw at a user by default, it would be a good
usability feature to allow a user to customize the output without having
to pipe it into other commands.
^ permalink raw reply
* Re: bnx2: rx_fw_discards: BCM5716 sporadically drops packets when update to driver version 2.2.6
From: maowenan @ 2018-11-02 6:14 UTC (permalink / raw)
To: Mody, Rasesh, netdev@vger.kernel.org, f.fainelli@gmail.com,
andrew@lunn.ch, linux-kernel@vger.kernel.org
Cc: Rahman, Ameen, maowenan
In-Reply-To: <090bad42-5ddb-2973-8cb1-bcdd03a05814@huawei.com>
@Mody, Rasesh
Could you help take a look at this?
On 2018/10/30 17:03, maowenan wrote:
>
>
> On 2018/10/30 14:47, Mody, Rasesh wrote:
>>> From: maowenan <maowenan@huawei.com>
>>> Sent: Thursday, October 25, 2018 8:16 PM
>>>
>>> Hi，
>>>
>>> After I update version of bnx2 driver from 2.2.1 to 2.2.6, I find BCM5716
>>> sporadically drops packets, which shows in rx_fw_discards.
>>> C36-141-5:~ # ethtool -S NIC0
>>>
>>> NIC statistics:
>>> rx_ucast_packets: 11902
>>>
>>> rx_mcast_packets: 217
>>>
>>> rx_bcast_packets: 4954320
>>>
>>> rx_filtered_packets: 328793
>>>
>>> rx_fw_discards: 2742
>>>
>>> C36-141-5:~ #
>>>
>>> 5s later:
>>>
>>> C36-141-5:~ # ethtool -S NIC0
>>>
>>> NIC statistics:
>>> rx_ucast_packets: 11910
>>>
>>> rx_mcast_packets: 217
>>>
>>> rx_bcast_packets: 4958117
>>>
>>> rx_filtered_packets: 328897
>>>
>>> rx_fw_discards: 2750
>>>
>>> C36-141-5:~ #
>>>
>>> so rx_fw_discards: 2742-----> rx_fw_discards: 2750, lost 8 packets.
>>>
>>> the information of bnx2
>>> C36-141-5:~ # modinfo bnx2
>>> kernel/drivers/net/ethernet/broadcom/bnx2.ko
>>>
>>> firmware: bnx2/bnx2-rv2p-09ax-6.0.17.fw
>>>
>>> firmware: bnx2/bnx2-rv2p-09-6.0.17.fw
>>>
>>> firmware: bnx2/bnx2-mips-09-6.2.1b.fw
>>>
>>> firmware: bnx2/bnx2-rv2p-06-6.0.15.fw
>>>
>>> firmware: bnx2/bnx2-mips-06-6.2.3.fw
>>> version: 2.2.6
>>>
>>>
>>> 1) Firstly, I check the patches from 2.2.1 to 2.2.6, below patch is interesting.
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id
>>> =0021850d0417a4dc38ed871d929b651b87e2ead9
>>> Do not enable filter SORT MODE in chip init routine. This patch addresses an
>>> issue where BCM5716 sporadically drops packets when changing multicast list.
>>>
>>> diff --git a/drivers/net/ethernet/broadcom/bnx2.c
>>> b/drivers/net/ethernet/broadcom/bnx2.c
>>> index 8957eb5f4478..8c9a8b7787d2 100644
>>> --- a/drivers/net/ethernet/broadcom/bnx2.c
>>> +++ b/drivers/net/ethernet/broadcom/bnx2.c
>>> @@ -4984,8 +4984,6 @@ bnx2_init_chip(struct bnx2 *bp)
>>>
>>> bp->idle_chk_status_idx = 0xffff;
>>>
>>> - bp->rx_mode = BNX2_EMAC_RX_MODE_SORT_MODE;
>>> -
>>> /* Set up how to generate a link change interrupt. */
>>> BNX2_WR(bp, BNX2_EMAC_ATTENTION_ENA,
>>> BNX2_EMAC_ATTENTION_ENA_LINK);
>>>
>>>
>>> 2) Secondly, I revert this patch, after verify it, I find rx_fw_discards does not
>>> increasing.
>>> so I think this patch can fix current issue. But I'm not sure the issue of this
>>> patch to fix will be reproduced?
>>> I'm not convinced that what factor will trigger rx_fw_discards increasing?
>>> And how to fix this?
>>
>> Can you please reword your point above? i.e. what is working and what is not. I am not sure if I understand it completely.
>>
>> Is the rx_fw_disacard count incrementing with 2.2.6 upstream driver on BCM5716? What is the kernel version?
>> Which test is being run?
>
>
> The case is based on BCM5716's driver version of 2.2.6, and both 3.10 and mainline kernel version exist this issue .
> The testing seems like using one port of BCM5716 to receive packets, it can be found rx_fw_disacard increasing occasionally.
> There is one patch in 2.2.6, 0021850d0417a4dc38ed871d929b651b87e2ead9, if I remove this patch, I can't reproduce this issue,
> rx_fw_disacard doesn't increase.
> So I don't know why this patch(0021850d0417a4dc38ed871d929b651b87e2ead9) can fix my problem(rx_fw_disacard increasing)?
> If I remove this patch, how to fix the issue that the patch had resolved?
>
>>
>>>
>>> Thanks a lot.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
>
> .
>
^ permalink raw reply
* Re: [PATCH bpf-next v2 02/13] bpf: btf: Add BTF_KIND_FUNC and BTF_KIND_FUNC_PROTO
From: Edward Cree @ 2018-11-01 21:08 UTC (permalink / raw)
To: Martin Lau
Cc: Yonghong Song, Alexei Starovoitov, daniel@iogearbox.net,
netdev@vger.kernel.org, Kernel Team
In-Reply-To: <20181019232704.j45f7fbhreyinjxk@kafai-mbp.dhcp.thefacebook.com>
I've spent a bit more time thinking about / sleeping on this, and I
still think there's a major disagreement here. Basically it seems
like I'm saying "the design of BTF is wrong" and you're saying "but
it's the design" (with the possible implication — I'm not entirely
sure — of "but that's what DWARF does").
So let's back away from the details about FUNC/PROTO, and talk in
more general terms about what a BTF record means.
There are two classes of things we might want to put in debug-info:
* There exists a type T
* I have an instance X (variable, subprogram, etc.) of type T
Both of these may need to reference other types, and have the same
space of possible things T could be, but there the similarity ends:
they are semantically different things.
Indeed, the only reason for any record of the first class is to
define types referenced by records of the second class. Some
concrete examples of records of the second class are:
1) I have a map named "foo" with key-type T1 and value-type T2
2) I have a subprogram named "bar" with prototype T3
3) I am using stack slot fp-8 to store a value of type T4
4) I am casting ctx+8 to a pointer type T5 before dereferencing it
Currently we have (1) and this patch series adds (2), both done
through records that look like they are just defining a type (i.e.
the first class of record) but have 'magic' semantics (in the case
of (1), special names of the form ____btf_map_foo. How anyone
thought that was a clean and tasteful design is beyond me.)
What IMHO the design *should* be, is that we have a 'types'
subsection that *only* contains records of the first class, and
then other subsections to hold records of the second class that
reference records of the first class by ID. So for (1) you'd have
either additional fields in struct bpf_map_def (we've extended that
several times before, after all), or you'd have a maps table in
.BTF that links map names ("foo", not "____btf_map_foo"!) with type
IDs for its key and value:
struct btf_map_record {
__u32 name_off; /* name of map */
__u32 key_type_id; /* index in "types" table */
__u32 value_type_id; /* ditto */
}
(Note the absence of any meaningless struct type as created by
BPF_ANNOTATE_KV_PAIR. That kind of source-level hack should be
converted by the compiler's BTF output module into something less
magic, rather than baked into the format definition.)
Then for (2) you'd have a functions table in .BTF that links subprog
names, start offsets, and signatures/prototypes:
struct btf_func_record {
__u32 name_off; /* name of function */
__u16 subprog_secn; /* section index in which func appears */
__u16 subprog_start; /* offset in section of func entry point */
__u32 type_id; /* index in "types" table of func signature */
}
I believe this is a much cleaner design, which will be easier to extend
in the future to add things like (3) and (4) for source-line-level
debug information. I also believe that if someone had written
documentation describing the original design, semantics of the various
BTF records, etc., it would have been immediately obvious that the
design was needlessly confusing and ad-hoc.
On 20/10/18 00:27, Martin Lau wrote:
> Like struct, the member's names of struct is part of the btf_type.
> A struct with the same member's types but different member's names
> is a different btf_type.
Yes, but that's not what I'm talking about. I'm talking about structs
with the same member names, but with different names of the structs.
As in the following C snippet:
struct foo {
int i;
};
int main(void)
{
struct foo x;
struct foo y;
x.i = 0
y.i = x.i;
return y.i;
}
We have one type 'struct foo' (name "foo"), but two _instances_ of
that type (names "x", "y"). We *cannot* use a single BTF record to
express both "x" and its type, because its type has a name of its
own ("foo") and there is only room in struct btf_type for one name.
Thus we must have one record for the instance "x" and another record
for the type "foo", with the former referencing the latter.
> Having two id spaces for debug-info is confusing. They are
> all debug-info at the end.
But they have different semantics! Just because you have a term,
"debug-info", that's defined to cover both, doesn't mean that they
are the same thing. You might as well say that passport numbers and
telephone numbers should be drawn from the same numbering space,
because they're both "personal information", and never mind that one
identifies a person and the other identifies a telephone.
It's having the _same_ id space for entities that are almost, but not
quite, entirely unlike each other that's confusing.
-Ed
^ permalink raw reply
* Re: [PATCH iproute2 net-next 0/3] ss: Allow selection of columns to be displayed
From: Jakub Kicinski @ 2018-11-01 21:06 UTC (permalink / raw)
To: David Ahern; +Cc: Stefano Brivio, Yoann P., Stephen Hemminger, netdev
In-Reply-To: <adb1ade4-e284-73fc-ff77-c1b1b2b4c3f1@gmail.com>
On Wed, 31 Oct 2018 20:48:05 -0600, David Ahern wrote:
> > spacing with a special character in the format string, that is:
> >
> > "%S.%Qr.%Qs %Al:%Pl %Ar:%Pr %p\n"
> >
> > would mean "align everything to the right, distribute remaining
> > whitespace between %S, %Qr and %Qs". But it looks rather complicated
> > at a glance.
> >
>
> My concern here is that once this goes in for 1 command, the others in
> iproute2 need to follow suit - meaning same syntax style for all
> commands. Given that I'd prefer we get a reasonable consensus on syntax
> that will work across commands -- ss, ip, tc. If it is as simple as
> column names with a fixed order, that is fine but just give proper
> consideration given the impact.
FWIW I just started piping iproute2 commands to jq. Example:
tc -s -j qdisc show dev em1 | \
jq -r '.[] | [.kind,.parent,.handle,.offloaded,.bytes,.packets,.drops,.overlimits,.requeues,.backlog,.qlen,.marked] | @tsv'
JSONification would probably be quite an undertaking for ss :(
^ permalink raw reply
* Re: [RFC PATCH v3 06/10] udp: cope with UDP GRO packet misdirection
From: Willem de Bruijn @ 2018-11-01 21:01 UTC (permalink / raw)
To: Paolo Abeni
Cc: Network Development, David Miller, Willem de Bruijn,
steffen.klassert, Subash Abhinov Kasiviswanathan
In-Reply-To: <debe3f0794fa2c0c80890a24b56aa6785a648acb.camel@redhat.com>
On Wed, Oct 31, 2018 at 5:57 AM Paolo Abeni <pabeni@redhat.com> wrote:
>
> On Tue, 2018-10-30 at 18:24 +0100, Paolo Abeni wrote:
> > --- a/include/net/udp.h
> > +++ b/include/net/udp.h
> > @@ -406,17 +406,24 @@ static inline int copy_linear_skb(struct sk_buff *skb, int len, int off,
> > } while(0)
> >
> > #if IS_ENABLED(CONFIG_IPV6)
> > -#define __UDPX_INC_STATS(sk, field) \
> > -do { \
> > - if ((sk)->sk_family == AF_INET) \
> > - __UDP_INC_STATS(sock_net(sk), field, 0); \
> > - else \
> > - __UDP6_INC_STATS(sock_net(sk), field, 0); \
> > -} while (0)
> > +#define __UDPX_MIB(sk, ipv4) \
> > +({ \
> > + ipv4 ? (IS_UDPLITE(sk) ? sock_net(sk)->mib.udplite_statistics : \
> > + sock_net(sk)->mib.udp_statistics) : \
> > + (IS_UDPLITE(sk) ? sock_net(sk)->mib.udplite_stats_in6 : \
> > + sock_net(sk)->mib.udp_stats_in6); \
> > +})
> > #else
> > -#define __UDPX_INC_STATS(sk, field) __UDP_INC_STATS(sock_net(sk), field, 0)
> > +#define __UDPX_MIB(sk, ipv4) \
> > +({ \
> > + IS_UDPLITE(sk) ? sock_net(sk)->mib.udplite_statistics : \
> > + sock_net(sk)->mib.udp_statistics; \
> > +})
> > #endif
> >
> > +#define __UDPX_INC_STATS(sk, field) \
> > + __SNMP_INC_STATS(__UDPX_MIB(sk, (sk)->sk_family == AF_INET, field)
> > +
>
> This is broken (complains only if CONFIG_AF_RXRPC is set), will fix in
> next iteration (thanks kbuildbot).
>
> But I'd prefer to keep the above helper: it can be used in a follow-up
> patch to cleanup a bit udp6_recvmsg().
>
> > #ifdef CONFIG_PROC_FS
> > struct udp_seq_afinfo {
> > sa_family_t family;
> > @@ -450,4 +457,32 @@ DECLARE_STATIC_KEY_FALSE(udpv6_encap_needed_key);
> > void udpv6_encap_enable(void);
> > #endif
> >
> > +static inline struct sk_buff *udp_rcv_segment(struct sock *sk,
> > + struct sk_buff *skb)
> > +{
> > + bool ipv4 = skb->protocol == htons(ETH_P_IP);
>
> And this cause a compile warning when # CONFIG_IPV6 is not set, I will
> fix in the next iteration (again thanks kbuildbot)
Can also just pass it as argument. This skb->protocol should work correctly
with tunneled packets, but it wasn't as immediately obvious to me.
Also
+ if (unlikely(!segs))
+ goto drop;
this should not happen. But if it could and the caller treats it the
same as error (both now return NULL), then skb needs to be freed.
^ permalink raw reply
* Re: [RFC PATCH v3 08/10] selftests: conditionally enable XDP support in udpgso_bench_rx
From: Willem de Bruijn @ 2018-11-01 21:00 UTC (permalink / raw)
To: Paolo Abeni
Cc: Network Development, David Miller, Willem de Bruijn,
steffen.klassert, Subash Abhinov Kasiviswanathan
In-Reply-To: <a94e1c791730c157ee97a051a3d8892bb32d9664.1540920083.git.pabeni@redhat.com>
On Tue, Oct 30, 2018 at 1:28 PM Paolo Abeni <pabeni@redhat.com> wrote:
>
> XDP support will be used by a later patch to test the GRO path
> in a net namespace, leveraging the veth XDP implementation.
> To avoid breaking existing setup, XDP support is conditionally
> enabled and build only if llc is locally available.
>
> rfc v2 -> rfc v3:
> - move 'x' option handling here
>
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> ---
> tools/testing/selftests/net/Makefile | 69 +++++++++++++++++++
> tools/testing/selftests/net/udpgso_bench_rx.c | 41 ++++++++++-
> tools/testing/selftests/net/xdp_dummy.c | 13 ++++
> 3 files changed, 121 insertions(+), 2 deletions(-)
> create mode 100644 tools/testing/selftests/net/xdp_dummy.c
>
> diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
> index 256d82d5fa87..176459b7c4d6 100644
> --- a/tools/testing/selftests/net/Makefile
> +++ b/tools/testing/selftests/net/Makefile
> @@ -16,8 +16,77 @@ TEST_GEN_PROGS = reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa
> TEST_GEN_PROGS += reuseport_dualstack reuseaddr_conflict tls
>
> KSFT_KHDR_INSTALL := 1
> +
> +# Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline:
> +# make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang
> +LLC ?= llc
> +CLANG ?= clang
> +LLVM_OBJCOPY ?= llvm-objcopy
> +BTF_PAHOLE ?= pahole
> +HAS_LLC := $(shell which $(LLC) 2>/dev/null)
> +
> +# conditional enable testes requiring llc
> +ifneq (, $(HAS_LLC))
> +TEST_GEN_FILES += xdp_dummy.o
> +endif
> +
> include ../lib.mk
>
> +ifneq (, $(HAS_LLC))
> +
> +# Detect that we're cross compiling and use the cross compiler
> +ifdef CROSS_COMPILE
> +CLANG_ARCH_ARGS = -target $(ARCH)
> +endif
> +
> +PROBE := $(shell $(LLC) -march=bpf -mcpu=probe -filetype=null /dev/null 2>&1)
> +
> +# Let newer LLVM versions transparently probe the kernel for availability
> +# of full BPF instruction set.
> +ifeq ($(PROBE),)
> + CPU ?= probe
> +else
> + CPU ?= generic
> +endif
> +
> +SRC_PATH := $(abspath ../../../..)
> +LIB_PATH := $(SRC_PATH)/tools/lib
> +XDP_CFLAGS := -D SUPPORT_XDP=1 -I$(LIB_PATH)
> +LIBBPF = $(LIB_PATH)/bpf/libbpf.a
> +BTF_LLC_PROBE := $(shell $(LLC) -march=bpf -mattr=help 2>&1 | grep dwarfris)
> +BTF_PAHOLE_PROBE := $(shell $(BTF_PAHOLE) --help 2>&1 | grep BTF)
> +BTF_OBJCOPY_PROBE := $(shell $(LLVM_OBJCOPY) --help 2>&1 | grep -i 'usage.*llvm')
> +CLANG_SYS_INCLUDES := $(shell $(CLANG) -v -E - </dev/null 2>&1 \
> + | sed -n '/<...> search starts here:/,/End of search list./{ s| \(/.*\)|-idirafter \1|p }')
> +CLANG_FLAGS = -I. -I$(SRC_PATH)/include -I../bpf/ \
> + $(CLANG_SYS_INCLUDES) -Wno-compare-distinct-pointer-types
> +
> +ifneq ($(and $(BTF_LLC_PROBE),$(BTF_PAHOLE_PROBE),$(BTF_OBJCOPY_PROBE)),)
> + CLANG_CFLAGS += -g
> + LLC_FLAGS += -mattr=dwarfris
> + DWARF2BTF = y
> +endif
> +
> +$(LIBBPF): FORCE
> +# Fix up variables inherited from Kbuild that tools/ build system won't like
> + $(MAKE) -C $(dir $@) RM='rm -rf' LDFLAGS= srctree=$(SRC_PATH) O= $(nodir $@)
> +
> +$(OUTPUT)/udpgso_bench_rx: $(OUTPUT)/udpgso_bench_rx.c $(LIBBPF)
> + $(CC) -o $@ $(XDP_CFLAGS) $(CFLAGS) $(LOADLIBES) $(LDLIBS) $^ -lelf
> +
> +FORCE:
> +
> +# bpf program[s] generation
> +$(OUTPUT)/%.o: %.c
> + $(CLANG) $(CLANG_FLAGS) \
> + -O2 -target bpf -emit-llvm -c $< -o - | \
> + $(LLC) -march=bpf -mcpu=$(CPU) $(LLC_FLAGS) -filetype=obj -o $@
> +ifeq ($(DWARF2BTF),y)
> + $(BTF_PAHOLE) -J $@
> +endif
> +
> +endif
> +
To get around having to add all this Makefile boilerplate, perhaps
don't integrate the xdp loader with udpgso_bench_rx, but add a
standalone trivial loader to /tools/testing/selftests/bpf. Akin to
samples/xdp1_user.c, but even simpler because without map. Then also
move xdp_dummy.c there.
That does add a cross-directory dependency, but that is not
significantly different from the SUPPORT_XDP conditional dependency
today.
On which note, the test initially silently failed for me because the
binary compiled, but without that option. It is probably better to
fail the test hard and with a clear error if XDP is not supported.
With that aside,
Tested-by: Willem de Bruijn <willemb@google.com>
^ permalink raw reply
* Re: [RFC PATCH v3 01/10] udp: implement complete book-keeping for encap_needed
From: Willem de Bruijn @ 2018-11-01 20:59 UTC (permalink / raw)
To: Paolo Abeni
Cc: Network Development, David Miller, Willem de Bruijn,
steffen.klassert, Subash Abhinov Kasiviswanathan
In-Reply-To: <f6e079479fb72b715af20644a915783f0e692a6c.1540920083.git.pabeni@redhat.com>
On Tue, Oct 30, 2018 at 1:28 PM Paolo Abeni <pabeni@redhat.com> wrote:
>
> The *encap_needed static keys are enabled by UDP tunnels
> and several UDP encapsulations type, but they are never
> turned off. This can cause unneeded overall performance
> degradation for systems where such features are used
> transiently.
>
> This patch introduces complete book-keeping for such keys,
> decreasing the usage at socket destruction time, if needed,
> and avoiding that the same socket could increase the key
> usage multiple times.
>
> rfc v2 - rfc v3:
> - use udp_tunnel_encap_enable() in setsockopt()
>
> Signed-off-by: Paolo Abeni <pabeni@redhat.com>
> @@ -2447,7 +2452,7 @@ int udp_lib_setsockopt(struct sock *sk, int level, int optname,
> /* FALLTHROUGH */
> case UDP_ENCAP_L2TPINUDP:
> up->encap_type = val;
> - udp_encap_enable();
> + udp_tunnel_encap_enable(sk->sk_socket);
this now also needs lock_sock?
^ permalink raw reply
* Re: [Patch net] net: drop skb on failure in ip_check_defrag()
From: David Miller @ 2018-11-01 20:56 UTC (permalink / raw)
To: xiyou.wangcong; +Cc: netdev, edumazet
In-Reply-To: <20181101190237.10830-1-xiyou.wangcong@gmail.com>
From: Cong Wang <xiyou.wangcong@gmail.com>
Date: Thu, 1 Nov 2018 12:02:37 -0700
> Most callers of pskb_trim_rcsum() simply drop the skb when
> it fails, however, ip_check_defrag() still continues to pass
> the skb up to stack. This is suspicious.
>
> In ip_check_defrag(), after we learn the skb is an IP fragment,
> passing the skb to callers makes no sense, because callers expect
> fragments are defrag'ed on success. So, dropping the skb when we
> can't defrag it is reasonable.
>
> Note, prior to commit 88078d98d1bb, this is not a big problem as
> checksum will be fixed up anyway. After it, the checksum is not
> correct on failure.
>
> Found this during code review.
>
> Fixes: 88078d98d1bb ("net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends")
> Cc: Eric Dumazet <edumazet@google.com>
> Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Applied and queued up for -stable, thanks!
^ permalink raw reply
* Re: [RFC 0/2] Delayed binding of UDP sockets for Quic per-connection sockets
From: Leif Hedstrom @ 2018-11-01 17:58 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Christoph Paasch, netdev, Ian Swett, Jana Iyengar
In-Reply-To: <0ce864f0-38b9-59cc-18ea-e071afca347d@gmail.com>
> On Oct 31, 2018, at 6:53 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
>
>
> On 10/31/2018 04:26 PM, Christoph Paasch wrote:
>> Implementations of Quic might want to create a separate socket for each
>> Quic-connection by creating a connected UDP-socket.
>>
>
> Nice proposal, but I doubt a QUIC server can afford having one UDP socket per connection ?
First thing: This is an idea we’ve been floating, and it’s not completed yet, so we don’t have any performance numbers etc. to share. The ideas for the implementation came up after a discussion with Ian and Jana re: their implementation of a QUIC server.
That much said, the general rationale for this is that having a socket for each QUIC connection could simplify integrating QUIC into existing software that already does epoll() over TCP sockets. This is how e.g. Apache Traffic Server works, which is our target implementation for QUIC.
>
> It would add a huge overhead in term of memory usage in the kernel,
> and lots of epoll events to manage (say a QUIC server with one million flows, receiving
> very few packets per second per flow)
Our use case is not millions of sockets, rather, 10’s of thousands. There would be one socket for each QUIC Connection, not per stream (obviously). At ~80Gbps on a box, we definitely see much less than 100k TCP connections.
Question: is there additional memory overhead here for the UDP sockets vs a normal TCP socket for e.g. HTTP or HTTP/2 ?
>
> Maybe you could elaborate on the need of having one UDP socket per connection.
We had a couple reasons:
1) Easier to integrate with existing epoll() based event processing
2) Possibly less CPU usage / faster handling, since scheduling is simplified with the epoll integration (untested)
Ian and Jana also had a couple of reasons why this delayed bind could be useful for their implementations, but I’ll leave it to them to go into details.
Cheers,
— leif
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Saeed Mahameed @ 2018-11-01 20:37 UTC (permalink / raw)
To: pstaszewski@itcare.pl, netdev@vger.kernel.org
In-Reply-To: <6486d01d-7a50-33c4-e27f-4ace8aa8e150@itcare.pl>
On Thu, 2018-11-01 at 12:09 +0100, Paweł Staszewski wrote:
>
> W dniu 01.11.2018 o 10:50, Saeed Mahameed pisze:
> > On Wed, 2018-10-31 at 22:57 +0100, Paweł Staszewski wrote:
> > > Hi
> > >
> > > So maybee someone will be interested how linux kernel handles
> > > normal
> > > traffic (not pktgen :) )
> > >
> > >
> > > Server HW configuration:
> > >
> > > CPU : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
> > >
> > > NIC's: 2x 100G Mellanox ConnectX-4 (connected to x16 pcie 8GT)
> > >
> > >
> > > Server software:
> > >
> > > FRR - as routing daemon
> > >
> > > enp175s0f0 (100G) - 16 vlans from upstreams (28 RSS binded to
> > > local
> > > numa
> > > node)
> > >
> > > enp175s0f1 (100G) - 343 vlans to clients (28 RSS binded to local
> > > numa
> > > node)
> > >
> > >
> > > Maximum traffic that server can handle:
> > >
> > > Bandwidth
> > >
> > > bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> > > input: /proc/net/dev type: rate
> > > \ iface Rx Tx Total
> > > =================================================================
> > > ====
> > > =========
> > > enp175s0f1: 28.51 Gb/s 37.24
> > > Gb/s
> > > 65.74 Gb/s
> > > enp175s0f0: 38.07 Gb/s 28.44
> > > Gb/s
> > > 66.51 Gb/s
> > > ---------------------------------------------------------------
> > > ----
> > > -----------
> > > total: 66.58 Gb/s 65.67
> > > Gb/s
> > > 132.25 Gb/s
> > >
> > >
> > > Packets per second:
> > >
> > > bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> > > input: /proc/net/dev type: rate
> > > - iface Rx Tx Total
> > > =================================================================
> > > ====
> > > =========
> > > enp175s0f1: 5248589.00 P/s 3486617.75 P/s
> > > 8735207.00 P/s
> > > enp175s0f0: 3557944.25 P/s 5232516.00 P/s
> > > 8790460.00 P/s
> > > ---------------------------------------------------------------
> > > ----
> > > -----------
> > > total: 8806533.00 P/s 8719134.00 P/s
> > > 17525668.00 P/s
> > >
> > >
> > > After reaching that limits nics on the upstream side (more RX
> > > traffic)
> > > start to drop packets
> > >
> > >
> > > I just dont understand that server can't handle more bandwidth
> > > (~40Gbit/s is limit where all cpu's are 100% util) - where pps on
> > > RX
> > > side are increasing.
> > >
> >
> > Where do you see 40 Gb/s ? you showed that both ports on the same
> > NIC (
> > same pcie link) are doing 66.58 Gb/s (RX) + 65.67 Gb/s (TX) =
> > 132.25
> > Gb/s which aligns with your pcie link limit, what am i missing ?
>
> hmm yes that was my concern also - cause cant find anywhere
> informations
> about that bandwidth is uni or bidirectional - so if 126Gbit for x16
> 8GT
> is unidir - then bidir will be 126/2 ~68Gbit - which will fit total
> bw
> on both ports
i think it is bidir
> This can explain maybee also why cpuload is rising rapidly from
> 120Gbit/s in total to 132Gbit (counters of bwmng are from /proc/net -
> so
> there can be some error in reading them when offloading (gro/gso/tso)
> on
> nic's is enabled that is why
>
> >
> > > Was thinking that maybee reached some pcie x16 limit - but x16
> > > 8GT
> > > is
> > > 126Gbit - and also when testing with pktgen i can reach more bw
> > > and
> > > pps
> > > (like 4x more comparing to normal internet traffic)
> > >
> >
> > Are you forwarding when using pktgen as well or you just testing
> > the RX
> > side pps ?
>
> Yes pktgen was tested on single port RX
> Can check also forwarding to eliminate pciex limits
>
So this explains why you have more RX pps, since tx is idle and pcie
will be free to do only rx.
[...]
> >
> > > ethtool -S enp175s0f1
> > > NIC statistics:
> > > rx_packets: 173730800927
> > > rx_bytes: 99827422751332
> > > tx_packets: 142532009512
> > > tx_bytes: 184633045911222
> > > tx_tso_packets: 25989113891
> > > tx_tso_bytes: 132933363384458
> > > tx_tso_inner_packets: 0
> > > tx_tso_inner_bytes: 0
> > > tx_added_vlan_packets: 74630239613
> > > tx_nop: 2029817748
> > > rx_lro_packets: 0
> > > rx_lro_bytes: 0
> > > rx_ecn_mark: 0
> > > rx_removed_vlan_packets: 173730800927
> > > rx_csum_unnecessary: 0
> > > rx_csum_none: 434357
> > > rx_csum_complete: 173730366570
> > > rx_csum_unnecessary_inner: 0
> > > rx_xdp_drop: 0
> > > rx_xdp_redirect: 0
> > > rx_xdp_tx_xmit: 0
> > > rx_xdp_tx_full: 0
> > > rx_xdp_tx_err: 0
> > > rx_xdp_tx_cqe: 0
> > > tx_csum_none: 38260960853
> > > tx_csum_partial: 36369278774
> > > tx_csum_partial_inner: 0
> > > tx_queue_stopped: 1
> > > tx_queue_dropped: 0
> > > tx_xmit_more: 748638099
> > > tx_recover: 0
> > > tx_cqes: 73881645031
> > > tx_queue_wake: 1
> > > tx_udp_seg_rem: 0
> > > tx_cqe_err: 0
> > > tx_xdp_xmit: 0
> > > tx_xdp_full: 0
> > > tx_xdp_err: 0
> > > tx_xdp_cqes: 0
> > > rx_wqe_err: 0
> > > rx_mpwqe_filler_cqes: 0
> > > rx_mpwqe_filler_strides: 0
> > > rx_buff_alloc_err: 0
> > > rx_cqe_compress_blks: 0
> > > rx_cqe_compress_pkts: 0
> >
> > If this is a pcie bottleneck it might be useful to enable CQE
> > compression (to reduce PCIe completion descriptors transactions)
> > you should see the above rx_cqe_compress_pkts increasing when
> > enabled.
> >
> > $ ethtool --set-priv-flags enp175s0f1 rx_cqe_compress on
> > $ ethtool --show-priv-flags enp175s0f1
> > Private flags for p6p1:
> > rx_cqe_moder : on
> > cqe_moder : off
> > rx_cqe_compress : on
> > ...
> >
> > try this on both interfaces.
>
> Done
> ethtool --show-priv-flags enp175s0f1
> Private flags for enp175s0f1:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : on
> rx_striding_rq : off
> rx_no_csum_complete: off
>
> ethtool --show-priv-flags enp175s0f0
> Private flags for enp175s0f0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : on
> rx_striding_rq : off
> rx_no_csum_complete: off
>
did it help reduce the load on the pcie ? do you see more pps ?
what is the ratio between rx_cqe_compress_pkts and over all rx packets
?
[...]
> > > ethtool -S enp175s0f0
> > > NIC statistics:
> > > rx_packets: 141574897253
> > > rx_bytes: 184445040406258
> > > tx_packets: 172569543894
> > > tx_bytes: 99486882076365
> > > tx_tso_packets: 9367664195
> > > tx_tso_bytes: 56435233992948
> > > tx_tso_inner_packets: 0
> > > tx_tso_inner_bytes: 0
> > > tx_added_vlan_packets: 141297671626
> > > tx_nop: 2102916272
> > > rx_lro_packets: 0
> > > rx_lro_bytes: 0
> > > rx_ecn_mark: 0
> > > rx_removed_vlan_packets: 141574897252
> > > rx_csum_unnecessary: 0
> > > rx_csum_none: 23135854
> > > rx_csum_complete: 141551761398
> > > rx_csum_unnecessary_inner: 0
> > > rx_xdp_drop: 0
> > > rx_xdp_redirect: 0
> > > rx_xdp_tx_xmit: 0
> > > rx_xdp_tx_full: 0
> > > rx_xdp_tx_err: 0
> > > rx_xdp_tx_cqe: 0
> > > tx_csum_none: 127934791664
> >
> > It is a good idea to look into this, tx is not requesting hw tx
> > csumming for a lot of packets, maybe you are wasting a lot of cpu
> > on
> > calculating csum, or maybe this is just the rx csum complete..
> >
> > > tx_csum_partial: 13362879974
> > > tx_csum_partial_inner: 0
> > > tx_queue_stopped: 232561
> >
> > TX queues are stalling, could be an indentation for the pcie
> > bottelneck.
> >
> > > tx_queue_dropped: 0
> > > tx_xmit_more: 1266021946
> > > tx_recover: 0
> > > tx_cqes: 140031716469
> > > tx_queue_wake: 232561
> > > tx_udp_seg_rem: 0
> > > tx_cqe_err: 0
> > > tx_xdp_xmit: 0
> > > tx_xdp_full: 0
> > > tx_xdp_err: 0
> > > tx_xdp_cqes: 0
> > > rx_wqe_err: 0
> > > rx_mpwqe_filler_cqes: 0
> > > rx_mpwqe_filler_strides: 0
> > > rx_buff_alloc_err: 0
> > > rx_cqe_compress_blks: 0
> > > rx_cqe_compress_pkts: 0
> > > rx_page_reuse: 0
> > > rx_cache_reuse: 16625975793
> > > rx_cache_full: 54161465914
> > > rx_cache_empty: 258048
> > > rx_cache_busy: 54161472735
> > > rx_cache_waive: 0
> > > rx_congst_umr: 0
> > > rx_arfs_err: 0
> > > ch_events: 40572621887
> > > ch_poll: 40885650979
> > > ch_arm: 40429276692
> > > ch_aff_change: 0
> > > ch_eq_rearm: 0
> > > rx_out_of_buffer: 2791690
> > > rx_if_down_packets: 74
> > > rx_vport_unicast_packets: 141843476308
> > > rx_vport_unicast_bytes: 185421265403318
> > > tx_vport_unicast_packets: 172569484005
> > > tx_vport_unicast_bytes: 100019940094298
> > > rx_vport_multicast_packets: 85122935
> > > rx_vport_multicast_bytes: 5761316431
> > > tx_vport_multicast_packets: 6452
> > > tx_vport_multicast_bytes: 643540
> > > rx_vport_broadcast_packets: 22423624
> > > rx_vport_broadcast_bytes: 1390127090
> > > tx_vport_broadcast_packets: 22024
> > > tx_vport_broadcast_bytes: 1321440
> > > rx_vport_rdma_unicast_packets: 0
> > > rx_vport_rdma_unicast_bytes: 0
> > > tx_vport_rdma_unicast_packets: 0
> > > tx_vport_rdma_unicast_bytes: 0
> > > rx_vport_rdma_multicast_packets: 0
> > > rx_vport_rdma_multicast_bytes: 0
> > > tx_vport_rdma_multicast_packets: 0
> > > tx_vport_rdma_multicast_bytes: 0
> > > tx_packets_phy: 172569501577
> > > rx_packets_phy: 142871314588
> > > rx_crc_errors_phy: 0
> > > tx_bytes_phy: 100710212814151
> > > rx_bytes_phy: 187209224289564
> > > tx_multicast_phy: 6452
> > > tx_broadcast_phy: 22024
> > > rx_multicast_phy: 85122933
> > > rx_broadcast_phy: 22423623
> > > rx_in_range_len_errors_phy: 2
> > > rx_out_of_range_len_phy: 0
> > > rx_oversize_pkts_phy: 0
> > > rx_symbol_err_phy: 0
> > > tx_mac_control_phy: 0
> > > rx_mac_control_phy: 0
> > > rx_unsupported_op_phy: 0
> > > rx_pause_ctrl_phy: 0
> > > tx_pause_ctrl_phy: 0
> > > rx_discards_phy: 920161423
> >
> > Ok, this port seem to be suffering more, RX is congested, maybe due
> > to
> > the pcie bottleneck.
>
> Yes this side is receiving more traffic - second port is +10G more tx
>
[...]
> > > Average: 17 0.00 0.00 16.60 0.00 0.00 52.10
> > > 0.00 0.00 0.00 31.30
> > > Average: 18 0.00 0.00 13.90 0.00 0.00 61.20
> > > 0.00 0.00 0.00 24.90
> > > Average: 19 0.00 0.00 9.99 0.00 0.00 70.33
> > > 0.00 0.00 0.00 19.68
> > > Average: 20 0.00 0.00 9.00 0.00 0.00 73.00
> > > 0.00 0.00 0.00 18.00
> > > Average: 21 0.00 0.00 8.70 0.00 0.00 73.90
> > > 0.00 0.00 0.00 17.40
> > > Average: 22 0.00 0.00 15.42 0.00 0.00 58.56
> > > 0.00 0.00 0.00 26.03
> > > Average: 23 0.00 0.00 10.81 0.00 0.00 71.67
> > > 0.00 0.00 0.00 17.52
> > > Average: 24 0.00 0.00 10.00 0.00 0.00 71.80
> > > 0.00 0.00 0.00 18.20
> > > Average: 25 0.00 0.00 11.19 0.00 0.00 71.13
> > > 0.00 0.00 0.00 17.68
> > > Average: 26 0.00 0.00 11.00 0.00 0.00 70.80
> > > 0.00 0.00 0.00 18.20
> > > Average: 27 0.00 0.00 10.01 0.00 0.00 69.57
> > > 0.00 0.00 0.00 20.42
> >
> > The numa cores are not at 100% util, you have around 20% of idle on
> > each one.
>
> Yes - no 100% cpu - but the difference between 80% and 100% is like
> push
> aditional 1-2Gbit/s
>
yes but, it doens't look like the bottleneck is the cpu, although it is
close to be :)..
> >
> > > Average: 28 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 33 0.00 0.00 3.90 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 96.10
> > > Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 36 0.10 0.00 0.20 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.70
> > > Average: 37 0.20 0.00 0.30 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.50
> > > Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 100.00
> > > Average: 39 0.00 0.00 2.60 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 97.40
> > > Average: 40 0.00 0.00 0.90 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.10
> > > Average: 41 0.10 0.00 0.50 0.00 0.00 0.00
> > > 0.00
> > > 0.00 0.00 99.40
> > > Average: 42 0.00 0.00 9.91 0.00 0.00 70.67
> > > 0.00 0.00 0.00 19.42
> > > Average: 43 0.00 0.00 15.90 0.00 0.00 57.50
> > > 0.00 0.00 0.00 26.60
> > > Average: 44 0.00 0.00 12.20 0.00 0.00 66.20
> > > 0.00 0.00 0.00 21.60
> > > Average: 45 0.00 0.00 12.00 0.00 0.00 67.50
> > > 0.00 0.00 0.00 20.50
> > > Average: 46 0.00 0.00 12.90 0.00 0.00 65.50
> > > 0.00 0.00 0.00 21.60
> > > Average: 47 0.00 0.00 14.59 0.00 0.00 60.84
> > > 0.00 0.00 0.00 24.58
> > > Average: 48 0.00 0.00 13.59 0.00 0.00 61.74
> > > 0.00 0.00 0.00 24.68
> > > Average: 49 0.00 0.00 18.36 0.00 0.00 53.29
> > > 0.00 0.00 0.00 28.34
> > > Average: 50 0.00 0.00 15.32 0.00 0.00 58.86
> > > 0.00 0.00 0.00 25.83
> > > Average: 51 0.00 0.00 17.60 0.00 0.00 55.20
> > > 0.00 0.00 0.00 27.20
> > > Average: 52 0.00 0.00 15.92 0.00 0.00 56.06
> > > 0.00 0.00 0.00 28.03
> > > Average: 53 0.00 0.00 13.00 0.00 0.00 62.30
> > > 0.00 0.00 0.00 24.70
> > > Average: 54 0.00 0.00 13.20 0.00 0.00 61.50
> > > 0.00 0.00 0.00 25.30
> > > Average: 55 0.00 0.00 14.59 0.00 0.00 58.64
> > > 0.00 0.00 0.00 26.77
> > >
> > >
> > > ethtool -k enp175s0f0
> > > Features for enp175s0f0:
> > > rx-checksumming: on
> > > tx-checksumming: on
> > > tx-checksum-ipv4: on
> > > tx-checksum-ip-generic: off [fixed]
> > > tx-checksum-ipv6: on
> > > tx-checksum-fcoe-crc: off [fixed]
> > > tx-checksum-sctp: off [fixed]
> > > scatter-gather: on
> > > tx-scatter-gather: on
> > > tx-scatter-gather-fraglist: off [fixed]
> > > tcp-segmentation-offload: on
> > > tx-tcp-segmentation: on
> > > tx-tcp-ecn-segmentation: off [fixed]
> > > tx-tcp-mangleid-segmentation: off
> > > tx-tcp6-segmentation: on
> > > udp-fragmentation-offload: off
> > > generic-segmentation-offload: on
> > > generic-receive-offload: on
> > > large-receive-offload: off [fixed]
> > > rx-vlan-offload: on
> > > tx-vlan-offload: on
> > > ntuple-filters: off
> > > receive-hashing: on
> > > highdma: on [fixed]
> > > rx-vlan-filter: on
> > > vlan-challenged: off [fixed]
> > > tx-lockless: off [fixed]
> > > netns-local: off [fixed]
> > > tx-gso-robust: off [fixed]
> > > tx-fcoe-segmentation: off [fixed]
> > > tx-gre-segmentation: on
> > > tx-gre-csum-segmentation: on
> > > tx-ipxip4-segmentation: off [fixed]
> > > tx-ipxip6-segmentation: off [fixed]
> > > tx-udp_tnl-segmentation: on
> > > tx-udp_tnl-csum-segmentation: on
> > > tx-gso-partial: on
> > > tx-sctp-segmentation: off [fixed]
> > > tx-esp-segmentation: off [fixed]
> > > tx-udp-segmentation: on
> > > fcoe-mtu: off [fixed]
> > > tx-nocache-copy: off
> > > loopback: off [fixed]
> > > rx-fcs: off
> > > rx-all: off
> > > tx-vlan-stag-hw-insert: on
> > > rx-vlan-stag-hw-parse: off [fixed]
> > > rx-vlan-stag-filter: on [fixed]
> > > l2-fwd-offload: off [fixed]
> > > hw-tc-offload: off
> > > esp-hw-offload: off [fixed]
> > > esp-tx-csum-hw-offload: off [fixed]
> > > rx-udp_tunnel-port-offload: on
> > > tls-hw-tx-offload: off [fixed]
> > > tls-hw-rx-offload: off [fixed]
> > > rx-gro-hw: off [fixed]
> > > tls-hw-record: off [fixed]
> > >
> > > ethtool -c enp175s0f0
> > > Coalesce parameters for enp175s0f0:
> > > Adaptive RX: off TX: on
> > > stats-block-usecs: 0
> > > sample-interval: 0
> > > pkt-rate-low: 0
> > > pkt-rate-high: 0
> > > dmac: 32703
> > >
> > > rx-usecs: 256
> > > rx-frames: 128
> > > rx-usecs-irq: 0
> > > rx-frames-irq: 0
> > >
> > > tx-usecs: 8
> > > tx-frames: 128
> > > tx-usecs-irq: 0
> > > tx-frames-irq: 0
> > >
> > > rx-usecs-low: 0
> > > rx-frame-low: 0
> > > tx-usecs-low: 0
> > > tx-frame-low: 0
> > >
> > > rx-usecs-high: 0
> > > rx-frame-high: 0
> > > tx-usecs-high: 0
> > > tx-frame-high: 0
> > >
> > > ethtool -g enp175s0f0
> > > Ring parameters for enp175s0f0:
> > > Pre-set maximums:
> > > RX: 8192
> > > RX Mini: 0
> > > RX Jumbo: 0
> > > TX: 8192
> > > Current hardware settings:
> > > RX: 4096
> > > RX Mini: 0
> > > RX Jumbo: 0
> > > TX: 4096
> > >
> > >
> > >
> > >
> > >
> > >
>
> Also changed a little coalesce params - and best for this config are:
> ethtool -c enp175s0f0
> Coalesce parameters for enp175s0f0:
> Adaptive RX: off TX: off
> stats-block-usecs: 0
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
> dmac: 32573
>
> rx-usecs: 40
> rx-frames: 128
> rx-usecs-irq: 0
> rx-frames-irq: 0
>
> tx-usecs: 8
> tx-frames: 8
> tx-usecs-irq: 0
> tx-frames-irq: 0
>
> rx-usecs-low: 0
> rx-frame-low: 0
> tx-usecs-low: 0
> tx-frame-low: 0
>
> rx-usecs-high: 0
> rx-frame-high: 0
> tx-usecs-high: 0
> tx-frame-high: 0
>
>
> Less drops on RX side - and more pps in overall forwarded.
>
how much improvement ? maybe we can improve our adaptive rx coal to be
efficient for this work load.
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Saeed Mahameed @ 2018-11-01 20:23 UTC (permalink / raw)
To: aaron.lu@intel.com, brouer@redhat.com
Cc: pstaszewski@itcare.pl, eric.dumazet@gmail.com,
netdev@vger.kernel.org, Tariq Toukan, ilias.apalodimas@linaro.org,
yoel@kviknet.dk, mgorman@techsingularity.net
In-Reply-To: <20181101152716.GA13895@intel.com>
On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> wrote:
> ... ...
> > Section copied out:
> >
> > mlx5e_poll_tx_cq
> > |
> > --16.34%--napi_consume_skb
> > |
> > |--12.65%--__free_pages_ok
> > | |
> > | --11.86%--free_one_page
> > | |
> > | |--10.10%
> > --queued_spin_lock_slowpath
> > | |
> > | --0.65%--_raw_spin_lock
>
> This callchain looks like it is freeing higher order pages than order
> 0:
> __free_pages_ok is only called for pages whose order are bigger than
> 0.
mlx5 rx uses only order 0 pages, so i don't know where these high order
tx SKBs are coming from..
>
> > |
> > |--1.55%--page_frag_free
> > |
> > --1.44%--skb_release_data
> >
> >
> > Let me explain what (I think) happens. The mlx5 driver RX-page
> > recycle
> > mechanism is not effective in this workload, and pages have to go
> > through the page allocator. The lock contention happens during
> > mlx5
> > DMA TX completion cycle. And the page allocator cannot keep up at
> > these speeds.
> >
> > One solution is extend page allocator with a bulk free API. (This
> > have
> > been on my TODO list for a long time, but I don't have a
> > micro-benchmark that trick the driver page-recycle to fail). It
> > should
> > fit nicely, as I can see that kmem_cache_free_bulk() does get
> > activated (bulk freeing SKBs), which means that DMA TX completion
> > do
> > have a bulk of packets.
> >
> > We can (and should) also improve the page recycle scheme in the
> > driver.
> > After LPC, I have a project with Tariq and Ilias (Cc'ed) to improve
> > the
> > page_pool, and we will (attempt) to generalize this, for both high-
> > end
> > mlx5 and more low-end ARM64-boards (macchiatobin and espressobin).
> >
> > The MM-people is in parallel working to improve the performance of
> > order-0 page returns. Thus, the explicit page bulk free API might
> > actually become less important. I actually think (Cc.) Aaron have
> > a
> > patchset he would like you to test, which removes the (zone->)lock
> > you hit in free_one_page().
>
> Thanks Jesper.
>
> Yes, the said patchset is in this branch:
> https://github.com/aaronlu/linux no_merge_cluster_alloc_4.19-rc5
>
> But as I said above, I think the lock contention here is for
> order > 0 pages so my current patchset will not work here,
> unfortunately.
>
> BTW, Mel Gorman has suggested an alternative way to improve page
> allocator's scalability and I'm working on it right now, it will
> improve page allocator's scalability for all order pages. I might be
> able to post it some time next week, will CC all of you when it's
> ready.
^ permalink raw reply
* Re: Help with the BPF verifier
From: Edward Cree @ 2018-11-01 20:05 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Yonghong Song
Cc: Daniel Borkmann, Jiri Olsa, Martin Lau, Alexei Starovoitov,
Linux Networking Development Mailing List
In-Reply-To: <20181101185217.GA20495@kernel.org>
On 01/11/18 18:52, Arnaldo Carvalho de Melo wrote:
> R0=inv(id=0) R1=inv6 R2=inv6 R3=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv64 R10=fp0,call_-1
> 15: (b7) r2 = 0
> 16: (63) *(u32 *)(r10 -260) = r2
> 17: (67) r1 <<= 32
> 18: (77) r1 >>= 32
> 19: (67) r1 <<= 3
> 20: (bf) r2 = r6
> 21: (0f) r2 += r1
> 22: (79) r3 = *(u64 *)(r2 +16)
> R2 invalid mem access 'inv'
I wonder if you could run this with verifier log level 2? (I'm not sure how
you would go about plumbing that through the perf tooling.) It seems very
odd that it ends up with R2=inv, and I'm wondering whether R1 becomes unknown
during the shifts or whether the addition in insn 21 somehow produces the
unknown-ness. (I know we used to have a thing[1] where doing ptr += K and
then also having an offset in the LDX produced an error about
ptr+const+const, but that seems to have been fixed at some point.)
Note however that even if we get past this, R1 at this point holds 6, so it
looks like the verifier is walking the impossible path where we're inside the
'if' even though filename_arg = 6. This is a (slightly annoying) verifier
limitation, that it walks paths with impossible combinations of constraints
(we've previously had cases where assertions in the verifier would blow up
because of this, e.g. registers with max_val less than min_val). So if the
check_ctx_access() is going to worry about whether you're off the end of the
array (I'm not sure what your program type is and thus which is_valid_access
callback is involved), then it'll complain about this.
If filename_arg came from some external source you'd have a different
problem, because then it would have a totally unknown value, that on entering
the 'if' becomes "unknown but < 6", which is still too variable to have as
the offset of a ctx access. Those have to be at a known constant offset, so
that we can determine the type of the returned value.
As a way to fix this, how about [UNTESTED!]:
const void *filename_arg = NULL;
/* ... */
switch (augmented_args.args.syscall_nr) {
case SYS_OPEN: filename_arg = args->args[0]; break;
case SYS_OPENAT: filename_arg = args->args[1]; break;
}
/* ... */
if (filename_arg) {
/* stuff */
blah = probe_read_str(/* ... */, filename_arg);
} else {
/* the other stuff */
}
That way, you're only ever dealing in constant pointers (although judging by
an old thread I found[1] about ptr+const+const, the compiler might decide to
make some optimisations that end up looking like your existing code).
As for what you want to do with the index coming from userspace, the verifier
will not like that at all, as mentioned above, so I think you'll need to do
something like:
switch (filename_arg_from_userspace) {
case 0: filename_arg = args->args[0]; break;
case 1: filename_arg = args->args[1]; break;
/* etc */
default: filename_arg = NULL;
}
thus ensuring that you only ever have ctx pointers with constant offsets.
-Ed
[1]: https://lists.iovisor.org/g/iovisor-dev/topic/21386327#1302
^ permalink raw reply
* Re: [PATCH iproute2] ip rule: Require at least one argument for add
From: Stephen Hemminger @ 2018-11-01 19:54 UTC (permalink / raw)
To: David Ahern; +Cc: netdev, David Ahern
In-Reply-To: <20181030205905.14209-1-dsahern@kernel.org>
On Tue, 30 Oct 2018 13:59:05 -0700
David Ahern <dsahern@kernel.org> wrote:
> From: David Ahern <dsahern@gmail.com>
>
> 'ip rule add' with no additional arguments just adds another rule
> for the main table - which exists by default. Require at least
> 1 argument similar to delete.
>
> Signed-off-by: David Ahern <dsahern@gmail.com>
Applied these two
^ permalink raw reply
* Re: [PATCH net] qmi_wwan: Support dynamic config on Quectel EP06
From: Kristian Evensen @ 2018-11-01 19:37 UTC (permalink / raw)
To: Network Development, Bjørn Mork
In-Reply-To: <CAKfDRXj2RuGZ-JpGN71kBT_wZvvGihcZVpbCU-Z4HnPW5Lt=TA@mail.gmail.com>
On Thu, Nov 1, 2018 at 8:30 PM Kristian Evensen
<kristian.evensen@gmail.com> wrote:
>
> On Thu, Nov 1, 2018 at 6:40 PM Kristian Evensen
> <kristian.evensen@gmail.com> wrote:
> >
> > Hi,
> >
> > On Sat, Sep 8, 2018 at 1:50 PM Kristian Evensen
> > <kristian.evensen@gmail.com> wrote:
> > > Quectel EP06 (and EM06/EG06) supports dynamic configuration of USB
> > > interfaces, without the device changing VID/PID or configuration number.
> > > When the configuration is updated and interfaces are added/removed, the
> > > interface numbers change. This means that the current code for matching
> > > EP06 does not work.
> >
> > Would it be possible to have this patch added to stable? I checked
> > both the 4.14-tree and the stable queue, but could not find the
> > patch/upstream commit.
>
> Please ignore this request. I discovered patch does not apply to 4.14,
> so I will do a manual backport and send to stable.
Blah, it is clearly not my day today. I discovered that my 4.14 build
directory was dirty and that the patch applies fined on top of
4.14.78. Sorry about the extra noise and please do not ignore my
request for stable :)
BR,
Kristian
^ permalink raw reply
* Re: [PATCH net] qmi_wwan: Support dynamic config on Quectel EP06
From: Kristian Evensen @ 2018-11-01 19:30 UTC (permalink / raw)
To: Network Development, Bjørn Mork
In-Reply-To: <CAKfDRXjpSEOqbKXFiB7E4p_K5oPbJpvfoQqG7gzTC6BaOyFr8A@mail.gmail.com>
On Thu, Nov 1, 2018 at 6:40 PM Kristian Evensen
<kristian.evensen@gmail.com> wrote:
>
> Hi,
>
> On Sat, Sep 8, 2018 at 1:50 PM Kristian Evensen
> <kristian.evensen@gmail.com> wrote:
> > Quectel EP06 (and EM06/EG06) supports dynamic configuration of USB
> > interfaces, without the device changing VID/PID or configuration number.
> > When the configuration is updated and interfaces are added/removed, the
> > interface numbers change. This means that the current code for matching
> > EP06 does not work.
>
> Would it be possible to have this patch added to stable? I checked
> both the 4.14-tree and the stable queue, but could not find the
> patch/upstream commit.
Please ignore this request. I discovered patch does not apply to 4.14,
so I will do a manual backport and send to stable.
BR,
Kristian
^ permalink raw reply
* Re: Help with the BPF verifier
From: David Miller @ 2018-11-01 19:28 UTC (permalink / raw)
To: acme; +Cc: yhs, daniel, jolsa, kafai, alexei.starovoitov, netdev
In-Reply-To: <20181101191310.GB20495@kernel.org>
From: Arnaldo Carvalho de Melo <acme@kernel.org>
Date: Thu, 1 Nov 2018 16:13:10 -0300
> Nope... this is inside an if:
>
> if (filename_arg <= 5) {
> augmented_args.filename.reserved = 0;
> augmented_args.filename.size = probe_read_str(&augmented_args.filename.value,
> sizeof(augmented_args.filename.value),
> (const void *)args->args[filename_arg]);
> if (augmented_args.filename.size < sizeof(augmented_args.filename.value)) {
> len -= sizeof(augmented_args.filename.value) - augmented_args.filename.size;
> len &= sizeof(augmented_args.filename.value) - 1;
> }
> } else {
>
> I use 6 to mean "hey, this syscall doesn't have any string argument, don't
> bother with it".
Really weird. And it's unsigned so I can't imagine it wants you to
check that it's >= 0...
Maybe Deniel or someone else can figure it out.
^ permalink raw reply
* RE: [Intel-wired-lan] [PATCH][next] igc: fix error return handling from call to netif_set_real_num_tx_queues
From: Brown, Aaron F @ 2018-11-01 19:26 UTC (permalink / raw)
To: Colin King, Neftin, Sasha, Kirsher, Jeffrey T, David S . Miller,
intel-wired-lan@lists.osuosl.org
Cc: netdev@vger.kernel.org, kernel-janitors@vger.kernel.org
In-Reply-To: <20181019181615.27528-1-colin.king@canonical.com>
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Colin King
> Sent: Friday, October 19, 2018 11:16 AM
> To: Neftin, Sasha <sasha.neftin@intel.com>; Kirsher, Jeffrey T
> <jeffrey.t.kirsher@intel.com>; David S . Miller <davem@davemloft.net>;
> intel-wired-lan@lists.osuosl.org
> Cc: netdev@vger.kernel.org; kernel-janitors@vger.kernel.org
> Subject: [Intel-wired-lan] [PATCH][next] igc: fix error return handling from
> call to netif_set_real_num_tx_queues
>
> From: Colin Ian King <colin.king@canonical.com>
>
> The call to netif_set_real_num_tx_queues is not assigning the error
> return to variable err even though the next line checks err for an
> error. Fix this by adding the missing err assignment.
>
> Detected by CoverityScan, CID#1474551 ("Logically dead code")
>
> Fixes: 3df25e4c1e66 ("igc: Add interrupt support")
> Signed-off-by: Colin Ian King <colin.king@canonical.com>
> ---
> drivers/net/ethernet/intel/igc/igc_main.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
Tested-by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [PATCH net-next] igc: Remove set but not used variable 'pci_using_dac'
From: Brown, Aaron F @ 2018-11-01 19:24 UTC (permalink / raw)
To: YueHaibing, Kirsher, Jeffrey T, David S. Miller, Neftin, Sasha
Cc: intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org,
kernel-janitors@vger.kernel.org
In-Reply-To: <1539953291-189201-1-git-send-email-yuehaibing@huawei.com>
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of YueHaibing
> Sent: Friday, October 19, 2018 5:48 AM
> To: Kirsher, Jeffrey T <jeffrey.t.kirsher@intel.com>; David S. Miller
> <davem@davemloft.net>; Neftin, Sasha <sasha.neftin@intel.com>
> Cc: YueHaibing <yuehaibing@huawei.com>; intel-wired-lan@lists.osuosl.org;
> netdev@vger.kernel.org; kernel-janitors@vger.kernel.org
> Subject: [PATCH net-next] igc: Remove set but not used variable
> 'pci_using_dac'
>
> Fixes gcc '-Wunused-but-set-variable' warning:
>
> drivers/net/ethernet/intel/igc/igc_main.c: In function 'igc_probe':
> drivers/net/ethernet/intel/igc/igc_main.c:3535:11: warning:
> variable 'pci_using_dac' set but not used [-Wunused-but-set-variable]
>
> It never used since introduction in commit
> d89f88419f99 ("igc: Add skeletal frame for Intel(R) 2.5G Ethernet Controller
> support")
>
> Signed-off-by: YueHaibing <yuehaibing@huawei.com>
> ---
> drivers/net/ethernet/intel/igc/igc_main.c | 5 +----
> 1 file changed, 1 insertion(+), 4 deletions(-)
>
Tested-by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* RE: [PATCH net-next] igc: Remove set but not used variables 'ctrl_ext, link_mode'
From: Brown, Aaron F @ 2018-11-01 19:24 UTC (permalink / raw)
To: YueHaibing, Kirsher, Jeffrey T, Neftin, Sasha
Cc: intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org,
kernel-janitors@vger.kernel.org
In-Reply-To: <1539952830-187358-1-git-send-email-yuehaibing@huawei.com>
> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of YueHaibing
> Sent: Friday, October 19, 2018 5:41 AM
> To: Kirsher, Jeffrey T <jeffrey.t.kirsher@intel.com>; Neftin, Sasha
> <sasha.neftin@intel.com>
> Cc: YueHaibing <yuehaibing@huawei.com>; intel-wired-lan@lists.osuosl.org;
> netdev@vger.kernel.org; kernel-janitors@vger.kernel.org
> Subject: [PATCH net-next] igc: Remove set but not used variables 'ctrl_ext,
> link_mode'
>
> Fixes gcc '-Wunused-but-set-variable' warning:
>
> drivers/net/ethernet/intel/igc/igc_base.c: In function
> 'igc_init_phy_params_base':
> drivers/net/ethernet/intel/igc/igc_base.c:240:6: warning:
> variable 'ctrl_ext' set but not used [-Wunused-but-set-variable]
> u32 ctrl_ext;
>
> drivers/net/ethernet/intel/igc/igc_base.c: In function
> 'igc_get_invariants_base':
> drivers/net/ethernet/intel/igc/igc_base.c:290:6: warning:
> variable 'link_mode' set but not used [-Wunused-but-set-variable]
> u32 link_mode = 0;
>
> It never used since introduction in
> commit c0071c7aa5fe ("igc: Add HW initialization code")
>
> Signed-off-by: YueHaibing <yuehaibing@huawei.com>
> ---
> I'm not sure that reading IGC_CTRL_EXT is necessary.
> ---
> drivers/net/ethernet/intel/igc/igc_base.c | 8 --------
> 1 file changed, 8 deletions(-)
>
Tested-by: Aaron Brown <aaron.f.brown@intel.com>
^ permalink raw reply
* Re: [PATCH bpf 1/4] bpf: fix partial copy of map_ptr when dst is scalar
From: Arnaldo Carvalho de Melo @ 2018-11-01 19:18 UTC (permalink / raw)
To: Edward Cree; +Cc: Daniel Borkmann, ast, netdev
In-Reply-To: <eb077319-4b9a-d48d-b467-fb11301c4470@solarflare.com>
Em Thu, Nov 01, 2018 at 07:17:29PM +0000, Edward Cree escreveu:
> On 31/10/18 23:05, Daniel Borkmann wrote:
> > ALU operations on pointers such as scalar_reg += map_value_ptr are
> > handled in adjust_ptr_min_max_vals(). Problem is however that map_ptr
> > and range in the register state share a union, so transferring state
> > through dst_reg->range = ptr_reg->range is just buggy as any new
> > map_ptr in the dst_reg is then truncated (or null) for subsequent
> > checks. Fix this by adding a raw member and use it for copying state
> > over to dst_reg.
> >
> > Fixes: f1174f77b50c ("bpf/verifier: rework value tracking")
> > Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> > Cc: Edward Cree <ecree@solarflare.com>
> > Acked-by: Alexei Starovoitov <ast@kernel.org>
> > ---
> Acked-by: Edward Cree <ecree@solarflare.com>
> (though I apparently missed the 63-minute window to hit the git record...)
Those guys are fast! :-)
- Arnaldo
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox