* [net 1/1] tipc: fix link re-establish failure
From: Jon Maloy @ 2018-11-10 22:30 UTC (permalink / raw)
To: netdev, davem
Cc: gordan.mihaljevic, tung.q.nguyen, hoang.h.le, jon.maloy, maloy,
xinl, ying.xue, tipc-discussion
When a link failure is detected locally, the link is reset, the flag
link->in_session is set to false, and a RESET_MSG with the 'stopping'
bit set is sent to the peer.
The purpose of this bit is to inform the peer that this endpoint just
is going down, and that the peer should handle the reception of this
particular RESET message as a local failure. This forces the peer to
accept another RESET or ACTIVATE message from this endpoint before it
can re-establish the link. This again is necessary to ensure that
link session numbers are properly exchanged before the link comes up
again.
If a failure is detected locally at the same time at the peer endpoint
this will do the same, which is also a correct behavior.
However, when receiving such messages, the endpoints will not
distinguish between 'stopping' RESETs and ordinary ones when it comes
to updating session numbers. Both endpoints will copy the received
session number and set their 'in_session' flags to true at the
reception, while they are still expecting another RESET from the
peer before they can go ahead and re-establish. This is contradictory,
since, after applying the validation check referred to below, the
'in_session' flag will cause rejection of all such messages, and the
link will never come up again.
We now fix this by not only handling received RESET/STOPPING messages
as a local failure, but also by omitting to set a new session number
and the 'in_session' flag in such cases.
Fixes: 7ea817f4e832 ("tipc: check session number before accepting link protocol messages")
Signed-off-by: Jon Maloy <jon.maloy@ericsson.com>
---
net/tipc/link.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/net/tipc/link.c b/net/tipc/link.c
index 201c3b5..836727e 100644
--- a/net/tipc/link.c
+++ b/net/tipc/link.c
@@ -1594,14 +1594,17 @@ static int tipc_link_proto_rcv(struct tipc_link *l, struct sk_buff *skb,
if (in_range(peers_prio, l->priority + 1, TIPC_MAX_LINK_PRI))
l->priority = peers_prio;
- /* ACTIVATE_MSG serves as PEER_RESET if link is already down */
- if (msg_peer_stopping(hdr))
+ /* If peer is going down we want full re-establish cycle */
+ if (msg_peer_stopping(hdr)) {
rc = tipc_link_fsm_evt(l, LINK_FAILURE_EVT);
- else if ((mtyp == RESET_MSG) || !link_is_up(l))
+ break;
+ }
+ /* ACTIVATE_MSG serves as PEER_RESET if link is already down */
+ if (mtyp == RESET_MSG || !link_is_up(l))
rc = tipc_link_fsm_evt(l, LINK_PEER_RESET_EVT);
/* ACTIVATE_MSG takes up link if it was already locally reset */
- if ((mtyp == ACTIVATE_MSG) && (l->state == LINK_ESTABLISHING))
+ if (mtyp == ACTIVATE_MSG && l->state == LINK_ESTABLISHING)
rc = TIPC_LINK_UP_EVT;
l->peer_session = msg_session(hdr);
--
1.8.3.1
^ permalink raw reply related
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 22:19 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <20181110230630.0daeba8e@redhat.com>
W dniu 10.11.2018 o 23:06, Jesper Dangaard Brouer pisze:
> On Sat, 10 Nov 2018 20:56:02 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
>>>
>>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
>>>> <pstaszewski@itcare.pl> wrote:
>>>>
>>>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>>>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>>>>> limit is the same :)
>>>>>> But also after reaching 60Gbit/60Gbit
>>>>>>
>>>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>>>> input: /proc/net/dev type: rate
>>>>>> - iface Rx Tx Total
>>>>>> ===================================================================
>>>>>>
>>>>>>
>>>>>> enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
>>>>>> enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
>>>>>> -------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>>>>> Today reached 65/65Gbit/s
>>>>>
>>>>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>>>>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
>>>> This is weird!
>>>>
>>>> How do you see / measure these drops?
>>> Simple icmp test like ping -i 0.1
>>> And im testing by icmp management ip address on vlan that is attacked
>>> to one NIC (the side that is more stressed with RX)
>>> And another icmp test is forward thru this router - host behind it
>>>
>>> Both measurements shows same loss ratio from 0.1 to 0.5% after
>>> reaching ~45Gbit/s RX side - depends how much RX side is pushed drops
>>> vary between 0.1 to 0.5 - even 0.6%:)
>>>
> Okay good to know, you use an external measurement for this. I do
> think packets are getting dropped by the NIC.
>
>>>>> So checked other stats.
>>>>> softnet_stats shows average 1k squeezed per sec:
>>>> Is below output the raw counters? not per sec?
>>>>
>>>> It would be valuable to see the per sec stats instead...
>>>> I use this tool:
>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
>> CPU total/sec dropped/sec squeezed/sec collision/sec rx_rps/sec flow_limit/sec
>> CPU:00 0 0 0 0 0 0
> [...]
>> CPU:13 0 0 0 0 0 0
>> CPU:14 485538 0 43 0 0 0
>> CPU:15 474794 0 51 0 0 0
>> CPU:16 449322 0 41 0 0 0
>> CPU:17 476420 0 46 0 0 0
>> CPU:18 440436 0 38 0 0 0
>> CPU:19 501499 0 49 0 0 0
>> CPU:20 459468 0 49 0 0 0
>> CPU:21 438928 0 47 0 0 0
>> CPU:22 468983 0 40 0 0 0
>> CPU:23 446253 0 47 0 0 0
>> CPU:24 451909 0 46 0 0 0
>> CPU:25 479373 0 55 0 0 0
>> CPU:26 467848 0 49 0 0 0
>> CPU:27 453153 0 51 0 0 0
>> CPU:28 0 0 0 0 0 0
> [...]
>> CPU:40 0 0 0 0 0 0
>> CPU:41 0 0 0 0 0 0
>> CPU:42 466853 0 43 0 0 0
>> CPU:43 453059 0 54 0 0 0
>> CPU:44 363219 0 34 0 0 0
>> CPU:45 353632 0 38 0 0 0
>> CPU:46 371618 0 40 0 0 0
>> CPU:47 350518 0 46 0 0 0
>> CPU:48 397544 0 40 0 0 0
>> CPU:49 364873 0 38 0 0 0
>> CPU:50 383630 0 38 0 0 0
>> CPU:51 358771 0 39 0 0 0
>> CPU:52 372547 0 38 0 0 0
>> CPU:53 372882 0 36 0 0 0
>> CPU:54 366244 0 43 0 0 0
>> CPU:55 365886 0 39 0 0 0
>>
>> Summed: 11835201 0 1217 0 0 0
>
> Do notice, the per CPU squeeze is not too large.
Yes - but im searching invisible thing now :) something invisible is
slowing down packet processing :)
So trying to find any counter that have something to do with packet
processing.
> The summed 11.8 Mpps is a little high compared to:
>
> Ethtool(enp216s0) stat: 4971677 (4,971,677) <= rx_packets /sec
> Ethtool(enp175s0) stat: 3717148 (3,717,148) <= rx_packets /sec
> Sum: 3717148+4971677 = 8688825 (8,688,825)
Yes i was mentioning this that stats from /net/dev for nics are weird if
u compare them to ethtool - there are big differences for mellanox nic's
Especially with packets/s
For example when i change
- cqe to compress i have more interrupts - same as more packets/s - but
same bw
- change ring settings - like half hour before - changed TX fing from
4096 to 256 and i have less interrupts and less packets but more
bandwidth... weird...
Cause in normal traffic more packets/s need to be more bandwidth - if
average frame is 500-600 if I gain like 1M+pps - then it should mean in
average +5/6Gbit/s more
But it looks like it is more comparable to number of interrupts not
number of packets.
>
> [...]
>>>>> Remember those tests are now on two separate connectx5 connected to
>>>>> two separate pcie x16 gen 3.0
>>>> That is strange... I still suspect some HW NIC issue, can you provide
>>>> ethtool stats info via tool:
>>>>
>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>>>
>>>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>>>>
>>>> The tool remove zero-stats counters and report per sec stats. It makes
>>>> it easier to spot that is relevant for the given workload.
>>> yes mlnx have just too many counters that are always 0 for my case :)
>>> Will try this also
>>>
>> But still alot of non 0 counters
>> Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
>> Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
> [...]
>
> I have copied the stats over in another document so I can better looks
> at it... and I've found some interesting stats.
>
> E.g. we can see that the NIC hardware is dropping packets.
>
> RX-drops on enp175s0:
>
> (enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
> (enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
> -218308971 ( -218,308,971) Dropped bytes /sec
>
> (enp175s0) stat: 139602 ( 139,602) <= rx_discards_phy /sec
>
> (enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
> (enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy /sec
> -145272 ( -145,272) Dropped packets /sec
>
>
> RX-drops on enp216s0 is less:
>
> (enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
> (enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
> -41288962 ( -41,288,962) Dropped bytes /sec
>
> (enp216s0) stat: 464 (464) <= rx_discards_phy /sec
>
> (enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
> (enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy /sec
> -3886 ( -3,886) Dropped packets /sec
>
^ permalink raw reply
* Re: SACK compression patch causing performance drop
From: Eric Dumazet @ 2018-11-10 22:07 UTC (permalink / raw)
To: Eric Dumazet, Jean-Louis Dupond, netdev, edumazet
In-Reply-To: <cdbdedbf-bd48-5e3f-465a-81929e7db27f@gmail.com>
On 11/08/2018 07:14 AM, Eric Dumazet wrote:
>
>
> On 11/08/2018 12:23 AM, Jean-Louis Dupond wrote:
>> Hi,
>>
>> Was somebody able to check this?
>> Really think this should be fixed :)
>>
>> Thanks
>> Jean-Louis
>>
>
> I somehow missed this email.
>
> Packet captures might help, please send me
>
> tcpdump -s 128 -i ethX -w sack.pcap
>
> of some samples with or without the sack compression enabled.
Thanks for the traces.
It seems one of your host (A in following traces) is advertising SACK support
but does not really react to incoming SACK information. (It only reacts to the ACK number)
Only receiving consecutive DUPACK triggers a fast retransmit.
With Sack compression enabled, we can see sender resorts to RTO for recovery
01:04:20.609980 IP A > B: Flags [.], seq 133901705:133946445, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 44740
01:04:20.610016 IP A > B: Flags [.], seq 133946445:133963069, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 16624
01:04:20.610027 IP A > B: Flags [.], seq 133963069:133977845, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 14776
01:04:20.610035 IP A > B: Flags [.], seq 133977845:133986793, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 8948
01:04:20.610055 IP B > A: Flags [.], ack 133812253, win 51011, options [nop,nop,TS val 2206848846 ecr 1112736761,nop,nop,sack 1 {133946445:133986793}], length 0
01:04:20.610074 IP A > B: Flags [.], seq 133986793:134016297, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 29504
01:04:20.610118 IP B > A: Flags [.], ack 133812253, win 51011, options [nop,nop,TS val 2206848846 ecr 1112736761,nop,nop,sack 1 {133946445:134016297}], length 0
01:04:20.819665 IP A > B: Flags [.], seq 133812253:133821201, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 8948
01:04:20.819767 IP B > A: Flags [.], ack 133821201, win 48774, options [nop,nop,TS val 2206849056 ecr 1112736761,nop,nop,sack 1 {133946445:134016297}], length 0
01:04:20.819924 IP A > B: Flags [.], seq 133821201:133839097, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 17896
01:04:20.819966 IP B > A: Flags [.], ack 133839097, win 44300, options [nop,nop,TS val 2206849056 ecr 1112736761,nop,nop,sack 1 {133946445:134016297}], length 0
01:04:20.820134 IP A > B: Flags [.], seq 133839097:133874889, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 35792
01:04:20.820185 IP B > A: Flags [.], ack 133874889, win 40896, options [nop,nop,TS val 2206849056 ecr 1112736761,nop,nop,sack 1 {133946445:134016297}], length 0
01:04:20.820296 IP A > B: Flags [.], seq 133874889:133901733, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 26844
01:04:20.820327 IP B > A: Flags [.], ack 133901733, win 40896, options [nop,nop,TS val 2206849057 ecr 1112736761,nop,nop,sack 1 {133946445:134016297}], length 0
01:04:20.820346 IP A > B: Flags [.], seq 133901733:133946473, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 44740
01:04:20.820430 IP B > A: Flags [.], ack 134016297, win 51011, options [nop,nop,TS val 2206849057 ecr 1112736761,nop,nop,sack 1 {133946445:133946473}], length 0
01:04:20.820452 IP A > B: Flags [.], seq 133901733:133946473, ack 7728, win 32768, options [nop,nop,TS val 1112736761 ecr 2206848846], length 44740
01:04:20.820462 IP B > A: Flags [.], ack 134016297, win 51011, options [nop,nop,TS val 2206849057 ecr 1112736761,nop,nop,sack 1 {133901733:133946473}], length 0
Sack compression disabled, no RTO is triggered, fast rtx are working "properly".
00:52:22.329658 IP A > B: Flags [.], seq 2360149:2369069, ack 1680, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 8920
00:52:22.329666 IP A > B: Flags [.], seq 2369069:2378045, ack 1680, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 8976
00:52:22.329694 IP B > A: Flags [.], ack 2280125, win 51011, options [nop,nop,TS val 1636504300 ecr 1112018058], length 0
00:52:22.329716 IP A > B: Flags [.], seq 2378045:2427265, ack 1680, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 49220
00:52:22.329723 IP A > B: Flags [P.], seq 2427265:2433601, ack 1680, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 6336
00:52:22.329728 IP A > B: Flags [.], seq 2433601:2442549, ack 1680, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 8948
00:52:22.329735 IP B > A: Flags [P.], seq 1680:1728, ack 2280125, win 51011, options [nop,nop,TS val 1636504300 ecr 1112018058], length 48
00:52:22.329749 IP B > A: Flags [.], ack 2280125, win 51011, options [nop,nop,TS val 1636504300 ecr 1112018058,nop,nop,sack 1 {2378045:2427265}], length 0
00:52:22.329757 IP B > A: Flags [.], ack 2280125, win 51011, options [nop,nop,TS val 1636504300 ecr 1112018058,nop,nop,sack 1 {2378045:2433601}], length 0
00:52:22.329767 IP B > A: Flags [.], ack 2280125, win 51011, options [nop,nop,TS val 1636504300 ecr 1112018058,nop,nop,sack 1 {2378045:2442549}], length 0
00:52:22.329776 IP A > B: Flags [.], seq 2442549:2448377, ack 1680, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 5828
00:52:22.329785 IP A > B: Flags [.], seq 2448377:2484169, ack 1680, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 35792
00:52:22.329795 IP B > A: Flags [.], ack 2280125, win 51011, options [nop,nop,TS val 1636504300 ecr 1112018058,nop,nop,sack 1 {2378045:2448377}], length 0
00:52:22.329805 IP B > A: Flags [.], ack 2280125, win 51011, options [nop,nop,TS val 1636504300 ecr 1112018058,nop,nop,sack 1 {2378045:2484169}], length 0
00:52:22.330001 IP A > B: Flags [.], seq 2280125:2289125, ack 1728, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 9000
00:52:22.330020 IP B > A: Flags [.], ack 2289125, win 48761, options [nop,nop,TS val 1636504301 ecr 1112018058,nop,nop,sack 1 {2378045:2484169}], length 0
00:52:22.330035 IP A > B: Flags [.], seq 2289073:2333813, ack 1728, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 44740
00:52:22.330067 IP B > A: Flags [.], ack 2333813, win 37589, options [nop,nop,TS val 1636504301 ecr 1112018058,nop,nop,sack 2 {2289073:2289125}{2378045:2484169}], length 0
00:52:22.330085 IP A > B: Flags [.], seq 2333813:2368017, ack 1728, win 32768, options [nop,nop,TS val 1112018058 ecr 1636504300], length 34204
00:52:22.330111 IP B > A: Flags [.], ack 2368017, win 34208, options [nop,nop,TS val 1636504301 ecr 1112018058,nop,nop,sack 1 {2378045:2484169}], length 0
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Jesper Dangaard Brouer @ 2018-11-10 22:06 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev@vger.kernel.org, brouer
In-Reply-To: <69c11e38-da50-15a3-2dfc-bc47ccc134b9@itcare.pl>
On Sat, 10 Nov 2018 20:56:02 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
> >
> >
> > W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> >> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
> >> <pstaszewski@itcare.pl> wrote:
> >>
> >>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> >>>> CPU load is lower than for connectx4 - but it looks like bandwidth
> >>>> limit is the same :)
> >>>> But also after reaching 60Gbit/60Gbit
> >>>>
> >>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> >>>> input: /proc/net/dev type: rate
> >>>> - iface Rx Tx Total
> >>>> ===================================================================
> >>>>
> >>>>
> >>>> enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
> >>>> enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
> >>>> -------------------------------------------------------------------
> >>>>
> >>>>
> >>>> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
> >>> Today reached 65/65Gbit/s
> >>>
> >>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
> >>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
> >> This is weird!
> >>
> >> How do you see / measure these drops?
> >
> > Simple icmp test like ping -i 0.1
> > And im testing by icmp management ip address on vlan that is attacked
> > to one NIC (the side that is more stressed with RX)
> > And another icmp test is forward thru this router - host behind it
> >
> > Both measurements shows same loss ratio from 0.1 to 0.5% after
> > reaching ~45Gbit/s RX side - depends how much RX side is pushed drops
> > vary between 0.1 to 0.5 - even 0.6%:)
> >
Okay good to know, you use an external measurement for this. I do
think packets are getting dropped by the NIC.
> >>> So checked other stats.
> >>> softnet_stats shows average 1k squeezed per sec:
> >> Is below output the raw counters? not per sec?
> >>
> >> It would be valuable to see the per sec stats instead...
> >> I use this tool:
> >> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
> CPU total/sec dropped/sec squeezed/sec collision/sec rx_rps/sec flow_limit/sec
> CPU:00 0 0 0 0 0 0
[...]
> CPU:13 0 0 0 0 0 0
> CPU:14 485538 0 43 0 0 0
> CPU:15 474794 0 51 0 0 0
> CPU:16 449322 0 41 0 0 0
> CPU:17 476420 0 46 0 0 0
> CPU:18 440436 0 38 0 0 0
> CPU:19 501499 0 49 0 0 0
> CPU:20 459468 0 49 0 0 0
> CPU:21 438928 0 47 0 0 0
> CPU:22 468983 0 40 0 0 0
> CPU:23 446253 0 47 0 0 0
> CPU:24 451909 0 46 0 0 0
> CPU:25 479373 0 55 0 0 0
> CPU:26 467848 0 49 0 0 0
> CPU:27 453153 0 51 0 0 0
> CPU:28 0 0 0 0 0 0
[...]
> CPU:40 0 0 0 0 0 0
> CPU:41 0 0 0 0 0 0
> CPU:42 466853 0 43 0 0 0
> CPU:43 453059 0 54 0 0 0
> CPU:44 363219 0 34 0 0 0
> CPU:45 353632 0 38 0 0 0
> CPU:46 371618 0 40 0 0 0
> CPU:47 350518 0 46 0 0 0
> CPU:48 397544 0 40 0 0 0
> CPU:49 364873 0 38 0 0 0
> CPU:50 383630 0 38 0 0 0
> CPU:51 358771 0 39 0 0 0
> CPU:52 372547 0 38 0 0 0
> CPU:53 372882 0 36 0 0 0
> CPU:54 366244 0 43 0 0 0
> CPU:55 365886 0 39 0 0 0
>
> Summed: 11835201 0 1217 0 0 0
Do notice, the per CPU squeeze is not too large.
The summed 11.8 Mpps is a little high compared to:
Ethtool(enp216s0) stat: 4971677 (4,971,677) <= rx_packets /sec
Ethtool(enp175s0) stat: 3717148 (3,717,148) <= rx_packets /sec
Sum: 3717148+4971677 = 8688825 (8,688,825)
[...]
> >>>
> >>> Remember those tests are now on two separate connectx5 connected to
> >>> two separate pcie x16 gen 3.0
> >> That is strange... I still suspect some HW NIC issue, can you provide
> >> ethtool stats info via tool:
> >>
> >> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> >>
> >> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
> >>
> >> The tool remove zero-stats counters and report per sec stats. It makes
> >> it easier to spot that is relevant for the given workload.
> > yes mlnx have just too many counters that are always 0 for my case :)
> > Will try this also
> >
> But still alot of non 0 counters
> Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
> Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
[...]
I have copied the stats over in another document so I can better looks
at it... and I've found some interesting stats.
E.g. we can see that the NIC hardware is dropping packets.
RX-drops on enp175s0:
(enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
(enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
-218308971 ( -218,308,971) Dropped bytes /sec
(enp175s0) stat: 139602 ( 139,602) <= rx_discards_phy /sec
(enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
(enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy /sec
-145272 ( -145,272) Dropped packets /sec
RX-drops on enp216s0 is less:
(enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
(enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
-41288962 ( -41,288,962) Dropped bytes /sec
(enp216s0) stat: 464 (464) <= rx_discards_phy /sec
(enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
(enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy /sec
-3886 ( -3,886) Dropped packets /sec
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 22:04 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <78c37b54-dfd8-d528-1152-abe9ddbb08b2@itcare.pl>
W dniu 10.11.2018 o 22:53, Paweł Staszewski pisze:
>
>
> W dniu 10.11.2018 o 22:01, Jesper Dangaard Brouer pisze:
>> On Sat, 10 Nov 2018 21:02:10 +0100
>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>>> I want you to experiment with:
>>>>
>>>> ethtool --set-priv-flags DEVICE rx_striding_rq off
>>> just checked that previously connectx4 was have thos disabled:
>>> ethtool --show-priv-flags enp175s0f0
>>>
>>> Private flags for enp175s0f0:
>>> rx_cqe_moder : on
>>> tx_cqe_moder : off
>>> rx_cqe_compress : off
>>> rx_striding_rq : off
>>> rx_no_csum_complete: off
>>>
>> The CX4 hardware does not have this feature (p.s. the CX4-Lx does).
>>
>>> So now we are on connectx5 and we have enabled - for sure connectx5
>>> changed cpu load - where i have now max 50/60% cpu where with connectx4
>>> there was sometimes near 100% with same configuration.
>> I (strongly) believe the CPU load was related to the page-alloactor
>> lock congestion, that Aaron fixed.
>>
> Yes i think both - most problems with cpu was due to page-allocator
> problems.
> But also after change connctx4 to connectx5 there is cpu load
> difference - about 10% in total - but yes most of this like 40% is
> cause of Aaron patch :) - rly good job :)
>
>
> Now im messing with ring configuration for connectx5 nics.
> And after reading that paper:
> https://netdevconf.org/2.1/slides/apr6/network-performance/
> 04-amir-RX_and_TX_bulking_v2.pdf
>
> changed from RX:8192 / TX: 4096 to RX:8192 / TX: 256
>
> after this i gain about 5Gbit/s RX and TX traffic and less cpu load....
> before change there was 59/59 Gbit/s
>
> After change there is 64/64 Gbit/s
>
> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> input: /proc/net/dev type: rate
> | iface Rx Tx Total
> ==============================================================================
>
> enp175s0: 44.45 Gb/s 19.69 Gb/s
> 64.14 Gb/s
> enp216s0: 19.69 Gb/s 44.49 Gb/s
> 64.19 Gb/s
> ------------------------------------------------------------------------------
>
> total: 64.14 Gb/s 64.18 Gb/s 128.33 Gb/s
>
>
Also after this change kernel freed some memory... like 500MB
Still squeezed but less with more traffic...
CPU total/sec dropped/sec squeezed/sec
collision/sec rx_rps/sec flow_limit/sec
CPU:00 0 0 0 0
0 0
CPU:01 0 0 0 0
0 0
CPU:02 0 0 0 0
0 0
CPU:03 0 0 0 0
0 0
CPU:04 0 0 0 0
0 0
CPU:05 0 0 0 0
0 0
CPU:06 0 0 0 0
0 0
CPU:07 0 0 0 0
0 0
CPU:08 0 0 0 0
0 0
CPU:09 0 0 0 0
0 0
CPU:10 0 0 0 0
0 0
CPU:11 0 0 0 0
0 0
CPU:12 0 0 0 0
0 0
CPU:13 0 0 0 0
0 0
CPU:14 389270 0 41 0
0 0
CPU:15 375543 0 32 0
0 0
CPU:16 385847 0 22 0
0 0
CPU:17 412293 0 34 0
0 0
CPU:18 401287 0 30 0
0 0
CPU:19 368345 0 30 0
0 0
CPU:20 395452 0 28 0
0 0
CPU:21 374032 0 38 0
0 0
CPU:22 342036 0 32 0
0 0
CPU:23 374773 0 34 0
0 0
CPU:24 356139 0 31 0
0 0
CPU:25 392725 0 32 0
0 0
CPU:26 385937 0 37 0
0 0
CPU:27 385282 0 37 0
0 0
CPU:28 0 0 0 0
0 0
CPU:29 0 0 0 0
0 0
CPU:30 0 0 0 0
0 0
CPU:31 0 0 0 0
0 0
CPU:32 0 0 0 0
0 0
CPU:33 0 0 0 0
0 0
CPU:34 0 0 0 0
0 0
CPU:35 0 0 0 0
0 0
CPU:36 0 0 0 0
0 0
CPU:37 0 0 0 0
0 0
CPU:38 0 0 0 0
0 0
CPU:39 0 0 0 0
0 0
CPU:40 0 0 0 0
0 0
CPU:41 0 0 0 0
0 0
CPU:42 340817 0 33 0
0 0
CPU:43 364805 0 42 0
0 0
CPU:44 298484 0 29 0
0 0
CPU:45 292798 0 30 0
0 0
CPU:46 301739 0 24 0
0 0
CPU:47 275116 0 20 0
0 0
CPU:48 319237 0 34 0
0 0
CPU:49 290350 0 29 0
0 0
CPU:50 307084 0 30 0
0 0
CPU:51 332908 0 24 0
0 0
CPU:52 300151 0 24 0
0 0
CPU:53 310140 0 28 0
0 0
CPU:54 341788 0 28 0
0 0
CPU:55 320344 0 28 0
0 0
Summed: 9734722 0 860 0
0 0
>
>
>
>
>
>
>
>
>
>
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 21:53 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <20181110220131.655f7cc0@redhat.com>
W dniu 10.11.2018 o 22:01, Jesper Dangaard Brouer pisze:
> On Sat, 10 Nov 2018 21:02:10 +0100
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>>> I want you to experiment with:
>>>
>>> ethtool --set-priv-flags DEVICE rx_striding_rq off
>> just checked that previously connectx4 was have thos disabled:
>> ethtool --show-priv-flags enp175s0f0
>>
>> Private flags for enp175s0f0:
>> rx_cqe_moder : on
>> tx_cqe_moder : off
>> rx_cqe_compress : off
>> rx_striding_rq : off
>> rx_no_csum_complete: off
>>
> The CX4 hardware does not have this feature (p.s. the CX4-Lx does).
>
>
>> So now we are on connectx5 and we have enabled - for sure connectx5
>> changed cpu load - where i have now max 50/60% cpu where with connectx4
>> there was sometimes near 100% with same configuration.
> I (strongly) believe the CPU load was related to the page-alloactor
> lock congestion, that Aaron fixed.
>
Yes i think both - most problems with cpu was due to page-allocator
problems.
But also after change connctx4 to connectx5 there is cpu load difference
- about 10% in total - but yes most of this like 40% is cause of Aaron
patch :) - rly good job :)
Now im messing with ring configuration for connectx5 nics.
And after reading that paper:
https://netdevconf.org/2.1/slides/apr6/network-performance/
04-amir-RX_and_TX_bulking_v2.pdf
changed from RX:8192 / TX: 4096 to RX:8192 / TX: 256
after this i gain about 5Gbit/s RX and TX traffic and less cpu load....
before change there was 59/59 Gbit/s
After change there is 64/64 Gbit/s
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
| iface Rx Tx Total
==============================================================================
enp175s0: 44.45 Gb/s 19.69 Gb/s
64.14 Gb/s
enp216s0: 19.69 Gb/s 44.49 Gb/s
64.19 Gb/s
------------------------------------------------------------------------------
total: 64.14 Gb/s 64.18 Gb/s 128.33 Gb/s
^ permalink raw reply
* Re: [PATCH iproute2] testsuite: ss: Fix spacing in expected output for ssfilter.t
From: Phil Sutter @ 2018-11-10 21:48 UTC (permalink / raw)
To: Stefano Brivio; +Cc: Stephen Hemminger, Yoann P., netdev
In-Reply-To: <d50e870d746ecb30f1ace3af40db2a249da6b4c2.1541841383.git.sbrivio@redhat.com>
Hi Stefano,
On Sat, Nov 10, 2018 at 10:21:59AM +0100, Stefano Brivio wrote:
> Since commit 00240899ec0b ("ss: Actually print left delimiter for
> columns") changes spacing in ss output, we also need to adjust for that in
> the ss filter test.
>
> Fixes: 00240899ec0b ("ss: Actually print left delimiter for columns")
> Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
> ---
> testsuite/tests/ss/ssfilter.t | 24 ++++++++++++------------
> 1 file changed, 12 insertions(+), 12 deletions(-)
>
> diff --git a/testsuite/tests/ss/ssfilter.t b/testsuite/tests/ss/ssfilter.t
> index e74f1765cb72..3091054f2892 100755
> --- a/testsuite/tests/ss/ssfilter.t
> +++ b/testsuite/tests/ss/ssfilter.t
> @@ -12,37 +12,37 @@ export TCPDIAG_FILE="$(dirname $0)/ss1.dump"
> ts_log "[Testing ssfilter]"
>
> ts_ss "$0" "Match dport = 22" -Htna dport = 22
> -test_on "ESTAB 0 0 10.0.0.1:36266 10.0.0.1:22"
> +test_on "ESTAB 0 0 10.0.0.1:36266 10.0.0.1:22"
How about using a regular expression ('test_on' calls grep with '-E')?
E.g. this instead of the above:
| test_on "ESTAB *0 *0 *10.0.0.1:36266 *10.0.0.1:22"
Note that I didn't test this change, just made sure 'grep -E' recognizes
' *' as expected.
Cheers, Phil
^ permalink raw reply
* Re: [PATCH net-next 0/6] Remove VLAN.CFI overload
From: David Miller @ 2018-11-10 21:47 UTC (permalink / raw)
To: mirq-linux
Cc: netdev, ast, benh, daniel, jhogan, linux-mips, linuxppc-dev, mpe,
paul.burton, paulus, ralf, sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
From: Michał Mirosław <mirq-linux@rere.qmqm.pl>
Date: Sat, 10 Nov 2018 19:58:29 +0100
> Fix BPF code/JITs to allow for separate VLAN_PRESENT flag
> storage and finally move the flag to separate storage in skbuff.
>
> This is final step to make CLAN.CFI transparent to core Linux
> networking stack.
>
> An #ifdef is introduced temporarily to mark fragments masking
> VLAN_TAG_PRESENT. This is removed altogether in the final patch.
Daniel and Alexei, please review.
^ permalink raw reply
* Re: [PATCH net-next] OVS: remove VLAN_TAG_PRESENT - fixup
From: David Miller @ 2018-11-10 21:42 UTC (permalink / raw)
To: mirq-linux; +Cc: netdev, pshelar, dev
In-Reply-To: <3a3f096a59e139d4149ef57d247639286bba9593.1541875851.git.mirq-linux@rere.qmqm.pl>
From: Michał Mirosław <mirq-linux@rere.qmqm.pl>
Date: Sat, 10 Nov 2018 19:55:34 +0100
> It turns out I missed one VLAN_TAG_PRESENT in OVS code while rebasing.
> This fixes it.
>
> Fixes: 9df46aefafa6 ("OVS: remove use of VLAN_TAG_PRESENT")
> Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH net-next v2] documentation of some IP/ICMP snmp counters
From: peng yu @ 2018-11-10 21:41 UTC (permalink / raw)
To: netdev, Cong Wang, rdunlap
In-Reply-To: <20181110213812.8260-1-yupeng0921@gmail.com>
my changes depend on previous feedback:
1 group counters by protocol
2 remove all text which is pasted from RFC
3 about the simple command as 'ping', I hope to provide a full test
steps, so I still keep them in the document
4 use capital letter for the abbreviated keywords
5 tried to fix grammar mistakes, but I re-organized the document, lots
of sentences are changed, it may have new mistakes.
6 only provide IP/ICMP counters, no TCP counters, will provide TCP
counters after the IP/ICMP counters are accepted.
On Sat, Nov 10, 2018 at 1:38 PM yupeng <yupeng0921@gmail.com> wrote:
>
> The snmp_counter.rst explains the meanings of snmp counters. It also
> provides a set of experiments (only 1 for this initial patch),
> combines the experiments' resutls and the snmp counters'
> meanings. This is an initial path, only explains a part of IP/ICMP
> counters and provide a simple ping test.
>
> Signed-off-by: yupeng <yupeng0921@gmail.com>
> ---
> Documentation/networking/index.rst | 1 +
> Documentation/networking/snmp_counter.rst | 222 ++++++++++++++++++++++
> 2 files changed, 223 insertions(+)
> create mode 100644 Documentation/networking/snmp_counter.rst
>
> diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst
> index bd89dae8d578..6a47629ef8ed 100644
> --- a/Documentation/networking/index.rst
> +++ b/Documentation/networking/index.rst
> @@ -31,6 +31,7 @@ Contents:
> net_failover
> alias
> bridge
> + snmp_counter
>
> .. only:: subproject
>
> diff --git a/Documentation/networking/snmp_counter.rst b/Documentation/networking/snmp_counter.rst
> new file mode 100644
> index 000000000000..b1cfc70cd5f6
> --- /dev/null
> +++ b/Documentation/networking/snmp_counter.rst
> @@ -0,0 +1,222 @@
> +===========
> +SNMP counter
> +===========
> +
> +This document explains the meaning of SNMP counters.
> +
> +General IPv4 counters
> +====================
> +All layer 4 packets and ICMP packets will change these counters, but
> +these counters won't be changed by layer 2 packets (such as STP) or
> +ARP packets.
> +
> +* IpInReceives
> +Defined in `RFC1213 ipInReceives`_
> +
> +.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
> +
> +The number of packets received by the IP layer. It gets increasing at the
> +beginning of ip_rcv function, always be updated together with
> +IpExtInOctets. It indicates the number of aggregated segments after
> +GRO/LRO.
> +
> +* IpInDelivers
> +Defined in `RFC1213 ipInDelivers`_
> +
> +.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
> +
> +The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
> +ICMP and so on. If no one listens on a raw socket, only kernel
> +supported protocols will be delivered, if someone listens on the raw
> +socket, all valid IP packets will be delivered.
> +
> +* IpOutRequests
> +Defined in `RFC1213 ipOutRequests`_
> +
> +.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
> +
> +The number of packets sent via IP layer, for both single cast and
> +multicast packets, and would always be updated together with
> +IpExtOutOctets.
> +
> +* IpExtInOctets and IpExtOutOctets
> +They are linux kernel extensions, no RFC definitions. Please note,
> +RFC1213 indeed defines ifInOctets and ifOutOctets, but they
> +are different things. The ifInOctets and ifOutOctets include the MAC
> +layer header size but IpExtInOctets and IpExtOutOctets don't, they
> +only include the IP layer header and the IP layer data.
> +
> +* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
> +They indicate the number of four kinds of ECN IP packets, please refer
> +`Explicit Congestion Notification`_ for more details.
> +
> +.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
> +
> +These 4 counters calculate how many packets received per ECN
> +status. They count the real frame number regardless the LRO/GRO. So
> +for the same packet, you might find that IpInReceives count 1, but
> +IpExtInNoECTPkts counts 2 or more.
> +
> +ICMP counters
> +============
> +* IcmpInMsgs and IcmpOutMsgs
> +Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
> +
> +.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
> +.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
> +
> +As mentioned in the RFC1213, these two counters include errors, they
> +would be increased even if the ICMP packet has an invalid type. The
> +ICMP output path will check the header of a raw socket, so the
> +IcmpOutMsgs would still be updated if the IP header is constructed by
> +a userspace program.
> +
> +* ICMP named types
> +| These counters include most of common ICMP types, they are:
> +| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
> +| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
> +| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
> +| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
> +| IcmpInRedirects: `RFC1213 icmpInRedirects`_
> +| IcmpInEchos: `RFC1213 icmpInEchos`_
> +| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
> +| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
> +| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
> +| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
> +| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
> +| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
> +| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
> +| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
> +| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
> +| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
> +| IcmpOutEchos: `RFC1213 icmpOutEchos`_
> +| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
> +| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
> +| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
> +| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
> +| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
> +
> +.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
> +.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
> +.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
> +.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
> +.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
> +.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
> +.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
> +.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
> +.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
> +.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
> +.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
> +
> +.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
> +.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
> +.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
> +.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
> +.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
> +.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
> +.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
> +.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
> +.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
> +.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
> +.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
> +
> +Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
> +Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
> +straightforward. The 'In' counter means kernel receives such a packet
> +and the 'Out' counter means kernel sends such a packet.
> +
> +* ICMP numeric types
> +They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
> +ICMP type number. These counters track all kinds of ICMP packets. The
> +ICMP type number definition could be found in the `ICMP parameters`_
> +document.
> +
> +.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
> +
> +For example, if the Linux kernel sends an ICMP Echo packet, the
> +IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
> +packet, IcmpMsgInType0 would increase 1.
> +
> +* IcmpInCsumErrors
> +This counter indicates the checksum of the ICMP packet is
> +wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
> +before updating IcmpMsgInType[N]. If a packet has bad checksum, the
> +IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
> +
> +* IcmpInErrors and IcmpOutErrors
> +Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
> +
> +.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
> +.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
> +
> +When an error occurs in the ICMP packet handler path, these two
> +counters would be updated. The receiving packet path use IcmpInErrors
> +and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
> +is increased, IcmpInErrors would always be increased too.
> +
> +relationship of the ICMP counters
> +-------------------------------
> +The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
> +are updated at the same time. The sum of IcmpMsgInType[N] plus
> +IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
> +receives an ICMP packet, kernel follows below logic:
> +
> +1. increase IcmpInMsgs
> +2. if has any error, update IcmpInErrors and finish the process
> +3. update IcmpMsgOutType[N]
> +4. handle the packet depending on the type, if has any error, update
> + IcmpInErrors and finish the process
> +
> +So if all errors occur in step (2), IcmpInMsgs should be equal to the
> +sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
> +step (4), IcmpInMsgs should be equal to the sum of
> +IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
> +IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
> +IcmpInErrors.
> +
> +examples
> +=======
> +
> +ping test
> +--------
> +Run the ping command against the public dns server 8.8.8.8::
> +
> + nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
> + PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
> + 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
> +
> + --- 8.8.8.8 ping statistics ---
> + 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> + rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
> +
> +The nstayt result::
> +
> + nstatuser@nstat-a:~$ nstat
> + #kernel
> + IpInReceives 1 0.0
> + IpInDelivers 1 0.0
> + IpOutRequests 1 0.0
> + IcmpInMsgs 1 0.0
> + IcmpInEchoReps 1 0.0
> + IcmpOutMsgs 1 0.0
> + IcmpOutEchos 1 0.0
> + IcmpMsgInType0 1 0.0
> + IcmpMsgOutType8 1 0.0
> + IpExtInOctets 84 0.0
> + IpExtOutOctets 84 0.0
> + IpExtInNoECTPkts 1 0.0
> +
> +The Linux server sent an ICMP Echo packet, so IpOutRequests,
> +IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
> +server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
> +IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
> +was passed to the ICMP layer via IP layer, so IpInDelivers was
> +increased 1. The default ping data size is 48, so an ICMP Echo packet
> +and its corresponding Echo Reply packet are constructed by:
> +
> +* 14 bytes MAC header
> +* 20 bytes IP header
> +* 16 bytes ICMP header
> +* 48 bytes data (default value of the ping command)
> +
> +So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
> --
> 2.17.1
>
^ permalink raw reply
* [PATCH net-next v2] documentation of some IP/ICMP snmp counters
From: yupeng @ 2018-11-10 21:38 UTC (permalink / raw)
To: netdev, xiyou.wangcong, rdunlap
The snmp_counter.rst explains the meanings of snmp counters. It also
provides a set of experiments (only 1 for this initial patch),
combines the experiments' resutls and the snmp counters'
meanings. This is an initial path, only explains a part of IP/ICMP
counters and provide a simple ping test.
Signed-off-by: yupeng <yupeng0921@gmail.com>
---
Documentation/networking/index.rst | 1 +
Documentation/networking/snmp_counter.rst | 222 ++++++++++++++++++++++
2 files changed, 223 insertions(+)
create mode 100644 Documentation/networking/snmp_counter.rst
diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst
index bd89dae8d578..6a47629ef8ed 100644
--- a/Documentation/networking/index.rst
+++ b/Documentation/networking/index.rst
@@ -31,6 +31,7 @@ Contents:
net_failover
alias
bridge
+ snmp_counter
.. only:: subproject
diff --git a/Documentation/networking/snmp_counter.rst b/Documentation/networking/snmp_counter.rst
new file mode 100644
index 000000000000..b1cfc70cd5f6
--- /dev/null
+++ b/Documentation/networking/snmp_counter.rst
@@ -0,0 +1,222 @@
+===========
+SNMP counter
+===========
+
+This document explains the meaning of SNMP counters.
+
+General IPv4 counters
+====================
+All layer 4 packets and ICMP packets will change these counters, but
+these counters won't be changed by layer 2 packets (such as STP) or
+ARP packets.
+
+* IpInReceives
+Defined in `RFC1213 ipInReceives`_
+
+.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
+
+The number of packets received by the IP layer. It gets increasing at the
+beginning of ip_rcv function, always be updated together with
+IpExtInOctets. It indicates the number of aggregated segments after
+GRO/LRO.
+
+* IpInDelivers
+Defined in `RFC1213 ipInDelivers`_
+
+.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
+
+The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
+ICMP and so on. If no one listens on a raw socket, only kernel
+supported protocols will be delivered, if someone listens on the raw
+socket, all valid IP packets will be delivered.
+
+* IpOutRequests
+Defined in `RFC1213 ipOutRequests`_
+
+.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
+
+The number of packets sent via IP layer, for both single cast and
+multicast packets, and would always be updated together with
+IpExtOutOctets.
+
+* IpExtInOctets and IpExtOutOctets
+They are linux kernel extensions, no RFC definitions. Please note,
+RFC1213 indeed defines ifInOctets and ifOutOctets, but they
+are different things. The ifInOctets and ifOutOctets include the MAC
+layer header size but IpExtInOctets and IpExtOutOctets don't, they
+only include the IP layer header and the IP layer data.
+
+* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
+They indicate the number of four kinds of ECN IP packets, please refer
+`Explicit Congestion Notification`_ for more details.
+
+.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
+
+These 4 counters calculate how many packets received per ECN
+status. They count the real frame number regardless the LRO/GRO. So
+for the same packet, you might find that IpInReceives count 1, but
+IpExtInNoECTPkts counts 2 or more.
+
+ICMP counters
+============
+* IcmpInMsgs and IcmpOutMsgs
+Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
+
+.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
+.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
+
+As mentioned in the RFC1213, these two counters include errors, they
+would be increased even if the ICMP packet has an invalid type. The
+ICMP output path will check the header of a raw socket, so the
+IcmpOutMsgs would still be updated if the IP header is constructed by
+a userspace program.
+
+* ICMP named types
+| These counters include most of common ICMP types, they are:
+| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
+| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
+| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
+| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
+| IcmpInRedirects: `RFC1213 icmpInRedirects`_
+| IcmpInEchos: `RFC1213 icmpInEchos`_
+| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
+| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
+| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
+| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
+| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
+| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
+| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
+| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
+| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
+| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
+| IcmpOutEchos: `RFC1213 icmpOutEchos`_
+| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
+| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
+| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
+| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
+| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
+
+.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
+.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
+.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
+.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
+.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
+.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
+.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
+.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
+.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
+.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
+.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
+
+.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
+.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
+.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
+.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
+.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
+.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
+.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
+.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
+.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
+.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
+.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
+
+Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
+Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
+straightforward. The 'In' counter means kernel receives such a packet
+and the 'Out' counter means kernel sends such a packet.
+
+* ICMP numeric types
+They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
+ICMP type number. These counters track all kinds of ICMP packets. The
+ICMP type number definition could be found in the `ICMP parameters`_
+document.
+
+.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
+
+For example, if the Linux kernel sends an ICMP Echo packet, the
+IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
+packet, IcmpMsgInType0 would increase 1.
+
+* IcmpInCsumErrors
+This counter indicates the checksum of the ICMP packet is
+wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
+before updating IcmpMsgInType[N]. If a packet has bad checksum, the
+IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
+
+* IcmpInErrors and IcmpOutErrors
+Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
+
+.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
+.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
+
+When an error occurs in the ICMP packet handler path, these two
+counters would be updated. The receiving packet path use IcmpInErrors
+and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
+is increased, IcmpInErrors would always be increased too.
+
+relationship of the ICMP counters
+-------------------------------
+The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
+are updated at the same time. The sum of IcmpMsgInType[N] plus
+IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
+receives an ICMP packet, kernel follows below logic:
+
+1. increase IcmpInMsgs
+2. if has any error, update IcmpInErrors and finish the process
+3. update IcmpMsgOutType[N]
+4. handle the packet depending on the type, if has any error, update
+ IcmpInErrors and finish the process
+
+So if all errors occur in step (2), IcmpInMsgs should be equal to the
+sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
+step (4), IcmpInMsgs should be equal to the sum of
+IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
+IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
+IcmpInErrors.
+
+examples
+=======
+
+ping test
+--------
+Run the ping command against the public dns server 8.8.8.8::
+
+ nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
+ PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
+ 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
+
+ --- 8.8.8.8 ping statistics ---
+ 1 packets transmitted, 1 received, 0% packet loss, time 0ms
+ rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
+
+The nstayt result::
+
+ nstatuser@nstat-a:~$ nstat
+ #kernel
+ IpInReceives 1 0.0
+ IpInDelivers 1 0.0
+ IpOutRequests 1 0.0
+ IcmpInMsgs 1 0.0
+ IcmpInEchoReps 1 0.0
+ IcmpOutMsgs 1 0.0
+ IcmpOutEchos 1 0.0
+ IcmpMsgInType0 1 0.0
+ IcmpMsgOutType8 1 0.0
+ IpExtInOctets 84 0.0
+ IpExtOutOctets 84 0.0
+ IpExtInNoECTPkts 1 0.0
+
+The Linux server sent an ICMP Echo packet, so IpOutRequests,
+IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
+server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
+IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
+was passed to the ICMP layer via IP layer, so IpInDelivers was
+increased 1. The default ping data size is 48, so an ICMP Echo packet
+and its corresponding Echo Reply packet are constructed by:
+
+* 14 bytes MAC header
+* 20 bytes IP header
+* 16 bytes ICMP header
+* 48 bytes data (default value of the ping command)
+
+So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
--
2.17.1
^ permalink raw reply related
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Jesper Dangaard Brouer @ 2018-11-10 21:01 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev@vger.kernel.org, brouer
In-Reply-To: <5d44e0d9-d8d3-1b07-5392-838e4096ca74@itcare.pl>
On Sat, 10 Nov 2018 21:02:10 +0100
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> > I want you to experiment with:
> >
> > ethtool --set-priv-flags DEVICE rx_striding_rq off
>
> just checked that previously connectx4 was have thos disabled:
> ethtool --show-priv-flags enp175s0f0
>
> Private flags for enp175s0f0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : off
> rx_striding_rq : off
> rx_no_csum_complete: off
>
The CX4 hardware does not have this feature (p.s. the CX4-Lx does).
> So now we are on connectx5 and we have enabled - for sure connectx5
> changed cpu load - where i have now max 50/60% cpu where with connectx4
> there was sometimes near 100% with same configuration.
I (strongly) believe the CPU load was related to the page-alloactor
lock congestion, that Aaron fixed.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* From Mr Hamzak Wadrago;
From: Hamzak Wadrago @ 2018-11-10 20:56 UTC (permalink / raw)
I am Mr hamzak wadrago, the chief operating officer with my bank and I
want to inform you that an amount of US$15.5 million will be moved on
your name as the Foreign Business Partner to our late deceased
customer. I need your help to receive this money as we shall share the
money in the ratio of 60:40%. You will receive this amount through a
bank wire transfer.
Please send your full names, direct telephone numbers, and physical
address, more details and text of application form will be given to
you upon your reply.
Your quick response will be highly appreciated.
Yours sincerely,
Mr. Hamzak Wadrago,
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 20:02 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <20181110203409.482f39ec@redhat.com>
W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> I want you to experiment with:
>
> ethtool --set-priv-flags DEVICE rx_striding_rq off
just checked that previously connectx4 was have thos disabled:
ethtool --show-priv-flags enp175s0f0
Private flags for enp175s0f0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : off
rx_striding_rq : off
rx_no_csum_complete: off
So now we are on connectx5 and we have enabled - for sure connectx5
changed cpu load - where i have now max 50/60% cpu where with connectx4
there was sometimes near 100% with same configuration.
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 19:56 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <7037c58d-d77d-bdd5-6c91-19cea3cbe539@itcare.pl>
W dniu 10.11.2018 o 20:49, Paweł Staszewski pisze:
>
>
> W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
>> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>>> limit is the same :)
>>>> But also after reaching 60Gbit/60Gbit
>>>>
>>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> - iface Rx Tx Total
>>>> ==========================================================================
>>>>
>>>>
>>>> enp175s0: 45.09 Gb/s 15.09 Gb/s
>>>> 60.18 Gb/s
>>>> enp216s0: 15.14 Gb/s 45.19 Gb/s
>>>> 60.33 Gb/s
>>>> --------------------------------------------------------------------------
>>>>
>>>>
>>>> total: 60.45 Gb/s 60.48 Gb/s 120.93
>>>> Gb/s
>>> Today reached 65/65Gbit/s
>>>
>>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
>> This is weird!
>>
>> How do you see / measure these drops?
> Simple icmp test like ping -i 0.1
> And im testing by icmp management ip address on vlan that is attacked
> to one NIC (the side that is more stressed with RX)
> And another icmp test is forward thru this router - host behind it
>
> Both measurements shows same loss ratio from 0.1 to 0.5% after
> reaching ~45Gbit/s RX side - depends how much RX side is pushed drops
> vary between 0.1 to 0.5 - even 0.6%:)
>
>
>>
>>
>>> So checked other stats.
>>> softnet_stats shows average 1k squeezed per sec:
>> Is below output the raw counters? not per sec?
>>
>> It would be valuable to see the per sec stats instead...
>> I use this tool:
>> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
CPU total/sec dropped/sec squeezed/sec
collision/sec rx_rps/sec flow_limit/sec
CPU:00 0 0 0 0
0 0
CPU:01 0 0 0 0
0 0
CPU:02 0 0 0 0
0 0
CPU:03 0 0 0 0
0 0
CPU:04 0 0 0 0
0 0
CPU:05 0 0 0 0
0 0
CPU:06 0 0 0 0
0 0
CPU:07 0 0 0 0
0 0
CPU:08 0 0 0 0
0 0
CPU:09 0 0 0 0
0 0
CPU:10 0 0 0 0
0 0
CPU:11 0 0 0 0
0 0
CPU:12 0 0 0 0
0 0
CPU:13 0 0 0 0
0 0
CPU:14 485538 0 43 0
0 0
CPU:15 474794 0 51 0
0 0
CPU:16 449322 0 41 0
0 0
CPU:17 476420 0 46 0
0 0
CPU:18 440436 0 38 0
0 0
CPU:19 501499 0 49 0
0 0
CPU:20 459468 0 49 0
0 0
CPU:21 438928 0 47 0
0 0
CPU:22 468983 0 40 0
0 0
CPU:23 446253 0 47 0
0 0
CPU:24 451909 0 46 0
0 0
CPU:25 479373 0 55 0
0 0
CPU:26 467848 0 49 0
0 0
CPU:27 453153 0 51 0
0 0
CPU:28 0 0 0 0
0 0
CPU:29 0 0 0 0
0 0
CPU:30 0 0 0 0
0 0
CPU:31 0 0 0 0
0 0
CPU:32 0 0 0 0
0 0
CPU:33 0 0 0 0
0 0
CPU:34 0 0 0 0
0 0
CPU:35 0 0 0 0
0 0
CPU:36 0 0 0 0
0 0
CPU:37 0 0 0 0
0 0
CPU:38 0 0 0 0
0 0
CPU:39 0 0 0 0
0 0
CPU:40 0 0 0 0
0 0
CPU:41 0 0 0 0
0 0
CPU:42 466853 0 43 0
0 0
CPU:43 453059 0 54 0
0 0
CPU:44 363219 0 34 0
0 0
CPU:45 353632 0 38 0
0 0
CPU:46 371618 0 40 0
0 0
CPU:47 350518 0 46 0
0 0
CPU:48 397544 0 40 0
0 0
CPU:49 364873 0 38 0
0 0
CPU:50 383630 0 38 0
0 0
CPU:51 358771 0 39 0
0 0
CPU:52 372547 0 38 0
0 0
CPU:53 372882 0 36 0
0 0
CPU:54 366244 0 43 0
0 0
CPU:55 365886 0 39 0
0 0
Summed: 11835201 0 1217 0
0 0
>>
>>> cpu total dropped squeezed collision rps flow_limit
>>> 0 18554 0 1 0 0 0
>>> 1 16728 0 1 0 0 0
>>> 2 18033 0 1 0 0 0
>>> 3 17757 0 1 0 0 0
>>> 4 18861 0 0 0 0 0
>>> 5 0 0 1 0 0 0
>>> 6 2 0 1 0 0 0
>>> 7 0 0 1 0 0 0
>>> 8 0 0 0 0 0 0
>>> 9 0 0 1 0 0 0
>>> 10 0 0 0 0 0 0
>>> 11 0 0 1 0 0 0
>>> 12 50 0 1 0 0 0
>>> 13 257 0 0 0 0 0
>>> 14 3629115363 0 3353259 0 0 0
>>> 15 255167835 0 3138271 0 0 0
>>> 16 4240101961 0 3036130 0 0 0
>>> 17 599810018 0 3072169 0 0 0
>>> 18 432796524 0 3034191 0 0 0
>>> 19 41803906 0 3037405 0 0 0
>>> 20 900382666 0 3112294 0 0 0
>>> 21 620926085 0 3086009 0 0 0
>>> 22 41861198 0 3023142 0 0 0
>>> 23 4090425574 0 2990412 0 0 0
>>> 24 4264870218 0 3010272 0 0 0
>>> 25 141401811 0 3027153 0 0 0
>>> 26 104155188 0 3051251 0 0 0
>>> 27 4261258691 0 3039765 0 0 0
>>> 28 4 0 1 0 0 0
>>> 29 4 0 0 0 0 0
>>> 30 0 0 1 0 0 0
>>> 31 0 0 0 0 0 0
>>> 32 3 0 1 0 0 0
>>> 33 1 0 1 0 0 0
>>> 34 0 0 1 0 0 0
>>> 35 0 0 0 0 0 0
>>> 36 0 0 1 0 0 0
>>> 37 0 0 1 0 0 0
>>> 38 0 0 1 0 0 0
>>> 39 0 0 1 0 0 0
>>> 40 0 0 0 0 0 0
>>> 41 0 0 1 0 0 0
>>> 42 299758202 0 3139693 0 0 0
>>> 43 4254727979 0 3103577 0 0 0
>>> 44 1959555543 0 2554885 0 0 0
>>> 45 1675702723 0 2513481 0 0 0
>>> 46 1908435503 0 2519698 0 0 0
>>> 47 1877799710 0 2537768 0 0 0
>>> 48 2384274076 0 2584673 0 0 0
>>> 49 2598104878 0 2593616 0 0 0
>>> 50 1897566829 0 2530857 0 0 0
>>> 51 1712741629 0 2489089 0 0 0
>>> 52 1704033648 0 2495892 0 0 0
>>> 53 1636781820 0 2499783 0 0 0
>>> 54 1861997734 0 2541060 0 0 0
>>> 55 2113521616 0 2555673 0 0 0
>>>
>>>
>>> So i rised netdev backlog and budged to rly high values
>>> 524288 for netdev_budget and same for backlog
>> Does it affect the squeezed counters?
> a little - but not much
> After change budget from 65536 to to 524k - number of squeezed
> counters for all cpus changed from 1.5k per second to 0.9-1k per
> second - but increasing it more like above 524k change nothing - same
> 0.9 to 1k/s squeezed
>>
>> Notice, this (crazy) huge netdev_budget limit will also be limited
>> by /proc/sys/net/core/netdev_budget_usecs.
> Yes changed that also to 1000 / 2000 / 3000 / 4000 not much
> difference on squeezed - even cant see the difference
>
>>
>>> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
>> Hmmm, this could indicated not enough NAPI bulking is occurring.
>>
>> I have a BPF tool, that can give you some insight into NAPI bulking and
>> softirq idle/kthread starting. Called 'napi_monitor', could you try to
>> run this, so can try to understand this? You find the tool here:
>>
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
>> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
> yes will try it
>
>>> But after this changes i have less packets drops.
>>>
>>>
>>> Below perf top from max traffic reached:
>>> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>> ------------------------------------------------------------------------------------------
>>>
>>>
>>> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
>>> 8.44% [kernel] [k] mlx5e_sq_xmit
>>> 6.69% [kernel] [k] build_skb
>>> 5.21% [kernel] [k] fib_table_lookup
>>> 3.54% [kernel] [k] memcpy_erms
>>> 3.20% [kernel] [k] mlx5e_poll_rx_cq
>>> 2.25% [kernel] [k] vlan_do_receive
>>> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
>>> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
>>> 1.95% [kernel] [k] __dev_queue_xmit
>>> 1.83% [kernel] [k] dev_gro_receive
>>> 1.79% [kernel] [k] tcp_gro_receive
>>> 1.73% [kernel] [k] ip_finish_output2
>>> 1.63% [kernel] [k] mlx5e_poll_tx_cq
>>> 1.49% [kernel] [k] ipt_do_table
>>> 1.38% [kernel] [k] inet_gro_receive
>>> 1.31% [kernel] [k] __netif_receive_skb_core
>>> 1.30% [kernel] [k] _raw_spin_lock
>>> 1.28% [kernel] [k] mlx5_eq_int
>>> 1.24% [kernel] [k] irq_entries_start
>>> 1.19% [kernel] [k] __build_skb
>>> 1.15% [kernel] [k] swiotlb_map_page
>>> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
>>> 0.94% [kernel] [k] pfifo_fast_dequeue
>>> 0.92% [kernel] [k] ip_route_input_rcu
>>> 0.86% [kernel] [k] kmem_cache_alloc
>>> 0.80% [kernel] [k] mlx5e_xmit
>>> 0.79% [kernel] [k] dev_hard_start_xmit
>>> 0.78% [kernel] [k] _raw_spin_lock_irqsave
>>> 0.74% [kernel] [k] ip_forward
>>> 0.72% [kernel] [k] tasklet_action_common.isra.21
>>> 0.68% [kernel] [k] pfifo_fast_enqueue
>>> 0.67% [kernel] [k] netif_skb_features
>>> 0.66% [kernel] [k] skb_segment
>>> 0.60% [kernel] [k] skb_gro_receive
>>> 0.56% [kernel] [k] validate_xmit_skb.isra.142
>>> 0.53% [kernel] [k] skb_release_data
>>> 0.51% [kernel] [k] mlx5e_page_release
>>> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>>> 0.51% [kernel] [k] __qdisc_run
>>> 0.50% [kernel] [k] tcp4_gro_receive
>>> 0.49% [kernel] [k] page_frag_free
>>> 0.46% [kernel] [k] kmem_cache_free_bulk
>>> 0.43% [kernel] [k] kmem_cache_free
>>> 0.42% [kernel] [k] try_to_wake_up
>>> 0.39% [kernel] [k] _raw_spin_lock_irq
>>> 0.39% [kernel] [k] find_busiest_group
>>> 0.37% [kernel] [k] __memcpy
>>>
>>>
>>>
>>> Remember those tests are now on two separate connectx5 connected to
>>> two separate pcie x16 gen 3.0
>> That is strange... I still suspect some HW NIC issue, can you provide
>> ethtool stats info via tool:
>>
>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>
>> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>>
>> The tool remove zero-stats counters and report per sec stats. It makes
>> it easier to spot that is relevant for the given workload.
> yes mlnx have just too many counters that are always 0 for my case :)
> Will try this also
>
But still alot of non 0 counters
Show adapter(s) (enp175s0 enp216s0) statistics (ONLY that changed!)
Ethtool(enp175s0) stat: 8891 ( 8,891) <= ch0_arm /sec
Ethtool(enp175s0) stat: 10265 ( 10,265) <= ch0_events /sec
Ethtool(enp175s0) stat: 11072 ( 11,072) <= ch0_poll /sec
Ethtool(enp175s0) stat: 9003 ( 9,003) <= ch10_arm /sec
Ethtool(enp175s0) stat: 10476 ( 10,476) <= ch10_events /sec
Ethtool(enp175s0) stat: 11284 ( 11,284) <= ch10_poll /sec
Ethtool(enp175s0) stat: 11211 ( 11,211) <= ch11_arm /sec
Ethtool(enp175s0) stat: 12645 ( 12,645) <= ch11_events /sec
Ethtool(enp175s0) stat: 13450 ( 13,450) <= ch11_poll /sec
Ethtool(enp175s0) stat: 9012 ( 9,012) <= ch12_arm /sec
Ethtool(enp175s0) stat: 10366 ( 10,366) <= ch12_events /sec
Ethtool(enp175s0) stat: 11074 ( 11,074) <= ch12_poll /sec
Ethtool(enp175s0) stat: 8810 ( 8,810) <= ch13_arm /sec
Ethtool(enp175s0) stat: 10177 ( 10,177) <= ch13_events /sec
Ethtool(enp175s0) stat: 10886 ( 10,886) <= ch13_poll /sec
Ethtool(enp175s0) stat: 9794 ( 9,794) <= ch14_arm /sec
Ethtool(enp175s0) stat: 11159 ( 11,159) <= ch14_events /sec
Ethtool(enp175s0) stat: 11932 ( 11,932) <= ch14_poll /sec
Ethtool(enp175s0) stat: 8703 ( 8,703) <= ch15_arm /sec
Ethtool(enp175s0) stat: 10052 ( 10,052) <= ch15_events /sec
Ethtool(enp175s0) stat: 10774 ( 10,774) <= ch15_poll /sec
Ethtool(enp175s0) stat: 6429 ( 6,429) <= ch16_arm /sec
Ethtool(enp175s0) stat: 7591 ( 7,591) <= ch16_events /sec
Ethtool(enp175s0) stat: 8223 ( 8,223) <= ch16_poll /sec
Ethtool(enp175s0) stat: 8981 ( 8,981) <= ch17_arm /sec
Ethtool(enp175s0) stat: 10229 ( 10,229) <= ch17_events /sec
Ethtool(enp175s0) stat: 10887 ( 10,887) <= ch17_poll /sec
Ethtool(enp175s0) stat: 6786 ( 6,786) <= ch18_arm /sec
Ethtool(enp175s0) stat: 7887 ( 7,887) <= ch18_events /sec
Ethtool(enp175s0) stat: 8484 ( 8,484) <= ch18_poll /sec
Ethtool(enp175s0) stat: 6080 ( 6,080) <= ch19_arm /sec
Ethtool(enp175s0) stat: 7377 ( 7,377) <= ch19_events /sec
Ethtool(enp175s0) stat: 8124 ( 8,124) <= ch19_poll /sec
Ethtool(enp175s0) stat: 7715 ( 7,715) <= ch1_arm /sec
Ethtool(enp175s0) stat: 9109 ( 9,109) <= ch1_events /sec
Ethtool(enp175s0) stat: 9923 ( 9,923) <= ch1_poll /sec
Ethtool(enp175s0) stat: 7303 ( 7,303) <= ch20_arm /sec
Ethtool(enp175s0) stat: 8514 ( 8,514) <= ch20_events /sec
Ethtool(enp175s0) stat: 9169 ( 9,169) <= ch20_poll /sec
Ethtool(enp175s0) stat: 8972 ( 8,972) <= ch21_arm /sec
Ethtool(enp175s0) stat: 10060 ( 10,060) <= ch21_events /sec
Ethtool(enp175s0) stat: 10647 ( 10,647) <= ch21_poll /sec
Ethtool(enp175s0) stat: 7729 ( 7,729) <= ch22_arm /sec
Ethtool(enp175s0) stat: 8932 ( 8,932) <= ch22_events /sec
Ethtool(enp175s0) stat: 9585 ( 9,585) <= ch22_poll /sec
Ethtool(enp175s0) stat: 8125 ( 8,125) <= ch23_arm /sec
Ethtool(enp175s0) stat: 9218 ( 9,218) <= ch23_events /sec
Ethtool(enp175s0) stat: 9805 ( 9,805) <= ch23_poll /sec
Ethtool(enp175s0) stat: 7212 ( 7,212) <= ch24_arm /sec
Ethtool(enp175s0) stat: 8369 ( 8,369) <= ch24_events /sec
Ethtool(enp175s0) stat: 8993 ( 8,993) <= ch24_poll /sec
Ethtool(enp175s0) stat: 6328 ( 6,328) <= ch25_arm /sec
Ethtool(enp175s0) stat: 7567 ( 7,567) <= ch25_events /sec
Ethtool(enp175s0) stat: 8274 ( 8,274) <= ch25_poll /sec
Ethtool(enp175s0) stat: 6210 ( 6,210) <= ch26_arm /sec
Ethtool(enp175s0) stat: 7409 ( 7,409) <= ch26_events /sec
Ethtool(enp175s0) stat: 8062 ( 8,062) <= ch26_poll /sec
Ethtool(enp175s0) stat: 7035 ( 7,035) <= ch27_arm /sec
Ethtool(enp175s0) stat: 8203 ( 8,203) <= ch27_events /sec
Ethtool(enp175s0) stat: 8840 ( 8,840) <= ch27_poll /sec
Ethtool(enp175s0) stat: 11278 ( 11,278) <= ch2_arm /sec
Ethtool(enp175s0) stat: 12632 ( 12,632) <= ch2_events /sec
Ethtool(enp175s0) stat: 13348 ( 13,348) <= ch2_poll /sec
Ethtool(enp175s0) stat: 10612 ( 10,612) <= ch3_arm /sec
Ethtool(enp175s0) stat: 11900 ( 11,900) <= ch3_events /sec
Ethtool(enp175s0) stat: 12567 ( 12,567) <= ch3_poll /sec
Ethtool(enp175s0) stat: 8936 ( 8,936) <= ch4_arm /sec
Ethtool(enp175s0) stat: 10248 ( 10,248) <= ch4_events /sec
Ethtool(enp175s0) stat: 10962 ( 10,962) <= ch4_poll /sec
Ethtool(enp175s0) stat: 11631 ( 11,631) <= ch5_arm /sec
Ethtool(enp175s0) stat: 12953 ( 12,953) <= ch5_events /sec
Ethtool(enp175s0) stat: 13629 ( 13,629) <= ch5_poll /sec
Ethtool(enp175s0) stat: 9877 ( 9,877) <= ch6_arm /sec
Ethtool(enp175s0) stat: 11114 ( 11,114) <= ch6_events /sec
Ethtool(enp175s0) stat: 11800 ( 11,800) <= ch6_poll /sec
Ethtool(enp175s0) stat: 8228 ( 8,228) <= ch7_arm /sec
Ethtool(enp175s0) stat: 9577 ( 9,577) <= ch7_events /sec
Ethtool(enp175s0) stat: 10320 ( 10,320) <= ch7_poll /sec
Ethtool(enp175s0) stat: 11808 ( 11,808) <= ch8_arm /sec
Ethtool(enp175s0) stat: 13135 ( 13,135) <= ch8_events /sec
Ethtool(enp175s0) stat: 13828 ( 13,828) <= ch8_poll /sec
Ethtool(enp175s0) stat: 10566 ( 10,566) <= ch9_arm /sec
Ethtool(enp175s0) stat: 11904 ( 11,904) <= ch9_events /sec
Ethtool(enp175s0) stat: 12634 ( 12,634) <= ch9_poll /sec
Ethtool(enp175s0) stat: 243256 ( 243,256) <= ch_arm /sec
Ethtool(enp175s0) stat: 279057 ( 279,057) <= ch_events /sec
Ethtool(enp175s0) stat: 298563 ( 298,563) <= ch_poll /sec
Ethtool(enp175s0) stat: 186677525 ( 186,677,525) <= rx0_bytes /sec
Ethtool(enp175s0) stat: 72870 ( 72,870) <=
rx0_cache_reuse /sec
Ethtool(enp175s0) stat: 145627 ( 145,627) <=
rx0_csum_complete /sec
Ethtool(enp175s0) stat: 88 ( 88) <= rx0_csum_none /sec
Ethtool(enp175s0) stat: 145715 ( 145,715) <= rx0_packets /sec
Ethtool(enp175s0) stat: 145715 ( 145,715) <=
rx0_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 198552827 ( 198,552,827) <= rx10_bytes /sec
Ethtool(enp175s0) stat: 75553 ( 75,553) <=
rx10_cache_reuse /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <=
rx10_csum_complete /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <= rx10_packets /sec
Ethtool(enp175s0) stat: 151021 ( 151,021) <=
rx10_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 200924148 ( 200,924,148) <= rx11_bytes /sec
Ethtool(enp175s0) stat: 76589 ( 76,589) <=
rx11_cache_reuse /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <=
rx11_csum_complete /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <= rx11_packets /sec
Ethtool(enp175s0) stat: 153221 ( 153,221) <=
rx11_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 186259790 ( 186,259,790) <= rx12_bytes /sec
Ethtool(enp175s0) stat: 70675 ( 70,675) <=
rx12_cache_reuse /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <=
rx12_csum_complete /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <= rx12_packets /sec
Ethtool(enp175s0) stat: 141440 ( 141,440) <=
rx12_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 189627451 ( 189,627,451) <= rx13_bytes /sec
Ethtool(enp175s0) stat: 72626 ( 72,626) <=
rx13_cache_reuse /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <=
rx13_csum_complete /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <= rx13_packets /sec
Ethtool(enp175s0) stat: 145327 ( 145,327) <=
rx13_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 199246096 ( 199,246,096) <= rx14_bytes /sec
Ethtool(enp175s0) stat: 77992 ( 77,992) <=
rx14_cache_reuse /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <=
rx14_csum_complete /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <= rx14_packets /sec
Ethtool(enp175s0) stat: 156043 ( 156,043) <=
rx14_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 189698176 ( 189,698,176) <= rx15_bytes /sec
Ethtool(enp175s0) stat: 72382 ( 72,382) <=
rx15_cache_reuse /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <=
rx15_csum_complete /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <= rx15_packets /sec
Ethtool(enp175s0) stat: 144658 ( 144,658) <=
rx15_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 143896232 ( 143,896,232) <= rx16_bytes /sec
Ethtool(enp175s0) stat: 55369 ( 55,369) <=
rx16_cache_reuse /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <=
rx16_csum_complete /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <= rx16_packets /sec
Ethtool(enp175s0) stat: 110745 ( 110,745) <=
rx16_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 171449483 ( 171,449,483) <= rx17_bytes /sec
Ethtool(enp175s0) stat: 65308 ( 65,308) <=
rx17_cache_reuse /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <=
rx17_csum_complete /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <= rx17_packets /sec
Ethtool(enp175s0) stat: 130563 ( 130,563) <=
rx17_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 141033264 ( 141,033,264) <= rx18_bytes /sec
Ethtool(enp175s0) stat: 54515 ( 54,515) <=
rx18_cache_reuse /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <=
rx18_csum_complete /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <= rx18_packets /sec
Ethtool(enp175s0) stat: 108966 ( 108,966) <=
rx18_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 163097410 ( 163,097,410) <= rx19_bytes /sec
Ethtool(enp175s0) stat: 61894 ( 61,894) <=
rx19_cache_reuse /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <=
rx19_csum_complete /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <= rx19_packets /sec
Ethtool(enp175s0) stat: 123773 ( 123,773) <=
rx19_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 181707418 ( 181,707,418) <= rx1_bytes /sec
Ethtool(enp175s0) stat: 71223 ( 71,223) <=
rx1_cache_reuse /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <=
rx1_csum_complete /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <= rx1_packets /sec
Ethtool(enp175s0) stat: 142445 ( 142,445) <=
rx1_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 161626368 ( 161,626,368) <= rx20_bytes /sec
Ethtool(enp175s0) stat: 61345 ( 61,345) <=
rx20_cache_reuse /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <=
rx20_csum_complete /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <= rx20_packets /sec
Ethtool(enp175s0) stat: 122724 ( 122,724) <=
rx20_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 138593554 ( 138,593,554) <= rx21_bytes /sec
Ethtool(enp175s0) stat: 53478 ( 53,478) <=
rx21_cache_reuse /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <=
rx21_csum_complete /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <= rx21_packets /sec
Ethtool(enp175s0) stat: 106949 ( 106,949) <=
rx21_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 149217722 ( 149,217,722) <= rx22_bytes /sec
Ethtool(enp175s0) stat: 58174 ( 58,174) <=
rx22_cache_reuse /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <=
rx22_csum_complete /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <= rx22_packets /sec
Ethtool(enp175s0) stat: 116342 ( 116,342) <=
rx22_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 147968086 ( 147,968,086) <= rx23_bytes /sec
Ethtool(enp175s0) stat: 55979 ( 55,979) <=
rx23_cache_reuse /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <=
rx23_csum_complete /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <= rx23_packets /sec
Ethtool(enp175s0) stat: 111901 ( 111,901) <=
rx23_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 145955524 ( 145,955,524) <= rx24_bytes /sec
Ethtool(enp175s0) stat: 55491 ( 55,491) <=
rx24_cache_reuse /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <=
rx24_csum_complete /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <= rx24_packets /sec
Ethtool(enp175s0) stat: 110980 ( 110,980) <=
rx24_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 155552699 ( 155,552,699) <= rx25_bytes /sec
Ethtool(enp175s0) stat: 59028 ( 59,028) <=
rx25_cache_reuse /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <=
rx25_csum_complete /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <= rx25_packets /sec
Ethtool(enp175s0) stat: 118074 ( 118,074) <=
rx25_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 144880442 ( 144,880,442) <= rx26_bytes /sec
Ethtool(enp175s0) stat: 56223 ( 56,223) <=
rx26_cache_reuse /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <=
rx26_csum_complete /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <= rx26_packets /sec
Ethtool(enp175s0) stat: 112334 ( 112,334) <=
rx26_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 154545288 ( 154,545,288) <= rx27_bytes /sec
Ethtool(enp175s0) stat: 58784 ( 58,784) <=
rx27_cache_reuse /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <=
rx27_csum_complete /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <= rx27_packets /sec
Ethtool(enp175s0) stat: 117627 ( 117,627) <=
rx27_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 182425129 ( 182,425,129) <= rx2_bytes /sec
Ethtool(enp175s0) stat: 71406 ( 71,406) <=
rx2_cache_reuse /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <=
rx2_csum_complete /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <= rx2_packets /sec
Ethtool(enp175s0) stat: 142872 ( 142,872) <=
rx2_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 188368405 ( 188,368,405) <= rx3_bytes /sec
Ethtool(enp175s0) stat: 72138 ( 72,138) <=
rx3_cache_reuse /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <=
rx3_csum_complete /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <= rx3_packets /sec
Ethtool(enp175s0) stat: 144259 ( 144,259) <=
rx3_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 186009984 ( 186,009,984) <= rx4_bytes /sec
Ethtool(enp175s0) stat: 70004 ( 70,004) <=
rx4_cache_reuse /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <=
rx4_csum_complete /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <= rx4_packets /sec
Ethtool(enp175s0) stat: 139939 ( 139,939) <=
rx4_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 198040550 ( 198,040,550) <= rx5_bytes /sec
Ethtool(enp175s0) stat: 75492 ( 75,492) <=
rx5_cache_reuse /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <=
rx5_csum_complete /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <= rx5_packets /sec
Ethtool(enp175s0) stat: 150950 ( 150,950) <=
rx5_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 182607101 ( 182,607,101) <= rx6_bytes /sec
Ethtool(enp175s0) stat: 69699 ( 69,699) <=
rx6_cache_reuse /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <=
rx6_csum_complete /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <= rx6_packets /sec
Ethtool(enp175s0) stat: 139335 ( 139,335) <=
rx6_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 174999243 ( 174,999,243) <= rx7_bytes /sec
Ethtool(enp175s0) stat: 66650 ( 66,650) <=
rx7_cache_reuse /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <=
rx7_csum_complete /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <= rx7_packets /sec
Ethtool(enp175s0) stat: 133323 ( 133,323) <=
rx7_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 204109286 ( 204,109,286) <= rx8_bytes /sec
Ethtool(enp175s0) stat: 76711 ( 76,711) <=
rx8_cache_reuse /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <=
rx8_csum_complete /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <= rx8_packets /sec
Ethtool(enp175s0) stat: 153481 ( 153,481) <=
rx8_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 183703752 ( 183,703,752) <= rx9_bytes /sec
Ethtool(enp175s0) stat: 71101 ( 71,101) <=
rx9_cache_reuse /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <=
rx9_csum_complete /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <= rx9_packets /sec
Ethtool(enp175s0) stat: 142172 ( 142,172) <=
rx9_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 2024072 ( 2,024,072) <=
rx_1024_to_1518_bytes_phy /sec
Ethtool(enp175s0) stat: 106582 ( 106,582) <=
rx_128_to_255_bytes_phy /sec
Ethtool(enp175s0) stat: 1296735 ( 1,296,735) <=
rx_1519_to_2047_bytes_phy /sec
Ethtool(enp175s0) stat: 59460 ( 59,460) <=
rx_256_to_511_bytes_phy /sec
Ethtool(enp175s0) stat: 57326 ( 57,326) <=
rx_512_to_1023_bytes_phy /sec
Ethtool(enp175s0) stat: 7159 ( 7,159) <=
rx_64_bytes_phy /sec
Ethtool(enp175s0) stat: 310730 ( 310,730) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp175s0) stat: 232 ( 232) <=
rx_broadcast_phy /sec
Ethtool(enp175s0) stat: 4850734036 ( 4,850,734,036) <= rx_bytes /sec
Ethtool(enp175s0) stat: 5069043007 ( 5,069,043,007) <= rx_bytes_phy /sec
Ethtool(enp175s0) stat: 1858636 ( 1,858,636) <= rx_cache_reuse
/sec
Ethtool(enp175s0) stat: 3717060 ( 3,717,060) <=
rx_csum_complete /sec
Ethtool(enp175s0) stat: 88 ( 88) <= rx_csum_none /sec
Ethtool(enp175s0) stat: 139602 ( 139,602) <=
rx_discards_phy /sec
Ethtool(enp175s0) stat: 354 ( 354) <=
rx_multicast_phy /sec
Ethtool(enp175s0) stat: 3717148 ( 3,717,148) <= rx_packets /sec
Ethtool(enp175s0) stat: 3862420 ( 3,862,420) <= rx_packets_phy
/sec
Ethtool(enp175s0) stat: 5063355121 ( 5,063,355,121) <= rx_prio0_bytes
/sec
Ethtool(enp175s0) stat: 3718759 ( 3,718,759) <=
rx_prio0_packets /sec
Ethtool(enp175s0) stat: 7193190 ( 7,193,190) <= rx_prio1_bytes
/sec
Ethtool(enp175s0) stat: 5031 ( 5,031) <=
rx_prio1_packets /sec
Ethtool(enp175s0) stat: 557 ( 557) <= rx_prio2_bytes
/sec
Ethtool(enp175s0) stat: 5 ( 5) <=
rx_prio2_packets /sec
Ethtool(enp175s0) stat: 61 ( 61) <= rx_prio3_bytes
/sec
Ethtool(enp175s0) stat: 1 ( 1) <=
rx_prio3_packets /sec
Ethtool(enp175s0) stat: 21010 ( 21,010) <= rx_prio4_bytes
/sec
Ethtool(enp175s0) stat: 39 ( 39) <=
rx_prio4_packets /sec
Ethtool(enp175s0) stat: 187 ( 187) <= rx_prio5_bytes
/sec
Ethtool(enp175s0) stat: 2 ( 2) <=
rx_prio5_packets /sec
Ethtool(enp175s0) stat: 1711 ( 1,711) <= rx_prio6_bytes
/sec
Ethtool(enp175s0) stat: 15 ( 15) <=
rx_prio6_packets /sec
Ethtool(enp175s0) stat: 19498 ( 19,498) <= rx_prio7_bytes
/sec
Ethtool(enp175s0) stat: 273 ( 273) <=
rx_prio7_packets /sec
Ethtool(enp175s0) stat: 3717148 ( 3,717,148) <=
rx_removed_vlan_packets /sec
Ethtool(enp175s0) stat: 5737 ( 5,737) <=
rx_steer_missed_packets /sec
Ethtool(enp175s0) stat: 14573 ( 14,573) <=
rx_vport_broadcast_bytes /sec
Ethtool(enp175s0) stat: 232 ( 232) <=
rx_vport_broadcast_packets /sec
Ethtool(enp175s0) stat: 25491 ( 25,491) <=
rx_vport_multicast_bytes /sec
Ethtool(enp175s0) stat: 354 ( 354) <=
rx_vport_multicast_packets /sec
Ethtool(enp175s0) stat: 4872354516 ( 4,872,354,516) <=
rx_vport_unicast_bytes /sec
Ethtool(enp175s0) stat: 3721920 ( 3,721,920) <=
rx_vport_unicast_packets /sec
Ethtool(enp175s0) stat: 158883 ( 158,883) <=
tx0_added_vlan_packets /sec
Ethtool(enp175s0) stat: 93790423 ( 93,790,423) <= tx0_bytes /sec
Ethtool(enp175s0) stat: 158854 ( 158,854) <= tx0_cqes /sec
Ethtool(enp175s0) stat: 146499 ( 146,499) <= tx0_csum_none /sec
Ethtool(enp175s0) stat: 12384 ( 12,384) <=
tx0_csum_partial /sec
Ethtool(enp175s0) stat: 2144 ( 2,144) <= tx0_nop /sec
Ethtool(enp175s0) stat: 188173 ( 188,173) <= tx0_packets /sec
Ethtool(enp175s0) stat: 53068613 ( 53,068,613) <= tx0_tso_bytes /sec
Ethtool(enp175s0) stat: 8839 ( 8,839) <=
tx0_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx0_xmit_more /sec
Ethtool(enp175s0) stat: 165538 ( 165,538) <=
tx10_added_vlan_packets /sec
Ethtool(enp175s0) stat: 102395057 ( 102,395,057) <= tx10_bytes /sec
Ethtool(enp175s0) stat: 165303 ( 165,303) <= tx10_cqes /sec
Ethtool(enp175s0) stat: 151089 ( 151,089) <= tx10_csum_none
/sec
Ethtool(enp175s0) stat: 14448 ( 14,448) <=
tx10_csum_partial /sec
Ethtool(enp175s0) stat: 2391 ( 2,391) <= tx10_nop /sec
Ethtool(enp175s0) stat: 198798 ( 198,798) <= tx10_packets /sec
Ethtool(enp175s0) stat: 58951449 ( 58,951,449) <= tx10_tso_bytes
/sec
Ethtool(enp175s0) stat: 8987 ( 8,987) <=
tx10_tso_packets /sec
Ethtool(enp175s0) stat: 234 ( 234) <= tx10_xmit_more
/sec
Ethtool(enp175s0) stat: 166402 ( 166,402) <=
tx11_added_vlan_packets /sec
Ethtool(enp175s0) stat: 98591304 ( 98,591,304) <= tx11_bytes /sec
Ethtool(enp175s0) stat: 166384 ( 166,384) <= tx11_cqes /sec
Ethtool(enp175s0) stat: 152456 ( 152,456) <= tx11_csum_none
/sec
Ethtool(enp175s0) stat: 13946 ( 13,946) <=
tx11_csum_partial /sec
Ethtool(enp175s0) stat: 2386 ( 2,386) <= tx11_nop /sec
Ethtool(enp175s0) stat: 201615 ( 201,615) <= tx11_packets /sec
Ethtool(enp175s0) stat: 63844660 ( 63,844,660) <= tx11_tso_bytes
/sec
Ethtool(enp175s0) stat: 10515 ( 10,515) <=
tx11_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx11_xmit_more
/sec
Ethtool(enp175s0) stat: 156312 ( 156,312) <=
tx12_added_vlan_packets /sec
Ethtool(enp175s0) stat: 97068537 ( 97,068,537) <= tx12_bytes /sec
Ethtool(enp175s0) stat: 156302 ( 156,302) <= tx12_cqes /sec
Ethtool(enp175s0) stat: 142468 ( 142,468) <= tx12_csum_none
/sec
Ethtool(enp175s0) stat: 13844 ( 13,844) <=
tx12_csum_partial /sec
Ethtool(enp175s0) stat: 2278 ( 2,278) <= tx12_nop /sec
Ethtool(enp175s0) stat: 187535 ( 187,535) <= tx12_packets /sec
Ethtool(enp175s0) stat: 58368798 ( 58,368,798) <= tx12_tso_bytes
/sec
Ethtool(enp175s0) stat: 10398 ( 10,398) <=
tx12_tso_packets /sec
Ethtool(enp175s0) stat: 10 ( 10) <= tx12_xmit_more
/sec
Ethtool(enp175s0) stat: 161768 ( 161,768) <=
tx13_added_vlan_packets /sec
Ethtool(enp175s0) stat: 120232518 ( 120,232,518) <= tx13_bytes /sec
Ethtool(enp175s0) stat: 161584 ( 161,584) <= tx13_cqes /sec
Ethtool(enp175s0) stat: 144388 ( 144,388) <= tx13_csum_none
/sec
Ethtool(enp175s0) stat: 17380 ( 17,380) <=
tx13_csum_partial /sec
Ethtool(enp175s0) stat: 2425 ( 2,425) <= tx13_nop /sec
Ethtool(enp175s0) stat: 202823 ( 202,823) <= tx13_packets /sec
Ethtool(enp175s0) stat: 72804507 ( 72,804,507) <= tx13_tso_bytes
/sec
Ethtool(enp175s0) stat: 10865 ( 10,865) <=
tx13_tso_packets /sec
Ethtool(enp175s0) stat: 185 ( 185) <= tx13_xmit_more
/sec
Ethtool(enp175s0) stat: 165762 ( 165,762) <=
tx14_added_vlan_packets /sec
Ethtool(enp175s0) stat: 99271622 ( 99,271,622) <= tx14_bytes /sec
Ethtool(enp175s0) stat: 165688 ( 165,688) <= tx14_cqes /sec
Ethtool(enp175s0) stat: 153195 ( 153,195) <= tx14_csum_none
/sec
Ethtool(enp175s0) stat: 12566 ( 12,566) <=
tx14_csum_partial /sec
Ethtool(enp175s0) stat: 2195 ( 2,195) <= tx14_nop /sec
Ethtool(enp175s0) stat: 195504 ( 195,504) <= tx14_packets /sec
Ethtool(enp175s0) stat: 53717277 ( 53,717,277) <= tx14_tso_bytes
/sec
Ethtool(enp175s0) stat: 8743 ( 8,743) <=
tx14_tso_packets /sec
Ethtool(enp175s0) stat: 32 ( 32) <= tx14_xmit_more
/sec
Ethtool(enp175s0) stat: 162803 ( 162,803) <=
tx15_added_vlan_packets /sec
Ethtool(enp175s0) stat: 105591893 ( 105,591,893) <= tx15_bytes /sec
Ethtool(enp175s0) stat: 162673 ( 162,673) <= tx15_cqes /sec
Ethtool(enp175s0) stat: 147080 ( 147,080) <= tx15_csum_none
/sec
Ethtool(enp175s0) stat: 15723 ( 15,723) <=
tx15_csum_partial /sec
Ethtool(enp175s0) stat: 2355 ( 2,355) <= tx15_nop /sec
Ethtool(enp175s0) stat: 198282 ( 198,282) <= tx15_packets /sec
Ethtool(enp175s0) stat: 64278573 ( 64,278,573) <= tx15_tso_bytes
/sec
Ethtool(enp175s0) stat: 10704 ( 10,704) <=
tx15_tso_packets /sec
Ethtool(enp175s0) stat: 183 ( 183) <= tx15_xmit_more
/sec
Ethtool(enp175s0) stat: 125282 ( 125,282) <=
tx16_added_vlan_packets /sec
Ethtool(enp175s0) stat: 81835815 ( 81,835,815) <= tx16_bytes /sec
Ethtool(enp175s0) stat: 125264 ( 125,264) <= tx16_cqes /sec
Ethtool(enp175s0) stat: 113284 ( 113,284) <= tx16_csum_none
/sec
Ethtool(enp175s0) stat: 11998 ( 11,998) <=
tx16_csum_partial /sec
Ethtool(enp175s0) stat: 1773 ( 1,773) <= tx16_nop /sec
Ethtool(enp175s0) stat: 150812 ( 150,812) <= tx16_packets /sec
Ethtool(enp175s0) stat: 46027767 ( 46,027,767) <= tx16_tso_bytes
/sec
Ethtool(enp175s0) stat: 7361 ( 7,361) <=
tx16_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx16_xmit_more
/sec
Ethtool(enp175s0) stat: 131865 ( 131,865) <=
tx17_added_vlan_packets /sec
Ethtool(enp175s0) stat: 89213555 ( 89,213,555) <= tx17_bytes /sec
Ethtool(enp175s0) stat: 131409 ( 131,409) <= tx17_cqes /sec
Ethtool(enp175s0) stat: 117199 ( 117,199) <= tx17_csum_none
/sec
Ethtool(enp175s0) stat: 14665 ( 14,665) <=
tx17_csum_partial /sec
Ethtool(enp175s0) stat: 1933 ( 1,933) <= tx17_nop /sec
Ethtool(enp175s0) stat: 161576 ( 161,576) <= tx17_packets /sec
Ethtool(enp175s0) stat: 53682297 ( 53,682,297) <= tx17_tso_bytes
/sec
Ethtool(enp175s0) stat: 8825 ( 8,825) <=
tx17_tso_packets /sec
Ethtool(enp175s0) stat: 459 ( 459) <= tx17_xmit_more
/sec
Ethtool(enp175s0) stat: 122492 ( 122,492) <=
tx18_added_vlan_packets /sec
Ethtool(enp175s0) stat: 87373476 ( 87,373,476) <= tx18_bytes /sec
Ethtool(enp175s0) stat: 122138 ( 122,138) <= tx18_cqes /sec
Ethtool(enp175s0) stat: 109037 ( 109,037) <= tx18_csum_none
/sec
Ethtool(enp175s0) stat: 13455 ( 13,455) <=
tx18_csum_partial /sec
Ethtool(enp175s0) stat: 1933 ( 1,933) <= tx18_nop /sec
Ethtool(enp175s0) stat: 152821 ( 152,821) <= tx18_packets /sec
Ethtool(enp175s0) stat: 54163102 ( 54,163,102) <= tx18_tso_bytes
/sec
Ethtool(enp175s0) stat: 8674 ( 8,674) <=
tx18_tso_packets /sec
Ethtool(enp175s0) stat: 354 ( 354) <= tx18_xmit_more
/sec
Ethtool(enp175s0) stat: 119294 ( 119,294) <=
tx19_added_vlan_packets /sec
Ethtool(enp175s0) stat: 67719609 ( 67,719,609) <= tx19_bytes /sec
Ethtool(enp175s0) stat: 119262 ( 119,262) <= tx19_cqes /sec
Ethtool(enp175s0) stat: 108591 ( 108,591) <= tx19_csum_none
/sec
Ethtool(enp175s0) stat: 10703 ( 10,703) <=
tx19_csum_partial /sec
Ethtool(enp175s0) stat: 1551 ( 1,551) <= tx19_nop /sec
Ethtool(enp175s0) stat: 141344 ( 141,344) <= tx19_packets /sec
Ethtool(enp175s0) stat: 39539778 ( 39,539,778) <= tx19_tso_bytes
/sec
Ethtool(enp175s0) stat: 6310 ( 6,310) <=
tx19_tso_packets /sec
Ethtool(enp175s0) stat: 31 ( 31) <= tx19_xmit_more
/sec
Ethtool(enp175s0) stat: 157094 ( 157,094) <=
tx1_added_vlan_packets /sec
Ethtool(enp175s0) stat: 93505806 ( 93,505,806) <= tx1_bytes /sec
Ethtool(enp175s0) stat: 156935 ( 156,935) <= tx1_cqes /sec
Ethtool(enp175s0) stat: 144805 ( 144,805) <= tx1_csum_none /sec
Ethtool(enp175s0) stat: 12289 ( 12,289) <=
tx1_csum_partial /sec
Ethtool(enp175s0) stat: 2201 ( 2,201) <= tx1_nop /sec
Ethtool(enp175s0) stat: 184561 ( 184,561) <= tx1_packets /sec
Ethtool(enp175s0) stat: 50513729 ( 50,513,729) <= tx1_tso_bytes /sec
Ethtool(enp175s0) stat: 8699 ( 8,699) <=
tx1_tso_packets /sec
Ethtool(enp175s0) stat: 159 ( 159) <= tx1_xmit_more /sec
Ethtool(enp175s0) stat: 134411 ( 134,411) <=
tx20_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88898658 ( 88,898,658) <= tx20_bytes /sec
Ethtool(enp175s0) stat: 134221 ( 134,221) <= tx20_cqes /sec
Ethtool(enp175s0) stat: 120787 ( 120,787) <= tx20_csum_none
/sec
Ethtool(enp175s0) stat: 13624 ( 13,624) <=
tx20_csum_partial /sec
Ethtool(enp175s0) stat: 2064 ( 2,064) <= tx20_nop /sec
Ethtool(enp175s0) stat: 165064 ( 165,064) <= tx20_packets /sec
Ethtool(enp175s0) stat: 54707259 ( 54,707,259) <= tx20_tso_bytes
/sec
Ethtool(enp175s0) stat: 8769 ( 8,769) <=
tx20_tso_packets /sec
Ethtool(enp175s0) stat: 188 ( 188) <= tx20_xmit_more
/sec
Ethtool(enp175s0) stat: 127029 ( 127,029) <=
tx21_added_vlan_packets /sec
Ethtool(enp175s0) stat: 98012630 ( 98,012,630) <= tx21_bytes /sec
Ethtool(enp175s0) stat: 126990 ( 126,990) <= tx21_cqes /sec
Ethtool(enp175s0) stat: 111553 ( 111,553) <= tx21_csum_none
/sec
Ethtool(enp175s0) stat: 15476 ( 15,476) <=
tx21_csum_partial /sec
Ethtool(enp175s0) stat: 2002 ( 2,002) <= tx21_nop /sec
Ethtool(enp175s0) stat: 159688 ( 159,688) <= tx21_packets /sec
Ethtool(enp175s0) stat: 59637304 ( 59,637,304) <= tx21_tso_bytes
/sec
Ethtool(enp175s0) stat: 9988 ( 9,988) <=
tx21_tso_packets /sec
Ethtool(enp175s0) stat: 39 ( 39) <= tx21_xmit_more
/sec
Ethtool(enp175s0) stat: 122610 ( 122,610) <=
tx22_added_vlan_packets /sec
Ethtool(enp175s0) stat: 70972052 ( 70,972,052) <= tx22_bytes /sec
Ethtool(enp175s0) stat: 122600 ( 122,600) <= tx22_cqes /sec
Ethtool(enp175s0) stat: 111526 ( 111,526) <= tx22_csum_none
/sec
Ethtool(enp175s0) stat: 11085 ( 11,085) <=
tx22_csum_partial /sec
Ethtool(enp175s0) stat: 1544 ( 1,544) <= tx22_nop /sec
Ethtool(enp175s0) stat: 144874 ( 144,874) <= tx22_packets /sec
Ethtool(enp175s0) stat: 40175057 ( 40,175,057) <= tx22_tso_bytes
/sec
Ethtool(enp175s0) stat: 6517 ( 6,517) <=
tx22_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx22_xmit_more
/sec
Ethtool(enp175s0) stat: 126809 ( 126,809) <=
tx23_added_vlan_packets /sec
Ethtool(enp175s0) stat: 76906314 ( 76,906,314) <= tx23_bytes /sec
Ethtool(enp175s0) stat: 126791 ( 126,791) <= tx23_cqes /sec
Ethtool(enp175s0) stat: 116656 ( 116,656) <= tx23_csum_none
/sec
Ethtool(enp175s0) stat: 10153 ( 10,153) <=
tx23_csum_partial /sec
Ethtool(enp175s0) stat: 1686 ( 1,686) <= tx23_nop /sec
Ethtool(enp175s0) stat: 148551 ( 148,551) <= tx23_packets /sec
Ethtool(enp175s0) stat: 39913764 ( 39,913,764) <= tx23_tso_bytes
/sec
Ethtool(enp175s0) stat: 6978 ( 6,978) <=
tx23_tso_packets /sec
Ethtool(enp175s0) stat: 18 ( 18) <= tx23_xmit_more
/sec
Ethtool(enp175s0) stat: 133910 ( 133,910) <=
tx24_added_vlan_packets /sec
Ethtool(enp175s0) stat: 69609913 ( 69,609,913) <= tx24_bytes /sec
Ethtool(enp175s0) stat: 133890 ( 133,890) <= tx24_cqes /sec
Ethtool(enp175s0) stat: 124462 ( 124,462) <= tx24_csum_none
/sec
Ethtool(enp175s0) stat: 9448 ( 9,448) <=
tx24_csum_partial /sec
Ethtool(enp175s0) stat: 1639 ( 1,639) <= tx24_nop /sec
Ethtool(enp175s0) stat: 154475 ( 154,475) <= tx24_packets /sec
Ethtool(enp175s0) stat: 37456148 ( 37,456,148) <= tx24_tso_bytes
/sec
Ethtool(enp175s0) stat: 6247 ( 6,247) <=
tx24_tso_packets /sec
Ethtool(enp175s0) stat: 20 ( 20) <= tx24_xmit_more
/sec
Ethtool(enp175s0) stat: 118528 ( 118,528) <=
tx25_added_vlan_packets /sec
Ethtool(enp175s0) stat: 62435525 ( 62,435,525) <= tx25_bytes /sec
Ethtool(enp175s0) stat: 118471 ( 118,471) <= tx25_cqes /sec
Ethtool(enp175s0) stat: 108887 ( 108,887) <= tx25_csum_none
/sec
Ethtool(enp175s0) stat: 9640 ( 9,640) <=
tx25_csum_partial /sec
Ethtool(enp175s0) stat: 1528 ( 1,528) <= tx25_nop /sec
Ethtool(enp175s0) stat: 138748 ( 138,748) <= tx25_packets /sec
Ethtool(enp175s0) stat: 36592045 ( 36,592,045) <= tx25_tso_bytes
/sec
Ethtool(enp175s0) stat: 5993 ( 5,993) <=
tx25_tso_packets /sec
Ethtool(enp175s0) stat: 44 ( 44) <= tx25_xmit_more
/sec
Ethtool(enp175s0) stat: 119890 ( 119,890) <=
tx26_added_vlan_packets /sec
Ethtool(enp175s0) stat: 76743929 ( 76,743,929) <= tx26_bytes /sec
Ethtool(enp175s0) stat: 119873 ( 119,873) <= tx26_cqes /sec
Ethtool(enp175s0) stat: 108181 ( 108,181) <= tx26_csum_none
/sec
Ethtool(enp175s0) stat: 11709 ( 11,709) <=
tx26_csum_partial /sec
Ethtool(enp175s0) stat: 1690 ( 1,690) <= tx26_nop /sec
Ethtool(enp175s0) stat: 144495 ( 144,495) <= tx26_packets /sec
Ethtool(enp175s0) stat: 43922304 ( 43,922,304) <= tx26_tso_bytes
/sec
Ethtool(enp175s0) stat: 7043 ( 7,043) <=
tx26_tso_packets /sec
Ethtool(enp175s0) stat: 17 ( 17) <= tx26_xmit_more
/sec
Ethtool(enp175s0) stat: 130825 ( 130,825) <=
tx27_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88723162 ( 88,723,162) <= tx27_bytes /sec
Ethtool(enp175s0) stat: 130769 ( 130,769) <= tx27_cqes /sec
Ethtool(enp175s0) stat: 116555 ( 116,555) <= tx27_csum_none
/sec
Ethtool(enp175s0) stat: 14270 ( 14,270) <=
tx27_csum_partial /sec
Ethtool(enp175s0) stat: 2042 ( 2,042) <= tx27_nop /sec
Ethtool(enp175s0) stat: 161228 ( 161,228) <= tx27_packets /sec
Ethtool(enp175s0) stat: 55637023 ( 55,637,023) <= tx27_tso_bytes
/sec
Ethtool(enp175s0) stat: 9707 ( 9,707) <=
tx27_tso_packets /sec
Ethtool(enp175s0) stat: 71 ( 71) <= tx27_xmit_more
/sec
Ethtool(enp175s0) stat: 166973 ( 166,973) <=
tx2_added_vlan_packets /sec
Ethtool(enp175s0) stat: 103659503 ( 103,659,503) <= tx2_bytes /sec
Ethtool(enp175s0) stat: 166941 ( 166,941) <= tx2_cqes /sec
Ethtool(enp175s0) stat: 151389 ( 151,389) <= tx2_csum_none /sec
Ethtool(enp175s0) stat: 15585 ( 15,585) <=
tx2_csum_partial /sec
Ethtool(enp175s0) stat: 2455 ( 2,455) <= tx2_nop /sec
Ethtool(enp175s0) stat: 201854 ( 201,854) <= tx2_packets /sec
Ethtool(enp175s0) stat: 65384298 ( 65,384,298) <= tx2_tso_bytes /sec
Ethtool(enp175s0) stat: 11809 ( 11,809) <=
tx2_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx2_xmit_more /sec
Ethtool(enp175s0) stat: 172353 ( 172,353) <=
tx3_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88248541 ( 88,248,541) <= tx3_bytes /sec
Ethtool(enp175s0) stat: 172277 ( 172,277) <= tx3_cqes /sec
Ethtool(enp175s0) stat: 160714 ( 160,714) <= tx3_csum_none /sec
Ethtool(enp175s0) stat: 11639 ( 11,639) <=
tx3_csum_partial /sec
Ethtool(enp175s0) stat: 2258 ( 2,258) <= tx3_nop /sec
Ethtool(enp175s0) stat: 199853 ( 199,853) <= tx3_packets /sec
Ethtool(enp175s0) stat: 49776463 ( 49,776,463) <= tx3_tso_bytes /sec
Ethtool(enp175s0) stat: 8353 ( 8,353) <=
tx3_tso_packets /sec
Ethtool(enp175s0) stat: 76 ( 76) <= tx3_xmit_more /sec
Ethtool(enp175s0) stat: 157527 ( 157,527) <=
tx4_added_vlan_packets /sec
Ethtool(enp175s0) stat: 110770979 ( 110,770,979) <= tx4_bytes /sec
Ethtool(enp175s0) stat: 157492 ( 157,492) <= tx4_cqes /sec
Ethtool(enp175s0) stat: 141858 ( 141,858) <= tx4_csum_none /sec
Ethtool(enp175s0) stat: 15670 ( 15,670) <=
tx4_csum_partial /sec
Ethtool(enp175s0) stat: 2320 ( 2,320) <= tx4_nop /sec
Ethtool(enp175s0) stat: 192429 ( 192,429) <= tx4_packets /sec
Ethtool(enp175s0) stat: 64689503 ( 64,689,503) <= tx4_tso_bytes /sec
Ethtool(enp175s0) stat: 11367 ( 11,367) <=
tx4_tso_packets /sec
Ethtool(enp175s0) stat: 35 ( 35) <= tx4_xmit_more /sec
Ethtool(enp175s0) stat: 169077 ( 169,077) <=
tx5_added_vlan_packets /sec
Ethtool(enp175s0) stat: 121536690 ( 121,536,690) <= tx5_bytes /sec
Ethtool(enp175s0) stat: 168906 ( 168,906) <= tx5_cqes /sec
Ethtool(enp175s0) stat: 150099 ( 150,099) <= tx5_csum_none /sec
Ethtool(enp175s0) stat: 18978 ( 18,978) <=
tx5_csum_partial /sec
Ethtool(enp175s0) stat: 2678 ( 2,678) <= tx5_nop /sec
Ethtool(enp175s0) stat: 210733 ( 210,733) <= tx5_packets /sec
Ethtool(enp175s0) stat: 76448346 ( 76,448,346) <= tx5_tso_bytes /sec
Ethtool(enp175s0) stat: 13238 ( 13,238) <=
tx5_tso_packets /sec
Ethtool(enp175s0) stat: 171 ( 171) <= tx5_xmit_more /sec
Ethtool(enp175s0) stat: 156881 ( 156,881) <=
tx6_added_vlan_packets /sec
Ethtool(enp175s0) stat: 85752393 ( 85,752,393) <= tx6_bytes /sec
Ethtool(enp175s0) stat: 156859 ( 156,859) <= tx6_cqes /sec
Ethtool(enp175s0) stat: 144843 ( 144,843) <= tx6_csum_none /sec
Ethtool(enp175s0) stat: 12038 ( 12,038) <=
tx6_csum_partial /sec
Ethtool(enp175s0) stat: 2034 ( 2,034) <= tx6_nop /sec
Ethtool(enp175s0) stat: 181315 ( 181,315) <= tx6_packets /sec
Ethtool(enp175s0) stat: 44625307 ( 44,625,307) <= tx6_tso_bytes /sec
Ethtool(enp175s0) stat: 7752 ( 7,752) <=
tx6_tso_packets /sec
Ethtool(enp175s0) stat: 24 ( 24) <= tx6_xmit_more /sec
Ethtool(enp175s0) stat: 157744 ( 157,744) <=
tx7_added_vlan_packets /sec
Ethtool(enp175s0) stat: 88795890 ( 88,795,890) <= tx7_bytes /sec
Ethtool(enp175s0) stat: 157666 ( 157,666) <= tx7_cqes /sec
Ethtool(enp175s0) stat: 146023 ( 146,023) <= tx7_csum_none /sec
Ethtool(enp175s0) stat: 11720 ( 11,720) <=
tx7_csum_partial /sec
Ethtool(enp175s0) stat: 2190 ( 2,190) <= tx7_nop /sec
Ethtool(enp175s0) stat: 184845 ( 184,845) <= tx7_packets /sec
Ethtool(enp175s0) stat: 48986735 ( 48,986,735) <= tx7_tso_bytes /sec
Ethtool(enp175s0) stat: 8401 ( 8,401) <=
tx7_tso_packets /sec
Ethtool(enp175s0) stat: 78 ( 78) <= tx7_xmit_more /sec
Ethtool(enp175s0) stat: 165190 ( 165,190) <=
tx8_added_vlan_packets /sec
Ethtool(enp175s0) stat: 108303283 ( 108,303,283) <= tx8_bytes /sec
Ethtool(enp175s0) stat: 165145 ( 165,145) <= tx8_cqes /sec
Ethtool(enp175s0) stat: 150047 ( 150,047) <= tx8_csum_none /sec
Ethtool(enp175s0) stat: 15143 ( 15,143) <=
tx8_csum_partial /sec
Ethtool(enp175s0) stat: 2497 ( 2,497) <= tx8_nop /sec
Ethtool(enp175s0) stat: 201615 ( 201,615) <= tx8_packets /sec
Ethtool(enp175s0) stat: 67575199 ( 67,575,199) <= tx8_tso_bytes /sec
Ethtool(enp175s0) stat: 11828 ( 11,828) <=
tx8_tso_packets /sec
Ethtool(enp175s0) stat: 45 ( 45) <= tx8_xmit_more /sec
Ethtool(enp175s0) stat: 168256 ( 168,256) <=
tx9_added_vlan_packets /sec
Ethtool(enp175s0) stat: 115981227 ( 115,981,227) <= tx9_bytes /sec
Ethtool(enp175s0) stat: 168225 ( 168,225) <= tx9_cqes /sec
Ethtool(enp175s0) stat: 147980 ( 147,980) <= tx9_csum_none /sec
Ethtool(enp175s0) stat: 20275 ( 20,275) <=
tx9_csum_partial /sec
Ethtool(enp175s0) stat: 2522 ( 2,522) <= tx9_nop /sec
Ethtool(enp175s0) stat: 203398 ( 203,398) <= tx9_packets /sec
Ethtool(enp175s0) stat: 63891360 ( 63,891,360) <= tx9_tso_bytes /sec
Ethtool(enp175s0) stat: 10610 ( 10,610) <=
tx9_tso_packets /sec
Ethtool(enp175s0) stat: 30 ( 30) <= tx9_xmit_more /sec
Ethtool(enp175s0) stat: 4121491 ( 4,121,491) <=
tx_added_vlan_packets /sec
Ethtool(enp175s0) stat: 1 ( 1) <=
tx_broadcast_phy /sec
Ethtool(enp175s0) stat: 2591851137 ( 2,591,851,137) <= tx_bytes /sec
Ethtool(enp175s0) stat: 2626884154 ( 2,626,884,154) <= tx_bytes_phy /sec
Ethtool(enp175s0) stat: 4118829 ( 4,118,829) <= tx_cqes /sec
Ethtool(enp175s0) stat: 3741682 ( 3,741,682) <= tx_csum_none /sec
Ethtool(enp175s0) stat: 379810 ( 379,810) <=
tx_csum_partial /sec
Ethtool(enp175s0) stat: 58718 ( 58,718) <= tx_nop /sec
Ethtool(enp175s0) stat: 4956995 ( 4,956,995) <= tx_packets /sec
Ethtool(enp175s0) stat: 4956579 ( 4,956,579) <= tx_packets_phy
/sec
Ethtool(enp175s0) stat: 2627152852 ( 2,627,152,852) <= tx_prio0_bytes
/sec
Ethtool(enp175s0) stat: 4957649 ( 4,957,649) <=
tx_prio0_packets /sec
Ethtool(enp175s0) stat: 1518383905 ( 1,518,383,905) <= tx_tso_bytes /sec
Ethtool(enp175s0) stat: 253526 ( 253,526) <= tx_tso_packets
/sec
Ethtool(enp175s0) stat: 57 ( 57) <=
tx_vport_broadcast_bytes /sec
Ethtool(enp175s0) stat: 1 ( 1) <=
tx_vport_broadcast_packets /sec
Ethtool(enp175s0) stat: 2607061269 ( 2,607,061,269) <=
tx_vport_unicast_bytes /sec
Ethtool(enp175s0) stat: 4956492 ( 4,956,492) <=
tx_vport_unicast_packets /sec
Ethtool(enp175s0) stat: 2630 ( 2,630) <= tx_xmit_more /sec
Ethtool(enp216s0) stat: 6923 ( 6,923) <= ch0_arm /sec
Ethtool(enp216s0) stat: 8367 ( 8,367) <= ch0_events /sec
Ethtool(enp216s0) stat: 9797 ( 9,797) <= ch0_poll /sec
Ethtool(enp216s0) stat: 7703 ( 7,703) <= ch10_arm /sec
Ethtool(enp216s0) stat: 9253 ( 9,253) <= ch10_events /sec
Ethtool(enp216s0) stat: 10807 ( 10,807) <= ch10_poll /sec
Ethtool(enp216s0) stat: 7886 ( 7,886) <= ch11_arm /sec
Ethtool(enp216s0) stat: 9391 ( 9,391) <= ch11_events /sec
Ethtool(enp216s0) stat: 10896 ( 10,896) <= ch11_poll /sec
Ethtool(enp216s0) stat: 9793 ( 9,793) <= ch12_arm /sec
Ethtool(enp216s0) stat: 11299 ( 11,299) <= ch12_events /sec
Ethtool(enp216s0) stat: 12637 ( 12,637) <= ch12_poll /sec
Ethtool(enp216s0) stat: 9119 ( 9,119) <= ch13_arm /sec
Ethtool(enp216s0) stat: 10671 ( 10,671) <= ch13_events /sec
Ethtool(enp216s0) stat: 12205 ( 12,205) <= ch13_poll /sec
Ethtool(enp216s0) stat: 8784 ( 8,784) <= ch14_arm /sec
Ethtool(enp216s0) stat: 10189 ( 10,189) <= ch14_events /sec
Ethtool(enp216s0) stat: 11565 ( 11,565) <= ch14_poll /sec
Ethtool(enp216s0) stat: 9248 ( 9,248) <= ch15_arm /sec
Ethtool(enp216s0) stat: 10739 ( 10,739) <= ch15_events /sec
Ethtool(enp216s0) stat: 12238 ( 12,238) <= ch15_poll /sec
Ethtool(enp216s0) stat: 5201 ( 5,201) <= ch16_arm /sec
Ethtool(enp216s0) stat: 6554 ( 6,554) <= ch16_events /sec
Ethtool(enp216s0) stat: 7667 ( 7,667) <= ch16_poll /sec
Ethtool(enp216s0) stat: 7799 ( 7,799) <= ch17_arm /sec
Ethtool(enp216s0) stat: 9120 ( 9,120) <= ch17_events /sec
Ethtool(enp216s0) stat: 10194 ( 10,194) <= ch17_poll /sec
Ethtool(enp216s0) stat: 6889 ( 6,889) <= ch18_arm /sec
Ethtool(enp216s0) stat: 8266 ( 8,266) <= ch18_events /sec
Ethtool(enp216s0) stat: 9332 ( 9,332) <= ch18_poll /sec
Ethtool(enp216s0) stat: 5494 ( 5,494) <= ch19_arm /sec
Ethtool(enp216s0) stat: 6813 ( 6,813) <= ch19_events /sec
Ethtool(enp216s0) stat: 7837 ( 7,837) <= ch19_poll /sec
Ethtool(enp216s0) stat: 6672 ( 6,672) <= ch1_arm /sec
Ethtool(enp216s0) stat: 8124 ( 8,124) <= ch1_events /sec
Ethtool(enp216s0) stat: 9604 ( 9,604) <= ch1_poll /sec
Ethtool(enp216s0) stat: 7705 ( 7,705) <= ch20_arm /sec
Ethtool(enp216s0) stat: 9102 ( 9,102) <= ch20_events /sec
Ethtool(enp216s0) stat: 10287 ( 10,287) <= ch20_poll /sec
Ethtool(enp216s0) stat: 5929 ( 5,929) <= ch21_arm /sec
Ethtool(enp216s0) stat: 7333 ( 7,333) <= ch21_events /sec
Ethtool(enp216s0) stat: 8463 ( 8,463) <= ch21_poll /sec
Ethtool(enp216s0) stat: 5495 ( 5,495) <= ch22_arm /sec
Ethtool(enp216s0) stat: 6813 ( 6,813) <= ch22_events /sec
Ethtool(enp216s0) stat: 7843 ( 7,843) <= ch22_poll /sec
Ethtool(enp216s0) stat: 7091 ( 7,091) <= ch23_arm /sec
Ethtool(enp216s0) stat: 8367 ( 8,367) <= ch23_events /sec
Ethtool(enp216s0) stat: 9344 ( 9,344) <= ch23_poll /sec
Ethtool(enp216s0) stat: 5481 ( 5,481) <= ch24_arm /sec
Ethtool(enp216s0) stat: 6879 ( 6,879) <= ch24_events /sec
Ethtool(enp216s0) stat: 7995 ( 7,995) <= ch24_poll /sec
Ethtool(enp216s0) stat: 5642 ( 5,642) <= ch25_arm /sec
Ethtool(enp216s0) stat: 6959 ( 6,959) <= ch25_events /sec
Ethtool(enp216s0) stat: 7927 ( 7,927) <= ch25_poll /sec
Ethtool(enp216s0) stat: 5289 ( 5,289) <= ch26_arm /sec
Ethtool(enp216s0) stat: 6643 ( 6,643) <= ch26_events /sec
Ethtool(enp216s0) stat: 7691 ( 7,691) <= ch26_poll /sec
Ethtool(enp216s0) stat: 7313 ( 7,313) <= ch27_arm /sec
Ethtool(enp216s0) stat: 8719 ( 8,719) <= ch27_events /sec
Ethtool(enp216s0) stat: 9876 ( 9,876) <= ch27_poll /sec
Ethtool(enp216s0) stat: 7791 ( 7,791) <= ch2_arm /sec
Ethtool(enp216s0) stat: 9328 ( 9,328) <= ch2_events /sec
Ethtool(enp216s0) stat: 10838 ( 10,838) <= ch2_poll /sec
Ethtool(enp216s0) stat: 9595 ( 9,595) <= ch3_arm /sec
Ethtool(enp216s0) stat: 11015 ( 11,015) <= ch3_events /sec
Ethtool(enp216s0) stat: 12402 ( 12,402) <= ch3_poll /sec
Ethtool(enp216s0) stat: 9653 ( 9,653) <= ch4_arm /sec
Ethtool(enp216s0) stat: 11163 ( 11,163) <= ch4_events /sec
Ethtool(enp216s0) stat: 12553 ( 12,553) <= ch4_poll /sec
Ethtool(enp216s0) stat: 10269 ( 10,269) <= ch5_arm /sec
Ethtool(enp216s0) stat: 11727 ( 11,727) <= ch5_events /sec
Ethtool(enp216s0) stat: 13160 ( 13,160) <= ch5_poll /sec
Ethtool(enp216s0) stat: 8806 ( 8,806) <= ch6_arm /sec
Ethtool(enp216s0) stat: 10191 ( 10,191) <= ch6_events /sec
Ethtool(enp216s0) stat: 11431 ( 11,431) <= ch6_poll /sec
Ethtool(enp216s0) stat: 6866 ( 6,866) <= ch7_arm /sec
Ethtool(enp216s0) stat: 8412 ( 8,412) <= ch7_events /sec
Ethtool(enp216s0) stat: 9815 ( 9,815) <= ch7_poll /sec
Ethtool(enp216s0) stat: 10533 ( 10,533) <= ch8_arm /sec
Ethtool(enp216s0) stat: 11945 ( 11,945) <= ch8_events /sec
Ethtool(enp216s0) stat: 13286 ( 13,286) <= ch8_poll /sec
Ethtool(enp216s0) stat: 7126 ( 7,126) <= ch9_arm /sec
Ethtool(enp216s0) stat: 8643 ( 8,643) <= ch9_events /sec
Ethtool(enp216s0) stat: 10210 ( 10,210) <= ch9_poll /sec
Ethtool(enp216s0) stat: 212206 ( 212,206) <= ch_arm /sec
Ethtool(enp216s0) stat: 252118 ( 252,118) <= ch_events /sec
Ethtool(enp216s0) stat: 288009 ( 288,009) <= ch_poll /sec
Ethtool(enp216s0) stat: 93281134 ( 93,281,134) <= rx0_bytes /sec
Ethtool(enp216s0) stat: 94200 ( 94,200) <=
rx0_cache_reuse /sec
Ethtool(enp216s0) stat: 188327 ( 188,327) <=
rx0_csum_complete /sec
Ethtool(enp216s0) stat: 7 ( 7) <= rx0_csum_none /sec
Ethtool(enp216s0) stat: 188334 ( 188,334) <= rx0_packets /sec
Ethtool(enp216s0) stat: 188334 ( 188,334) <=
rx0_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 102443052 ( 102,443,052) <= rx10_bytes /sec
Ethtool(enp216s0) stat: 99816 ( 99,816) <=
rx10_cache_reuse /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <=
rx10_csum_complete /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <= rx10_packets /sec
Ethtool(enp216s0) stat: 199616 ( 199,616) <=
rx10_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 97655592 ( 97,655,592) <= rx11_bytes /sec
Ethtool(enp216s0) stat: 100854 ( 100,854) <=
rx11_cache_reuse /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <=
rx11_csum_complete /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <= rx11_packets /sec
Ethtool(enp216s0) stat: 201655 ( 201,655) <=
rx11_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 97076715 ( 97,076,715) <= rx12_bytes /sec
Ethtool(enp216s0) stat: 94078 ( 94,078) <=
rx12_cache_reuse /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <=
rx12_csum_complete /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <= rx12_packets /sec
Ethtool(enp216s0) stat: 188037 ( 188,037) <=
rx12_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 120547921 ( 120,547,921) <= rx13_bytes /sec
Ethtool(enp216s0) stat: 101709 ( 101,709) <=
rx13_cache_reuse /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <=
rx13_csum_complete /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <= rx13_packets /sec
Ethtool(enp216s0) stat: 203473 ( 203,473) <=
rx13_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 98367936 ( 98,367,936) <= rx14_bytes /sec
Ethtool(enp216s0) stat: 97741 ( 97,741) <=
rx14_cache_reuse /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <=
rx14_csum_complete /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <= rx14_packets /sec
Ethtool(enp216s0) stat: 195506 ( 195,506) <=
rx14_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 106726542 ( 106,726,542) <= rx15_bytes /sec
Ethtool(enp216s0) stat: 99694 ( 99,694) <=
rx15_cache_reuse /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <=
rx15_csum_complete /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <= rx15_packets /sec
Ethtool(enp216s0) stat: 199395 ( 199,395) <=
rx15_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 81928969 ( 81,928,969) <= rx16_bytes /sec
Ethtool(enp216s0) stat: 75580 ( 75,580) <=
rx16_cache_reuse /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <=
rx16_csum_complete /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <= rx16_packets /sec
Ethtool(enp216s0) stat: 151139 ( 151,139) <=
rx16_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 90509227 ( 90,509,227) <= rx17_bytes /sec
Ethtool(enp216s0) stat: 81196 ( 81,196) <=
rx17_cache_reuse /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <=
rx17_csum_complete /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <= rx17_packets /sec
Ethtool(enp216s0) stat: 162403 ( 162,403) <=
rx17_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 87854385 ( 87,854,385) <= rx18_bytes /sec
Ethtool(enp216s0) stat: 76923 ( 76,923) <=
rx18_cache_reuse /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <=
rx18_csum_complete /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <= rx18_packets /sec
Ethtool(enp216s0) stat: 153866 ( 153,866) <=
rx18_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 67849725 ( 67,849,725) <= rx19_bytes /sec
Ethtool(enp216s0) stat: 71001 ( 71,001) <=
rx19_cache_reuse /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <=
rx19_csum_complete /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <= rx19_packets /sec
Ethtool(enp216s0) stat: 142064 ( 142,064) <=
rx19_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 92611021 ( 92,611,021) <= rx1_bytes /sec
Ethtool(enp216s0) stat: 92307 ( 92,307) <=
rx1_cache_reuse /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <=
rx1_csum_complete /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <= rx1_packets /sec
Ethtool(enp216s0) stat: 184639 ( 184,639) <=
rx1_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88902617 ( 88,902,617) <= rx20_bytes /sec
Ethtool(enp216s0) stat: 82844 ( 82,844) <=
rx20_cache_reuse /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <=
rx20_csum_complete /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <= rx20_packets /sec
Ethtool(enp216s0) stat: 165764 ( 165,764) <=
rx20_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 98096942 ( 98,096,942) <= rx21_bytes /sec
Ethtool(enp216s0) stat: 79975 ( 79,975) <=
rx21_cache_reuse /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <=
rx21_csum_complete /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <= rx21_packets /sec
Ethtool(enp216s0) stat: 159908 ( 159,908) <=
rx21_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 71885445 ( 71,885,445) <= rx22_bytes /sec
Ethtool(enp216s0) stat: 73565 ( 73,565) <=
rx22_cache_reuse /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <=
rx22_csum_complete /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <= rx22_packets /sec
Ethtool(enp216s0) stat: 147136 ( 147,136) <=
rx22_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 77030721 ( 77,030,721) <= rx23_bytes /sec
Ethtool(enp216s0) stat: 74481 ( 74,481) <=
rx23_cache_reuse /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <=
rx23_csum_complete /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <= rx23_packets /sec
Ethtool(enp216s0) stat: 148989 ( 148,989) <=
rx23_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 69603951 ( 69,603,951) <= rx24_bytes /sec
Ethtool(enp216s0) stat: 77472 ( 77,472) <=
rx24_cache_reuse /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <=
rx24_csum_complete /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <= rx24_packets /sec
Ethtool(enp216s0) stat: 154916 ( 154,916) <=
rx24_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 62277522 ( 62,277,522) <= rx25_bytes /sec
Ethtool(enp216s0) stat: 69414 ( 69,414) <=
rx25_cache_reuse /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <=
rx25_csum_complete /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <= rx25_packets /sec
Ethtool(enp216s0) stat: 138835 ( 138,835) <=
rx25_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 77498258 ( 77,498,258) <= rx26_bytes /sec
Ethtool(enp216s0) stat: 72466 ( 72,466) <=
rx26_cache_reuse /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <=
rx26_csum_complete /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <= rx26_packets /sec
Ethtool(enp216s0) stat: 144925 ( 144,925) <=
rx26_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88962856 ( 88,962,856) <= rx27_bytes /sec
Ethtool(enp216s0) stat: 80952 ( 80,952) <=
rx27_cache_reuse /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <=
rx27_csum_complete /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <= rx27_packets /sec
Ethtool(enp216s0) stat: 161879 ( 161,879) <=
rx27_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 103624977 ( 103,624,977) <= rx2_bytes /sec
Ethtool(enp216s0) stat: 101098 ( 101,098) <=
rx2_cache_reuse /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <=
rx2_csum_complete /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <= rx2_packets /sec
Ethtool(enp216s0) stat: 202130 ( 202,130) <=
rx2_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 87213368 ( 87,213,368) <= rx3_bytes /sec
Ethtool(enp216s0) stat: 99877 ( 99,877) <=
rx3_cache_reuse /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <=
rx3_csum_complete /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <= rx3_packets /sec
Ethtool(enp216s0) stat: 199778 ( 199,778) <=
rx3_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 110458845 ( 110,458,845) <= rx4_bytes /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx4_cache_busy
/sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx4_cache_full
/sec
Ethtool(enp216s0) stat: 95523 ( 95,523) <=
rx4_cache_reuse /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <=
rx4_csum_complete /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <= rx4_packets /sec
Ethtool(enp216s0) stat: 192402 ( 192,402) <=
rx4_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 121222062 ( 121,222,062) <= rx5_bytes /sec
Ethtool(enp216s0) stat: 105616 ( 105,616) <=
rx5_cache_reuse /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <=
rx5_csum_complete /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <= rx5_packets /sec
Ethtool(enp216s0) stat: 211273 ( 211,273) <=
rx5_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 85541470 ( 85,541,470) <= rx6_bytes /sec
Ethtool(enp216s0) stat: 91147 ( 91,147) <=
rx6_cache_reuse /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <=
rx6_csum_complete /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <= rx6_packets /sec
Ethtool(enp216s0) stat: 182257 ( 182,257) <=
rx6_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 88443649 ( 88,443,649) <= rx7_bytes /sec
Ethtool(enp216s0) stat: 92368 ( 92,368) <=
rx7_cache_reuse /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <=
rx7_csum_complete /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <= rx7_packets /sec
Ethtool(enp216s0) stat: 184828 ( 184,828) <=
rx7_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 108419072 ( 108,419,072) <= rx8_bytes /sec
Ethtool(enp216s0) stat: 101098 ( 101,098) <=
rx8_cache_reuse /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <=
rx8_csum_complete /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <= rx8_packets /sec
Ethtool(enp216s0) stat: 202241 ( 202,241) <=
rx8_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 116210958 ( 116,210,958) <= rx9_bytes /sec
Ethtool(enp216s0) stat: 102014 ( 102,014) <=
rx9_cache_reuse /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <=
rx9_csum_complete /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <= rx9_packets /sec
Ethtool(enp216s0) stat: 204092 ( 204,092) <=
rx9_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 1065697 ( 1,065,697) <=
rx_1024_to_1518_bytes_phy /sec
Ethtool(enp216s0) stat: 141705 ( 141,705) <=
rx_128_to_255_bytes_phy /sec
Ethtool(enp216s0) stat: 512465 ( 512,465) <=
rx_1519_to_2047_bytes_phy /sec
Ethtool(enp216s0) stat: 49523 ( 49,523) <=
rx_256_to_511_bytes_phy /sec
Ethtool(enp216s0) stat: 56321 ( 56,321) <=
rx_512_to_1023_bytes_phy /sec
Ethtool(enp216s0) stat: 957286 ( 957,286) <=
rx_64_bytes_phy /sec
Ethtool(enp216s0) stat: 2193644 ( 2,193,644) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_broadcast_phy /sec
Ethtool(enp216s0) stat: 2592286809 ( 2,592,286,809) <= rx_bytes /sec
Ethtool(enp216s0) stat: 2633575771 ( 2,633,575,771) <= rx_bytes_phy /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx_cache_busy /sec
Ethtool(enp216s0) stat: 692 ( 692) <= rx_cache_full /sec
Ethtool(enp216s0) stat: 2484928 ( 2,484,928) <= rx_cache_reuse
/sec
Ethtool(enp216s0) stat: 4971670 ( 4,971,670) <=
rx_csum_complete /sec
Ethtool(enp216s0) stat: 7 ( 7) <= rx_csum_none /sec
Ethtool(enp216s0) stat: 464 ( 464) <=
rx_discards_phy /sec
Ethtool(enp216s0) stat: 1376 ( 1,376) <=
rx_multicast_phy /sec
Ethtool(enp216s0) stat: 4971677 ( 4,971,677) <= rx_packets /sec
Ethtool(enp216s0) stat: 4975563 ( 4,975,563) <= rx_packets_phy
/sec
Ethtool(enp216s0) stat: 2634727119 ( 2,634,727,119) <= rx_prio0_bytes
/sec
Ethtool(enp216s0) stat: 4975012 ( 4,975,012) <=
rx_prio0_packets /sec
Ethtool(enp216s0) stat: 61 ( 61) <= rx_prio1_bytes
/sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_prio1_packets /sec
Ethtool(enp216s0) stat: 2003 ( 2,003) <= rx_prio3_bytes
/sec
Ethtool(enp216s0) stat: 7 ( 7) <=
rx_prio3_packets /sec
Ethtool(enp216s0) stat: 2497 ( 2,497) <= rx_prio4_bytes
/sec
Ethtool(enp216s0) stat: 3 ( 3) <=
rx_prio4_packets /sec
Ethtool(enp216s0) stat: 10697 ( 10,697) <= rx_prio5_bytes
/sec
Ethtool(enp216s0) stat: 163 ( 163) <=
rx_prio5_packets /sec
Ethtool(enp216s0) stat: 3519 ( 3,519) <= rx_prio6_bytes
/sec
Ethtool(enp216s0) stat: 26 ( 26) <=
rx_prio6_packets /sec
Ethtool(enp216s0) stat: 95583 ( 95,583) <= rx_prio7_bytes
/sec
Ethtool(enp216s0) stat: 1365 ( 1,365) <=
rx_prio7_packets /sec
Ethtool(enp216s0) stat: 4971677 ( 4,971,677) <=
rx_removed_vlan_packets /sec
Ethtool(enp216s0) stat: 1377 ( 1,377) <=
rx_steer_missed_packets /sec
Ethtool(enp216s0) stat: 150 ( 150) <=
rx_vport_broadcast_bytes /sec
Ethtool(enp216s0) stat: 1 ( 1) <=
rx_vport_broadcast_packets /sec
Ethtool(enp216s0) stat: 90890 ( 90,890) <=
rx_vport_multicast_bytes /sec
Ethtool(enp216s0) stat: 1376 ( 1,376) <=
rx_vport_multicast_packets /sec
Ethtool(enp216s0) stat: 2612989221 ( 2,612,989,221) <=
rx_vport_unicast_bytes /sec
Ethtool(enp216s0) stat: 4973191 ( 4,973,191) <=
rx_vport_unicast_packets /sec
Ethtool(enp216s0) stat: 82580 ( 82,580) <=
tx0_added_vlan_packets /sec
Ethtool(enp216s0) stat: 186927644 ( 186,927,644) <= tx0_bytes /sec
Ethtool(enp216s0) stat: 81347 ( 81,347) <= tx0_cqes /sec
Ethtool(enp216s0) stat: 49836 ( 49,836) <= tx0_csum_none /sec
Ethtool(enp216s0) stat: 32745 ( 32,745) <=
tx0_csum_partial /sec
Ethtool(enp216s0) stat: 2093 ( 2,093) <= tx0_nop /sec
Ethtool(enp216s0) stat: 146450 ( 146,450) <= tx0_packets /sec
Ethtool(enp216s0) stat: 131547357 ( 131,547,357) <= tx0_tso_bytes /sec
Ethtool(enp216s0) stat: 28323 ( 28,323) <=
tx0_tso_packets /sec
Ethtool(enp216s0) stat: 1226 ( 1,226) <= tx0_xmit_more /sec
Ethtool(enp216s0) stat: 85851 ( 85,851) <=
tx10_added_vlan_packets /sec
Ethtool(enp216s0) stat: 198440153 ( 198,440,153) <= tx10_bytes /sec
Ethtool(enp216s0) stat: 84107 ( 84,107) <= tx10_cqes /sec
Ethtool(enp216s0) stat: 51644 ( 51,644) <= tx10_csum_none
/sec
Ethtool(enp216s0) stat: 34207 ( 34,207) <=
tx10_csum_partial /sec
Ethtool(enp216s0) stat: 2256 ( 2,256) <= tx10_nop /sec
Ethtool(enp216s0) stat: 151642 ( 151,642) <= tx10_packets /sec
Ethtool(enp216s0) stat: 135145744 ( 135,145,744) <= tx10_tso_bytes
/sec
Ethtool(enp216s0) stat: 29140 ( 29,140) <=
tx10_tso_packets /sec
Ethtool(enp216s0) stat: 1739 ( 1,739) <= tx10_xmit_more
/sec
Ethtool(enp216s0) stat: 85526 ( 85,526) <=
tx11_added_vlan_packets /sec
Ethtool(enp216s0) stat: 199463985 ( 199,463,985) <= tx11_bytes /sec
Ethtool(enp216s0) stat: 83793 ( 83,793) <= tx11_cqes /sec
Ethtool(enp216s0) stat: 50188 ( 50,188) <= tx11_csum_none
/sec
Ethtool(enp216s0) stat: 35338 ( 35,338) <=
tx11_csum_partial /sec
Ethtool(enp216s0) stat: 2227 ( 2,227) <= tx11_nop /sec
Ethtool(enp216s0) stat: 152815 ( 152,815) <= tx11_packets /sec
Ethtool(enp216s0) stat: 138134263 ( 138,134,263) <= tx11_tso_bytes
/sec
Ethtool(enp216s0) stat: 29948 ( 29,948) <=
tx11_tso_packets /sec
Ethtool(enp216s0) stat: 1733 ( 1,733) <= tx11_xmit_more
/sec
Ethtool(enp216s0) stat: 77464 ( 77,464) <=
tx12_added_vlan_packets /sec
Ethtool(enp216s0) stat: 185723223 ( 185,723,223) <= tx12_bytes /sec
Ethtool(enp216s0) stat: 75907 ( 75,907) <= tx12_cqes /sec
Ethtool(enp216s0) stat: 43727 ( 43,727) <= tx12_csum_none
/sec
Ethtool(enp216s0) stat: 33738 ( 33,738) <=
tx12_csum_partial /sec
Ethtool(enp216s0) stat: 2060 ( 2,060) <= tx12_nop /sec
Ethtool(enp216s0) stat: 141903 ( 141,903) <= tx12_packets /sec
Ethtool(enp216s0) stat: 133227574 ( 133,227,574) <= tx12_tso_bytes
/sec
Ethtool(enp216s0) stat: 29139 ( 29,139) <=
tx12_tso_packets /sec
Ethtool(enp216s0) stat: 1559 ( 1,559) <= tx12_xmit_more
/sec
Ethtool(enp216s0) stat: 79682 ( 79,682) <=
tx13_added_vlan_packets /sec
Ethtool(enp216s0) stat: 189943899 ( 189,943,899) <= tx13_bytes /sec
Ethtool(enp216s0) stat: 78110 ( 78,110) <= tx13_cqes /sec
Ethtool(enp216s0) stat: 46294 ( 46,294) <= tx13_csum_none
/sec
Ethtool(enp216s0) stat: 33388 ( 33,388) <=
tx13_csum_partial /sec
Ethtool(enp216s0) stat: 2109 ( 2,109) <= tx13_nop /sec
Ethtool(enp216s0) stat: 145877 ( 145,877) <= tx13_packets /sec
Ethtool(enp216s0) stat: 136076991 ( 136,076,991) <= tx13_tso_bytes
/sec
Ethtool(enp216s0) stat: 29367 ( 29,367) <=
tx13_tso_packets /sec
Ethtool(enp216s0) stat: 1572 ( 1,572) <= tx13_xmit_more
/sec
Ethtool(enp216s0) stat: 86096 ( 86,096) <=
tx14_added_vlan_packets /sec
Ethtool(enp216s0) stat: 199318635 ( 199,318,635) <= tx14_bytes /sec
Ethtool(enp216s0) stat: 84335 ( 84,335) <= tx14_cqes /sec
Ethtool(enp216s0) stat: 50934 ( 50,934) <= tx14_csum_none
/sec
Ethtool(enp216s0) stat: 35163 ( 35,163) <=
tx14_csum_partial /sec
Ethtool(enp216s0) stat: 2263 ( 2,263) <= tx14_nop /sec
Ethtool(enp216s0) stat: 156445 ( 156,445) <= tx14_packets /sec
Ethtool(enp216s0) stat: 142293555 ( 142,293,555) <= tx14_tso_bytes
/sec
Ethtool(enp216s0) stat: 29703 ( 29,703) <=
tx14_tso_packets /sec
Ethtool(enp216s0) stat: 1761 ( 1,761) <= tx14_xmit_more
/sec
Ethtool(enp216s0) stat: 79698 ( 79,698) <=
tx15_added_vlan_packets /sec
Ethtool(enp216s0) stat: 189620424 ( 189,620,424) <= tx15_bytes /sec
Ethtool(enp216s0) stat: 78356 ( 78,356) <= tx15_cqes /sec
Ethtool(enp216s0) stat: 45528 ( 45,528) <= tx15_csum_none
/sec
Ethtool(enp216s0) stat: 34170 ( 34,170) <=
tx15_csum_partial /sec
Ethtool(enp216s0) stat: 2156 ( 2,156) <= tx15_nop /sec
Ethtool(enp216s0) stat: 144935 ( 144,935) <= tx15_packets /sec
Ethtool(enp216s0) stat: 135023821 ( 135,023,821) <= tx15_tso_bytes
/sec
Ethtool(enp216s0) stat: 29400 ( 29,400) <=
tx15_tso_packets /sec
Ethtool(enp216s0) stat: 1344 ( 1,344) <= tx15_xmit_more
/sec
Ethtool(enp216s0) stat: 59598 ( 59,598) <=
tx16_added_vlan_packets /sec
Ethtool(enp216s0) stat: 143187495 ( 143,187,495) <= tx16_bytes /sec
Ethtool(enp216s0) stat: 58408 ( 58,408) <= tx16_cqes /sec
Ethtool(enp216s0) stat: 35002 ( 35,002) <= tx16_csum_none
/sec
Ethtool(enp216s0) stat: 24595 ( 24,595) <=
tx16_csum_partial /sec
Ethtool(enp216s0) stat: 1585 ( 1,585) <= tx16_nop /sec
Ethtool(enp216s0) stat: 110250 ( 110,250) <= tx16_packets /sec
Ethtool(enp216s0) stat: 101339613 ( 101,339,613) <= tx16_tso_bytes
/sec
Ethtool(enp216s0) stat: 20698 ( 20,698) <=
tx16_tso_packets /sec
Ethtool(enp216s0) stat: 1179 ( 1,179) <= tx16_xmit_more
/sec
Ethtool(enp216s0) stat: 69504 ( 69,504) <=
tx17_added_vlan_packets /sec
Ethtool(enp216s0) stat: 171534675 ( 171,534,675) <= tx17_bytes /sec
Ethtool(enp216s0) stat: 68155 ( 68,155) <= tx17_cqes /sec
Ethtool(enp216s0) stat: 39445 ( 39,445) <= tx17_csum_none
/sec
Ethtool(enp216s0) stat: 30059 ( 30,059) <=
tx17_csum_partial /sec
Ethtool(enp216s0) stat: 1886 ( 1,886) <= tx17_nop /sec
Ethtool(enp216s0) stat: 130910 ( 130,910) <= tx17_packets /sec
Ethtool(enp216s0) stat: 123978012 ( 123,978,012) <= tx17_tso_bytes
/sec
Ethtool(enp216s0) stat: 26215 ( 26,215) <=
tx17_tso_packets /sec
Ethtool(enp216s0) stat: 1349 ( 1,349) <= tx17_xmit_more
/sec
Ethtool(enp216s0) stat: 58880 ( 58,880) <=
tx18_added_vlan_packets /sec
Ethtool(enp216s0) stat: 141863299 ( 141,863,299) <= tx18_bytes /sec
Ethtool(enp216s0) stat: 57755 ( 57,755) <= tx18_cqes /sec
Ethtool(enp216s0) stat: 33875 ( 33,875) <= tx18_csum_none
/sec
Ethtool(enp216s0) stat: 25005 ( 25,005) <=
tx18_csum_partial /sec
Ethtool(enp216s0) stat: 1592 ( 1,592) <= tx18_nop /sec
Ethtool(enp216s0) stat: 110248 ( 110,248) <= tx18_packets /sec
Ethtool(enp216s0) stat: 103105544 ( 103,105,544) <= tx18_tso_bytes
/sec
Ethtool(enp216s0) stat: 21243 ( 21,243) <=
tx18_tso_packets /sec
Ethtool(enp216s0) stat: 1108 ( 1,108) <= tx18_xmit_more
/sec
Ethtool(enp216s0) stat: 69804 ( 69,804) <=
tx19_added_vlan_packets /sec
Ethtool(enp216s0) stat: 164225730 ( 164,225,730) <= tx19_bytes /sec
Ethtool(enp216s0) stat: 68756 ( 68,756) <= tx19_cqes /sec
Ethtool(enp216s0) stat: 42003 ( 42,003) <= tx19_csum_none
/sec
Ethtool(enp216s0) stat: 27801 ( 27,801) <=
tx19_csum_partial /sec
Ethtool(enp216s0) stat: 1790 ( 1,790) <= tx19_nop /sec
Ethtool(enp216s0) stat: 124794 ( 124,794) <= tx19_packets /sec
Ethtool(enp216s0) stat: 110814620 ( 110,814,620) <= tx19_tso_bytes
/sec
Ethtool(enp216s0) stat: 23278 ( 23,278) <=
tx19_tso_packets /sec
Ethtool(enp216s0) stat: 1045 ( 1,045) <= tx19_xmit_more
/sec
Ethtool(enp216s0) stat: 79346 ( 79,346) <=
tx1_added_vlan_packets /sec
Ethtool(enp216s0) stat: 181179251 ( 181,179,251) <= tx1_bytes /sec
Ethtool(enp216s0) stat: 78062 ( 78,062) <= tx1_cqes /sec
Ethtool(enp216s0) stat: 46525 ( 46,525) <= tx1_csum_none /sec
Ethtool(enp216s0) stat: 32821 ( 32,821) <=
tx1_csum_partial /sec
Ethtool(enp216s0) stat: 1996 ( 1,996) <= tx1_nop /sec
Ethtool(enp216s0) stat: 142716 ( 142,716) <= tx1_packets /sec
Ethtool(enp216s0) stat: 129507562 ( 129,507,562) <= tx1_tso_bytes /sec
Ethtool(enp216s0) stat: 27579 ( 27,579) <=
tx1_tso_packets /sec
Ethtool(enp216s0) stat: 1281 ( 1,281) <= tx1_xmit_more /sec
Ethtool(enp216s0) stat: 66641 ( 66,641) <=
tx20_added_vlan_packets /sec
Ethtool(enp216s0) stat: 161452661 ( 161,452,661) <= tx20_bytes /sec
Ethtool(enp216s0) stat: 65374 ( 65,374) <= tx20_cqes /sec
Ethtool(enp216s0) stat: 37657 ( 37,657) <= tx20_csum_none
/sec
Ethtool(enp216s0) stat: 28983 ( 28,983) <=
tx20_csum_partial /sec
Ethtool(enp216s0) stat: 1739 ( 1,739) <= tx20_nop /sec
Ethtool(enp216s0) stat: 122824 ( 122,824) <= tx20_packets /sec
Ethtool(enp216s0) stat: 115707977 ( 115,707,977) <= tx20_tso_bytes
/sec
Ethtool(enp216s0) stat: 24823 ( 24,823) <=
tx20_tso_packets /sec
Ethtool(enp216s0) stat: 1263 ( 1,263) <= tx20_xmit_more
/sec
Ethtool(enp216s0) stat: 60564 ( 60,564) <=
tx21_added_vlan_packets /sec
Ethtool(enp216s0) stat: 138260273 ( 138,260,273) <= tx21_bytes /sec
Ethtool(enp216s0) stat: 59611 ( 59,611) <= tx21_cqes /sec
Ethtool(enp216s0) stat: 36105 ( 36,105) <= tx21_csum_none
/sec
Ethtool(enp216s0) stat: 24459 ( 24,459) <=
tx21_csum_partial /sec
Ethtool(enp216s0) stat: 1490 ( 1,490) <= tx21_nop /sec
Ethtool(enp216s0) stat: 106913 ( 106,913) <= tx21_packets /sec
Ethtool(enp216s0) stat: 97216016 ( 97,216,016) <= tx21_tso_bytes
/sec
Ethtool(enp216s0) stat: 21944 ( 21,944) <=
tx21_tso_packets /sec
Ethtool(enp216s0) stat: 930 ( 930) <= tx21_xmit_more
/sec
Ethtool(enp216s0) stat: 66527 ( 66,527) <=
tx22_added_vlan_packets /sec
Ethtool(enp216s0) stat: 149686537 ( 149,686,537) <= tx22_bytes /sec
Ethtool(enp216s0) stat: 65262 ( 65,262) <= tx22_cqes /sec
Ethtool(enp216s0) stat: 40609 ( 40,609) <= tx22_csum_none
/sec
Ethtool(enp216s0) stat: 25918 ( 25,918) <=
tx22_csum_partial /sec
Ethtool(enp216s0) stat: 1676 ( 1,676) <= tx22_nop /sec
Ethtool(enp216s0) stat: 118071 ( 118,071) <= tx22_packets /sec
Ethtool(enp216s0) stat: 104822520 ( 104,822,520) <= tx22_tso_bytes
/sec
Ethtool(enp216s0) stat: 22046 ( 22,046) <=
tx22_tso_packets /sec
Ethtool(enp216s0) stat: 1265 ( 1,265) <= tx22_xmit_more
/sec
Ethtool(enp216s0) stat: 59973 ( 59,973) <=
tx23_added_vlan_packets /sec
Ethtool(enp216s0) stat: 147788578 ( 147,788,578) <= tx23_bytes /sec
Ethtool(enp216s0) stat: 58827 ( 58,827) <= tx23_cqes /sec
Ethtool(enp216s0) stat: 33197 ( 33,197) <= tx23_csum_none
/sec
Ethtool(enp216s0) stat: 26776 ( 26,776) <=
tx23_csum_partial /sec
Ethtool(enp216s0) stat: 1669 ( 1,669) <= tx23_nop /sec
Ethtool(enp216s0) stat: 112053 ( 112,053) <= tx23_packets /sec
Ethtool(enp216s0) stat: 106098597 ( 106,098,597) <= tx23_tso_bytes
/sec
Ethtool(enp216s0) stat: 22433 ( 22,433) <=
tx23_tso_packets /sec
Ethtool(enp216s0) stat: 1126 ( 1,126) <= tx23_xmit_more
/sec
Ethtool(enp216s0) stat: 58819 ( 58,819) <=
tx24_added_vlan_packets /sec
Ethtool(enp216s0) stat: 146231570 ( 146,231,570) <= tx24_bytes /sec
Ethtool(enp216s0) stat: 57661 ( 57,661) <= tx24_cqes /sec
Ethtool(enp216s0) stat: 32150 ( 32,150) <= tx24_csum_none
/sec
Ethtool(enp216s0) stat: 26669 ( 26,669) <=
tx24_csum_partial /sec
Ethtool(enp216s0) stat: 1578 ( 1,578) <= tx24_nop /sec
Ethtool(enp216s0) stat: 111230 ( 111,230) <= tx24_packets /sec
Ethtool(enp216s0) stat: 106359530 ( 106,359,530) <= tx24_tso_bytes
/sec
Ethtool(enp216s0) stat: 22402 ( 22,402) <=
tx24_tso_packets /sec
Ethtool(enp216s0) stat: 1158 ( 1,158) <= tx24_xmit_more
/sec
Ethtool(enp216s0) stat: 64116 ( 64,116) <=
tx25_added_vlan_packets /sec
Ethtool(enp216s0) stat: 156132090 ( 156,132,090) <= tx25_bytes /sec
Ethtool(enp216s0) stat: 62901 ( 62,901) <= tx25_cqes /sec
Ethtool(enp216s0) stat: 36357 ( 36,357) <= tx25_csum_none
/sec
Ethtool(enp216s0) stat: 27759 ( 27,759) <=
tx25_csum_partial /sec
Ethtool(enp216s0) stat: 1717 ( 1,717) <= tx25_nop /sec
Ethtool(enp216s0) stat: 118618 ( 118,618) <= tx25_packets /sec
Ethtool(enp216s0) stat: 110752893 ( 110,752,893) <= tx25_tso_bytes
/sec
Ethtool(enp216s0) stat: 23282 ( 23,282) <=
tx25_tso_packets /sec
Ethtool(enp216s0) stat: 1212 ( 1,212) <= tx25_xmit_more
/sec
Ethtool(enp216s0) stat: 62028 ( 62,028) <=
tx26_added_vlan_packets /sec
Ethtool(enp216s0) stat: 144966495 ( 144,966,495) <= tx26_bytes /sec
Ethtool(enp216s0) stat: 60906 ( 60,906) <= tx26_cqes /sec
Ethtool(enp216s0) stat: 36345 ( 36,345) <= tx26_csum_none
/sec
Ethtool(enp216s0) stat: 25684 ( 25,684) <=
tx26_csum_partial /sec
Ethtool(enp216s0) stat: 1615 ( 1,615) <= tx26_nop /sec
Ethtool(enp216s0) stat: 112487 ( 112,487) <= tx26_packets /sec
Ethtool(enp216s0) stat: 102083200 ( 102,083,200) <= tx26_tso_bytes
/sec
Ethtool(enp216s0) stat: 21598 ( 21,598) <=
tx26_tso_packets /sec
Ethtool(enp216s0) stat: 1123 ( 1,123) <= tx26_xmit_more
/sec
Ethtool(enp216s0) stat: 64029 ( 64,029) <=
tx27_added_vlan_packets /sec
Ethtool(enp216s0) stat: 154747343 ( 154,747,343) <= tx27_bytes /sec
Ethtool(enp216s0) stat: 62881 ( 62,881) <= tx27_cqes /sec
Ethtool(enp216s0) stat: 35343 ( 35,343) <= tx27_csum_none
/sec
Ethtool(enp216s0) stat: 28686 ( 28,686) <=
tx27_csum_partial /sec
Ethtool(enp216s0) stat: 1707 ( 1,707) <= tx27_nop /sec
Ethtool(enp216s0) stat: 118074 ( 118,074) <= tx27_packets /sec
Ethtool(enp216s0) stat: 110960676 ( 110,960,676) <= tx27_tso_bytes
/sec
Ethtool(enp216s0) stat: 24139 ( 24,139) <=
tx27_tso_packets /sec
Ethtool(enp216s0) stat: 1148 ( 1,148) <= tx27_xmit_more
/sec
Ethtool(enp216s0) stat: 82542 ( 82,542) <=
tx2_added_vlan_packets /sec
Ethtool(enp216s0) stat: 183007051 ( 183,007,051) <= tx2_bytes /sec
Ethtool(enp216s0) stat: 81320 ( 81,320) <= tx2_cqes /sec
Ethtool(enp216s0) stat: 49989 ( 49,989) <= tx2_csum_none /sec
Ethtool(enp216s0) stat: 32553 ( 32,553) <=
tx2_csum_partial /sec
Ethtool(enp216s0) stat: 2142 ( 2,142) <= tx2_nop /sec
Ethtool(enp216s0) stat: 143613 ( 143,613) <= tx2_packets /sec
Ethtool(enp216s0) stat: 126895404 ( 126,895,404) <= tx2_tso_bytes /sec
Ethtool(enp216s0) stat: 28173 ( 28,173) <=
tx2_tso_packets /sec
Ethtool(enp216s0) stat: 1220 ( 1,220) <= tx2_xmit_more /sec
Ethtool(enp216s0) stat: 78737 ( 78,737) <=
tx3_added_vlan_packets /sec
Ethtool(enp216s0) stat: 188546556 ( 188,546,556) <= tx3_bytes /sec
Ethtool(enp216s0) stat: 77389 ( 77,389) <= tx3_cqes /sec
Ethtool(enp216s0) stat: 46147 ( 46,147) <= tx3_csum_none /sec
Ethtool(enp216s0) stat: 32590 ( 32,590) <=
tx3_csum_partial /sec
Ethtool(enp216s0) stat: 2171 ( 2,171) <= tx3_nop /sec
Ethtool(enp216s0) stat: 144570 ( 144,570) <= tx3_packets /sec
Ethtool(enp216s0) stat: 134231500 ( 134,231,500) <= tx3_tso_bytes /sec
Ethtool(enp216s0) stat: 28857 ( 28,857) <=
tx3_tso_packets /sec
Ethtool(enp216s0) stat: 1348 ( 1,348) <= tx3_xmit_more /sec
Ethtool(enp216s0) stat: 76284 ( 76,284) <=
tx4_added_vlan_packets /sec
Ethtool(enp216s0) stat: 185633403 ( 185,633,403) <= tx4_bytes /sec
Ethtool(enp216s0) stat: 74710 ( 74,710) <= tx4_cqes /sec
Ethtool(enp216s0) stat: 42803 ( 42,803) <= tx4_csum_none /sec
Ethtool(enp216s0) stat: 33482 ( 33,482) <=
tx4_csum_partial /sec
Ethtool(enp216s0) stat: 2005 ( 2,005) <= tx4_nop /sec
Ethtool(enp216s0) stat: 139978 ( 139,978) <= tx4_packets /sec
Ethtool(enp216s0) stat: 131762891 ( 131,762,891) <= tx4_tso_bytes /sec
Ethtool(enp216s0) stat: 28843 ( 28,843) <=
tx4_tso_packets /sec
Ethtool(enp216s0) stat: 1563 ( 1,563) <= tx4_xmit_more /sec
Ethtool(enp216s0) stat: 83878 ( 83,878) <=
tx5_added_vlan_packets /sec
Ethtool(enp216s0) stat: 197920426 ( 197,920,426) <= tx5_bytes /sec
Ethtool(enp216s0) stat: 82146 ( 82,146) <= tx5_cqes /sec
Ethtool(enp216s0) stat: 48122 ( 48,122) <= tx5_csum_none /sec
Ethtool(enp216s0) stat: 35755 ( 35,755) <=
tx5_csum_partial /sec
Ethtool(enp216s0) stat: 2227 ( 2,227) <= tx5_nop /sec
Ethtool(enp216s0) stat: 151214 ( 151,214) <= tx5_packets /sec
Ethtool(enp216s0) stat: 139113072 ( 139,113,072) <= tx5_tso_bytes /sec
Ethtool(enp216s0) stat: 30390 ( 30,390) <=
tx5_tso_packets /sec
Ethtool(enp216s0) stat: 1730 ( 1,730) <= tx5_xmit_more /sec
Ethtool(enp216s0) stat: 76307 ( 76,307) <=
tx6_added_vlan_packets /sec
Ethtool(enp216s0) stat: 182153538 ( 182,153,538) <= tx6_bytes /sec
Ethtool(enp216s0) stat: 74744 ( 74,744) <= tx6_cqes /sec
Ethtool(enp216s0) stat: 44381 ( 44,381) <= tx6_csum_none /sec
Ethtool(enp216s0) stat: 31926 ( 31,926) <=
tx6_csum_partial /sec
Ethtool(enp216s0) stat: 2068 ( 2,068) <= tx6_nop /sec
Ethtool(enp216s0) stat: 139745 ( 139,745) <= tx6_packets /sec
Ethtool(enp216s0) stat: 129175625 ( 129,175,625) <= tx6_tso_bytes /sec
Ethtool(enp216s0) stat: 27349 ( 27,349) <=
tx6_tso_packets /sec
Ethtool(enp216s0) stat: 1562 ( 1,562) <= tx6_xmit_more /sec
Ethtool(enp216s0) stat: 72151 ( 72,151) <=
tx7_added_vlan_packets /sec
Ethtool(enp216s0) stat: 175340247 ( 175,340,247) <= tx7_bytes /sec
Ethtool(enp216s0) stat: 70565 ( 70,565) <= tx7_cqes /sec
Ethtool(enp216s0) stat: 40824 ( 40,824) <= tx7_csum_none /sec
Ethtool(enp216s0) stat: 31327 ( 31,327) <=
tx7_csum_partial /sec
Ethtool(enp216s0) stat: 1962 ( 1,962) <= tx7_nop /sec
Ethtool(enp216s0) stat: 133790 ( 133,790) <= tx7_packets /sec
Ethtool(enp216s0) stat: 124931972 ( 124,931,972) <= tx7_tso_bytes /sec
Ethtool(enp216s0) stat: 26314 ( 26,314) <=
tx7_tso_packets /sec
Ethtool(enp216s0) stat: 1586 ( 1,586) <= tx7_xmit_more /sec
Ethtool(enp216s0) stat: 83983 ( 83,983) <=
tx8_added_vlan_packets /sec
Ethtool(enp216s0) stat: 203970262 ( 203,970,262) <= tx8_bytes /sec
Ethtool(enp216s0) stat: 82475 ( 82,475) <= tx8_cqes /sec
Ethtool(enp216s0) stat: 47937 ( 47,937) <= tx8_csum_none /sec
Ethtool(enp216s0) stat: 36046 ( 36,046) <=
tx8_csum_partial /sec
Ethtool(enp216s0) stat: 2203 ( 2,203) <= tx8_nop /sec
Ethtool(enp216s0) stat: 153743 ( 153,743) <= tx8_packets /sec
Ethtool(enp216s0) stat: 143296525 ( 143,296,525) <= tx8_tso_bytes /sec
Ethtool(enp216s0) stat: 30936 ( 30,936) <=
tx8_tso_packets /sec
Ethtool(enp216s0) stat: 1500 ( 1,500) <= tx8_xmit_more /sec
Ethtool(enp216s0) stat: 79406 ( 79,406) <=
tx9_added_vlan_packets /sec
Ethtool(enp216s0) stat: 183286836 ( 183,286,836) <= tx9_bytes /sec
Ethtool(enp216s0) stat: 77769 ( 77,769) <= tx9_cqes /sec
Ethtool(enp216s0) stat: 48014 ( 48,014) <= tx9_csum_none /sec
Ethtool(enp216s0) stat: 31392 ( 31,392) <=
tx9_csum_partial /sec
Ethtool(enp216s0) stat: 2146 ( 2,146) <= tx9_nop /sec
Ethtool(enp216s0) stat: 142229 ( 142,229) <= tx9_packets /sec
Ethtool(enp216s0) stat: 127419807 ( 127,419,807) <= tx9_tso_bytes /sec
Ethtool(enp216s0) stat: 26644 ( 26,644) <=
tx9_tso_packets /sec
Ethtool(enp216s0) stat: 1633 ( 1,633) <= tx9_xmit_more /sec
Ethtool(enp216s0) stat: 2050335 ( 2,050,335) <=
tx_added_vlan_packets /sec
Ethtool(enp216s0) stat: 4851410368 ( 4,851,410,368) <= tx_bytes /sec
Ethtool(enp216s0) stat: 4881854912 ( 4,881,854,912) <= tx_bytes_phy /sec
Ethtool(enp216s0) stat: 2011989 ( 2,011,989) <= tx_cqes /sec
Ethtool(enp216s0) stat: 1191137 ( 1,191,137) <= tx_csum_none /sec
Ethtool(enp216s0) stat: 859197 ( 859,197) <=
tx_csum_partial /sec
Ethtool(enp216s0) stat: 54140 ( 54,140) <= tx_nop /sec
Ethtool(enp216s0) stat: 3728759 ( 3,728,759) <= tx_packets /sec
Ethtool(enp216s0) stat: 3729416 ( 3,729,416) <= tx_packets_phy
/sec
Ethtool(enp216s0) stat: 4882774572 ( 4,882,774,572) <= tx_prio0_bytes
/sec
Ethtool(enp216s0) stat: 3730195 ( 3,730,195) <=
tx_prio0_packets /sec
Ethtool(enp216s0) stat: 3431642829 ( 3,431,642,829) <= tx_tso_bytes /sec
Ethtool(enp216s0) stat: 734343 ( 734,343) <= tx_tso_packets
/sec
Ethtool(enp216s0) stat: 4866327192 ( 4,866,327,192) <=
tx_vport_unicast_bytes /sec
Ethtool(enp216s0) stat: 3728966 ( 3,728,966) <=
tx_vport_unicast_packets /sec
Ethtool(enp216s0) stat: 38268 ( 38,268) <= tx_xmit_more /sec
>>
>> Can you give output put from:
>> $ ethtool --show-priv-flag DEVICE
>>
>> I want you to experiment with:
> ethtool --show-priv-flags enp175s0
> Private flags for enp175s0:
> rx_cqe_moder : on
> tx_cqe_moder : off
> rx_cqe_compress : off
> rx_striding_rq : on
> rx_no_csum_complete: off
>
>>
>> ethtool --set-priv-flags DEVICE rx_striding_rq off
> ok i will first check on test server if this will reset my interface
> and will not produce kernel panic :)
>>
>> I think you already have played with 'rx_cqe_compress', right.
> yes - and compress increasing number of irq's but doing not much for
> bandwidth same limit 60-64Gbit/s total RX+TX on one 100G port
>
> And what is weird - that limit is in overall symetric - cause if for
> example 100G port is receiving 42G traffic and transmitting 20G
> traffic - when i flood rx side with pktgen or other for example icmp
> traffic 1/2/3/4/5G - then receiving side increase with 1/2/3/4/5Gbit
> of traffic but transmitting is going down for same lvl's
>
>
>
>
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Paweł Staszewski @ 2018-11-10 19:49 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Saeed Mahameed, netdev@vger.kernel.org
In-Reply-To: <20181110203409.482f39ec@redhat.com>
W dniu 10.11.2018 o 20:34, Jesper Dangaard Brouer pisze:
> On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
>>> CPU load is lower than for connectx4 - but it looks like bandwidth
>>> limit is the same :)
>>> But also after reaching 60Gbit/60Gbit
>>>
>>> bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
>>> input: /proc/net/dev type: rate
>>> - iface Rx Tx Total
>>> ==========================================================================
>>>
>>> enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
>>> enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
>>> --------------------------------------------------------------------------
>>>
>>> total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>> Today reached 65/65Gbit/s
>>
>> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
>> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
> This is weird!
>
> How do you see / measure these drops?
Simple icmp test like ping -i 0.1
And im testing by icmp management ip address on vlan that is attacked to
one NIC (the side that is more stressed with RX)
And another icmp test is forward thru this router - host behind it
Both measurements shows same loss ratio from 0.1 to 0.5% after reaching
~45Gbit/s RX side - depends how much RX side is pushed drops vary
between 0.1 to 0.5 - even 0.6%:)
>
>
>> So checked other stats.
>> softnet_stats shows average 1k squeezed per sec:
> Is below output the raw counters? not per sec?
>
> It would be valuable to see the per sec stats instead...
> I use this tool:
> https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
>
>> cpu total dropped squeezed collision rps flow_limit
>> 0 18554 0 1 0 0 0
>> 1 16728 0 1 0 0 0
>> 2 18033 0 1 0 0 0
>> 3 17757 0 1 0 0 0
>> 4 18861 0 0 0 0 0
>> 5 0 0 1 0 0 0
>> 6 2 0 1 0 0 0
>> 7 0 0 1 0 0 0
>> 8 0 0 0 0 0 0
>> 9 0 0 1 0 0 0
>> 10 0 0 0 0 0 0
>> 11 0 0 1 0 0 0
>> 12 50 0 1 0 0 0
>> 13 257 0 0 0 0 0
>> 14 3629115363 0 3353259 0 0 0
>> 15 255167835 0 3138271 0 0 0
>> 16 4240101961 0 3036130 0 0 0
>> 17 599810018 0 3072169 0 0 0
>> 18 432796524 0 3034191 0 0 0
>> 19 41803906 0 3037405 0 0 0
>> 20 900382666 0 3112294 0 0 0
>> 21 620926085 0 3086009 0 0 0
>> 22 41861198 0 3023142 0 0 0
>> 23 4090425574 0 2990412 0 0 0
>> 24 4264870218 0 3010272 0 0 0
>> 25 141401811 0 3027153 0 0 0
>> 26 104155188 0 3051251 0 0 0
>> 27 4261258691 0 3039765 0 0 0
>> 28 4 0 1 0 0 0
>> 29 4 0 0 0 0 0
>> 30 0 0 1 0 0 0
>> 31 0 0 0 0 0 0
>> 32 3 0 1 0 0 0
>> 33 1 0 1 0 0 0
>> 34 0 0 1 0 0 0
>> 35 0 0 0 0 0 0
>> 36 0 0 1 0 0 0
>> 37 0 0 1 0 0 0
>> 38 0 0 1 0 0 0
>> 39 0 0 1 0 0 0
>> 40 0 0 0 0 0 0
>> 41 0 0 1 0 0 0
>> 42 299758202 0 3139693 0 0 0
>> 43 4254727979 0 3103577 0 0 0
>> 44 1959555543 0 2554885 0 0 0
>> 45 1675702723 0 2513481 0 0 0
>> 46 1908435503 0 2519698 0 0 0
>> 47 1877799710 0 2537768 0 0 0
>> 48 2384274076 0 2584673 0 0 0
>> 49 2598104878 0 2593616 0 0 0
>> 50 1897566829 0 2530857 0 0 0
>> 51 1712741629 0 2489089 0 0 0
>> 52 1704033648 0 2495892 0 0 0
>> 53 1636781820 0 2499783 0 0 0
>> 54 1861997734 0 2541060 0 0 0
>> 55 2113521616 0 2555673 0 0 0
>>
>>
>> So i rised netdev backlog and budged to rly high values
>> 524288 for netdev_budget and same for backlog
> Does it affect the squeezed counters?
a little - but not much
After change budget from 65536 to to 524k - number of squeezed counters
for all cpus changed from 1.5k per second to 0.9-1k per second - but
increasing it more like above 524k change nothing - same 0.9 to 1k/s
squeezed
>
> Notice, this (crazy) huge netdev_budget limit will also be limited
> by /proc/sys/net/core/netdev_budget_usecs.
Yes changed that also to 1000 / 2000 / 3000 / 4000 not much difference
on squeezed - even cant see the difference
>
>> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
> Hmmm, this could indicated not enough NAPI bulking is occurring.
>
> I have a BPF tool, that can give you some insight into NAPI bulking and
> softirq idle/kthread starting. Called 'napi_monitor', could you try to
> run this, so can try to understand this? You find the tool here:
>
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
> https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
yes will try it
>
>> But after this changes i have less packets drops.
>>
>>
>> Below perf top from max traffic reached:
>> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> ------------------------------------------------------------------------------------------
>>
>> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
>> 8.44% [kernel] [k] mlx5e_sq_xmit
>> 6.69% [kernel] [k] build_skb
>> 5.21% [kernel] [k] fib_table_lookup
>> 3.54% [kernel] [k] memcpy_erms
>> 3.20% [kernel] [k] mlx5e_poll_rx_cq
>> 2.25% [kernel] [k] vlan_do_receive
>> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
>> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
>> 1.95% [kernel] [k] __dev_queue_xmit
>> 1.83% [kernel] [k] dev_gro_receive
>> 1.79% [kernel] [k] tcp_gro_receive
>> 1.73% [kernel] [k] ip_finish_output2
>> 1.63% [kernel] [k] mlx5e_poll_tx_cq
>> 1.49% [kernel] [k] ipt_do_table
>> 1.38% [kernel] [k] inet_gro_receive
>> 1.31% [kernel] [k] __netif_receive_skb_core
>> 1.30% [kernel] [k] _raw_spin_lock
>> 1.28% [kernel] [k] mlx5_eq_int
>> 1.24% [kernel] [k] irq_entries_start
>> 1.19% [kernel] [k] __build_skb
>> 1.15% [kernel] [k] swiotlb_map_page
>> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.94% [kernel] [k] pfifo_fast_dequeue
>> 0.92% [kernel] [k] ip_route_input_rcu
>> 0.86% [kernel] [k] kmem_cache_alloc
>> 0.80% [kernel] [k] mlx5e_xmit
>> 0.79% [kernel] [k] dev_hard_start_xmit
>> 0.78% [kernel] [k] _raw_spin_lock_irqsave
>> 0.74% [kernel] [k] ip_forward
>> 0.72% [kernel] [k] tasklet_action_common.isra.21
>> 0.68% [kernel] [k] pfifo_fast_enqueue
>> 0.67% [kernel] [k] netif_skb_features
>> 0.66% [kernel] [k] skb_segment
>> 0.60% [kernel] [k] skb_gro_receive
>> 0.56% [kernel] [k] validate_xmit_skb.isra.142
>> 0.53% [kernel] [k] skb_release_data
>> 0.51% [kernel] [k] mlx5e_page_release
>> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
>> 0.51% [kernel] [k] __qdisc_run
>> 0.50% [kernel] [k] tcp4_gro_receive
>> 0.49% [kernel] [k] page_frag_free
>> 0.46% [kernel] [k] kmem_cache_free_bulk
>> 0.43% [kernel] [k] kmem_cache_free
>> 0.42% [kernel] [k] try_to_wake_up
>> 0.39% [kernel] [k] _raw_spin_lock_irq
>> 0.39% [kernel] [k] find_busiest_group
>> 0.37% [kernel] [k] __memcpy
>>
>>
>>
>> Remember those tests are now on two separate connectx5 connected to
>> two separate pcie x16 gen 3.0
>
> That is strange... I still suspect some HW NIC issue, can you provide
> ethtool stats info via tool:
>
> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>
> $ ethtool_stats.pl --dev enp175s0 --dev enp216s0
>
> The tool remove zero-stats counters and report per sec stats. It makes
> it easier to spot that is relevant for the given workload.
yes mlnx have just too many counters that are always 0 for my case :)
Will try this also
>
> Can you give output put from:
> $ ethtool --show-priv-flag DEVICE
>
> I want you to experiment with:
ethtool --show-priv-flags enp175s0
Private flags for enp175s0:
rx_cqe_moder : on
tx_cqe_moder : off
rx_cqe_compress : off
rx_striding_rq : on
rx_no_csum_complete: off
>
> ethtool --set-priv-flags DEVICE rx_striding_rq off
ok i will first check on test server if this will reset my interface and
will not produce kernel panic :)
>
> I think you already have played with 'rx_cqe_compress', right.
yes - and compress increasing number of irq's but doing not much for
bandwidth same limit 60-64Gbit/s total RX+TX on one 100G port
And what is weird - that limit is in overall symetric - cause if for
example 100G port is receiving 42G traffic and transmitting 20G traffic
- when i flood rx side with pktgen or other for example icmp traffic
1/2/3/4/5G - then receiving side increase with 1/2/3/4/5Gbit of traffic
but transmitting is going down for same lvl's
^ permalink raw reply
* [BUG] xfrm: unable to handle kernel NULL pointer dereference
From: Jean-Philippe Menil @ 2018-11-10 19:34 UTC (permalink / raw)
To: steffen.klassert; +Cc: herbert, davem, netdev, kuznet, yoshfuji
[-- Attachment #1: Type: text/plain, Size: 3005 bytes --]
Hi guys,
we're seeing unexpected crashes from kernel 4.15 to 4.18.17, using IPsec
VTI interfaces, on several vpn hosts, since upgrade from 4.4.
Attached, the offended oops against 4.18.
Output of decodedecode:
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00
00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe
<f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
All code
========
0: 8b 44 24 70 mov 0x70(%rsp),%eax
4: 0f c8 bswap %eax
6: 89 87 b4 00 00 00 mov %eax,0xb4(%rdi)
c: 48 8b 86 20 05 00 00 mov 0x520(%rsi),%rax
13: 8b 80 f8 14 00 00 mov 0x14f8(%rax),%eax
19: 85 c0 test %eax,%eax
1b: 75 05 jne 0x22
1d: 48 85 d2 test %rdx,%rdx
20: 74 0e je 0x30
22: 48 8b 43 58 mov 0x58(%rbx),%rax
26: 48 83 e0 fe and $0xfffffffffffffffe,%rax
2a:* f6 40 38 04 testb $0x4,0x38(%rax) <--
trapping instruction
2e: 74 7d je 0xad
30: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
37: 49 8b 44 24 20 mov 0x20(%r12),%rax
3c: 48 rex.W
3d: 39 .byte 0x39
3e: 86 20 xchg %ah,(%rax)
Code starting with the faulting instruction
===========================================
0: f6 40 38 04 testb $0x4,0x38(%rax)
4: 74 7d je 0x83
6: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
d: 49 8b 44 24 20 mov 0x20(%r12),%rax
12: 48 rex.W
13: 39 .byte 0x39
14: 86 20 xchg %ah,(%rax)
if my understanding is correct, we fail here:
/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
0x0000000000000b19 <+185>: testb $0x4,0x38(%rax)
0x0000000000000b1d <+189>: je 0xb9c <vti_rcv_cb+316>
(gdb) list *0x0000000000000b19
0xb19 is in vti_rcv_cb
(/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:1169).
1164 int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
1165
1166 if (sk && sk->sk_policy[XFRM_POLICY_IN])
1167 return __xfrm_policy_check(sk, ndir, skb, family);
1168
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
1170 (skb_dst(skb)->flags & DST_NOPOLICY) ||
1171 __xfrm_policy_check(sk, ndir, skb, family);
1172 }
1173
I really have hard time to understand why skb seem to be freed twice.
I'm not able to repeat the bug in lab, but it happened regulary in prod,
seem to depend of the workload.
Any help will be appreciated.
Let me know if you need further informations.
Regards,
Jean-Philippe
[-- Attachment #2: oops.txt --]
[-- Type: text/plain, Size: 14750 bytes --]
[ 31.154360] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
[ 31.162233] PGD 0 P4D 0
[ 31.164786] Oops: 0000 [#1] SMP PTI
[ 31.168291] CPU: 5 PID: 42 Comm: ksoftirqd/5 Not tainted 4.18.0-11-generic #12~18.04.1-Ubuntu
[ 31.176854] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016
[ 31.184980] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.189962] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.208916] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.214160] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.221328] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.228485] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.235643] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.242803] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.249964] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 31.258077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.263848] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.271004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.278163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.285320] Call Trace:
[ 31.287789] xfrm4_rcv_cb+0x4a/0x70
[ 31.291297] xfrm_input+0x58f/0x8f0
[ 31.294807] vti_input+0xaa/0x110 [ip_vti]
[ 31.298926] vti_rcv+0x33/0x3c [ip_vti]
[ 31.302783] xfrm4_esp_rcv+0x39/0x50
[ 31.306375] ip_local_deliver_finish+0x62/0x200
[ 31.310923] ip_local_deliver+0xdf/0xf0
[ 31.314775] ? ip_rcv_finish+0x420/0x420
[ 31.318718] ip_rcv_finish+0x126/0x420
[ 31.322486] ip_rcv+0x28f/0x360
[ 31.325655] ? inet_del_offload+0x40/0x40
[ 31.329686] __netif_receive_skb_core+0x48c/0xb70
[ 31.334413] ? kmem_cache_alloc+0xb4/0x1d0
[ 31.338532] ? __build_skb+0x2b/0xf0
[ 31.342128] __netif_receive_skb+0x18/0x60
[ 31.346244] ? __netif_receive_skb+0x18/0x60
[ 31.350536] netif_receive_skb_internal+0x45/0xe0
[ 31.355263] napi_gro_receive+0xc5/0xf0
[ 31.359141] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 31.364476] ? skb_release_all+0x24/0x30
[ 31.368430] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 31.373432] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 31.378333] ? __switch_to_asm+0x34/0x70
[ 31.382270] ? __switch_to_asm+0x40/0x70
[ 31.386214] ? __switch_to_asm+0x34/0x70
[ 31.391056] ? __switch_to_asm+0x40/0x70
[ 31.395905] ? __switch_to_asm+0x34/0x70
[ 31.400743] net_rx_action+0x140/0x3a0
[ 31.405379] ? __switch_to+0xad/0x500
[ 31.409887] __do_softirq+0xe4/0x2bb
[ 31.414448] run_ksoftirqd+0x2b/0x40
[ 31.418862] smpboot_thread_fn+0xfc/0x170
[ 31.423700] kthread+0x121/0x140
[ 31.427701] ? sort_range+0x30/0x30
[ 31.432040] ? kthread_create_worker_on_cpu+0x70/0x70
[ 31.437816] ret_from_fork+0x35/0x40
[ 31.442219] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev mei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel
[ 31.519488] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 31.578877] CR2: 0000000000000038
[ 31.583249] ---[ end trace c4bada38847a0075 ]---
[ 31.737166] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.737167] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.737209] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.737212] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.737213] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.737216] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.737219] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.737220] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.737222] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 31.737224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.737225] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.737227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.737228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.737230] Kernel panic - not syncing: Fatal exception in interrupt
[ 31.737264] Kernel Offset: 0x3c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 36.558279] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[ 36.566769] ------------[ cut here ]------------
[ 36.572306] sched: Unexpected reschedule of offline CPU#3!
[ 36.578714] WARNING: CPU: 5 PID: 42 at /build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/arch/x86/kernel/smp.c:128 native_smp_send_resched
ule+0x3a/0x40
[ 36.594262] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack
_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev m
ei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid sch_fq_codel
[ 36.673307] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compres
s raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x
86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 36.733827] CPU: 5 PID: 42 Comm: ksoftirqd/5 Tainted: G D 4.18.0-11-generic #12~18.04.1-Ubuntu
[ 36.745027] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS 1.0b 11/21/2016
[ 36.754392] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 36.761143] Code: c6 62 01 73 17 48 8b 05 24 d2 17 01 be fd 00 00 00 48 8b 40 30 e8 96 96 ba 00 5d c3 89 fe 48 c7 c7 90 2a cd 85 e8 26 4a 0
3 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 20 65 48
[ 36.782546] RSP: 0018:ffff9a35bfd43b68 EFLAGS: 00010082
[ 36.789019] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000006
[ 36.797379] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9a35bfd564b0
[ 36.805721] RBP: ffff9a35bfd43b68 R08: 00000000000004b7 R09: 0000000000cdcdcd
[ 36.814046] R10: 0000000000000324 R11: 00000000ffffffff R12: ffff9a35bfce2c40
[ 36.822350] R13: ffff9a3598e35c00 R14: 0000000000000008 R15: 0000000000000003
[ 36.830637] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000) knlGS:0000000000000000
[ 36.839869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.846742] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 36.855009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 36.863275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 36.871510] Call Trace:
[ 36.875048] <IRQ>
[ 36.878136] resched_curr+0x5d/0xc0
[ 36.882689] check_preempt_wakeup+0x130/0x240
[ 36.888116] check_preempt_curr+0x2d/0x90
[ 36.893184] ttwu_do_wakeup+0x1e/0x140
[ 36.897984] ttwu_do_activate+0x77/0x80
[ 36.902857] try_to_wake_up+0x1d6/0x450
[ 36.907726] ? __netif_receive_skb_core+0x48c/0xb70
[ 36.913625] default_wake_function+0x12/0x20
[ 36.918914] __wake_up_common+0x73/0x130
[ 36.923851] __wake_up_locked+0x16/0x20
[ 36.928706] ep_poll_callback+0xcb/0x2b0
[ 36.933634] __wake_up_common+0x73/0x130
[ 36.938549] __wake_up_common_lock+0x80/0xc0
[ 36.943787] ? tick_sched_do_timer+0x60/0x60
[ 36.949011] __wake_up+0x13/0x20
[ 36.953190] wake_up_klogd_work_func+0x40/0x60
[ 36.958560] irq_work_run_list+0x52/0x80
[ 36.963388] irq_work_tick+0x3b/0x50
[ 36.967841] update_process_times+0x42/0x60
[ 36.972883] tick_sched_handle+0x25/0x70
[ 36.977644] tick_sched_timer+0x3c/0x80
[ 36.982285] __hrtimer_run_queues+0x10f/0x280
[ 36.987429] hrtimer_interrupt+0xe7/0x240
[ 36.992202] ? rcu_irq_exit+0x1d/0x20
[ 36.996602] smp_apic_timer_interrupt+0x6f/0x130
[ 37.001929] apic_timer_interrupt+0xf/0x20
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 0
0 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6 20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 00 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.073287] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000000
[ 37.080999] ? panic+0x1f7/0x244
[ 37.084812] oops_end+0xce/0xe0
[ 37.088531] no_context+0x17c/0x400
[ 37.092585] ? skcipher_walk_first+0x4c/0x110
[ 37.097500] __bad_area_nosemaphore+0x115/0x1d0
[ 37.102591] ? cbc_decrypt+0xb1/0xe0 [aesni_intel]
[ 37.107926] bad_area_nosemaphore+0x14/0x20
[ 37.112654] __do_page_fault+0xd4/0x4d0
[ 37.117030] do_page_fault+0x2d/0xf0
[ 37.121143] ? skb_copy_bits+0x61/0x260
[ 37.125511] page_fault+0x1e/0x30
[ 37.129362] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 37.154892] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 37.160711] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 37.168451] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 37.176196] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 37.183950] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 37.191716] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 37.199481] ? esp_input_done2+0x6b/0x320 [esp4]
[ 37.204740] xfrm4_rcv_cb+0x4a/0x70
[ 37.208873] xfrm_input+0x58f/0x8f0
[ 37.212996] vti_input+0xaa/0x110 [ip_vti]
[ 37.217735] vti_rcv+0x33/0x3c [ip_vti]
[ 37.222209] xfrm4_esp_rcv+0x39/0x50
[ 37.226424] ip_local_deliver_finish+0x62/0x200
[ 37.231599] ip_local_deliver+0xdf/0xf0
[ 37.236072] ? ip_rcv_finish+0x420/0x420
[ 37.240635] ip_rcv_finish+0x126/0x420
[ 37.245025] ip_rcv+0x28f/0x360
[ 37.248801] ? inet_del_offload+0x40/0x40
[ 37.253457] __netif_receive_skb_core+0x48c/0xb70
[ 37.258807] ? kmem_cache_alloc+0xb4/0x1d0
[ 37.263541] ? __build_skb+0x2b/0xf0
[ 37.267749] __netif_receive_skb+0x18/0x60
[ 37.272483] ? __netif_receive_skb+0x18/0x60
[ 37.277391] netif_receive_skb_internal+0x45/0xe0
[ 37.282732] napi_gro_receive+0xc5/0xf0
[ 37.287232] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 37.293170] ? skb_release_all+0x24/0x30
[ 37.297708] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 37.303281] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 37.308752] ? __switch_to_asm+0x34/0x70
[ 37.313266] ? __switch_to_asm+0x40/0x70
[ 37.317768] ? __switch_to_asm+0x34/0x70
[ 37.322259] ? __switch_to_asm+0x40/0x70
[ 37.326742] ? __switch_to_asm+0x34/0x70
[ 37.331214] net_rx_action+0x140/0x3a0
[ 37.335519] ? __switch_to+0xad/0x500
[ 37.339728] __do_softirq+0xe4/0x2bb
[ 37.343852] run_ksoftirqd+0x2b/0x40
[ 37.347971] smpboot_thread_fn+0xfc/0x170
[ 37.352522] kthread+0x121/0x140
[ 37.356282] ? sort_range+0x30/0x30
[ 37.360303] ? kthread_create_worker_on_cpu+0x70/0x70
[ 37.365895] ret_from_fork+0x35/0x40
[ 37.370010] ---[ end trace c4bada38847a0076 ]---
^ permalink raw reply
* Re: Kernel 4.19 network performance - forwarding/routing normal users traffic
From: Jesper Dangaard Brouer @ 2018-11-10 19:34 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Saeed Mahameed, netdev@vger.kernel.org, brouer
In-Reply-To: <e6ece370-bf62-5d36-0417-779d2345fc8d@itcare.pl>
On Fri, 9 Nov 2018 23:20:38 +0100 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 08.11.2018 o 20:12, Paweł Staszewski pisze:
> > CPU load is lower than for connectx4 - but it looks like bandwidth
> > limit is the same :)
> > But also after reaching 60Gbit/60Gbit
> >
> > bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
> > input: /proc/net/dev type: rate
> > - iface Rx Tx Total
> > ==========================================================================
> >
> > enp175s0: 45.09 Gb/s 15.09 Gb/s 60.18 Gb/s
> > enp216s0: 15.14 Gb/s 45.19 Gb/s 60.33 Gb/s
> > --------------------------------------------------------------------------
> >
> > total: 60.45 Gb/s 60.48 Gb/s 120.93 Gb/s
>
> Today reached 65/65Gbit/s
>
> But starting from 60Gbit/s RX / 60Gbit TX nics start to drop packets
> (with 50%CPU on all 28cores) - so still there is cpu power to use :).
This is weird!
How do you see / measure these drops?
> So checked other stats.
> softnet_stats shows average 1k squeezed per sec:
Is below output the raw counters? not per sec?
It would be valuable to see the per sec stats instead...
I use this tool:
https://github.com/netoptimizer/network-testing/blob/master/bin/softnet_stat.pl
> cpu total dropped squeezed collision rps flow_limit
> 0 18554 0 1 0 0 0
> 1 16728 0 1 0 0 0
> 2 18033 0 1 0 0 0
> 3 17757 0 1 0 0 0
> 4 18861 0 0 0 0 0
> 5 0 0 1 0 0 0
> 6 2 0 1 0 0 0
> 7 0 0 1 0 0 0
> 8 0 0 0 0 0 0
> 9 0 0 1 0 0 0
> 10 0 0 0 0 0 0
> 11 0 0 1 0 0 0
> 12 50 0 1 0 0 0
> 13 257 0 0 0 0 0
> 14 3629115363 0 3353259 0 0 0
> 15 255167835 0 3138271 0 0 0
> 16 4240101961 0 3036130 0 0 0
> 17 599810018 0 3072169 0 0 0
> 18 432796524 0 3034191 0 0 0
> 19 41803906 0 3037405 0 0 0
> 20 900382666 0 3112294 0 0 0
> 21 620926085 0 3086009 0 0 0
> 22 41861198 0 3023142 0 0 0
> 23 4090425574 0 2990412 0 0 0
> 24 4264870218 0 3010272 0 0 0
> 25 141401811 0 3027153 0 0 0
> 26 104155188 0 3051251 0 0 0
> 27 4261258691 0 3039765 0 0 0
> 28 4 0 1 0 0 0
> 29 4 0 0 0 0 0
> 30 0 0 1 0 0 0
> 31 0 0 0 0 0 0
> 32 3 0 1 0 0 0
> 33 1 0 1 0 0 0
> 34 0 0 1 0 0 0
> 35 0 0 0 0 0 0
> 36 0 0 1 0 0 0
> 37 0 0 1 0 0 0
> 38 0 0 1 0 0 0
> 39 0 0 1 0 0 0
> 40 0 0 0 0 0 0
> 41 0 0 1 0 0 0
> 42 299758202 0 3139693 0 0 0
> 43 4254727979 0 3103577 0 0 0
> 44 1959555543 0 2554885 0 0 0
> 45 1675702723 0 2513481 0 0 0
> 46 1908435503 0 2519698 0 0 0
> 47 1877799710 0 2537768 0 0 0
> 48 2384274076 0 2584673 0 0 0
> 49 2598104878 0 2593616 0 0 0
> 50 1897566829 0 2530857 0 0 0
> 51 1712741629 0 2489089 0 0 0
> 52 1704033648 0 2495892 0 0 0
> 53 1636781820 0 2499783 0 0 0
> 54 1861997734 0 2541060 0 0 0
> 55 2113521616 0 2555673 0 0 0
>
>
> So i rised netdev backlog and budged to rly high values
> 524288 for netdev_budget and same for backlog
Does it affect the squeezed counters?
Notice, this (crazy) huge netdev_budget limit will also be limited
by /proc/sys/net/core/netdev_budget_usecs.
> This rised sortirqs from about 600k/sec to 800k/sec for NET_TX/NET_RX
Hmmm, this could indicated not enough NAPI bulking is occurring.
I have a BPF tool, that can give you some insight into NAPI bulking and
softirq idle/kthread starting. Called 'napi_monitor', could you try to
run this, so can try to understand this? You find the tool here:
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_user.c
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/napi_monitor_kern.c
> But after this changes i have less packets drops.
>
>
> Below perf top from max traffic reached:
> PerfTop: 72230 irqs/sec kernel:99.4% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ------------------------------------------------------------------------------------------
>
> 12.62% [kernel] [k] mlx5e_skb_from_cqe_mpwrq_linear
> 8.44% [kernel] [k] mlx5e_sq_xmit
> 6.69% [kernel] [k] build_skb
> 5.21% [kernel] [k] fib_table_lookup
> 3.54% [kernel] [k] memcpy_erms
> 3.20% [kernel] [k] mlx5e_poll_rx_cq
> 2.25% [kernel] [k] vlan_do_receive
> 2.20% [kernel] [k] mlx5e_post_rx_mpwqes
> 2.02% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
> 1.95% [kernel] [k] __dev_queue_xmit
> 1.83% [kernel] [k] dev_gro_receive
> 1.79% [kernel] [k] tcp_gro_receive
> 1.73% [kernel] [k] ip_finish_output2
> 1.63% [kernel] [k] mlx5e_poll_tx_cq
> 1.49% [kernel] [k] ipt_do_table
> 1.38% [kernel] [k] inet_gro_receive
> 1.31% [kernel] [k] __netif_receive_skb_core
> 1.30% [kernel] [k] _raw_spin_lock
> 1.28% [kernel] [k] mlx5_eq_int
> 1.24% [kernel] [k] irq_entries_start
> 1.19% [kernel] [k] __build_skb
> 1.15% [kernel] [k] swiotlb_map_page
> 1.02% [kernel] [k] vlan_dev_hard_start_xmit
> 0.94% [kernel] [k] pfifo_fast_dequeue
> 0.92% [kernel] [k] ip_route_input_rcu
> 0.86% [kernel] [k] kmem_cache_alloc
> 0.80% [kernel] [k] mlx5e_xmit
> 0.79% [kernel] [k] dev_hard_start_xmit
> 0.78% [kernel] [k] _raw_spin_lock_irqsave
> 0.74% [kernel] [k] ip_forward
> 0.72% [kernel] [k] tasklet_action_common.isra.21
> 0.68% [kernel] [k] pfifo_fast_enqueue
> 0.67% [kernel] [k] netif_skb_features
> 0.66% [kernel] [k] skb_segment
> 0.60% [kernel] [k] skb_gro_receive
> 0.56% [kernel] [k] validate_xmit_skb.isra.142
> 0.53% [kernel] [k] skb_release_data
> 0.51% [kernel] [k] mlx5e_page_release
> 0.51% [kernel] [k] ip_rcv_core.isra.20.constprop.25
> 0.51% [kernel] [k] __qdisc_run
> 0.50% [kernel] [k] tcp4_gro_receive
> 0.49% [kernel] [k] page_frag_free
> 0.46% [kernel] [k] kmem_cache_free_bulk
> 0.43% [kernel] [k] kmem_cache_free
> 0.42% [kernel] [k] try_to_wake_up
> 0.39% [kernel] [k] _raw_spin_lock_irq
> 0.39% [kernel] [k] find_busiest_group
> 0.37% [kernel] [k] __memcpy
>
>
>
> Remember those tests are now on two separate connectx5 connected to
> two separate pcie x16 gen 3.0
That is strange... I still suspect some HW NIC issue, can you provide
ethtool stats info via tool:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
$ ethtool_stats.pl --dev enp175s0 --dev enp216s0
The tool remove zero-stats counters and report per sec stats. It makes
it easier to spot that is relevant for the given workload.
Can you give output put from:
$ ethtool --show-priv-flag DEVICE
I want you to experiment with:
ethtool --set-priv-flags DEVICE rx_striding_rq off
I think you already have played with 'rx_cqe_compress', right.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* [net-next 1/1] tipc: improve broadcast retransmission algorithm
From: Jon Maloy @ 2018-11-10 19:23 UTC (permalink / raw)
To: netdev, davem
Cc: gordan.mihaljevic, tung.q.nguyen, hoang.h.le, jon.maloy, maloy,
xinl, ying.xue, tipc-discussion, LUU Duc Canh
From: LUU Duc Canh <canh.d.luu@dektech.com.au>
Currently, the broadcast retransmission algorithm is using the
'prev_retr' field in struct tipc_link to time stamp the latest broadcast
retransmission occasion. This helps to restrict retransmission of
individual broadcast packets to max once per 10 milliseconds, even
though all other criteria for retransmission are met.
We now move this time stamp to the control block of each individual
packet, and remove other limiting criteria. This simplifies the
retransmission algorithm, and eliminates any risk of logical errors
in selecting which packets can be retransmitted.
Acked-by: Ying Xue <ying.xue@windriver.com>
Signed-off-by: LUU Duc Canh <canh.d.luu@dektech.com.au>
Signed-off-by: Jon Maloy <jon.maloy@ericsson.com>
---
net/tipc/link.c | 59 +++++++++++++--------------------------------------------
net/tipc/msg.h | 1 +
2 files changed, 14 insertions(+), 46 deletions(-)
diff --git a/net/tipc/link.c b/net/tipc/link.c
index 201c3b5..aefb5b4 100644
--- a/net/tipc/link.c
+++ b/net/tipc/link.c
@@ -105,7 +105,7 @@ struct tipc_stats {
* @transmitq: queue for sent, non-acked messages
* @backlogq: queue for messages waiting to be sent
* @snt_nxt: next sequence number to use for outbound messages
- * @last_retransmitted: sequence number of most recently retransmitted message
+ * @prev_from: sequence number of most previous retransmission request
* @stale_cnt: counter for number of identical retransmit attempts
* @stale_limit: time when repeated identical retransmits must force link reset
* @ackers: # of peers that needs to ack each packet before it can be released
@@ -163,7 +163,7 @@ struct tipc_link {
u16 limit;
} backlog[5];
u16 snd_nxt;
- u16 last_retransm;
+ u16 prev_from;
u16 window;
u16 stale_cnt;
unsigned long stale_limit;
@@ -186,9 +186,6 @@ struct tipc_link {
u16 acked;
struct tipc_link *bc_rcvlink;
struct tipc_link *bc_sndlink;
- unsigned long prev_retr;
- u16 prev_from;
- u16 prev_to;
u8 nack_state;
bool bc_peer_is_up;
@@ -210,7 +207,7 @@ enum {
BC_NACK_SND_SUPPRESS,
};
-#define TIPC_BC_RETR_LIMIT 10 /* [ms] */
+#define TIPC_BC_RETR_LIM msecs_to_jiffies(10) /* [ms] */
/*
* Interval between NACKs when packets arrive out of order
@@ -1036,10 +1033,12 @@ static int tipc_link_retrans(struct tipc_link *l, struct tipc_link *r,
if (!skb)
return 0;
+ if (less(to, from))
+ return 0;
/* Detect repeated retransmit failures on same packet */
- if (r->last_retransm != buf_seqno(skb)) {
- r->last_retransm = buf_seqno(skb);
+ if (r->prev_from != from) {
+ r->prev_from = from;
r->stale_limit = jiffies + msecs_to_jiffies(r->tolerance);
r->stale_cnt = 0;
} else if (++r->stale_cnt > 99 && time_after(jiffies, r->stale_limit)) {
@@ -1055,6 +1054,11 @@ static int tipc_link_retrans(struct tipc_link *l, struct tipc_link *r,
continue;
if (more(msg_seqno(hdr), to))
break;
+ if (link_is_bc_sndlink(l)) {
+ if (time_before(jiffies, TIPC_SKB_CB(skb)->nxt_retr))
+ continue;
+ TIPC_SKB_CB(skb)->nxt_retr = jiffies + TIPC_BC_RETR_LIM;
+ }
_skb = __pskb_copy(skb, MIN_H_SIZE, GFP_ATOMIC);
if (!_skb)
return 0;
@@ -1734,42 +1738,6 @@ void tipc_link_bc_init_rcv(struct tipc_link *l, struct tipc_msg *hdr)
l->rcv_nxt = peers_snd_nxt;
}
-/* link_bc_retr eval()- check if the indicated range can be retransmitted now
- * - Adjust permitted range if there is overlap with previous retransmission
- */
-static bool link_bc_retr_eval(struct tipc_link *l, u16 *from, u16 *to)
-{
- unsigned long elapsed = jiffies_to_msecs(jiffies - l->prev_retr);
-
- if (less(*to, *from))
- return false;
-
- /* New retransmission request */
- if ((elapsed > TIPC_BC_RETR_LIMIT) ||
- less(*to, l->prev_from) || more(*from, l->prev_to)) {
- l->prev_from = *from;
- l->prev_to = *to;
- l->prev_retr = jiffies;
- return true;
- }
-
- /* Inside range of previous retransmit */
- if (!less(*from, l->prev_from) && !more(*to, l->prev_to))
- return false;
-
- /* Fully or partially outside previous range => exclude overlap */
- if (less(*from, l->prev_from)) {
- *to = l->prev_from - 1;
- l->prev_from = *from;
- }
- if (more(*to, l->prev_to)) {
- *from = l->prev_to + 1;
- l->prev_to = *to;
- }
- l->prev_retr = jiffies;
- return true;
-}
-
/* tipc_link_bc_sync_rcv - update rcv link according to peer's send state
*/
int tipc_link_bc_sync_rcv(struct tipc_link *l, struct tipc_msg *hdr,
@@ -1800,8 +1768,7 @@ int tipc_link_bc_sync_rcv(struct tipc_link *l, struct tipc_msg *hdr,
if (more(peers_snd_nxt, l->rcv_nxt + l->window))
return rc;
- if (link_bc_retr_eval(snd_l, &from, &to))
- rc = tipc_link_retrans(snd_l, l, from, to, xmitq);
+ rc = tipc_link_retrans(snd_l, l, from, to, xmitq);
l->snd_nxt = peers_snd_nxt;
if (link_bc_rcv_gap(l))
diff --git a/net/tipc/msg.h b/net/tipc/msg.h
index a2879e6..a092495 100644
--- a/net/tipc/msg.h
+++ b/net/tipc/msg.h
@@ -105,6 +105,7 @@ struct tipc_skb_cb {
u32 bytes_read;
u32 orig_member;
struct sk_buff *tail;
+ unsigned long nxt_retr;
bool validated;
u16 chain_imp;
u16 ackers;
--
2.1.4
^ permalink raw reply related
* Re: [PATCH bpf-next 0/3] bpf: Allow narrow loads with offset > 0
From: Alexei Starovoitov @ 2018-11-10 19:13 UTC (permalink / raw)
To: Andrey Ignatov; +Cc: netdev, ast, daniel, yhs, kernel-team
In-Reply-To: <cover.1541553332.git.rdna@fb.com>
On Tue, Nov 06, 2018 at 05:23:25PM -0800, Andrey Ignatov wrote:
> This patch set adds support for narrow loads with offset > 0 to BPF
> verifier.
>
> Patch 1 provides more details and is the main patch in the set.
> Patches 2 and 3 add new test cases to test_verifier and test_sock_addr
> selftests.
Applied, Thanks
^ permalink raw reply
* [PATCH net-next 6/6] net: remove VLAN_TAG_PRESENT
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Alexei Starovoitov, Benjamin Herrenschmidt, Daniel Borkmann,
David S. Miller, James Hogan, linux-mips, linuxppc-dev,
Michael Ellerman, Paul Burton, Paul Mackerras, Ralf Baechle,
sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Replace VLAN_TAG_PRESENT with single bit flag and free up
VLAN.CFI overload. Now VLAN.CFI is visible in networking stack
and can be passed around intact.
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/mips/net/bpf_jit.c | 3 ---
arch/powerpc/net/bpf_jit_comp.c | 3 ---
arch/sparc/net/bpf_jit_comp_32.c | 4 ----
include/linux/if_vlan.h | 11 ++++++-----
include/linux/skbuff.h | 16 +++++++++-------
lib/test_bpf.c | 14 ++++++++------
net/core/filter.c | 6 ------
7 files changed, 23 insertions(+), 34 deletions(-)
diff --git a/arch/mips/net/bpf_jit.c b/arch/mips/net/bpf_jit.c
index de4c6372ad9a..3a0e34f4e615 100644
--- a/arch/mips/net/bpf_jit.c
+++ b/arch/mips/net/bpf_jit.c
@@ -1164,9 +1164,6 @@ static int build_body(struct jit_ctx *ctx)
vlan_tci) != 2);
off = offsetof(struct sk_buff, vlan_tci);
emit_half_load_unsigned(r_A, r_skb, off, ctx);
-#ifdef VLAN_TAG_PRESENT
- emit_andi(r_A, r_A, (u16)~VLAN_TAG_PRESENT, ctx);
-#endif
break;
case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
ctx->flags |= SEEN_SKB | SEEN_A;
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index dc4a2f54e829..91d223cf512b 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -383,9 +383,6 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
PPC_LHZ_OFFS(r_A, r_skb, offsetof(struct sk_buff,
vlan_tci));
-#ifdef VLAN_TAG_PRESENT
- PPC_ANDI(r_A, r_A, ~VLAN_TAG_PRESENT);
-#endif
break;
case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
PPC_LBZ_OFFS(r_A, r_skb, PKT_VLAN_PRESENT_OFFSET());
diff --git a/arch/sparc/net/bpf_jit_comp_32.c b/arch/sparc/net/bpf_jit_comp_32.c
index 48f3c04dd179..84cc8f7f83e9 100644
--- a/arch/sparc/net/bpf_jit_comp_32.c
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -553,10 +553,6 @@ void bpf_jit_compile(struct bpf_prog *fp)
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
emit_skb_load16(vlan_tci, r_A);
-#ifdef VLAN_TAG_PRESENT
- emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
- emit_and(r_A, r_TMP, r_A);
-#endif
break;
case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
__emit_skb_load8(__pkt_vlan_present_offset, r_A);
diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h
index 1be5230921b5..7a541eadf78e 100644
--- a/include/linux/if_vlan.h
+++ b/include/linux/if_vlan.h
@@ -66,7 +66,6 @@ static inline struct vlan_ethhdr *vlan_eth_hdr(const struct sk_buff *skb)
#define VLAN_PRIO_MASK 0xe000 /* Priority Code Point */
#define VLAN_PRIO_SHIFT 13
#define VLAN_CFI_MASK 0x1000 /* Canonical Format Indicator */
-#define VLAN_TAG_PRESENT VLAN_CFI_MASK
#define VLAN_VID_MASK 0x0fff /* VLAN Identifier */
#define VLAN_N_VID 4096
@@ -78,8 +77,8 @@ static inline bool is_vlan_dev(const struct net_device *dev)
return dev->priv_flags & IFF_802_1Q_VLAN;
}
-#define skb_vlan_tag_present(__skb) ((__skb)->vlan_tci & VLAN_TAG_PRESENT)
-#define skb_vlan_tag_get(__skb) ((__skb)->vlan_tci & ~VLAN_TAG_PRESENT)
+#define skb_vlan_tag_present(__skb) ((__skb)->vlan_present)
+#define skb_vlan_tag_get(__skb) ((__skb)->vlan_tci)
#define skb_vlan_tag_get_id(__skb) ((__skb)->vlan_tci & VLAN_VID_MASK)
#define skb_vlan_tag_get_prio(__skb) (((__skb)->vlan_tci & VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT)
@@ -480,7 +479,7 @@ static inline struct sk_buff *vlan_insert_tag_set_proto(struct sk_buff *skb,
*/
static inline void __vlan_hwaccel_clear_tag(struct sk_buff *skb)
{
- skb->vlan_tci = 0;
+ skb->vlan_present = 0;
}
/**
@@ -492,6 +491,7 @@ static inline void __vlan_hwaccel_clear_tag(struct sk_buff *skb)
*/
static inline void __vlan_hwaccel_copy_tag(struct sk_buff *dst, const struct sk_buff *src)
{
+ dst->vlan_present = src->vlan_present;
dst->vlan_proto = src->vlan_proto;
dst->vlan_tci = src->vlan_tci;
}
@@ -526,7 +526,8 @@ static inline void __vlan_hwaccel_put_tag(struct sk_buff *skb,
__be16 vlan_proto, u16 vlan_tci)
{
skb->vlan_proto = vlan_proto;
- skb->vlan_tci = VLAN_TAG_PRESENT | vlan_tci;
+ skb->vlan_tci = vlan_tci;
+ skb->vlan_present = 1;
}
/**
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 99f38779332c..b9aa0d1b21cf 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -777,6 +777,14 @@ struct sk_buff {
__u8 encap_hdr_csum:1;
__u8 csum_valid:1;
+#ifdef __BIG_ENDIAN_BITFIELD
+#define PKT_VLAN_PRESENT_BIT 7
+#else
+#define PKT_VLAN_PRESENT_BIT 0
+#endif
+#define PKT_VLAN_PRESENT_OFFSET() offsetof(struct sk_buff, __pkt_vlan_present_offset)
+ __u8 __pkt_vlan_present_offset[0];
+ __u8 vlan_present:1;
__u8 csum_complete_sw:1;
__u8 csum_level:2;
__u8 csum_not_inet:1;
@@ -784,8 +792,8 @@ struct sk_buff {
#ifdef CONFIG_IPV6_NDISC_NODETYPE
__u8 ndisc_nodetype:2;
#endif
+
__u8 ipvs_property:1;
-
__u8 inner_protocol_type:1;
__u8 remcsum_offload:1;
#ifdef CONFIG_NET_SWITCHDEV
@@ -816,12 +824,6 @@ struct sk_buff {
__u32 priority;
int skb_iif;
__u32 hash;
-#define PKT_VLAN_PRESENT_BIT 4 // CFI (12-th bit) in TCI
-#ifdef __BIG_ENDIAN
-#define PKT_VLAN_PRESENT_OFFSET() offsetof(struct sk_buff, vlan_tci)
-#else
-#define PKT_VLAN_PRESENT_OFFSET() (offsetof(struct sk_buff, vlan_tci) + 1)
-#endif
__be16 vlan_proto;
__u16 vlan_tci;
#if defined(CONFIG_NET_RX_BUSY_POLL) || defined(CONFIG_XPS)
diff --git a/lib/test_bpf.c b/lib/test_bpf.c
index aa22bcaec1dc..f3e570722a7e 100644
--- a/lib/test_bpf.c
+++ b/lib/test_bpf.c
@@ -39,6 +39,7 @@
#define SKB_HASH 0x1234aaab
#define SKB_QUEUE_MAP 123
#define SKB_VLAN_TCI 0xffff
+#define SKB_VLAN_PRESENT 1
#define SKB_DEV_IFINDEX 577
#define SKB_DEV_TYPE 588
@@ -725,8 +726,8 @@ static struct bpf_test tests[] = {
CLASSIC,
{ },
{
- { 1, SKB_VLAN_TCI & ~VLAN_TAG_PRESENT },
- { 10, SKB_VLAN_TCI & ~VLAN_TAG_PRESENT }
+ { 1, SKB_VLAN_TCI },
+ { 10, SKB_VLAN_TCI }
},
},
{
@@ -739,8 +740,8 @@ static struct bpf_test tests[] = {
CLASSIC,
{ },
{
- { 1, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) },
- { 10, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) }
+ { 1, SKB_VLAN_PRESENT },
+ { 10, SKB_VLAN_PRESENT }
},
},
{
@@ -5289,8 +5290,8 @@ static struct bpf_test tests[] = {
#endif
{ },
{
- { 1, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) },
- { 10, !!(SKB_VLAN_TCI & VLAN_TAG_PRESENT) }
+ { 1, SKB_VLAN_PRESENT },
+ { 10, SKB_VLAN_PRESENT }
},
.fill_helper = bpf_fill_maxinsns6,
.expected_errcode = -ENOTSUPP,
@@ -6493,6 +6494,7 @@ static struct sk_buff *populate_skb(char *buf, int size)
skb->hash = SKB_HASH;
skb->queue_mapping = SKB_QUEUE_MAP;
skb->vlan_tci = SKB_VLAN_TCI;
+ skb->vlan_present = SKB_VLAN_PRESENT;
skb->vlan_proto = htons(ETH_P_IP);
dev_net_set(&dev, &init_net);
skb->dev = &dev;
diff --git a/net/core/filter.c b/net/core/filter.c
index c151b906df53..10acbc00ff6c 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -301,9 +301,6 @@ static u32 convert_skb_access(int skb_field, int dst_reg, int src_reg,
/* dst_reg = *(u16 *) (src_reg + offsetof(vlan_tci)) */
*insn++ = BPF_LDX_MEM(BPF_H, dst_reg, src_reg,
offsetof(struct sk_buff, vlan_tci));
-#ifdef VLAN_TAG_PRESENT
- *insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg, ~VLAN_TAG_PRESENT);
-#endif
break;
case SKF_AD_VLAN_TAG_PRESENT:
*insn++ = BPF_LDX_MEM(BPF_B, dst_reg, src_reg, PKT_VLAN_PRESENT_OFFSET());
@@ -6152,9 +6149,6 @@ static u32 bpf_convert_ctx_access(enum bpf_access_type type,
*insn++ = BPF_LDX_MEM(BPF_H, si->dst_reg, si->src_reg,
bpf_target_off(struct sk_buff, vlan_tci, 2,
target_size));
-#ifdef VLAN_TAG_PRESENT
- *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, ~VLAN_TAG_PRESENT);
-#endif
break;
case offsetof(struct __sk_buff, cb[0]) ...
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 5/6] net/bpf_jit: SPARC: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: David S. Miller, sparclinux, Alexei Starovoitov,
Benjamin Herrenschmidt, Daniel Borkmann, James Hogan, linux-mips,
linuxppc-dev, Michael Ellerman, Paul Burton, Paul Mackerras,
Ralf Baechle
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/sparc/net/bpf_jit_comp_32.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/arch/sparc/net/bpf_jit_comp_32.c b/arch/sparc/net/bpf_jit_comp_32.c
index a5ff88643d5c..48f3c04dd179 100644
--- a/arch/sparc/net/bpf_jit_comp_32.c
+++ b/arch/sparc/net/bpf_jit_comp_32.c
@@ -552,15 +552,18 @@ void bpf_jit_compile(struct bpf_prog *fp)
emit_skb_load32(hash, r_A);
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
- case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
emit_skb_load16(vlan_tci, r_A);
- if (code != (BPF_ANC | SKF_AD_VLAN_TAG)) {
- emit_alu_K(SRL, 12);
+#ifdef VLAN_TAG_PRESENT
+ emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
+ emit_and(r_A, r_TMP, r_A);
+#endif
+ break;
+ case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
+ __emit_skb_load8(__pkt_vlan_present_offset, r_A);
+ if (PKT_VLAN_PRESENT_BIT)
+ emit_alu_K(SRL, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
emit_andi(r_A, 1, r_A);
- } else {
- emit_loadimm(~VLAN_TAG_PRESENT, r_TMP);
- emit_and(r_A, r_TMP, r_A);
- }
break;
case BPF_LD | BPF_W | BPF_LEN:
emit_skb_load32(len, r_A);
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 3/6] net/bpf_jit: MIPS: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Ralf Baechle, Paul Burton, James Hogan, linux-mips,
Alexei Starovoitov, Benjamin Herrenschmidt, Daniel Borkmann,
David S. Miller, linuxppc-dev, Michael Ellerman, Paul Mackerras,
sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/mips/net/bpf_jit.c | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/arch/mips/net/bpf_jit.c b/arch/mips/net/bpf_jit.c
index 4d8cb9bb8365..de4c6372ad9a 100644
--- a/arch/mips/net/bpf_jit.c
+++ b/arch/mips/net/bpf_jit.c
@@ -1159,19 +1159,22 @@ static int build_body(struct jit_ctx *ctx)
emit_load(r_A, r_skb, off, ctx);
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
- case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
ctx->flags |= SEEN_SKB | SEEN_A;
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff,
vlan_tci) != 2);
off = offsetof(struct sk_buff, vlan_tci);
- emit_half_load_unsigned(r_s0, r_skb, off, ctx);
- if (code == (BPF_ANC | SKF_AD_VLAN_TAG)) {
- emit_andi(r_A, r_s0, (u16)~VLAN_TAG_PRESENT, ctx);
- } else {
- emit_andi(r_A, r_s0, VLAN_TAG_PRESENT, ctx);
- /* return 1 if present */
- emit_sltu(r_A, r_zero, r_A, ctx);
- }
+ emit_half_load_unsigned(r_A, r_skb, off, ctx);
+#ifdef VLAN_TAG_PRESENT
+ emit_andi(r_A, r_A, (u16)~VLAN_TAG_PRESENT, ctx);
+#endif
+ break;
+ case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
+ ctx->flags |= SEEN_SKB | SEEN_A;
+ emit_load_byte(r_A, r_skb, PKT_VLAN_PRESENT_OFFSET(), ctx);
+ if (PKT_VLAN_PRESENT_BIT)
+ emit_srl(r_A, r_A, PKT_VLAN_PRESENT_BIT, ctx);
+ if (PKT_VLAN_PRESENT_BIT < 7)
+ emit_andi(r_A, r_A, 1, ctx);
break;
case BPF_ANC | SKF_AD_PKTTYPE:
ctx->flags |= SEEN_SKB;
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 4/6] net/bpf_jit: PPC: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman,
linuxppc-dev, Alexei Starovoitov, Daniel Borkmann,
David S. Miller, James Hogan, linux-mips, Paul Burton,
Ralf Baechle, sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
arch/powerpc/net/bpf_jit_comp.c | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index d5bfe24bb3b5..dc4a2f54e829 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -379,18 +379,20 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
hash));
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
- case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, vlan_tci) != 2);
- BUILD_BUG_ON(VLAN_TAG_PRESENT != 0x1000);
PPC_LHZ_OFFS(r_A, r_skb, offsetof(struct sk_buff,
vlan_tci));
- if (code == (BPF_ANC | SKF_AD_VLAN_TAG)) {
- PPC_ANDI(r_A, r_A, ~VLAN_TAG_PRESENT);
- } else {
- PPC_ANDI(r_A, r_A, VLAN_TAG_PRESENT);
- PPC_SRWI(r_A, r_A, 12);
- }
+#ifdef VLAN_TAG_PRESENT
+ PPC_ANDI(r_A, r_A, ~VLAN_TAG_PRESENT);
+#endif
+ break;
+ case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
+ PPC_LBZ_OFFS(r_A, r_skb, PKT_VLAN_PRESENT_OFFSET());
+ if (PKT_VLAN_PRESENT_BIT)
+ PPC_SRWI(r_A, r_A, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
+ PPC_ANDI(r_A, r_A, 1);
break;
case BPF_ANC | SKF_AD_QUEUE:
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff,
--
2.19.1
^ permalink raw reply related
* [PATCH net-next 2/6] net/bpf: split VLAN_PRESENT bit handling from VLAN_TCI
From: Michał Mirosław @ 2018-11-10 18:58 UTC (permalink / raw)
To: netdev
Cc: Alexei Starovoitov, Daniel Borkmann, Benjamin Herrenschmidt,
David S. Miller, James Hogan, linux-mips, linuxppc-dev,
Michael Ellerman, Paul Burton, Paul Mackerras, Ralf Baechle,
sparclinux
In-Reply-To: <cover.1541876179.git.mirq-linux@rere.qmqm.pl>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
---
net/core/filter.c | 40 +++++++++++++++++++++-------------------
1 file changed, 21 insertions(+), 19 deletions(-)
diff --git a/net/core/filter.c b/net/core/filter.c
index e521c5ebc7d1..c151b906df53 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -296,22 +296,21 @@ static u32 convert_skb_access(int skb_field, int dst_reg, int src_reg,
break;
case SKF_AD_VLAN_TAG:
- case SKF_AD_VLAN_TAG_PRESENT:
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, vlan_tci) != 2);
- BUILD_BUG_ON(VLAN_TAG_PRESENT != 0x1000);
/* dst_reg = *(u16 *) (src_reg + offsetof(vlan_tci)) */
*insn++ = BPF_LDX_MEM(BPF_H, dst_reg, src_reg,
offsetof(struct sk_buff, vlan_tci));
- if (skb_field == SKF_AD_VLAN_TAG) {
- *insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg,
- ~VLAN_TAG_PRESENT);
- } else {
- /* dst_reg >>= 12 */
- *insn++ = BPF_ALU32_IMM(BPF_RSH, dst_reg, 12);
- /* dst_reg &= 1 */
+#ifdef VLAN_TAG_PRESENT
+ *insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg, ~VLAN_TAG_PRESENT);
+#endif
+ break;
+ case SKF_AD_VLAN_TAG_PRESENT:
+ *insn++ = BPF_LDX_MEM(BPF_B, dst_reg, src_reg, PKT_VLAN_PRESENT_OFFSET());
+ if (PKT_VLAN_PRESENT_BIT)
+ *insn++ = BPF_ALU32_IMM(BPF_RSH, dst_reg, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
*insn++ = BPF_ALU32_IMM(BPF_AND, dst_reg, 1);
- }
break;
}
@@ -6140,19 +6139,22 @@ static u32 bpf_convert_ctx_access(enum bpf_access_type type,
break;
case offsetof(struct __sk_buff, vlan_present):
+ *target_size = 1;
+ *insn++ = BPF_LDX_MEM(BPF_B, si->dst_reg, si->src_reg,
+ PKT_VLAN_PRESENT_OFFSET());
+ if (PKT_VLAN_PRESENT_BIT)
+ *insn++ = BPF_ALU32_IMM(BPF_RSH, si->dst_reg, PKT_VLAN_PRESENT_BIT);
+ if (PKT_VLAN_PRESENT_BIT < 7)
+ *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, 1);
+ break;
+
case offsetof(struct __sk_buff, vlan_tci):
- BUILD_BUG_ON(VLAN_TAG_PRESENT != 0x1000);
-
*insn++ = BPF_LDX_MEM(BPF_H, si->dst_reg, si->src_reg,
bpf_target_off(struct sk_buff, vlan_tci, 2,
target_size));
- if (si->off == offsetof(struct __sk_buff, vlan_tci)) {
- *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg,
- ~VLAN_TAG_PRESENT);
- } else {
- *insn++ = BPF_ALU32_IMM(BPF_RSH, si->dst_reg, 12);
- *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, 1);
- }
+#ifdef VLAN_TAG_PRESENT
+ *insn++ = BPF_ALU32_IMM(BPF_AND, si->dst_reg, ~VLAN_TAG_PRESENT);
+#endif
break;
case offsetof(struct __sk_buff, cb[0]) ...
--
2.19.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox