* [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-01-08 10:49 Jaroslav Pulchart
2024-01-10 6:45 ` Jaroslav Pulchart
` (2 more replies)
0 siblings, 3 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-01-08 10:49 UTC (permalink / raw)
To: Jesse Brandeburg, Tony Nguyen, intel-wired-lan, Dave Ertman
Cc: Igor Raits, Daniel Secik
[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]
Hello
I would like to report a regression triggered by recent change in
Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
bisected and the regression is triggered by
fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
check for SRIOV and LAG" commit and originally reported as part of
https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
thread.
> However, after the following patch we see that more NUMA nodes have
> such a low amount of memory and that is causing constant reclaiming
> of memory because it looks like something inside of the kernel ate all
> the memory. This is right after the start of the system as well.
I'm reporting it here as it is a different problem than the original
thread. The commit introduces a low memory problem per each numa node
of the first socket (node0 .. node3 in our case) and cause constant
kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
memory issue is nicely visible in "numastat -m", see attached files:
* numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
* numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
the server "is fresh" (after reboot), without running any application load.
$ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
numastat_m-6.6.10_28GB_HP_no_revert.txt
numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
2756.89 2754.86 100.39 2278.43 < ice
fix is reverted, we have ~2GB free per numa, except one, like before
== no issue
numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
3551.29 1530.52 2212.04 3488.09
...
numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
127.52 66.49 120.23 263.47 <
ice fix is present, we see just few MB free per each node, this will
cause kswapd utilization!
numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
3322.18 3134.47 195.55 879.17
...
If you have some hints on how to debug what is actually occupying all
that memory and some fix of the problem will be nice. We can provide
testing and more reports if needed to analyze the issue. We reverted
the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
till we know a proper fix.
Best regards,
Jaroslav Pulchart
[-- Attachment #2: numastat_m-6.6.10_28GB_HP_ice_revert.txt --]
[-- Type: text/plain, Size: 6633 bytes --]
Node 0 Node 1 Node 2 Node 3
--------------- --------------- --------------- ---------------
MemTotal 32264.16 32701.87 32701.88 32686.87
MemFree 2756.89 2754.86 100.39 2278.43
MemUsed 29507.27 29947.01 32601.49 30408.44
Active 267.38 204.00 98.88 567.17
Inactive 22.58 10.24 3.62 38.77
Active(anon) 229.16 199.68 69.59 419.21
Inactive(anon) 0.00 0.00 0.00 0.00
Active(file) 38.22 4.32 29.29 147.96
Inactive(file) 22.58 10.24 3.62 38.77
Unevictable 10.79 0.32 0.01 2.87
Mlocked 10.79 0.32 0.01 2.87
Dirty 0.02 0.01 0.01 0.01
Writeback 0.00 0.00 0.00 0.00
FilePages 71.67 14.80 33.34 190.00
Mapped 23.41 1.25 18.92 145.39
AnonPages 211.82 187.18 68.89 400.52
Shmem 0.36 0.24 0.41 0.54
KernelStack 2.64 1.70 2.10 3.50
PageTables 1.09 0.87 0.45 2.71
NFS_Unstable 0.00 0.00 0.00 0.00
Bounce 0.00 0.00 0.00 0.00
WritebackTmp 0.00 0.00 0.00 0.00
Slab 50.88 32.03 20.61 88.15
SReclaimable 11.10 5.93 3.02 14.20
SUnreclaim 39.78 26.10 17.60 73.95
AnonHugePages 40.00 52.00 36.00 82.00
ShmemHugePages 0.00 0.00 0.00 0.00
ShmemPmdMapped 0.00 0.00 0.00 0.00
HugePages_Total 28672.00 28672.00 28672.00 28672.00
HugePages_Free 28672.00 28672.00 28672.00 28672.00
HugePages_Surp 0.00 0.00 0.00 0.00
KReclaimable 11.10 5.93 3.02 14.20
Node 4 Node 5 Node 6 Node 7
--------------- --------------- --------------- ---------------
MemTotal 32701.88 32701.87 32659.87 32696.75
MemFree 3551.29 1530.52 2212.04 3488.09
MemUsed 29150.59 31171.35 30447.83 29208.66
Active 287.19 195.57 257.13 181.00
Inactive 29.21 48.47 81.34 170.16
Active(anon) 251.84 138.33 205.82 122.58
Inactive(anon) 0.00 0.00 0.00 0.00
Active(file) 35.35 57.25 51.31 58.42
Inactive(file) 29.21 48.47 81.34 170.16
Unevictable 19.86 23.11 291.23 11.16
Mlocked 19.86 23.11 291.23 11.16
Dirty 0.02 0.02 0.02 0.01
Writeback 0.00 0.00 0.00 0.00
FilePages 79.42 116.06 136.51 233.12
Mapped 37.59 47.25 40.64 26.11
AnonPages 250.75 141.84 487.42 125.68
Shmem 0.49 8.59 0.50 0.23
KernelStack 2.35 4.29 3.48 1.84
PageTables 1.71 1.62 2.13 0.97
NFS_Unstable 0.00 0.00 0.00 0.00
Bounce 0.00 0.00 0.00 0.00
WritebackTmp 0.00 0.00 0.00 0.00
Slab 41.12 77.15 48.36 33.94
SReclaimable 16.09 15.53 12.07 6.84
SUnreclaim 25.04 61.62 36.29 27.11
AnonHugePages 170.00 92.00 184.00 80.00
ShmemHugePages 0.00 0.00 0.00 0.00
ShmemPmdMapped 0.00 0.00 0.00 0.00
HugePages_Total 28672.00 28672.00 28672.00 28672.00
HugePages_Free 28672.00 28672.00 28672.00 28672.00
HugePages_Surp 0.00 0.00 0.00 0.00
KReclaimable 16.09 15.53 12.07 6.84
Total
---------------
MemTotal 261115.14
MemFree 18672.50
MemUsed 242442.64
Active 2058.32
Inactive 404.39
Active(anon) 1636.21
Inactive(anon) 0.00
Active(file) 422.12
Inactive(file) 404.39
Unevictable 359.34
Mlocked 359.34
Dirty 0.12
Writeback 0.00
FilePages 874.93
Mapped 340.57
AnonPages 1874.10
Shmem 11.36
KernelStack 21.91
PageTables 11.55
NFS_Unstable 0.00
Bounce 0.00
WritebackTmp 0.00
Slab 392.25
SReclaimable 84.77
SUnreclaim 307.48
AnonHugePages 736.00
ShmemHugePages 0.00
ShmemPmdMapped 0.00
HugePages_Total 229376.00
HugePages_Free 229376.00
HugePages_Surp 0.00
KReclaimable 84.77
[-- Attachment #3: 6.6.9-kswapd_usage.png --]
[-- Type: image/png, Size: 189979 bytes --]
[-- Attachment #4: numastat_m-6.6.10_28GB_HP_no_revert.txt --]
[-- Type: text/plain, Size: 6633 bytes --]
Node 0 Node 1 Node 2 Node 3
--------------- --------------- --------------- ---------------
MemTotal 32264.16 32701.87 32701.88 32686.87
MemFree 127.52 66.49 120.23 263.47
MemUsed 32136.64 32635.38 32581.64 32423.40
Active 23.83 59.13 30.02 41.47
Inactive 19.67 31.62 43.53 103.41
Active(anon) 15.28 52.11 18.29 28.68
Inactive(anon) 18.08 25.77 26.57 7.29
Active(file) 8.55 7.02 11.73 12.79
Inactive(file) 1.59 5.85 16.96 96.12
Unevictable 1.94 12.32 0.50 19.13
Mlocked 1.94 12.32 0.50 19.13
Dirty 0.00 0.00 0.01 0.00
Writeback 0.00 0.00 0.00 0.00
FilePages 10.28 12.90 29.07 121.34
Mapped 6.10 4.62 5.55 26.80
AnonPages 34.06 90.17 43.08 39.95
Shmem 0.12 0.03 0.21 0.05
KernelStack 1.92 2.16 2.08 2.88
PageTables 0.34 0.62 0.30 1.97
NFS_Unstable 0.00 0.00 0.00 0.00
Bounce 0.00 0.00 0.00 0.00
WritebackTmp 0.00 0.00 0.00 0.00
Slab 42.06 29.30 35.05 166.25
SReclaimable 12.12 4.18 8.05 48.11
SUnreclaim 29.94 25.12 27.00 118.14
AnonHugePages 12.00 52.00 18.00 8.00
ShmemHugePages 0.00 0.00 0.00 0.00
ShmemPmdMapped 0.00 0.00 0.00 0.00
HugePages_Total 28672.00 28672.00 28672.00 28672.00
HugePages_Free 28672.00 28672.00 28672.00 28672.00
HugePages_Surp 0.00 0.00 0.00 0.00
KReclaimable 12.12 4.18 8.05 48.11
Node 4 Node 5 Node 6 Node 7
--------------- --------------- --------------- ---------------
MemTotal 32701.88 32659.86 32701.88 32696.75
MemFree 3322.18 3134.47 195.55 879.17
MemUsed 29379.69 29525.39 32506.32 31817.58
Active 350.11 460.66 424.54 433.99
Inactive 125.39 159.05 2362.56 2393.70
Active(anon) 250.39 370.38 282.00 353.43
Inactive(anon) 0.00 0.25 0.29 0.11
Active(file) 99.73 90.29 142.53 80.56
Inactive(file) 125.39 158.80 2362.28 2393.59
Unevictable 12.24 21.84 295.76 3.39
Mlocked 12.24 21.84 295.76 3.39
Dirty 0.00 0.02 0.01 0.01
Writeback 0.00 0.00 0.00 0.00
FilePages 229.27 264.37 2519.70 2476.24
Mapped 62.79 92.60 94.86 51.39
AnonPages 254.16 372.37 553.12 354.69
Shmem 0.33 0.52 8.82 0.64
KernelStack 2.49 2.71 4.30 3.56
PageTables 1.38 2.01 2.67 2.25
NFS_Unstable 0.00 0.00 0.00 0.00
Bounce 0.00 0.00 0.00 0.00
WritebackTmp 0.00 0.00 0.00 0.00
Slab 103.72 73.49 104.67 194.87
SReclaimable 62.41 20.92 69.53 145.89
SUnreclaim 41.31 52.57 35.14 48.98
AnonHugePages 122.00 90.00 176.00 182.00
ShmemHugePages 0.00 0.00 0.00 0.00
ShmemPmdMapped 0.00 0.00 0.00 0.00
HugePages_Total 28672.00 28672.00 28672.00 28672.00
HugePages_Free 28672.00 28672.00 28672.00 28672.00
HugePages_Surp 0.00 0.00 0.00 0.00
KReclaimable 62.41 20.92 69.53 145.89
Total
---------------
MemTotal 261115.14
MemFree 8109.09
MemUsed 253006.05
Active 1823.75
Inactive 5238.92
Active(anon) 1370.56
Inactive(anon) 78.36
Active(file) 453.19
Inactive(file) 5160.56
Unevictable 367.12
Mlocked 367.12
Dirty 0.05
Writeback 0.00
FilePages 5663.17
Mapped 344.71
AnonPages 1741.59
Shmem 10.73
KernelStack 22.09
PageTables 11.55
NFS_Unstable 0.00
Bounce 0.00
WritebackTmp 0.00
Slab 749.42
SReclaimable 371.21
SUnreclaim 378.20
AnonHugePages 660.00
ShmemHugePages 0.00
ShmemPmdMapped 0.00
HugePages_Total 229376.00
HugePages_Free 229376.00
HugePages_Surp 0.00
KReclaimable 371.21
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-08 10:49 [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping Jaroslav Pulchart
@ 2024-01-10 6:45 ` Jaroslav Pulchart
2024-01-10 18:07 ` Jesse Brandeburg
2024-01-19 10:23 ` Linux regression tracking (Thorsten Leemhuis)
2 siblings, 0 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-01-10 6:45 UTC (permalink / raw)
To: Jesse Brandeburg, Tony Nguyen, intel-wired-lan, Dave Ertman
Cc: Igor Raits, Daniel Secik
>
> Hello
>
> I would like to report a regression triggered by recent change in
> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
> bisected and the regression is triggered by
> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
> check for SRIOV and LAG" commit and originally reported as part of
> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
> thread.
>
> > However, after the following patch we see that more NUMA nodes have
> > such a low amount of memory and that is causing constant reclaiming
> > of memory because it looks like something inside of the kernel ate all
> > the memory. This is right after the start of the system as well.
>
> I'm reporting it here as it is a different problem than the original
> thread. The commit introduces a low memory problem per each numa node
> of the first socket (node0 .. node3 in our case) and cause constant
> kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
> memory issue is nicely visible in "numastat -m", see attached files:
> * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
> * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
> the server "is fresh" (after reboot), without running any application load.
>
> $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
> numastat_m-6.6.10_28GB_HP_no_revert.txt
> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
> 2756.89 2754.86 100.39 2278.43 < ice
> fix is reverted, we have ~2GB free per numa, except one, like before
> == no issue
> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
> 3551.29 1530.52 2212.04 3488.09
> ...
> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
> 127.52 66.49 120.23 263.47 <
> ice fix is present, we see just few MB free per each node, this will
> cause kswapd utilization!
> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
> 3322.18 3134.47 195.55 879.17
> ...
>
> If you have some hints on how to debug what is actually occupying all
> that memory and some fix of the problem will be nice. We can provide
> testing and more reports if needed to analyze the issue. We reverted
> the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
> till we know a proper fix.
>
> Best regards,
> Jaroslav Pulchart
Hello everyone
FYI: I try to use linuxk-6.7(.0) kernel and the issue is there as well.
Best regards,
Jaroslav Pulchart
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-08 10:49 [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping Jaroslav Pulchart
2024-01-10 6:45 ` Jaroslav Pulchart
@ 2024-01-10 18:07 ` Jesse Brandeburg
2024-01-11 8:26 ` Jaroslav Pulchart
2024-01-14 12:05 ` Igor Raits
2024-01-19 10:23 ` Linux regression tracking (Thorsten Leemhuis)
2 siblings, 2 replies; 23+ messages in thread
From: Jesse Brandeburg @ 2024-01-10 18:07 UTC (permalink / raw)
To: Jaroslav Pulchart, Tony Nguyen, intel-wired-lan, Dave Ertman
Cc: Igor Raits, Daniel Secik
On 1/8/2024 2:49 AM, Jaroslav Pulchart wrote:
> Hello
First, thank you for your work trying to chase this!
>
> I would like to report a regression triggered by recent change in
> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
> bisected and the regression is triggered by
> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
> check for SRIOV and LAG" commit and originally reported as part of
> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
> thread.
I think that's a bad bisect. There is no reason I could understand for
that change to cause a continuous or large leak, it really doesn't make
any sense. Reverting it consistently helps? You're not just rewinding
the tree back to that point, right? just running 6.6.9 without that
patch? (sorry for being pedantic, just trying to be certain)
>> However, after the following patch we see that more NUMA nodes have
>> such a low amount of memory and that is causing constant reclaiming
>> of memory because it looks like something inside of the kernel ate all
>> the memory. This is right after the start of the system as well.
>
> I'm reporting it here as it is a different problem than the original
> thread. The commit introduces a low memory problem per each numa node
> of the first socket (node0 .. node3 in our case) and cause constant
> kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
> memory issue is nicely visible in "numastat -m", see attached files:
> * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
> * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
> the server "is fresh" (after reboot), without running any application load.
OK, so the initial allocations of your system is running your system out
of memory.
Are you running jumbo frames on your ethernet interfaces?
Do you have /proc/slabinfo output from working/non-working boot?
>
> $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
> numastat_m-6.6.10_28GB_HP_no_revert.txt
> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
> 2756.89 2754.86 100.39 2278.43 < ice
> fix is reverted, we have ~2GB free per numa, except one, like before
> == no issue
> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
> 3551.29 1530.52 2212.04 3488.09
> ...
> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
> 127.52 66.49 120.23 263.47 <
> ice fix is present, we see just few MB free per each node, this will
> cause kswapd utilization!
> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
> 3322.18 3134.47 195.55 879.17
> ...
>
> If you have some hints on how to debug what is actually occupying all
> that memory and some fix of the problem will be nice. We can provide
> testing and more reports if needed to analyze the issue. We reverted
> the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
> till we know a proper fix.
My first suspicion is that we're contributing to the problem by running
out of receive descriptors memory.
Can we see the ethtool -S stats from the freshly booted system that's
running out of memory or doing OOM? Also, all the standard debugging
info (at least once please), devlink dev info, any other configuration
specifics? What networking config (bonding? anything else?)
Do you have a bugzilla.kernel.org bug yet where you can upload larger
files like dmesg and others?
Also, I'm curious if your problem goes away if you change / reduce the
number of queues per port. use ethtool -L eth0 combined 4 ?
You also said something about reproducing when launching / destroying
virtual machines with VF passthrough?
Can you reproduce the issue without starting qemu (just doing bare-metal
SR-IOV instance creation/destruction via
/sys/class/net/eth0/device/sriov_numvfs ?)
Thanks
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-10 18:07 ` Jesse Brandeburg
@ 2024-01-11 8:26 ` Jaroslav Pulchart
2024-01-12 10:23 ` Paul Menzel
2024-01-24 14:29 ` Linux regression tracking (Thorsten Leemhuis)
2024-01-14 12:05 ` Igor Raits
1 sibling, 2 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-01-11 8:26 UTC (permalink / raw)
To: Jesse Brandeburg
Cc: intel-wired-lan, Igor Raits, Daniel Secik, Tony Nguyen,
Dave Ertman
[-- Attachment #1: Type: text/plain, Size: 5641 bytes --]
>
> On 1/8/2024 2:49 AM, Jaroslav Pulchart wrote:
> > Hello
>
> First, thank you for your work trying to chase this!
>
> >
> > I would like to report a regression triggered by recent change in
> > Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
> > bisected and the regression is triggered by
> > fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
> > check for SRIOV and LAG" commit and originally reported as part of
> > https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
> > thread.
>
> I think that's a bad bisect. There is no reason I could understand for
> that change to cause a continuous or large leak, it really doesn't make
> any sense. Reverting it consistently helps? You're not just rewinding
> the tree back to that point, right? just running 6.6.9 without that
> patch? (sorry for being pedantic, just trying to be certain)
>
Reverting just the single bisected commit continuously helps for >=
6.6.9 and as well for current 6.7.
We cannot use any new linux kernel without reverting it due to this
extra memory utilization.
>
> >> However, after the following patch we see that more NUMA nodes have
> >> such a low amount of memory and that is causing constant reclaiming
> >> of memory because it looks like something inside of the kernel ate all
> >> the memory. This is right after the start of the system as well.
> >
> > I'm reporting it here as it is a different problem than the original
> > thread. The commit introduces a low memory problem per each numa node
> > of the first socket (node0 .. node3 in our case) and cause constant
> > kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
> > memory issue is nicely visible in "numastat -m", see attached files:
> > * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
> > * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
> > the server "is fresh" (after reboot), without running any application load.
>
> OK, so the initial allocations of your system is running your system out
> of memory.
>
> Are you running jumbo frames on your ethernet interfaces?
>
Yes, we are (MTU 9000).
> Do you have /proc/slabinfo output from working/non-working boot?
>
Yes, I have a complete sos report so I can pick-up files from there.
See attached
slabinfo.vanila (non-working)
slabinfo.reverted (working)
> >
> > $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
> > numastat_m-6.6.10_28GB_HP_no_revert.txt
> > numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
> > 2756.89 2754.86 100.39 2278.43 < ice
> > fix is reverted, we have ~2GB free per numa, except one, like before
> > == no issue
> > numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
> > 3551.29 1530.52 2212.04 3488.09
> > ...
> > numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
> > 127.52 66.49 120.23 263.47 <
>
>
> > ice fix is present, we see just few MB free per each node, this will
> > cause kswapd utilization!
> > numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
> > 3322.18 3134.47 195.55 879.17
> > ...
> >
> > If you have some hints on how to debug what is actually occupying all
> > that memory and some fix of the problem will be nice. We can provide
> > testing and more reports if needed to analyze the issue. We reverted
> > the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
> > till we know a proper fix.
>
> My first suspicion is that we're contributing to the problem by running
> out of receive descriptors memory.
>
> Can we see the ethtool -S stats from the freshly booted system that's
> running out of memory or doing OOM? Also, all the standard debugging
> info (at least once please), devlink dev info, any other configuration
> specifics? What networking config (bonding? anything else?)
>
The system is not in OOM, it starts to continuously utilize four
kswapd0-4 of each numa node from the first CPU socket processes (each
at 100% and all doing swap in/out) after the system start to be used
by application due to "low memory".
We have two 25G 2P E810-XXV Adapters. The first port of each (em1 +
p3p1) is connected and they're bonded in LACP. Second ports (em2 and
p3p2) are unused.
See attached file for working:
ethtool_-S_em1.reverted
ethtool_-S_em2.reverted
ethtool_-S_p3p1.reverted
ethtool_-S_p3p2.reverted
See attached file for non-working:
ethtool_-S_em1.vanila
ethtool_-S_em2.vanila
ethtool_-S_p3p1.vanila
ethtool_-S_p3p2.vanila
> Do you have a bugzilla.kernel.org bug yet where you can upload larger
> files like dmesg and others?
I do not have yet, I will create a new one and ping you then.
>
> Also, I'm curious if your problem goes away if you change / reduce the
> number of queues per port. use ethtool -L eth0 combined 4 ?
>
I will try and give you feedback soon.
> You also said something about reproducing when launching / destroying
> virtual machines with VF passthrough?
The memory usage is there from boot without running any VMs. The issue
is that the host has low memory for self and it starts to use kswapd
when we start to use it by starting vms.
>
> Can you reproduce the issue without starting qemu (just doing bare-metal
> SR-IOV instance creation/destruction via
> /sys/class/net/eth0/device/sriov_numvfs ?)
>
Yes we can reproduce it without qemu running, the extra memory usage
is from the beginning after boot, not depending on any running VM.
We do not use SR-IOV.
> Thanks
Thanks,
Jaroslav Pulchart
[-- Attachment #2: slabinfo.reverted --]
[-- Type: application/octet-stream, Size: 34756 bytes --]
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
nfs4_xattr_cache_cache 0 0 2128 15 8 : tunables 0 0 0 : slabdata 0 0 0
nfs_direct_cache 0 0 224 36 2 : tunables 0 0 0 : slabdata 0 0 0
nfs_commit_data 46 46 704 46 8 : tunables 0 0 0 : slabdata 1 1 0
nfs_write_data 34 34 960 34 8 : tunables 0 0 0 : slabdata 1 1 0
nfs_read_data 34 34 960 34 8 : tunables 0 0 0 : slabdata 1 1 0
nfs_inode_cache 60 60 1088 30 8 : tunables 0 0 0 : slabdata 2 2 0
nfs_page 32 32 128 32 1 : tunables 0 0 0 : slabdata 1 1 0
fscache_cookie_jar 0 0 176 46 2 : tunables 0 0 0 : slabdata 0 0 0
tcmu_cmd_cache 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0
lio_r2t_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
lio_ooo_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
lio_dr_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
lio_qr_cache 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
t10_alua_lba_map_mem_cache 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0
t10_alua_lba_map_cache 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
t10_alua_tg_pt_gp_cache 58 58 280 58 4 : tunables 0 0 0 : slabdata 1 1 0
t10_alua_lu_gp_mem_cache 85 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0
t10_alua_lu_gp_cache 42 42 192 42 2 : tunables 0 0 0 : slabdata 1 1 0
t10_pr_reg_cache 0 0 704 46 8 : tunables 0 0 0 : slabdata 0 0 0
se_ua_cache 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0
se_sess_cache 0 0 160 51 2 : tunables 0 0 0 : slabdata 0 0 0
sw_flow_stats 1664 1664 32 128 1 : tunables 0 0 0 : slabdata 13 13 0
sw_flow 919 924 1144 28 8 : tunables 0 0 0 : slabdata 33 33 0
nf_conncount_rb 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
nf_conncount_tuple 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
rpc_inode_cache 92 92 704 46 8 : tunables 0 0 0 : slabdata 2 2 0
rpc_buffers 112 112 2048 16 8 : tunables 0 0 0 : slabdata 7 7 0
rpc_tasks 320 320 256 32 2 : tunables 0 0 0 : slabdata 10 10 0
fat_inode_cache 84 84 776 42 8 : tunables 0 0 0 : slabdata 2 2 0
fat_cache 102 102 40 102 1 : tunables 0 0 0 : slabdata 1 1 0
btrfs_prelim_ref 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_extent_op 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_data_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_tree_ref 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_ref_head 0 0 136 60 2 : tunables 0 0 0 : slabdata 0 0 0
btrfs_inode_defrag 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_node 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
btrfs_ordered_extent 0 0 424 38 4 : tunables 0 0 0 : slabdata 0 0 0
btrfs_extent_map 0 0 144 56 2 : tunables 0 0 0 : slabdata 0 0 0
bio-312 51 51 320 51 4 : tunables 0 0 0 : slabdata 1 1 0
bio-376 42 42 384 42 4 : tunables 0 0 0 : slabdata 1 1 0
btrfs_extent_buffer 0 0 240 34 2 : tunables 0 0 0 : slabdata 0 0 0
btrfs_extent_state 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_free_space_bitmap 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
btrfs_free_space 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_path 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_trans_handle 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0
bio-392 36 36 448 36 4 : tunables 0 0 0 : slabdata 1 1 0
btrfs_inode 0 0 1120 29 8 : tunables 0 0 0 : slabdata 0 0 0
bio-448 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0
kvm_async_pf 0 0 136 60 2 : tunables 0 0 0 : slabdata 0 0 0
kvm_vcpu 12 12 7288 4 8 : tunables 0 0 0 : slabdata 3 3 0
kvm_mmu_page_header 0 0 184 44 2 : tunables 0 0 0 : slabdata 0 0 0
pte_list_desc 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
x86_emulator 36 36 2656 12 8 : tunables 0 0 0 : slabdata 3 3 0
bio-176 504 504 192 42 2 : tunables 0 0 0 : slabdata 12 12 0
bio-256 2805 2805 320 51 4 : tunables 0 0 0 : slabdata 55 55 0
zspage 2820 2993 56 73 1 : tunables 0 0 0 : slabdata 41 41 0
zs_handle 22552 25600 8 512 1 : tunables 0 0 0 : slabdata 50 50 0
0000:e2:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:e1:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:c2:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:c1:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:a2:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:a1:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:82:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:81:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:65:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:64:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:43:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:42:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:22:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:21:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:04:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:03:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
fuse_request 0 0 152 53 2 : tunables 0 0 0 : slabdata 0 0 0
fuse_inode 0 0 832 39 8 : tunables 0 0 0 : slabdata 0 0 0
ext4_groupinfo_4k 2640 2640 184 44 2 : tunables 0 0 0 : slabdata 60 60 0
ext4_fc_dentry_update 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
ext4_inode_cache 45528 45528 1168 28 8 : tunables 0 0 0 : slabdata 1626 1626 0
ext4_free_data 3504 3504 56 73 1 : tunables 0 0 0 : slabdata 48 48 0
ext4_allocation_context 3286 3286 152 53 2 : tunables 0 0 0 : slabdata 62 62 0
ext4_prealloc_space 2196 2196 112 36 1 : tunables 0 0 0 : slabdata 61 61 0
ext4_system_zone 306 306 40 102 1 : tunables 0 0 0 : slabdata 3 3 0
ext4_io_end_vec 7424 7424 32 128 1 : tunables 0 0 0 : slabdata 58 58 0
ext4_io_end 4096 4096 64 64 1 : tunables 0 0 0 : slabdata 64 64 0
bio_post_read_ctx 170 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0
pending_reservation 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
extent_status 47404 47430 40 102 1 : tunables 0 0 0 : slabdata 465 465 0
mbcache 730 730 56 73 1 : tunables 0 0 0 : slabdata 10 10 0
jbd2_transaction_s 1248 1248 256 32 2 : tunables 0 0 0 : slabdata 39 39 0
jbd2_inode 14528 14528 64 64 1 : tunables 0 0 0 : slabdata 227 227 0
jbd2_journal_handle 4672 4672 56 73 1 : tunables 0 0 0 : slabdata 64 64 0
jbd2_journal_head 4999 5134 120 34 1 : tunables 0 0 0 : slabdata 151 151 0
jbd2_revoke_table_s 512 512 16 256 1 : tunables 0 0 0 : slabdata 2 2 0
jbd2_revoke_record_s 2944 2944 32 128 1 : tunables 0 0 0 : slabdata 23 23 0
bio-992 32 32 1024 32 8 : tunables 0 0 0 : slabdata 1 1 0
bio-107 3960 4770 1088 30 8 : tunables 0 0 0 : slabdata 159 159 0
bio-136 1344 1344 192 42 2 : tunables 0 0 0 : slabdata 32 32 0
scsi_sense_cache 5976 6304 128 32 1 : tunables 0 0 0 : slabdata 197 197 0
kcopyd_job 0 0 3240 10 8 : tunables 0 0 0 : slabdata 0 0 0
io 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dm_uevent 0 0 2888 11 8 : tunables 0 0 0 : slabdata 0 0 0
nf_conntrack_expect 0 0 208 39 2 : tunables 0 0 0 : slabdata 0 0 0
nf_conntrack 2032 2272 256 32 2 : tunables 0 0 0 : slabdata 71 71 0
nf-frags 0 0 184 44 2 : tunables 0 0 0 : slabdata 0 0 0
bridge_fdb_cache 192 192 128 32 1 : tunables 0 0 0 : slabdata 6 6 0
MPTCPv6 0 0 2112 15 8 : tunables 0 0 0 : slabdata 0 0 0
ip6-frags 0 0 184 44 2 : tunables 0 0 0 : slabdata 0 0 0
fib6_nodes 320 320 64 64 1 : tunables 0 0 0 : slabdata 5 5 0
ip6_dst_cache 640 640 256 32 2 : tunables 0 0 0 : slabdata 20 20 0
ip6_mrt_cache 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
PINGv6 0 0 1216 26 8 : tunables 0 0 0 : slabdata 0 0 0
RAWv6 156 156 1216 26 8 : tunables 0 0 0 : slabdata 6 6 0
UDPLITEv6 0 0 1344 24 8 : tunables 0 0 0 : slabdata 0 0 0
UDPv6 648 648 1344 24 8 : tunables 0 0 0 : slabdata 27 27 0
tw_sock_TCPv6 0 0 272 60 4 : tunables 0 0 0 : slabdata 0 0 0
request_sock_TCPv6 0 0 312 52 4 : tunables 0 0 0 : slabdata 0 0 0
TCPv6 533 533 2432 13 8 : tunables 0 0 0 : slabdata 41 41 0
uhci_urb_priv 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0
btree_node 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
io_kiocb 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
bfq_io_cq 0 0 1360 24 8 : tunables 0 0 0 : slabdata 0 0 0
bfq_queue 0 0 576 56 8 : tunables 0 0 0 : slabdata 0 0 0
mqueue_inode_cache 34 34 960 34 8 : tunables 0 0 0 : slabdata 1 1 0
kioctx 1568 1568 576 56 8 : tunables 0 0 0 : slabdata 28 28 0
aio_kiocb 1176 1176 192 42 2 : tunables 0 0 0 : slabdata 28 28 0
userfaultfd_ctx_cache 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
fanotify_perm_event 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
fanotify_path_event 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
fsnotify_mark 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0
dnotify_struct 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
dio 0 0 640 51 8 : tunables 0 0 0 : slabdata 0 0 0
fasync_cache 85 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0
audit_tree_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0
pid_namespace 0 0 144 56 2 : tunables 0 0 0 : slabdata 0 0 0
posix_timers_cache 2048 2048 256 32 2 : tunables 0 0 0 : slabdata 64 64 0
UNIX-STREAM 1980 1980 1088 30 8 : tunables 0 0 0 : slabdata 66 66 0
UNIX 1890 1890 1088 30 8 : tunables 0 0 0 : slabdata 63 63 0
ip4-frags 0 0 200 40 2 : tunables 0 0 0 : slabdata 0 0 0
ip_mrt_cache 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
UDP-Lite 0 0 1152 28 8 : tunables 0 0 0 : slabdata 0 0 0
MPTCP 0 0 1984 16 8 : tunables 0 0 0 : slabdata 0 0 0
request_sock_subflow_v6 0 0 384 42 4 : tunables 0 0 0 : slabdata 0 0 0
request_sock_subflow_v4 0 0 384 42 4 : tunables 0 0 0 : slabdata 0 0 0
tcp_bind2_bucket 3520 3520 64 64 1 : tunables 0 0 0 : slabdata 55 55 0
tcp_bind_bucket 1728 1728 128 32 1 : tunables 0 0 0 : slabdata 54 54 0
inet_peer_cache 42 42 192 42 2 : tunables 0 0 0 : slabdata 1 1 0
xfrm_dst_cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
xfrm_state 0 0 768 42 8 : tunables 0 0 0 : slabdata 0 0 0
ip_fib_trie 510 510 48 85 1 : tunables 0 0 0 : slabdata 6 6 0
ip_fib_alias 438 438 56 73 1 : tunables 0 0 0 : slabdata 6 6 0
ip_dst_cache 2478 2478 192 42 2 : tunables 0 0 0 : slabdata 59 59 0
PING 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
RAW 128 128 1024 32 8 : tunables 0 0 0 : slabdata 4 4 0
UDP 1736 1736 1152 28 8 : tunables 0 0 0 : slabdata 62 62 0
tw_sock_TCP 1380 1380 272 60 4 : tunables 0 0 0 : slabdata 23 23 0
request_sock_TCP 1560 1560 312 52 4 : tunables 0 0 0 : slabdata 30 30 0
TCP 742 742 2304 14 8 : tunables 0 0 0 : slabdata 53 53 0
hugetlbfs_inode_cache 300 300 648 50 8 : tunables 0 0 0 : slabdata 6 6 0
dquot 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
bio-264 1530 1530 320 51 4 : tunables 0 0 0 : slabdata 30 30 0
ep_head 16384 16384 16 256 1 : tunables 0 0 0 : slabdata 64 64 0
eventpoll_pwq 4352 4352 64 64 1 : tunables 0 0 0 : slabdata 68 68 0
eventpoll_epi 2944 2944 128 32 1 : tunables 0 0 0 : slabdata 92 92 0
inotify_inode_mark 2499 2499 80 51 1 : tunables 0 0 0 : slabdata 49 49 0
dax_cache 294 294 768 42 8 : tunables 0 0 0 : slabdata 7 7 0
sgpool-128 448 448 4096 8 8 : tunables 0 0 0 : slabdata 56 56 0
sgpool-64 896 896 2048 16 8 : tunables 0 0 0 : slabdata 56 56 0
sgpool-32 2016 2016 1024 32 8 : tunables 0 0 0 : slabdata 63 63 0
sgpool-16 1984 1984 512 32 4 : tunables 0 0 0 : slabdata 62 62 0
sgpool-8 2048 2048 256 32 2 : tunables 0 0 0 : slabdata 64 64 0
bio_crypt_ctx 204 204 40 102 1 : tunables 0 0 0 : slabdata 2 2 0
request_queue 476 476 960 34 8 : tunables 0 0 0 : slabdata 14 14 0
blkdev_ioc 92 92 88 46 1 : tunables 0 0 0 : slabdata 2 2 0
bio-200 6722 7616 256 32 2 : tunables 0 0 0 : slabdata 238 238 0
biovec-max 2736 2808 4096 8 8 : tunables 0 0 0 : slabdata 351 351 0
biovec-128 800 800 2048 16 8 : tunables 0 0 0 : slabdata 50 50 0
biovec-64 1952 1952 1024 32 8 : tunables 0 0 0 : slabdata 61 61 0
biovec-16 2016 2016 256 32 2 : tunables 0 0 0 : slabdata 63 63 0
bio_integrity_payload 42 42 192 42 2 : tunables 0 0 0 : slabdata 1 1 0
khugepaged_mm_slot 6528 6528 40 102 1 : tunables 0 0 0 : slabdata 64 64 0
ksm_mm_slot 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
ksm_stable_node 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
ksm_rmap_item 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
user_namespace 0 0 624 52 8 : tunables 0 0 0 : slabdata 0 0 0
uid_cache 966 966 192 42 2 : tunables 0 0 0 : slabdata 23 23 0
iommu_iova 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dmaengine-unmap-256 15 15 2112 15 8 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-128 30 30 1088 30 8 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-16 42 42 192 42 2 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-2 64 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0
audit_buffer 9180 9180 24 170 1 : tunables 0 0 0 : slabdata 54 54 0
sock_inode_cache 4134 4134 832 39 8 : tunables 0 0 0 : slabdata 106 106 0
skbuff_ext_cache 798 798 192 42 2 : tunables 0 0 0 : slabdata 19 19 0
skbuff_small_head 4592 5208 576 56 8 : tunables 0 0 0 : slabdata 93 93 0
skbuff_fclone_cache 1984 1984 512 32 4 : tunables 0 0 0 : slabdata 62 62 0
skbuff_head_cache 3662 4000 256 32 2 : tunables 0 0 0 : slabdata 125 125 0
tracefs_inode_cache 17799 17799 640 51 8 : tunables 0 0 0 : slabdata 349 349 0
configfs_dir_cache 4646 4646 88 46 1 : tunables 0 0 0 : slabdata 101 101 0
file_lock_cache 2368 2368 216 37 2 : tunables 0 0 0 : slabdata 64 64 0
file_lock_ctx 4088 4088 56 73 1 : tunables 0 0 0 : slabdata 56 56 0
fsnotify_mark_connector 5760 5760 32 128 1 : tunables 0 0 0 : slabdata 45 45 0
buffer_head 26364 26598 104 39 1 : tunables 0 0 0 : slabdata 682 682 0
task_delay_info 0 0 144 56 2 : tunables 0 0 0 : slabdata 0 0 0
taskstats 1554 1554 432 37 4 : tunables 0 0 0 : slabdata 42 42 0
proc_dir_entry 7464 7476 192 42 2 : tunables 0 0 0 : slabdata 178 178 0
pde_opener 6222 6222 40 102 1 : tunables 0 0 0 : slabdata 61 61 0
proc_inode_cache 68000 68338 696 47 8 : tunables 0 0 0 : slabdata 1454 1454 0
seq_file 2176 2176 120 34 1 : tunables 0 0 0 : slabdata 64 64 0
sigqueue 3366 3366 80 51 1 : tunables 0 0 0 : slabdata 66 66 0
bdev_cache 200 200 1600 20 8 : tunables 0 0 0 : slabdata 10 10 0
shmem_inode_cache 5562 5617 784 41 8 : tunables 0 0 0 : slabdata 137 137 0
kernfs_iattrs_cache 3111 3111 80 51 1 : tunables 0 0 0 : slabdata 61 61 0
kernfs_node_cache 92304 92576 128 32 1 : tunables 0 0 0 : slabdata 2893 2893 0
mnt_cache 3213 3213 320 51 4 : tunables 0 0 0 : slabdata 63 63 0
filp 9878 10880 256 32 2 : tunables 0 0 0 : slabdata 340 340 0
inode_cache 75244 75244 624 52 8 : tunables 0 0 0 : slabdata 1447 1447 0
dentry 225845 226086 192 42 2 : tunables 0 0 0 : slabdata 5383 5383 0
names_cache 512 512 4096 8 8 : tunables 0 0 0 : slabdata 64 64 0
net_namespace 21 21 4160 7 8 : tunables 0 0 0 : slabdata 3 3 0
iint_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
hashtab_node 17680 17680 24 170 1 : tunables 0 0 0 : slabdata 104 104 0
ebitmap_node 48128 48128 64 64 1 : tunables 0 0 0 : slabdata 752 752 0
avtab_extended_perms 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0
avtab_node 90440 90440 24 170 1 : tunables 0 0 0 : slabdata 532 532 0
avc_xperms_data 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
avc_xperms_decision_node 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
avc_xperms_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0
avc_node 7000 7000 72 56 1 : tunables 0 0 0 : slabdata 125 125 0
lsm_inode_cache 206448 206517 56 73 1 : tunables 0 0 0 : slabdata 2829 2829 0
lsm_file_cache 19170 19200 16 256 1 : tunables 0 0 0 : slabdata 75 75 0
key_jar 1728 1728 256 32 2 : tunables 0 0 0 : slabdata 54 54 0
uts_namespace 185 185 432 37 4 : tunables 0 0 0 : slabdata 5 5 0
nsproxy 1400 1400 72 56 1 : tunables 0 0 0 : slabdata 25 25 0
vma_lock 27318 29172 40 102 1 : tunables 0 0 0 : slabdata 286 286 0
vm_area_struct 23161 23980 184 44 2 : tunables 0 0 0 : slabdata 545 545 0
fs_cache 4224 4224 64 64 1 : tunables 0 0 0 : slabdata 66 66 0
files_cache 3128 3128 704 46 8 : tunables 0 0 0 : slabdata 68 68 0
signal_cache 2772 2828 1152 28 8 : tunables 0 0 0 : slabdata 101 101 0
sighand_cache 1980 2055 2112 15 8 : tunables 0 0 0 : slabdata 137 137 0
task_struct 1906 2000 6656 4 8 : tunables 0 0 0 : slabdata 500 500 0
cred_jar 5124 5124 192 42 2 : tunables 0 0 0 : slabdata 122 122 0
anon_vma_chain 19129 20096 64 64 1 : tunables 0 0 0 : slabdata 314 314 0
anon_vma 14277 14976 104 39 1 : tunables 0 0 0 : slabdata 384 384 0
pid 3296 3296 128 32 1 : tunables 0 0 0 : slabdata 103 103 0
irq_remap_cache 72 72 8192 4 8 : tunables 0 0 0 : slabdata 18 18 0
Acpi-Operand 19712 19712 72 56 1 : tunables 0 0 0 : slabdata 352 352 0
Acpi-ParseExt 1683 1683 80 51 1 : tunables 0 0 0 : slabdata 33 33 0
Acpi-Parse 2409 2409 56 73 1 : tunables 0 0 0 : slabdata 33 33 0
Acpi-State 1785 1785 80 51 1 : tunables 0 0 0 : slabdata 35 35 0
Acpi-Namespace 6205 6205 48 85 1 : tunables 0 0 0 : slabdata 73 73 0
shared_policy_node 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
numa_policy 900 900 272 60 4 : tunables 0 0 0 : slabdata 15 15 0
perf_event 3539 13225 1264 25 8 : tunables 0 0 0 : slabdata 529 529 0
trace_event_file 2814 2814 96 42 1 : tunables 0 0 0 : slabdata 67 67 0
ftrace_event_field 7008 7008 56 73 1 : tunables 0 0 0 : slabdata 96 96 0
pool_workqueue 6358 10464 512 32 4 : tunables 0 0 0 : slabdata 327 327 0
maple_node 7719 8608 256 32 2 : tunables 0 0 0 : slabdata 269 269 0
radix_tree_node 17286 17360 584 56 8 : tunables 0 0 0 : slabdata 310 310 0
task_group 3162 3162 640 51 8 : tunables 0 0 0 : slabdata 62 62 0
mm_struct 1679 1679 1408 23 8 : tunables 0 0 0 : slabdata 73 73 0
vmap_area 8719 9128 72 56 1 : tunables 0 0 0 : slabdata 163 163 0
kmalloc-cg-8k 64 64 8192 4 8 : tunables 0 0 0 : slabdata 16 16 0
kmalloc-cg-4k 640 640 4096 8 8 : tunables 0 0 0 : slabdata 80 80 0
kmalloc-cg-2k 1472 1472 2048 16 8 : tunables 0 0 0 : slabdata 92 92 0
kmalloc-cg-1k 2144 2144 1024 32 8 : tunables 0 0 0 : slabdata 67 67 0
kmalloc-cg-512 2112 2112 512 32 4 : tunables 0 0 0 : slabdata 66 66 0
kmalloc-cg-256 832 832 256 32 2 : tunables 0 0 0 : slabdata 26 26 0
kmalloc-cg-192 2772 2772 192 42 2 : tunables 0 0 0 : slabdata 66 66 0
kmalloc-cg-128 1408 1408 128 32 1 : tunables 0 0 0 : slabdata 44 44 0
kmalloc-cg-96 7692 7770 96 42 1 : tunables 0 0 0 : slabdata 185 185 0
kmalloc-cg-64 4224 4224 64 64 1 : tunables 0 0 0 : slabdata 66 66 0
kmalloc-cg-32 13653 13696 32 128 1 : tunables 0 0 0 : slabdata 107 107 0
kmalloc-cg-16 15104 15104 16 256 1 : tunables 0 0 0 : slabdata 59 59 0
kmalloc-cg-8 26112 26112 8 512 1 : tunables 0 0 0 : slabdata 51 51 0
dma-kmalloc-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-2k 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-1k 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-2k 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-1k 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-512 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-128 1024 1024 128 32 1 : tunables 0 0 0 : slabdata 32 32 0
kmalloc-rcl-96 5796 5796 96 42 1 : tunables 0 0 0 : slabdata 138 138 0
kmalloc-rcl-64 7232 7232 64 64 1 : tunables 0 0 0 : slabdata 113 113 0
kmalloc-rcl-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-8k 481 500 8192 4 8 : tunables 0 0 0 : slabdata 125 125 0
kmalloc-4k 2963 3040 4096 8 8 : tunables 0 0 0 : slabdata 380 380 0
kmalloc-2k 3742 3824 2048 16 8 : tunables 0 0 0 : slabdata 239 239 0
kmalloc-1k 7640 7968 1024 32 8 : tunables 0 0 0 : slabdata 249 249 0
kmalloc-512 35088 80064 512 32 4 : tunables 0 0 0 : slabdata 2502 2502 0
kmalloc-256 11315 11552 256 32 2 : tunables 0 0 0 : slabdata 361 361 0
kmalloc-192 15160 19950 192 42 2 : tunables 0 0 0 : slabdata 475 475 0
kmalloc-128 23291 23360 128 32 1 : tunables 0 0 0 : slabdata 730 730 0
kmalloc-96 24608 25662 96 42 1 : tunables 0 0 0 : slabdata 611 611 0
kmalloc-64 35100 35584 64 64 1 : tunables 0 0 0 : slabdata 556 556 0
kmalloc-32 37137 38272 32 128 1 : tunables 0 0 0 : slabdata 299 299 0
kmalloc-16 52212 52480 16 256 1 : tunables 0 0 0 : slabdata 205 205 0
kmalloc-8 65280 67072 8 512 1 : tunables 0 0 0 : slabdata 131 131 0
kmem_cache_node 4409 4416 64 64 1 : tunables 0 0 0 : slabdata 69 69 0
kmem_cache 1785 1785 320 51 4 : tunables 0 0 0 : slabdata 35 35 0
[-- Attachment #3: slabinfo.vanila --]
[-- Type: application/octet-stream, Size: 34756 bytes --]
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
nfs4_xattr_cache_cache 0 0 2128 15 8 : tunables 0 0 0 : slabdata 0 0 0
nfs_direct_cache 0 0 224 36 2 : tunables 0 0 0 : slabdata 0 0 0
nfs_commit_data 46 46 704 46 8 : tunables 0 0 0 : slabdata 1 1 0
nfs_write_data 34 34 960 34 8 : tunables 0 0 0 : slabdata 1 1 0
nfs_read_data 34 34 960 34 8 : tunables 0 0 0 : slabdata 1 1 0
nfs_inode_cache 60 60 1088 30 8 : tunables 0 0 0 : slabdata 2 2 0
nfs_page 32 32 128 32 1 : tunables 0 0 0 : slabdata 1 1 0
fscache_cookie_jar 0 0 176 46 2 : tunables 0 0 0 : slabdata 0 0 0
tcmu_cmd_cache 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0
lio_r2t_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
lio_ooo_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
lio_dr_cache 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
lio_qr_cache 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
t10_alua_lba_map_mem_cache 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0
t10_alua_lba_map_cache 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
t10_alua_tg_pt_gp_cache 58 58 280 58 4 : tunables 0 0 0 : slabdata 1 1 0
t10_alua_lu_gp_mem_cache 85 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0
t10_alua_lu_gp_cache 42 42 192 42 2 : tunables 0 0 0 : slabdata 1 1 0
t10_pr_reg_cache 0 0 704 46 8 : tunables 0 0 0 : slabdata 0 0 0
se_ua_cache 0 0 24 170 1 : tunables 0 0 0 : slabdata 0 0 0
se_sess_cache 0 0 160 51 2 : tunables 0 0 0 : slabdata 0 0 0
sw_flow_stats 2589 24704 32 128 1 : tunables 0 0 0 : slabdata 193 193 0
sw_flow 1971 1988 1144 28 8 : tunables 0 0 0 : slabdata 71 71 0
nf_conncount_rb 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
nf_conncount_tuple 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
rpc_inode_cache 92 92 704 46 8 : tunables 0 0 0 : slabdata 2 2 0
rpc_buffers 128 128 2048 16 8 : tunables 0 0 0 : slabdata 8 8 0
rpc_tasks 384 384 256 32 2 : tunables 0 0 0 : slabdata 12 12 0
fat_inode_cache 168 168 776 42 8 : tunables 0 0 0 : slabdata 4 4 0
fat_cache 204 204 40 102 1 : tunables 0 0 0 : slabdata 2 2 0
btrfs_prelim_ref 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_extent_op 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_data_ref 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_tree_ref 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_ref_head 0 0 136 60 2 : tunables 0 0 0 : slabdata 0 0 0
btrfs_inode_defrag 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_delayed_node 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
btrfs_ordered_extent 0 0 424 38 4 : tunables 0 0 0 : slabdata 0 0 0
btrfs_extent_map 0 0 144 56 2 : tunables 0 0 0 : slabdata 0 0 0
bio-312 51 51 320 51 4 : tunables 0 0 0 : slabdata 1 1 0
bio-376 42 42 384 42 4 : tunables 0 0 0 : slabdata 1 1 0
btrfs_extent_buffer 0 0 240 34 2 : tunables 0 0 0 : slabdata 0 0 0
btrfs_extent_state 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_free_space_bitmap 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
btrfs_free_space 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_path 0 0 112 36 1 : tunables 0 0 0 : slabdata 0 0 0
btrfs_trans_handle 0 0 104 39 1 : tunables 0 0 0 : slabdata 0 0 0
bio-392 36 36 448 36 4 : tunables 0 0 0 : slabdata 1 1 0
btrfs_inode 0 0 1120 29 8 : tunables 0 0 0 : slabdata 0 0 0
bio-448 32 32 512 32 4 : tunables 0 0 0 : slabdata 1 1 0
bio-176 504 504 192 42 2 : tunables 0 0 0 : slabdata 12 12 0
bio-256 3009 3009 320 51 4 : tunables 0 0 0 : slabdata 59 59 0
kvm_async_pf 0 0 136 60 2 : tunables 0 0 0 : slabdata 0 0 0
kvm_vcpu 12 12 7288 4 8 : tunables 0 0 0 : slabdata 3 3 0
kvm_mmu_page_header 0 0 184 44 2 : tunables 0 0 0 : slabdata 0 0 0
pte_list_desc 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
x86_emulator 36 36 2656 12 8 : tunables 0 0 0 : slabdata 3 3 0
0000:e2:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:e1:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:c2:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:c1:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:a2:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:a1:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:82:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:81:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:65:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:64:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:43:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:42:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:22:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
zspage 2844 2993 56 73 1 : tunables 0 0 0 : slabdata 41 41 0
zs_handle 22530 26112 8 512 1 : tunables 0 0 0 : slabdata 51 51 0
0000:21:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:04:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
0000:03:00.2-dmaengine-desc-cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
fuse_request 0 0 152 53 2 : tunables 0 0 0 : slabdata 0 0 0
fuse_inode 0 0 832 39 8 : tunables 0 0 0 : slabdata 0 0 0
ext4_groupinfo_4k 2640 2640 184 44 2 : tunables 0 0 0 : slabdata 60 60 0
ext4_fc_dentry_update 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
ext4_inode_cache 40861 40936 1168 28 8 : tunables 0 0 0 : slabdata 1462 1462 0
ext4_free_data 3723 3723 56 73 1 : tunables 0 0 0 : slabdata 51 51 0
ext4_allocation_context 3286 3286 152 53 2 : tunables 0 0 0 : slabdata 62 62 0
ext4_prealloc_space 2160 2160 112 36 1 : tunables 0 0 0 : slabdata 60 60 0
ext4_system_zone 306 306 40 102 1 : tunables 0 0 0 : slabdata 3 3 0
ext4_io_end_vec 7296 7296 32 128 1 : tunables 0 0 0 : slabdata 57 57 0
ext4_io_end 4416 4672 64 64 1 : tunables 0 0 0 : slabdata 73 73 0
bio_post_read_ctx 170 170 48 85 1 : tunables 0 0 0 : slabdata 2 2 0
pending_reservation 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
extent_status 28952 29478 40 102 1 : tunables 0 0 0 : slabdata 289 289 0
mbcache 657 657 56 73 1 : tunables 0 0 0 : slabdata 9 9 0
jbd2_transaction_s 1216 1216 256 32 2 : tunables 0 0 0 : slabdata 38 38 0
jbd2_inode 13034 13056 64 64 1 : tunables 0 0 0 : slabdata 204 204 0
jbd2_journal_handle 4818 4818 56 73 1 : tunables 0 0 0 : slabdata 66 66 0
jbd2_journal_head 4828 4828 120 34 1 : tunables 0 0 0 : slabdata 142 142 0
jbd2_revoke_table_s 512 512 16 256 1 : tunables 0 0 0 : slabdata 2 2 0
jbd2_revoke_record_s 2944 2944 32 128 1 : tunables 0 0 0 : slabdata 23 23 0
bio-992 32 32 1024 32 8 : tunables 0 0 0 : slabdata 1 1 0
bio-107 3930 4620 1088 30 8 : tunables 0 0 0 : slabdata 154 154 0
bio-136 1050 1050 192 42 2 : tunables 0 0 0 : slabdata 25 25 0
scsi_sense_cache 5921 6272 128 32 1 : tunables 0 0 0 : slabdata 196 196 0
kcopyd_job 0 0 3240 10 8 : tunables 0 0 0 : slabdata 0 0 0
io 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dm_uevent 0 0 2888 11 8 : tunables 0 0 0 : slabdata 0 0 0
nf_conntrack_expect 0 0 208 39 2 : tunables 0 0 0 : slabdata 0 0 0
nf_conntrack 2335 2688 256 32 2 : tunables 0 0 0 : slabdata 84 84 0
nf-frags 0 0 184 44 2 : tunables 0 0 0 : slabdata 0 0 0
bridge_fdb_cache 224 224 128 32 1 : tunables 0 0 0 : slabdata 7 7 0
MPTCPv6 0 0 2112 15 8 : tunables 0 0 0 : slabdata 0 0 0
ip6-frags 0 0 184 44 2 : tunables 0 0 0 : slabdata 0 0 0
fib6_nodes 256 256 64 64 1 : tunables 0 0 0 : slabdata 4 4 0
ip6_dst_cache 608 608 256 32 2 : tunables 0 0 0 : slabdata 19 19 0
ip6_mrt_cache 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
PINGv6 0 0 1216 26 8 : tunables 0 0 0 : slabdata 0 0 0
RAWv6 156 156 1216 26 8 : tunables 0 0 0 : slabdata 6 6 0
UDPLITEv6 0 0 1344 24 8 : tunables 0 0 0 : slabdata 0 0 0
UDPv6 672 672 1344 24 8 : tunables 0 0 0 : slabdata 28 28 0
tw_sock_TCPv6 0 0 272 60 4 : tunables 0 0 0 : slabdata 0 0 0
request_sock_TCPv6 0 0 312 52 4 : tunables 0 0 0 : slabdata 0 0 0
TCPv6 546 546 2432 13 8 : tunables 0 0 0 : slabdata 42 42 0
uhci_urb_priv 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0
btree_node 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
io_kiocb 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
bfq_io_cq 0 0 1360 24 8 : tunables 0 0 0 : slabdata 0 0 0
bfq_queue 0 0 576 56 8 : tunables 0 0 0 : slabdata 0 0 0
mqueue_inode_cache 34 34 960 34 8 : tunables 0 0 0 : slabdata 1 1 0
kioctx 1288 1288 576 56 8 : tunables 0 0 0 : slabdata 23 23 0
aio_kiocb 1008 1008 192 42 2 : tunables 0 0 0 : slabdata 24 24 0
userfaultfd_ctx_cache 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
fanotify_perm_event 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
fanotify_path_event 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
fanotify_fid_event 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
fsnotify_mark 0 0 72 56 1 : tunables 0 0 0 : slabdata 0 0 0
dnotify_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0
dnotify_struct 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
dio 0 0 640 51 8 : tunables 0 0 0 : slabdata 0 0 0
fasync_cache 85 85 48 85 1 : tunables 0 0 0 : slabdata 1 1 0
audit_tree_mark 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0
pid_namespace 0 0 144 56 2 : tunables 0 0 0 : slabdata 0 0 0
posix_timers_cache 2016 2016 256 32 2 : tunables 0 0 0 : slabdata 63 63 0
UNIX-STREAM 1980 1980 1088 30 8 : tunables 0 0 0 : slabdata 66 66 0
UNIX 1830 1830 1088 30 8 : tunables 0 0 0 : slabdata 61 61 0
ip4-frags 0 0 200 40 2 : tunables 0 0 0 : slabdata 0 0 0
ip_mrt_cache 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
UDP-Lite 0 0 1152 28 8 : tunables 0 0 0 : slabdata 0 0 0
MPTCP 0 0 1984 16 8 : tunables 0 0 0 : slabdata 0 0 0
request_sock_subflow_v6 0 0 384 42 4 : tunables 0 0 0 : slabdata 0 0 0
request_sock_subflow_v4 0 0 384 42 4 : tunables 0 0 0 : slabdata 0 0 0
tcp_bind2_bucket 3520 3520 64 64 1 : tunables 0 0 0 : slabdata 55 55 0
tcp_bind_bucket 1760 1760 128 32 1 : tunables 0 0 0 : slabdata 55 55 0
inet_peer_cache 84 84 192 42 2 : tunables 0 0 0 : slabdata 2 2 0
xfrm_dst_cache 0 0 320 51 4 : tunables 0 0 0 : slabdata 0 0 0
xfrm_state 0 0 768 42 8 : tunables 0 0 0 : slabdata 0 0 0
ip_fib_trie 425 425 48 85 1 : tunables 0 0 0 : slabdata 5 5 0
ip_fib_alias 365 365 56 73 1 : tunables 0 0 0 : slabdata 5 5 0
ip_dst_cache 2646 2646 192 42 2 : tunables 0 0 0 : slabdata 63 63 0
PING 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
RAW 128 128 1024 32 8 : tunables 0 0 0 : slabdata 4 4 0
UDP 1764 1764 1152 28 8 : tunables 0 0 0 : slabdata 63 63 0
tw_sock_TCP 1380 1380 272 60 4 : tunables 0 0 0 : slabdata 23 23 0
request_sock_TCP 1820 1820 312 52 4 : tunables 0 0 0 : slabdata 35 35 0
TCP 770 770 2304 14 8 : tunables 0 0 0 : slabdata 55 55 0
hugetlbfs_inode_cache 300 300 648 50 8 : tunables 0 0 0 : slabdata 6 6 0
dquot 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
bio-264 1326 1326 320 51 4 : tunables 0 0 0 : slabdata 26 26 0
ep_head 16384 16384 16 256 1 : tunables 0 0 0 : slabdata 64 64 0
eventpoll_pwq 4352 4352 64 64 1 : tunables 0 0 0 : slabdata 68 68 0
eventpoll_epi 2816 2816 128 32 1 : tunables 0 0 0 : slabdata 88 88 0
inotify_inode_mark 3009 3009 80 51 1 : tunables 0 0 0 : slabdata 59 59 0
dax_cache 252 252 768 42 8 : tunables 0 0 0 : slabdata 6 6 0
sgpool-128 432 432 4096 8 8 : tunables 0 0 0 : slabdata 54 54 0
sgpool-64 896 896 2048 16 8 : tunables 0 0 0 : slabdata 56 56 0
sgpool-32 1984 1984 1024 32 8 : tunables 0 0 0 : slabdata 62 62 0
sgpool-16 2016 2016 512 32 4 : tunables 0 0 0 : slabdata 63 63 0
sgpool-8 2048 2048 256 32 2 : tunables 0 0 0 : slabdata 64 64 0
bio_crypt_ctx 204 204 40 102 1 : tunables 0 0 0 : slabdata 2 2 0
request_queue 408 408 960 34 8 : tunables 0 0 0 : slabdata 12 12 0
blkdev_ioc 92 92 88 46 1 : tunables 0 0 0 : slabdata 2 2 0
bio-200 7298 7776 256 32 2 : tunables 0 0 0 : slabdata 243 243 0
biovec-max 2792 2880 4096 8 8 : tunables 0 0 0 : slabdata 360 360 0
biovec-128 896 896 2048 16 8 : tunables 0 0 0 : slabdata 56 56 0
biovec-64 2080 2080 1024 32 8 : tunables 0 0 0 : slabdata 65 65 0
biovec-16 2016 2016 256 32 2 : tunables 0 0 0 : slabdata 63 63 0
bio_integrity_payload 42 42 192 42 2 : tunables 0 0 0 : slabdata 1 1 0
khugepaged_mm_slot 6528 6528 40 102 1 : tunables 0 0 0 : slabdata 64 64 0
ksm_mm_slot 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
ksm_stable_node 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
ksm_rmap_item 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
user_namespace 0 0 624 52 8 : tunables 0 0 0 : slabdata 0 0 0
uid_cache 756 756 192 42 2 : tunables 0 0 0 : slabdata 18 18 0
iommu_iova 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dmaengine-unmap-256 15 15 2112 15 8 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-128 30 30 1088 30 8 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-16 42 42 192 42 2 : tunables 0 0 0 : slabdata 1 1 0
dmaengine-unmap-2 64 64 64 64 1 : tunables 0 0 0 : slabdata 1 1 0
audit_buffer 9690 9690 24 170 1 : tunables 0 0 0 : slabdata 57 57 0
sock_inode_cache 4329 4329 832 39 8 : tunables 0 0 0 : slabdata 111 111 0
skbuff_ext_cache 798 798 192 42 2 : tunables 0 0 0 : slabdata 19 19 0
skbuff_small_head 4200 4760 576 56 8 : tunables 0 0 0 : slabdata 85 85 0
skbuff_fclone_cache 2016 2016 512 32 4 : tunables 0 0 0 : slabdata 63 63 0
skbuff_head_cache 3744 3872 256 32 2 : tunables 0 0 0 : slabdata 121 121 0
tracefs_inode_cache 1173 1377 640 51 8 : tunables 0 0 0 : slabdata 27 27 0
configfs_dir_cache 4646 4646 88 46 1 : tunables 0 0 0 : slabdata 101 101 0
file_lock_cache 2368 2368 216 37 2 : tunables 0 0 0 : slabdata 64 64 0
file_lock_ctx 4015 4015 56 73 1 : tunables 0 0 0 : slabdata 55 55 0
fsnotify_mark_connector 7296 7296 32 128 1 : tunables 0 0 0 : slabdata 57 57 0
buffer_head 23822 24024 104 39 1 : tunables 0 0 0 : slabdata 616 616 0
task_delay_info 0 0 144 56 2 : tunables 0 0 0 : slabdata 0 0 0
taskstats 1924 1924 432 37 4 : tunables 0 0 0 : slabdata 52 52 0
proc_dir_entry 7919 7938 192 42 2 : tunables 0 0 0 : slabdata 189 189 0
pde_opener 6528 6528 40 102 1 : tunables 0 0 0 : slabdata 64 64 0
proc_inode_cache 59072 66505 696 47 8 : tunables 0 0 0 : slabdata 1415 1415 0
seq_file 2176 2176 120 34 1 : tunables 0 0 0 : slabdata 64 64 0
sigqueue 3366 3366 80 51 1 : tunables 0 0 0 : slabdata 66 66 0
bdev_cache 180 180 1600 20 8 : tunables 0 0 0 : slabdata 9 9 0
shmem_inode_cache 5410 5494 784 41 8 : tunables 0 0 0 : slabdata 134 134 0
kernfs_iattrs_cache 3928 3978 80 51 1 : tunables 0 0 0 : slabdata 78 78 0
kernfs_node_cache 93015 93088 128 32 1 : tunables 0 0 0 : slabdata 2909 2909 0
mnt_cache 3009 3009 320 51 4 : tunables 0 0 0 : slabdata 59 59 0
filp 10502 11296 256 32 2 : tunables 0 0 0 : slabdata 353 353 0
inode_cache 76260 76492 624 52 8 : tunables 0 0 0 : slabdata 1471 1471 0
dentry 182338 188958 192 42 2 : tunables 0 0 0 : slabdata 4499 4499 0
names_cache 512 512 4096 8 8 : tunables 0 0 0 : slabdata 64 64 0
net_namespace 14 14 4160 7 8 : tunables 0 0 0 : slabdata 2 2 0
iint_cache 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
hashtab_node 17680 17680 24 170 1 : tunables 0 0 0 : slabdata 104 104 0
ebitmap_node 48128 48128 64 64 1 : tunables 0 0 0 : slabdata 752 752 0
avtab_extended_perms 0 0 40 102 1 : tunables 0 0 0 : slabdata 0 0 0
avtab_node 90440 90440 24 170 1 : tunables 0 0 0 : slabdata 532 532 0
avc_xperms_data 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
avc_xperms_decision_node 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
avc_xperms_node 0 0 56 73 1 : tunables 0 0 0 : slabdata 0 0 0
avc_node 7056 7056 72 56 1 : tunables 0 0 0 : slabdata 126 126 0
lsm_inode_cache 176912 180310 56 73 1 : tunables 0 0 0 : slabdata 2470 2470 0
lsm_file_cache 18944 18944 16 256 1 : tunables 0 0 0 : slabdata 74 74 0
key_jar 1728 1728 256 32 2 : tunables 0 0 0 : slabdata 54 54 0
uts_namespace 185 185 432 37 4 : tunables 0 0 0 : slabdata 5 5 0
nsproxy 1176 1176 72 56 1 : tunables 0 0 0 : slabdata 21 21 0
vma_lock 25908 27642 40 102 1 : tunables 0 0 0 : slabdata 271 271 0
vm_area_struct 22162 23100 184 44 2 : tunables 0 0 0 : slabdata 525 525 0
fs_cache 4288 4288 64 64 1 : tunables 0 0 0 : slabdata 67 67 0
files_cache 3174 3174 704 46 8 : tunables 0 0 0 : slabdata 69 69 0
signal_cache 2800 2912 1152 28 8 : tunables 0 0 0 : slabdata 104 104 0
sighand_cache 1935 1995 2112 15 8 : tunables 0 0 0 : slabdata 133 133 0
task_struct 1897 1972 6656 4 8 : tunables 0 0 0 : slabdata 493 493 0
cred_jar 5376 5376 192 42 2 : tunables 0 0 0 : slabdata 128 128 0
anon_vma_chain 17717 19328 64 64 1 : tunables 0 0 0 : slabdata 302 302 0
anon_vma 14980 15600 104 39 1 : tunables 0 0 0 : slabdata 400 400 0
pid 3328 3328 128 32 1 : tunables 0 0 0 : slabdata 104 104 0
irq_remap_cache 72 72 8192 4 8 : tunables 0 0 0 : slabdata 18 18 0
Acpi-Operand 19712 19712 72 56 1 : tunables 0 0 0 : slabdata 352 352 0
Acpi-ParseExt 2040 2040 80 51 1 : tunables 0 0 0 : slabdata 40 40 0
Acpi-Parse 2920 2920 56 73 1 : tunables 0 0 0 : slabdata 40 40 0
Acpi-State 2091 2091 80 51 1 : tunables 0 0 0 : slabdata 41 41 0
Acpi-Namespace 6715 6715 48 85 1 : tunables 0 0 0 : slabdata 79 79 0
shared_policy_node 0 0 48 85 1 : tunables 0 0 0 : slabdata 0 0 0
numa_policy 1020 1020 272 60 4 : tunables 0 0 0 : slabdata 17 17 0
perf_event 3537 25850 1264 25 8 : tunables 0 0 0 : slabdata 1034 1034 0
trace_event_file 2856 2856 96 42 1 : tunables 0 0 0 : slabdata 68 68 0
ftrace_event_field 7154 7154 56 73 1 : tunables 0 0 0 : slabdata 98 98 0
pool_workqueue 6615 12064 512 32 4 : tunables 0 0 0 : slabdata 377 377 0
maple_node 7852 8704 256 32 2 : tunables 0 0 0 : slabdata 272 272 0
radix_tree_node 18362 18648 584 56 8 : tunables 0 0 0 : slabdata 333 333 0
task_group 3162 3162 640 51 8 : tunables 0 0 0 : slabdata 62 62 0
mm_struct 1702 1702 1408 23 8 : tunables 0 0 0 : slabdata 74 74 0
vmap_area 8051 8792 72 56 1 : tunables 0 0 0 : slabdata 157 157 0
kmalloc-cg-8k 80 80 8192 4 8 : tunables 0 0 0 : slabdata 20 20 0
kmalloc-cg-4k 632 632 4096 8 8 : tunables 0 0 0 : slabdata 79 79 0
kmalloc-cg-2k 1504 1504 2048 16 8 : tunables 0 0 0 : slabdata 94 94 0
kmalloc-cg-1k 2208 2208 1024 32 8 : tunables 0 0 0 : slabdata 69 69 0
kmalloc-cg-512 2112 2112 512 32 4 : tunables 0 0 0 : slabdata 66 66 0
kmalloc-cg-256 676 704 256 32 2 : tunables 0 0 0 : slabdata 22 22 0
kmalloc-cg-192 2730 2730 192 42 2 : tunables 0 0 0 : slabdata 65 65 0
kmalloc-cg-128 1344 1344 128 32 1 : tunables 0 0 0 : slabdata 42 42 0
kmalloc-cg-96 8242 8358 96 42 1 : tunables 0 0 0 : slabdata 199 199 0
kmalloc-cg-64 4416 4416 64 64 1 : tunables 0 0 0 : slabdata 69 69 0
kmalloc-cg-32 14592 14592 32 128 1 : tunables 0 0 0 : slabdata 114 114 0
kmalloc-cg-16 14336 14336 16 256 1 : tunables 0 0 0 : slabdata 56 56 0
kmalloc-cg-8 23552 23552 8 512 1 : tunables 0 0 0 : slabdata 46 46 0
dma-kmalloc-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-2k 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-1k 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-512 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8k 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-4k 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-2k 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-1k 0 0 1024 32 8 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-512 0 0 512 32 4 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-256 0 0 256 32 2 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-192 0 0 192 42 2 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-128 1152 1152 128 32 1 : tunables 0 0 0 : slabdata 36 36 0
kmalloc-rcl-96 7182 7182 96 42 1 : tunables 0 0 0 : slabdata 171 171 0
kmalloc-rcl-64 5770 5952 64 64 1 : tunables 0 0 0 : slabdata 93 93 0
kmalloc-rcl-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-rcl-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0
kmalloc-8k 512 548 8192 4 8 : tunables 0 0 0 : slabdata 137 137 0
kmalloc-4k 3080 3240 4096 8 8 : tunables 0 0 0 : slabdata 405 405 0
kmalloc-2k 3780 3872 2048 16 8 : tunables 0 0 0 : slabdata 242 242 0
kmalloc-1k 8047 8768 1024 32 8 : tunables 0 0 0 : slabdata 274 274 0
kmalloc-512 35966 87904 512 32 4 : tunables 0 0 0 : slabdata 2747 2747 0
kmalloc-256 11607 12608 256 32 2 : tunables 0 0 0 : slabdata 394 394 0
kmalloc-192 16002 23058 192 42 2 : tunables 0 0 0 : slabdata 549 549 0
kmalloc-128 23659 24096 128 32 1 : tunables 0 0 0 : slabdata 753 753 0
kmalloc-96 26261 26712 96 42 1 : tunables 0 0 0 : slabdata 636 636 0
kmalloc-64 34663 35520 64 64 1 : tunables 0 0 0 : slabdata 555 555 0
kmalloc-32 38182 39168 32 128 1 : tunables 0 0 0 : slabdata 306 306 0
kmalloc-16 52206 52224 16 256 1 : tunables 0 0 0 : slabdata 204 204 0
kmalloc-8 66997 68096 8 512 1 : tunables 0 0 0 : slabdata 133 133 0
kmem_cache_node 4089 4096 64 64 1 : tunables 0 0 0 : slabdata 64 64 0
kmem_cache 1530 1530 320 51 4 : tunables 0 0 0 : slabdata 30 30 0
[-- Attachment #4: ethtool_-S_em1.reverted --]
[-- Type: application/octet-stream, Size: 9717 bytes --]
NIC statistics:
rx_unicast: 1426
tx_unicast: 808
rx_multicast: 32
tx_multicast: 6
rx_broadcast: 5489
tx_broadcast: 4
rx_bytes: 1622225
tx_bytes: 501646
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_1_packets: 63
tx_queue_1_bytes: 14294
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 102
tx_queue_5_bytes: 297493
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 24
tx_queue_7_bytes: 2919
tx_queue_8_packets: 2
tx_queue_8_bytes: 120
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 79
tx_queue_12_bytes: 31945
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_14_packets: 24
tx_queue_14_bytes: 2905
tx_queue_15_packets: 8
tx_queue_15_bytes: 1111
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 23
tx_queue_19_bytes: 2667
tx_queue_20_packets: 92
tx_queue_20_bytes: 19132
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 24
tx_queue_25_bytes: 2600
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 10
tx_queue_29_bytes: 1300
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 24
tx_queue_32_bytes: 2905
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 57
tx_queue_42_bytes: 13550
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 105
tx_queue_44_bytes: 8621
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 24
tx_queue_48_bytes: 2807
tx_queue_49_packets: 2
tx_queue_49_bytes: 120
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 53
tx_queue_53_bytes: 5197
tx_queue_54_packets: 52
tx_queue_54_bytes: 42729
tx_queue_55_packets: 30
tx_queue_55_bytes: 43391
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 10
tx_queue_60_bytes: 1304
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 10
tx_queue_62_bytes: 1304
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 5567
rx_queue_0_bytes: 380033
rx_queue_1_packets: 1
rx_queue_1_bytes: 1139
rx_queue_2_packets: 0
rx_queue_2_bytes: 0
rx_queue_3_packets: 52
rx_queue_3_bytes: 4511
rx_queue_4_packets: 0
rx_queue_4_bytes: 0
rx_queue_5_packets: 10
rx_queue_5_bytes: 4501
rx_queue_6_packets: 1
rx_queue_6_bytes: 452
rx_queue_7_packets: 0
rx_queue_7_bytes: 0
rx_queue_8_packets: 8
rx_queue_8_bytes: 12133
rx_queue_9_packets: 34
rx_queue_9_bytes: 5131
rx_queue_10_packets: 43
rx_queue_10_bytes: 14602
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 0
rx_queue_12_bytes: 0
rx_queue_13_packets: 16
rx_queue_13_bytes: 6377
rx_queue_14_packets: 87
rx_queue_14_bytes: 143166
rx_queue_15_packets: 29
rx_queue_15_bytes: 21852
rx_queue_16_packets: 9
rx_queue_16_bytes: 924
rx_queue_17_packets: 43
rx_queue_17_bytes: 8653
rx_queue_18_packets: 15
rx_queue_18_bytes: 1663
rx_queue_19_packets: 47
rx_queue_19_bytes: 9004
rx_queue_20_packets: 1
rx_queue_20_bytes: 345
rx_queue_21_packets: 63
rx_queue_21_bytes: 12208
rx_queue_22_packets: 32
rx_queue_22_bytes: 7975
rx_queue_23_packets: 1
rx_queue_23_bytes: 204
rx_queue_24_packets: 3
rx_queue_24_bytes: 485
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 1
rx_queue_26_bytes: 172
rx_queue_27_packets: 159
rx_queue_27_bytes: 14567
rx_queue_28_packets: 1
rx_queue_28_bytes: 197
rx_queue_29_packets: 1
rx_queue_29_bytes: 87
rx_queue_30_packets: 47
rx_queue_30_bytes: 8983
rx_queue_31_packets: 4
rx_queue_31_bytes: 838
rx_queue_32_packets: 1
rx_queue_32_bytes: 90
rx_queue_33_packets: 9
rx_queue_33_bytes: 1487
rx_queue_34_packets: 1
rx_queue_34_bytes: 113
rx_queue_35_packets: 11
rx_queue_35_bytes: 5706
rx_queue_36_packets: 22
rx_queue_36_bytes: 2350
rx_queue_37_packets: 25
rx_queue_37_bytes: 6230
rx_queue_38_packets: 2
rx_queue_38_bytes: 443
rx_queue_39_packets: 25
rx_queue_39_bytes: 6108
rx_queue_40_packets: 1
rx_queue_40_bytes: 97
rx_queue_41_packets: 1
rx_queue_41_bytes: 87
rx_queue_42_packets: 2
rx_queue_42_bytes: 166
rx_queue_43_packets: 22
rx_queue_43_bytes: 2371
rx_queue_44_packets: 11
rx_queue_44_bytes: 2665
rx_queue_45_packets: 12
rx_queue_45_bytes: 1501
rx_queue_46_packets: 1
rx_queue_46_bytes: 231
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 10
rx_queue_48_bytes: 5563
rx_queue_49_packets: 11
rx_queue_49_bytes: 2665
rx_queue_50_packets: 131
rx_queue_50_bytes: 502138
rx_queue_51_packets: 10
rx_queue_51_bytes: 6429
rx_queue_52_packets: 27
rx_queue_52_bytes: 8166
rx_queue_53_packets: 12
rx_queue_53_bytes: 2752
rx_queue_54_packets: 9
rx_queue_54_bytes: 7073
rx_queue_55_packets: 1
rx_queue_55_bytes: 450
rx_queue_56_packets: 2
rx_queue_56_bytes: 1158
rx_queue_57_packets: 276
rx_queue_57_bytes: 356389
rx_queue_58_packets: 5
rx_queue_58_bytes: 300
rx_queue_59_packets: 25
rx_queue_59_bytes: 2776
rx_queue_60_packets: 1
rx_queue_60_bytes: 143
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 3
rx_queue_62_bytes: 992
rx_queue_63_packets: 3
rx_queue_63_bytes: 1476
rx_bytes.nic: 1744217
tx_bytes.nic: 504918
rx_unicast.nic: 1426
tx_unicast.nic: 808
rx_multicast.nic: 1154
tx_multicast.nic: 6
rx_broadcast.nic: 6128
tx_broadcast.nic: 4
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 6033
tx_size_64.nic: 4
rx_size_127.nic: 2055
tx_size_127.nic: 534
rx_size_255.nic: 177
tx_size_255.nic: 66
rx_size_511.nic: 147
tx_size_511.nic: 71
rx_size_1023.nic: 82
tx_size_1023.nic: 38
rx_size_1522.nic: 53
tx_size_1522.nic: 23
rx_size_big.nic: 161
tx_size_big.nic: 82
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
[-- Attachment #5: ethtool_-S_em2.reverted --]
[-- Type: application/octet-stream, Size: 9373 bytes --]
NIC statistics:
rx_unicast: 0
tx_unicast: 0
rx_multicast: 0
tx_multicast: 0
rx_broadcast: 0
tx_broadcast: 0
rx_bytes: 0
tx_bytes: 0
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_1_packets: 0
tx_queue_1_bytes: 0
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 0
tx_queue_8_bytes: 0
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 0
rx_queue_0_bytes: 0
rx_queue_1_packets: 0
rx_queue_1_bytes: 0
rx_queue_2_packets: 0
rx_queue_2_bytes: 0
rx_queue_3_packets: 0
rx_queue_3_bytes: 0
rx_queue_4_packets: 0
rx_queue_4_bytes: 0
rx_queue_5_packets: 0
rx_queue_5_bytes: 0
rx_queue_6_packets: 0
rx_queue_6_bytes: 0
rx_queue_7_packets: 0
rx_queue_7_bytes: 0
rx_queue_8_packets: 0
rx_queue_8_bytes: 0
rx_queue_9_packets: 0
rx_queue_9_bytes: 0
rx_queue_10_packets: 0
rx_queue_10_bytes: 0
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 0
rx_queue_12_bytes: 0
rx_queue_13_packets: 0
rx_queue_13_bytes: 0
rx_queue_14_packets: 0
rx_queue_14_bytes: 0
rx_queue_15_packets: 0
rx_queue_15_bytes: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
rx_bytes.nic: 0
tx_bytes.nic: 0
rx_unicast.nic: 0
tx_unicast.nic: 0
rx_multicast.nic: 0
tx_multicast.nic: 0
rx_broadcast.nic: 0
tx_broadcast.nic: 0
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 0
tx_size_64.nic: 0
rx_size_127.nic: 0
tx_size_127.nic: 0
rx_size_255.nic: 0
tx_size_255.nic: 0
rx_size_511.nic: 0
tx_size_511.nic: 0
rx_size_1023.nic: 0
tx_size_1023.nic: 0
rx_size_1522.nic: 0
tx_size_1522.nic: 0
rx_size_big.nic: 0
tx_size_big.nic: 0
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
[-- Attachment #6: ethtool_-S_em1.vanila --]
[-- Type: application/octet-stream, Size: 9789 bytes --]
NIC statistics:
rx_unicast: 2410
tx_unicast: 795
rx_multicast: 81
tx_multicast: 7
rx_broadcast: 47318
tx_broadcast: 4
rx_bytes: 4028974
tx_bytes: 517682
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_1_packets: 0
tx_queue_1_bytes: 0
tx_queue_2_packets: 77
tx_queue_2_bytes: 18436
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 8
tx_queue_8_bytes: 1110
tx_queue_9_packets: 25
tx_queue_9_bytes: 2873
tx_queue_10_packets: 110
tx_queue_10_bytes: 117632
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 36
tx_queue_12_bytes: 4467
tx_queue_13_packets: 25
tx_queue_13_bytes: 3013
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 12
tx_queue_17_bytes: 1389
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 20
tx_queue_19_bytes: 10787
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 155
tx_queue_22_bytes: 299998
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 25
tx_queue_25_bytes: 2971
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 25
tx_queue_32_bytes: 2666
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 7
tx_queue_34_bytes: 710
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 21
tx_queue_36_bytes: 2676
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 9
tx_queue_40_bytes: 830
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 25
tx_queue_42_bytes: 2971
tx_queue_43_packets: 8
tx_queue_43_bytes: 1111
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 11
tx_queue_48_bytes: 1366
tx_queue_49_packets: 25
tx_queue_49_bytes: 2771
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 25
tx_queue_53_bytes: 3219
tx_queue_54_packets: 10
tx_queue_54_bytes: 1265
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 97
tx_queue_58_bytes: 26439
tx_queue_59_packets: 25
tx_queue_59_bytes: 2901
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 25
tx_queue_62_bytes: 2901
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 47450
rx_queue_0_bytes: 2857586
rx_queue_1_packets: 126
rx_queue_1_bytes: 24586
rx_queue_2_packets: 30
rx_queue_2_bytes: 6761
rx_queue_3_packets: 3
rx_queue_3_bytes: 210
rx_queue_4_packets: 16
rx_queue_4_bytes: 4414
rx_queue_5_packets: 20
rx_queue_5_bytes: 4370
rx_queue_6_packets: 328
rx_queue_6_bytes: 30493
rx_queue_7_packets: 66
rx_queue_7_bytes: 52978
rx_queue_8_packets: 68
rx_queue_8_bytes: 16610
rx_queue_9_packets: 28
rx_queue_9_bytes: 3645
rx_queue_10_packets: 12
rx_queue_10_bytes: 5666
rx_queue_11_packets: 46
rx_queue_11_bytes: 13992
rx_queue_12_packets: 10
rx_queue_12_bytes: 5577
rx_queue_13_packets: 30
rx_queue_13_bytes: 13706
rx_queue_14_packets: 32
rx_queue_14_bytes: 10884
rx_queue_15_packets: 62
rx_queue_15_bytes: 13169
rx_queue_16_packets: 25
rx_queue_16_bytes: 7314
rx_queue_17_packets: 2
rx_queue_17_bytes: 261
rx_queue_18_packets: 34
rx_queue_18_bytes: 7594
rx_queue_19_packets: 29
rx_queue_19_bytes: 11790
rx_queue_20_packets: 26
rx_queue_20_bytes: 5596
rx_queue_21_packets: 56
rx_queue_21_bytes: 41339
rx_queue_22_packets: 13
rx_queue_22_bytes: 6206
rx_queue_23_packets: 11
rx_queue_23_bytes: 5741
rx_queue_24_packets: 29
rx_queue_24_bytes: 18359
rx_queue_25_packets: 23
rx_queue_25_bytes: 7551
rx_queue_26_packets: 57
rx_queue_26_bytes: 7200
rx_queue_27_packets: 51
rx_queue_27_bytes: 12942
rx_queue_28_packets: 12
rx_queue_28_bytes: 2725
rx_queue_29_packets: 3
rx_queue_29_bytes: 465
rx_queue_30_packets: 43
rx_queue_30_bytes: 4053
rx_queue_31_packets: 27
rx_queue_31_bytes: 3041
rx_queue_32_packets: 15
rx_queue_32_bytes: 3016
rx_queue_33_packets: 25
rx_queue_33_bytes: 5600
rx_queue_34_packets: 23
rx_queue_34_bytes: 8342
rx_queue_35_packets: 10
rx_queue_35_bytes: 5540
rx_queue_36_packets: 27
rx_queue_36_bytes: 2881
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 11
rx_queue_38_bytes: 729
rx_queue_39_packets: 40
rx_queue_39_bytes: 10748
rx_queue_40_packets: 2
rx_queue_40_bytes: 120
rx_queue_41_packets: 42
rx_queue_41_bytes: 8835
rx_queue_42_packets: 12
rx_queue_42_bytes: 2725
rx_queue_43_packets: 1
rx_queue_43_bytes: 541
rx_queue_44_packets: 34
rx_queue_44_bytes: 11266
rx_queue_45_packets: 25
rx_queue_45_bytes: 12599
rx_queue_46_packets: 12
rx_queue_46_bytes: 5686
rx_queue_47_packets: 63
rx_queue_47_bytes: 17075
rx_queue_48_packets: 2
rx_queue_48_bytes: 120
rx_queue_49_packets: 4
rx_queue_49_bytes: 246
rx_queue_50_packets: 22
rx_queue_50_bytes: 6387
rx_queue_51_packets: 25
rx_queue_51_bytes: 7633
rx_queue_52_packets: 17
rx_queue_52_bytes: 5175
rx_queue_53_packets: 14
rx_queue_53_bytes: 2845
rx_queue_54_packets: 36
rx_queue_54_bytes: 4330
rx_queue_55_packets: 13
rx_queue_55_bytes: 3588
rx_queue_56_packets: 36
rx_queue_56_bytes: 3406
rx_queue_57_packets: 457
rx_queue_57_bytes: 471551
rx_queue_58_packets: 2
rx_queue_58_bytes: 203
rx_queue_59_packets: 24
rx_queue_59_bytes: 7172
rx_queue_60_packets: 3
rx_queue_60_bytes: 180
rx_queue_61_packets: 3
rx_queue_61_bytes: 250
rx_queue_62_packets: 40
rx_queue_62_bytes: 4237
rx_queue_63_packets: 1
rx_queue_63_bytes: 60
rx_bytes.nic: 4135212
tx_bytes.nic: 520906
rx_unicast.nic: 2417
tx_unicast.nic: 795
rx_multicast.nic: 944
tx_multicast.nic: 7
rx_broadcast.nic: 47943
tx_broadcast.nic: 4
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 47914
tx_size_64.nic: 4
rx_size_127.nic: 2401
tx_size_127.nic: 524
rx_size_255.nic: 215
tx_size_255.nic: 92
rx_size_511.nic: 195
tx_size_511.nic: 20
rx_size_1023.nic: 127
tx_size_1023.nic: 52
rx_size_1522.nic: 440
tx_size_1522.nic: 4
rx_size_big.nic: 12
tx_size_big.nic: 110
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
[-- Attachment #7: ethtool_-S_p3p1.reverted --]
[-- Type: application/octet-stream, Size: 9949 bytes --]
NIC statistics:
rx_unicast: 2620
tx_unicast: 3973
rx_multicast: 12958
tx_multicast: 6
rx_broadcast: 3287
tx_broadcast: 6
rx_bytes: 4483342
tx_bytes: 1089545
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 76
tx_queue_0_bytes: 11324
tx_queue_1_packets: 20
tx_queue_1_bytes: 2674
tx_queue_2_packets: 29
tx_queue_2_bytes: 6360
tx_queue_3_packets: 566
tx_queue_3_bytes: 35875
tx_queue_4_packets: 8
tx_queue_4_bytes: 1484
tx_queue_5_packets: 20
tx_queue_5_bytes: 3374
tx_queue_6_packets: 10
tx_queue_6_bytes: 763
tx_queue_7_packets: 48
tx_queue_7_bytes: 8166
tx_queue_8_packets: 19
tx_queue_8_bytes: 2213
tx_queue_9_packets: 42
tx_queue_9_bytes: 39486
tx_queue_10_packets: 71
tx_queue_10_bytes: 50539
tx_queue_11_packets: 19
tx_queue_11_bytes: 3163
tx_queue_12_packets: 39
tx_queue_12_bytes: 4726
tx_queue_13_packets: 3
tx_queue_13_bytes: 292
tx_queue_14_packets: 110
tx_queue_14_bytes: 97401
tx_queue_15_packets: 85
tx_queue_15_bytes: 55694
tx_queue_16_packets: 12
tx_queue_16_bytes: 1783
tx_queue_17_packets: 48
tx_queue_17_bytes: 19004
tx_queue_18_packets: 985
tx_queue_18_bytes: 190936
tx_queue_19_packets: 37
tx_queue_19_bytes: 6379
tx_queue_20_packets: 62
tx_queue_20_bytes: 7777
tx_queue_21_packets: 2
tx_queue_21_bytes: 186
tx_queue_22_packets: 38
tx_queue_22_bytes: 5008
tx_queue_23_packets: 57
tx_queue_23_bytes: 8104
tx_queue_24_packets: 11
tx_queue_24_bytes: 1671
tx_queue_25_packets: 9
tx_queue_25_bytes: 1636
tx_queue_26_packets: 43
tx_queue_26_bytes: 6676
tx_queue_27_packets: 29
tx_queue_27_bytes: 5153
tx_queue_28_packets: 47
tx_queue_28_bytes: 7157
tx_queue_29_packets: 140
tx_queue_29_bytes: 65794
tx_queue_30_packets: 12
tx_queue_30_bytes: 1923
tx_queue_31_packets: 25
tx_queue_31_bytes: 3636
tx_queue_32_packets: 8
tx_queue_32_bytes: 1565
tx_queue_33_packets: 22
tx_queue_33_bytes: 3368
tx_queue_34_packets: 77
tx_queue_34_bytes: 53573
tx_queue_35_packets: 25
tx_queue_35_bytes: 3277
tx_queue_36_packets: 23
tx_queue_36_bytes: 3569
tx_queue_37_packets: 10
tx_queue_37_bytes: 1630
tx_queue_38_packets: 59
tx_queue_38_bytes: 43718
tx_queue_39_packets: 57
tx_queue_39_bytes: 42737
tx_queue_40_packets: 4
tx_queue_40_bytes: 370
tx_queue_41_packets: 3
tx_queue_41_bytes: 213
tx_queue_42_packets: 15
tx_queue_42_bytes: 1713
tx_queue_43_packets: 264
tx_queue_43_bytes: 78806
tx_queue_44_packets: 228
tx_queue_44_bytes: 98692
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 3
tx_queue_46_bytes: 288
tx_queue_47_packets: 29
tx_queue_47_bytes: 4889
tx_queue_48_packets: 1
tx_queue_48_bytes: 98
tx_queue_49_packets: 6
tx_queue_49_bytes: 428
tx_queue_50_packets: 34
tx_queue_50_bytes: 5193
tx_queue_51_packets: 34
tx_queue_51_bytes: 4506
tx_queue_52_packets: 27
tx_queue_52_bytes: 12351
tx_queue_53_packets: 13
tx_queue_53_bytes: 1944
tx_queue_54_packets: 15
tx_queue_54_bytes: 2067
tx_queue_55_packets: 17
tx_queue_55_bytes: 1926
tx_queue_56_packets: 1
tx_queue_56_bytes: 91
tx_queue_57_packets: 45
tx_queue_57_bytes: 10424
tx_queue_58_packets: 15
tx_queue_58_bytes: 2016
tx_queue_59_packets: 13
tx_queue_59_bytes: 1828
tx_queue_60_packets: 2
tx_queue_60_bytes: 171
tx_queue_61_packets: 179
tx_queue_61_bytes: 32376
tx_queue_62_packets: 11
tx_queue_62_bytes: 2274
tx_queue_63_packets: 23
tx_queue_63_bytes: 3468
rx_queue_0_packets: 16182
rx_queue_0_bytes: 1028323
rx_queue_1_packets: 21
rx_queue_1_bytes: 1394
rx_queue_2_packets: 71
rx_queue_2_bytes: 86673
rx_queue_3_packets: 30
rx_queue_3_bytes: 15994
rx_queue_4_packets: 1
rx_queue_4_bytes: 90
rx_queue_5_packets: 20
rx_queue_5_bytes: 11168
rx_queue_6_packets: 45
rx_queue_6_bytes: 7911
rx_queue_7_packets: 12
rx_queue_7_bytes: 6247
rx_queue_8_packets: 11
rx_queue_8_bytes: 5716
rx_queue_9_packets: 3
rx_queue_9_bytes: 759
rx_queue_10_packets: 12
rx_queue_10_bytes: 2971
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 14
rx_queue_12_bytes: 10363
rx_queue_13_packets: 9
rx_queue_13_bytes: 5480
rx_queue_14_packets: 11
rx_queue_14_bytes: 1455
rx_queue_15_packets: 19
rx_queue_15_bytes: 12190
rx_queue_16_packets: 32
rx_queue_16_bytes: 8034
rx_queue_17_packets: 237
rx_queue_17_bytes: 1246278
rx_queue_18_packets: 13
rx_queue_18_bytes: 2831
rx_queue_19_packets: 18
rx_queue_19_bytes: 3331
rx_queue_20_packets: 4
rx_queue_20_bytes: 2054
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 21
rx_queue_22_bytes: 8254
rx_queue_23_packets: 10
rx_queue_23_bytes: 5711
rx_queue_24_packets: 34
rx_queue_24_bytes: 8630
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 35
rx_queue_26_bytes: 12235
rx_queue_27_packets: 2
rx_queue_27_bytes: 552
rx_queue_28_packets: 25
rx_queue_28_bytes: 9820
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 144
rx_queue_30_bytes: 96107
rx_queue_31_packets: 24
rx_queue_31_bytes: 6260
rx_queue_32_packets: 23
rx_queue_32_bytes: 2566
rx_queue_33_packets: 1
rx_queue_33_bytes: 547
rx_queue_34_packets: 46
rx_queue_34_bytes: 9601
rx_queue_35_packets: 12
rx_queue_35_bytes: 5810
rx_queue_36_packets: 48
rx_queue_36_bytes: 4169
rx_queue_37_packets: 13
rx_queue_37_bytes: 5969
rx_queue_38_packets: 12
rx_queue_38_bytes: 6520
rx_queue_39_packets: 151
rx_queue_39_bytes: 15113
rx_queue_40_packets: 32
rx_queue_40_bytes: 7139
rx_queue_41_packets: 12
rx_queue_41_bytes: 6843
rx_queue_42_packets: 18
rx_queue_42_bytes: 6738
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 10
rx_queue_44_bytes: 5571
rx_queue_45_packets: 22
rx_queue_45_bytes: 5330
rx_queue_46_packets: 32
rx_queue_46_bytes: 7900
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 1
rx_queue_48_bytes: 248
rx_queue_49_packets: 1
rx_queue_49_bytes: 171
rx_queue_50_packets: 14
rx_queue_50_bytes: 1597
rx_queue_51_packets: 2
rx_queue_51_bytes: 180
rx_queue_52_packets: 1
rx_queue_52_bytes: 130
rx_queue_53_packets: 32
rx_queue_53_bytes: 6179
rx_queue_54_packets: 23
rx_queue_54_bytes: 2701
rx_queue_55_packets: 63
rx_queue_55_bytes: 5768
rx_queue_56_packets: 86
rx_queue_56_bytes: 191534
rx_queue_57_packets: 44
rx_queue_57_bytes: 13764
rx_queue_58_packets: 108
rx_queue_58_bytes: 12422
rx_queue_59_packets: 928
rx_queue_59_bytes: 1388417
rx_queue_60_packets: 12
rx_queue_60_bytes: 5811
rx_queue_61_packets: 20
rx_queue_61_bytes: 8145
rx_queue_62_packets: 29
rx_queue_62_bytes: 6875
rx_queue_63_packets: 9
rx_queue_63_bytes: 5501
rx_bytes.nic: 4619986
tx_bytes.nic: 1105485
rx_unicast.nic: 2620
tx_unicast.nic: 3973
rx_multicast.nic: 14155
tx_multicast.nic: 6
rx_broadcast.nic: 4069
tx_broadcast.nic: 6
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 4149
tx_size_64.nic: 574
rx_size_127.nic: 15042
tx_size_127.nic: 1385
rx_size_255.nic: 136
tx_size_255.nic: 1313
rx_size_511.nic: 145
tx_size_511.nic: 167
rx_size_1023.nic: 96
tx_size_1023.nic: 226
rx_size_1522.nic: 1093
tx_size_1522.nic: 308
rx_size_big.nic: 183
tx_size_big.nic: 12
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
[-- Attachment #8: ethtool_-S_em2.vanila --]
[-- Type: application/octet-stream, Size: 9373 bytes --]
NIC statistics:
rx_unicast: 0
tx_unicast: 0
rx_multicast: 0
tx_multicast: 0
rx_broadcast: 0
tx_broadcast: 0
rx_bytes: 0
tx_bytes: 0
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_1_packets: 0
tx_queue_1_bytes: 0
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 0
tx_queue_8_bytes: 0
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 0
rx_queue_0_bytes: 0
rx_queue_1_packets: 0
rx_queue_1_bytes: 0
rx_queue_2_packets: 0
rx_queue_2_bytes: 0
rx_queue_3_packets: 0
rx_queue_3_bytes: 0
rx_queue_4_packets: 0
rx_queue_4_bytes: 0
rx_queue_5_packets: 0
rx_queue_5_bytes: 0
rx_queue_6_packets: 0
rx_queue_6_bytes: 0
rx_queue_7_packets: 0
rx_queue_7_bytes: 0
rx_queue_8_packets: 0
rx_queue_8_bytes: 0
rx_queue_9_packets: 0
rx_queue_9_bytes: 0
rx_queue_10_packets: 0
rx_queue_10_bytes: 0
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 0
rx_queue_12_bytes: 0
rx_queue_13_packets: 0
rx_queue_13_bytes: 0
rx_queue_14_packets: 0
rx_queue_14_bytes: 0
rx_queue_15_packets: 0
rx_queue_15_bytes: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
rx_bytes.nic: 0
tx_bytes.nic: 0
rx_unicast.nic: 0
tx_unicast.nic: 0
rx_multicast.nic: 0
tx_multicast.nic: 0
rx_broadcast.nic: 0
tx_broadcast.nic: 0
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 0
tx_size_64.nic: 0
rx_size_127.nic: 0
tx_size_127.nic: 0
rx_size_255.nic: 0
tx_size_255.nic: 0
rx_size_511.nic: 0
tx_size_511.nic: 0
rx_size_1023.nic: 0
tx_size_1023.nic: 0
rx_size_1522.nic: 0
tx_size_1522.nic: 0
rx_size_big.nic: 0
tx_size_big.nic: 0
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
[-- Attachment #9: ethtool_-S_p3p2.reverted --]
[-- Type: application/octet-stream, Size: 9373 bytes --]
NIC statistics:
rx_unicast: 0
tx_unicast: 0
rx_multicast: 0
tx_multicast: 0
rx_broadcast: 0
tx_broadcast: 0
rx_bytes: 0
tx_bytes: 0
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_1_packets: 0
tx_queue_1_bytes: 0
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 0
tx_queue_8_bytes: 0
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 0
rx_queue_0_bytes: 0
rx_queue_1_packets: 0
rx_queue_1_bytes: 0
rx_queue_2_packets: 0
rx_queue_2_bytes: 0
rx_queue_3_packets: 0
rx_queue_3_bytes: 0
rx_queue_4_packets: 0
rx_queue_4_bytes: 0
rx_queue_5_packets: 0
rx_queue_5_bytes: 0
rx_queue_6_packets: 0
rx_queue_6_bytes: 0
rx_queue_7_packets: 0
rx_queue_7_bytes: 0
rx_queue_8_packets: 0
rx_queue_8_bytes: 0
rx_queue_9_packets: 0
rx_queue_9_bytes: 0
rx_queue_10_packets: 0
rx_queue_10_bytes: 0
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 0
rx_queue_12_bytes: 0
rx_queue_13_packets: 0
rx_queue_13_bytes: 0
rx_queue_14_packets: 0
rx_queue_14_bytes: 0
rx_queue_15_packets: 0
rx_queue_15_bytes: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
rx_bytes.nic: 0
tx_bytes.nic: 0
rx_unicast.nic: 0
tx_unicast.nic: 0
rx_multicast.nic: 0
tx_multicast.nic: 0
rx_broadcast.nic: 0
tx_broadcast.nic: 0
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 0
tx_size_64.nic: 0
rx_size_127.nic: 0
tx_size_127.nic: 0
rx_size_255.nic: 0
tx_size_255.nic: 0
rx_size_511.nic: 0
tx_size_511.nic: 0
rx_size_1023.nic: 0
tx_size_1023.nic: 0
rx_size_1522.nic: 0
tx_size_1522.nic: 0
rx_size_big.nic: 0
tx_size_big.nic: 0
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
[-- Attachment #10: ethtool_-S_p3p2.vanila --]
[-- Type: application/octet-stream, Size: 9373 bytes --]
NIC statistics:
rx_unicast: 0
tx_unicast: 0
rx_multicast: 0
tx_multicast: 0
rx_broadcast: 0
tx_broadcast: 0
rx_bytes: 0
tx_bytes: 0
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 0
tx_queue_0_bytes: 0
tx_queue_1_packets: 0
tx_queue_1_bytes: 0
tx_queue_2_packets: 0
tx_queue_2_bytes: 0
tx_queue_3_packets: 0
tx_queue_3_bytes: 0
tx_queue_4_packets: 0
tx_queue_4_bytes: 0
tx_queue_5_packets: 0
tx_queue_5_bytes: 0
tx_queue_6_packets: 0
tx_queue_6_bytes: 0
tx_queue_7_packets: 0
tx_queue_7_bytes: 0
tx_queue_8_packets: 0
tx_queue_8_bytes: 0
tx_queue_9_packets: 0
tx_queue_9_bytes: 0
tx_queue_10_packets: 0
tx_queue_10_bytes: 0
tx_queue_11_packets: 0
tx_queue_11_bytes: 0
tx_queue_12_packets: 0
tx_queue_12_bytes: 0
tx_queue_13_packets: 0
tx_queue_13_bytes: 0
tx_queue_14_packets: 0
tx_queue_14_bytes: 0
tx_queue_15_packets: 0
tx_queue_15_bytes: 0
tx_queue_16_packets: 0
tx_queue_16_bytes: 0
tx_queue_17_packets: 0
tx_queue_17_bytes: 0
tx_queue_18_packets: 0
tx_queue_18_bytes: 0
tx_queue_19_packets: 0
tx_queue_19_bytes: 0
tx_queue_20_packets: 0
tx_queue_20_bytes: 0
tx_queue_21_packets: 0
tx_queue_21_bytes: 0
tx_queue_22_packets: 0
tx_queue_22_bytes: 0
tx_queue_23_packets: 0
tx_queue_23_bytes: 0
tx_queue_24_packets: 0
tx_queue_24_bytes: 0
tx_queue_25_packets: 0
tx_queue_25_bytes: 0
tx_queue_26_packets: 0
tx_queue_26_bytes: 0
tx_queue_27_packets: 0
tx_queue_27_bytes: 0
tx_queue_28_packets: 0
tx_queue_28_bytes: 0
tx_queue_29_packets: 0
tx_queue_29_bytes: 0
tx_queue_30_packets: 0
tx_queue_30_bytes: 0
tx_queue_31_packets: 0
tx_queue_31_bytes: 0
tx_queue_32_packets: 0
tx_queue_32_bytes: 0
tx_queue_33_packets: 0
tx_queue_33_bytes: 0
tx_queue_34_packets: 0
tx_queue_34_bytes: 0
tx_queue_35_packets: 0
tx_queue_35_bytes: 0
tx_queue_36_packets: 0
tx_queue_36_bytes: 0
tx_queue_37_packets: 0
tx_queue_37_bytes: 0
tx_queue_38_packets: 0
tx_queue_38_bytes: 0
tx_queue_39_packets: 0
tx_queue_39_bytes: 0
tx_queue_40_packets: 0
tx_queue_40_bytes: 0
tx_queue_41_packets: 0
tx_queue_41_bytes: 0
tx_queue_42_packets: 0
tx_queue_42_bytes: 0
tx_queue_43_packets: 0
tx_queue_43_bytes: 0
tx_queue_44_packets: 0
tx_queue_44_bytes: 0
tx_queue_45_packets: 0
tx_queue_45_bytes: 0
tx_queue_46_packets: 0
tx_queue_46_bytes: 0
tx_queue_47_packets: 0
tx_queue_47_bytes: 0
tx_queue_48_packets: 0
tx_queue_48_bytes: 0
tx_queue_49_packets: 0
tx_queue_49_bytes: 0
tx_queue_50_packets: 0
tx_queue_50_bytes: 0
tx_queue_51_packets: 0
tx_queue_51_bytes: 0
tx_queue_52_packets: 0
tx_queue_52_bytes: 0
tx_queue_53_packets: 0
tx_queue_53_bytes: 0
tx_queue_54_packets: 0
tx_queue_54_bytes: 0
tx_queue_55_packets: 0
tx_queue_55_bytes: 0
tx_queue_56_packets: 0
tx_queue_56_bytes: 0
tx_queue_57_packets: 0
tx_queue_57_bytes: 0
tx_queue_58_packets: 0
tx_queue_58_bytes: 0
tx_queue_59_packets: 0
tx_queue_59_bytes: 0
tx_queue_60_packets: 0
tx_queue_60_bytes: 0
tx_queue_61_packets: 0
tx_queue_61_bytes: 0
tx_queue_62_packets: 0
tx_queue_62_bytes: 0
tx_queue_63_packets: 0
tx_queue_63_bytes: 0
rx_queue_0_packets: 0
rx_queue_0_bytes: 0
rx_queue_1_packets: 0
rx_queue_1_bytes: 0
rx_queue_2_packets: 0
rx_queue_2_bytes: 0
rx_queue_3_packets: 0
rx_queue_3_bytes: 0
rx_queue_4_packets: 0
rx_queue_4_bytes: 0
rx_queue_5_packets: 0
rx_queue_5_bytes: 0
rx_queue_6_packets: 0
rx_queue_6_bytes: 0
rx_queue_7_packets: 0
rx_queue_7_bytes: 0
rx_queue_8_packets: 0
rx_queue_8_bytes: 0
rx_queue_9_packets: 0
rx_queue_9_bytes: 0
rx_queue_10_packets: 0
rx_queue_10_bytes: 0
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 0
rx_queue_12_bytes: 0
rx_queue_13_packets: 0
rx_queue_13_bytes: 0
rx_queue_14_packets: 0
rx_queue_14_bytes: 0
rx_queue_15_packets: 0
rx_queue_15_bytes: 0
rx_queue_16_packets: 0
rx_queue_16_bytes: 0
rx_queue_17_packets: 0
rx_queue_17_bytes: 0
rx_queue_18_packets: 0
rx_queue_18_bytes: 0
rx_queue_19_packets: 0
rx_queue_19_bytes: 0
rx_queue_20_packets: 0
rx_queue_20_bytes: 0
rx_queue_21_packets: 0
rx_queue_21_bytes: 0
rx_queue_22_packets: 0
rx_queue_22_bytes: 0
rx_queue_23_packets: 0
rx_queue_23_bytes: 0
rx_queue_24_packets: 0
rx_queue_24_bytes: 0
rx_queue_25_packets: 0
rx_queue_25_bytes: 0
rx_queue_26_packets: 0
rx_queue_26_bytes: 0
rx_queue_27_packets: 0
rx_queue_27_bytes: 0
rx_queue_28_packets: 0
rx_queue_28_bytes: 0
rx_queue_29_packets: 0
rx_queue_29_bytes: 0
rx_queue_30_packets: 0
rx_queue_30_bytes: 0
rx_queue_31_packets: 0
rx_queue_31_bytes: 0
rx_queue_32_packets: 0
rx_queue_32_bytes: 0
rx_queue_33_packets: 0
rx_queue_33_bytes: 0
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 0
rx_queue_35_bytes: 0
rx_queue_36_packets: 0
rx_queue_36_bytes: 0
rx_queue_37_packets: 0
rx_queue_37_bytes: 0
rx_queue_38_packets: 0
rx_queue_38_bytes: 0
rx_queue_39_packets: 0
rx_queue_39_bytes: 0
rx_queue_40_packets: 0
rx_queue_40_bytes: 0
rx_queue_41_packets: 0
rx_queue_41_bytes: 0
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 0
rx_queue_43_bytes: 0
rx_queue_44_packets: 0
rx_queue_44_bytes: 0
rx_queue_45_packets: 0
rx_queue_45_bytes: 0
rx_queue_46_packets: 0
rx_queue_46_bytes: 0
rx_queue_47_packets: 0
rx_queue_47_bytes: 0
rx_queue_48_packets: 0
rx_queue_48_bytes: 0
rx_queue_49_packets: 0
rx_queue_49_bytes: 0
rx_queue_50_packets: 0
rx_queue_50_bytes: 0
rx_queue_51_packets: 0
rx_queue_51_bytes: 0
rx_queue_52_packets: 0
rx_queue_52_bytes: 0
rx_queue_53_packets: 0
rx_queue_53_bytes: 0
rx_queue_54_packets: 0
rx_queue_54_bytes: 0
rx_queue_55_packets: 0
rx_queue_55_bytes: 0
rx_queue_56_packets: 0
rx_queue_56_bytes: 0
rx_queue_57_packets: 0
rx_queue_57_bytes: 0
rx_queue_58_packets: 0
rx_queue_58_bytes: 0
rx_queue_59_packets: 0
rx_queue_59_bytes: 0
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 0
rx_queue_62_bytes: 0
rx_queue_63_packets: 0
rx_queue_63_bytes: 0
rx_bytes.nic: 0
tx_bytes.nic: 0
rx_unicast.nic: 0
tx_unicast.nic: 0
rx_multicast.nic: 0
tx_multicast.nic: 0
rx_broadcast.nic: 0
tx_broadcast.nic: 0
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 0
tx_size_64.nic: 0
rx_size_127.nic: 0
tx_size_127.nic: 0
rx_size_255.nic: 0
tx_size_255.nic: 0
rx_size_511.nic: 0
tx_size_511.nic: 0
rx_size_1023.nic: 0
tx_size_1023.nic: 0
rx_size_1522.nic: 0
tx_size_1522.nic: 0
rx_size_big.nic: 0
tx_size_big.nic: 0
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
[-- Attachment #11: ethtool_-S_p3p1.vanila --]
[-- Type: application/octet-stream, Size: 9970 bytes --]
NIC statistics:
rx_unicast: 3318
tx_unicast: 6036
rx_multicast: 17512
tx_multicast: 7
rx_broadcast: 9708
tx_broadcast: 7
rx_bytes: 7018146
tx_bytes: 1828728
rx_dropped: 0
rx_unknown_protocol: 0
rx_alloc_fail: 0
rx_pg_alloc_fail: 0
tx_errors: 0
tx_linearize: 0
tx_busy: 0
tx_restart: 0
tx_queue_0_packets: 127
tx_queue_0_bytes: 42547
tx_queue_1_packets: 43
tx_queue_1_bytes: 6596
tx_queue_2_packets: 64
tx_queue_2_bytes: 6323
tx_queue_3_packets: 37
tx_queue_3_bytes: 5448
tx_queue_4_packets: 11
tx_queue_4_bytes: 1711
tx_queue_5_packets: 575
tx_queue_5_bytes: 39247
tx_queue_6_packets: 393
tx_queue_6_bytes: 277407
tx_queue_7_packets: 30
tx_queue_7_bytes: 4728
tx_queue_8_packets: 32
tx_queue_8_bytes: 5039
tx_queue_9_packets: 21
tx_queue_9_bytes: 1682
tx_queue_10_packets: 68
tx_queue_10_bytes: 9895
tx_queue_11_packets: 13
tx_queue_11_bytes: 1929
tx_queue_12_packets: 625
tx_queue_12_bytes: 285489
tx_queue_13_packets: 70
tx_queue_13_bytes: 14720
tx_queue_14_packets: 21
tx_queue_14_bytes: 2446
tx_queue_15_packets: 48
tx_queue_15_bytes: 46082
tx_queue_16_packets: 74
tx_queue_16_bytes: 10274
tx_queue_17_packets: 30
tx_queue_17_bytes: 2941
tx_queue_18_packets: 15
tx_queue_18_bytes: 2021
tx_queue_19_packets: 40
tx_queue_19_bytes: 6560
tx_queue_20_packets: 61
tx_queue_20_bytes: 10463
tx_queue_21_packets: 75
tx_queue_21_bytes: 51182
tx_queue_22_packets: 52
tx_queue_22_bytes: 43475
tx_queue_23_packets: 16
tx_queue_23_bytes: 2221
tx_queue_24_packets: 27
tx_queue_24_bytes: 3840
tx_queue_25_packets: 27
tx_queue_25_bytes: 3655
tx_queue_26_packets: 45
tx_queue_26_bytes: 23434
tx_queue_27_packets: 8
tx_queue_27_bytes: 688
tx_queue_28_packets: 120
tx_queue_28_bytes: 22419
tx_queue_29_packets: 21
tx_queue_29_bytes: 3420
tx_queue_30_packets: 1451
tx_queue_30_bytes: 278072
tx_queue_31_packets: 5
tx_queue_31_bytes: 386
tx_queue_32_packets: 61
tx_queue_32_bytes: 8173
tx_queue_33_packets: 65
tx_queue_33_bytes: 30957
tx_queue_34_packets: 25
tx_queue_34_bytes: 3418
tx_queue_35_packets: 32
tx_queue_35_bytes: 5076
tx_queue_36_packets: 59
tx_queue_36_bytes: 51349
tx_queue_37_packets: 86
tx_queue_37_bytes: 54777
tx_queue_38_packets: 18
tx_queue_38_bytes: 2584
tx_queue_39_packets: 34
tx_queue_39_bytes: 4254
tx_queue_40_packets: 29
tx_queue_40_bytes: 3826
tx_queue_41_packets: 13
tx_queue_41_bytes: 1047
tx_queue_42_packets: 24
tx_queue_42_bytes: 3637
tx_queue_43_packets: 50
tx_queue_43_bytes: 8555
tx_queue_44_packets: 49
tx_queue_44_bytes: 6574
tx_queue_45_packets: 9
tx_queue_45_bytes: 694
tx_queue_46_packets: 292
tx_queue_46_bytes: 83483
tx_queue_47_packets: 18
tx_queue_47_bytes: 1942
tx_queue_48_packets: 28
tx_queue_48_bytes: 3936
tx_queue_49_packets: 7
tx_queue_49_bytes: 1701
tx_queue_50_packets: 29
tx_queue_50_bytes: 3381
tx_queue_51_packets: 32
tx_queue_51_bytes: 4109
tx_queue_52_packets: 57
tx_queue_52_bytes: 8571
tx_queue_53_packets: 41
tx_queue_53_bytes: 5786
tx_queue_54_packets: 75
tx_queue_54_bytes: 11598
tx_queue_55_packets: 6
tx_queue_55_bytes: 444
tx_queue_56_packets: 47
tx_queue_56_bytes: 6697
tx_queue_57_packets: 57
tx_queue_57_bytes: 48814
tx_queue_58_packets: 257
tx_queue_58_bytes: 68114
tx_queue_59_packets: 14
tx_queue_59_bytes: 1953
tx_queue_60_packets: 5
tx_queue_60_bytes: 401
tx_queue_61_packets: 34
tx_queue_61_bytes: 6218
tx_queue_62_packets: 190
tx_queue_62_bytes: 102086
tx_queue_63_packets: 62
tx_queue_63_bytes: 46326
rx_queue_0_packets: 27114
rx_queue_0_bytes: 1705011
rx_queue_1_packets: 11
rx_queue_1_bytes: 5620
rx_queue_2_packets: 22
rx_queue_2_bytes: 10290
rx_queue_3_packets: 2
rx_queue_3_bytes: 120
rx_queue_4_packets: 3
rx_queue_4_bytes: 287
rx_queue_5_packets: 279
rx_queue_5_bytes: 514728
rx_queue_6_packets: 5
rx_queue_6_bytes: 1666
rx_queue_7_packets: 23
rx_queue_7_bytes: 13853
rx_queue_8_packets: 61
rx_queue_8_bytes: 5865
rx_queue_9_packets: 88
rx_queue_9_bytes: 8383
rx_queue_10_packets: 16
rx_queue_10_bytes: 3366
rx_queue_11_packets: 0
rx_queue_11_bytes: 0
rx_queue_12_packets: 27
rx_queue_12_bytes: 3475
rx_queue_13_packets: 27
rx_queue_13_bytes: 5770
rx_queue_14_packets: 2
rx_queue_14_bytes: 523
rx_queue_15_packets: 5
rx_queue_15_bytes: 586
rx_queue_16_packets: 14
rx_queue_16_bytes: 3187
rx_queue_17_packets: 34
rx_queue_17_bytes: 4344
rx_queue_18_packets: 2
rx_queue_18_bytes: 264
rx_queue_19_packets: 19
rx_queue_19_bytes: 7905
rx_queue_20_packets: 83
rx_queue_20_bytes: 11280
rx_queue_21_packets: 387
rx_queue_21_bytes: 2134737
rx_queue_22_packets: 11
rx_queue_22_bytes: 5614
rx_queue_23_packets: 18
rx_queue_23_bytes: 7348
rx_queue_24_packets: 1
rx_queue_24_bytes: 60
rx_queue_25_packets: 32
rx_queue_25_bytes: 4077
rx_queue_26_packets: 3
rx_queue_26_bytes: 733
rx_queue_27_packets: 43
rx_queue_27_bytes: 75918
rx_queue_28_packets: 26
rx_queue_28_bytes: 8072
rx_queue_29_packets: 16
rx_queue_29_bytes: 3533
rx_queue_30_packets: 5
rx_queue_30_bytes: 547
rx_queue_31_packets: 98
rx_queue_31_bytes: 193839
rx_queue_32_packets: 35
rx_queue_32_bytes: 8124
rx_queue_33_packets: 11
rx_queue_33_bytes: 5845
rx_queue_34_packets: 0
rx_queue_34_bytes: 0
rx_queue_35_packets: 1
rx_queue_35_bytes: 60
rx_queue_36_packets: 24
rx_queue_36_bytes: 6255
rx_queue_37_packets: 21
rx_queue_37_bytes: 11183
rx_queue_38_packets: 133
rx_queue_38_bytes: 187239
rx_queue_39_packets: 2
rx_queue_39_bytes: 173
rx_queue_40_packets: 37
rx_queue_40_bytes: 4395
rx_queue_41_packets: 2
rx_queue_41_bytes: 264
rx_queue_42_packets: 0
rx_queue_42_bytes: 0
rx_queue_43_packets: 20
rx_queue_43_bytes: 9015
rx_queue_44_packets: 12
rx_queue_44_bytes: 5962
rx_queue_45_packets: 981
rx_queue_45_bytes: 1398044
rx_queue_46_packets: 3
rx_queue_46_bytes: 283
rx_queue_47_packets: 38
rx_queue_47_bytes: 9589
rx_queue_48_packets: 14
rx_queue_48_bytes: 4287
rx_queue_49_packets: 20
rx_queue_49_bytes: 7283
rx_queue_50_packets: 3
rx_queue_50_bytes: 210
rx_queue_51_packets: 13
rx_queue_51_bytes: 3031
rx_queue_52_packets: 28
rx_queue_52_bytes: 4496
rx_queue_53_packets: 35
rx_queue_53_bytes: 5220
rx_queue_54_packets: 275
rx_queue_54_bytes: 360750
rx_queue_55_packets: 21
rx_queue_55_bytes: 8205
rx_queue_56_packets: 76
rx_queue_56_bytes: 5364
rx_queue_57_packets: 4
rx_queue_57_bytes: 882
rx_queue_58_packets: 2
rx_queue_58_bytes: 173
rx_queue_59_packets: 10
rx_queue_59_bytes: 5570
rx_queue_60_packets: 0
rx_queue_60_bytes: 0
rx_queue_61_packets: 0
rx_queue_61_bytes: 0
rx_queue_62_packets: 1
rx_queue_62_bytes: 60
rx_queue_63_packets: 239
rx_queue_63_bytes: 20631
rx_bytes.nic: 7152984
tx_bytes.nic: 1852928
rx_unicast.nic: 3328
tx_unicast.nic: 6036
rx_multicast.nic: 18646
tx_multicast.nic: 7
rx_broadcast.nic: 10470
tx_broadcast.nic: 7
tx_errors.nic: 0
tx_timeout.nic: 0
rx_size_64.nic: 10602
tx_size_64.nic: 566
rx_size_127.nic: 19883
tx_size_127.nic: 2381
rx_size_255.nic: 223
tx_size_255.nic: 1863
rx_size_511.nic: 170
tx_size_511.nic: 321
rx_size_1023.nic: 120
tx_size_1023.nic: 322
rx_size_1522.nic: 996
tx_size_1522.nic: 580
rx_size_big.nic: 450
tx_size_big.nic: 17
link_xon_rx.nic: 0
link_xon_tx.nic: 0
link_xoff_rx.nic: 0
link_xoff_tx.nic: 0
tx_dropped_link_down.nic: 0
rx_undersize.nic: 0
rx_fragments.nic: 0
rx_oversize.nic: 0
rx_jabber.nic: 0
rx_csum_bad.nic: 0
rx_length_errors.nic: 0
rx_dropped.nic: 0
rx_crc_errors.nic: 0
illegal_bytes.nic: 0
mac_local_faults.nic: 0
mac_remote_faults.nic: 0
fdir_sb_match.nic: 0
fdir_sb_status.nic: 1
tx_hwtstamp_skipped: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_flushed: 0
tx_hwtstamp_discarded: 0
late_cached_phc_updates: 0
tx_priority_0_xon.nic: 0
tx_priority_0_xoff.nic: 0
tx_priority_1_xon.nic: 0
tx_priority_1_xoff.nic: 0
tx_priority_2_xon.nic: 0
tx_priority_2_xoff.nic: 0
tx_priority_3_xon.nic: 0
tx_priority_3_xoff.nic: 0
tx_priority_4_xon.nic: 0
tx_priority_4_xoff.nic: 0
tx_priority_5_xon.nic: 0
tx_priority_5_xoff.nic: 0
tx_priority_6_xon.nic: 0
tx_priority_6_xoff.nic: 0
tx_priority_7_xon.nic: 0
tx_priority_7_xoff.nic: 0
rx_priority_0_xon.nic: 0
rx_priority_0_xoff.nic: 0
rx_priority_1_xon.nic: 0
rx_priority_1_xoff.nic: 0
rx_priority_2_xon.nic: 0
rx_priority_2_xoff.nic: 0
rx_priority_3_xon.nic: 0
rx_priority_3_xoff.nic: 0
rx_priority_4_xon.nic: 0
rx_priority_4_xoff.nic: 0
rx_priority_5_xon.nic: 0
rx_priority_5_xoff.nic: 0
rx_priority_6_xon.nic: 0
rx_priority_6_xoff.nic: 0
rx_priority_7_xon.nic: 0
rx_priority_7_xoff.nic: 0
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-11 8:26 ` Jaroslav Pulchart
@ 2024-01-12 10:23 ` Paul Menzel
2024-01-24 14:29 ` Linux regression tracking (Thorsten Leemhuis)
1 sibling, 0 replies; 23+ messages in thread
From: Paul Menzel @ 2024-01-12 10:23 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: regressions, Daniel Secik, Jesse Brandeburg, Tony Nguyen,
Dave Ertman, intel-wired-lan, Igor Raits
[Cc: +regressions@lists.linux.dev]
Am 11.01.24 um 09:26 schrieb Jaroslav Pulchart:
>>
>> On 1/8/2024 2:49 AM, Jaroslav Pulchart wrote:
>>> Hello
>>
>> First, thank you for your work trying to chase this!
>>
>>>
>>> I would like to report a regression triggered by recent change in
>>> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
>>> bisected and the regression is triggered by
>>> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
>>> check for SRIOV and LAG" commit and originally reported as part of
>>> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
>>> thread.
>>
>> I think that's a bad bisect. There is no reason I could understand for
>> that change to cause a continuous or large leak, it really doesn't make
>> any sense. Reverting it consistently helps? You're not just rewinding
>> the tree back to that point, right? just running 6.6.9 without that
>> patch? (sorry for being pedantic, just trying to be certain)
>>
>
> Reverting just the single bisected commit continuously helps for >=
> 6.6.9 and as well for current 6.7.
> We cannot use any new linux kernel without reverting it due to this
> extra memory utilization.
>
>>
>>>> However, after the following patch we see that more NUMA nodes have
>>>> such a low amount of memory and that is causing constant reclaiming
>>>> of memory because it looks like something inside of the kernel ate all
>>>> the memory. This is right after the start of the system as well.
>>>
>>> I'm reporting it here as it is a different problem than the original
>>> thread. The commit introduces a low memory problem per each numa node
>>> of the first socket (node0 .. node3 in our case) and cause constant
>>> kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
>>> memory issue is nicely visible in "numastat -m", see attached files:
>>> * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
>>> * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
>>> the server "is fresh" (after reboot), without running any application load.
>>
>> OK, so the initial allocations of your system is running your system out
>> of memory.
>>
>> Are you running jumbo frames on your ethernet interfaces?
>>
>
> Yes, we are (MTU 9000).
>
>> Do you have /proc/slabinfo output from working/non-working boot?
>>
>
> Yes, I have a complete sos report so I can pick-up files from there.
> See attached
> slabinfo.vanila (non-working)
> slabinfo.reverted (working)
>
>>>
>>> $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt
>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>> 2756.89 2754.86 100.39 2278.43 < ice
>>> fix is reverted, we have ~2GB free per numa, except one, like before
>>> == no issue
>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>> 3551.29 1530.52 2212.04 3488.09
>>> ...
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>> 127.52 66.49 120.23 263.47 <
>>
>>
>>> ice fix is present, we see just few MB free per each node, this will
>>> cause kswapd utilization!
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>> 3322.18 3134.47 195.55 879.17
>>> ...
>>>
>>> If you have some hints on how to debug what is actually occupying all
>>> that memory and some fix of the problem will be nice. We can provide
>>> testing and more reports if needed to analyze the issue. We reverted
>>> the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
>>> till we know a proper fix.
>>
>> My first suspicion is that we're contributing to the problem by running
>> out of receive descriptors memory.
>>
>> Can we see the ethtool -S stats from the freshly booted system that's
>> running out of memory or doing OOM? Also, all the standard debugging
>> info (at least once please), devlink dev info, any other configuration
>> specifics? What networking config (bonding? anything else?)
>>
>
> The system is not in OOM, it starts to continuously utilize four
> kswapd0-4 of each numa node from the first CPU socket processes (each
> at 100% and all doing swap in/out) after the system start to be used
> by application due to "low memory".
>
> We have two 25G 2P E810-XXV Adapters. The first port of each (em1 +
> p3p1) is connected and they're bonded in LACP. Second ports (em2 and
> p3p2) are unused.
>
> See attached file for working:
> ethtool_-S_em1.reverted
> ethtool_-S_em2.reverted
> ethtool_-S_p3p1.reverted
> ethtool_-S_p3p2.reverted
>
> See attached file for non-working:
> ethtool_-S_em1.vanila
> ethtool_-S_em2.vanila
> ethtool_-S_p3p1.vanila
> ethtool_-S_p3p2.vanila
>
>
>> Do you have a bugzilla.kernel.org bug yet where you can upload larger
>> files like dmesg and others?
>
> I do not have yet, I will create a new one and ping you then.
>
>>
>> Also, I'm curious if your problem goes away if you change / reduce the
>> number of queues per port. use ethtool -L eth0 combined 4 ?
>>
>
> I will try and give you feedback soon.
>
>> You also said something about reproducing when launching / destroying
>> virtual machines with VF passthrough?
>
> The memory usage is there from boot without running any VMs. The issue
> is that the host has low memory for self and it starts to use kswapd
> when we start to use it by starting vms.
>
>>
>> Can you reproduce the issue without starting qemu (just doing bare-metal
>> SR-IOV instance creation/destruction via
>> /sys/class/net/eth0/device/sriov_numvfs ?)
>>
>
> Yes we can reproduce it without qemu running, the extra memory usage
> is from the beginning after boot, not depending on any running VM.
>
> We do not use SR-IOV.
>
>> Thanks
>
> Thanks,
> Jaroslav Pulchart
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-01-12 10:23 ` Paul Menzel
0 siblings, 0 replies; 23+ messages in thread
From: Paul Menzel @ 2024-01-12 10:23 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: Jesse Brandeburg, intel-wired-lan, Igor Raits, Daniel Secik,
Tony Nguyen, Dave Ertman, regressions
[Cc: +regressions@lists.linux.dev]
Am 11.01.24 um 09:26 schrieb Jaroslav Pulchart:
>>
>> On 1/8/2024 2:49 AM, Jaroslav Pulchart wrote:
>>> Hello
>>
>> First, thank you for your work trying to chase this!
>>
>>>
>>> I would like to report a regression triggered by recent change in
>>> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
>>> bisected and the regression is triggered by
>>> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
>>> check for SRIOV and LAG" commit and originally reported as part of
>>> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
>>> thread.
>>
>> I think that's a bad bisect. There is no reason I could understand for
>> that change to cause a continuous or large leak, it really doesn't make
>> any sense. Reverting it consistently helps? You're not just rewinding
>> the tree back to that point, right? just running 6.6.9 without that
>> patch? (sorry for being pedantic, just trying to be certain)
>>
>
> Reverting just the single bisected commit continuously helps for >=
> 6.6.9 and as well for current 6.7.
> We cannot use any new linux kernel without reverting it due to this
> extra memory utilization.
>
>>
>>>> However, after the following patch we see that more NUMA nodes have
>>>> such a low amount of memory and that is causing constant reclaiming
>>>> of memory because it looks like something inside of the kernel ate all
>>>> the memory. This is right after the start of the system as well.
>>>
>>> I'm reporting it here as it is a different problem than the original
>>> thread. The commit introduces a low memory problem per each numa node
>>> of the first socket (node0 .. node3 in our case) and cause constant
>>> kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
>>> memory issue is nicely visible in "numastat -m", see attached files:
>>> * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
>>> * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
>>> the server "is fresh" (after reboot), without running any application load.
>>
>> OK, so the initial allocations of your system is running your system out
>> of memory.
>>
>> Are you running jumbo frames on your ethernet interfaces?
>>
>
> Yes, we are (MTU 9000).
>
>> Do you have /proc/slabinfo output from working/non-working boot?
>>
>
> Yes, I have a complete sos report so I can pick-up files from there.
> See attached
> slabinfo.vanila (non-working)
> slabinfo.reverted (working)
>
>>>
>>> $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt
>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>> 2756.89 2754.86 100.39 2278.43 < ice
>>> fix is reverted, we have ~2GB free per numa, except one, like before
>>> == no issue
>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>> 3551.29 1530.52 2212.04 3488.09
>>> ...
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>> 127.52 66.49 120.23 263.47 <
>>
>>
>>> ice fix is present, we see just few MB free per each node, this will
>>> cause kswapd utilization!
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>> 3322.18 3134.47 195.55 879.17
>>> ...
>>>
>>> If you have some hints on how to debug what is actually occupying all
>>> that memory and some fix of the problem will be nice. We can provide
>>> testing and more reports if needed to analyze the issue. We reverted
>>> the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
>>> till we know a proper fix.
>>
>> My first suspicion is that we're contributing to the problem by running
>> out of receive descriptors memory.
>>
>> Can we see the ethtool -S stats from the freshly booted system that's
>> running out of memory or doing OOM? Also, all the standard debugging
>> info (at least once please), devlink dev info, any other configuration
>> specifics? What networking config (bonding? anything else?)
>>
>
> The system is not in OOM, it starts to continuously utilize four
> kswapd0-4 of each numa node from the first CPU socket processes (each
> at 100% and all doing swap in/out) after the system start to be used
> by application due to "low memory".
>
> We have two 25G 2P E810-XXV Adapters. The first port of each (em1 +
> p3p1) is connected and they're bonded in LACP. Second ports (em2 and
> p3p2) are unused.
>
> See attached file for working:
> ethtool_-S_em1.reverted
> ethtool_-S_em2.reverted
> ethtool_-S_p3p1.reverted
> ethtool_-S_p3p2.reverted
>
> See attached file for non-working:
> ethtool_-S_em1.vanila
> ethtool_-S_em2.vanila
> ethtool_-S_p3p1.vanila
> ethtool_-S_p3p2.vanila
>
>
>> Do you have a bugzilla.kernel.org bug yet where you can upload larger
>> files like dmesg and others?
>
> I do not have yet, I will create a new one and ping you then.
>
>>
>> Also, I'm curious if your problem goes away if you change / reduce the
>> number of queues per port. use ethtool -L eth0 combined 4 ?
>>
>
> I will try and give you feedback soon.
>
>> You also said something about reproducing when launching / destroying
>> virtual machines with VF passthrough?
>
> The memory usage is there from boot without running any VMs. The issue
> is that the host has low memory for self and it starts to use kswapd
> when we start to use it by starting vms.
>
>>
>> Can you reproduce the issue without starting qemu (just doing bare-metal
>> SR-IOV instance creation/destruction via
>> /sys/class/net/eth0/device/sriov_numvfs ?)
>>
>
> Yes we can reproduce it without qemu running, the extra memory usage
> is from the beginning after boot, not depending on any running VM.
>
> We do not use SR-IOV.
>
>> Thanks
>
> Thanks,
> Jaroslav Pulchart
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-10 18:07 ` Jesse Brandeburg
2024-01-11 8:26 ` Jaroslav Pulchart
@ 2024-01-14 12:05 ` Igor Raits
1 sibling, 0 replies; 23+ messages in thread
From: Igor Raits @ 2024-01-14 12:05 UTC (permalink / raw)
To: Jesse Brandeburg
Cc: intel-wired-lan, Jaroslav Pulchart, Tony Nguyen, Daniel Secik,
Dave Ertman
Hi Jesse,
On Wed, Jan 10, 2024 at 7:08 PM Jesse Brandeburg
<jesse.brandeburg@intel.com> wrote:
>
> Also, I'm curious if your problem goes away if you change / reduce the
> number of queues per port. use ethtool -L eth0 combined 4 ?
I've tried a similar thing out on our servers and it does reclaim a
lot of memory.
# nodestats | awk '/MemFree:/ { for (i = 2; i <= NF; i++) total += $i
} END { print "Total MemFree: " total " MiB" }'
Total MemFree: 53934 MiB
# for i in p3p2 em2; do ethtool -L $i combined 2; done # <--- these
are the ports we don't use at all (they are DOWN, not part of any LAG,
etc.)
# nodestats | awk '/MemFree:/ { for (i = 2; i <= NF; i++) total += $i
} END { print "Total MemFree: " total " MiB" }'
Total MemFree: 55279 MiB
# echo "Hey, here is my 1.4GiB"
# for i in p3p1 em1; do ethtool -L $i combined 2; done # <--- these
are the ports that we do use
# nodestats | awk '/MemFree:/ { for (i = 2; i <= NF; i++) total += $i
} END { print "Total MemFree: " total " MiB" }'
Total MemFree: 58371 MiB
# echo "Hey, here is another 3GiB"
P.S> I've done these tests on a slightly different server that has
more memory (hence more HPs and different amounts of memory per NUMA)
but we have the same problem on both setups.
Maybe an important point, we set irq affinity to one specific NUMA as
that was increasing performance when we did tests so maybe for us the
better setup would be to set up combined 6 for ports that we use and
combined 2 for ports that we do not use.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-08 10:49 [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping Jaroslav Pulchart
2024-01-10 6:45 ` Jaroslav Pulchart
2024-01-10 18:07 ` Jesse Brandeburg
@ 2024-01-19 10:23 ` Linux regression tracking (Thorsten Leemhuis)
2 siblings, 0 replies; 23+ messages in thread
From: Linux regression tracking (Thorsten Leemhuis) @ 2024-01-19 10:23 UTC (permalink / raw)
To: intel-wired-lan, Linux kernel regressions list
[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]
On 08.01.24 11:49, Jaroslav Pulchart wrote:
>
> I would like to report a regression triggered by recent change in
> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
> bisected and the regression is triggered by
> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
> check for SRIOV and LAG" commit and originally reported as part of
> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
> thread.
Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:
#regzbot ^introduced 4d50fcdc2476ee
#regzbot title net: ice: extra memory consumption and cause continous
kswapd* usage and continuous swapping
#regzbot ignore-activity
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-11 8:26 ` Jaroslav Pulchart
2024-01-12 10:23 ` Paul Menzel
@ 2024-01-24 14:29 ` Linux regression tracking (Thorsten Leemhuis)
2024-01-24 16:30 ` Paul Menzel
2024-02-01 14:57 ` Jakub Kicinski
1 sibling, 2 replies; 23+ messages in thread
From: Linux regression tracking (Thorsten Leemhuis) @ 2024-01-24 14:29 UTC (permalink / raw)
To: Jaroslav Pulchart, Jesse Brandeburg
Cc: Tony Nguyen, Igor Raits, Daniel Secik, intel-wired-lan,
Dave Ertman
Hi, Thorsten here, the Linux kernel's regression tracker.
On 11.01.24 09:26, Jaroslav Pulchart wrote:
>> On 1/8/2024 2:49 AM, Jaroslav Pulchart wrote:
>> First, thank you for your work trying to chase this!
>>> I would like to report a regression triggered by recent change in
>>> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
>>> bisected and the regression is triggered by
>>> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
>>> check for SRIOV and LAG" commit and originally reported as part of
>>> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
>>> thread.
>>
>> I think that's a bad bisect. There is no reason I could understand for
>> that change to cause a continuous or large leak, it really doesn't make
>> any sense. Reverting it consistently helps? You're not just rewinding
>> the tree back to that point, right? just running 6.6.9 without that
>> patch? (sorry for being pedantic, just trying to be certain)
>
> Reverting just the single bisected commit continuously helps for >=
> 6.6.9 and as well for current 6.7.
> We cannot use any new linux kernel without reverting it due to this
> extra memory utilization.
Quick query: what's the status wrt to this regression? Looks like
nothing happened in the past week.
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
#regzbot poke
>>>> However, after the following patch we see that more NUMA nodes have
>>>> such a low amount of memory and that is causing constant reclaiming
>>>> of memory because it looks like something inside of the kernel ate all
>>>> the memory. This is right after the start of the system as well.
>>>
>>> I'm reporting it here as it is a different problem than the original
>>> thread. The commit introduces a low memory problem per each numa node
>>> of the first socket (node0 .. node3 in our case) and cause constant
>>> kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
>>> memory issue is nicely visible in "numastat -m", see attached files:
>>> * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
>>> * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
>>> the server "is fresh" (after reboot), without running any application load.
>>
>> OK, so the initial allocations of your system is running your system out
>> of memory.
>>
>> Are you running jumbo frames on your ethernet interfaces?
>>
>
> Yes, we are (MTU 9000).
>
>> Do you have /proc/slabinfo output from working/non-working boot?
>>
>
> Yes, I have a complete sos report so I can pick-up files from there.
> See attached
> slabinfo.vanila (non-working)
> slabinfo.reverted (working)
>
>>>
>>> $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt
>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>> 2756.89 2754.86 100.39 2278.43 < ice
>>> fix is reverted, we have ~2GB free per numa, except one, like before
>>> == no issue
>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>> 3551.29 1530.52 2212.04 3488.09
>>> ...
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>> 127.52 66.49 120.23 263.47 <
>>
>>
>>> ice fix is present, we see just few MB free per each node, this will
>>> cause kswapd utilization!
>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>> 3322.18 3134.47 195.55 879.17
>>> ...
>>>
>>> If you have some hints on how to debug what is actually occupying all
>>> that memory and some fix of the problem will be nice. We can provide
>>> testing and more reports if needed to analyze the issue. We reverted
>>> the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
>>> till we know a proper fix.
>>
>> My first suspicion is that we're contributing to the problem by running
>> out of receive descriptors memory.
>>
>> Can we see the ethtool -S stats from the freshly booted system that's
>> running out of memory or doing OOM? Also, all the standard debugging
>> info (at least once please), devlink dev info, any other configuration
>> specifics? What networking config (bonding? anything else?)
>>
>
> The system is not in OOM, it starts to continuously utilize four
> kswapd0-4 of each numa node from the first CPU socket processes (each
> at 100% and all doing swap in/out) after the system start to be used
> by application due to "low memory".
>
> We have two 25G 2P E810-XXV Adapters. The first port of each (em1 +
> p3p1) is connected and they're bonded in LACP. Second ports (em2 and
> p3p2) are unused.
>
> See attached file for working:
> ethtool_-S_em1.reverted
> ethtool_-S_em2.reverted
> ethtool_-S_p3p1.reverted
> ethtool_-S_p3p2.reverted
>
> See attached file for non-working:
> ethtool_-S_em1.vanila
> ethtool_-S_em2.vanila
> ethtool_-S_p3p1.vanila
> ethtool_-S_p3p2.vanila
>
>
>> Do you have a bugzilla.kernel.org bug yet where you can upload larger
>> files like dmesg and others?
>
> I do not have yet, I will create a new one and ping you then.
>
>>
>> Also, I'm curious if your problem goes away if you change / reduce the
>> number of queues per port. use ethtool -L eth0 combined 4 ?
>>
>
> I will try and give you feedback soon.
>
>> You also said something about reproducing when launching / destroying
>> virtual machines with VF passthrough?
>
> The memory usage is there from boot without running any VMs. The issue
> is that the host has low memory for self and it starts to use kswapd
> when we start to use it by starting vms.
>
>>
>> Can you reproduce the issue without starting qemu (just doing bare-metal
>> SR-IOV instance creation/destruction via
>> /sys/class/net/eth0/device/sriov_numvfs ?)
>>
>
> Yes we can reproduce it without qemu running, the extra memory usage
> is from the beginning after boot, not depending on any running VM.
>
> We do not use SR-IOV.
>
>> Thanks
>
> Thanks,
> Jaroslav Pulchart
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-24 14:29 ` Linux regression tracking (Thorsten Leemhuis)
@ 2024-01-24 16:30 ` Paul Menzel
2024-02-01 14:57 ` Jakub Kicinski
1 sibling, 0 replies; 23+ messages in thread
From: Paul Menzel @ 2024-01-24 16:30 UTC (permalink / raw)
To: Jaroslav Pulchart, regressions, Jesse Brandeburg
Cc: intel-wired-lan, Igor Raits, Daniel Secik, Tony Nguyen,
Dave Ertman
Dear Jaroslav,
Am 24.01.24 um 15:29 schrieb Linux regression tracking (Thorsten Leemhuis):
> On 11.01.24 09:26, Jaroslav Pulchart wrote:
>>> On 1/8/2024 2:49 AM, Jaroslav Pulchart wrote:
>>> First, thank you for your work trying to chase this!
>>>> I would like to report a regression triggered by recent change in
>>>> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
>>>> bisected and the regression is triggered by
>>>> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
>>>> check for SRIOV and LAG" commit and originally reported as part of
>>>> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
>>>> thread.
>>>
>>> I think that's a bad bisect. There is no reason I could understand for
>>> that change to cause a continuous or large leak, it really doesn't make
>>> any sense. Reverting it consistently helps? You're not just rewinding
>>> the tree back to that point, right? just running 6.6.9 without that
>>> patch? (sorry for being pedantic, just trying to be certain)
>>
>> Reverting just the single bisected commit continuously helps for >=
>> 6.6.9 and as well for current 6.7.
>> We cannot use any new linux kernel without reverting it due to this
>> extra memory utilization.
>
> Quick query: what's the status wrt to this regression? Looks like
> nothing happened in the past week.
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
>
> #regzbot poke
According to Linux’ “no regression rule” [1], I recommend to send in a
revert for the bisected commit.
Kind regards,
Paul
[1]:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-regressions.html
>>>>> However, after the following patch we see that more NUMA nodes have
>>>>> such a low amount of memory and that is causing constant reclaiming
>>>>> of memory because it looks like something inside of the kernel ate all
>>>>> the memory. This is right after the start of the system as well.
>>>>
>>>> I'm reporting it here as it is a different problem than the original
>>>> thread. The commit introduces a low memory problem per each numa node
>>>> of the first socket (node0 .. node3 in our case) and cause constant
>>>> kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
>>>> memory issue is nicely visible in "numastat -m", see attached files:
>>>> * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
>>>> * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
>>>> the server "is fresh" (after reboot), without running any application load.
>>>
>>> OK, so the initial allocations of your system is running your system out
>>> of memory.
>>>
>>> Are you running jumbo frames on your ethernet interfaces?
>>
>> Yes, we are (MTU 9000).
>>
>>> Do you have /proc/slabinfo output from working/non-working boot?
>>
>> Yes, I have a complete sos report so I can pick-up files from there.
>> See attached
>> slabinfo.vanila (non-working)
>> slabinfo.reverted (working)
>>
>>>> $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
>>>> numastat_m-6.6.10_28GB_HP_no_revert.txt
>>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>>> 2756.89 2754.86 100.39 2278.43 < ice
>>>> fix is reverted, we have ~2GB free per numa, except one, like before
>>>> == no issue
>>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>>> 3551.29 1530.52 2212.04 3488.09
>>>> ...
>>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>>> 127.52 66.49 120.23 263.47 <
>>>
>>>
>>>> ice fix is present, we see just few MB free per each node, this will
>>>> cause kswapd utilization!
>>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>>> 3322.18 3134.47 195.55 879.17
>>>> ...
>>>>
>>>> If you have some hints on how to debug what is actually occupying all
>>>> that memory and some fix of the problem will be nice. We can provide
>>>> testing and more reports if needed to analyze the issue. We reverted
>>>> the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
>>>> till we know a proper fix.
>>>
>>> My first suspicion is that we're contributing to the problem by running
>>> out of receive descriptors memory.
>>>
>>> Can we see the ethtool -S stats from the freshly booted system that's
>>> running out of memory or doing OOM? Also, all the standard debugging
>>> info (at least once please), devlink dev info, any other configuration
>>> specifics? What networking config (bonding? anything else?)
>>
>> The system is not in OOM, it starts to continuously utilize four
>> kswapd0-4 of each numa node from the first CPU socket processes (each
>> at 100% and all doing swap in/out) after the system start to be used
>> by application due to "low memory".
>>
>> We have two 25G 2P E810-XXV Adapters. The first port of each (em1 +
>> p3p1) is connected and they're bonded in LACP. Second ports (em2 and
>> p3p2) are unused.
>>
>> See attached file for working:
>> ethtool_-S_em1.reverted
>> ethtool_-S_em2.reverted
>> ethtool_-S_p3p1.reverted
>> ethtool_-S_p3p2.reverted
>>
>> See attached file for non-working:
>> ethtool_-S_em1.vanila
>> ethtool_-S_em2.vanila
>> ethtool_-S_p3p1.vanila
>> ethtool_-S_p3p2.vanila
>>
>>> Do you have a bugzilla.kernel.org bug yet where you can upload larger
>>> files like dmesg and others?
>>
>> I do not have yet, I will create a new one and ping you then.
>>
>>> Also, I'm curious if your problem goes away if you change / reduce the
>>> number of queues per port. use ethtool -L eth0 combined 4 ?
>>
>> I will try and give you feedback soon.
>>
>>> You also said something about reproducing when launching / destroying
>>> virtual machines with VF passthrough?
>>
>> The memory usage is there from boot without running any VMs. The issue
>> is that the host has low memory for self and it starts to use kswapd
>> when we start to use it by starting vms.
>>
>>>
>>> Can you reproduce the issue without starting qemu (just doing bare-metal
>>> SR-IOV instance creation/destruction via
>>> /sys/class/net/eth0/device/sriov_numvfs ?)
>>>
>>
>> Yes we can reproduce it without qemu running, the extra memory usage
>> is from the beginning after boot, not depending on any running VM.
>>
>> We do not use SR-IOV.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-01-24 16:30 ` Paul Menzel
0 siblings, 0 replies; 23+ messages in thread
From: Paul Menzel @ 2024-01-24 16:30 UTC (permalink / raw)
To: Jaroslav Pulchart, regressions, Jesse Brandeburg
Cc: Tony Nguyen, Igor Raits, Daniel Secik, intel-wired-lan,
Dave Ertman
Dear Jaroslav,
Am 24.01.24 um 15:29 schrieb Linux regression tracking (Thorsten Leemhuis):
> On 11.01.24 09:26, Jaroslav Pulchart wrote:
>>> On 1/8/2024 2:49 AM, Jaroslav Pulchart wrote:
>>> First, thank you for your work trying to chase this!
>>>> I would like to report a regression triggered by recent change in
>>>> Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
>>>> bisected and the regression is triggered by
>>>> fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
>>>> check for SRIOV and LAG" commit and originally reported as part of
>>>> https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
>>>> thread.
>>>
>>> I think that's a bad bisect. There is no reason I could understand for
>>> that change to cause a continuous or large leak, it really doesn't make
>>> any sense. Reverting it consistently helps? You're not just rewinding
>>> the tree back to that point, right? just running 6.6.9 without that
>>> patch? (sorry for being pedantic, just trying to be certain)
>>
>> Reverting just the single bisected commit continuously helps for >=
>> 6.6.9 and as well for current 6.7.
>> We cannot use any new linux kernel without reverting it due to this
>> extra memory utilization.
>
> Quick query: what's the status wrt to this regression? Looks like
> nothing happened in the past week.
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
>
> #regzbot poke
According to Linux’ “no regression rule” [1], I recommend to send in a
revert for the bisected commit.
Kind regards,
Paul
[1]:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-regressions.html
>>>>> However, after the following patch we see that more NUMA nodes have
>>>>> such a low amount of memory and that is causing constant reclaiming
>>>>> of memory because it looks like something inside of the kernel ate all
>>>>> the memory. This is right after the start of the system as well.
>>>>
>>>> I'm reporting it here as it is a different problem than the original
>>>> thread. The commit introduces a low memory problem per each numa node
>>>> of the first socket (node0 .. node3 in our case) and cause constant
>>>> kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
>>>> memory issue is nicely visible in "numastat -m", see attached files:
>>>> * numastat_m-6.6.10_28GB_HP_ice_revert.txt >= 6.6.9 with reverted ice commit
>>>> * numastat_m-6.6.10_28GB_HP_no_revert.txt >= 6.6.9 vanilla
>>>> the server "is fresh" (after reboot), without running any application load.
>>>
>>> OK, so the initial allocations of your system is running your system out
>>> of memory.
>>>
>>> Are you running jumbo frames on your ethernet interfaces?
>>
>> Yes, we are (MTU 9000).
>>
>>> Do you have /proc/slabinfo output from working/non-working boot?
>>
>> Yes, I have a complete sos report so I can pick-up files from there.
>> See attached
>> slabinfo.vanila (non-working)
>> slabinfo.reverted (working)
>>
>>>> $ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
>>>> numastat_m-6.6.10_28GB_HP_no_revert.txt
>>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>>> 2756.89 2754.86 100.39 2278.43 < ice
>>>> fix is reverted, we have ~2GB free per numa, except one, like before
>>>> == no issue
>>>> numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
>>>> 3551.29 1530.52 2212.04 3488.09
>>>> ...
>>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>>> 127.52 66.49 120.23 263.47 <
>>>
>>>
>>>> ice fix is present, we see just few MB free per each node, this will
>>>> cause kswapd utilization!
>>>> numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
>>>> 3322.18 3134.47 195.55 879.17
>>>> ...
>>>>
>>>> If you have some hints on how to debug what is actually occupying all
>>>> that memory and some fix of the problem will be nice. We can provide
>>>> testing and more reports if needed to analyze the issue. We reverted
>>>> the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
>>>> till we know a proper fix.
>>>
>>> My first suspicion is that we're contributing to the problem by running
>>> out of receive descriptors memory.
>>>
>>> Can we see the ethtool -S stats from the freshly booted system that's
>>> running out of memory or doing OOM? Also, all the standard debugging
>>> info (at least once please), devlink dev info, any other configuration
>>> specifics? What networking config (bonding? anything else?)
>>
>> The system is not in OOM, it starts to continuously utilize four
>> kswapd0-4 of each numa node from the first CPU socket processes (each
>> at 100% and all doing swap in/out) after the system start to be used
>> by application due to "low memory".
>>
>> We have two 25G 2P E810-XXV Adapters. The first port of each (em1 +
>> p3p1) is connected and they're bonded in LACP. Second ports (em2 and
>> p3p2) are unused.
>>
>> See attached file for working:
>> ethtool_-S_em1.reverted
>> ethtool_-S_em2.reverted
>> ethtool_-S_p3p1.reverted
>> ethtool_-S_p3p2.reverted
>>
>> See attached file for non-working:
>> ethtool_-S_em1.vanila
>> ethtool_-S_em2.vanila
>> ethtool_-S_p3p1.vanila
>> ethtool_-S_p3p2.vanila
>>
>>> Do you have a bugzilla.kernel.org bug yet where you can upload larger
>>> files like dmesg and others?
>>
>> I do not have yet, I will create a new one and ping you then.
>>
>>> Also, I'm curious if your problem goes away if you change / reduce the
>>> number of queues per port. use ethtool -L eth0 combined 4 ?
>>
>> I will try and give you feedback soon.
>>
>>> You also said something about reproducing when launching / destroying
>>> virtual machines with VF passthrough?
>>
>> The memory usage is there from boot without running any VMs. The issue
>> is that the host has low memory for self and it starts to use kswapd
>> when we start to use it by starting vms.
>>
>>>
>>> Can you reproduce the issue without starting qemu (just doing bare-metal
>>> SR-IOV instance creation/destruction via
>>> /sys/class/net/eth0/device/sriov_numvfs ?)
>>>
>>
>> Yes we can reproduce it without qemu running, the extra memory usage
>> is from the beginning after boot, not depending on any running VM.
>>
>> We do not use SR-IOV.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-01-24 14:29 ` Linux regression tracking (Thorsten Leemhuis)
@ 2024-02-01 14:57 ` Jakub Kicinski
2024-02-01 14:57 ` Jakub Kicinski
1 sibling, 0 replies; 23+ messages in thread
From: Jakub Kicinski @ 2024-02-01 14:57 UTC (permalink / raw)
To: Jesse Brandeburg, Dave Ertman
Cc: Linux regressions mailing list, Daniel Secik,
Linux regression tracking (Thorsten Leemhuis), Tony Nguyen,
intel-wired-lan, Igor Raits, Jaroslav Pulchart
On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
Leemhuis) wrote:
> >> I think that's a bad bisect. There is no reason I could understand for
> >> that change to cause a continuous or large leak, it really doesn't make
> >> any sense. Reverting it consistently helps? You're not just rewinding
> >> the tree back to that point, right? just running 6.6.9 without that
> >> patch? (sorry for being pedantic, just trying to be certain)
> >
> > Reverting just the single bisected commit continuously helps for >=
> > 6.6.9 and as well for current 6.7.
> > We cannot use any new linux kernel without reverting it due to this
> > extra memory utilization.
>
> Quick query: what's the status wrt to this regression? Looks like
> nothing happened in the past week.
Is someone working on this? Indeed the commit in question looks
harmless but can't argue with the revert helping :S
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-02-01 14:57 ` Jakub Kicinski
0 siblings, 0 replies; 23+ messages in thread
From: Jakub Kicinski @ 2024-02-01 14:57 UTC (permalink / raw)
To: Jesse Brandeburg, Dave Ertman
Cc: Linux regression tracking (Thorsten Leemhuis),
Linux regressions mailing list, Jaroslav Pulchart, Tony Nguyen,
Igor Raits, Daniel Secik, intel-wired-lan
On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
Leemhuis) wrote:
> >> I think that's a bad bisect. There is no reason I could understand for
> >> that change to cause a continuous or large leak, it really doesn't make
> >> any sense. Reverting it consistently helps? You're not just rewinding
> >> the tree back to that point, right? just running 6.6.9 without that
> >> patch? (sorry for being pedantic, just trying to be certain)
> >
> > Reverting just the single bisected commit continuously helps for >=
> > 6.6.9 and as well for current 6.7.
> > We cannot use any new linux kernel without reverting it due to this
> > extra memory utilization.
>
> Quick query: what's the status wrt to this regression? Looks like
> nothing happened in the past week.
Is someone working on this? Indeed the commit in question looks
harmless but can't argue with the revert helping :S
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-02-01 14:57 ` Jakub Kicinski
@ 2024-02-01 17:19 ` Jaroslav Pulchart
-1 siblings, 0 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-02-01 17:19 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Linux regressions mailing list, Daniel Secik, Jesse Brandeburg,
Linux regression tracking (Thorsten Leemhuis), Tony Nguyen,
Dave Ertman, Igor Raits, intel-wired-lan
>
> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
> Leemhuis) wrote:
> > >> I think that's a bad bisect. There is no reason I could understand for
> > >> that change to cause a continuous or large leak, it really doesn't make
> > >> any sense. Reverting it consistently helps? You're not just rewinding
> > >> the tree back to that point, right? just running 6.6.9 without that
> > >> patch? (sorry for being pedantic, just trying to be certain)
> > >
> > > Reverting just the single bisected commit continuously helps for >=
> > > 6.6.9 and as well for current 6.7.
> > > We cannot use any new linux kernel without reverting it due to this
> > > extra memory utilization.
> >
> > Quick query: what's the status wrt to this regression? Looks like
> > nothing happened in the past week.
>
> Is someone working on this? Indeed the commit in question looks
> harmless but can't argue with the revert helping :S
No clue if someone is working on it, however the commit itself is a
trigger of some other issue.
The analysis of my colleague Igor (see previous email) shows the
memory consumption is caused by queues of each ice network interface
(even the unused ones). Our final fix was to lower the queues to 6 for
used interfaces and 2 of unused interfaces manually.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-02-01 17:19 ` Jaroslav Pulchart
0 siblings, 0 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-02-01 17:19 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Jesse Brandeburg, Dave Ertman,
Linux regression tracking (Thorsten Leemhuis),
Linux regressions mailing list, Tony Nguyen, Igor Raits,
Daniel Secik, intel-wired-lan
>
> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
> Leemhuis) wrote:
> > >> I think that's a bad bisect. There is no reason I could understand for
> > >> that change to cause a continuous or large leak, it really doesn't make
> > >> any sense. Reverting it consistently helps? You're not just rewinding
> > >> the tree back to that point, right? just running 6.6.9 without that
> > >> patch? (sorry for being pedantic, just trying to be certain)
> > >
> > > Reverting just the single bisected commit continuously helps for >=
> > > 6.6.9 and as well for current 6.7.
> > > We cannot use any new linux kernel without reverting it due to this
> > > extra memory utilization.
> >
> > Quick query: what's the status wrt to this regression? Looks like
> > nothing happened in the past week.
>
> Is someone working on this? Indeed the commit in question looks
> harmless but can't argue with the revert helping :S
No clue if someone is working on it, however the commit itself is a
trigger of some other issue.
The analysis of my colleague Igor (see previous email) shows the
memory consumption is caused by queues of each ice network interface
(even the unused ones). Our final fix was to lower the queues to 6 for
used interfaces and 2 of unused interfaces manually.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-02-01 17:19 ` Jaroslav Pulchart
@ 2024-02-19 11:29 ` Thorsten Leemhuis
-1 siblings, 0 replies; 23+ messages in thread
From: Thorsten Leemhuis @ 2024-02-19 11:29 UTC (permalink / raw)
To: Jaroslav Pulchart, Jakub Kicinski
Cc: Linux regressions mailing list, Daniel Secik, Jesse Brandeburg,
Tony Nguyen, Dave Ertman, Igor Raits, intel-wired-lan
On 01.02.24 18:19, Jaroslav Pulchart wrote:
>>
>> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
>> Leemhuis) wrote:
>>>>> I think that's a bad bisect. There is no reason I could understand for
>>>>> that change to cause a continuous or large leak, it really doesn't make
>>>>> any sense. Reverting it consistently helps? You're not just rewinding
>>>>> the tree back to that point, right? just running 6.6.9 without that
>>>>> patch? (sorry for being pedantic, just trying to be certain)
>>>>
>>>> Reverting just the single bisected commit continuously helps for >=
>>>> 6.6.9 and as well for current 6.7.
>>>> We cannot use any new linux kernel without reverting it due to this
>>>> extra memory utilization.
>>>
>>> Quick query: what's the status wrt to this regression? Looks like
>>> nothing happened in the past week.
>>
>> Is someone working on this? Indeed the commit in question looks
>> harmless but can't argue with the revert helping :S
>
> No clue if someone is working on it,
Yeah, a quick public status update would be really helpful. And maybe
some debugging tips that might enable Jaroslav to pinpoint the real
problem.
> however the commit itself is a
> trigger of some other issue.
>
> The analysis of my colleague Igor (see previous email) shows the
> memory consumption is caused by queues of each ice network interface
> (even the unused ones). Our final fix was to lower the queues to 6 for
> used interfaces and 2 of unused interfaces manually.
Despite the above allow me to ask: Can you live with that workaround?
Ideally of course this should be fixed, but well, the world sometimes is
a tricky place. :-/
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
#regzbot poke
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-02-19 11:29 ` Thorsten Leemhuis
0 siblings, 0 replies; 23+ messages in thread
From: Thorsten Leemhuis @ 2024-02-19 11:29 UTC (permalink / raw)
To: Jaroslav Pulchart, Jakub Kicinski
Cc: Jesse Brandeburg, Dave Ertman, Linux regressions mailing list,
Tony Nguyen, Igor Raits, Daniel Secik, intel-wired-lan
On 01.02.24 18:19, Jaroslav Pulchart wrote:
>>
>> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
>> Leemhuis) wrote:
>>>>> I think that's a bad bisect. There is no reason I could understand for
>>>>> that change to cause a continuous or large leak, it really doesn't make
>>>>> any sense. Reverting it consistently helps? You're not just rewinding
>>>>> the tree back to that point, right? just running 6.6.9 without that
>>>>> patch? (sorry for being pedantic, just trying to be certain)
>>>>
>>>> Reverting just the single bisected commit continuously helps for >=
>>>> 6.6.9 and as well for current 6.7.
>>>> We cannot use any new linux kernel without reverting it due to this
>>>> extra memory utilization.
>>>
>>> Quick query: what's the status wrt to this regression? Looks like
>>> nothing happened in the past week.
>>
>> Is someone working on this? Indeed the commit in question looks
>> harmless but can't argue with the revert helping :S
>
> No clue if someone is working on it,
Yeah, a quick public status update would be really helpful. And maybe
some debugging tips that might enable Jaroslav to pinpoint the real
problem.
> however the commit itself is a
> trigger of some other issue.
>
> The analysis of my colleague Igor (see previous email) shows the
> memory consumption is caused by queues of each ice network interface
> (even the unused ones). Our final fix was to lower the queues to 6 for
> used interfaces and 2 of unused interfaces manually.
Despite the above allow me to ask: Can you live with that workaround?
Ideally of course this should be fixed, but well, the world sometimes is
a tricky place. :-/
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
#regzbot poke
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-02-19 11:29 ` Thorsten Leemhuis
@ 2024-02-19 11:40 ` Jaroslav Pulchart
-1 siblings, 0 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-02-19 11:40 UTC (permalink / raw)
To: Thorsten Leemhuis
Cc: Linux regressions mailing list, Daniel Secik, Jesse Brandeburg,
Tony Nguyen, Dave Ertman, Jakub Kicinski, Igor Raits,
intel-wired-lan
If the question is for me then my opinion about it is this:
I'm fine with the behaviour of a driver about memory consumption if it
is predictable/described with the possibility to change it from
default values. My understanding of "predictable" is something like
this:
The ICE driver is going to
* Setup 64 queues per each port, not active included.
* Each queue consumes "xxxx MB" amount of kernel memory per each numa node.
example: Two 2 ports Intel NICs using ICE driver will consume ~6GB of
RAM of each NUMA node, please consider changing the defaults values to
avoid OOM :-).
po 19. 2. 2024 v 12:29 odesílatel Thorsten Leemhuis
<regressions@leemhuis.info> napsal:
>
> On 01.02.24 18:19, Jaroslav Pulchart wrote:
> >>
> >> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
> >> Leemhuis) wrote:
> >>>>> I think that's a bad bisect. There is no reason I could understand for
> >>>>> that change to cause a continuous or large leak, it really doesn't make
> >>>>> any sense. Reverting it consistently helps? You're not just rewinding
> >>>>> the tree back to that point, right? just running 6.6.9 without that
> >>>>> patch? (sorry for being pedantic, just trying to be certain)
> >>>>
> >>>> Reverting just the single bisected commit continuously helps for >=
> >>>> 6.6.9 and as well for current 6.7.
> >>>> We cannot use any new linux kernel without reverting it due to this
> >>>> extra memory utilization.
> >>>
> >>> Quick query: what's the status wrt to this regression? Looks like
> >>> nothing happened in the past week.
> >>
> >> Is someone working on this? Indeed the commit in question looks
> >> harmless but can't argue with the revert helping :S
> >
> > No clue if someone is working on it,
>
> Yeah, a quick public status update would be really helpful. And maybe
> some debugging tips that might enable Jaroslav to pinpoint the real
> problem.
>
> > however the commit itself is a
> > trigger of some other issue.
> >
> > The analysis of my colleague Igor (see previous email) shows the
> > memory consumption is caused by queues of each ice network interface
> > (even the unused ones). Our final fix was to lower the queues to 6 for
> > used interfaces and 2 of unused interfaces manually.
>
> Despite the above allow me to ask: Can you live with that workaround?
> Ideally of course this should be fixed, but well, the world sometimes is
> a tricky place. :-/
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
>
> #regzbot poke
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-02-19 11:40 ` Jaroslav Pulchart
0 siblings, 0 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-02-19 11:40 UTC (permalink / raw)
To: Thorsten Leemhuis
Cc: Jakub Kicinski, Jesse Brandeburg, Dave Ertman,
Linux regressions mailing list, Tony Nguyen, Igor Raits,
Daniel Secik, intel-wired-lan
If the question is for me then my opinion about it is this:
I'm fine with the behaviour of a driver about memory consumption if it
is predictable/described with the possibility to change it from
default values. My understanding of "predictable" is something like
this:
The ICE driver is going to
* Setup 64 queues per each port, not active included.
* Each queue consumes "xxxx MB" amount of kernel memory per each numa node.
example: Two 2 ports Intel NICs using ICE driver will consume ~6GB of
RAM of each NUMA node, please consider changing the defaults values to
avoid OOM :-).
po 19. 2. 2024 v 12:29 odesílatel Thorsten Leemhuis
<regressions@leemhuis.info> napsal:
>
> On 01.02.24 18:19, Jaroslav Pulchart wrote:
> >>
> >> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
> >> Leemhuis) wrote:
> >>>>> I think that's a bad bisect. There is no reason I could understand for
> >>>>> that change to cause a continuous or large leak, it really doesn't make
> >>>>> any sense. Reverting it consistently helps? You're not just rewinding
> >>>>> the tree back to that point, right? just running 6.6.9 without that
> >>>>> patch? (sorry for being pedantic, just trying to be certain)
> >>>>
> >>>> Reverting just the single bisected commit continuously helps for >=
> >>>> 6.6.9 and as well for current 6.7.
> >>>> We cannot use any new linux kernel without reverting it due to this
> >>>> extra memory utilization.
> >>>
> >>> Quick query: what's the status wrt to this regression? Looks like
> >>> nothing happened in the past week.
> >>
> >> Is someone working on this? Indeed the commit in question looks
> >> harmless but can't argue with the revert helping :S
> >
> > No clue if someone is working on it,
>
> Yeah, a quick public status update would be really helpful. And maybe
> some debugging tips that might enable Jaroslav to pinpoint the real
> problem.
>
> > however the commit itself is a
> > trigger of some other issue.
> >
> > The analysis of my colleague Igor (see previous email) shows the
> > memory consumption is caused by queues of each ice network interface
> > (even the unused ones). Our final fix was to lower the queues to 6 for
> > used interfaces and 2 of unused interfaces manually.
>
> Despite the above allow me to ask: Can you live with that workaround?
> Ideally of course this should be fixed, but well, the world sometimes is
> a tricky place. :-/
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
>
> #regzbot poke
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-02-19 11:40 ` Jaroslav Pulchart
@ 2024-02-22 9:51 ` Thorsten Leemhuis
-1 siblings, 0 replies; 23+ messages in thread
From: Thorsten Leemhuis @ 2024-02-22 9:51 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: Linux regressions mailing list, Daniel Secik, Jesse Brandeburg,
Tony Nguyen, Dave Ertman, Jakub Kicinski, Igor Raits,
intel-wired-lan
On 19.02.24 12:40, Jaroslav Pulchart wrote:
> If the question is for me then my opinion about it is this:
>
> I'm fine with the behaviour of a driver about memory consumption if it
> is predictable/described with the possibility to change it from
> default values. My understanding of "predictable" is something like
> this:
>
> The ICE driver is going to
> * Setup 64 queues per each port, not active included.
> * Each queue consumes "xxxx MB" amount of kernel memory per each numa node.
> example: Two 2 ports Intel NICs using ICE driver will consume ~6GB of
> RAM of each NUMA node, please consider changing the defaults values to
> avoid OOM :-).
6GB of RAM for each NUMA node? That sounds, well, a whole lot :-D Makes
me also wonder a bit why nobody else reported this (and if you have any
debug option enable in your .config or something like that that which is
rarely used).
Whatever: to me it still feels like this regression is not handled as
Linus would want it, but I'm not totally sure and guess I have to admit
that I'm out of my depth here. I'll let my regression tracking bot
continue monitor this, but will most likely leave things to the network
and driver maintainers from here on unless something changes.
Ciao, Thorsten
> po 19. 2. 2024 v 12:29 odesílatel Thorsten Leemhuis
> <regressions@leemhuis.info> napsal:
>>
>> On 01.02.24 18:19, Jaroslav Pulchart wrote:
>>>>
>>>> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
>>>> Leemhuis) wrote:
>>>>>>> I think that's a bad bisect. There is no reason I could understand for
>>>>>>> that change to cause a continuous or large leak, it really doesn't make
>>>>>>> any sense. Reverting it consistently helps? You're not just rewinding
>>>>>>> the tree back to that point, right? just running 6.6.9 without that
>>>>>>> patch? (sorry for being pedantic, just trying to be certain)
>>>>>>
>>>>>> Reverting just the single bisected commit continuously helps for >=
>>>>>> 6.6.9 and as well for current 6.7.
>>>>>> We cannot use any new linux kernel without reverting it due to this
>>>>>> extra memory utilization.
>>>>>
>>>>> Quick query: what's the status wrt to this regression? Looks like
>>>>> nothing happened in the past week.
>>>>
>>>> Is someone working on this? Indeed the commit in question looks
>>>> harmless but can't argue with the revert helping :S
>>>
>>> No clue if someone is working on it,
>>
>> Yeah, a quick public status update would be really helpful. And maybe
>> some debugging tips that might enable Jaroslav to pinpoint the real
>> problem.
>>
>>> however the commit itself is a
>>> trigger of some other issue.
>>>
>>> The analysis of my colleague Igor (see previous email) shows the
>>> memory consumption is caused by queues of each ice network interface
>>> (even the unused ones). Our final fix was to lower the queues to 6 for
>>> used interfaces and 2 of unused interfaces manually.
>>
>> Despite the above allow me to ask: Can you live with that workaround?
>> Ideally of course this should be fixed, but well, the world sometimes is
>> a tricky place. :-/
>>
>> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>> --
>> Everything you wanna know about Linux kernel regression tracking:
>> https://linux-regtracking.leemhuis.info/about/#tldr
>> If I did something stupid, please tell me, as explained on that page.
>>
>> #regzbot poke
>
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-02-22 9:51 ` Thorsten Leemhuis
0 siblings, 0 replies; 23+ messages in thread
From: Thorsten Leemhuis @ 2024-02-22 9:51 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: Jakub Kicinski, Jesse Brandeburg, Dave Ertman,
Linux regressions mailing list, Tony Nguyen, Igor Raits,
Daniel Secik, intel-wired-lan
On 19.02.24 12:40, Jaroslav Pulchart wrote:
> If the question is for me then my opinion about it is this:
>
> I'm fine with the behaviour of a driver about memory consumption if it
> is predictable/described with the possibility to change it from
> default values. My understanding of "predictable" is something like
> this:
>
> The ICE driver is going to
> * Setup 64 queues per each port, not active included.
> * Each queue consumes "xxxx MB" amount of kernel memory per each numa node.
> example: Two 2 ports Intel NICs using ICE driver will consume ~6GB of
> RAM of each NUMA node, please consider changing the defaults values to
> avoid OOM :-).
6GB of RAM for each NUMA node? That sounds, well, a whole lot :-D Makes
me also wonder a bit why nobody else reported this (and if you have any
debug option enable in your .config or something like that that which is
rarely used).
Whatever: to me it still feels like this regression is not handled as
Linus would want it, but I'm not totally sure and guess I have to admit
that I'm out of my depth here. I'll let my regression tracking bot
continue monitor this, but will most likely leave things to the network
and driver maintainers from here on unless something changes.
Ciao, Thorsten
> po 19. 2. 2024 v 12:29 odesílatel Thorsten Leemhuis
> <regressions@leemhuis.info> napsal:
>>
>> On 01.02.24 18:19, Jaroslav Pulchart wrote:
>>>>
>>>> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
>>>> Leemhuis) wrote:
>>>>>>> I think that's a bad bisect. There is no reason I could understand for
>>>>>>> that change to cause a continuous or large leak, it really doesn't make
>>>>>>> any sense. Reverting it consistently helps? You're not just rewinding
>>>>>>> the tree back to that point, right? just running 6.6.9 without that
>>>>>>> patch? (sorry for being pedantic, just trying to be certain)
>>>>>>
>>>>>> Reverting just the single bisected commit continuously helps for >=
>>>>>> 6.6.9 and as well for current 6.7.
>>>>>> We cannot use any new linux kernel without reverting it due to this
>>>>>> extra memory utilization.
>>>>>
>>>>> Quick query: what's the status wrt to this regression? Looks like
>>>>> nothing happened in the past week.
>>>>
>>>> Is someone working on this? Indeed the commit in question looks
>>>> harmless but can't argue with the revert helping :S
>>>
>>> No clue if someone is working on it,
>>
>> Yeah, a quick public status update would be really helpful. And maybe
>> some debugging tips that might enable Jaroslav to pinpoint the real
>> problem.
>>
>>> however the commit itself is a
>>> trigger of some other issue.
>>>
>>> The analysis of my colleague Igor (see previous email) shows the
>>> memory consumption is caused by queues of each ice network interface
>>> (even the unused ones). Our final fix was to lower the queues to 6 for
>>> used interfaces and 2 of unused interfaces manually.
>>
>> Despite the above allow me to ask: Can you live with that workaround?
>> Ideally of course this should be fixed, but well, the world sometimes is
>> a tricky place. :-/
>>
>> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>> --
>> Everything you wanna know about Linux kernel regression tracking:
>> https://linux-regtracking.leemhuis.info/about/#tldr
>> If I did something stupid, please tell me, as explained on that page.
>>
>> #regzbot poke
>
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
2024-02-22 9:51 ` Thorsten Leemhuis
@ 2024-03-05 13:00 ` Linux regression tracking (Thorsten Leemhuis)
-1 siblings, 0 replies; 23+ messages in thread
From: Linux regression tracking (Thorsten Leemhuis) @ 2024-03-05 13:00 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: Linux regressions mailing list, Daniel Secik, Tony Nguyen,
Dave Ertman, Jakub Kicinski, Igor Raits, intel-wired-lan
On 22.02.24 10:51, Thorsten Leemhuis wrote:
> On 19.02.24 12:40, Jaroslav Pulchart wrote:
>> If the question is for me then my opinion about it is this:
>>
>> I'm fine with the behaviour of a driver about memory consumption if it
>> is predictable/described with the possibility to change it from
>> default values. My understanding of "predictable" is something like
>> this:
>>
>> The ICE driver is going to
>> * Setup 64 queues per each port, not active included.
>> * Each queue consumes "xxxx MB" amount of kernel memory per each numa node.
>> example: Two 2 ports Intel NICs using ICE driver will consume ~6GB of
>> RAM of each NUMA node, please consider changing the defaults values to
>> avoid OOM :-).
> [...]
> Whatever: to me it still feels like this regression is not handled as
> Linus would want it, but I'm not totally sure and guess I have to admit
> that I'm out of my depth here. I'll let my regression tracking bot
> continue monitor this, but will most likely leave things to the network
> and driver maintainers from here on unless something changes.
FWIW, it seems nobody really cares, so I'll strop tracking this issue:
#regzbot inconclusive: might not qualify as a regression
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
>> po 19. 2. 2024 v 12:29 odesílatel Thorsten Leemhuis
>> <regressions@leemhuis.info> napsal:
>>>
>>> On 01.02.24 18:19, Jaroslav Pulchart wrote:
>>>>>
>>>>> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
>>>>> Leemhuis) wrote:
>>>>>>>> I think that's a bad bisect. There is no reason I could understand for
>>>>>>>> that change to cause a continuous or large leak, it really doesn't make
>>>>>>>> any sense. Reverting it consistently helps? You're not just rewinding
>>>>>>>> the tree back to that point, right? just running 6.6.9 without that
>>>>>>>> patch? (sorry for being pedantic, just trying to be certain)
>>>>>>>
>>>>>>> Reverting just the single bisected commit continuously helps for >=
>>>>>>> 6.6.9 and as well for current 6.7.
>>>>>>> We cannot use any new linux kernel without reverting it due to this
>>>>>>> extra memory utilization.
>>>>>>
>>>>>> Quick query: what's the status wrt to this regression? Looks like
>>>>>> nothing happened in the past week.
>>>>>
>>>>> Is someone working on this? Indeed the commit in question looks
>>>>> harmless but can't argue with the revert helping :S
>>>>
>>>> No clue if someone is working on it,
>>>
>>> Yeah, a quick public status update would be really helpful. And maybe
>>> some debugging tips that might enable Jaroslav to pinpoint the real
>>> problem.
>>>
>>>> however the commit itself is a
>>>> trigger of some other issue.
>>>>
>>>> The analysis of my colleague Igor (see previous email) shows the
>>>> memory consumption is caused by queues of each ice network interface
>>>> (even the unused ones). Our final fix was to lower the queues to 6 for
>>>> used interfaces and 2 of unused interfaces manually.
>>>
>>> Despite the above allow me to ask: Can you live with that workaround?
>>> Ideally of course this should be fixed, but well, the world sometimes is
>>> a tricky place. :-/
>>>
>>> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>>> --
>>> Everything you wanna know about Linux kernel regression tracking:
>>> https://linux-regtracking.leemhuis.info/about/#tldr
>>> If I did something stupid, please tell me, as explained on that page.
>>>
>>> #regzbot poke
>>
>>
>
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-03-05 13:00 ` Linux regression tracking (Thorsten Leemhuis)
0 siblings, 0 replies; 23+ messages in thread
From: Linux regression tracking (Thorsten Leemhuis) @ 2024-03-05 13:00 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: Jakub Kicinski, Jesse Brandeburg, Dave Ertman,
Linux regressions mailing list, Tony Nguyen, Igor Raits,
Daniel Secik, intel-wired-lan
On 22.02.24 10:51, Thorsten Leemhuis wrote:
> On 19.02.24 12:40, Jaroslav Pulchart wrote:
>> If the question is for me then my opinion about it is this:
>>
>> I'm fine with the behaviour of a driver about memory consumption if it
>> is predictable/described with the possibility to change it from
>> default values. My understanding of "predictable" is something like
>> this:
>>
>> The ICE driver is going to
>> * Setup 64 queues per each port, not active included.
>> * Each queue consumes "xxxx MB" amount of kernel memory per each numa node.
>> example: Two 2 ports Intel NICs using ICE driver will consume ~6GB of
>> RAM of each NUMA node, please consider changing the defaults values to
>> avoid OOM :-).
> [...]
> Whatever: to me it still feels like this regression is not handled as
> Linus would want it, but I'm not totally sure and guess I have to admit
> that I'm out of my depth here. I'll let my regression tracking bot
> continue monitor this, but will most likely leave things to the network
> and driver maintainers from here on unless something changes.
FWIW, it seems nobody really cares, so I'll strop tracking this issue:
#regzbot inconclusive: might not qualify as a regression
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
>> po 19. 2. 2024 v 12:29 odesílatel Thorsten Leemhuis
>> <regressions@leemhuis.info> napsal:
>>>
>>> On 01.02.24 18:19, Jaroslav Pulchart wrote:
>>>>>
>>>>> On Wed, 24 Jan 2024 15:29:38 +0100 Linux regression tracking (Thorsten
>>>>> Leemhuis) wrote:
>>>>>>>> I think that's a bad bisect. There is no reason I could understand for
>>>>>>>> that change to cause a continuous or large leak, it really doesn't make
>>>>>>>> any sense. Reverting it consistently helps? You're not just rewinding
>>>>>>>> the tree back to that point, right? just running 6.6.9 without that
>>>>>>>> patch? (sorry for being pedantic, just trying to be certain)
>>>>>>>
>>>>>>> Reverting just the single bisected commit continuously helps for >=
>>>>>>> 6.6.9 and as well for current 6.7.
>>>>>>> We cannot use any new linux kernel without reverting it due to this
>>>>>>> extra memory utilization.
>>>>>>
>>>>>> Quick query: what's the status wrt to this regression? Looks like
>>>>>> nothing happened in the past week.
>>>>>
>>>>> Is someone working on this? Indeed the commit in question looks
>>>>> harmless but can't argue with the revert helping :S
>>>>
>>>> No clue if someone is working on it,
>>>
>>> Yeah, a quick public status update would be really helpful. And maybe
>>> some debugging tips that might enable Jaroslav to pinpoint the real
>>> problem.
>>>
>>>> however the commit itself is a
>>>> trigger of some other issue.
>>>>
>>>> The analysis of my colleague Igor (see previous email) shows the
>>>> memory consumption is caused by queues of each ice network interface
>>>> (even the unused ones). Our final fix was to lower the queues to 6 for
>>>> used interfaces and 2 of unused interfaces manually.
>>>
>>> Despite the above allow me to ask: Can you live with that workaround?
>>> Ideally of course this should be fixed, but well, the world sometimes is
>>> a tricky place. :-/
>>>
>>> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>>> --
>>> Everything you wanna know about Linux kernel regression tracking:
>>> https://linux-regtracking.leemhuis.info/about/#tldr
>>> If I did something stupid, please tell me, as explained on that page.
>>>
>>> #regzbot poke
>>
>>
>
>
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2024-03-05 13:01 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-01-08 10:49 [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping Jaroslav Pulchart
2024-01-10 6:45 ` Jaroslav Pulchart
2024-01-10 18:07 ` Jesse Brandeburg
2024-01-11 8:26 ` Jaroslav Pulchart
2024-01-12 10:23 ` Paul Menzel
2024-01-12 10:23 ` Paul Menzel
2024-01-24 14:29 ` Linux regression tracking (Thorsten Leemhuis)
2024-01-24 16:30 ` Paul Menzel
2024-01-24 16:30 ` Paul Menzel
2024-02-01 14:57 ` Jakub Kicinski
2024-02-01 14:57 ` Jakub Kicinski
2024-02-01 17:19 ` Jaroslav Pulchart
2024-02-01 17:19 ` Jaroslav Pulchart
2024-02-19 11:29 ` Thorsten Leemhuis
2024-02-19 11:29 ` Thorsten Leemhuis
2024-02-19 11:40 ` Jaroslav Pulchart
2024-02-19 11:40 ` Jaroslav Pulchart
2024-02-22 9:51 ` Thorsten Leemhuis
2024-02-22 9:51 ` Thorsten Leemhuis
2024-03-05 13:00 ` Linux regression tracking (Thorsten Leemhuis)
2024-03-05 13:00 ` Linux regression tracking (Thorsten Leemhuis)
2024-01-14 12:05 ` Igor Raits
2024-01-19 10:23 ` Linux regression tracking (Thorsten Leemhuis)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.