All of lore.kernel.org
 help / color / mirror / Atom feed
* [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping
@ 2024-01-08 10:49 Jaroslav Pulchart
  2024-01-10  6:45 ` Jaroslav Pulchart
                   ` (2 more replies)
  0 siblings, 3 replies; 23+ messages in thread
From: Jaroslav Pulchart @ 2024-01-08 10:49 UTC (permalink / raw)
  To: Jesse Brandeburg, Tony Nguyen, intel-wired-lan, Dave Ertman
  Cc: Igor Raits, Daniel Secik

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

Hello

I would like to report a regression triggered by recent change in
Intel ICE Ethernet driver in the 6.6.9 linux kernel. The problem was
bisected and the regression is triggered by
fc4d6d136d42fab207b3ce20a8ebfd61a13f931f "ice: alter feature support
check for SRIOV and LAG" commit and originally reported as part of
https://lore.kernel.org/linux-mm/CAK8fFZ4DY+GtBA40Pm7Nn5xCHy+51w3sfxPqkqpqakSXYyX+Wg@mail.gmail.com/T/#m5217c62beb03b3bc75d7dd4b1d9bab64a3e68826
thread.

> However, after the following patch we see that more NUMA nodes have
> such a low amount of memory and  that is causing constant reclaiming
> of memory because it looks like something inside of the kernel ate all
> the memory. This is right after the start of the system as well.

 I'm reporting it here as it is a different problem than the original
thread. The commit introduces a low memory problem per each numa node
of the first socket (node0 .. node3 in our case) and cause constant
kswapd* 100% CPU usage. See attached 6.6.9-kswapd_usage.png. The low
memory issue is nicely visible in "numastat -m", see attached files:
* numastat_m-6.6.10_28GB_HP_ice_revert.txt   >= 6.6.9 with reverted ice commit
* numastat_m-6.6.10_28GB_HP_no_revert.txt    >= 6.6.9 vanilla
the server "is fresh" (after reboot), without running any application load.

$ grep MemFree numastat_m-6.6.10_28GB_HP_ice_revert.txt
numastat_m-6.6.10_28GB_HP_no_revert.txt
numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
2756.89         2754.86          100.39         2278.43         < ice
fix is reverted, we have ~2GB free per numa, except one, like before
== no issue
numastat_m-6.6.10_28GB_HP_ice_revert.txt:MemFree
3551.29         1530.52         2212.04         3488.09
...
numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
127.52           66.49          120.23          263.47               <
ice fix is present, we see just few MB free per each node, this will
cause kswapd utilization!
numastat_m-6.6.10_28GB_HP_no_revert.txt:MemFree
3322.18         3134.47          195.55          879.17
...

If you have some hints on how to debug what is actually occupying all
that memory and some fix of the problem will be nice. We can provide
testing and more reports if needed to analyze the issue. We reverted
the commit fc4d6d136d42fab207b3ce20a8ebfd61a13f931f as a workaround
till we know a proper fix.

Best regards,
Jaroslav Pulchart

[-- Attachment #2: numastat_m-6.6.10_28GB_HP_ice_revert.txt --]
[-- Type: text/plain, Size: 6633 bytes --]

                          Node 0          Node 1          Node 2          Node 3
                 --------------- --------------- --------------- ---------------
MemTotal                32264.16        32701.87        32701.88        32686.87
MemFree                  2756.89         2754.86          100.39         2278.43
MemUsed                 29507.27        29947.01        32601.49        30408.44
Active                    267.38          204.00           98.88          567.17
Inactive                   22.58           10.24            3.62           38.77
Active(anon)              229.16          199.68           69.59          419.21
Inactive(anon)              0.00            0.00            0.00            0.00
Active(file)               38.22            4.32           29.29          147.96
Inactive(file)             22.58           10.24            3.62           38.77
Unevictable                10.79            0.32            0.01            2.87
Mlocked                    10.79            0.32            0.01            2.87
Dirty                       0.02            0.01            0.01            0.01
Writeback                   0.00            0.00            0.00            0.00
FilePages                  71.67           14.80           33.34          190.00
Mapped                     23.41            1.25           18.92          145.39
AnonPages                 211.82          187.18           68.89          400.52
Shmem                       0.36            0.24            0.41            0.54
KernelStack                 2.64            1.70            2.10            3.50
PageTables                  1.09            0.87            0.45            2.71
NFS_Unstable                0.00            0.00            0.00            0.00
Bounce                      0.00            0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00            0.00
Slab                       50.88           32.03           20.61           88.15
SReclaimable               11.10            5.93            3.02           14.20
SUnreclaim                 39.78           26.10           17.60           73.95
AnonHugePages              40.00           52.00           36.00           82.00
ShmemHugePages              0.00            0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00            0.00
HugePages_Total         28672.00        28672.00        28672.00        28672.00
HugePages_Free          28672.00        28672.00        28672.00        28672.00
HugePages_Surp              0.00            0.00            0.00            0.00
KReclaimable               11.10            5.93            3.02           14.20

                          Node 4          Node 5          Node 6          Node 7
                 --------------- --------------- --------------- ---------------
MemTotal                32701.88        32701.87        32659.87        32696.75
MemFree                  3551.29         1530.52         2212.04         3488.09
MemUsed                 29150.59        31171.35        30447.83        29208.66
Active                    287.19          195.57          257.13          181.00
Inactive                   29.21           48.47           81.34          170.16
Active(anon)              251.84          138.33          205.82          122.58
Inactive(anon)              0.00            0.00            0.00            0.00
Active(file)               35.35           57.25           51.31           58.42
Inactive(file)             29.21           48.47           81.34          170.16
Unevictable                19.86           23.11          291.23           11.16
Mlocked                    19.86           23.11          291.23           11.16
Dirty                       0.02            0.02            0.02            0.01
Writeback                   0.00            0.00            0.00            0.00
FilePages                  79.42          116.06          136.51          233.12
Mapped                     37.59           47.25           40.64           26.11
AnonPages                 250.75          141.84          487.42          125.68
Shmem                       0.49            8.59            0.50            0.23
KernelStack                 2.35            4.29            3.48            1.84
PageTables                  1.71            1.62            2.13            0.97
NFS_Unstable                0.00            0.00            0.00            0.00
Bounce                      0.00            0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00            0.00
Slab                       41.12           77.15           48.36           33.94
SReclaimable               16.09           15.53           12.07            6.84
SUnreclaim                 25.04           61.62           36.29           27.11
AnonHugePages             170.00           92.00          184.00           80.00
ShmemHugePages              0.00            0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00            0.00
HugePages_Total         28672.00        28672.00        28672.00        28672.00
HugePages_Free          28672.00        28672.00        28672.00        28672.00
HugePages_Surp              0.00            0.00            0.00            0.00
KReclaimable               16.09           15.53           12.07            6.84

                           Total
                 ---------------
MemTotal               261115.14
MemFree                 18672.50
MemUsed                242442.64
Active                   2058.32
Inactive                  404.39
Active(anon)             1636.21
Inactive(anon)              0.00
Active(file)              422.12
Inactive(file)            404.39
Unevictable               359.34
Mlocked                   359.34
Dirty                       0.12
Writeback                   0.00
FilePages                 874.93
Mapped                    340.57
AnonPages                1874.10
Shmem                      11.36
KernelStack                21.91
PageTables                 11.55
NFS_Unstable                0.00
Bounce                      0.00
WritebackTmp                0.00
Slab                      392.25
SReclaimable               84.77
SUnreclaim                307.48
AnonHugePages             736.00
ShmemHugePages              0.00
ShmemPmdMapped              0.00
HugePages_Total        229376.00
HugePages_Free         229376.00
HugePages_Surp              0.00
KReclaimable               84.77


[-- Attachment #3: 6.6.9-kswapd_usage.png --]
[-- Type: image/png, Size: 189979 bytes --]

[-- Attachment #4: numastat_m-6.6.10_28GB_HP_no_revert.txt --]
[-- Type: text/plain, Size: 6633 bytes --]

                          Node 0          Node 1          Node 2          Node 3
                 --------------- --------------- --------------- ---------------
MemTotal                32264.16        32701.87        32701.88        32686.87
MemFree                   127.52           66.49          120.23          263.47
MemUsed                 32136.64        32635.38        32581.64        32423.40
Active                     23.83           59.13           30.02           41.47
Inactive                   19.67           31.62           43.53          103.41
Active(anon)               15.28           52.11           18.29           28.68
Inactive(anon)             18.08           25.77           26.57            7.29
Active(file)                8.55            7.02           11.73           12.79
Inactive(file)              1.59            5.85           16.96           96.12
Unevictable                 1.94           12.32            0.50           19.13
Mlocked                     1.94           12.32            0.50           19.13
Dirty                       0.00            0.00            0.01            0.00
Writeback                   0.00            0.00            0.00            0.00
FilePages                  10.28           12.90           29.07          121.34
Mapped                      6.10            4.62            5.55           26.80
AnonPages                  34.06           90.17           43.08           39.95
Shmem                       0.12            0.03            0.21            0.05
KernelStack                 1.92            2.16            2.08            2.88
PageTables                  0.34            0.62            0.30            1.97
NFS_Unstable                0.00            0.00            0.00            0.00
Bounce                      0.00            0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00            0.00
Slab                       42.06           29.30           35.05          166.25
SReclaimable               12.12            4.18            8.05           48.11
SUnreclaim                 29.94           25.12           27.00          118.14
AnonHugePages              12.00           52.00           18.00            8.00
ShmemHugePages              0.00            0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00            0.00
HugePages_Total         28672.00        28672.00        28672.00        28672.00
HugePages_Free          28672.00        28672.00        28672.00        28672.00
HugePages_Surp              0.00            0.00            0.00            0.00
KReclaimable               12.12            4.18            8.05           48.11

                          Node 4          Node 5          Node 6          Node 7
                 --------------- --------------- --------------- ---------------
MemTotal                32701.88        32659.86        32701.88        32696.75
MemFree                  3322.18         3134.47          195.55          879.17
MemUsed                 29379.69        29525.39        32506.32        31817.58
Active                    350.11          460.66          424.54          433.99
Inactive                  125.39          159.05         2362.56         2393.70
Active(anon)              250.39          370.38          282.00          353.43
Inactive(anon)              0.00            0.25            0.29            0.11
Active(file)               99.73           90.29          142.53           80.56
Inactive(file)            125.39          158.80         2362.28         2393.59
Unevictable                12.24           21.84          295.76            3.39
Mlocked                    12.24           21.84          295.76            3.39
Dirty                       0.00            0.02            0.01            0.01
Writeback                   0.00            0.00            0.00            0.00
FilePages                 229.27          264.37         2519.70         2476.24
Mapped                     62.79           92.60           94.86           51.39
AnonPages                 254.16          372.37          553.12          354.69
Shmem                       0.33            0.52            8.82            0.64
KernelStack                 2.49            2.71            4.30            3.56
PageTables                  1.38            2.01            2.67            2.25
NFS_Unstable                0.00            0.00            0.00            0.00
Bounce                      0.00            0.00            0.00            0.00
WritebackTmp                0.00            0.00            0.00            0.00
Slab                      103.72           73.49          104.67          194.87
SReclaimable               62.41           20.92           69.53          145.89
SUnreclaim                 41.31           52.57           35.14           48.98
AnonHugePages             122.00           90.00          176.00          182.00
ShmemHugePages              0.00            0.00            0.00            0.00
ShmemPmdMapped              0.00            0.00            0.00            0.00
HugePages_Total         28672.00        28672.00        28672.00        28672.00
HugePages_Free          28672.00        28672.00        28672.00        28672.00
HugePages_Surp              0.00            0.00            0.00            0.00
KReclaimable               62.41           20.92           69.53          145.89

                           Total
                 ---------------
MemTotal               261115.14
MemFree                  8109.09
MemUsed                253006.05
Active                   1823.75
Inactive                 5238.92
Active(anon)             1370.56
Inactive(anon)             78.36
Active(file)              453.19
Inactive(file)           5160.56
Unevictable               367.12
Mlocked                   367.12
Dirty                       0.05
Writeback                   0.00
FilePages                5663.17
Mapped                    344.71
AnonPages                1741.59
Shmem                      10.73
KernelStack                22.09
PageTables                 11.55
NFS_Unstable                0.00
Bounce                      0.00
WritebackTmp                0.00
Slab                      749.42
SReclaimable              371.21
SUnreclaim                378.20
AnonHugePages             660.00
ShmemHugePages              0.00
ShmemPmdMapped              0.00
HugePages_Total        229376.00
HugePages_Free         229376.00
HugePages_Surp              0.00
KReclaimable              371.21


^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2024-03-05 13:01 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-01-08 10:49 [Intel-wired-lan] [REGRESSION] Intel ICE Ethernet driver in linux >= 6.6.9 triggers extra memory consumption and cause continous kswapd* usage and continuous swapping Jaroslav Pulchart
2024-01-10  6:45 ` Jaroslav Pulchart
2024-01-10 18:07 ` Jesse Brandeburg
2024-01-11  8:26   ` Jaroslav Pulchart
2024-01-12 10:23     ` Paul Menzel
2024-01-12 10:23       ` Paul Menzel
2024-01-24 14:29     ` Linux regression tracking (Thorsten Leemhuis)
2024-01-24 16:30       ` Paul Menzel
2024-01-24 16:30         ` Paul Menzel
2024-02-01 14:57       ` Jakub Kicinski
2024-02-01 14:57         ` Jakub Kicinski
2024-02-01 17:19         ` Jaroslav Pulchart
2024-02-01 17:19           ` Jaroslav Pulchart
2024-02-19 11:29           ` Thorsten Leemhuis
2024-02-19 11:29             ` Thorsten Leemhuis
2024-02-19 11:40             ` Jaroslav Pulchart
2024-02-19 11:40               ` Jaroslav Pulchart
2024-02-22  9:51               ` Thorsten Leemhuis
2024-02-22  9:51                 ` Thorsten Leemhuis
2024-03-05 13:00                 ` Linux regression tracking (Thorsten Leemhuis)
2024-03-05 13:00                   ` Linux regression tracking (Thorsten Leemhuis)
2024-01-14 12:05   ` Igor Raits
2024-01-19 10:23 ` Linux regression tracking (Thorsten Leemhuis)

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.