* [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 2/5] controllers/memcg: accept range of usage_in_bytes Krzysztof Kozlowski
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
Several Linux kernel versions report higher max_usage_in_bytes than
expected size of 1024 pages. For example v5.4, v5.8, v5.10
and 5.13.0-rc5:
memcg_max_usage_in_bytes_test 1 TINFO: Test memory.max_usage_in_bytes
memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 1393215
memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1393215
memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4325376, 4194304 expected
It seems that recent Linux kernel caches the statistics more
aggressively (especially on multi-CPU systems) and the batch updates of
32 pages are visible in usage_in_bytes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../controllers/memcg/functional/memcg_lib.sh | 20 +++++++++++++++++--
.../memcg_max_usage_in_bytes_test.sh | 3 ++-
2 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
index d9bb6d94324d..4c47e63745ff 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
@@ -27,6 +27,15 @@ fi
# Post 4.16 kernel updates stat in batch (> 32 pages) every time
PAGESIZES=$(($PAGESIZE * 33))
+# On recent Linux kernels (at least v5.4) updating stats happens in batches
+# (PAGESIZES) and also might depend on workload and number of CPUs. The kernel
+# caches the data and does not prioritize stats precision. This is especially
+# visible for max_usage_in_bytes where it usually exceeds
+# actual memory allocation.
+# When checking for usage_in_bytes and max_usage_in_bytes accept also higher values
+# from given range:
+MEM_USAGE_RANGE=$((PAGESIZES))
+
HUGEPAGESIZE=$(awk '/Hugepagesize/ {print $2}' /proc/meminfo)
[ -z $HUGEPAGESIZE ] && HUGEPAGESIZE=0
HUGEPAGESIZE=$(($HUGEPAGESIZE * 1024))
@@ -140,7 +149,8 @@ shmmax_cleanup()
# Check size in memcg
# $1 - Item name
-# $2 - Expected size
+# $2 - Expected size lower bound
+# $3 - Expected size upper bound (optional)
check_mem_stat()
{
local item_size
@@ -151,7 +161,13 @@ check_mem_stat()
item_size=$(grep -w $1 memory.stat | cut -d " " -f 2)
fi
- if [ "$2" = "$item_size" ]; then
+ if [ "$3" ]; then
+ if [ $item_size -ge $2 ] && [ $item_size -le $3 ]; then
+ tst_res TPASS "$1 is ${2}-${3} as expected"
+ else
+ tst_res TFAIL "$1 is $item_size, ${2}-${3} expected"
+ fi
+ elif [ "$2" = "$item_size" ]; then
tst_res TPASS "$1 is $2 as expected"
else
tst_res TFAIL "$1 is $item_size, $2 expected"
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
index 14daa4651798..8831f1937070 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
@@ -12,6 +12,7 @@ TST_CNT=4
. memcg_lib.sh
MEM_TO_ALLOC=$((PAGESIZE * 1024))
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE))
MEM_LIMIT=$((MEM_TO_ALLOC * 2))
# Run test cases which checks memory.[memsw.]max_usage_in_bytes after make
@@ -32,7 +33,7 @@ test_max_usage_in_bytes()
signal_memcg_process $MEM_TO_ALLOC
signal_memcg_process $MEM_TO_ALLOC
- check_mem_stat $item $MEM_TO_ALLOC
+ check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER
if [ $check_after_reset -eq 1 ]; then
echo 0 > $item
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [LTP] [PATCH v4 2/5] controllers/memcg: accept range of usage_in_bytes
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 3/5] controllers/memcg: accept non-zero max_usage_in_bytes after reset Krzysztof Kozlowski
` (2 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
Several Linux kernel versions report higher usage_in_bytes than
expected size of 1024 pages. For example v5.4, v5.8, v5.10
and 5.13.0-rc5:
memcg_usage_in_bytes_test 1 TINFO: Test memory.usage_in_bytes
memcg_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_usage_in_bytes_test 1 TINFO: Warming up pid: 1160
memcg_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 1160
memcg_usage_in_bytes_test 1 TFAIL: memory.usage_in_bytes is 4325376, 4194304 expected
It seems that recent Linux kernel caches the statistics more
aggressively (especially on multi-CPU systems) and the batch updates of
32 pages are visible in usage_in_bytes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../controllers/memcg/functional/memcg_lib.sh | 11 +++++++---
| 20 +++++++++----------
.../memcg/functional/memcg_stat_test.sh | 8 ++++----
.../functional/memcg_usage_in_bytes_test.sh | 7 +++++--
4 files changed, 27 insertions(+), 19 deletions(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
index 4c47e63745ff..9634eb7e9f42 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
@@ -248,8 +248,9 @@ test_mem_stat()
local size=$2
local total_size=$3
local stat_name=$4
- local exp_stat_size=$5
- local check_after_free=$6
+ local exp_stat_size_low=$5
+ local exp_stat_size_up=$6
+ local check_after_free=$7
start_memcg_process $memtypes -s $size
@@ -260,7 +261,11 @@ test_mem_stat()
echo $MEMCG_PROCESS_PID > tasks
signal_memcg_process $size
- check_mem_stat $stat_name $exp_stat_size
+ if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then
+ check_mem_stat $stat_name $exp_stat_size_low
+ else
+ check_mem_stat $stat_name $exp_stat_size_low $exp_stat_size_up
+ fi
signal_memcg_process $size
if $check_after_free; then
--git a/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh b/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh
index 1a6128a6dba8..d9b4ec287b5f 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_stat_rss.sh
@@ -18,54 +18,54 @@ TST_CNT=10
# Test the management and counting of memory
test1()
{
- test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+ test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
}
test2()
{
- test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 false
+ test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 false
}
test3()
{
- test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 false
+ test_mem_stat "--shm -k 3" $PAGESIZE $PAGESIZE "rss" 0 0 false
}
test4()
{
test_mem_stat "--mmap-anon --mmap-file --shm" \
- $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES false
+ $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES false
}
test5()
{
- test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES false
+ test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES false
}
test6()
{
- test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+ test_mem_stat "--mmap-anon" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
}
test7()
{
- test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 true
+ test_mem_stat "--mmap-file" $PAGESIZE $PAGESIZE "rss" 0 0 true
}
test8()
{
- test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 true
+ test_mem_stat "--shm -k 8" $PAGESIZE $PAGESIZE "rss" 0 0 true
}
test9()
{
test_mem_stat "--mmap-anon --mmap-file --shm" \
- $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES true
+ $PAGESIZES $((PAGESIZES * 3)) "rss" $PAGESIZES $PAGESIZES true
}
test10()
{
- test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES true
+ test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES "rss" $PAGESIZES $PAGESIZES true
}
tst_run
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh
index 925c4ecf87bc..e5eb7e5d0001 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_stat_test.sh
@@ -14,28 +14,28 @@ TST_CNT=8
test1()
{
tst_res TINFO "Test cache"
- test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES false
+ test_mem_stat "--shm -k 3" $PAGESIZES $PAGESIZES "cache" $PAGESIZES $PAGESIZES false
}
test2()
{
tst_res TINFO "Test mapped_file"
test_mem_stat "--mmap-file" $PAGESIZES $PAGESIZES \
- "mapped_file" $PAGESIZES false
+ "mapped_file" $PAGESIZES $PAGESIZES false
}
test3()
{
tst_res TINFO "Test unevictable with MAP_LOCKED"
test_mem_stat "--mmap-lock1" $PAGESIZES $PAGESIZES \
- "unevictable" $PAGESIZES false
+ "unevictable" $PAGESIZES $PAGESIZES false
}
test4()
{
tst_res TINFO "Test unevictable with mlock"
test_mem_stat "--mmap-lock2" $PAGESIZES $PAGESIZES \
- "unevictable" $PAGESIZES false
+ "unevictable" $PAGESIZES $PAGESIZES false
}
test5()
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh
index e77d6bf2ef23..9140fd9d1fd7 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_usage_in_bytes_test.sh
@@ -12,13 +12,15 @@ TST_CNT=2
. memcg_lib.sh
MEM_TO_ALLOC=$((PAGESIZE * 1024))
+MEM_EXPECTED_UPPER=$((MEM_TO_ALLOC + MEM_USAGE_RANGE))
MEM_LIMIT=$((MEM_TO_ALLOC * 2))
test1()
{
tst_res TINFO "Test memory.usage_in_bytes"
test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
- "memory.usage_in_bytes" $MEM_TO_ALLOC false
+ "memory.usage_in_bytes" $MEM_TO_ALLOC \
+ $MEM_EXPECTED_UPPER false
}
test2()
@@ -29,7 +31,8 @@ test2()
echo $MEM_LIMIT > memory.limit_in_bytes
echo $MEM_LIMIT > memory.memsw.limit_in_bytes
test_mem_stat "--mmap-anon" $MEM_TO_ALLOC $MEM_TO_ALLOC \
- "memory.memsw.usage_in_bytes" $MEM_TO_ALLOC false
+ "memory.memsw.usage_in_bytes" $MEM_TO_ALLOC \
+ $MEM_EXPECTED_UPPER false
}
tst_run
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [LTP] [PATCH v4 3/5] controllers/memcg: accept non-zero max_usage_in_bytes after reset
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 1/5] controllers/memcg: accept range of max_usage_in_bytes Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 2/5] controllers/memcg: accept range of usage_in_bytes Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge Krzysztof Kozlowski
2021-07-13 9:22 ` [LTP] [PATCH v4 5/5] controllers/memcg: offset kernel memory Krzysztof Kozlowski
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
Several Linux kernel versions report a non-zero max_usage_in_bytes after
resetting the counter. For example v5.4, v5.8, v5.10, v5.11, v5.12 and
5.13.0-rc5:
memcg_max_usage_in_bytes_test 4 TINFO: Test reset memory.memsw.max_usage_in_bytes
memcg_max_usage_in_bytes_test 4 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_max_usage_in_bytes_test 4 TINFO: Warming up pid: 1416
memcg_max_usage_in_bytes_test 4 TINFO: Process is still here after warm up: 1416
memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 4325376, 4194304 expected
memcg_max_usage_in_bytes_test 4 TFAIL: memory.memsw.max_usage_in_bytes is 122880, 0 expected
It seems that recent Linux kernel caches the statistics more
aggressively (especially on multi-CPU systems) and the batch updates of
32 pages are visible in usage_in_bytes.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../memcg/functional/memcg_max_usage_in_bytes_test.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
index 8831f1937070..2d494ac3a78f 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
@@ -37,7 +37,7 @@ test_max_usage_in_bytes()
if [ $check_after_reset -eq 1 ]; then
echo 0 > $item
- check_mem_stat $item 0
+ check_mem_stat $item 0 $PAGESIZES
fi
stop_memcg_process
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
` (2 preceding siblings ...)
2021-07-13 9:22 ` [LTP] [PATCH v4 3/5] controllers/memcg: accept non-zero max_usage_in_bytes after reset Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
2021-07-13 9:40 ` Richard Palethorpe
2021-07-13 9:22 ` [LTP] [PATCH v4 5/5] controllers/memcg: offset kernel memory Krzysztof Kozlowski
4 siblings, 1 reply; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
with:
memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
memcg_subgroup_charge 1 TPASS: rss is 0 as expected
In dmesg one could see that OOM killer killed the process even though
group memory limit was matching the usage:
memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
...
memory: usage 132kB, limit 132kB, failcnt 9
memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
...
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 19289] 0 19289 669 389 40960 0 0 memcg_process
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
It seems actual kernel memory usage by a given group depends on number
of CPUs, where at least one page is allocated per-cpu beside regular
(expected) allocation. Fix the test on machines with more CPUs by
including this per-CPU memory in group limits, plus some slack of 4
PAGES. Increase also memory allocation from 32 to 64 pages to be more
distinctive from kernel per-CPU memory.
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++-----
1 file changed, 25 insertions(+), 8 deletions(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
index 9b23177a4dc5..0d2b235aff7c 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
@@ -14,16 +14,33 @@ TST_CNT=3
. memcg_lib.sh
+# Allocate memory bigger than per-cpu kernel memory
+MEM_TO_ALLOC=$((PAGESIZES * 2))
+
# Test the memory charge won't move to subgroup
# $1 - memory.limit_in_bytes in parent group
# $2 - memory.limit_in_bytes in sub group
test_subgroup()
{
+ local limit_parent=$1
+ local limit_subgroup=$2
+ local total_cpus=`tst_ncpus`
+
+ # Kernel memory allocated for the process is also charged.
+ # It might depend on the number of CPUs. For example on kernel v5.11
+ # additionally total_cpus plus 1-2 pages are charged to the group.
+ if [ $limit_parent -ne 0 ]; then
+ limit_parent=$((limit_parent + 4 * PAGESIZE + total_cpus * PAGESIZE))
+ fi
+ if [ $limit_subgroup -ne 0 ]; then
+ limit_subgroup=$((limit_subgroup + 4 * PAGESIZE + total_cpus * PAGESIZE))
+ fi
+
mkdir subgroup
- echo $1 > memory.limit_in_bytes
- echo $2 > subgroup/memory.limit_in_bytes
+ echo $limit_parent > memory.limit_in_bytes
+ echo $limit_subgroup > subgroup/memory.limit_in_bytes
- start_memcg_process --mmap-anon -s $PAGESIZES
+ start_memcg_process --mmap-anon -s $MEM_TO_ALLOC
warmup
if [ $? -ne 0 ]; then
@@ -31,8 +48,8 @@ test_subgroup()
fi
echo $MEMCG_PROCESS_PID > tasks
- signal_memcg_process $PAGESIZES
- check_mem_stat "rss" $PAGESIZES
+ signal_memcg_process $MEM_TO_ALLOC
+ check_mem_stat "rss" $MEM_TO_ALLOC
cd subgroup
echo $MEMCG_PROCESS_PID > tasks
@@ -47,17 +64,17 @@ test_subgroup()
test1()
{
tst_res TINFO "Test that group and subgroup have no relationship"
- test_subgroup $PAGESIZES $((2 * PAGESIZES))
+ test_subgroup $MEM_TO_ALLOC $((2 * MEM_TO_ALLOC))
}
test2()
{
- test_subgroup $PAGESIZES $PAGESIZES
+ test_subgroup $MEM_TO_ALLOC $MEM_TO_ALLOC
}
test3()
{
- test_subgroup $PAGESIZES 0
+ test_subgroup $MEM_TO_ALLOC 0
}
tst_run
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:22 ` [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge Krzysztof Kozlowski
@ 2021-07-13 9:40 ` Richard Palethorpe
2021-07-13 9:48 ` Krzysztof Kozlowski
0 siblings, 1 reply; 10+ messages in thread
From: Richard Palethorpe @ 2021-07-13 9:40 UTC (permalink / raw)
To: ltp
Hello Krzysztof,
Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
> The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
> with:
>
> memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
> memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
> memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
> memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
> memcg_subgroup_charge 1 TPASS: rss is 0 as expected
>
> In dmesg one could see that OOM killer killed the process even though
> group memory limit was matching the usage:
>
> memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
> CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
> ...
> memory: usage 132kB, limit 132kB, failcnt 9
> memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
> kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
> ...
> Tasks state (memory values in pages):
> [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
> [ 19289] 0 19289 669 389 40960 0 0 memcg_process
> oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
> Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
> oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
>
> It seems actual kernel memory usage by a given group depends on number
> of CPUs, where at least one page is allocated per-cpu beside regular
> (expected) allocation. Fix the test on machines with more CPUs by
> including this per-CPU memory in group limits, plus some slack of 4
> PAGES. Increase also memory allocation from 32 to 64 pages to be more
> distinctive from kernel per-CPU memory.
Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
>
> Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
> ---
> .../memcg/functional/memcg_subgroup_charge.sh | 33 ++++++++++++++-----
> 1 file changed, 25 insertions(+), 8 deletions(-)
>
> diff --git a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
> index 9b23177a4dc5..0d2b235aff7c 100755
> --- a/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
> +++ b/testcases/kernel/controllers/memcg/functional/memcg_subgroup_charge.sh
> @@ -14,16 +14,33 @@ TST_CNT=3
>
> . memcg_lib.sh
>
> +# Allocate memory bigger than per-cpu kernel memory
> +MEM_TO_ALLOC=$((PAGESIZES * 2))
> +
--
Thank you,
Richard.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:40 ` Richard Palethorpe
@ 2021-07-13 9:48 ` Krzysztof Kozlowski
2021-07-13 14:17 ` Richard Palethorpe
0 siblings, 1 reply; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:48 UTC (permalink / raw)
To: ltp
On 13/07/2021 11:40, Richard Palethorpe wrote:
> Hello Krzysztof,
>
> Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
>
>> The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
>> with:
>>
>> memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
>> memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
>> memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
>> memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
>> memcg_subgroup_charge 1 TPASS: rss is 0 as expected
>>
>> In dmesg one could see that OOM killer killed the process even though
>> group memory limit was matching the usage:
>>
>> memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
>> CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
>> ...
>> memory: usage 132kB, limit 132kB, failcnt 9
>> memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
>> kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
>> ...
>> Tasks state (memory values in pages):
>> [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
>> [ 19289] 0 19289 669 389 40960 0 0 memcg_process
>> oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
>> Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
>> oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
>>
>> It seems actual kernel memory usage by a given group depends on number
>> of CPUs, where at least one page is allocated per-cpu beside regular
>> (expected) allocation. Fix the test on machines with more CPUs by
>> including this per-CPU memory in group limits, plus some slack of 4
>> PAGES. Increase also memory allocation from 32 to 64 pages to be more
>> distinctive from kernel per-CPU memory.
>
> Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
>
Yes, right. Maybe this could be fixed when applying - no need for resend.
Best regards,
Krzysztof
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge
2021-07-13 9:48 ` Krzysztof Kozlowski
@ 2021-07-13 14:17 ` Richard Palethorpe
2021-07-14 6:05 ` Li Wang
0 siblings, 1 reply; 10+ messages in thread
From: Richard Palethorpe @ 2021-07-13 14:17 UTC (permalink / raw)
To: ltp
Hello,
Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
> On 13/07/2021 11:40, Richard Palethorpe wrote:
>> Hello Krzysztof,
>>
>> Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com> writes:
>>
>>> The memcg_subgroup_charge was failing on kernel v5.8 in around 10% cases
>>> with:
>>>
>>> memcg_subgroup_charge 1 TINFO: Running memcg_process --mmap-anon -s 135168
>>> memcg_subgroup_charge 1 TINFO: Warming up pid: 19289
>>> memcg_subgroup_charge 1 TINFO: Process is still here after warm up: 19289
>>> memcg_subgroup_charge 1 TFAIL: rss is 0, 135168 expected
>>> memcg_subgroup_charge 1 TPASS: rss is 0 as expected
>>>
>>> In dmesg one could see that OOM killer killed the process even though
>>> group memory limit was matching the usage:
>>>
>>> memcg_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
>>> CPU: 4 PID: 19289 Comm: memcg_process Not tainted 5.8.0-1031-oracle #32~20.04.2-Ubuntu
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
>>> ...
>>> memory: usage 132kB, limit 132kB, failcnt 9
>>> memory+swap: usage 132kB, limit 9007199254740988kB, failcnt 0
>>> kmem: usage 4kB, limit 9007199254740988kB, failcnt 0
>>> ...
>>> Tasks state (memory values in pages):
>>> [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
>>> [ 19289] 0 19289 669 389 40960 0 0 memcg_process
>>> oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ltp_19257,task_memcg=/ltp_19257,task=memcg_process,pid=19289,uid=0
>>> Memory cgroup out of memory: Killed process 19289 (memcg_process) total-vm:2676kB, anon-rss:84kB, file-rss:1468kB, shmem-rss:4kB, UID:0 pgtables:40kB oom_score_adj:0
>>> oom_reaper: reaped process 19289 (memcg_process), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
>>>
>>> It seems actual kernel memory usage by a given group depends on number
>>> of CPUs, where at least one page is allocated per-cpu beside regular
>>> (expected) allocation. Fix the test on machines with more CPUs by
>>> including this per-CPU memory in group limits, plus some slack of 4
>>> PAGES. Increase also memory allocation from 32 to 64 pages to be more
>>> distinctive from kernel per-CPU memory.
>>
>> Actually I think it is 66 pages? Because PAGESIZES=pagesize*33.
>>
>
> Yes, right. Maybe this could be fixed when applying - no need for resend.
>
>
> Best regards,
> Krzysztof
Well I am more than happy with the patchset.
Acked-by: Richard Palethorpe <rpalethorpe@suse.com>
--
Thank you,
Richard.
^ permalink raw reply [flat|nested] 10+ messages in thread
* [LTP] [PATCH v4 5/5] controllers/memcg: offset kernel memory
2021-07-13 9:22 [LTP] [PATCH v4 0/5] controllers/memcg: fixes for newer kernels Krzysztof Kozlowski
` (3 preceding siblings ...)
2021-07-13 9:22 ` [LTP] [PATCH v4 4/5] controllers/memcg: increase memory limit in subgroup charge Krzysztof Kozlowski
@ 2021-07-13 9:22 ` Krzysztof Kozlowski
4 siblings, 0 replies; 10+ messages in thread
From: Krzysztof Kozlowski @ 2021-07-13 9:22 UTC (permalink / raw)
To: ltp
The memory allocated by kernel with __GFP_ACCOUNT is counted as well in
memory group usage limits. Add it to fix failures like:
memcg_max_usage_in_bytes_test 1 TINFO: Running memcg_process --mmap-anon -s 4194304
memcg_max_usage_in_bytes_test 1 TINFO: Warming up pid: 925811
memcg_max_usage_in_bytes_test 1 TINFO: Process is still here after warm up: 925811
memcg_max_usage_in_bytes_test 1 TFAIL: memory.max_usage_in_bytes is 4333568, 4194304-4325376 expected
(where memory.kmem.max_usage_in_bytes:8192)
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
---
.../controllers/memcg/functional/memcg_lib.sh | 10 ++++++++++
.../functional/memcg_max_usage_in_bytes_test.sh | 15 ++++++++++++++-
2 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
index 9634eb7e9f42..dad66c798e19 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_lib.sh
@@ -251,6 +251,7 @@ test_mem_stat()
local exp_stat_size_low=$5
local exp_stat_size_up=$6
local check_after_free=$7
+ local kmem_stat_name="${stat_name##*.}"
start_memcg_process $memtypes -s $size
@@ -261,6 +262,15 @@ test_mem_stat()
echo $MEMCG_PROCESS_PID > tasks
signal_memcg_process $size
+ if [ "$kmem_stat_name" = "max_usage_in_bytes" ] ||
+ [ "$kmem_stat_name" = "usage_in_bytes" ]; then
+ local kmem=$(cat "memory.kmem.${kmem_stat_name}")
+ if [ $? -eq 0 ]; then
+ exp_stat_size_low=$((exp_stat_size_low + kmem))
+ exp_stat_size_up=$((exp_stat_size_up + kmem))
+ fi
+ fi
+
if [ "$exp_stat_size_low" = "$exp_stat_size_up" ]; then
check_mem_stat $stat_name $exp_stat_size_low
else
diff --git a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
index 2d494ac3a78f..24e9d115c19e 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_max_usage_in_bytes_test.sh
@@ -22,6 +22,10 @@ test_max_usage_in_bytes()
local item="memory.max_usage_in_bytes"
[ $1 -eq 1 ] && item="memory.memsw.max_usage_in_bytes"
local check_after_reset=$2
+ local exp_stat_size_low=$MEM_TO_ALLOC
+ local exp_stat_size_up=$MEM_EXPECTED_UPPER
+ local kmem_stat_name="${item##*.}"
+
start_memcg_process --mmap-anon -s $MEM_TO_ALLOC
warmup
@@ -33,7 +37,16 @@ test_max_usage_in_bytes()
signal_memcg_process $MEM_TO_ALLOC
signal_memcg_process $MEM_TO_ALLOC
- check_mem_stat $item $MEM_TO_ALLOC $MEM_EXPECTED_UPPER
+ if [ "$kmem_stat_name" = "max_usage_in_bytes" ] ||
+ [ "$kmem_stat_name" = "usage_in_bytes" ]; then
+ local kmem=$(cat "memory.kmem.${kmem_stat_name}")
+ if [ $? -eq 0 ]; then
+ exp_stat_size_low=$((exp_stat_size_low + kmem))
+ exp_stat_size_up=$((exp_stat_size_up + kmem))
+ fi
+ fi
+
+ check_mem_stat $item $exp_stat_size_low $exp_stat_size_up
if [ $check_after_reset -eq 1 ]; then
echo 0 > $item
--
2.27.0
^ permalink raw reply related [flat|nested] 10+ messages in thread