[LTP] [PATCH] memcg_stress: survive OOM by targeting the stressors

[LTP] [PATCH] memcg_stress: survive OOM by targeting the stressors

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

The single-cgroup subtest intentionally lets one process fault in the
whole memory budget to push the system to its memory ceiling, so hitting
the OOM killer is an expected part of the stress. The problem is that the
OOM killer could reap the driver script, turning that expected pressure
into a spurious TBROK.

Mark the stress processes as the preferred OOM victims. A killed stressor
is already reaped and cleaned up by the test, while the driver survives
and reports the result.

Fixes: 02961a7b2bb4 ("memcg_stress_test.sh: Fix reserved mem calculate")
Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
On some systems this might happens:

[  244.207623][T117130] kirk[3061]: memcg_stress: start (command: memcg_stress_test.sh)
[  244.391438][T117164] memcg_stress_te (117164): drop_caches: 3
[ 1444.509722][T119900] kirk[3061]: memcg_stress: end (returncode: -1)
[ 1444.629434][T119934] kirk[3061]: memcg_control: start (command: memcg_control_test.sh)
[ 1445.856012][T120020] mem_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
---
 testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh b/testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh
index 47cac9af8bec970b7fb338c5469ab73d55267740..14a5ddeeaa45a1bc5e30b2fa7640bcb1effbd94b 100755
--- a/testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh
+++ b/testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh
@@ -71,8 +71,12 @@ run_stress()
 	for i in $(seq 0 $(($cgroups-1))); do
 		ROD mkdir "$test_path/$i"
 		memcg_process_stress $mem_size $interval &
-		ROD echo $! \> "$test_path/$i/$task_list"
-		pids="$pids $!"
+		pid=$!
+		ROD echo $pid \> "$test_path/$i/$task_list"
+		# Make the stress process the preferred OOM victim so that memory
+		# pressure kills a stressor rather than the driver script.
+		echo 1000 > "/proc/$pid/oom_score_adj" 2>/dev/null
+		pids="$pids $pid"
 	done
 
 	for pid in $pids; do

---
base-commit: 4898cb95b01b8a3ab5e1ea752990deca7a66ca34
change-id: 20260703-fix_memcg_stress-0c6669bfa5f6

Best regards,
-- 
Andrea Cervesato <andrea.cervesato@suse.com>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] memcg_stress: survive OOM by targeting the stressors

[linuxtestproject.agent]

Hi Andrea,

On Fri, 3 Jul 2026, Andrea Cervesato wrote:
> memcg_stress: survive OOM by targeting the stressors

Verdict - Reviewed

---
Note:

The agent can sometimes produce false positives although often its
findings are genuine. If you find issues with the review, please
comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] memcg_stress: survive OOM by targeting the stressors

[Li Wang]

Hi Andrea,

Andrea Cervesato wrote:

> From: Andrea Cervesato <andrea.cervesato@suse.com>
> 
> The single-cgroup subtest intentionally lets one process fault in the
> whole memory budget to push the system to its memory ceiling, so hitting
> the OOM killer is an expected part of the stress. The problem is that the
> OOM killer could reap the driver script, turning that expected pressure
> into a spurious TBROK.
> 
> Mark the stress processes as the preferred OOM victims. A killed stressor
> is already reaped and cleaned up by the test, while the driver survives
> and reports the result.
> 
> Fixes: 02961a7b2bb4 ("memcg_stress_test.sh: Fix reserved mem calculate")
> Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
> ---
> On some systems this might happens:
> 
> [  244.207623][T117130] kirk[3061]: memcg_stress: start (command: memcg_stress_test.sh)
> [  244.391438][T117164] memcg_stress_te (117164): drop_caches: 3
> [ 1444.509722][T119900] kirk[3061]: memcg_stress: end (returncode: -1)
> [ 1444.629434][T119934] kirk[3061]: memcg_control: start (command: memcg_control_test.sh)
> [ 1445.856012][T120020] mem_process invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0

I agree with the underlying goal, keep the driver alive and let the
stressors absorb the OOM pressure, but I'd like to suggest a different
way that mirrors what we already do in the C harness, and that fixes
this at the lib level rather than in the individual test.

In lib/tst_test.c we don't raise the score of the stress/test processes.
Instead we protect the harness and then explicitly drop that protection
in the child before it runs the actual test:

$ grep -A 8 -B 9 tst_enable_oom_protection lib/tst_test.c

void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
{
	unsigned int test_variants = 1;
	struct utsname uval;

	tst_test = self;

	do_setup(argc, argv);
	tst_enable_oom_protection(context->lib_pid);

	SAFE_SIGNAL(SIGALRM, alarm_handler);
	SAFE_SIGNAL(SIGUSR1, heartbeat_handler);

	tst_res(TINFO, "LTP version: "LTP_VERSION);

	uname(&uval);
	tst_res(TINFO, "Tested kernel: %s %s %s", uval.release, uval.version, uval.machine);

$ grep -A 8 -B 9 tst_disable_oom_protection lib/tst_test.c
	alarm(context->overall_time);

	show_failure_hints = 1;

	test_pid = fork();
	if (test_pid < 0)
		tst_brk(TBROK | TERRNO, "fork()");

	if (!test_pid) {
		tst_disable_oom_protection(0);
		SAFE_SIGNAL(SIGALRM, SIG_DFL);
		SAFE_SIGNAL(SIGUSR1, SIG_DFL);
		SAFE_SIGNAL(SIGTERM, SIG_DFL);
		SAFE_SIGNAL(SIGINT, SIG_DFL);
		SAFE_SETPGID(0, 0);
		testrun();
	}

The important detail is that oom_score_adj is inherited across fork.
So the model is: the library process sets itself to -1000, and child
resets itself back to 0 before doing any real work. The harness survives,
the workload stays a normal OOM candidate.

What do you think about doing this in the shell lib so all shell
tests benefit, rather than only memcg_stress?

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] memcg_stress: survive OOM by targeting the stressors

[Andrea Cervesato via ltp]

Hi Li,

> I agree with the underlying goal, keep the driver alive and let the
> stressors absorb the OOM pressure, but I'd like to suggest a different
> way that mirrors what we already do in the C harness, and that fixes
> this at the lib level rather than in the individual test.
> 
> In lib/tst_test.c we don't raise the score of the stress/test processes.
> Instead we protect the harness and then explicitly drop that protection
> in the child before it runs the actual test:
> 
> $ grep -A 8 -B 9 tst_enable_oom_protection lib/tst_test.c
> 
> void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
> {
> 	unsigned int test_variants = 1;
> 	struct utsname uval;
> 
> 	tst_test = self;
> 
> 	do_setup(argc, argv);
> 	tst_enable_oom_protection(context->lib_pid);
> 
> 	SAFE_SIGNAL(SIGALRM, alarm_handler);
> 	SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
> 
> 	tst_res(TINFO, "LTP version: "LTP_VERSION);
> 
> 	uname(&uval);
> 	tst_res(TINFO, "Tested kernel: %s %s %s", uval.release, uval.version, uval.machine);
> 
> $ grep -A 8 -B 9 tst_disable_oom_protection lib/tst_test.c
> 	alarm(context->overall_time);
> 
> 	show_failure_hints = 1;
> 
> 	test_pid = fork();
> 	if (test_pid < 0)
> 		tst_brk(TBROK | TERRNO, "fork()");
> 
> 	if (!test_pid) {
> 		tst_disable_oom_protection(0);
> 		SAFE_SIGNAL(SIGALRM, SIG_DFL);
> 		SAFE_SIGNAL(SIGUSR1, SIG_DFL);
> 		SAFE_SIGNAL(SIGTERM, SIG_DFL);
> 		SAFE_SIGNAL(SIGINT, SIG_DFL);
> 		SAFE_SETPGID(0, 0);
> 		testrun();
> 	}
> 
> The important detail is that oom_score_adj is inherited across fork.
> So the model is: the library process sets itself to -1000, and child
> resets itself back to 0 before doing any real work. The harness survives,
> the workload stays a normal OOM candidate.
> 
> What do you think about doing this in the shell lib so all shell
> tests benefit, rather than only memcg_stress?

this patch was sent with a wrong assumption indeed. As you can see
from the log, the OOM message is wrong and I pasted the wrong example :)

But this raised a good point that I think you spotted quite well. It
would be nice to implement this indeed, since shell tests are often
causing issues when it comes to containers or stressing the memory.

I will leave to @Petr the last word because he's more expert than me
on the shell tests.

Thanks,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] memcg_stress: survive OOM by targeting the stressors

[Li Wang]

Andrea Cervesato wrote:

> > What do you think about doing this in the shell lib so all shell
> > tests benefit, rather than only memcg_stress?
> 
> this patch was sent with a wrong assumption indeed. As you can see
> from the log, the OOM message is wrong and I pasted the wrong example :)

Ah, the memcg_stress test already done and kirk start a new
one memcg_control, that oom trigged by there.

> But this raised a good point that I think you spotted quite well. It
> would be nice to implement this indeed, since shell tests are often
> causing issues when it comes to containers or stressing the memory.

Sure, if the shell part encounters same situation, appropriate
measures should be taken.

> I will leave to @Petr the last word because he's more expert than me
> on the shell tests.

+1

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp