[LTP] [PATCH v4] shell: enable OOM protection by default

[LTP] [PATCH v4] shell: enable OOM protection by default

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

The shell harness shields itself from the OOM killer and runs the
test in a child process, so it survives memory pressure and can
still report results (e.g. during memcg stress tests).

Suggested-by: Li Wang <liwang@redhat.com>
Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
Under Li's idea, implement a OOM protection mechanism for the shell
tests so we can avoid OOM for memcg stress tests.
---
Changes in v4:
- fix with Petr suggestions
- fix with AI agent suggestions
- Link to v3: https://lore.kernel.org/20260804-shell_oom_protection-v3-1-fe42b15d034c@suse.com

Changes in v3:
- simplify oom code
- enable OOM protection by default
- Link to v2: https://lore.kernel.org/20260730-shell_oom_protection-v2-0-be1de2baa83d@suse.com

Changes in v2:
- update shell OOM protection functional test
- Link to v1: https://lore.kernel.org/20260713-shell_oom_protection-v1-0-b732e8647894@suse.com

To: Linux Test Project <ltp@lists.linux.it>
---
 lib/newlib_tests/runtest.sh                  |  1 +
 lib/newlib_tests/shell/tst_oom_protection.sh | 34 +++++++++++
 testcases/lib/tst_test.sh                    | 84 ++++++++++++++++++++++++++--
 3 files changed, 115 insertions(+), 4 deletions(-)

diff --git a/lib/newlib_tests/runtest.sh b/lib/newlib_tests/runtest.sh
index 71808ef8b..7e2d0a2ac 100755
--- a/lib/newlib_tests/runtest.sh
+++ b/lib/newlib_tests/runtest.sh
@@ -44,6 +44,7 @@ shell/tst_check_driver.sh
 shell/tst_check_kconfig0[1-5].sh
 shell/tst_mount_device.sh
 shell/tst_mount_device_tmpfs.sh
+shell/tst_oom_protection.sh
 shell/tst_skip_filesystems.sh
 shell/net/*.sh}"
 
diff --git a/lib/newlib_tests/shell/tst_oom_protection.sh b/lib/newlib_tests/shell/tst_oom_protection.sh
new file mode 100755
index 000000000..564680700
--- /dev/null
+++ b/lib/newlib_tests/shell/tst_oom_protection.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2026 Linux Test Project
+
+TST_TESTFUNC=do_test
+
+read_oom_score_adj() {
+	cat "/proc/$1/oom_score_adj"
+}
+
+do_test() {
+	local harness_score body_score
+
+	# $$ points to the protected harness, while /proc/self is the
+	# unprotected child that actually runs the test body.
+	harness_score=$(read_oom_score_adj "$$")
+	body_score=$(read_oom_score_adj self)
+
+	if [ "$harness_score" != -1000 ]; then
+		tst_res TCONF "shell harness OOM protection unavailable"
+		return
+	fi
+
+	tst_res TPASS "shell harness is protected from OOM by default"
+
+	if [ "$body_score" = 0 ]; then
+		tst_res TPASS "test body runs in an unprotected child (oom_score_adj=0)"
+	else
+		tst_res TFAIL "test body oom_score_adj is $body_score, expected 0"
+	fi
+}
+
+. tst_test.sh
+tst_run
diff --git a/testcases/lib/tst_test.sh b/testcases/lib/tst_test.sh
index b3e7e29bb..e42e562f5 100644
--- a/testcases/lib/tst_test.sh
+++ b/testcases/lib/tst_test.sh
@@ -28,6 +28,48 @@ export TST_USR_GID="${LTP_USR_GID:-65534}"
 trap "tst_brk TBROK 'test interrupted'" INT
 trap "unset _tst_setup_timer_pid; tst_brk TBROK 'test terminated'" TERM
 
+_tst_set_oom_score_adj()
+{
+	local value="$1"
+	local path="/proc/self/oom_score_adj"
+
+	if [ ! -e "$path" ]; then
+		tst_res TINFO "oom_score_adj does not exist, skipping the adjustment"
+		return
+	fi
+
+	if ! echo "$value" > "$path"; then
+		tst_res TWARN "Can't adjust score"
+	fi
+}
+
+_tst_enable_oom_protection()
+{
+	_tst_set_oom_score_adj -1000
+}
+
+_tst_disable_oom_protection()
+{
+	_tst_set_oom_score_adj 0
+}
+
+_tst_write_results()
+{
+	[ "$TST_CHILD" = 1 ] || return 0
+	[ -n "$TST_RESULTS_FILE" ] || return 0
+
+	echo "$TST_PASS $TST_FAIL $TST_BROK $TST_WARN $TST_CONF $TST_COUNT" \
+		> "$TST_RESULTS_FILE"
+}
+
+_tst_read_results()
+{
+	[ -s "$TST_RESULTS_FILE" ] || return 0
+
+	read TST_PASS TST_FAIL TST_BROK TST_WARN TST_CONF TST_COUNT \
+		< "$TST_RESULTS_FILE"
+}
+
 _tst_do_cleanup()
 {
 	if [ -n "$TST_DO_CLEANUP" -a -n "$TST_CLEANUP" -a -z "$LTP_NO_CLEANUP" ]; then
@@ -48,6 +90,15 @@ _tst_do_exit()
 
 	_tst_do_cleanup
 
+	# When running as the unprotected test child, only propagate the
+	# results back to the protected harness which does the teardown and
+	# prints the summary.
+	if [ "$TST_CHILD" = 1 ]; then
+		_tst_cleanup_timer
+		_tst_write_results
+		exit 0
+	fi
+
 	cd "$LTPROOT"
 	[ "$TST_MOUNT_FLAG" = 1 ] && tst_umount
 
@@ -788,10 +839,34 @@ tst_run()
 
 	TST_MNTPOINT="${TST_MNTPOINT:-$PWD/mntpoint}"
 
-	if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
-		_tst_run_tcases_per_fs
-	else
-		_tst_run_iterations
+	# Protect the harness so it survives memory pressure and can
+	# still report results, then run the test body in a child that
+	# resets OOM protection. This way workloads forked by the test
+	# remain killable, mirroring the C test harness.
+	_tst_enable_oom_protection
+
+	TST_RESULTS_FILE=$(mktemp "${TMPDIR:-/tmp}/ltp_${TST_ID}_res.XXXXXX")
+
+	(
+		TST_CHILD=1
+		_tst_disable_oom_protection
+		trap '_tst_cleanup_timer; _tst_write_results' EXIT
+
+		if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
+			_tst_run_tcases_per_fs
+		else
+			_tst_run_iterations
+		fi
+	) &
+
+	wait $!
+	ret=$?
+
+	_tst_read_results
+	rm -f "$TST_RESULTS_FILE"
+
+	if [ $ret -gt 128 ]; then
+		tst_res TBROK "Test killed by signal $((ret - 128))!"
 	fi
 
 	_tst_do_exit
@@ -865,6 +940,7 @@ _tst_run_test()
 	$_tst_fnc "$@"
 	_tst_rescmp "$_tst_res"
 	TST_COUNT=$((TST_COUNT+1))
+	_tst_write_results
 }
 
 export LC_ALL=C

---
base-commit: ee945bb71ba7015dace7d983cf98a530b3eb53a4
change-id: 20260713-shell_oom_protection-6221ab825220

Best regards,
--  
Andrea Cervesato <andrea.cervesato@suse.com>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] shell: enable OOM protection by default

[linuxtestproject.agent]

Hi Andrea,

On Aug 10, 2026, Andrea Cervesato wrote:
> shell: enable OOM protection by default

> +	if ! echo "$value" > "$path"; then
> +		tst_res TWARN "Can't adjust score"
> +	fi

Could this avoid reporting TWARN when CAP_SYS_RESOURCE is absent, as
tst_memutils.c does? Lowering oom_score_adj requires that capability, so
every unprivileged shell test now gets a warning result. The new self-test
itself reports TCONF but exits 36 because this TWARN remains counted.

> +	(
> +		TST_CHILD=1
> +		_tst_disable_oom_protection
> +		trap '_tst_cleanup_timer; _tst_write_results' EXIT
> +
> +		if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
> +			_tst_run_tcases_per_fs
> +		else
> +			_tst_run_iterations
> +		fi
> +	) &
> +
> +	wait $!
> +	ret=$?

How can the protected parent perform cleanup if this child is OOM-killed
or times out? TST_DO_CLEANUP and TST_MOUNT_FLAG are set only in the
subshell. On an abnormal child exit, the parent therefore reports TBROK
without calling TST_CLEANUP or unmounting a device mounted by
_prepare_device(), leaving external test state behind.

Verdict - Needs revision

---
Note:

The agent can sometimes produce false positives although often its
findings are genuine. If you find issues with the review, please
comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] shell: enable OOM protection by default

[Petr Vorel]

Hi Andrea,

> Hi Andrea,

> On Aug 10, 2026, Andrea Cervesato wrote:
> > shell: enable OOM protection by default

> > +	if ! echo "$value" > "$path"; then
> > +		tst_res TWARN "Can't adjust score"
> > +	fi

> Could this avoid reporting TWARN when CAP_SYS_RESOURCE is absent, as
> tst_memutils.c does? Lowering oom_score_adj requires that capability, so
> every unprivileged shell test now gets a warning result. The new self-test
> itself reports TCONF but exits 36 because this TWARN remains counted.

Ah, this really needs to be guarded behind  if [ "$(id -ru)" != 0 ]; then 

If you're in a mood for it, you could factor out [ "$(id -ru)" != 0 ] check into
it's own function tst_check_root or tst_is_root which would be then used in
tst_require_root and in this function.

I repeat my suggestion from last revision: add a helper into testcases/lib which
will just call C function. That way the behavior will be the same (use
capabilities). That would be more clearer solution.

Kind regards,
Petr

> > +	(
> > +		TST_CHILD=1
> > +		_tst_disable_oom_protection
> > +		trap '_tst_cleanup_timer; _tst_write_results' EXIT
> > +
> > +		if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
> > +			_tst_run_tcases_per_fs
> > +		else
> > +			_tst_run_iterations
> > +		fi
> > +	) &
> > +
> > +	wait $!
> > +	ret=$?

> How can the protected parent perform cleanup if this child is OOM-killed
> or times out? TST_DO_CLEANUP and TST_MOUNT_FLAG are set only in the
> subshell. On an abnormal child exit, the parent therefore reports TBROK
> without calling TST_CLEANUP or unmounting a device mounted by
> _prepare_device(), leaving external test state behind.

Sounds serious :). I'll comment on the patch itself.

Kind regards,
Petr

> Verdict - Needs revision

> ---
> Note:

> The agent can sometimes produce false positives although often its
> findings are genuine. If you find issues with the review, please
> comment this email or ignore the suggestions.

> Regards,
> LTP AI Reviewer

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v4] shell: enable OOM protection by default

[Petr Vorel]

Hi Andrea,

...
> diff --git a/lib/newlib_tests/shell/tst_oom_protection.sh b/lib/newlib_tests/shell/tst_oom_protection.sh
> new file mode 100755
> index 000000000..564680700
> --- /dev/null
> +++ b/lib/newlib_tests/shell/tst_oom_protection.sh
> @@ -0,0 +1,34 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (c) 2026 Linux Test Project
> +
> +TST_TESTFUNC=do_test
> +
> +read_oom_score_adj() {
> +	cat "/proc/$1/oom_score_adj"
> +}
> +
> +do_test() {
> +	local harness_score body_score
> +
> +	# $$ points to the protected harness, while /proc/self is the
> +	# unprotected child that actually runs the test body.
> +	harness_score=$(read_oom_score_adj "$$")
> +	body_score=$(read_oom_score_adj self)
> +
> +	if [ "$harness_score" != -1000 ]; then
> +		tst_res TCONF "shell harness OOM protection unavailable"
> +		return
> +	fi
> +
> +	tst_res TPASS "shell harness is protected from OOM by default"
> +
> +	if [ "$body_score" = 0 ]; then
> +		tst_res TPASS "test body runs in an unprotected child (oom_score_adj=0)"
> +	else
> +		tst_res TFAIL "test body oom_score_adj is $body_score, expected 0"
> +	fi
> +}
> +
> +. tst_test.sh
> +tst_run
> diff --git a/testcases/lib/tst_test.sh b/testcases/lib/tst_test.sh
> index b3e7e29bb..e42e562f5 100644
> --- a/testcases/lib/tst_test.sh
> +++ b/testcases/lib/tst_test.sh
> @@ -28,6 +28,48 @@ export TST_USR_GID="${LTP_USR_GID:-65534}"
>  trap "tst_brk TBROK 'test interrupted'" INT
>  trap "unset _tst_setup_timer_pid; tst_brk TBROK 'test terminated'" TERM

> +_tst_set_oom_score_adj()
> +{
> +	local value="$1"
> +	local path="/proc/self/oom_score_adj"
> +
> +	if [ ! -e "$path" ]; then
> +		tst_res TINFO "oom_score_adj does not exist, skipping the adjustment"
> +		return
> +	fi
> +
> +	if ! echo "$value" > "$path"; then
> +		tst_res TWARN "Can't adjust score"
> +	fi
> +}
> +
> +_tst_enable_oom_protection()
> +{
> +	_tst_set_oom_score_adj -1000
> +}
> +
> +_tst_disable_oom_protection()
> +{
> +	_tst_set_oom_score_adj 0
> +}
> +
> +_tst_write_results()
> +{
> +	[ "$TST_CHILD" = 1 ] || return 0
Yes, $TST_CHILD is here visible, because it's run from the child...

> +	[ -n "$TST_RESULTS_FILE" ] || return 0
> +
> +	echo "$TST_PASS $TST_FAIL $TST_BROK $TST_WARN $TST_CONF $TST_COUNT" \
> +		> "$TST_RESULTS_FILE"
> +}
> +
> +_tst_read_results()
> +{
> +	[ -s "$TST_RESULTS_FILE" ] || return 0
> +
> +	read TST_PASS TST_FAIL TST_BROK TST_WARN TST_CONF TST_COUNT \
> +		< "$TST_RESULTS_FILE"
> +}
> +
>  _tst_do_cleanup()
>  {
>  	if [ -n "$TST_DO_CLEANUP" -a -n "$TST_CLEANUP" -a -z "$LTP_NO_CLEANUP" ]; then
> @@ -48,6 +90,15 @@ _tst_do_exit()

>  	_tst_do_cleanup

> +	# When running as the unprotected test child, only propagate the
> +	# results back to the protected harness which does the teardown and
> +	# prints the summary.
> +	if [ "$TST_CHILD" = 1 ]; then
... but as agent noted $TST_CHILD is not visible here in _tst_do_exit() because
that's run from parent shell => nack.

> +		_tst_cleanup_timer
> +		_tst_write_results
> +		exit 0
> +	fi
> +
>  	cd "$LTPROOT"
>  	[ "$TST_MOUNT_FLAG" = 1 ] && tst_umount

> @@ -788,10 +839,34 @@ tst_run()

>  	TST_MNTPOINT="${TST_MNTPOINT:-$PWD/mntpoint}"

> -	if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
> -		_tst_run_tcases_per_fs
> -	else
> -		_tst_run_iterations
> +	# Protect the harness so it survives memory pressure and can
> +	# still report results, then run the test body in a child that
> +	# resets OOM protection. This way workloads forked by the test
> +	# remain killable, mirroring the C test harness.
> +	_tst_enable_oom_protection
> +
> +	TST_RESULTS_FILE=$(mktemp "${TMPDIR:-/tmp}/ltp_${TST_ID}_res.XXXXXX")
Hm, writing temporary file directly to TMPDIR is not optimal, but if that was
the only problem I'd be ok with it.

> +	(
> +		TST_CHILD=1
Agent is correct, this is not visible in the parent shell => I doubt it will
work.

Also this is supposed to fix memcg stress tests, but it touches all shell test
=> very effective way to broke many tests in single commit with great change to
get it quickly reverted :).

IMHO we should really rewrite the tests which does not work in tst_test.sh API
into C API (better) or to the shell loader. Further complicate tst_test.sh is
a way to hell.

Kind regards,
Petr

> +		_tst_disable_oom_protection
> +		trap '_tst_cleanup_timer; _tst_write_results' EXIT
> +
> +		if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
> +			_tst_run_tcases_per_fs
> +		else
> +			_tst_run_iterations
> +		fi
> +	) &
> +
> +	wait $!
> +	ret=$?
> +
> +	_tst_read_results
> +	rm -f "$TST_RESULTS_FILE"
> +
> +	if [ $ret -gt 128 ]; then
> +		tst_res TBROK "Test killed by signal $((ret - 128))!"
>  	fi

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v4] shell: enable OOM protection by default

[Li Wang]

Hi Andrea, Petr,

After looking into it more, now I slightly think doing OOM protection
in the shell harness isn't worth it: the benefit doesn't justify the
complexity. Forking the whole test body means dealing with cleanup and
timer ownership across the fork, plus passing results back from the
child, and all of that would need revalidating for every shell test
that goes through tst_test.sh.

Given that only a few tests actually produce memory pressure, I'd
rather keep it simple and handle it locally in those tests instead of
touching the common harness. Something like a small helper the test
can wrap the memory hungry workload with:

    tst_oom_run()
    {
        ( echo 0 > /proc/self/oom_score_adj; exec "$@" )
    }

The harness (or the setup) protects itself with oom_score_adj -1000,
and only the workload started via tst_oom_run stays killable.
This keeps the harness alive to report results without forking the
whole test body, and matches the C harness model where the worker is
the unprotected part.

Does this direction make sense to you, or do you see a case that
really needs the protection to be harness-wide?
 
> Suggested-by: Li Wang <liwang@redhat.com>

This email address is no longer in use :).

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v4] shell: enable OOM protection by default

[Petr Vorel]

Hi Li, Andrea,

> Hi Andrea, Petr,

> After looking into it more, now I slightly think doing OOM protection
> in the shell harness isn't worth it: the benefit doesn't justify the
> complexity. Forking the whole test body means dealing with cleanup and
> timer ownership across the fork, plus passing results back from the
> child, and all of that would need revalidating for every shell test
> that goes through tst_test.sh.

+1

> Given that only a few tests actually produce memory pressure, I'd
> rather keep it simple and handle it locally in those tests instead of
> touching the common harness. Something like a small helper the test
> can wrap the memory hungry workload with:

>     tst_oom_run()
>     {
>         ( echo 0 > /proc/self/oom_score_adj; exec "$@" )
>     }

+1

Maybe test should declare through TST_* variable that it needs OOM killer
(approach on Andrea's v2 [1]). I'm sorry Andrea, it takes time to find a right
solution.

[1] https://lore.kernel.org/ltp/20260730-shell_oom_protection-v2-0-be1de2baa83d@suse.com/

And again, IMHO the real solution would be to rewrite tests into C API or shell
loader API.

> The harness (or the setup) protects itself with oom_score_adj -1000,
> and only the workload started via tst_oom_run stays killable.
> This keeps the harness alive to report results without forking the
> whole test body, and matches the C harness model where the worker is
> the unprotected part.

Yes, forking (the subshell) part looked to me fragile.

> Does this direction make sense to you, or do you see a case that
> really needs the protection to be harness-wide?

> > Suggested-by: Li Wang <liwang@redhat.com>

> This email address is no longer in use :).

+1

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp