[PATCH] selftests/mm: Exit on invalid command-line options

[PATCH] selftests/mm: Exit on invalid command-line options

[hi]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

getopts reports unknown options and missing option arguments, but

run_vmtests.sh ignores its error result and continues running tests.

For example, passing -z or omitting the argument to -t can run the

default test selection instead of rejecting the command.




Exit with failure when getopts reports an error, before any tests or

their setup run. Keep the diagnostic provided by getopts.




Verified that -z, -t and -t mmap -z exit with status 1, while -h still

exits with status 0. The mmap category's three test programs pass in

a VM running the kernel built from the same source tree.




Fixes: 85463321e726 ("selftests/vm: enable running select groups of tests")

Assisted-by: LLM
Signed-off-by: Tianyi Chen <hi@tychen.cc>


---

 tools/testing/selftests/mm/run_vmtests.sh | 1 +

 1 file changed, 1 insertion(+)




diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh

index d09f9f6a384..189927f87e6 100755

--- a/tools/testing/selftests/mm/run_vmtests.sh

+++ b/tools/testing/selftests/mm/run_vmtests.sh

@@ -110,6 +110,7 @@ while getopts "aht:nd" OPT; do

            "t") VM_SELFTEST_ITEMS=${OPTARG} ;;

            "n") TAP_PREFIX= ;;

            "d") RUN_DESTRUCTIVE=true ;;

+           "?") exit 1 ;;

      esac

 done

 shift $((OPTIND -1))

-- 

2.55.0


 (https://www.spaceship.com/business-email/?utm_source=def_email_sign&utm_medium=email&utm_campaign=def_email_sign)

[-- Attachment #2: Type: text/html, Size: 3742 bytes --]

[PATCH RESEND] selftests/mm: Exit on invalid command-line options

[Tianyi Chen]

getopts reports unknown options and missing option arguments, but
run_vmtests.sh ignores its error result and continues running tests.
For example, passing -z or omitting the argument to -t can run the
default test selection instead of rejecting the command.

Exit with failure when getopts reports an error, before any tests or
their setup run. Keep the diagnostic provided by getopts.

Verified that -z, -t and -t mmap -z exit with status 1, while -h still
exits with status 0. The mmap category's three test programs pass in
a VM running the kernel built from the same source tree.

Fixes: 85463321e726 ("selftests/vm: enable running select groups of tests")
Assisted-by: LLM
Signed-off-by: Tianyi Chen <hi@tychen.cc>
---
Resending because the webmail editor damaged the whitespace in the
previous submission. The code change is unchanged.

 tools/testing/selftests/mm/run_vmtests.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index d09f9f6a384..189927f87e6 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -110,6 +110,7 @@ while getopts "aht:nd" OPT; do
 		"t") VM_SELFTEST_ITEMS=${OPTARG} ;;
 		"n") TAP_PREFIX= ;;
 		"d") RUN_DESTRUCTIVE=true ;;
+		"?") exit 1 ;;
 	esac
 done
 shift $((OPTIND -1))
-- 
2.55.0

Re: [PATCH RESEND] selftests/mm: Exit on invalid command-line options

[David Hildenbrand (Arm)]

On 9/6/26 15:54, Tianyi Chen wrote:
> getopts reports unknown options and missing option arguments, but
> run_vmtests.sh ignores its error result and continues running tests.
> For example, passing -z or omitting the argument to -t can run the
> default test selection instead of rejecting the command.
> 
> Exit with failure when getopts reports an error, before any tests or
> their setup run. Keep the diagnostic provided by getopts.
> 
> Verified that -z, -t and -t mmap -z exit with status 1, while -h still
> exits with status 0. The mmap category's three test programs pass in
> a VM running the kernel built from the same source tree.

I wonder whether we also bail out on wrong/missing categories?

It's also interesting that we do the /proc/sys/kernel/yama/ptrace_scope stuff
even when not actually running the memfd_secret test.

-- 
Cheers,

David

[PATCH v2 0/2] selftests/mm: Validate selections and scope memfd_secret setup

[Tianyi Chen]

run_vmtests.sh can reach test setup after an invalid category selection,
and its memfd_secret preparation can change ptrace_scope even when that
category was not selected. These two patches address David's follow-up
questions on the earlier getopts fix.

Patch 1 extends the getopts error handling to reject empty, whitespace-only
and unknown category selections before setup. It uses the existing category
list in usage() and normalizes separators, while preserving the default
when -t is omitted and the existing environment-selection behavior.

Patch 2 gates memfd_secret preparation on category selection and executable
presence. It retains the existing behavior when memfd_secret is selected;
it does not add configuration restoration or otherwise change that test.

Changes in v2:
- Rebase onto current mm-unstable.
- Extend the original option-error patch with category validation.
- Add a separate patch for the memfd_secret setup side effect.

Focused validation:
- Bash syntax and strict checkpatch passed. All 30 documented categories
  match the categories used by the script.
- Parser checks cover omitted/default selections, empty environment values,
  explicit empty -t, invalid options/categories, whitespace-separated lists
  and command-line overrides of environment selections.
- Twelve invalid-input cases and help were run through the full script in
  a KVM guest. Invalid inputs exited with status 1 before test setup.
- The real mmap category's three programs and memfd_secret passed in an
  x86-64 KVM guest running Linux 7.3.0-rc1, including multi-category and
  environment-selected runs.
- The guest kernel lacks Yama. A disposable file bind-mounted at the actual
  ptrace_scope path verified the shell's write condition: the old script
  writes for an unrelated selection, the new script does not, and selected
  executable memfd_secret retains the write. This checks script gating,
  not Yama policy enforcement. Missing-executable behavior was also checked.

This was focused validation, not a run of the complete MM selftest suite.

Previous submission:
https://lore.kernel.org/r/20260906135444.749144-1-hi@tychen.cc
Review:
https://lore.kernel.org/r/519ef70a-679c-455f-ba5f-3841730873a4@kernel.org


Tianyi Chen (2):
  selftests/mm: Reject invalid test selections before running tests
  selftests/mm: Only prepare ptrace_scope when memfd_secret is selected

 tools/testing/selftests/mm/run_vmtests.sh | 29 +++++++++++++++++++----
 1 file changed, 24 insertions(+), 5 deletions(-)

-- 
2.55.0

[PATCH v2 1/2] selftests/mm: Reject invalid test selections before running tests

[Tianyi Chen]

getopts reports unknown options and missing arguments, but run_vmtests.sh
ignores its error result and continues with test setup. An empty -t
argument also falls back to the default selection, while unknown category
names can silently select no tests and still reach setup code.

Exit on getopts errors and validate category names against the existing
list in usage() before any test setup. Reject empty and whitespace-only
selections, and normalize category separators so validation and execution
agree. Keep the default selection for an unset or empty VM_SELFTEST_ITEMS
environment variable, and preserve the existing standalone default value.

Fixes: 85463321e726 ("selftests/vm: enable running select groups of tests")
Assisted-by: Codex:GPT-6
Signed-off-by: Tianyi Chen <hi@tychen.cc>
---
 tools/testing/selftests/mm/run_vmtests.sh | 27 +++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index d09f9f6a384..836c7454cba 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -96,26 +96,45 @@ separated by spaces:
 
 example: ./run_vmtests.sh -t "hmm mmap ksm"
 EOF
-	exit 0
 }
 
 RUN_ALL=false
 RUN_DESTRUCTIVE=false
 TAP_PREFIX="# "
 
+# An unset or empty environment selection keeps the default behavior.
+VM_SELFTEST_ITEMS=${VM_SELFTEST_ITEMS:-default}
+
 while getopts "aht:nd" OPT; do
 	case ${OPT} in
 		"a") RUN_ALL=true ;;
-		"h") usage ;;
+		"h") usage; exit 0 ;;
 		"t") VM_SELFTEST_ITEMS=${OPTARG} ;;
 		"n") TAP_PREFIX= ;;
 		"d") RUN_DESTRUCTIVE=true ;;
+		"?") exit 1 ;;
 	esac
 done
 shift $((OPTIND -1))
 
-# default behavior: run all tests
-VM_SELFTEST_ITEMS=${VM_SELFTEST_ITEMS:-default}
+# Normalize whitespace so validation and test_selected() use the same names.
+read -r -a selected_categories <<< "${VM_SELFTEST_ITEMS//$'\n'/ }"
+VM_SELFTEST_ITEMS="${selected_categories[*]}"
+if [ -z "$VM_SELFTEST_ITEMS" ]; then
+	echo "No test categories specified" >&2
+	exit 1
+fi
+
+if [ "$VM_SELFTEST_ITEMS" != "default" ]; then
+	# Keep the documented category list as the source of valid names.
+	valid_categories=$(usage | sed -n 's/^- //p')
+	for category in "${selected_categories[@]}"; do
+		if ! grep -Fxq -- "$category" <<< "$valid_categories"; then
+			echo "Unknown test category: $category" >&2
+			exit 1
+		fi
+	done
+fi
 
 test_selected() {
 	if [ "$VM_SELFTEST_ITEMS" == "default" ]; then
-- 
2.55.0

[PATCH v2 2/2] selftests/mm: Only prepare ptrace_scope when memfd_secret is selected

[Tianyi Chen]

The memfd_secret setup clears ptrace_scope whenever its test binary is
executable, even when a different category was selected. run_test() filters
the test invocation, but it does not protect the preceding setup.

Check the category selection before entering the memfd_secret block so
running unrelated categories does not change ptrace_scope. Keep the
existing executable check and the behavior when memfd_secret is selected.

Assisted-by: Codex:GPT-6
Signed-off-by: Tianyi Chen <hi@tychen.cc>
---
 tools/testing/selftests/mm/run_vmtests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 836c7454cba..9932df252f4 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -371,7 +371,7 @@ CATEGORY="process_madv" run_test ./process_madv
 
 CATEGORY="vma_merge" run_test ./merge
 
-if [ -x ./memfd_secret ]
+if test_selected "memfd_secret" && [ -x ./memfd_secret ]
 then
 if [ -f /proc/sys/kernel/yama/ptrace_scope ]; then
 	(echo 0 > /proc/sys/kernel/yama/ptrace_scope 2>&1) | tap_prefix
-- 
2.55.0

Re: [PATCH RESEND] selftests/mm: Exit on invalid command-line options

[Tianyi Chen]

Thanks for pointing these out.

Unknown categories were not validated, and an explicitly empty -t
argument fell back to the default selection. A missing -t argument is
covered by getopts error handling. I have extended the first patch to
reject empty and unknown selections before setup, while preserving the
default behavior when -t is omitted. Validation uses the existing category
list in usage(), so there is no second category list to keep synchronized.

The ptrace_scope setup was indeed outside the category check. I have added
a separate patch to run it only when memfd_secret is selected and its
executable is present. The behavior when that test is selected is unchanged.

The v2 two-patch series and focused validation details are here:
https://lore.kernel.org/r/178886112560.138404.1278745290043112298.mm-cli-v2-0@tychen.cc

The real mmap and memfd_secret tests passed in the guest. The configuration
write check used a disposable file at the sysctl path because that kernel
does not enable Yama; it verifies the script's gating, not Yama enforcement.

Thanks,
Tianyi

Re: [PATCH v2 1/2] selftests/mm: Reject invalid test selections before running tests

[David Hildenbrand (Arm)]

On 9/8/26 11:55, Tianyi Chen wrote:
> getopts reports unknown options and missing arguments, but run_vmtests.sh
> ignores its error result and continues with test setup. An empty -t
> argument also falls back to the default selection, while unknown category
> names can silently select no tests and still reach setup code.
> 
> Exit on getopts errors and validate category names against the existing
> list in usage() before any test setup. Reject empty and whitespace-only
> selections, and normalize category separators so validation and execution
> agree. Keep the default selection for an unset or empty VM_SELFTEST_ITEMS
> environment variable, and preserve the existing standalone default value.
> 
> Fixes: 85463321e726 ("selftests/vm: enable running select groups of tests")
> Assisted-by: Codex:GPT-6
> Signed-off-by: Tianyi Chen <hi@tychen.cc>
> ---
>  tools/testing/selftests/mm/run_vmtests.sh | 27 +++++++++++++++++++----
>  1 file changed, 23 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
> index d09f9f6a384..836c7454cba 100755
> --- a/tools/testing/selftests/mm/run_vmtests.sh
> +++ b/tools/testing/selftests/mm/run_vmtests.sh
> @@ -96,26 +96,45 @@ separated by spaces:
>  
>  example: ./run_vmtests.sh -t "hmm mmap ksm"
>  EOF
> -	exit 0
>  }
>  
>  RUN_ALL=false
>  RUN_DESTRUCTIVE=false
>  TAP_PREFIX="# "
>  
> +# An unset or empty environment selection keeps the default behavior.
> +VM_SELFTEST_ITEMS=${VM_SELFTEST_ITEMS:-default}

Who would set it at this point?

Can't this just be

VM_SELFTEST_ITEMS="default"

> +
>  while getopts "aht:nd" OPT; do
>  	case ${OPT} in
>  		"a") RUN_ALL=true ;;
> -		"h") usage ;;
> +		"h") usage; exit 0 ;;
>  		"t") VM_SELFTEST_ITEMS=${OPTARG} ;;
>  		"n") TAP_PREFIX= ;;
>  		"d") RUN_DESTRUCTIVE=true ;;
> +		"?") exit 1 ;;
>  	esac
>  done
>  shift $((OPTIND -1))
>  
> -# default behavior: run all tests
> -VM_SELFTEST_ITEMS=${VM_SELFTEST_ITEMS:-default}
> +# Normalize whitespace so validation and test_selected() use the same names.
> +read -r -a selected_categories <<< "${VM_SELFTEST_ITEMS//$'\n'/ }"
> +VM_SELFTEST_ITEMS="${selected_categories[*]}"
> +if [ -z "$VM_SELFTEST_ITEMS" ]; then
> +	echo "No test categories specified" >&2
> +	exit 1
> +fi
> +
> +if [ "$VM_SELFTEST_ITEMS" != "default" ]; then
> +	# Keep the documented category list as the source of valid names.

That makes sense to me.

It's interesting what would happen when trying to run a test that does not even
exist for the architecture. Getting some feedback that the test was skipped
would be nice. But that's something for another day :)

-- 
Cheers,

David

Re: [PATCH v2 2/2] selftests/mm: Only prepare ptrace_scope when memfd_secret is selected

[David Hildenbrand (Arm)]

On 9/8/26 11:55, Tianyi Chen wrote:
> The memfd_secret setup clears ptrace_scope whenever its test binary is
> executable, even when a different category was selected. run_test() filters
> the test invocation, but it does not protect the preceding setup.
> 
> Check the category selection before entering the memfd_secret block so
> running unrelated categories does not change ptrace_scope. Keep the
> existing executable check and the behavior when memfd_secret is selected.
> 
> Assisted-by: Codex:GPT-6

We prefer

Assisted-by: LLM

now

> Signed-off-by: Tianyi Chen <hi@tychen.cc>
> ---
>  tools/testing/selftests/mm/run_vmtests.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
> index 836c7454cba..9932df252f4 100755
> --- a/tools/testing/selftests/mm/run_vmtests.sh
> +++ b/tools/testing/selftests/mm/run_vmtests.sh
> @@ -371,7 +371,7 @@ CATEGORY="process_madv" run_test ./process_madv
>  
>  CATEGORY="vma_merge" run_test ./merge
>  
> -if [ -x ./memfd_secret ]
> +if test_selected "memfd_secret" && [ -x ./memfd_secret ]
>  then
>  if [ -f /proc/sys/kernel/yama/ptrace_scope ]; then
>  	(echo 0 > /proc/sys/kernel/yama/ptrace_scope 2>&1) | tap_prefix

Acked-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH v2 0/2] selftests/mm: Validate selections and scope memfd_secret setup

[David Hildenbrand (Arm)]

On 9/8/26 11:55, Tianyi Chen wrote:
> run_vmtests.sh can reach test setup after an invalid category selection,
> and its memfd_secret preparation can change ptrace_scope even when that
> category was not selected. These two patches address David's follow-up
> questions on the earlier getopts fix.
> 
> Patch 1 extends the getopts error handling to reject empty, whitespace-only
> and unknown category selections before setup. It uses the existing category
> list in usage() and normalizes separators, while preserving the default
> when -t is omitted and the existing environment-selection behavior.
> 
> Patch 2 gates memfd_secret preparation on category selection and executable
> presence. It retains the existing behavior when memfd_secret is selected;
> it does not add configuration restoration or otherwise change that test.
> 
> Changes in v2:
> - Rebase onto current mm-unstable.
> - Extend the original option-error patch with category validation.
> - Add a separate patch for the memfd_secret setup side effect.
> 
> Focused validation:
> - Bash syntax and strict checkpatch passed. All 30 documented categories
>   match the categories used by the script.
> - Parser checks cover omitted/default selections, empty environment values,
>   explicit empty -t, invalid options/categories, whitespace-separated lists
>   and command-line overrides of environment selections.
> - Twelve invalid-input cases and help were run through the full script in
>   a KVM guest. Invalid inputs exited with status 1 before test setup.
> - The real mmap category's three programs and memfd_secret passed in an
>   x86-64 KVM guest running Linux 7.3.0-rc1, including multi-category and
>   environment-selected runs.
> - The guest kernel lacks Yama. A disposable file bind-mounted at the actual
>   ptrace_scope path verified the shell's write condition: the old script
>   writes for an unrelated selection, the new script does not, and selected
>   executable memfd_secret retains the write. This checks script gating,
>   not Yama policy enforcement. Missing-executable behavior was also checked.
> 
> This was focused validation, not a run of the complete MM selftest suite.
> 
> Previous submission:
> https://lore.kernel.org/r/20260906135444.749144-1-hi@tychen.cc
> Review:
> https://lore.kernel.org/r/519ef70a-679c-455f-ba5f-3841730873a4@kernel.org
> 
> 
> Tianyi Chen (2):

For the future, don't send new versions in reply to old versions of a patch set.

-- 
Cheers,

David

Re: [PATCH v2 1/2] selftests/mm: Reject invalid test selections before running tests

[Tianyi Chen]

Hi David,

> Who would set it at this point?
> Can't this just be
> VM_SELFTEST_ITEMS="default"

I kept the expansion to preserve the old script's implicit environment
override. I could not find an in-tree caller relying on it; the wrappers
use -t, which is also the documented interface. I have made the change
you suggested in v3. An explicit empty -t still fails validation.

I have also changed both trailers to Assisted-by: LLM and added your
Acked-by to patch 2. Thanks for the review and the ack.

Sorry about threading v2 under v1. V3 starts a new thread, and I will
keep future revisions separate as well:
https://lore.kernel.org/r/178886714088.169611.14957244271354428473.mm-cli-v3-0@tychen.cc

Thanks,
Tianyi