[PATCH 0/9] rtla/tests: Extend runtime test coverage

[PATCH 0/9] rtla/tests: Extend runtime test coverage

[Tomas Glozar]

This patchset introduces some new tests to cover more options, especially
histogram and thread options. Most of the new tests use positive and negative
output matches, sometimes in combination with action scripts, to verify that
RTLA is applying the settings correctly.

Tests were reorganized a little, adding two new sections: thread tests and
histogram tests, next to basic tests.

Additionally, coverage of existing tests is extended by adding new matches and
by extending tests to cover both top and hist tools where possible. For the
latter, new helpers check_top_hist and check_top_q_hist are added to engine.sh.

As part of the new action scripts, detection of measurement threads is made more
robust by following child processes of either RTLA (user workload) or kthreadd
(kernel workload) rather than grepping through the comms of all processes, which
might have lead to false positives.

These changes significantly improve test coverage and make the test suite more
against false positives from unrelated processes.

Tomas Glozar (9):
  rtla/tests: Cover both top and hist tools where possible
  rtla/tests: Add get_workload_pids() helper
  rtla/tests: Check -c/--cpus thread affinity
  rtla/tests: Use negative match when testing --aa-only
  rtla/tests: Extend timerlat top --aa-only coverage
  rtla/tests: Cover all hist options in runtime tests
  rtla/tests: Add runtime test for -H/--house-keeping
  rtla/tests: Add runtime test for -k and -u options
  rtla/tests: Add runtime tests for -C/--cgroup

 tools/tracing/rtla/tests/engine.sh            |  15 +++
 tools/tracing/rtla/tests/osnoise.t            |  73 +++++++----
 .../rtla/tests/scripts/check-cgroup-match.sh  |  17 +++
 .../tracing/rtla/tests/scripts/check-cpus.sh  |   9 ++
 .../tests/scripts/check-housekeeping-cpus.sh  |   4 +
 .../rtla/tests/scripts/check-priority.sh      |   8 +-
 .../scripts/check-user-kernel-threads.sh      |  16 +++
 .../tests/scripts/lib/get_workload_pids.sh    |  11 ++
 tools/tracing/rtla/tests/timerlat.t           | 113 +++++++++++-------
 9 files changed, 194 insertions(+), 72 deletions(-)
 create mode 100755 tools/tracing/rtla/tests/scripts/check-cgroup-match.sh
 create mode 100755 tools/tracing/rtla/tests/scripts/check-cpus.sh
 create mode 100755 tools/tracing/rtla/tests/scripts/check-housekeeping-cpus.sh
 create mode 100755 tools/tracing/rtla/tests/scripts/check-user-kernel-threads.sh
 create mode 100644 tools/tracing/rtla/tests/scripts/lib/get_workload_pids.sh

-- 
2.53.0

[PATCH 1/9] rtla/tests: Cover both top and hist tools where possible

[Tomas Glozar]

RTLA runtime tests currently do not cover both tool variants for osnoise
and timerlat properly. Many tests applicable to both tools are only
tested for one tool, selected randomly.

Introduce two new shell functions, check_top_hist() and
check_top_q_hist(). The functions use the same syntax as check() and run
check() on the arguments twice: once replacing the "TOOL" string in the
command with "top" (or "top -q"), once replacing it with "hist". The top
-q variant is used for tests relying on messages printed after aborting
the RTLA main loop with a starting new line, which only happens for top
tools in quiet mode; without -q, the top output is printed on the same
line and the matches would fail.

Tests that are applicable to both top and hist tools were modified to
the run for both; additionally, tests that were already done for both
tools were migrated to the new shell functions, unless the test command
or matches differ between the tools. Additional tests were added to test
tool-specific help messages.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/engine.sh  | 15 ++++++
 tools/tracing/rtla/tests/osnoise.t  | 46 +++++++++--------
 tools/tracing/rtla/tests/timerlat.t | 76 ++++++++++++++---------------
 3 files changed, 73 insertions(+), 64 deletions(-)

diff --git a/tools/tracing/rtla/tests/engine.sh b/tools/tracing/rtla/tests/engine.sh
index ed261e07c6d9..27d92f19a322 100644
--- a/tools/tracing/rtla/tests/engine.sh
+++ b/tools/tracing/rtla/tests/engine.sh
@@ -112,6 +112,21 @@ check_with_osnoise_options() {
 	NO_RESET_OSNOISE=1 check "$arg1" "$arg2" "$arg3"
 }
 
+check_top_hist() {
+	# Test one command with both "top" and "hist" tools, replacing "TOOL" in
+	# command with either "top" or "hist" respectively, and prefixing the test
+	# names with "top " and "hist ".
+	check "top $1" "$(echo "$2" | sed 's/TOOL/top/g')" "${@:3}"
+	check "hist $1" "$(echo "$2" | sed 's/TOOL/hist/g')" "${@:3}"
+}
+
+check_top_q_hist() {
+	# Same as above, but pass "-q" to top so that strings printed in main
+	# loop are on their own line for top too, not only for hist.
+	check "top $1" "$(echo "$2" | sed 's/TOOL/top -q/g')" "${@:3}"
+	check "hist $1" "$(echo "$2" | sed 's/TOOL/hist/g')" "${@:3}"
+}
+
 set_timeout() {
 	TIMEOUT="timeout -v -k 15s $1"
 }
diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index 396334608920..ce3a448b1f87 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -7,13 +7,15 @@ set_timeout 2m
 
 check "verify help page" \
 	"osnoise --help" 0 "osnoise version"
-check "verify the --priority/-P param" \
-	"osnoise top -P F:1 -c 0 -r 900000 -d 10s -q -S 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh osnoise/ SCHED_FIFO 1\"" \
+check_top_hist "verify help page" \
+	"osnoise TOOL --help" 0 "rtla osnoise"
+check_top_q_hist "verify the --priority/-P param" \
+	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh osnoise/ SCHED_FIFO 1\"" \
 	2 "Priorities are set correctly"
-check "verify the --stop/-s param" \
-	"osnoise top -s 30 -T 1" 2 "osnoise hit stop tracing"
-check "verify the  --trace param" \
-	"osnoise hist -s 30 -T 1 -t" 2 "Saving trace to osnoise_trace.txt"
+check_top_q_hist "verify the --stop/-s param" \
+	"osnoise TOOL -s 30 -T 1" 2 "osnoise hit stop tracing"
+check_top_q_hist "verify the --trace param" \
+	"osnoise TOOL -s 30 -T 1 -t" 2 "Saving trace to osnoise_trace.txt"
 check "verify the --entries/-E param" \
 	"osnoise hist -P F:1 -c 0 -r 900000 -d 10s -b 10 -E 25"
 
@@ -24,27 +26,23 @@ check_with_osnoise_options "apply default period" \
 	"osnoise hist -s 1" 2 period_us=600000000
 
 # Actions tests
-check "trace output through -t with custom filename" \
-	"osnoise hist -S 2 -t custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
-check "trace output through --on-threshold trace" \
-	"osnoise hist -S 2 --on-threshold trace" 2 "^  Saving trace to osnoise_trace.txt$"
-check "trace output through --on-threshold trace with custom filename" \
-	"osnoise hist -S 2 --on-threshold trace,file=custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
-check "exec command" \
-	"osnoise hist -S 2 --on-threshold shell,command='echo TestOutput'" 2 "^TestOutput$"
-check "multiple actions" \
-	"osnoise hist -S 2 --on-threshold shell,command='echo -n 1' --on-threshold shell,command='echo 2'" 2 "^12$"
+check_top_q_hist "trace output through -t with custom filename" \
+	"osnoise TOOL -S 2 -t custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
+check_top_q_hist "trace output through --on-threshold trace" \
+	"osnoise TOOL -S 2 --on-threshold trace" 2 "^  Saving trace to osnoise_trace.txt$"
+check_top_q_hist "trace output through --on-threshold trace with custom filename" \
+	"osnoise TOOL -S 2 --on-threshold trace,file=custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
+check_top_q_hist "exec command" \
+	"osnoise TOOL -S 2 --on-threshold shell,command='echo TestOutput'" 2 "^TestOutput$"
+check_top_q_hist "multiple actions" \
+	"osnoise TOOL -S 2 --on-threshold shell,command='echo -n 1' --on-threshold shell,command='echo 2'" 2 "^12$"
 check "hist stop at failed action" \
 	"osnoise hist -S 2 --on-threshold shell,command='echo -n 1; false' --on-threshold shell,command='echo -n 2'" 2 "^1# RTLA osnoise histogram$"
 check "top stop at failed action" \
 	"osnoise top -S 2 --on-threshold shell,command='echo -n abc; false' --on-threshold shell,command='echo -n defgh'" 2 "^abc" "defgh"
-check "hist with continue" \
-	"osnoise hist -S 2 -d 5s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$"
-check "top with continue" \
-	"osnoise top -q -S 2 -d 5s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$"
-check "hist with trace output at end" \
-	"osnoise hist -d 1s --on-end trace" 0 "^  Saving trace to osnoise_trace.txt$"
-check "top with trace output at end" \
-	"osnoise top -d 1s --on-end trace" 0 "^  Saving trace to osnoise_trace.txt$"
+check_top_q_hist "with continue" \
+	"osnoise TOOL -S 2 -d 5s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$"
+check_top_hist "with trace output at end" \
+	"osnoise TOOL -d 1s --on-end trace" 0 "^  Saving trace to osnoise_trace.txt$"
 
 test_end
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index fd4935fd7b49..d7944710a859 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -22,64 +22,60 @@ export RTLA_NO_BPF=$option
 # Basic tests
 check "verify help page" \
 	"timerlat --help" 0 "timerlat version"
-check "verify -s/--stack" \
-	"timerlat top -s 3 -T 10 -t" 2 "Blocking thread stack trace"
-check "verify -P/--priority" \
-	"timerlat top -P F:1 -c 0 -d 10s -q -T 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh timerlatu/ SCHED_FIFO 1\"" \
+check_top_hist "verify help page" \
+	"timerlat TOOL --help" 0 "rtla timerlat"
+check_top_hist "verify -s/--stack" \
+	"timerlat TOOL -s 3 -T 10 -t" 2 "Blocking thread stack trace"
+check_top_hist "verify -P/--priority" \
+	"timerlat TOOL -P F:1 -c 0 -d 10s -T 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh timerlatu/ SCHED_FIFO 1\"" \
 	2 "Priorities are set correctly"
-check "test in nanoseconds" \
-	"timerlat top -i 2 -c 0 -n -d 10s" 2 "ns"
-check "set the automatic trace mode" \
-	"timerlat top -a 5" 2 "analyzing it"
-check "dump tasks" \
-	"timerlat top -a 5 --dump-tasks" 2 "Printing CPU tasks"
+check_top_hist "test in nanoseconds" \
+	"timerlat TOOL -i 2 -c 0 -n -d 10s" 2 "ns"
+check_top_hist "set the automatic trace mode" \
+	"timerlat TOOL -a 5" 2 "analyzing it"
+check_top_hist "dump tasks" \
+	"timerlat TOOL -a 5 --dump-tasks" 2 "Printing CPU tasks"
 check "print the auto-analysis if hits the stop tracing condition" \
 	"timerlat top --aa-only 5" 2
-check "disable auto-analysis" \
-	"timerlat top -s 3 -T 10 -t --no-aa" 2
-check "verify -c/--cpus" \
-	"timerlat hist -c 0 -d 10s"
-check "hist test in nanoseconds" \
-	"timerlat hist -i 2 -c 0 -n -d 10s" 2 "ns"
+check_top_hist "disable auto-analysis" \
+	"timerlat TOOL -s 3 -T 10 -t --no-aa" 2
+check_top_hist "verify -c/--cpus" \
+	"timerlat TOOL -c 0 -d 10s"
 
 # Actions tests
-check "trace output through -t" \
-	"timerlat hist -T 2 -t" 2 "^  Saving trace to timerlat_trace.txt$"
-check "trace output through -t with custom filename" \
-	"timerlat hist -T 2 -t custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
-check "trace output through --on-threshold trace" \
-	"timerlat hist -T 2 --on-threshold trace" 2 "^  Saving trace to timerlat_trace.txt$"
-check "trace output through --on-threshold trace with custom filename" \
-	"timerlat hist -T 2 --on-threshold trace,file=custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
-check "exec command" \
-	"timerlat hist -T 2 --on-threshold shell,command='echo TestOutput'" 2 "^TestOutput$"
-check "multiple actions" \
-	"timerlat hist -T 2 --on-threshold shell,command='echo -n 1' --on-threshold shell,command='echo 2'" 2 "^12$"
+check_top_q_hist "trace output through -t" \
+	"timerlat TOOL -T 2 -t" 2 "^  Saving trace to timerlat_trace.txt$"
+check_top_q_hist "trace output through -t with custom filename" \
+	"timerlat TOOL -T 2 -t custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
+check_top_q_hist "trace output through --on-threshold trace" \
+	"timerlat TOOL -T 2 --on-threshold trace" 2 "^  Saving trace to timerlat_trace.txt$"
+check_top_q_hist "trace output through --on-threshold trace with custom filename" \
+	"timerlat TOOL -T 2 --on-threshold trace,file=custom_filename.txt" 2 "^  Saving trace to custom_filename.txt$"
+check_top_q_hist "exec command" \
+	"timerlat TOOL -T 2 --on-threshold shell,command='echo TestOutput'" 2 "^TestOutput$"
+check_top_q_hist "multiple actions" \
+	"timerlat TOOL -T 2 --on-threshold shell,command='echo -n 1' --on-threshold shell,command='echo 2'" 2 "^12$"
 check "hist stop at failed action" \
 	"timerlat hist -T 2 --on-threshold shell,command='echo -n 1; false' --on-threshold shell,command='echo -n 2'" 2 "^1# RTLA timerlat histogram$"
 check "top stop at failed action" \
 	"timerlat top -T 2 --on-threshold shell,command='echo -n abc; false' --on-threshold shell,command='echo -n defgh'" 2 "^abc" "defgh"
-check "hist with continue" \
-	"timerlat hist -T 2 -d 5s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$"
-check "top with continue" \
-	"timerlat top -q -T 2 -d 5s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$"
-check "hist with trace output at end" \
-	"timerlat hist -d 1s --on-end trace" 0 "^  Saving trace to timerlat_trace.txt$"
-check "top with trace output at end" \
-	"timerlat top -d 1s --on-end trace" 0 "^  Saving trace to timerlat_trace.txt$"
+check_top_q_hist "with continue" \
+	"timerlat TOOL -T 2 -d 5s --on-threshold shell,command='echo TestOutput' --on-threshold continue" 0 "^TestOutput$"
+check_top_hist "with trace output at end" \
+	"timerlat TOOL -d 1s --on-end trace" 0 "^  Saving trace to timerlat_trace.txt$"
 
 # BPF action program tests
 if [ "$option" -eq 0 ]
 then
 	# Test BPF action program properly in BPF mode
 	[ -z "$BPFTOOL" ] && BPFTOOL=bpftool
-	check "hist with BPF action program (BPF mode)" \
-		"timerlat hist -T 2 --bpf-action tests/bpf/bpf_action_map.o --on-threshold shell,command='$BPFTOOL map dump name rtla_test_map'" \
+	check_top_q_hist "with BPF action program (BPF mode)" \
+		"timerlat TOOL -T 2 --bpf-action tests/bpf/bpf_action_map.o --on-threshold shell,command='$BPFTOOL map dump name rtla_test_map'" \
 		2 '"value": 42'
 else
 	# Test BPF action program failure in non-BPF mode
-	check "hist with BPF action program (non-BPF mode)" \
-		"timerlat hist -T 2 --bpf-action tests/bpf/bpf_action_map.o" \
+	check_top_q_hist "with BPF action program (non-BPF mode)" \
+		"timerlat TOOL -T 2 --bpf-action tests/bpf/bpf_action_map.o" \
 		1 "BPF actions are not supported in tracefs-only mode"
 fi
 done
-- 
2.53.0

[PATCH 2/9] rtla/tests: Add get_workload_pids() helper

[Tomas Glozar]

RTLA runtime tests that check workload processes (currently the test
case "verify -P/--priority" of timerlat.t and "verify the --priority/-P
param" of osnoise.t) use "pgrep timerlatu/" or "pgrep osnoise/"
respectively to identify the workload.

Make them more robust by adding a get_workload_pids() helper that
finds the main rtla process and returns the PIDs of all siblings other
than the test script itself, plus all child processes of kthreadd that
have the osnoise/timerlat kthread pattern comm.

This filters out any spurious processes not related to the running test
that happen to have "timerlatu/" or "osnoise/" in their command, for
example, a user grepping the same names at the time of the running of
the test.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/osnoise.t                    |  2 +-
 tools/tracing/rtla/tests/scripts/check-priority.sh    |  8 ++++----
 .../rtla/tests/scripts/lib/get_workload_pids.sh       | 11 +++++++++++
 tools/tracing/rtla/tests/timerlat.t                   |  2 +-
 4 files changed, 17 insertions(+), 6 deletions(-)
 create mode 100644 tools/tracing/rtla/tests/scripts/lib/get_workload_pids.sh

diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index ce3a448b1f87..ed6ff0cc3329 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -10,7 +10,7 @@ check "verify help page" \
 check_top_hist "verify help page" \
 	"osnoise TOOL --help" 0 "rtla osnoise"
 check_top_q_hist "verify the --priority/-P param" \
-	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh osnoise/ SCHED_FIFO 1\"" \
+	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
 	2 "Priorities are set correctly"
 check_top_q_hist "verify the --stop/-s param" \
 	"osnoise TOOL -s 30 -T 1" 2 "osnoise hit stop tracing"
diff --git a/tools/tracing/rtla/tests/scripts/check-priority.sh b/tools/tracing/rtla/tests/scripts/check-priority.sh
index 79b702a34a96..b51d5232a868 100755
--- a/tools/tracing/rtla/tests/scripts/check-priority.sh
+++ b/tools/tracing/rtla/tests/scripts/check-priority.sh
@@ -1,8 +1,8 @@
 #!/bin/bash
 # SPDX-License-Identifier: GPL-2.0
-pids="$(pgrep ^$1)" || exit 1
-for pid in $pids
+. "$(dirname $0)/lib/get_workload_pids.sh"
+for pid in $(get_workload_pids)
 do
-  chrt -p $pid | cut -d ':' -f 2 | head -n1 | grep "^ $2\$" >/dev/null
-  chrt -p $pid | cut -d ':' -f 2 | tail -n1 | grep "^ $3\$" >/dev/null
+  chrt -p $pid | cut -d ':' -f 2 | head -n1 | grep "^ $1\$" >/dev/null
+  chrt -p $pid | cut -d ':' -f 2 | tail -n1 | grep "^ $2\$" >/dev/null
 done && echo "Priorities are set correctly"
diff --git a/tools/tracing/rtla/tests/scripts/lib/get_workload_pids.sh b/tools/tracing/rtla/tests/scripts/lib/get_workload_pids.sh
new file mode 100644
index 000000000000..8aff98cd2c1f
--- /dev/null
+++ b/tools/tracing/rtla/tests/scripts/lib/get_workload_pids.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+get_workload_pids() {
+    local shell_pid=$$
+    local rtla_pid=$(ps -o ppid= $shell_pid)
+
+    # kernel threads
+    pgrep -P $(pgrep ^kthreadd$) -f '^(osnoise|timerlat)/[0-9]+$'
+    # user threads
+    pgrep -P $rtla_pid | grep -v "^$shell_pid$"
+}
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index d7944710a859..765dffd9d42a 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -27,7 +27,7 @@ check_top_hist "verify help page" \
 check_top_hist "verify -s/--stack" \
 	"timerlat TOOL -s 3 -T 10 -t" 2 "Blocking thread stack trace"
 check_top_hist "verify -P/--priority" \
-	"timerlat TOOL -P F:1 -c 0 -d 10s -T 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh timerlatu/ SCHED_FIFO 1\"" \
+	"timerlat TOOL -P F:1 -c 0 -d 10s -T 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
 	2 "Priorities are set correctly"
 check_top_hist "test in nanoseconds" \
 	"timerlat TOOL -i 2 -c 0 -n -d 10s" 2 "ns"
-- 
2.53.0

[PATCH 3/9] rtla/tests: Check -c/--cpus thread affinity

[Tomas Glozar]

RTLA runtime tests verify the -c/--cpus options, but do not check
whether the correct affinity is actually applied.

Add a script named check-cpus.sh that retrieves the affinity of all
workload threads and use it to check the -c/--cpus option for both
osnoise and timerlat tools.

Also add missing -c/--cpus test for osnoise.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/osnoise.t             | 2 ++
 tools/tracing/rtla/tests/scripts/check-cpus.sh | 9 +++++++++
 tools/tracing/rtla/tests/timerlat.t            | 4 ++--
 3 files changed, 13 insertions(+), 2 deletions(-)
 create mode 100755 tools/tracing/rtla/tests/scripts/check-cpus.sh

diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index ed6ff0cc3329..5edffb23981b 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -18,6 +18,8 @@ check_top_q_hist "verify the --trace param" \
 	"osnoise TOOL -s 30 -T 1 -t" 2 "Saving trace to osnoise_trace.txt"
 check "verify the --entries/-E param" \
 	"osnoise hist -P F:1 -c 0 -r 900000 -d 10s -b 10 -E 25"
+check_top_q_hist "verify the -c/--cpus param" \
+	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 
 # Test setting default period by putting an absurdly high period
 # and stopping on threshold.
diff --git a/tools/tracing/rtla/tests/scripts/check-cpus.sh b/tools/tracing/rtla/tests/scripts/check-cpus.sh
new file mode 100755
index 000000000000..0b016d4a7945
--- /dev/null
+++ b/tools/tracing/rtla/tests/scripts/check-cpus.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+. "$(dirname $0)/lib/get_workload_pids.sh"
+echo -n "Affinity of threads: "
+for pid in $(get_workload_pids)
+do
+    echo -n $(taskset -c -p $pid | cut -d ':' -f 2)
+done
+echo
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index 765dffd9d42a..fb60022aaa64 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -39,8 +39,8 @@ check "print the auto-analysis if hits the stop tracing condition" \
 	"timerlat top --aa-only 5" 2
 check_top_hist "disable auto-analysis" \
 	"timerlat TOOL -s 3 -T 10 -t --no-aa" 2
-check_top_hist "verify -c/--cpus" \
-	"timerlat TOOL -c 0 -d 10s"
+check_top_q_hist "verify -c/--cpus" \
+	"timerlat TOOL -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 
 # Actions tests
 check_top_q_hist "trace output through -t" \
-- 
2.53.0

[PATCH 4/9] rtla/tests: Use negative match when testing --aa-only

[Tomas Glozar]

For testing the -a/--auto option in timerlat tool, the string "analyzing
it" is matched against to make sure auto-analysis was triggered.

Use the same string as a negative match for --aa-only option test.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/timerlat.t | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index fb60022aaa64..f47a82c115c7 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -38,7 +38,7 @@ check_top_hist "dump tasks" \
 check "print the auto-analysis if hits the stop tracing condition" \
 	"timerlat top --aa-only 5" 2
 check_top_hist "disable auto-analysis" \
-	"timerlat TOOL -s 3 -T 10 -t --no-aa" 2
+	"timerlat TOOL -s 3 -T 10 -t --no-aa" 2 "" "analyzing it"
 check_top_q_hist "verify -c/--cpus" \
 	"timerlat TOOL -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 
-- 
2.53.0

[PATCH 5/9] rtla/tests: Extend timerlat top --aa-only coverage

[Tomas Glozar]

rtla-timerlat-top's --aa-only option is currently only tested for return
value.

Extend the tests to also check that only auto-analysis is being done via
a negative match for the "Timer Latency" text in the top header, and
further split the test case into two:

- one test case for --aa-only stopping on threshold
- one test case for --aa-only exiting without threshold being hit

For both cases, the expected output ("analyzing it" or "Max latency was"
respectively) is checked against in addition to the negative match.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/timerlat.t | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index f47a82c115c7..28c01d8b299d 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -35,8 +35,10 @@ check_top_hist "set the automatic trace mode" \
 	"timerlat TOOL -a 5" 2 "analyzing it"
 check_top_hist "dump tasks" \
 	"timerlat TOOL -a 5 --dump-tasks" 2 "Printing CPU tasks"
-check "print the auto-analysis if hits the stop tracing condition" \
-	"timerlat top --aa-only 5" 2
+check "verify --aa-only stop on threshold" \
+	"timerlat top --aa-only 5" 2 "analyzing it" "Timer Latency"
+check "verify --aa-only max latency" \
+	"timerlat top --aa-only 2000000 -d 1s" 0 "^  Max latency was" "Timer Latency"
 check_top_hist "disable auto-analysis" \
 	"timerlat TOOL -s 3 -T 10 -t --no-aa" 2 "" "analyzing it"
 check_top_q_hist "verify -c/--cpus" \
-- 
2.53.0

[PATCH 6/9] rtla/tests: Cover all hist options in runtime tests

[Tomas Glozar]

Cover all options regarding histogram formatting for both
rtla-osnoise-hist and rtla-timerlat-hist tools. All options also have
output checking using positive or negative match, except for
-b/--bucket-size and -E/--entries, which cannot be tested in isolated
due to the output depending on the actual data collected.

Old -E/--entries test for rtla-osnoise was replaced with a new one
equivalent to the timerlat one.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/osnoise.t  | 18 ++++++++++++++++--
 tools/tracing/rtla/tests/timerlat.t | 20 ++++++++++++++++++++
 2 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index 5edffb23981b..773a46e2dc5f 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -16,11 +16,25 @@ check_top_q_hist "verify the --stop/-s param" \
 	"osnoise TOOL -s 30 -T 1" 2 "osnoise hit stop tracing"
 check_top_q_hist "verify the --trace param" \
 	"osnoise TOOL -s 30 -T 1 -t" 2 "Saving trace to osnoise_trace.txt"
-check "verify the --entries/-E param" \
-	"osnoise hist -P F:1 -c 0 -r 900000 -d 10s -b 10 -E 25"
 check_top_q_hist "verify the -c/--cpus param" \
 	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 
+# Histogram tests
+check "hist with -b/--bucket-size" \
+	"osnoise hist -b 1 -d 1s"
+check "hist with -E/--entries" \
+	"osnoise hist -E 10 -d 1s"
+check "hist with -E/--entries out of range" \
+	"osnoise hist -E 1 -d 1s" 1 "^Entries must be > 10 and < 9999999$"
+check "hist with --no-header" \
+	"osnoise hist --no-header -d 1s" 0 "" "RTLA osnoise histogram"
+check "hist with --with-zeros" \
+	"osnoise hist --with-zeros -b 100000 -E 21 -d 1s" 0 '^2000000\s+0\s+'
+check "hist with --no-index" \
+	"osnoise hist --no-index --with-zeros -d 1s" 0 "" "^count:"
+check "hist with --no-summary" \
+	"osnoise hist --no-summary -d 1s" 0 "" "^count:"
+
 # Test setting default period by putting an absurdly high period
 # and stopping on threshold.
 # If default period is not set, this will time out.
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index 28c01d8b299d..a14d9ec32ede 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -44,6 +44,26 @@ check_top_hist "disable auto-analysis" \
 check_top_q_hist "verify -c/--cpus" \
 	"timerlat TOOL -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 
+# Histogram tests
+check "hist with -b/--bucket-size" \
+	"timerlat hist -b 1 -d 1s"
+check "hist with -E/--entries" \
+	"timerlat hist -E 10 -d 1s"
+check "hist with -E/--entries out of range" \
+	"timerlat hist -E 1 -d 1s" 1 "^Entries must be > 10 and < 9999999$"
+check "hist with --no-header" \
+	"timerlat hist --no-header -d 1s" 0 "" "RTLA timerlat histogram"
+check "hist with --with-zeros" \
+	"timerlat hist --with-zeros -b 100000 -E 21 -d 1s" 0 '^2000000\s+0\s+'
+check "hist with --no-index" \
+	"timerlat hist --no-index --with-zeros -d 1s" 0 "" "^count:"
+check "hist with --no-summary" \
+	"timerlat hist --no-summary -d 1s" 0 "" "^ALL:"
+check "hist with --no-irq" \
+	"timerlat hist --no-irq -d 1s" 0 "" "IRQ-"
+check "hist with --no-thread" \
+	"timerlat hist --no-thread -d 1s" 0 "" "Thr-"
+
 # Actions tests
 check_top_q_hist "trace output through -t" \
 	"timerlat TOOL -T 2 -t" 2 "^  Saving trace to timerlat_trace.txt$"
-- 
2.53.0

[PATCH 7/9] rtla/tests: Add runtime test for -H/--house-keeping

[Tomas Glozar]

Add a runtime test for -H/--house-keeping option for both osnoise and
timerlat tools, with affinity checking similar to what is done for
-c/--cpus.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/osnoise.t                          | 2 ++
 tools/tracing/rtla/tests/scripts/check-housekeeping-cpus.sh | 4 ++++
 tools/tracing/rtla/tests/timerlat.t                         | 2 ++
 3 files changed, 8 insertions(+)
 create mode 100755 tools/tracing/rtla/tests/scripts/check-housekeeping-cpus.sh

diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index 773a46e2dc5f..cdea84914345 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -18,6 +18,8 @@ check_top_q_hist "verify the --trace param" \
 	"osnoise TOOL -s 30 -T 1 -t" 2 "Saving trace to osnoise_trace.txt"
 check_top_q_hist "verify the -c/--cpus param" \
 	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
+check_top_q_hist "verify the -H/--house-keeping param" \
+	"osnoise TOOL -P F:1 -H 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=tests/scripts/check-housekeeping-cpus.sh" 2 "^Affinity of threads: 0$"
 
 # Histogram tests
 check "hist with -b/--bucket-size" \
diff --git a/tools/tracing/rtla/tests/scripts/check-housekeeping-cpus.sh b/tools/tracing/rtla/tests/scripts/check-housekeeping-cpus.sh
new file mode 100755
index 000000000000..4742f34efb49
--- /dev/null
+++ b/tools/tracing/rtla/tests/scripts/check-housekeeping-cpus.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+pid=$(ps -o ppid= $$)
+echo "Affinity of threads:$(taskset -c -p $pid | cut -d ':' -f 2)"
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index a14d9ec32ede..20f68bcbcb27 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -43,6 +43,8 @@ check_top_hist "disable auto-analysis" \
 	"timerlat TOOL -s 3 -T 10 -t --no-aa" 2 "" "analyzing it"
 check_top_q_hist "verify -c/--cpus" \
 	"timerlat TOOL -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
+check_top_q_hist "verify -H/--house-keeping" \
+	"timerlat TOOL -H 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-housekeeping-cpus.sh" 2 "^Affinity of threads: 0$"
 
 # Histogram tests
 check "hist with -b/--bucket-size" \
-- 
2.53.0

[PATCH 8/9] rtla/tests: Add runtime test for -k and -u options

[Tomas Glozar]

Add runtime test for rtla-timerlat's -k/--kernel-threads and
-u/--user-threads options using get_workload_pids.sh to check whether
the appropriate threads are being created.

The tests are implemented for both top and hist. Additionally, all tests
related to timerlat threads are moved to a separate section in the test
files. The latter is also done for rtla-osnoise tests.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/osnoise.t               |  8 +++++---
 .../tests/scripts/check-user-kernel-threads.sh   | 16 ++++++++++++++++
 tools/tracing/rtla/tests/timerlat.t              | 12 +++++++++---
 3 files changed, 30 insertions(+), 6 deletions(-)
 create mode 100755 tools/tracing/rtla/tests/scripts/check-user-kernel-threads.sh

diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index cdea84914345..d0b623233db5 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -9,13 +9,15 @@ check "verify help page" \
 	"osnoise --help" 0 "osnoise version"
 check_top_hist "verify help page" \
 	"osnoise TOOL --help" 0 "rtla osnoise"
-check_top_q_hist "verify the --priority/-P param" \
-	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
-	2 "Priorities are set correctly"
 check_top_q_hist "verify the --stop/-s param" \
 	"osnoise TOOL -s 30 -T 1" 2 "osnoise hit stop tracing"
 check_top_q_hist "verify the --trace param" \
 	"osnoise TOOL -s 30 -T 1 -t" 2 "Saving trace to osnoise_trace.txt"
+
+# Thread tests
+check_top_q_hist "verify the --priority/-P param" \
+	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
+	2 "Priorities are set correctly"
 check_top_q_hist "verify the -c/--cpus param" \
 	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 check_top_q_hist "verify the -H/--house-keeping param" \
diff --git a/tools/tracing/rtla/tests/scripts/check-user-kernel-threads.sh b/tools/tracing/rtla/tests/scripts/check-user-kernel-threads.sh
new file mode 100755
index 000000000000..bb7ac510a735
--- /dev/null
+++ b/tools/tracing/rtla/tests/scripts/check-user-kernel-threads.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+. "$(dirname $0)/lib/get_workload_pids.sh"
+kthreadd_pid=$(pgrep ^kthreadd$)
+cnt_kernel=0
+cnt_user=0
+for pid in $(get_workload_pids)
+do
+    if [ "$(echo $(ps -o ppid= $pid))" = "$kthreadd_pid" ]
+    then
+        ((++cnt_kernel))
+    else
+        ((++cnt_user))
+    fi
+done
+echo "$cnt_kernel kernel threads, $cnt_user user threads"
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index 20f68bcbcb27..3557adbdebae 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -26,9 +26,6 @@ check_top_hist "verify help page" \
 	"timerlat TOOL --help" 0 "rtla timerlat"
 check_top_hist "verify -s/--stack" \
 	"timerlat TOOL -s 3 -T 10 -t" 2 "Blocking thread stack trace"
-check_top_hist "verify -P/--priority" \
-	"timerlat TOOL -P F:1 -c 0 -d 10s -T 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
-	2 "Priorities are set correctly"
 check_top_hist "test in nanoseconds" \
 	"timerlat TOOL -i 2 -c 0 -n -d 10s" 2 "ns"
 check_top_hist "set the automatic trace mode" \
@@ -41,10 +38,19 @@ check "verify --aa-only max latency" \
 	"timerlat top --aa-only 2000000 -d 1s" 0 "^  Max latency was" "Timer Latency"
 check_top_hist "disable auto-analysis" \
 	"timerlat TOOL -s 3 -T 10 -t --no-aa" 2 "" "analyzing it"
+
+# Thread tests
+check_top_hist "verify -P/--priority" \
+	"timerlat TOOL -P F:1 -c 0 -d 10s -T 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
+	2 "Priorities are set correctly"
 check_top_q_hist "verify -c/--cpus" \
 	"timerlat TOOL -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 check_top_q_hist "verify -H/--house-keeping" \
 	"timerlat TOOL -H 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-housekeeping-cpus.sh" 2 "^Affinity of threads: 0$"
+check_top_q_hist "verify -k/--kernel-threads" \
+	"timerlat TOOL -k -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-user-kernel-threads.sh" 2 "1 kernel threads, 0 user threads"
+check_top_q_hist "verify -u/--user-threads" \
+	"timerlat TOOL -u -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-user-kernel-threads.sh" 2 "0 kernel threads, 1 user threads"
 
 # Histogram tests
 check "hist with -b/--bucket-size" \
-- 
2.53.0

[PATCH 9/9] rtla/tests: Add runtime tests for -C/--cgroup

[Tomas Glozar]

Add a new script check-cgroup-match.sh that retrieves the cgroup of the
main rtla process and compares it to the cgroup of the rtla workload
threads.

Add a new test based on this script, for both osnoise and timerlat
tools, testing the variant of -C without argument (which sets the cgroup
of the workload to the cgroup of the rtla main process).

Note that this has to be tested in kernel mode to be significant for
timerlat tool, as user workloads inherit the parent rtla process cgroup
even without the option.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/tests/osnoise.t              |  3 +++
 .../rtla/tests/scripts/check-cgroup-match.sh    | 17 +++++++++++++++++
 tools/tracing/rtla/tests/timerlat.t             |  3 +++
 3 files changed, 23 insertions(+)
 create mode 100755 tools/tracing/rtla/tests/scripts/check-cgroup-match.sh

diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index d0b623233db5..06787471d0e8 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -18,6 +18,9 @@ check_top_q_hist "verify the --trace param" \
 check_top_q_hist "verify the --priority/-P param" \
 	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
 	2 "Priorities are set correctly"
+check_top_q_hist "verify the -C/--cgroup param" \
+	"osnoise TOOL -C -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=\"tests/scripts/check-cgroup-match.sh\"" \
+	2 "cgroup matches for all workload PIDs"
 check_top_q_hist "verify the -c/--cpus param" \
 	"osnoise TOOL -P F:1 -c 0 -r 900000 -d 10s -S 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 check_top_q_hist "verify the -H/--house-keeping param" \
diff --git a/tools/tracing/rtla/tests/scripts/check-cgroup-match.sh b/tools/tracing/rtla/tests/scripts/check-cgroup-match.sh
new file mode 100755
index 000000000000..fdc2c68c5957
--- /dev/null
+++ b/tools/tracing/rtla/tests/scripts/check-cgroup-match.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+. "$(dirname $0)/lib/get_workload_pids.sh"
+rtla_pid=$(echo $(ps -o ppid= $$))
+rtla_cgroup=$(</proc/$rtla_pid/cgroup)
+echo "RTLA cgroup: $rtla_cgroup"
+for pid in $(get_workload_pids)
+do
+    pid_cgroup=$(</proc/$pid/cgroup)
+    echo "PID $pid cgroup: $pid_cgroup"
+    if ! [ "$pid_cgroup" = "$rtla_cgroup" ]
+    then
+        echo "Mismatch!"
+        exit 0
+    fi
+done
+echo "cgroup matches for all workload PIDs"
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index 3557adbdebae..3ebfe316b39e 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -43,6 +43,9 @@ check_top_hist "disable auto-analysis" \
 check_top_hist "verify -P/--priority" \
 	"timerlat TOOL -P F:1 -c 0 -d 10s -T 1 --on-threshold shell,command=\"tests/scripts/check-priority.sh SCHED_FIFO 1\"" \
 	2 "Priorities are set correctly"
+check_top_hist "verify -C/--cgroup" \
+	"timerlat TOOL -k -C -c 0 -d 10s -T 1 --on-threshold shell,command=\"tests/scripts/check-cgroup-match.sh\"" \
+	2 "cgroup matches for all workload PIDs"
 check_top_q_hist "verify -c/--cpus" \
 	"timerlat TOOL -c 0 -d 10s -T 1 --on-threshold shell,command=tests/scripts/check-cpus.sh" 2 "^Affinity of threads: 0$"
 check_top_q_hist "verify -H/--house-keeping" \
-- 
2.53.0