[PATCH stalld 00/36] tests: Replace timing-dependent synchronization with event-driven helpers

[PATCH stalld 00/36] tests: Replace timing-dependent synchronization with event-driven helpers

[Wander Lairson Costa]

The stalld test suite relies heavily on hardcoded sleep calls for
synchronization: waiting for stalld to initialize, for starvation
to be detected, and for boosting to occur. These fixed delays make
the tests both slow (sleeping the full duration even when the event
happens instantly) and unreliable (too short on slow systems, too
long on fast ones).

This series replaces approximately 120 sleep calls across 20 test
files with event-driven helpers that return as soon as the expected
event occurs. The core mechanism is wait_for_log_message(), which
uses process substitution with tail -f to detect log patterns
instantly rather than polling. On top of this, domain-specific
helpers like wait_for_starvation_detected(), wait_for_boost_detected(),
start_starvation_gen(), and start_stalld_with_log() encapsulate
common synchronization patterns into reusable functions with proper
readiness detection and cleanup.

The series also introduces pass() and fail() helpers to standardize
test result reporting, replaces duplicated stalld invocation
boilerplate with shared helpers, and fixes several pre-existing test
bugs exposed by the tighter synchronization: invalid argument tests
that never actually failed, a multi-CPU detection grep that was too
restrictive, and FIFO fallback behavior that was incorrectly treated
as an error. A stalld source fix is included to die on invalid CPU
affinity, which was silently ignored.

Clark Williams (1):
  tests: Add pre-test and post-test cleanup of stalld processes

Wander Lairson Costa (35):
  tests/helpers: Fix stalld daemon detection in start_stalld()
  tests/helpers: Remove duplicate log() function
  tests: Add per-test runtime measurement to run_tests.sh
  tests/functional: Fix and refactor test_backend_selection.sh
  tests/functional: Fix test_logging_destinations.sh path and backend
  tests/helpers: Replace sleep with poll in start_stalld_with_log()
  tests/helpers: Fix stop_stalld() timeout and shutdown logic
  tests/helpers: Fix relative path in backend detection functions
  tests/functional: Remove redundant post-stop_stalld sleeps
  tests/functional: Fix false positive log matching in
    test_logging_destinations
  tests/helpers: Rewrite wait_for_log_message() with process
    substitution
  tests/helpers: Add wait_for_stalld_ready() and use in
    start_stalld_with_log()
  tests/helpers: Fix fractional sleep timeout bugs
  tests/helpers: Flush stdout after starvation_gen startup messages
  tests/helpers: Add start_starvation_gen() helper function
  tests/helpers: Add wait_for_starvation_detected() and
    wait_for_boost_detected()
  tests/functional: Use start_starvation_gen() helper
  tests/functional: Replace detection sleeps with event-driven helpers
  tests/functional: Remove duplicated -a flag in
    test_fifo_priority_starvation
  tests/functional: Add missing -a flag in test_starvation_detection
  tests/functional: Use start_stalld_with_log() in test_log_only
  tests/functional: Use start_stalld_with_log() in
    test_logging_destinations
  tests/functional: Use start_stalld_with_log() in test_cpu_selection
  tests/functional: Use wait_for_stalld_ready() in
    test_backend_selection
  tests/functional: Use timeout for error path in test_force_fifo
  tests/functional: Use timeout for invalid argument tests
  tests: Add pass() helper and replace assert_equals hack
  tests/functional: Use pass() for all test pass reporting
  tests: Add fail() helper and use for all test failures
  tests/helpers: Use pass()/fail() in assert functions
  tests/functional: Fix multi-CPU detection in test_starvation_detection
  tests/functional: Accept FIFO fallback in test_fifo_boosting
  tests/functional: Fix readiness detection and FIFO fallback in
    test_force_fifo
  tests/functional: Fix invalid pidfile test in test_pidfile
  stalld: die on invalid CPU affinity

 src/stalld.c                                  |   6 +-
 tests/functional/test_affinity.sh             |  43 +--
 tests/functional/test_backend_selection.sh    | 193 +++-------
 tests/functional/test_boost_duration.sh       | 118 ++----
 tests/functional/test_boost_period.sh         | 116 ++----
 tests/functional/test_boost_restoration.sh    |  79 ++--
 tests/functional/test_boost_runtime.sh        | 132 ++-----
 tests/functional/test_cpu_selection.sh        |  79 +---
 tests/functional/test_deadline_boosting.sh    | 116 +++---
 tests/functional/test_fifo_boosting.sh        | 133 +++----
 .../test_fifo_priority_starvation.sh          | 106 ++----
 tests/functional/test_force_fifo.sh           | 101 ++---
 tests/functional/test_foreground.sh           |  18 +-
 tests/functional/test_idle_detection.sh       |  40 +-
 tests/functional/test_log_only.sh             |  46 +--
 tests/functional/test_logging_destinations.sh |  51 ++-
 tests/functional/test_pidfile.sh              |  75 ++--
 tests/functional/test_runqueue_parsing.sh     | 118 +++---
 tests/functional/test_starvation_detection.sh | 136 +++----
 tests/functional/test_starvation_threshold.sh |  98 ++---
 tests/functional/test_task_merging.sh         |  69 ++--
 tests/helpers/starvation_gen.c                |   1 +
 tests/helpers/test_helpers.sh                 | 351 ++++++++++++------
 tests/run_tests.sh                            |  61 ++-
 24 files changed, 905 insertions(+), 1381 deletions(-)

-- 
2.53.0

[PATCH stalld 01/36] tests: Add pre-test and post-test cleanup of stalld processes

[Wander Lairson Costa]

From: Clark Williams <williams@redhat.com>

Add kill_existing_stalld() function to test_helpers.sh and
kill_existing_stalld_processes() function to run_tests.sh to ensure
no stale stalld processes from previous test runs interfere with new
tests. This prevents orphaned processes from interrupted test runs
from causing false negatives or unexpected behavior.

Changes:
- test_helpers.sh: Add kill_existing_stalld() with graceful then
  forced shutdown, export the function, and call from
  setup_test_environment()
- run_tests.sh: Add kill_existing_stalld_processes() with logging,
  call from init_tests() before tests start, and call from
  cleanup_runner() on exit

This ensures clean test environment initialization and proper cleanup
even when test runs are interrupted.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Assisted-By: git-scm-master (claude-sonnet-4 / 36f1d1f06f2e)
Signed-off-by: Clark Williams <williams@redhat.com>
Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 35 ++++++++++++++++++++++++++++-
 tests/run_tests.sh            | 42 +++++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+), 1 deletion(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 630601a..c42dab3 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -407,6 +407,36 @@ stop_stalld() {
 	fi
 }
 
+# Kill any existing stalld processes (cleanup from previous runs)
+# This ensures a clean slate before starting tests
+kill_existing_stalld() {
+	local pids=$(pgrep -x stalld 2>/dev/null)
+	if [ -n "${pids}" ]; then
+		echo "Killing existing stalld processes: ${pids}"
+		for pid in ${pids}; do
+			# Try graceful shutdown first
+			kill ${pid} 2>/dev/null || true
+		done
+		sleep 0.5
+		# Force kill any remaining
+		pids=$(pgrep -x stalld 2>/dev/null)
+		if [ -n "${pids}" ]; then
+			for pid in ${pids}; do
+				kill -9 ${pid} 2>/dev/null || true
+			done
+			sleep 0.2
+		fi
+		# Verify all killed
+		pids=$(pgrep -x stalld 2>/dev/null)
+		if [ -n "${pids}" ]; then
+			echo -e "${YELLOW}WARNING: Could not kill all stalld processes: ${pids}${NC}"
+			return 1
+		fi
+		echo "All existing stalld processes killed"
+	fi
+	return 0
+}
+
 # Cleanup function (call in trap)
 cleanup() {
 	local exit_code=$?
@@ -712,6 +742,9 @@ disable_dl_server() {
 setup_test_environment() {
 	echo "Setting up test environment..."
 
+	# Kill any existing stalld processes from previous runs
+	kill_existing_stalld
+
 	# Save and disable RT throttling
 	save_rt_throttling
 	disable_rt_throttling
@@ -1022,7 +1055,7 @@ export -f start_test end_test
 export -f assert_equals assert_contains assert_not_contains
 export -f assert_file_exists assert_file_not_exists
 export -f assert_process_running assert_process_not_running
-export -f start_stalld stop_stalld cleanup
+export -f start_stalld stop_stalld kill_existing_stalld cleanup
 export -f wait_for_log_message
 export -f get_thread_policy get_thread_priority
 export -f create_cpu_load
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index c6adc49..16b6750 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -220,6 +220,15 @@ cleanup_runner() {
 	fi
 
 	if [ $EUID -eq 0 ]; then
+		# Kill any remaining stalld processes
+		local pids=$(pgrep -x stalld 2>/dev/null)
+		if [ -n "${pids}" ]; then
+			echo -e "${BLUE}Cleaning up remaining stalld processes: ${pids}${NC}"
+			for pid in ${pids}; do
+				kill -9 ${pid} 2>/dev/null || true
+			done
+		fi
+
 		restore_dl_server_state
 		restore_rt_throttling_state
 	fi
@@ -236,6 +245,34 @@ handle_interrupt() {
 trap cleanup_runner EXIT
 trap handle_interrupt INT TERM
 
+# Kill any existing stalld processes before tests
+kill_existing_stalld_processes() {
+	local pids=$(pgrep -x stalld 2>/dev/null)
+	if [ -n "${pids}" ]; then
+		echo -e "${BLUE}Killing existing stalld processes: ${pids}${NC}" | tee -a "${LOG_FILE}"
+		for pid in ${pids}; do
+			kill ${pid} 2>/dev/null || true
+		done
+		sleep 0.5
+		# Force kill any remaining
+		pids=$(pgrep -x stalld 2>/dev/null)
+		if [ -n "${pids}" ]; then
+			for pid in ${pids}; do
+				kill -9 ${pid} 2>/dev/null || true
+			done
+			sleep 0.2
+		fi
+		# Verify
+		pids=$(pgrep -x stalld 2>/dev/null)
+		if [ -n "${pids}" ]; then
+			echo -e "${YELLOW}WARNING: Could not kill all stalld processes: ${pids}${NC}" | tee -a "${LOG_FILE}"
+		else
+			echo -e "${GREEN}All existing stalld processes killed${NC}" | tee -a "${LOG_FILE}"
+		fi
+		echo "" | tee -a "${LOG_FILE}"
+	fi
+}
+
 # Initialize
 init_tests() {
 	mkdir -p "${RESULTS_DIR}"
@@ -243,6 +280,11 @@ init_tests() {
 	print_banner | tee "${LOG_FILE}"
 	echo "" | tee -a "${LOG_FILE}"
 
+	# Kill any existing stalld processes from previous runs
+	if [ $EUID -eq 0 ]; then
+		kill_existing_stalld_processes
+	fi
+
 	# Save and disable RT throttling if running as root
 	if [ $EUID -eq 0 ]; then
 		save_and_disable_rt_throttling
-- 
2.53.0

[PATCH stalld 02/36] tests/helpers: Fix stalld daemon detection in start_stalld()

[Wander Lairson Costa]

The start_stalld() test helper function incorrectly assumed that
daemonized processes would always have a parent PID (PPID) of 1 (init)
or 2 (kthreadd). This assumption is no longer valid on modern Linux
systems utilizing systemd, where orphaned processes are commonly
re-parented to the user's systemd session, not directly to the global
init process.

This led to failures in the test_foreground test on both sched_debug
and queue_track backends due to the helper failing to correctly
identify the daemonized stalld process.

To address this, the daemon detection logic has been refactored.
Instead of relying on PPID checks, the helper now first waits for the
initial shell background process (shell_pid) that started stalld to
exit. This exit signals that stalld has successfully performed its
double-fork and daemonized. Once the parent shell process has exited,
the helper then uses pgrep -n -x stalld to reliably find the newest
running stalld instance, which corresponds to the daemonized process.
This approach provides a more robust and platform-agnostic method for
detecting stalld when it runs in daemon mode.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 45 ++++++++++-------------------------
 1 file changed, 13 insertions(+), 32 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index c42dab3..5a384a7 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -321,48 +321,29 @@ start_stalld() {
 				fi
 			fi
 		else
-			# Daemon mode: parent forks, child becomes daemon (ppid=1), parent exits
-			# We need to wait for and find the DAEMON child, not the exiting parent
+			# Daemon mode: stalld double-forks, so the shell_pid will exit
+			# and the daemon will be re-parented. Wait for the shell process
+			# to exit, then find the newest stalld process.
 			while [ $attempt -lt $max_attempts ]; do
 				sleep 0.5
 
-				# Get all stalld processes
-				local pids=$(pgrep -x stalld 2>/dev/null)
-
-				for pid in $pids; do
-					if kill -0 $pid 2>/dev/null; then
-						# Check parent PID - daemon should have ppid=1 (init) or ppid=2 (kthreadd)
-						local ppid=$(ps -o ppid= -p $pid 2>/dev/null | tr -d ' ')
-
-						# For daemonized processes, wait for ppid=1 or 2
-						if [ "$ppid" = "1" ] || [ "$ppid" = "2" ]; then
-							STALLD_PID=$pid
-							# Wait a bit to ensure daemon is stable
-							sleep 0.5
-							# Verify it's still running
-							if kill -0 $pid 2>/dev/null; then
-								break 2  # Break out of both loops
-							else
-								# Daemon died, keep looking
-								STALLD_PID=""
-							fi
-						fi
+				# Check if shell_pid has exited (daemonization complete)
+				if ! kill -0 ${shell_pid} 2>/dev/null; then
+					# Shell process exited, daemon should be running
+					# Use pgrep -n to find the newest stalld process
+					STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
+					if [ -n "${STALLD_PID}" ] && kill -0 ${STALLD_PID} 2>/dev/null; then
+						break
 					fi
-				done
-
-				# If we found a daemonized process, we're done
-				if [ -n "${STALLD_PID}" ]; then
-					break
 				fi
 
 				attempt=$((attempt + 1))
 			done
 
-			# Last resort: use the shell PID if nothing else worked
+			# If we still don't have a PID, try one more time
 			if [ -z "${STALLD_PID}" ]; then
-				if kill -0 ${shell_pid} 2>/dev/null; then
-					STALLD_PID=${shell_pid}
-				fi
+				sleep 1
+				STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
 			fi
 		fi
 	fi
-- 
2.53.0

[PATCH stalld 03/36] tests/helpers: Remove duplicate log() function

[Wander Lairson Costa]

test_helpers.sh defines log() twice: the full version at line 43
with journal correlation via logger, and a simplified version at
line 895 that only echoes to stdout. In bash, the second definition
shadows the first, so all tests were silently losing journal
logging that enables correlating test activities with stalld
behavior in journalctl.

Remove the duplicate to restore journal logging for all tests.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 5a384a7..3fa8d07 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -891,11 +891,6 @@ start_stalld_with_backend() {
 # Consolidated helper functions (previously duplicated across tests)
 #
 
-# Logging with timestamp (used by many functional tests)
-log() {
-	echo "[$(date +'%H:%M:%S')] $*"
-}
-
 # Get scheduling policy (0=OTHER, 1=FIFO, 2=RR, 6=DEADLINE)
 get_sched_policy() {
 	local pid=$1
-- 
2.53.0

[PATCH stalld 04/36] tests: Add per-test runtime measurement to run_tests.sh

[Wander Lairson Costa]

The test runner provides no visibility into how long individual
tests take or the total suite runtime. This makes it difficult to
identify slow tests and to measure the impact of future
optimizations to test determinism.

Add elapsed time tracking to run_unit_test() and run_shell_test()
using bash's built-in SECONDS variable. Each test result now
includes the elapsed time in its output, changing the format from
"PASS" to "PASS (12s)" for all outcomes (PASS, FAIL, SKIP). The
total suite runtime is also displayed in print_summary().

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/run_tests.sh | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 16b6750..bbb3b16 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -464,8 +464,10 @@ run_unit_test() {
 		test_log="${RESULTS_DIR}/${backend_mode//:/_}_${test_name}.log"
 	fi
 
+	local start_time=$SECONDS
 	if "${test_path}" > "${test_log}" 2>&1; then
-		echo -e "${GREEN}PASS${NC}" | tee -a "${LOG_FILE}"
+		local elapsed=$((SECONDS - start_time))
+		echo -e "${GREEN}PASS${NC} (${elapsed}s)" | tee -a "${LOG_FILE}"
 		PASSED_TESTS=$((PASSED_TESTS + 1))
 		if [ -n "${backend}" ]; then
 			BACKEND_PASSED["${backend}"]=$((BACKEND_PASSED["${backend}"] + 1))
@@ -474,7 +476,8 @@ run_unit_test() {
 			MODE_PASSED["${mode}"]=$((MODE_PASSED["${mode}"] + 1))
 		fi
 	else
-		echo -e "${RED}FAIL${NC}" | tee -a "${LOG_FILE}"
+		local elapsed=$((SECONDS - start_time))
+		echo -e "${RED}FAIL${NC} (${elapsed}s)" | tee -a "${LOG_FILE}"
 		echo "  See ${test_log} for details" | tee -a "${LOG_FILE}"
 		FAILED_TESTS=$((FAILED_TESTS + 1))
 		if [ -n "${backend}" ]; then
@@ -601,8 +604,10 @@ run_shell_test() {
 		test_log="${RESULTS_DIR}/${backend_mode//:/_}_${test_name}.log"
 	fi
 
+	local start_time=$SECONDS
 	if bash "${test_path}" > "${test_log}" 2>&1; then
-		echo -e "${GREEN}PASS${NC}" | tee -a "${LOG_FILE}"
+		local elapsed=$((SECONDS - start_time))
+		echo -e "${GREEN}PASS${NC} (${elapsed}s)" | tee -a "${LOG_FILE}"
 		PASSED_TESTS=$((PASSED_TESTS + 1))
 		if [ -n "${backend}" ]; then
 			BACKEND_PASSED["${backend}"]=$((BACKEND_PASSED["${backend}"] + 1))
@@ -612,9 +617,10 @@ run_shell_test() {
 		fi
 	else
 		local exit_code=$?
+		local elapsed=$((SECONDS - start_time))
 		if [ ${exit_code} -eq 77 ]; then
 			# Exit code 77 = SKIP (autotools convention)
-			echo -e "${YELLOW}SKIP${NC}" | tee -a "${LOG_FILE}"
+			echo -e "${YELLOW}SKIP${NC} (${elapsed}s)" | tee -a "${LOG_FILE}"
 			SKIPPED_TESTS=$((SKIPPED_TESTS + 1))
 			if [ -n "${backend}" ]; then
 				BACKEND_SKIPPED["${backend}"]=$((BACKEND_SKIPPED["${backend}"] + 1))
@@ -623,7 +629,7 @@ run_shell_test() {
 				MODE_SKIPPED["${mode}"]=$((MODE_SKIPPED["${mode}"] + 1))
 			fi
 		else
-			echo -e "${RED}FAIL${NC}" | tee -a "${LOG_FILE}"
+			echo -e "${RED}FAIL${NC} (${elapsed}s)" | tee -a "${LOG_FILE}"
 			echo "  See ${test_log} for details" | tee -a "${LOG_FILE}"
 			FAILED_TESTS=$((FAILED_TESTS + 1))
 			if [ -n "${backend}" ]; then
@@ -644,6 +650,7 @@ print_summary() {
 	echo -e "  ${GREEN}Passed:  ${PASSED_TESTS}${NC}" | tee -a "${LOG_FILE}"
 	echo -e "  ${RED}Failed:  ${FAILED_TESTS}${NC}" | tee -a "${LOG_FILE}"
 	echo -e "  ${YELLOW}Skipped: ${SKIPPED_TESTS}${NC}" | tee -a "${LOG_FILE}"
+	echo "  Time:    $((SECONDS - SUITE_START_TIME))s" | tee -a "${LOG_FILE}"
 
 	# Print per-backend statistics if matrix testing enabled
 	if [ ${BACKEND_MATRIX} -eq 1 ] && [ ${#BACKEND_TOTAL[@]} -gt 0 ]; then
@@ -866,6 +873,8 @@ run_specific_test() {
 
 # Main execution
 main() {
+	SUITE_START_TIME=$SECONDS
+
 	# Export BACKEND and THREADING_MODE for use by test scripts
 	export STALLD_TEST_BACKEND="${BACKEND}"
 	export STALLD_TEST_THREADING_MODE="${THREADING_MODE}"
-- 
2.53.0

[PATCH stalld 05/36] tests/functional: Fix and refactor test_backend_selection.sh

[Wander Lairson Costa]

test_backend_selection.sh fails when invoked by run_tests.sh because
it uses a relative path (../stalld) that resolves incorrectly when the
working directory is not tests/functional/. It also lacks
parse_test_options for backend passthrough from the test runner, does
not track temporary log files for cleanup, and contains four
copy-pasted 30-line blocks that implement the same
start/verify/check/stop pattern.

Extract the repeated pattern into a test_backend_flag() helper
function that starts stalld with a given -b flag, verifies the
expected backend message appears in the log, and calls stop_stalld.
The helper uses $! instead of pgrep for PID capture since stalld
runs in foreground mode. Fix the stalld path to use ${TEST_ROOT},
add parse_test_options and CLEANUP_FILES tracking, and correct the
test numbering.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_backend_selection.sh | 193 +++++++--------------
 1 file changed, 61 insertions(+), 132 deletions(-)

diff --git a/tests/functional/test_backend_selection.sh b/tests/functional/test_backend_selection.sh
index 6227756..5d60072 100755
--- a/tests/functional/test_backend_selection.sh
+++ b/tests/functional/test_backend_selection.sh
@@ -10,6 +10,7 @@
 
 TEST_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 source "${TEST_ROOT}/helpers/test_helpers.sh"
+parse_test_options "$@" || exit $?
 
 start_test "Backend Selection"
 
@@ -18,155 +19,83 @@ setup_test_environment
 
 require_root
 
-# Test 1: Start stalld with sched_debug backend
-echo "Test 1: Starting stalld with sched_debug backend"
-STALLD_LOG="/tmp/stalld_backend_sched_debug_$$.log"
-# Call stalld directly to capture its actual stderr output
-# IMPORTANT: -v must come BEFORE -b so verbose mode is enabled when backend message is logged
-../stalld -v -f -l -b sched_debug -t 60 > "${STALLD_LOG}" 2>&1 &
-sleep 1
-# Get the actual stalld PID
-STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
-if [ -z "${STALLD_PID}" ]; then
-	TEST_FAILED=$((TEST_FAILED + 1))
-	echo -e "  ${RED}FAIL${NC}: stalld failed to start with sched_debug backend"
-else
+# Helper: start stalld with a specific backend flag, verify the expected
+# backend message appears in the log. This test intentionally bypasses
+# start_stalld_with_log() because it needs to control the -b flag
+# directly rather than inheriting it from STALLD_TEST_BACKEND.
+#
+# Usage: test_backend_flag <backend_flag> <expected_message> <description>
+test_backend_flag() {
+	local backend_flag=$1
+	local expected_msg=$2
+	local description=$3
+	local log_file="/tmp/stalld_backend_${backend_flag}_$$.log"
+
+	CLEANUP_FILES+=("${log_file}")
+
+	"${TEST_ROOT}/../stalld" -v -f -l -b "${backend_flag}" -t 60 \
+		> "${log_file}" 2>&1 &
+	STALLD_PID=$!
 	CLEANUP_PIDS+=("${STALLD_PID}")
 	sleep 1
-	if kill -0 ${STALLD_PID} 2>/dev/null; then
-		# Check if log contains backend message
-		if grep -q "using sched_debug backend" "${STALLD_LOG}"; then
-			assert_equals "0" "0" "sched_debug backend selected"
-		else
-			TEST_FAILED=$((TEST_FAILED + 1))
-			echo -e "  ${RED}FAIL${NC}: Backend message not found in log"
-			echo "  Log contents:"
-			cat "${STALLD_LOG}"
-		fi
-		stop_stalld
+
+	if ! kill -0 ${STALLD_PID} 2>/dev/null; then
+		TEST_FAILED=$((TEST_FAILED + 1))
+		echo -e "  ${RED}FAIL${NC}: stalld failed to start (${description})"
+		return 1
+	fi
+
+	if grep -q "${expected_msg}" "${log_file}"; then
+		assert_equals "0" "0" "${description}"
 	else
 		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: stalld failed to start with sched_debug backend"
+		echo -e "  ${RED}FAIL${NC}: Backend message not found (${description})"
+		echo "  Expected: ${expected_msg}"
+		echo "  Log contents:"
+		cat "${log_file}"
 	fi
-fi
-rm -f "${STALLD_LOG}"
 
+	stop_stalld
+}
+
+# Test 1: sched_debug backend (full name)
+echo "Test 1: Starting stalld with sched_debug backend"
+test_backend_flag "sched_debug" "using sched_debug backend" \
+	"sched_debug backend selected"
+
+# Test 2: queue_track backend (if available)
 echo ""
-echo "=== Test 4: Check queue_track (BPF) backend availability ==="
+echo "Test 2: Check queue_track (BPF) backend"
 if is_backend_available "queue_track"; then
-	echo "Test 2a: Starting stalld with queue_track backend"
-	STALLD_LOG="/tmp/stalld_backend_queue_track_$$.log"
-	# Call stalld directly to capture its actual stderr output
-	# IMPORTANT: -v must come BEFORE -b so verbose mode is enabled when backend message is logged
-	../stalld -v -f -l -b queue_track -t 60 > "${STALLD_LOG}" 2>&1 &
-	sleep 1
-	# Get the actual stalld PID
-	STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
-	if [ -z "${STALLD_PID}" ]; then
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: stalld failed to start with queue_track backend"
-	else
-		CLEANUP_PIDS+=("${STALLD_PID}")
-		sleep 1
-		if kill -0 ${STALLD_PID} 2>/dev/null; then
-			# Check if log contains backend message
-			if grep -q "using queue_track backend" "${STALLD_LOG}"; then
-				assert_equals "0" "0" "queue_track backend selected"
-			else
-				TEST_FAILED=$((TEST_FAILED + 1))
-				echo -e "  ${RED}FAIL${NC}: Backend message not found in log"
-				echo "  Log contents:"
-				cat "${STALLD_LOG}"
-			fi
-			stop_stalld
-		else
-			TEST_FAILED=$((TEST_FAILED + 1))
-			echo -e "  ${RED}FAIL${NC}: stalld failed to start with queue_track backend"
-		fi
-	fi
-	rm -f "${STALLD_LOG}"
+	test_backend_flag "queue_track" "using queue_track backend" \
+		"queue_track backend selected"
 else
-	echo "ℹ queue_track (BPF) backend not available"
-	echo "  (This is expected on i686, powerpc, ppc64le, or kernels ≤3.x)"
+	echo "  queue_track (BPF) backend not available"
+	echo "  (This is expected on i686, powerpc, ppc64le, or kernels <=3.x)"
 	TEST_PASSED=$((TEST_PASSED + 1))
 fi
 
-# Test 3: Test short names (S for sched_debug)
+# Test 3: Short name 'S' for sched_debug
+echo ""
 echo "Test 3: Testing short name 'S' for sched_debug"
-STALLD_LOG="/tmp/stalld_backend_short_S_$$.log"
-# Call stalld directly to capture its actual stderr output
-# IMPORTANT: -v must come BEFORE -b so verbose mode is enabled when backend message is logged
-../stalld -v -f -l -b S -t 60 > "${STALLD_LOG}" 2>&1 &
-sleep 1
-# Get the actual stalld PID
-STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
-if [ -z "${STALLD_PID}" ]; then
-	TEST_FAILED=$((TEST_FAILED + 1))
-	echo -e "  ${RED}FAIL${NC}: stalld failed to start with short name 'S'"
-else
-	CLEANUP_PIDS+=("${STALLD_PID}")
-	sleep 1
-	if kill -0 ${STALLD_PID} 2>/dev/null; then
-		# Check if log contains backend message
-		if grep -q "using sched_debug backend" "${STALLD_LOG}"; then
-			assert_equals "0" "0" "Short name 'S' works for sched_debug"
-		else
-			TEST_FAILED=$((TEST_FAILED + 1))
-			echo -e "  ${RED}FAIL${NC}: Backend message not found for short name"
-			echo "  Log contents:"
-			cat "${STALLD_LOG}"
-		fi
-		stop_stalld
-	else
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: stalld failed to start with short name 'S'"
-	fi
-fi
-rm -f "${STALLD_LOG}"
+test_backend_flag "S" "using sched_debug backend" \
+	"Short name 'S' works for sched_debug"
 
-# Test 4: Test STALLD_TEST_BACKEND environment variable
+# Test 4: STALLD_TEST_BACKEND environment variable
+echo ""
 if [ -n "${STALLD_TEST_BACKEND}" ]; then
 	echo "Test 4: Testing STALLD_TEST_BACKEND=${STALLD_TEST_BACKEND}"
-	STALLD_LOG="/tmp/stalld_backend_env_$$.log"
-	# Call stalld directly to capture its actual stderr output
-	# start_stalld adds -b based on STALLD_TEST_BACKEND, so we mimic that here
-	# IMPORTANT: -v must come BEFORE -b so verbose mode is enabled when backend message is logged
-	../stalld -v -f -l -b "${STALLD_TEST_BACKEND}" -t 60 > "${STALLD_LOG}" 2>&1 &
-	sleep 1
-	# Get the actual stalld PID
-	STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
-	if [ -z "${STALLD_PID}" ]; then
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: stalld failed to start with STALLD_TEST_BACKEND"
-	else
-		CLEANUP_PIDS+=("${STALLD_PID}")
-		sleep 1
-		if kill -0 ${STALLD_PID} 2>/dev/null; then
-			# Normalize backend name for comparison
-			BACKEND_NORMALIZED="${STALLD_TEST_BACKEND}"
-			case "${STALLD_TEST_BACKEND}" in
-				S) BACKEND_NORMALIZED="sched_debug" ;;
-				Q) BACKEND_NORMALIZED="queue_track" ;;
-			esac
-
-			# Check if log contains backend message
-			if grep -q "using ${BACKEND_NORMALIZED} backend" "${STALLD_LOG}"; then
-				assert_equals "0" "0" "STALLD_TEST_BACKEND environment variable respected"
-			else
-				TEST_FAILED=$((TEST_FAILED + 1))
-				echo -e "  ${RED}FAIL${NC}: Backend ${BACKEND_NORMALIZED} not used from environment"
-				echo "  Log contents:"
-				cat "${STALLD_LOG}"
-			fi
-			stop_stalld
-		else
-			echo "ℹ Could not verify backend in logs (may not be logged)"
-			TEST_PASSED=$((TEST_PASSED + 1))
-		fi
-	fi
-	rm -f "${STALLD_LOG}"
+	# Normalize short names for expected message
+	BACKEND_NORMALIZED="${STALLD_TEST_BACKEND}"
+	case "${STALLD_TEST_BACKEND}" in
+		S) BACKEND_NORMALIZED="sched_debug" ;;
+		Q) BACKEND_NORMALIZED="queue_track" ;;
+	esac
+	test_backend_flag "${STALLD_TEST_BACKEND}" \
+		"using ${BACKEND_NORMALIZED} backend" \
+		"STALLD_TEST_BACKEND environment variable respected"
 else
-	echo "ℹ Skipping queue_track test - backend not available on this system"
+	echo "Test 4: Skipping (STALLD_TEST_BACKEND not set)"
 	TEST_PASSED=$((TEST_PASSED + 1))
 fi
 
-- 
2.53.0

[PATCH stalld 06/36] tests/functional: Fix test_logging_destinations.sh path and backend

[Wander Lairson Costa]

Tests 1 and 4 invoke stalld directly to capture its stdout for
verification. They used a relative path (../stalld) that breaks
when run_tests.sh invokes the test from a different working
directory, and used pgrep to find the PID instead of capturing $!
from the background launch. They also lacked backend passthrough
for matrix testing.

Fix the stalld path to use ${TEST_ROOT}, capture the PID via $!,
add BACKEND_FLAG construction and passthrough for the direct
invocations, and remove a redundant rm -f already handled by
CLEANUP_FILES. Tests 2 and 3 use start_stalld and need no changes.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_logging_destinations.sh | 22 +++++++++----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/tests/functional/test_logging_destinations.sh b/tests/functional/test_logging_destinations.sh
index 1d41464..6fa4907 100755
--- a/tests/functional/test_logging_destinations.sh
+++ b/tests/functional/test_logging_destinations.sh
@@ -28,12 +28,15 @@ LOG_FILE="/tmp/stalld_test_verbose_$$.log"
 CLEANUP_FILES+=("${LOG_FILE}")
 
 # Start stalld directly (not using start_stalld helper) to capture output
-../stalld -f -v -l -t 5 > "${LOG_FILE}" 2>&1 &
-sleep 2
-STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
-if [ -n "${STALLD_PID}" ]; then
-	CLEANUP_PIDS+=("${STALLD_PID}")
+# Add backend flag if specified via test runner
+BACKEND_FLAG=""
+if [ -n "${STALLD_TEST_BACKEND}" ]; then
+	BACKEND_FLAG="-b ${STALLD_TEST_BACKEND}"
 fi
+"${TEST_ROOT}/../stalld" -f -v ${BACKEND_FLAG} -l -t 5 > "${LOG_FILE}" 2>&1 &
+STALLD_PID=$!
+CLEANUP_PIDS+=("${STALLD_PID}")
+sleep 2
 
 if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 	# Check that output was written to our log file
@@ -51,7 +54,6 @@ if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 fi
 
 stop_stalld
-rm -f "${LOG_FILE}"
 
 # Test 2: Kernel message log (-k)
 echo ""
@@ -153,12 +155,10 @@ LOG_FILE="/tmp/stalld_test_combined_$$.log"
 CLEANUP_FILES+=("${LOG_FILE}")
 
 # Start stalld directly (not using start_stalld helper) to capture output
-../stalld -f -v -k -s -l -t 5 > "${LOG_FILE}" 2>&1 &
+"${TEST_ROOT}/../stalld" -f -v -k -s ${BACKEND_FLAG} -l -t 5 > "${LOG_FILE}" 2>&1 &
+STALLD_PID=$!
+CLEANUP_PIDS+=("${STALLD_PID}")
 sleep 2
-STALLD_PID=$(pgrep -n -x stalld 2>/dev/null)
-if [ -n "${STALLD_PID}" ]; then
-	CLEANUP_PIDS+=("${STALLD_PID}")
-fi
 
 if assert_process_running "${STALLD_PID}" "stalld with combined logging should be running"; then
 	# Verify verbose output
-- 
2.53.0

[PATCH stalld 07/36] tests/helpers: Replace sleep with poll in start_stalld_with_log()

[Wander Lairson Costa]

start_stalld_with_log() uses a hardcoded sleep 1 after launching
stalld, which is both fragile and wasteful. On systems where BPF
backend initialization takes longer than one second, it is
insufficient. On fast systems, it delays every test unnecessarily.

Replace the fixed sleep with a polling loop that checks whether
stalld has started writing to the log file. A brief initial sleep
covers the fast path, then 1-second polling with a 15-second
timeout handles slow startup such as BPF initialization. On
timeout, the process is killed with SIGTERM and escalated to
SIGKILL if it does not exit within one second.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 3fa8d07..f6d6ce2 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -948,11 +948,37 @@ start_stalld_with_log() {
 		echo "Using backend: ${STALLD_TEST_BACKEND}"
 	fi
 
-	# Start stalld with output redirected
-	${TEST_ROOT}/../stalld ${stalld_args} > "${log_file}" 2>&1 &
+	# Start stalld with line-buffered output so tail -f can detect
+	# readiness immediately instead of waiting for the buffer to fill.
+	stdbuf -oL ${TEST_ROOT}/../stalld ${stalld_args} > "${log_file}" 2>&1 &
 	STALLD_PID=$!
 	CLEANUP_PIDS+=("${STALLD_PID}")
+
+	# Poll for stalld to start writing to the log file rather than
+	# using a fixed sleep. Brief initial sleep covers the fast path,
+	# then 1-second polling for slow systems (e.g. BPF init).
+	sleep 0.01
+	local timeout=15
+	local elapsed=0
+	while [ $elapsed -lt $timeout ]; do
+		if ! kill -0 ${STALLD_PID} 2>/dev/null; then
+			echo -e "${RED}ERROR: stalld exited during startup${NC}"
+			return 1
+		fi
+		if [ -s "${log_file}" ]; then
+			return 0
+		fi
+		sleep 1
+		elapsed=$((elapsed + 1))
+	done
+
+	echo -e "${RED}ERROR: stalld did not produce output within ${timeout}s${NC}"
+	kill ${STALLD_PID} 2>/dev/null
 	sleep 1
+	if kill -0 ${STALLD_PID} 2>/dev/null; then
+		kill -9 ${STALLD_PID} 2>/dev/null
+	fi
+	return 1
 }
 
 # Wait for scheduling policy to change to expected value
-- 
2.53.0

[PATCH stalld 08/36] tests/helpers: Fix stop_stalld() timeout and shutdown logic

[Wander Lairson Costa]

stop_stalld() has two bugs. The polling loop after SIGKILL uses
sleep 0.1 with an integer counter, so the nominal timeout of 10
actually gives only 1 second. Additionally, SIGTERM only gets
0.2 seconds before escalating to SIGKILL, which is insufficient
for stalld to perform graceful cleanup such as removing its
pidfile or flushing logs.

Restructure stop_stalld() into two clear phases. First send
SIGTERM and poll for graceful exit with 1-second intervals for
up to 5 seconds, giving stalld time to clean up. Then escalate
to SIGKILL only if the process is still alive, and poll again
for up to 5 seconds with correct timeout arithmetic. Document
the guarantee that the process is dead before the function
returns so callers do not need post-stop sleeps.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index f6d6ce2..1750423 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -365,24 +365,33 @@ start_stalld() {
 }
 
 # Stop stalld
+# Sends SIGTERM and polls for graceful exit, then escalates to
+# SIGKILL if needed. Guarantees the process is dead before
+# returning so callers do not need post-stop sleeps.
 stop_stalld() {
 	if [ -n "${STALLD_PID}" ]; then
 		if kill -0 ${STALLD_PID} 2>/dev/null; then
-			# Try graceful shutdown first
+			# Try graceful shutdown first (SIGTERM)
 			kill ${STALLD_PID} 2>/dev/null || true
-			# Give it a moment to exit gracefully
-			sleep 0.2
-			# Force kill if still running
-			if kill -0 ${STALLD_PID} 2>/dev/null; then
-				kill -9 ${STALLD_PID} 2>/dev/null || true
-			fi
-			# Poll for process termination (don't use wait - might not be a child)
-			local timeout=10
+
+			# Poll for graceful exit (up to 5 seconds)
+			local timeout=5
 			local elapsed=0
 			while kill -0 ${STALLD_PID} 2>/dev/null && [ ${elapsed} -lt ${timeout} ]; do
-				sleep 0.1
+				sleep 1
 				elapsed=$((elapsed + 1))
 			done
+
+			# Escalate to SIGKILL if still running
+			if kill -0 ${STALLD_PID} 2>/dev/null; then
+				kill -9 ${STALLD_PID} 2>/dev/null || true
+				# Poll for forced termination (up to 5 seconds)
+				elapsed=0
+				while kill -0 ${STALLD_PID} 2>/dev/null && [ ${elapsed} -lt ${timeout} ]; do
+					sleep 1
+					elapsed=$((elapsed + 1))
+				done
+			fi
 		fi
 		STALLD_PID=""
 	fi
-- 
2.53.0

[PATCH stalld 09/36] tests/helpers: Fix relative path in backend detection functions

[Wander Lairson Costa]

detect_default_backend() and is_backend_available() use a
hardcoded relative path "../stalld" to locate the binary for
backend detection via nm. This breaks when tests are invoked from
a different working directory, such as when run_tests.sh executes
them from the project root.

Use ${TEST_ROOT}/../stalld instead, consistent with the path
resolution used in start_stalld_with_log() and
start_starvation_gen().

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 1750423..8ff1b80 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -816,7 +816,7 @@ pick_test_cpu() {
 
 # Detect which backend stalld was compiled with (default)
 detect_default_backend() {
-	local stalld_bin="../stalld"
+	local stalld_bin="${TEST_ROOT}/../stalld"
 	if [ ! -x "${stalld_bin}" ]; then
 		echo "unknown"
 		return 1
@@ -839,7 +839,7 @@ detect_default_backend() {
 # Check if a specific backend is available
 is_backend_available() {
 	local backend=$1
-	local stalld_bin="../stalld"
+	local stalld_bin="${TEST_ROOT}/../stalld"
 
 	if [ ! -x "${stalld_bin}" ]; then
 		return 1
-- 
2.53.0

[PATCH stalld 10/36] tests/functional: Remove redundant post-stop_stalld sleeps

[Wander Lairson Costa]

stop_stalld() now guarantees the process is dead before returning,
making the sleep 1 calls that follow it in test scripts redundant.
Remove all 30 occurrences across test_affinity.sh,
test_boost_duration.sh, test_boost_runtime.sh, test_force_fifo.sh,
test_foreground.sh, test_pidfile.sh, and
test_starvation_threshold.sh.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_affinity.sh             | 7 -------
 tests/functional/test_boost_duration.sh       | 4 ----
 tests/functional/test_boost_runtime.sh        | 4 ----
 tests/functional/test_force_fifo.sh           | 6 ------
 tests/functional/test_foreground.sh           | 1 -
 tests/functional/test_pidfile.sh              | 5 -----
 tests/functional/test_starvation_threshold.sh | 3 ---
 7 files changed, 30 deletions(-)

diff --git a/tests/functional/test_affinity.sh b/tests/functional/test_affinity.sh
index 90dd69f..c3888ba 100755
--- a/tests/functional/test_affinity.sh
+++ b/tests/functional/test_affinity.sh
@@ -89,7 +89,6 @@ else
 fi
 
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 2: Single CPU affinity
@@ -115,7 +114,6 @@ else
 fi
 
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 3: Multi-CPU affinity (CPU list)
@@ -143,7 +141,6 @@ if [ "$num_cpus" -ge 4 ]; then
     fi
 
     stop_stalld
-    sleep 1
 else
     log "⊘ SKIP: Test 3 requires at least 4 CPUs"
 fi
@@ -173,7 +170,6 @@ if [ "$num_cpus" -ge 4 ]; then
     fi
 
     stop_stalld
-    sleep 1
 else
     log "⊘ SKIP: Test 4 requires at least 4 CPUs"
 fi
@@ -210,7 +206,6 @@ if [ "$num_cpus" -ge 2 ]; then
     fi
 
     stop_stalld
-    sleep 1
 else
     log "⊘ SKIP: Test 5 requires at least 2 CPUs"
 fi
@@ -247,7 +242,6 @@ if [ "$num_cpus" -ge 2 ]; then
     fi
 
     stop_stalld
-    sleep 1
 else
     log "⊘ SKIP: Test 6 requires at least 2 CPUs"
 fi
@@ -318,7 +312,6 @@ else
 fi
 
 stop_stalld
-sleep 1
 
 log ""
 log "All affinity tests completed"
diff --git a/tests/functional/test_boost_duration.sh b/tests/functional/test_boost_duration.sh
index e9f8928..4df5489 100755
--- a/tests/functional/test_boost_duration.sh
+++ b/tests/functional/test_boost_duration.sh
@@ -84,7 +84,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 2: Short duration (1 second)
@@ -123,7 +122,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 3: Long duration (10 seconds)
@@ -163,7 +161,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 4: Verify task policy is restored after boost duration
@@ -201,7 +198,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 5: Invalid duration values
diff --git a/tests/functional/test_boost_runtime.sh b/tests/functional/test_boost_runtime.sh
index 60ccdc6..0475da5 100755
--- a/tests/functional/test_boost_runtime.sh
+++ b/tests/functional/test_boost_runtime.sh
@@ -84,7 +84,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 2: Custom runtime (10,000 ns = 10 microseconds, less than default)
@@ -123,7 +122,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 3: Larger runtime (100,000 ns = 100 microseconds)
@@ -162,7 +160,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 4: Runtime < period (valid configuration)
@@ -203,7 +200,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null || true
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 5: Runtime > period (should error or be rejected)
diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index 704d605..cab8003 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -85,7 +85,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 2: Force FIFO mode (-F)
@@ -134,7 +133,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 3: Verify FIFO priority setting
@@ -175,7 +173,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 4: Verify FIFO emulation behavior (sleep runtime, restore, sleep remainder)
@@ -223,7 +220,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 5: Single-threaded mode with FIFO (should fail/exit)
@@ -285,7 +281,6 @@ log "ℹ INFO: SCHED_DEADLINE boosts: $deadline_boosts"
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 # Run with FIFO
 STALLD_LOG_FIFO="/tmp/stalld_test_force_fifo_comparison_$$.log"
@@ -304,7 +299,6 @@ log "ℹ INFO: SCHED_FIFO boosts: $fifo_boosts"
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 log "ℹ INFO: Comparison complete (DEADLINE: $deadline_boosts, FIFO: $fifo_boosts)"
 
diff --git a/tests/functional/test_foreground.sh b/tests/functional/test_foreground.sh
index 693d4b1..c5afbba 100755
--- a/tests/functional/test_foreground.sh
+++ b/tests/functional/test_foreground.sh
@@ -45,7 +45,6 @@ if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 fi
 
 stop_stalld
-sleep 1
 
 # Test 2: With -f flag, stalld should stay in foreground
 echo ""
diff --git a/tests/functional/test_pidfile.sh b/tests/functional/test_pidfile.sh
index 675b15e..155855e 100755
--- a/tests/functional/test_pidfile.sh
+++ b/tests/functional/test_pidfile.sh
@@ -68,7 +68,6 @@ if [ $default_found -eq 0 ]; then
 fi
 
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 2: Custom pidfile location
@@ -112,7 +111,6 @@ fi
 log ""
 log "Test 3: Verify pidfile removed on clean shutdown"
 stop_stalld
-sleep 1
 
 if [ ! -f "${custom_pidfile}" ]; then
     log "✓ PASS: Pidfile removed on clean shutdown"
@@ -156,7 +154,6 @@ else
 fi
 
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 5: Test with foreground mode
@@ -192,7 +189,6 @@ else
 fi
 
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 6: Invalid pidfile path (permission denied)
@@ -278,7 +274,6 @@ else
 fi
 
 stop_stalld
-sleep 1
 
 log ""
 log "All pidfile tests completed"
diff --git a/tests/functional/test_starvation_threshold.sh b/tests/functional/test_starvation_threshold.sh
index c2dda57..41a2cca 100755
--- a/tests/functional/test_starvation_threshold.sh
+++ b/tests/functional/test_starvation_threshold.sh
@@ -98,7 +98,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 2: Verify no detection before threshold
@@ -149,7 +148,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 3: Shorter threshold (3 seconds)
@@ -197,7 +195,6 @@ fi
 kill -TERM "${STARVE_PID}" 2>/dev/null
 wait "${STARVE_PID}" 2>/dev/null || true
 stop_stalld
-sleep 1
 
 #=============================================================================
 # Test 4: Invalid threshold values
-- 
2.53.0

[PATCH stalld 11/36] tests/functional: Fix false positive log matching in test_logging_destinations

[Wander Lairson Costa]

The test framework's log() function sends entries to the system
journal tagged with [stalld] via logger -t stalld. The logging
destination tests were using grep "stalld" to verify daemon
output in dmesg, syslog, and journalctl, which also matched
these framework entries and caused false passes.

Add a local has_stalld_log() filter function that matches
"stalld" while excluding lines containing "stalld". Use grep -F
for fixed-string matching throughout to avoid unnecessary regex
interpretation.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_logging_destinations.sh | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/tests/functional/test_logging_destinations.sh b/tests/functional/test_logging_destinations.sh
index 6fa4907..171be8b 100755
--- a/tests/functional/test_logging_destinations.sh
+++ b/tests/functional/test_logging_destinations.sh
@@ -21,6 +21,12 @@ setup_test_environment
 # Require root for this test
 require_root
 
+# Filter stdin for stalld daemon messages, excluding the test
+# framework's own log entries tagged with [TEST].
+has_stalld_log() {
+	grep -F "stalld" | grep -Fqv "[TEST]"
+}
+
 # Test 1: Verbose mode (-v) logs to stdout/stderr
 echo "Test 1: Verbose mode (-v) logs to stdout"
 
@@ -74,7 +80,7 @@ if command -v dmesg >/dev/null 2>&1; then
 		# Note: This might not work in all environments
 		if [ ${DMESG_AFTER} -gt ${DMESG_BEFORE} ]; then
 			# Check if recent dmesg contains stalld messages
-			if dmesg | tail -10 | grep -q "stalld"; then
+			if dmesg | tail -10 | has_stalld_log; then
 				assert_equals "1" "1" "stalld messages in kernel log"
 			else
 				echo -e "  ${YELLOW}SKIP${NC}: cannot verify kernel log messages"
@@ -113,7 +119,7 @@ if [ -n "${SYSLOG_FILE}" ]; then
 
 		if [ ${SYSLOG_AFTER} -gt ${SYSLOG_BEFORE} ]; then
 			# Check for stalld messages in recent syslog
-			if tail -20 "${SYSLOG_FILE}" | grep -q "stalld"; then
+			if tail -20 "${SYSLOG_FILE}" | has_stalld_log; then
 				assert_equals "1" "1" "stalld messages in syslog"
 			else
 				echo -e "  ${YELLOW}SKIP${NC}: no stalld messages found in syslog"
@@ -133,9 +139,9 @@ elif command -v journalctl >/dev/null 2>&1; then
 
 	if assert_process_running "${STALLD_PID}" "stalld with -s should be running"; then
 		# Check journalctl for stalld messages
-		if journalctl -u stalld --since "1 minute ago" 2>/dev/null | grep -q "stalld"; then
+		if journalctl -u stalld --since "1 minute ago" 2>/dev/null | has_stalld_log; then
 			assert_equals "1" "1" "stalld messages in journalctl"
-		elif journalctl --since "1 minute ago" 2>/dev/null | grep -q "stalld"; then
+		elif journalctl --since "1 minute ago" 2>/dev/null | has_stalld_log; then
 			assert_equals "1" "1" "stalld messages in system journal"
 		else
 			echo -e "  ${YELLOW}SKIP${NC}: no stalld messages in journal (may take time to appear)"
-- 
2.53.0

[PATCH stalld 12/36] tests/helpers: Rewrite wait_for_log_message() with process substitution

[Wander Lairson Costa]

wait_for_log_message() uses a 1-second polling loop to check for
a pattern in a log file, adding up to 1 second of latency even
when the message appears instantly. It also has a journalctl
fallback that is unused by any caller and a misleading default
log file parameter (/var/log/syslog).

Rewrite the function to use process substitution instead of polling.
By feeding the output of a timeout-wrapped tail -f into grep -m1 -q
via process substitution, the function returns immediately when the
pattern appears. Process substitution is specifically required here
rather than a standard pipeline. In a pipeline, bash waits for both
processes to exit. Because tail -f blocks on inotify and does not
receive SIGPIPE, a pipeline would hang for the full timeout duration
even after grep finds a match.

Additionally, remove the unused journalctl fallback and make the log
file a required parameter since all callers pass one explicitly.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 41 +++++++++++++++--------------------
 1 file changed, 18 insertions(+), 23 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 8ff1b80..08838a2 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -500,36 +500,31 @@ handle_signal() {
 trap cleanup EXIT
 trap handle_signal INT TERM
 
-# Parse stalld log for specific message
+# Wait for a specific message to appear in a log file.
+# Uses tail -f piped through grep for instant detection -- returns
+# immediately when the pattern appears instead of sleeping between
+# polling intervals. Replays existing file content so messages
+# written before this function is called are also matched.
+#
+# Usage: wait_for_log_message <pattern> <timeout> <log_file>
 wait_for_log_message() {
 	local pattern=$1
 	local timeout=${2:-10}
-	local log_file=${3:-/var/log/syslog}
+	local log_file=$3
 
-	# If log_file doesn't exist, try journalctl
-	if [ ! -f "${log_file}" ]; then
-		# Using journalctl instead
-		local elapsed=0
-		while [ ${elapsed} -lt ${timeout} ]; do
-			if journalctl -u stalld --since "1 minute ago" 2>/dev/null | grep -q "${pattern}"; then
-				return 0
-			fi
-			sleep 1
-			elapsed=$((elapsed + 1))
-		done
+	if [ -z "${log_file}" ]; then
+		echo -e "${RED}ERROR: wait_for_log_message requires a log file${NC}"
 		return 1
 	fi
 
-	local elapsed=0
-	while [ ${elapsed} -lt ${timeout} ]; do
-		if grep -q "${pattern}" "${log_file}"; then
-			return 0
-		fi
-		sleep 1
-		elapsed=$((elapsed + 1))
-	done
-
-	return 1
+	# Process substitution runs tail in the background so bash
+	# only waits for grep to finish. A pipeline (tail | grep)
+	# would block until timeout kills tail even after grep has
+	# matched, because tail -f is blocked on inotify and never
+	# receives SIGPIPE.
+	grep -m1 -q "${pattern}" \
+		< <(timeout "${timeout}" tail -f -n +1 "${log_file}" 2>/dev/null)
+	return $?
 }
 
 # Get thread scheduling policy
-- 
2.53.0

[PATCH stalld 13/36] tests/helpers: Add wait_for_stalld_ready() and use in start_stalld_with_log()

[Wander Lairson Costa]

start_stalld_with_log() checks for a non-empty log file to
determine if stalld has started, but this is a weak signal that
does not guarantee initialization is complete. Backend loading,
boost method detection, and thread setup may still be in progress.

Add wait_for_stalld_ready() that watches for a stalld log message,
which stalld prints at the end of its init sequence after all setup
is complete. Integrate it into start_stalld_with_log() to replace
the non-empty file check, and reuse stop_stalld() in the error path
instead of duplicating the kill logic.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 45 +++++++++++++----------------------
 1 file changed, 16 insertions(+), 29 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 08838a2..e4e9b23 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -501,10 +501,8 @@ trap cleanup EXIT
 trap handle_signal INT TERM
 
 # Wait for a specific message to appear in a log file.
-# Uses tail -f piped through grep for instant detection -- returns
-# immediately when the pattern appears instead of sleeping between
-# polling intervals. Replays existing file content so messages
-# written before this function is called are also matched.
+# Returns immediately when the pattern is found, or returns 1
+# after the timeout expires.
 #
 # Usage: wait_for_log_message <pattern> <timeout> <log_file>
 wait_for_log_message() {
@@ -527,6 +525,15 @@ wait_for_log_message() {
 	return $?
 }
 
+# Wait for stalld to complete initialization.
+#
+# Usage: wait_for_stalld_ready <log_file> [timeout]
+wait_for_stalld_ready() {
+	local log_file=$1
+	local timeout=${2:-15}
+	wait_for_log_message "checking cpu\|waiting tasks" "${timeout}" "${log_file}"
+}
+
 # Get thread scheduling policy
 get_thread_policy() {
 	local pid=$1
@@ -958,31 +965,11 @@ start_stalld_with_log() {
 	STALLD_PID=$!
 	CLEANUP_PIDS+=("${STALLD_PID}")
 
-	# Poll for stalld to start writing to the log file rather than
-	# using a fixed sleep. Brief initial sleep covers the fast path,
-	# then 1-second polling for slow systems (e.g. BPF init).
-	sleep 0.01
-	local timeout=15
-	local elapsed=0
-	while [ $elapsed -lt $timeout ]; do
-		if ! kill -0 ${STALLD_PID} 2>/dev/null; then
-			echo -e "${RED}ERROR: stalld exited during startup${NC}"
-			return 1
-		fi
-		if [ -s "${log_file}" ]; then
-			return 0
-		fi
-		sleep 1
-		elapsed=$((elapsed + 1))
-	done
-
-	echo -e "${RED}ERROR: stalld did not produce output within ${timeout}s${NC}"
-	kill ${STALLD_PID} 2>/dev/null
-	sleep 1
-	if kill -0 ${STALLD_PID} 2>/dev/null; then
-		kill -9 ${STALLD_PID} 2>/dev/null
+	if ! wait_for_stalld_ready "${log_file}" 15; then
+		echo -e "${RED}ERROR: stalld did not initialize within 15s${NC}"
+		stop_stalld
+		return 1
 	fi
-	return 1
 }
 
 # Wait for scheduling policy to change to expected value
@@ -1062,7 +1049,7 @@ export -f assert_equals assert_contains assert_not_contains
 export -f assert_file_exists assert_file_not_exists
 export -f assert_process_running assert_process_not_running
 export -f start_stalld stop_stalld kill_existing_stalld cleanup
-export -f wait_for_log_message
+export -f wait_for_log_message wait_for_stalld_ready
 export -f get_thread_policy get_thread_priority
 export -f create_cpu_load
 export -f detect_default_backend is_backend_available get_available_backends start_stalld_with_backend
-- 
2.53.0

[PATCH stalld 14/36] tests/helpers: Fix fractional sleep timeout bugs

[Wander Lairson Costa]

The cleanup() process termination loop and start_stalld() pidfile
wait both use fractional sleeps (0.1s and 0.5s) with integer
counters, causing the actual timeouts to be a fraction of the
nominal values. cleanup() with timeout=5 and sleep 0.1 gives only
0.5 seconds, and start_stalld() with timeout=15 and sleep 0.5
gives only 7.5 seconds.

Change both loops to use sleep 1, making the timeout arithmetic
correct. Restructure the cleanup loop to use the same two-phase
SIGTERM/SIGKILL pattern established in stop_stalld().

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index e4e9b23..06c0471 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -288,7 +288,7 @@ start_stalld() {
 		local timeout=15
 		local elapsed=0
 		while [ ! -f "$pidfile" ] && [ $elapsed -lt $timeout ]; do
-			sleep 0.5
+			sleep 1
 			elapsed=$((elapsed + 1))
 		done
 
@@ -452,21 +452,23 @@ cleanup() {
 		if [ -n "${pid}" ] && [ "${pid}" -gt 0 ] 2>/dev/null; then
 			# Check if process exists
 			if kill -0 ${pid} 2>/dev/null; then
-				# Try gentle kill first
 				kill ${pid} 2>/dev/null || true
-				# Give it a moment
-				sleep 0.1
-				# Force kill if still running
-				if kill -0 ${pid} 2>/dev/null; then
-					kill -9 ${pid} 2>/dev/null || true
-				fi
-				# Poll for termination (don't use wait - might not be a child)
+
 				local timeout=5
 				local elapsed=0
 				while kill -0 ${pid} 2>/dev/null && [ ${elapsed} -lt ${timeout} ]; do
-					sleep 0.1
+					sleep 1
 					elapsed=$((elapsed + 1))
 				done
+
+				if kill -0 ${pid} 2>/dev/null; then
+					kill -9 ${pid} 2>/dev/null || true
+					elapsed=0
+					while kill -0 ${pid} 2>/dev/null && [ ${elapsed} -lt ${timeout} ]; do
+						sleep 1
+						elapsed=$((elapsed + 1))
+					done
+				fi
 			fi
 		fi
 	done
-- 
2.53.0

[PATCH stalld 15/36] tests/helpers: Flush stdout after starvation_gen startup messages

[Wander Lairson Costa]

starvation_gen prints its startup configuration (iteration target and
stop instructions) via printf, but does not flush stdout afterward.
When stdout is not connected to a terminal, libc switches to block
buffering, so these messages may not reach the test harness until the
buffer fills or the process exits.

Add an explicit fflush(stdout) after the startup printf calls to
ensure immediate delivery regardless of buffering mode.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/starvation_gen.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/helpers/starvation_gen.c b/tests/helpers/starvation_gen.c
index dc0e3de..81ee714 100644
--- a/tests/helpers/starvation_gen.c
+++ b/tests/helpers/starvation_gen.c
@@ -345,6 +345,7 @@ int main(int argc, char **argv) {
 	}
 	printf("\nBlockees will complete if boosted enough to finish %lu iterations\n", cfg.work_target);
 	printf("Press Ctrl+C to stop early\n");
+	fflush(stdout);
 
 	/*
 	 * Wait for specified duration OR until all blockees complete (whichever comes first).
-- 
2.53.0

[PATCH stalld 16/36] tests/helpers: Add start_starvation_gen() helper function

[Wander Lairson Costa]

Tests currently launch starvation_gen in the background and sleep
1-2 seconds hoping all threads are created and pinned. This is
non-deterministic and wastes time.

Add start_starvation_gen() that redirects starvation_gen stdout to
a log file and polls for the "Press Ctrl+C to stop early\n" message
emitted after all threads pass the pthread barrier. A brief initial
sleep covers the fast path, then the function falls back to
1-second polling with a 10-second timeout. On timeout, the process
is killed with SIGTERM and escalated to SIGKILL if it does not exit
within one second. The function sets STARVE_PID and STARVE_LOG for
test use and tracks both in CLEANUP_PIDS and CLEANUP_FILES.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 58 ++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 06c0471..276ffe8 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -1045,6 +1045,62 @@ init_functional_test() {
 	export TEST_CPU STALLD_CPU STARVE_GEN STALLD_LOG
 }
 
+# Start starvation_gen in background with readiness detection
+# Launches starvation_gen, redirects its stdout to a log file, and polls
+# for the "ready" message that starvation_gen prints after all threads
+# have passed the pthread barrier and are actively running/starving.
+#
+# Usage: start_starvation_gen [starvation_gen_args...]
+# Sets: STARVE_PID, STARVE_LOG
+# Example: start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d 15
+start_starvation_gen() {
+	local starve_bin="${TEST_ROOT}/helpers/starvation_gen"
+	if [ ! -x "${starve_bin}" ]; then
+		echo -e "${RED}ERROR: starvation_gen not found at ${starve_bin}${NC}"
+		return 1
+	fi
+
+	STARVE_LOG="/tmp/stalld_starvgen_$$.log"
+	CLEANUP_FILES+=("${STARVE_LOG}")
+
+	"${starve_bin}" "$@" > "${STARVE_LOG}" 2>&1 &
+	STARVE_PID=$!
+	CLEANUP_PIDS+=("${STARVE_PID}")
+
+	# Poll for "ready" message with timeout
+	# starvation_gen prints "ready" after all threads pass the barrier.
+	# Brief initial sleep covers the fast path, then 1-second polling
+	# for slow/loaded systems.
+	sleep 0.01
+
+	local timeout=10
+	local elapsed=0
+	while [ $elapsed -lt $timeout ]; do
+		if ! kill -0 ${STARVE_PID} 2>/dev/null; then
+			echo -e "${RED}ERROR: starvation_gen exited prematurely${NC}"
+			echo "  Log contents:"
+			cat "${STARVE_LOG}"
+			return 1
+		fi
+		if grep -q "Press Ctrl+C to stop early" "${STARVE_LOG}" 2>/dev/null; then
+			echo "starvation_gen ready (PID ${STARVE_PID})"
+			return 0
+		fi
+		sleep 1
+		elapsed=$((elapsed + 1))
+	done
+
+	echo -e "${RED}ERROR: starvation_gen did not become ready within ${timeout}s${NC}"
+	echo "  Log contents:"
+	cat "${STARVE_LOG}"
+	kill ${STARVE_PID} 2>/dev/null
+	sleep 1
+	if kill -0 ${STARVE_PID} 2>/dev/null; then
+		kill -9 ${STARVE_PID} 2>/dev/null
+	fi
+	return 1
+}
+
 # Export functions for use in tests
 export -f start_test end_test
 export -f assert_equals assert_contains assert_not_contains
@@ -1061,5 +1117,5 @@ export -f save_dl_server restore_dl_server disable_dl_server
 export -f setup_test_environment
 export -f get_num_cpus get_online_cpus pick_test_cpu
 export -f log get_sched_policy get_sched_priority get_nice_value get_ctxt_switches
-export -f start_stalld_with_log wait_for_policy_change
+export -f start_stalld_with_log start_starvation_gen wait_for_policy_change
 export -f calculate_detection_timeout init_functional_test
-- 
2.53.0

[PATCH stalld 17/36] tests/helpers: Add wait_for_starvation_detected() and wait_for_boost_detected()

[Wander Lairson Costa]

Tests that verify starvation detection and boosting currently use
hardcoded sleep calls to wait for stalld to act, making them
timing-dependent and flaky on slow systems. Subsequent commits will
replace ~48 such sleep calls across 10 test files.

Add two event-driven helpers that wrap wait_for_log_message() with
domain-specific patterns matching the log messages emitted by stalld
during starvation detection and task boosting. Both use a default
timeout of 30 seconds and return immediately when the event occurs.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 276ffe8..864036d 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -536,6 +536,24 @@ wait_for_stalld_ready() {
 	wait_for_log_message "checking cpu\|waiting tasks" "${timeout}" "${log_file}"
 }
 
+# Wait for stalld to detect a starving task.
+#
+# Usage: wait_for_starvation_detected <log_file> [timeout]
+wait_for_starvation_detected() {
+	local log_file=$1
+	local timeout=${2:-30}
+	wait_for_log_message "starved on CPU" "${timeout}" "${log_file}"
+}
+
+# Wait for stalld to boost a starving task.
+#
+# Usage: wait_for_boost_detected <log_file> [timeout]
+wait_for_boost_detected() {
+	local log_file=$1
+	local timeout=${2:-30}
+	wait_for_log_message "boosted pid" "${timeout}" "${log_file}"
+}
+
 # Get thread scheduling policy
 get_thread_policy() {
 	local pid=$1
@@ -1107,7 +1125,7 @@ export -f assert_equals assert_contains assert_not_contains
 export -f assert_file_exists assert_file_not_exists
 export -f assert_process_running assert_process_not_running
 export -f start_stalld stop_stalld kill_existing_stalld cleanup
-export -f wait_for_log_message wait_for_stalld_ready
+export -f wait_for_log_message wait_for_stalld_ready wait_for_starvation_detected wait_for_boost_detected
 export -f get_thread_policy get_thread_priority
 export -f create_cpu_load
 export -f detect_default_backend is_backend_available get_available_backends start_stalld_with_backend
-- 
2.53.0

[PATCH stalld 18/36] tests/functional: Use start_starvation_gen() helper

[Wander Lairson Costa]

Every functional test that creates starvation scenarios launches
starvation_gen manually: backgrounding the process, capturing the
PID, registering it for cleanup, and sleeping 1-3 seconds to wait
for startup. This duplicated boilerplate is fragile because the
sleep duration is an arbitrary guess that may be too short on slow
systems or too long on fast ones.

Replace all raw starvation_gen invocations across 14 test files
with the start_starvation_gen() helper introduced in Phase B. The
helper handles process management internally and uses event-driven
readiness detection instead of a fixed sleep, returning only after
starvation_gen confirms all threads are running. For tests that
need multiple simultaneous instances or use alternate PID variable
names, the global STARVE_PID is aliased immediately after the call.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_boost_duration.sh       | 16 ++------
 tests/functional/test_boost_period.sh         | 16 ++------
 tests/functional/test_boost_restoration.sh    | 24 +++---------
 tests/functional/test_boost_runtime.sh        | 16 ++------
 tests/functional/test_deadline_boosting.sh    | 26 ++++---------
 tests/functional/test_fifo_boosting.sh        | 35 +++---------------
 .../test_fifo_priority_starvation.sh          | 35 +++---------------
 tests/functional/test_force_fifo.sh           | 24 +++---------
 tests/functional/test_idle_detection.sh       |  8 +---
 tests/functional/test_log_only.sh             |  8 +---
 tests/functional/test_runqueue_parsing.sh     | 24 +++---------
 tests/functional/test_starvation_detection.sh | 37 ++++---------------
 tests/functional/test_starvation_threshold.sh | 21 ++---------
 tests/functional/test_task_merging.sh         | 22 ++++-------
 14 files changed, 70 insertions(+), 242 deletions(-)

diff --git a/tests/functional/test_boost_duration.sh b/tests/functional/test_boost_duration.sh
index 4df5489..c835fd3 100755
--- a/tests/functional/test_boost_duration.sh
+++ b/tests/functional/test_boost_duration.sh
@@ -63,9 +63,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -l > "${STA
 # Create starvation
 starvation_duration=15
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 wait_time=$((threshold + 2))
@@ -102,9 +100,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${short_
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 log "Waiting ${wait_time}s for detection"
@@ -141,9 +137,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${long_d
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${long_starvation}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${long_starvation} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${long_starvation}
 
 # Wait for detection
 log "Waiting ${wait_time}s for detection"
@@ -179,9 +173,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${durati
 
 # Create starvation with a specific task we can track
 log "Creating starvation on CPU ${TEST_CPU} for 15s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 1 -d 15 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 1 -d 15
 
 # Wait for detection
 log "Waiting ${wait_time}s for detection"
diff --git a/tests/functional/test_boost_period.sh b/tests/functional/test_boost_period.sh
index 97e0427..aab4436 100755
--- a/tests/functional/test_boost_period.sh
+++ b/tests/functional/test_boost_period.sh
@@ -60,9 +60,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -N > "${STALL
 # Create starvation
 starvation_duration=$((threshold + 5))
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 wait_time=$((threshold + 2))
@@ -104,9 +102,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $custom_pe
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 sleep ${wait_time}
@@ -139,9 +135,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $short_per
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 sleep ${wait_time}
@@ -174,9 +168,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $long_peri
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 sleep ${wait_time}
diff --git a/tests/functional/test_boost_restoration.sh b/tests/functional/test_boost_restoration.sh
index 1b8f809..beadf47 100755
--- a/tests/functional/test_boost_restoration.sh
+++ b/tests/functional/test_boost_restoration.sh
@@ -64,12 +64,9 @@ start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_dura
 
 # Create starvation (starvation_gen creates SCHED_FIFO blocker prio 80, blockee prio 1)
 log "Creating starvation with SCHED_FIFO tasks (blocker prio 80, blockee prio 1)"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -b 1 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -b 1 -n 1 -d 20
 
 # Find the starved task
-sleep 2
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
 tracked_pid=""
 for child_pid in ${STARVE_CHILDREN}; do
@@ -201,11 +198,8 @@ chmod +x /tmp/fifo_task_$$.sh /tmp/fifo_task_running_$$.sh
 CLEANUP_FILES+=("/tmp/fifo_task_$$.sh" "/tmp/fifo_task_running_$$.sh")
 
 # Also create the blocker that will starve our FIFO task
-"${STARVE_GEN}" -c ${TEST_CPU} -p 90 -n 1 -d 20 &
-BLOCKER_PID=$!
-CLEANUP_PIDS+=("${BLOCKER_PID}")
-
-sleep 1
+start_starvation_gen -c ${TEST_CPU} -p 90 -n 1 -d 20
+BLOCKER_PID=${STARVE_PID}
 
 # Start our FIFO task on the same CPU (it will starve)
 bash /tmp/fifo_task_running_$$.sh &
@@ -294,9 +288,7 @@ start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_dura
 
 # Use -o flag to create SCHED_OTHER blockees
 log "Creating SCHED_OTHER starvation (RT blocker prio 80, SCHED_OTHER blockee)"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -o -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -o -n 1 -d 20
 
 # Wait for starvation_gen to complete
 log "Waiting for starvation test to complete..."
@@ -326,9 +318,7 @@ start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_dura
 
 # Create starvation
 log "Creating starvation"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
 # Wait for starvation detection
 sleep $((threshold + 1))
@@ -389,9 +379,7 @@ start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_dura
 # This ensures the task exits DURING the boost period
 short_duration=$((threshold - 2))
 log "Creating starvation that will exit after ${short_duration}s"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d ${short_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d ${short_duration}
 
 # Give stalld time to detect starvation and start boosting
 # Need: threshold (10s) + buffer for detection (2s) = 12s
diff --git a/tests/functional/test_boost_runtime.sh b/tests/functional/test_boost_runtime.sh
index 0475da5..b8dfd1f 100755
--- a/tests/functional/test_boost_runtime.sh
+++ b/tests/functional/test_boost_runtime.sh
@@ -63,9 +63,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -l > "${STA
 # Create starvation
 starvation_duration=10
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 wait_time=$((threshold + 2))
@@ -102,9 +100,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${custom
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 log "Waiting ${wait_time}s for detection and boosting"
@@ -140,9 +136,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${large_
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 log "Waiting ${wait_time}s for detection and boosting"
@@ -180,9 +174,7 @@ start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${valid_
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 log "Waiting ${wait_time}s for detection and boosting"
diff --git a/tests/functional/test_deadline_boosting.sh b/tests/functional/test_deadline_boosting.sh
index d715186..de72100 100755
--- a/tests/functional/test_deadline_boosting.sh
+++ b/tests/functional/test_deadline_boosting.sh
@@ -63,9 +63,7 @@ start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALL
 # Create starvation
 starvation_duration=$((threshold + 8))
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting (threshold + granularity + buffer)
 # With -g 1, stalld checks every 1 second. In worst case, it checks just before
@@ -131,9 +129,7 @@ start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} \
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 15 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 15
 
 # Wait for boosting (threshold + granularity + buffer)
 wait_time=$((threshold + 1 + 3))
@@ -192,9 +188,7 @@ start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
 # Wait for starvation detection (threshold + granularity + buffer)
 wait_time=$((threshold + 1 + 3))
@@ -267,9 +261,7 @@ start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
 # Find a starved task and verify initial policy
 sleep 2
@@ -362,15 +354,13 @@ else
 
     # Create starvation on CPU0
     log "Creating starvation on CPU ${CPU0}"
-    "${STARVE_GEN}" -c ${CPU0} -p 80 -n 1 -d 15 &
-    STARVE_PID0=$!
-    CLEANUP_PIDS+=("${STARVE_PID0}")
+    start_starvation_gen -c ${CPU0} -p 80 -n 1 -d 15
+    STARVE_PID0=${STARVE_PID}
 
     # Create starvation on CPU1
     log "Creating starvation on CPU ${CPU1}"
-    "${STARVE_GEN}" -c ${CPU1} -p 80 -n 1 -d 15 &
-    STARVE_PID1=$!
-    CLEANUP_PIDS+=("${STARVE_PID1}")
+    start_starvation_gen -c ${CPU1} -p 80 -n 1 -d 15
+    STARVE_PID1=${STARVE_PID}
 
     # Wait for detection and boosting (threshold + granularity + buffer)
     wait_time=$((threshold + 1 + 3))
diff --git a/tests/functional/test_fifo_boosting.sh b/tests/functional/test_fifo_boosting.sh
index 7fbe297..a6168e9 100755
--- a/tests/functional/test_fifo_boosting.sh
+++ b/tests/functional/test_fifo_boosting.sh
@@ -56,12 +56,7 @@ threshold=5
 # Create starvation FIRST (before stalld starts)
 starvation_duration=$((threshold + 8))
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and create actual starvation
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with -F flag to force SCHED_FIFO boosting"
 # Note: -F requires non-single-threaded mode (aggressive mode)
@@ -109,12 +104,7 @@ rm -f "${STALLD_LOG}"
 
 # Create starvation FIRST
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 15 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and get child PIDs while they exist
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 15
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
 log "Starvation generator children PIDs: ${STARVE_CHILDREN}"
 
@@ -183,12 +173,7 @@ rm -f "${STALLD_LOG}"
 
 # Create starvation FIRST
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
 start_stalld -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} \
     -d ${boost_duration} -p ${boost_period} -r ${boost_runtime} \
@@ -233,12 +218,7 @@ STALLD_LOG_DEADLINE="/tmp/stalld_test_deadline_compare_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG_DEADLINE}")
 
 # Create starvation FIRST
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d 15 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Find a starved task and capture context switches immediately
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d 15
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
 deadline_tracked_pid=""
 for child_pid in ${STARVE_CHILDREN}; do
@@ -281,12 +261,7 @@ STALLD_LOG_FIFO="/tmp/stalld_test_fifo_compare_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG_FIFO}")
 
 # Create starvation FIRST
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d 15 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Find a starved task and capture context switches immediately
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d 15
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
 fifo_tracked_pid=""
 for child_pid in ${STARVE_CHILDREN}; do
diff --git a/tests/functional/test_fifo_priority_starvation.sh b/tests/functional/test_fifo_priority_starvation.sh
index 3459f3c..cf47971 100755
--- a/tests/functional/test_fifo_priority_starvation.sh
+++ b/tests/functional/test_fifo_priority_starvation.sh
@@ -64,12 +64,7 @@ starvation_duration=$((threshold + 5))
 log "Creating FIFO-on-FIFO starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 log "  Blocker: SCHED_FIFO priority 10"
 log "  Blockee: SCHED_FIFO priority 5"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 10 -b 5 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and pin to CPU
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 10 -b 5 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
@@ -124,12 +119,7 @@ threshold=5
 boost_duration=3
 
 log "Creating FIFO-on-FIFO starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 10 -b 5 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 10 -b 5 -n 1 -d 20
 
 # Find the starved task (blockee) PID
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
@@ -198,12 +188,7 @@ threshold=3
 # Create long starvation to trigger multiple detection cycles
 starvation_duration=15
 log "Creating long FIFO-on-FIFO starvation for ${starvation_duration}s"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 10 -b 5 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 10 -b 5 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 log "Will monitor for multiple detection cycles to verify timestamp preservation"
@@ -267,12 +252,7 @@ threshold=5
 log "Creating FIFO-on-FIFO starvation with close priorities"
 log "  Blocker: SCHED_FIFO priority 6"
 log "  Blockee: SCHED_FIFO priority 5"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 6 -b 5 -n 1 -d $((threshold + 5)) &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 6 -b 5 -n 1 -d $((threshold + 5))
 
 log "Starting stalld with ${threshold}s threshold"
 start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
@@ -309,12 +289,7 @@ rm -f "${STALLD_LOG}"
 threshold=5
 
 log "Creating FIFO-on-FIFO starvation"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 10 -b 5 -n 2 -d 20 -v &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and print PIDs
-sleep 3
+start_starvation_gen -c ${TEST_CPU} -p 10 -b 5 -n 2 -d 20 -v
 
 # Extract blocker and blockee PIDs from starvation_gen output
 # The output shows "Blocker TID: <pid>" and "Blockee N TID: <pid>"
diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index cab8003..4c43101 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -56,9 +56,7 @@ start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} > "${STALLD_LOG}" 2>&1
 # Create starvation
 starvation_duration=10
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 wait_time=$((threshold + 2))
@@ -104,9 +102,7 @@ start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A > "${STALLD_LOG2}" 2>&
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 log "Waiting ${wait_time}s for detection and boosting"
@@ -150,9 +146,7 @@ start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A > "${STALLD_LOG3}" 2>&
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 log "Waiting ${wait_time}s for detection and boosting"
@@ -192,9 +186,7 @@ start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A -d ${boost_duration} >
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${long_starvation}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${long_starvation} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${long_starvation}
 
 # Wait for detection and boosting
 log "Waiting ${wait_time}s for detection and boosting"
@@ -270,9 +262,7 @@ CLEANUP_FILES+=("${STALLD_LOG_DL}")
 
 log "Running comparison test with SCHED_DEADLINE"
 start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -d ${comparison_duration} > "${STALLD_LOG_DL}" 2>&1
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${comparison_starvation} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${comparison_starvation}
 sleep $((threshold + 3))
 
 deadline_boosts=$(grep -c "boost" "${STALLD_LOG_DL}" || echo 0)
@@ -288,9 +278,7 @@ CLEANUP_FILES+=("${STALLD_LOG_FIFO}")
 
 log "Running comparison test with SCHED_FIFO"
 start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A -d ${comparison_duration} > "${STALLD_LOG_FIFO}" 2>&1
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${comparison_starvation} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${comparison_starvation}
 sleep $((threshold + 3))
 
 fifo_boosts=$(grep -c "boost" "${STALLD_LOG_FIFO}" || echo 0)
diff --git a/tests/functional/test_idle_detection.sh b/tests/functional/test_idle_detection.sh
index c3aa565..755a4b5 100755
--- a/tests/functional/test_idle_detection.sh
+++ b/tests/functional/test_idle_detection.sh
@@ -160,9 +160,7 @@ sleep 3
 
 # Now create load to make CPU busy
 log "Creating load on CPU ${TEST_CPU} to make it busy"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d 12 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d 12
 
 # Wait for stalld to detect the busy CPU and starvation (threshold + granularity + buffer)
 log "Waiting for stalld to detect busy CPU and starvation..."
@@ -232,9 +230,7 @@ else
 
     # Create load only on CPU1, leave CPU0 idle
     log "Creating load on CPU ${CPU1} only"
-    "${STARVE_GEN}" -c ${CPU1} -p 80 -n 2 -d 12 &
-    STARVE_PID=$!
-    CLEANUP_PIDS+=("${STARVE_PID}")
+    start_starvation_gen -c ${CPU1} -p 80 -n 2 -d 12
 
     # Wait for detection (threshold + granularity + buffer)
     wait_time=$((threshold + 1 + 3))
diff --git a/tests/functional/test_log_only.sh b/tests/functional/test_log_only.sh
index cfd1b93..93c71cb 100755
--- a/tests/functional/test_log_only.sh
+++ b/tests/functional/test_log_only.sh
@@ -47,12 +47,8 @@ CLEANUP_FILES+=("${LOG_FILE}")
 echo "Creating starvation on CPU ${TEST_CPU} (will run for 15 seconds)"
 
 # Start starvation generator BEFORE stalld to ensure CPU is busy from the start
-${TEST_ROOT}/helpers/starvation_gen -c ${TEST_CPU} -p 10 -n 1 -d 15 &
-STARVGEN_PID=$!
-CLEANUP_PIDS+=("${STARVGEN_PID}")
-
-# Give the starvation generator time to start and monopolize the CPU
-sleep 1
+start_starvation_gen -c ${TEST_CPU} -p 10 -n 1 -d 15
+STARVGEN_PID=${STARVE_PID}
 
 # Start stalld in log-only mode with verbose output to capture logs
 echo "Starting stalld in log-only mode with 5 second threshold"
diff --git a/tests/functional/test_runqueue_parsing.sh b/tests/functional/test_runqueue_parsing.sh
index 2f23910..7edf936 100755
--- a/tests/functional/test_runqueue_parsing.sh
+++ b/tests/functional/test_runqueue_parsing.sh
@@ -98,9 +98,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ]; then
     # Create starvation to generate task data
     starvation_duration=$((threshold + 5))
     log "Creating starvation for ${starvation_duration}s"
-    "${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-    STARVE_PID=$!
-    CLEANUP_PIDS+=("${STARVE_PID}")
+    start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
     # Wait for detection (threshold + granularity + buffer)
     wait_time=$((threshold + 1 + 3))
@@ -159,9 +157,7 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     # Create starvation
     starvation_duration=$((threshold + 5))
     log "Creating starvation for ${starvation_duration}s"
-    "${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-    STARVE_PID=$!
-    CLEANUP_PIDS+=("${STARVE_PID}")
+    start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
     # Wait for detection (threshold + granularity + buffer)
     wait_time=$((threshold + 1 + 3))
@@ -229,9 +225,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ] && [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     rm -f "${STALLD_LOG_BPF}"
     start_stalld -f -v -g 1 -l -b queue_track -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG_BPF}" 2>&1
 
-    "${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-    STARVE_PID=$!
-    CLEANUP_PIDS+=("${STARVE_PID}")
+    start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
     wait_time=$((threshold + 1 + 3))
     sleep ${wait_time}
@@ -251,9 +245,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ] && [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     rm -f "${STALLD_LOG_SCHED}"
     start_stalld -f -v -g 1 -l -b sched_debug -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG_SCHED}" 2>&1
 
-    "${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-    STARVE_PID=$!
-    CLEANUP_PIDS+=("${STARVE_PID}")
+    start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
     wait_time=$((threshold + 1 + 3))
     sleep ${wait_time}
@@ -320,9 +312,7 @@ if [ -n "$test_backend" ]; then
 
     # Create starvation with known parameters
     log "Creating starvation with known task name (starvation_gen)"
-    "${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 10 -v &
-    STARVE_PID=$!
-    CLEANUP_PIDS+=("${STARVE_PID}")
+    start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 10 -v
 
     # Wait for detection (threshold + granularity + buffer)
     wait_time=$((threshold + 1 + 3))
@@ -384,9 +374,7 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     start_stalld -f -v -g 1 -l -b sched_debug -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG_SCHED}" 2>&1
 
     # Create brief starvation just to initialize the backend
-    "${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 8 &
-    STARVE_PID=$!
-    CLEANUP_PIDS+=("${STARVE_PID}")
+    start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 8
 
     wait_time=$((threshold + 1 + 3))
     sleep ${wait_time}
diff --git a/tests/functional/test_starvation_detection.sh b/tests/functional/test_starvation_detection.sh
index 89901d3..6a1821d 100755
--- a/tests/functional/test_starvation_detection.sh
+++ b/tests/functional/test_starvation_detection.sh
@@ -67,12 +67,7 @@ threshold=5
 # Create starvation BEFORE starting stalld to avoid idle detection race
 starvation_duration=$((threshold + 5))
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and pin to CPU
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
@@ -129,12 +124,7 @@ threshold=5
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 15 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 15
 
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
@@ -194,12 +184,7 @@ threshold=3
 # Create long starvation to trigger multiple detection cycles
 starvation_duration=15
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start
-sleep 2
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 log "Will monitor for multiple detection cycles to verify timestamp preservation"
@@ -292,18 +277,13 @@ else
 
     # Create starvation on CPU0
     log "Creating starvation on CPU ${CPU0}"
-    "${STARVE_GEN}" -c ${CPU0} -p 80 -n 1 -d 12 &
-    STARVE_PID0=$!
-    CLEANUP_PIDS+=("${STARVE_PID0}")
+    start_starvation_gen -c ${CPU0} -p 80 -n 1 -d 12
+    STARVE_PID0=${STARVE_PID}
 
     # Create starvation on CPU1
     log "Creating starvation on CPU ${CPU1}"
-    "${STARVE_GEN}" -c ${CPU1} -p 80 -n 1 -d 12 &
-    STARVE_PID1=$!
-    CLEANUP_PIDS+=("${STARVE_PID1}")
-
-    # Give starvation generators time to start
-    sleep 2
+    start_starvation_gen -c ${CPU1} -p 80 -n 1 -d 12
+    STARVE_PID1=${STARVE_PID}
 
     start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${CPU0},${CPU1} -a ${STALLD_CPU_MULTI}
 
@@ -394,8 +374,7 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a $
 
 # Create short-lived starvation that exits before threshold
 log "Creating short-lived starvation (3s, less than ${threshold}s threshold)"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 3 &
-STARVE_PID=$!
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 3
 
 # Wait for task to exit
 sleep 5
diff --git a/tests/functional/test_starvation_threshold.sh b/tests/functional/test_starvation_threshold.sh
index 41a2cca..45a4679 100755
--- a/tests/functional/test_starvation_threshold.sh
+++ b/tests/functional/test_starvation_threshold.sh
@@ -65,12 +65,7 @@ threshold=5
 # Create starvation BEFORE starting stalld (avoid detecting kworker tasks)
 starvation_duration=10
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and create actual starvation
-sleep 2
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold"
 # Use -i to ignore kernel workers that may starve before our test tasks
@@ -115,12 +110,7 @@ CLEANUP_FILES+=("${STALLD_LOG2}")
 # Create starvation that will last 6 seconds (less than threshold)
 starvation_duration=6
 log "Creating short starvation (${starvation_duration}s) with threshold of ${threshold}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and create actual starvation
-sleep 2
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold"
 start_stalld -f -v -N -M -g 1 -c "${TEST_CPU}" -a "${STALLD_CPU}" -t ${threshold} > "${STALLD_LOG2}" 2>&1
@@ -165,12 +155,7 @@ CLEANUP_FILES+=("${STALLD_LOG3}")
 # Create starvation for 8 seconds
 starvation_duration=8
 log "Creating starvation for ${starvation_duration}s with threshold of ${threshold}s"
-"${STARVE_GEN}" -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
-
-# Give starvation generator time to start and create actual starvation
-sleep 2
+start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold"
 # Use -i to ignore kernel workers that may starve before our test tasks
diff --git a/tests/functional/test_task_merging.sh b/tests/functional/test_task_merging.sh
index 053c648..1bea759 100755
--- a/tests/functional/test_task_merging.sh
+++ b/tests/functional/test_task_merging.sh
@@ -70,9 +70,7 @@ start_stalld -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${ST
 # Create long starvation to span multiple monitoring cycles
 starvation_duration=18
 log "Creating starvation for ${starvation_duration}s (multiple detection cycles)"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration} &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for first detection (threshold + granularity + buffer)
 log "Waiting for first detection cycle..."
@@ -150,9 +148,7 @@ start_stalld -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${ST
 
 # Create starvation
 log "Creating starvation"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
 # Wait for detection (threshold + granularity + buffer)
 wait_time=$((threshold + 1 + 3))
@@ -226,9 +222,7 @@ start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d 2 > "${
 
 # Create starvation that will get boosted (allowing progress)
 log "Creating starvation that will be boosted"
-"${STARVE_GEN}" -c ${TEST_CPU} -p 80 -n 1 -d 20 &
-STARVE_PID=$!
-CLEANUP_PIDS+=("${STARVE_PID}")
+start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
 # Wait for first detection and boost (threshold + granularity + buffer)
 wait_time=$((threshold + 1 + 3))
@@ -296,14 +290,12 @@ else
 
     # Create starvation on both CPUs
     log "Creating starvation on CPU ${CPU0}"
-    "${STARVE_GEN}" -c ${CPU0} -p 80 -n 1 -d 15 &
-    STARVE_PID0=$!
-    CLEANUP_PIDS+=("${STARVE_PID0}")
+    start_starvation_gen -c ${CPU0} -p 80 -n 1 -d 15
+    STARVE_PID0=${STARVE_PID}
 
     log "Creating starvation on CPU ${CPU1}"
-    "${STARVE_GEN}" -c ${CPU1} -p 80 -n 1 -d 15 &
-    STARVE_PID1=$!
-    CLEANUP_PIDS+=("${STARVE_PID1}")
+    start_starvation_gen -c ${CPU1} -p 80 -n 1 -d 15
+    STARVE_PID1=${STARVE_PID}
 
     # Wait for multiple detection cycles (threshold + granularity + buffer, then more)
     wait_time=$((threshold + 1 + 3))
-- 
2.53.0

[PATCH stalld 19/36] tests/functional: Replace detection sleeps with event-driven helpers

[Wander Lairson Costa]

Functional tests wait for stalld to detect starvation or boost tasks
by sleeping a fixed duration then grepping the log. These hardcoded
sleeps add unnecessary latency on fast systems and can be too short
on slow ones, causing flaky results.

Replace the sleep-then-grep pattern with wait_for_starvation_detected()
and wait_for_boost_detected(), which use tail -f internally to return
as soon as the expected log message appears. For tests that combine
detection and boost-expiry waits in a single sleep, split them into
a helper call for detection followed by a duration sleep for expiry.
Sleeps for negative tests, boost expiry, and measurement windows are
intentionally preserved.

Additionally, migrate all start_stalld calls that redirect to a log
file to use start_stalld_with_log(), which provides event-driven
readiness detection and adds -g 1 for faster scan cycles. Without
this, stalld's default 5-second granularity caused detection timeouts
on systems where the starvation window was too short. Fix a pre-existing
bug in test_boost_duration where -d 10 exceeded -t 3, causing stalld
to reject the arguments. Fix a pre-existing bug in test_boost_period
where -p values of 100ms and 10s were rejected by stalld.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_boost_duration.sh       | 42 +++++----------
 tests/functional/test_boost_period.sh         | 46 ++++++----------
 tests/functional/test_boost_restoration.sh    | 28 ++++------
 tests/functional/test_boost_runtime.sh        | 33 +++---------
 tests/functional/test_deadline_boosting.sh    | 52 +++++++------------
 tests/functional/test_fifo_boosting.sh        | 50 +++++++++---------
 .../test_fifo_priority_starvation.sh          | 36 ++++++-------
 tests/functional/test_force_fifo.sh           | 37 +++++--------
 tests/functional/test_idle_detection.sh       | 17 +++---
 tests/functional/test_log_only.sh             |  5 +-
 tests/functional/test_runqueue_parsing.sh     | 31 ++++-------
 tests/functional/test_starvation_detection.sh | 39 +++++---------
 tests/functional/test_starvation_threshold.sh | 25 ++++-----
 tests/functional/test_task_merging.sh         | 28 +++++-----
 14 files changed, 176 insertions(+), 293 deletions(-)

diff --git a/tests/functional/test_boost_duration.sh b/tests/functional/test_boost_duration.sh
index c835fd3..af657db 100755
--- a/tests/functional/test_boost_duration.sh
+++ b/tests/functional/test_boost_duration.sh
@@ -58,20 +58,15 @@ log "=========================================="
 
 threshold=3
 log "Starting stalld with ${threshold}s threshold (default boost duration)"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -l > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -l
 
 # Create starvation
 starvation_duration=15
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for detection and boosting
-wait_time=$((threshold + 2))
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-# Check if boosting occurred (in log-only mode we look for detection messages)
-if grep -qi "detect\|starv" "${STALLD_LOG}"; then
+# Wait for starvation detection
+if wait_for_starvation_detected "${STALLD_LOG}"; then
     log "✓ PASS: Starvation detection occurred with default duration"
 else
     log "✗ FAIL: No starvation detection"
@@ -96,18 +91,14 @@ STALLD_LOG2="/tmp/stalld_test_boost_duration_test2_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG2}")
 
 log "Starting stalld with ${threshold}s threshold and ${short_duration}s boost duration"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${short_duration} -l > "${STALLD_LOG2}" 2>&1
+start_stalld_with_log "${STALLD_LOG2}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${short_duration} -l
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for detection and boosting
-log "Waiting ${wait_time}s for detection"
-sleep ${wait_time}
-
-# Check if detection occurred
-if grep -qi "detect\|starv" "${STALLD_LOG2}"; then
+# Wait for starvation detection
+if wait_for_starvation_detected "${STALLD_LOG2}"; then
     log "✓ PASS: Starvation detection with ${short_duration}s duration"
 else
     log "✗ FAIL: No starvation detection with short duration"
@@ -129,22 +120,19 @@ log "=========================================="
 
 long_duration=10
 long_starvation=20
+threshold=10
 STALLD_LOG3="/tmp/stalld_test_boost_duration_test3_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG3}")
 
 log "Starting stalld with ${threshold}s threshold and ${long_duration}s boost duration"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${long_duration} -l > "${STALLD_LOG3}" 2>&1
+start_stalld_with_log "${STALLD_LOG3}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${long_duration} -l
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${long_starvation}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${long_starvation}
 
-# Wait for detection
-log "Waiting ${wait_time}s for detection"
-sleep ${wait_time}
-
-# Check if detection occurred
-if grep -qi "detect\|starv" "${STALLD_LOG3}"; then
+# Wait for starvation detection
+if wait_for_starvation_detected "${STALLD_LOG3}"; then
     log "✓ PASS: Starvation detection with ${long_duration}s duration"
 else
     log "✗ FAIL: No starvation detection with long duration"
@@ -164,22 +152,20 @@ log "=========================================="
 log "Test 4: Verify policy restoration after boost duration"
 log "=========================================="
 
+threshold=3
 duration=2
 STALLD_LOG4="/tmp/stalld_test_boost_duration_test4_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG4}")
 
 log "Starting stalld with ${threshold}s threshold and ${duration}s boost duration"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${duration} -l > "${STALLD_LOG4}" 2>&1
+start_stalld_with_log "${STALLD_LOG4}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -d ${duration} -l
 
 # Create starvation with a specific task we can track
 log "Creating starvation on CPU ${TEST_CPU} for 15s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 1 -d 15
 
-# Wait for detection
-log "Waiting ${wait_time}s for detection"
-sleep ${wait_time}
-
-if grep -qi "detect\|starv" "${STALLD_LOG4}"; then
+# Wait for starvation detection
+if wait_for_starvation_detected "${STALLD_LOG4}"; then
     log "✓ PASS: Starvation detection with ${duration}s boost duration"
 else
     log "✗ FAIL: No starvation detection"
diff --git a/tests/functional/test_boost_period.sh b/tests/functional/test_boost_period.sh
index aab4436..4838672 100755
--- a/tests/functional/test_boost_period.sh
+++ b/tests/functional/test_boost_period.sh
@@ -55,20 +55,15 @@ log "=========================================="
 
 threshold=5
 log "Starting stalld with default period"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -N > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -N
 
 # Create starvation
 starvation_duration=$((threshold + 5))
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for detection and boosting
-wait_time=$((threshold + 2))
-log "Waiting ${wait_time}s for starvation detection and boosting..."
-sleep ${wait_time}
-
-# Check if boosting occurred
-if grep -q "boost" "${STALLD_LOG}"; then
+# Wait for starvation detection and boosting
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boosting occurred with default period"
 
     # Try to find period value in logs
@@ -98,17 +93,14 @@ log "=========================================="
 custom_period=500000000
 rm -f "${STALLD_LOG}"
 log "Starting stalld with custom period ${custom_period} ns"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $custom_period -N > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $custom_period -N
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for detection and boosting
-sleep ${wait_time}
-
-# Check if boosting occurred
-if grep -q "boost" "${STALLD_LOG}"; then
+# Wait for starvation detection and boosting
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boosting occurred with custom period ${custom_period} ns"
 else
     log "✗ FAIL: No boosting with custom period"
@@ -125,23 +117,20 @@ stop_stalld
 #=============================================================================
 log ""
 log "=========================================="
-log "Test 3: Very short period of 100,000,000 ns (100ms)"
+log "Test 3: Short period of 200,000,000 ns (200ms)"
 log "=========================================="
 
-short_period=100000000
+short_period=200000000
 rm -f "${STALLD_LOG}"
 log "Starting stalld with short period ${short_period} ns"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $short_period -N > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $short_period -N
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for detection and boosting
-sleep ${wait_time}
-
-# Check if boosting occurred
-if grep -q "boost" "${STALLD_LOG}"; then
+# Wait for starvation detection and boosting
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boosting occurred with short period ${short_period} ns"
 else
     log "✗ FAIL: No boosting with short period"
@@ -158,23 +147,20 @@ stop_stalld
 #=============================================================================
 log ""
 log "=========================================="
-log "Test 4: Very long period of 10,000,000,000 ns (10s)"
+log "Test 4: Long period of 3,000,000,000 ns (3s)"
 log "=========================================="
 
-long_period=10000000000
+long_period=3000000000
 rm -f "${STALLD_LOG}"
 log "Starting stalld with long period ${long_period} ns"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $long_period -N > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t $threshold -p $long_period -N
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for detection and boosting
-sleep ${wait_time}
-
-# Check if boosting occurred
-if grep -q "boost" "${STALLD_LOG}"; then
+# Wait for starvation detection and boosting
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boosting occurred with long period ${long_period} ns"
 else
     log "✗ FAIL: No boosting with long period"
diff --git a/tests/functional/test_boost_restoration.sh b/tests/functional/test_boost_restoration.sh
index beadf47..fc5a25a 100755
--- a/tests/functional/test_boost_restoration.sh
+++ b/tests/functional/test_boost_restoration.sh
@@ -60,7 +60,7 @@ boost_duration=3
 
 log "Starting stalld with ${boost_duration}s boost duration"
 # Use -i to ignore kworkers so stalld focuses on our test workload
-start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker" > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker"
 
 # Create starvation (starvation_gen creates SCHED_FIFO blocker prio 80, blockee prio 1)
 log "Creating starvation with SCHED_FIFO tasks (blocker prio 80, blockee prio 1)"
@@ -92,7 +92,7 @@ if [ -n "${tracked_pid}" ]; then
 
     # Wait for starvation detection and boosting
     log "Waiting for starvation detection and boost..."
-    sleep $((threshold + 1))
+    wait_for_boost_detected "${STALLD_LOG}"
 
     # Check policy during boost (should be DEADLINE=6)
     if [ -f "/proc/${tracked_pid}/sched" ]; then
@@ -170,7 +170,7 @@ boost_duration=3
 
 log "Starting stalld"
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker" > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker"
 
 # Create a SCHED_FIFO task that will starve
 # We'll create our own RT task instead of using starvation_gen
@@ -226,7 +226,7 @@ if chrt -f -p 10 ${FIFO_TASK_PID} 2>/dev/null; then
 
     # Wait for starvation detection
     log "Waiting for starvation detection and boost..."
-    sleep $((threshold + 1))
+    wait_for_boost_detected "${STALLD_LOG}"
 
     # Check if boosted
     if [ -f "/proc/${FIFO_TASK_PID}/sched" ]; then
@@ -284,7 +284,7 @@ boost_duration=3
 
 log "Starting stalld"
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker" > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker"
 
 # Use -o flag to create SCHED_OTHER blockees
 log "Creating SCHED_OTHER starvation (RT blocker prio 80, SCHED_OTHER blockee)"
@@ -314,17 +314,14 @@ boost_duration=4  # 4 second boost
 
 log "Starting stalld with ${boost_duration}s boost duration"
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker" > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker"
 
 # Create starvation
 log "Creating starvation"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
-# Wait for starvation detection
-sleep $((threshold + 1))
-
-# Check when boost occurred
-if grep -q "boosted" "${STALLD_LOG}"; then
+# Wait for starvation detection and boosting
+if wait_for_boost_detected "${STALLD_LOG}"; then
     boost_time=$(date +%s)
     log "Boost detected at timestamp: ${boost_time}"
 
@@ -373,7 +370,7 @@ boost_duration=5  # Task will exit during boost (after 8s, boost is 5s)
 
 log "Starting stalld with ${threshold}s threshold, ${boost_duration}s boost (task will exit during boost)"
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker" > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} -N -i "kworker"
 
 # Create starvation that exits after threshold - 2s (so 8s)
 # This ensures the task exits DURING the boost period
@@ -381,11 +378,8 @@ short_duration=$((threshold - 2))
 log "Creating starvation that will exit after ${short_duration}s"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d ${short_duration}
 
-# Give stalld time to detect starvation and start boosting
-# Need: threshold (10s) + buffer for detection (2s) = 12s
-sleep $((threshold + 2))
-
-if grep -q "boosted" "${STALLD_LOG}"; then
+# Wait for starvation detection and boosting
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boost occurred"
 
     # At this point (12s), starvation_gen has exited (at 8s) during the boost
diff --git a/tests/functional/test_boost_runtime.sh b/tests/functional/test_boost_runtime.sh
index b8dfd1f..734a6f8 100755
--- a/tests/functional/test_boost_runtime.sh
+++ b/tests/functional/test_boost_runtime.sh
@@ -58,7 +58,7 @@ log "=========================================="
 
 threshold=3
 log "Starting stalld with ${threshold}s threshold (default boost runtime)"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -l > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -l
 
 # Create starvation
 starvation_duration=10
@@ -66,12 +66,7 @@ log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
-wait_time=$((threshold + 2))
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-# Check if detection occurred
-if grep -qi "detect\|starv" "${STALLD_LOG}"; then
+if wait_for_starvation_detected "${STALLD_LOG}"; then
     log "✓ PASS: Starvation detection with default runtime"
 else
     log "✗ FAIL: No starvation detection"
@@ -96,18 +91,14 @@ STALLD_LOG2="/tmp/stalld_test_boost_runtime_test2_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG2}")
 
 log "Starting stalld with ${threshold}s threshold and ${custom_runtime}ns runtime"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${custom_runtime} -l > "${STALLD_LOG2}" 2>&1
+start_stalld_with_log "${STALLD_LOG2}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${custom_runtime} -l
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-# Check if detection occurred
-if grep -qi "detect\|starv" "${STALLD_LOG2}"; then
+if wait_for_starvation_detected "${STALLD_LOG2}"; then
     log "✓ PASS: Starvation detection with custom runtime ${custom_runtime}ns"
 else
     log "✗ FAIL: No starvation detection with custom runtime"
@@ -132,18 +123,14 @@ STALLD_LOG3="/tmp/stalld_test_boost_runtime_test3_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG3}")
 
 log "Starting stalld with ${threshold}s threshold and ${large_runtime}ns runtime"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${large_runtime} -l > "${STALLD_LOG3}" 2>&1
+start_stalld_with_log "${STALLD_LOG3}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${large_runtime} -l
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-# Check if detection occurred
-if grep -qi "detect\|starv" "${STALLD_LOG3}"; then
+if wait_for_starvation_detected "${STALLD_LOG3}"; then
     log "✓ PASS: Starvation detection with large runtime ${large_runtime}ns"
 else
     log "✗ FAIL: No starvation detection with large runtime"
@@ -170,18 +157,14 @@ STALLD_LOG4="/tmp/stalld_test_boost_runtime_test4_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG4}")
 
 log "Starting stalld with runtime ${valid_runtime}ns < period ${period}ns"
-start_stalld -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${valid_runtime} -p ${period} -l > "${STALLD_LOG4}" 2>&1
+start_stalld_with_log "${STALLD_LOG4}" -f -v -c "${TEST_CPU}" -a ${STALLD_CPU} -t ${threshold} -r ${valid_runtime} -p ${period} -l
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-# Check if detection occurred
-if grep -qi "detect\|starv" "${STALLD_LOG4}"; then
+if wait_for_starvation_detected "${STALLD_LOG4}"; then
     log "✓ PASS: Starvation detection with runtime < period"
 else
     log "✗ FAIL: No starvation detection when runtime < period"
diff --git a/tests/functional/test_deadline_boosting.sh b/tests/functional/test_deadline_boosting.sh
index de72100..a5727da 100755
--- a/tests/functional/test_deadline_boosting.sh
+++ b/tests/functional/test_deadline_boosting.sh
@@ -58,22 +58,16 @@ log "=========================================="
 threshold=5
 log "Starting stalld with ${threshold}s threshold (default DEADLINE boosting)"
 # Use -g 1 for 1-second granularity to ensure timely detection
-start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
 # Create starvation
 starvation_duration=$((threshold + 8))
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for detection and boosting (threshold + granularity + buffer)
-# With -g 1, stalld checks every 1 second. In worst case, it checks just before
-# threshold is reached, then waits another granularity period.
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for starvation detection and boosting (threshold: ${threshold}s, granularity: 1s)..."
-sleep ${wait_time}
-
-# Verify boosting occurred
-if grep -q "boosted" "${STALLD_LOG}"; then
+# Wait for boosting
+log "Waiting for boost detection..."
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boosting occurred"
 
     # Verify SCHED_DEADLINE was used
@@ -123,18 +117,16 @@ log "  Runtime: ${boost_runtime}ns (50µs)"
 log "  Duration: ${boost_duration}s"
 
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} \
-    -p ${boost_period} -r ${boost_runtime} -d ${boost_duration} \
-    > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} \
+    -p ${boost_period} -r ${boost_runtime} -d ${boost_duration}
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 15
 
-# Wait for boosting (threshold + granularity + buffer)
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for detection..."
-sleep ${wait_time}
+# Wait for boosting
+log "Waiting for boost detection..."
+wait_for_boost_detected "${STALLD_LOG}"
 
 # Try to find the boosted task PID
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
@@ -184,16 +176,15 @@ boost_duration=5
 
 log "Starting stalld with ${boost_duration}s boost duration"
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration}
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
-# Wait for starvation detection (threshold + granularity + buffer)
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for detection..."
-sleep ${wait_time}
+# Wait for boosting
+log "Waiting for boost detection..."
+wait_for_boost_detected "${STALLD_LOG}"
 
 # Find a starved task
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
@@ -257,7 +248,7 @@ boost_duration=3
 
 log "Starting stalld with ${boost_duration}s boost duration"
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration}
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU}"
@@ -285,10 +276,8 @@ if [ -n "${tracked_pid}" ]; then
         log "⚠ WARNING: Initial policy is not SCHED_OTHER (got ${initial_policy})"
     fi
 
-    # Wait for starvation detection and boosting (threshold + granularity + buffer)
-    wait_time=$((threshold + 1 + 3))
-    log "Waiting ${wait_time}s for detection..."
-    sleep ${wait_time}
+    # Wait for starvation detection and boosting
+    wait_for_boost_detected "${STALLD_LOG}"
 
     # Check if policy changed to DEADLINE during boost
     boosted_policy=$(get_sched_policy ${tracked_pid})
@@ -350,7 +339,7 @@ else
     log "Testing simultaneous boosts on CPU ${CPU0} and CPU ${CPU1}"
 
     rm -f "${STALLD_LOG}"
-    start_stalld -f -v -g 1 -t $threshold -c ${CPU0},${CPU1} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+    start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -t $threshold -c ${CPU0},${CPU1} -a ${STALLD_CPU}
 
     # Create starvation on CPU0
     log "Creating starvation on CPU ${CPU0}"
@@ -362,10 +351,9 @@ else
     start_starvation_gen -c ${CPU1} -p 80 -n 1 -d 15
     STARVE_PID1=${STARVE_PID}
 
-    # Wait for detection and boosting (threshold + granularity + buffer)
-    wait_time=$((threshold + 1 + 3))
-    log "Waiting ${wait_time}s for detection..."
-    sleep ${wait_time}
+    # Wait for boosting on both CPUs
+    log "Waiting for boost detection..."
+    wait_for_boost_detected "${STALLD_LOG}"
 
     # Count boost messages
     boost_count=$(grep -c "boosted" "${STALLD_LOG}")
diff --git a/tests/functional/test_fifo_boosting.sh b/tests/functional/test_fifo_boosting.sh
index a6168e9..3088141 100755
--- a/tests/functional/test_fifo_boosting.sh
+++ b/tests/functional/test_fifo_boosting.sh
@@ -61,15 +61,11 @@ start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 log "Starting stalld with -F flag to force SCHED_FIFO boosting"
 # Note: -F requires non-single-threaded mode (aggressive mode)
 # Use -g 1 for 1-second granularity to ensure timely detection
-start_stalld -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
-# Wait for detection and boosting (threshold + granularity + buffer)
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for starvation detection and boosting (threshold: ${threshold}s, granularity: 1s)..."
-sleep ${wait_time}
-
-# Verify FIFO boosting occurred
-if grep -qiE "boosted.*(SCHED_FIFO|FIFO)|FIFO.*boost" "${STALLD_LOG}"; then
+# Wait for boosting
+log "Waiting for boost detection..."
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boosting occurred with -F flag"
 
     # Verify SCHED_FIFO was used
@@ -109,12 +105,11 @@ STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
 log "Starvation generator children PIDs: ${STARVE_CHILDREN}"
 
 log "Starting stalld with -F flag (FIFO boosting)"
-start_stalld -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
-# Wait for boosting (threshold + granularity + buffer)
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for detection..."
-sleep ${wait_time}
+# Wait for boosting
+log "Waiting for boost detection..."
+wait_for_boost_detected "${STALLD_LOG}"
 
 fifo_task_found=0
 for child_pid in ${STARVE_CHILDREN}; do
@@ -175,13 +170,16 @@ rm -f "${STALLD_LOG}"
 log "Creating starvation on CPU ${TEST_CPU}"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
-start_stalld -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} \
-    -d ${boost_duration} -p ${boost_period} -r ${boost_runtime} \
-    > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} \
+    -d ${boost_duration} -p ${boost_period} -r ${boost_runtime}
+
+# Wait for boosting to start
+log "Waiting for boost detection..."
+wait_for_boost_detected "${STALLD_LOG}"
 
-# Wait for boosting to complete (threshold + granularity + boost_duration + buffer)
+# Wait for FIFO emulation cycles to complete (boost_duration + buffer)
 log "Waiting for FIFO emulation cycles to complete..."
-sleep $((threshold + 1 + boost_duration + 2))
+sleep $((boost_duration + 2))
 
 # Count boost events (FIFO emulation creates multiple boosts)
 boost_count=$(grep -c "boosted.*SCHED_FIFO" "${STALLD_LOG}")
@@ -234,10 +232,12 @@ if [ -n "${deadline_tracked_pid}" ]; then
 fi
 
 # NOW start stalld
-start_stalld -f -v -g 1 -N -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} > "${STALLD_LOG_DEADLINE}" 2>&1
+start_stalld_with_log "${STALLD_LOG_DEADLINE}" -f -v -g 1 -N -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration}
 
-# Wait for detection, boost, and some progress (threshold + granularity + boost_duration)
-sleep $((threshold + 1 + boost_duration))
+# Wait for boost detection, then let boost run to completion
+log "Waiting for DEADLINE boost detection..."
+wait_for_boost_detected "${STALLD_LOG_DEADLINE}"
+sleep $((boost_duration + 1))
 
 ctxt_after_deadline=0
 if [ -n "${deadline_tracked_pid}" ] && [ -f "/proc/${deadline_tracked_pid}/status" ]; then
@@ -277,10 +277,12 @@ if [ -n "${fifo_tracked_pid}" ]; then
 fi
 
 # NOW start stalld
-start_stalld -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration} > "${STALLD_LOG_FIFO}" 2>&1
+start_stalld_with_log "${STALLD_LOG_FIFO}" -f -v -g 1 -N -F -A -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration}
 
-# Wait for detection, boost, and some progress (threshold + granularity + boost_duration)
-sleep $((threshold + 1 + boost_duration))
+# Wait for boost detection, then let boost run to completion
+log "Waiting for FIFO boost detection..."
+wait_for_boost_detected "${STALLD_LOG_FIFO}"
+sleep $((boost_duration + 1))
 
 ctxt_after_fifo=0
 if [ -n "${fifo_tracked_pid}" ] && [ -f "/proc/${fifo_tracked_pid}/status" ]; then
diff --git a/tests/functional/test_fifo_priority_starvation.sh b/tests/functional/test_fifo_priority_starvation.sh
index cf47971..adb7b44 100755
--- a/tests/functional/test_fifo_priority_starvation.sh
+++ b/tests/functional/test_fifo_priority_starvation.sh
@@ -69,13 +69,9 @@ start_starvation_gen -c ${TEST_CPU} -p 10 -b 5 -n 2 -d ${starvation_duration}
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
-# Wait for detection (threshold + small buffer)
-wait_time=$((threshold + 2))
-log "Waiting ${wait_time}s for starvation detection..."
-sleep ${wait_time}
-
-# Verify starvation was detected
-if grep -q "starved on CPU" "${STALLD_LOG}"; then
+# Wait for starvation detection
+log "Waiting for starvation detection..."
+if wait_for_starvation_detected "${STALLD_LOG}"; then
     log "✓ PASS: FIFO-on-FIFO starvation detected"
 
     # Verify correct CPU is logged
@@ -195,12 +191,14 @@ log "Will monitor for multiple detection cycles to verify timestamp preservation
 start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
 # Wait for multiple detection cycles
-log "Waiting for multiple detection cycles..."
-sleep $((threshold + 2))
+log "Waiting for first detection cycle..."
+wait_for_starvation_detected "${STALLD_LOG}"
 log "First detection cycle should have occurred"
-sleep 3
+log "Waiting for second detection cycle..."
+wait_for_starvation_detected "${STALLD_LOG}"
 log "Second detection cycle should have occurred"
-sleep 3
+log "Waiting for third detection cycle..."
+wait_for_starvation_detected "${STALLD_LOG}"
 log "Third detection cycle should have occurred"
 
 # Check if we see accumulating starvation time in logs
@@ -257,11 +255,9 @@ start_starvation_gen -c ${TEST_CPU} -p 6 -b 5 -n 1 -d $((threshold + 5))
 log "Starting stalld with ${threshold}s threshold"
 start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
-# Wait for detection
-sleep $((threshold + 2))
-
-# Verify detection works even with close priorities
-if grep -q "starved on CPU" "${STALLD_LOG}"; then
+# Wait for starvation detection
+log "Waiting for starvation detection..."
+if wait_for_starvation_detected "${STALLD_LOG}"; then
     log "✓ PASS: Starvation detected even with close priority gap (6 vs 5)"
 else
     log "⚠ WARNING: Starvation not detected with close priority gap"
@@ -298,11 +294,9 @@ log "Starvation generator PID: ${STARVE_PID}"
 log "Starting stalld with boosting enabled"
 start_stalld_with_log "${STALLD_LOG}" -f -v -N -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
-# Wait for detection and boosting
-sleep $((threshold + 2))
-
-# Verify boosting occurred
-if grep -q "boosted" "${STALLD_LOG}"; then
+# Wait for boosting
+log "Waiting for boost detection..."
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "✓ PASS: Boosting occurred"
 
     # Try to verify the correct task was boosted
diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index 4c43101..17bc66e 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -51,7 +51,7 @@ log "=========================================="
 
 threshold=3
 log "Starting stalld with ${threshold}s threshold (default, no -F)"
-start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -c "${TEST_CPU}" -t ${threshold}
 
 # Create starvation
 starvation_duration=10
@@ -59,12 +59,7 @@ log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
-wait_time=$((threshold + 2))
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-# Check if boosting occurred and look for DEADLINE mentions
-if grep -q "boost" "${STALLD_LOG}"; then
+if wait_for_boost_detected "${STALLD_LOG}"; then
     log "Boosting occurred"
 
     # Look for SCHED_DEADLINE indicators
@@ -98,18 +93,14 @@ STALLD_LOG2="/tmp/stalld_test_force_fifo_test2_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG2}")
 
 log "Starting stalld with -F flag and aggressive mode (-A)"
-start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A > "${STALLD_LOG2}" 2>&1
+start_stalld_with_log "${STALLD_LOG2}" -f -v -c "${TEST_CPU}" -t ${threshold} -F -A
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-# Check if boosting occurred and look for FIFO mentions
-if grep -q "boost" "${STALLD_LOG2}"; then
+if wait_for_boost_detected "${STALLD_LOG2}"; then
     log "Boosting occurred with -F flag"
 
     # Look for SCHED_FIFO indicators
@@ -142,15 +133,14 @@ STALLD_LOG3="/tmp/stalld_test_force_fifo_test3_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG3}")
 
 log "Starting stalld with -F and -A flags"
-start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A > "${STALLD_LOG3}" 2>&1
+start_stalld_with_log "${STALLD_LOG3}" -f -v -c "${TEST_CPU}" -t ${threshold} -F -A
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${starvation_duration}s"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
+wait_for_boost_detected "${STALLD_LOG3}"
 
 # Check logs for priority information
 if grep -qi "priority\|prio" "${STALLD_LOG3}"; then
@@ -182,17 +172,14 @@ STALLD_LOG4="/tmp/stalld_test_force_fifo_test4_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG4}")
 
 log "Starting stalld with -F, -A, and ${boost_duration}s boost duration"
-start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A -d ${boost_duration} > "${STALLD_LOG4}" 2>&1
+start_stalld_with_log "${STALLD_LOG4}" -f -v -c "${TEST_CPU}" -t ${threshold} -F -A -d ${boost_duration}
 
 # Create starvation
 log "Creating starvation on CPU ${TEST_CPU} for ${long_starvation}s"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${long_starvation}
 
 # Wait for detection and boosting
-log "Waiting ${wait_time}s for detection and boosting"
-sleep ${wait_time}
-
-if grep -q "boost" "${STALLD_LOG4}"; then
+if wait_for_boost_detected "${STALLD_LOG4}"; then
     log "Boosting detected, waiting for duration cycle"
 
     # Wait for boost duration + buffer to see restoration
@@ -261,9 +248,9 @@ STALLD_LOG_DL="/tmp/stalld_test_force_fifo_deadline_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG_DL}")
 
 log "Running comparison test with SCHED_DEADLINE"
-start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -d ${comparison_duration} > "${STALLD_LOG_DL}" 2>&1
+start_stalld_with_log "${STALLD_LOG_DL}" -f -v -c "${TEST_CPU}" -t ${threshold} -d ${comparison_duration}
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${comparison_starvation}
-sleep $((threshold + 3))
+wait_for_boost_detected "${STALLD_LOG_DL}"
 
 deadline_boosts=$(grep -c "boost" "${STALLD_LOG_DL}" || echo 0)
 log "ℹ INFO: SCHED_DEADLINE boosts: $deadline_boosts"
@@ -277,9 +264,9 @@ STALLD_LOG_FIFO="/tmp/stalld_test_force_fifo_comparison_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG_FIFO}")
 
 log "Running comparison test with SCHED_FIFO"
-start_stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F -A -d ${comparison_duration} > "${STALLD_LOG_FIFO}" 2>&1
+start_stalld_with_log "${STALLD_LOG_FIFO}" -f -v -c "${TEST_CPU}" -t ${threshold} -F -A -d ${comparison_duration}
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${comparison_starvation}
-sleep $((threshold + 3))
+wait_for_boost_detected "${STALLD_LOG_FIFO}"
 
 fifo_boosts=$(grep -c "boost" "${STALLD_LOG_FIFO}" || echo 0)
 log "ℹ INFO: SCHED_FIFO boosts: $fifo_boosts"
diff --git a/tests/functional/test_idle_detection.sh b/tests/functional/test_idle_detection.sh
index 755a4b5..ef1c572 100755
--- a/tests/functional/test_idle_detection.sh
+++ b/tests/functional/test_idle_detection.sh
@@ -84,7 +84,7 @@ log "Idle CPUs should be skipped to reduce overhead"
 threshold=5
 log "Starting stalld with verbose logging"
 # Use -g 1 for 1-second granularity
-start_stalld -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
 # Let stalld run while CPU is idle (no load)
 log "CPU ${TEST_CPU} should be idle (no load created)"
@@ -152,7 +152,7 @@ log "=========================================="
 threshold=5
 rm -f "${STALLD_LOG}"
 log "Starting stalld"
-start_stalld -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
 # Initially idle
 log "CPU ${TEST_CPU} initially idle"
@@ -162,13 +162,11 @@ sleep 3
 log "Creating load on CPU ${TEST_CPU} to make it busy"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d 12
 
-# Wait for stalld to detect the busy CPU and starvation (threshold + granularity + buffer)
+# Wait for stalld to detect the busy CPU and starvation
 log "Waiting for stalld to detect busy CPU and starvation..."
-wait_time=$((threshold + 1 + 3))
-sleep ${wait_time}
 
 # Verify stalld detected starvation (meaning it resumed monitoring)
-if grep -q "starved" "${STALLD_LOG}"; then
+if wait_for_starvation_detected "${STALLD_LOG}"; then
     log "✓ PASS: stalld detected starvation on now-busy CPU"
     log "        Monitoring resumed when CPU became busy"
 else
@@ -226,15 +224,14 @@ else
 
     threshold=5
     rm -f "${STALLD_LOG}"
-    start_stalld -f -v -g 1 -l -t $threshold -c ${CPU0},${CPU1} > "${STALLD_LOG}" 2>&1
+    start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -l -t $threshold -c ${CPU0},${CPU1}
 
     # Create load only on CPU1, leave CPU0 idle
     log "Creating load on CPU ${CPU1} only"
     start_starvation_gen -c ${CPU1} -p 80 -n 2 -d 12
 
-    # Wait for detection (threshold + granularity + buffer)
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
+    # Wait for detection
+    wait_for_starvation_detected "${STALLD_LOG}"
 
     # Check which CPU had starvation detected
     cpu0_detections=$(grep -c "starved on CPU ${CPU0}" "${STALLD_LOG}")
diff --git a/tests/functional/test_log_only.sh b/tests/functional/test_log_only.sh
index 93c71cb..655560c 100755
--- a/tests/functional/test_log_only.sh
+++ b/tests/functional/test_log_only.sh
@@ -76,11 +76,10 @@ fi
 echo "stalld started with PID ${STALLD_PID}"
 
 echo "Starvation generator started (PID ${STARVGEN_PID})"
-echo "Waiting 7 seconds for starvation detection..."
-sleep 7
+echo "Waiting for starvation detection..."
 
 # Check if stalld detected the starvation (should log it)
-if grep -q "starved" "${LOG_FILE}"; then
+if wait_for_starvation_detected "${LOG_FILE}"; then
 	assert_equals "1" "1" "stalld detected and logged starvation"
 else
 	TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_runqueue_parsing.sh b/tests/functional/test_runqueue_parsing.sh
index 7edf936..ccb1982 100755
--- a/tests/functional/test_runqueue_parsing.sh
+++ b/tests/functional/test_runqueue_parsing.sh
@@ -100,12 +100,8 @@ if [ ${BPF_AVAILABLE} -eq 1 ]; then
     log "Creating starvation for ${starvation_duration}s"
     start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
-    # Wait for detection (threshold + granularity + buffer)
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
-
-    # Verify eBPF backend detected starvation
-    if grep -q "starved on CPU" "${STALLD_LOG_BPF}"; then
+    # Wait for starvation detection
+    if wait_for_starvation_detected "${STALLD_LOG_BPF}"; then
         log "✓ PASS: eBPF backend detected starving tasks"
 
         # Verify task info is present (PID, comm)
@@ -159,12 +155,8 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     log "Creating starvation for ${starvation_duration}s"
     start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
-    # Wait for detection (threshold + granularity + buffer)
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
-
-    # Verify sched_debug backend detected starvation
-    if grep -q "starved on CPU" "${STALLD_LOG_SCHED}"; then
+    # Wait for starvation detection
+    if wait_for_starvation_detected "${STALLD_LOG_SCHED}"; then
         log "✓ PASS: sched_debug backend detected starving tasks"
 
         # Verify task info is present
@@ -227,8 +219,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ] && [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
 
     start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
+    wait_for_starvation_detected "${STALLD_LOG_BPF}"
     bpf_detections=$(count_detected_tasks "${STALLD_LOG_BPF}")
     log "eBPF backend detected: ${bpf_detections} starvation events"
 
@@ -247,8 +238,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ] && [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
 
     start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
+    wait_for_starvation_detected "${STALLD_LOG_SCHED}"
     sched_detections=$(count_detected_tasks "${STALLD_LOG_SCHED}")
     log "sched_debug backend detected: ${sched_detections} starvation events"
 
@@ -314,9 +304,8 @@ if [ -n "$test_backend" ]; then
     log "Creating starvation with known task name (starvation_gen)"
     start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 10 -v
 
-    # Wait for detection (threshold + granularity + buffer)
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
+    # Wait for starvation detection
+    wait_for_starvation_detected "${log_file}"
 
     # Verify fields are present
     log ""
@@ -376,8 +365,8 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     # Create brief starvation just to initialize the backend
     start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 8
 
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
+    # Wait for starvation detection
+    wait_for_starvation_detected "${STALLD_LOG_SCHED}"
 
     # Check for format detection messages
     if grep -q "detect_task_format" "${STALLD_LOG_SCHED}"; then
diff --git a/tests/functional/test_starvation_detection.sh b/tests/functional/test_starvation_detection.sh
index 6a1821d..6b7d330 100755
--- a/tests/functional/test_starvation_detection.sh
+++ b/tests/functional/test_starvation_detection.sh
@@ -72,16 +72,9 @@ start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
-# Wait for detection (threshold + granularity + buffer)
-# With -g 1, stalld checks every 1 second. In worst case, it checks just before
-# threshold is reached, then waits another granularity period.
-# So we need: threshold + granularity + buffer for processing
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for starvation detection (threshold: ${threshold}s, granularity: 1s)..."
-sleep ${wait_time}
-
-# Verify starvation was detected
-if grep -qE "starvation_gen.*starved on CPU|starved on CPU.*starvation_gen" "${STALLD_LOG}"; then
+# Wait for starvation detection
+log "Waiting for starvation detection..."
+if wait_for_starvation_detected "${STALLD_LOG}"; then
     log "✓ PASS: Starvation detected"
 
     # Verify correct CPU is logged
@@ -129,10 +122,9 @@ start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 15
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
-# Wait for detection (threshold + granularity + buffer)
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for detection..."
-sleep ${wait_time}
+# Wait for starvation detection
+log "Waiting for starvation detection..."
+wait_for_starvation_detected "${STALLD_LOG}"
 
 # Try to find the starved task PID from starvation_gen children
 # The blockee thread is what gets starved
@@ -190,16 +182,14 @@ log "Starting stalld with ${threshold}s threshold (log-only mode)"
 log "Will monitor for multiple detection cycles to verify timestamp preservation"
 start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU}
 
-# Wait for multiple detection cycles
-# First detection: threshold + granularity + buffer
+# Wait for first detection cycle
 log "Waiting for first detection cycle..."
-sleep $((threshold + 1 + 3))
+wait_for_starvation_detected "${STALLD_LOG}"
 log "First detection cycle should have occurred"
-# Second detection: wait for another granularity period + buffer
-sleep $((1 + 2))
+# Wait for additional detection cycles
+sleep 4
 log "Second detection cycle should have occurred"
-# Third detection: wait for another granularity period + buffer
-sleep $((1 + 2))
+sleep 4
 log "Third detection cycle should have occurred"
 
 # Stop stalld to flush output buffers before checking log
@@ -287,10 +277,9 @@ else
 
     start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${CPU0},${CPU1} -a ${STALLD_CPU_MULTI}
 
-    # Wait for detection (threshold + granularity + buffer)
-    wait_time=$((threshold + 1 + 3))
-    log "Waiting ${wait_time}s for detection..."
-    sleep ${wait_time}
+    # Wait for starvation detection
+    log "Waiting for starvation detection..."
+    wait_for_starvation_detected "${STALLD_LOG}"
 
     # Check both CPUs detected - specifically look for starvation_gen tasks
     if grep -qE "starvation_gen.*starved on CPU ${CPU0}|starved on CPU ${CPU0}.*starvation_gen" "${STALLD_LOG}"; then
diff --git a/tests/functional/test_starvation_threshold.sh b/tests/functional/test_starvation_threshold.sh
index 45a4679..9b14009 100755
--- a/tests/functional/test_starvation_threshold.sh
+++ b/tests/functional/test_starvation_threshold.sh
@@ -69,18 +69,13 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold"
 # Use -i to ignore kernel workers that may starve before our test tasks
-start_stalld -f -v -N -M -g 1 -i "ksoftirqd,kworker" -c "${TEST_CPU}" -a "${STALLD_CPU}" -t ${threshold} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -N -M -g 1 -i "ksoftirqd,kworker" -c "${TEST_CPU}" -a "${STALLD_CPU}" -t ${threshold}
 
-# Wait for threshold + granularity + buffer time
-# With -g 1, stalld checks every 1 second. In worst case, it checks just before
-# threshold is reached, then waits another granularity period.
-# So we need: threshold + granularity + buffer for processing
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for detection (threshold: ${threshold}s, granularity: 1s)"
-sleep ${wait_time}
+# Wait for starvation detection
+log "Waiting for detection (threshold: ${threshold}s)"
 
 # Check if starvation was detected - specifically look for starvation_gen tasks
-if grep -qE "starvation_gen.*starved on CPU ${TEST_CPU}|starved on CPU ${TEST_CPU}.*starvation_gen" "${STALLD_LOG}"; then
+if wait_for_starvation_detected "${STALLD_LOG}"; then
     log "✓ PASS: Starvation detected after ${threshold}s threshold"
 else
     log "✗ FAIL: Starvation not detected after ${threshold}s threshold"
@@ -113,7 +108,7 @@ log "Creating short starvation (${starvation_duration}s) with threshold of ${thr
 start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold"
-start_stalld -f -v -N -M -g 1 -c "${TEST_CPU}" -a "${STALLD_CPU}" -t ${threshold} > "${STALLD_LOG2}" 2>&1
+start_stalld_with_log "${STALLD_LOG2}" -f -v -N -M -g 1 -c "${TEST_CPU}" -a "${STALLD_CPU}" -t ${threshold}
 
 # Wait for starvation duration + small buffer
 sleep 8
@@ -159,15 +154,13 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold"
 # Use -i to ignore kernel workers that may starve before our test tasks
-start_stalld -f -v -N -M -g 1 -i "ksoftirqd,kworker" -c "${TEST_CPU}" -a "${STALLD_CPU}" -t ${threshold} > "${STALLD_LOG3}" 2>&1
+start_stalld_with_log "${STALLD_LOG3}" -f -v -N -M -g 1 -i "ksoftirqd,kworker" -c "${TEST_CPU}" -a "${STALLD_CPU}" -t ${threshold}
 
-# Wait for threshold + granularity + buffer
-wait_time=$((threshold + 1 + 3))
-log "Waiting ${wait_time}s for detection (threshold: ${threshold}s, granularity: 1s)"
-sleep ${wait_time}
+# Wait for starvation detection
+log "Waiting for detection (threshold: ${threshold}s)"
 
 # Check if starvation_gen was detected
-if grep -qE "starvation_gen.*starved on CPU ${TEST_CPU}|starved on CPU ${TEST_CPU}.*starvation_gen" "${STALLD_LOG3}"; then
+if wait_for_starvation_detected "${STALLD_LOG3}"; then
     log "✓ PASS: Starvation detected with ${threshold}s threshold"
 else
     log "✗ FAIL: Starvation not detected with ${threshold}s threshold"
diff --git a/tests/functional/test_task_merging.sh b/tests/functional/test_task_merging.sh
index 1bea759..5cd6406 100755
--- a/tests/functional/test_task_merging.sh
+++ b/tests/functional/test_task_merging.sh
@@ -65,17 +65,16 @@ log "Task merging: same PID + same ctxsw = preserved timestamp"
 threshold=3
 log "Starting stalld with ${threshold}s threshold (log-only, verbose)"
 # Use -g 1 for 1-second granularity
-start_stalld -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
 # Create long starvation to span multiple monitoring cycles
 starvation_duration=18
 log "Creating starvation for ${starvation_duration}s (multiple detection cycles)"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
-# Wait for first detection (threshold + granularity + buffer)
+# Wait for first detection
 log "Waiting for first detection cycle..."
-wait_time=$((threshold + 1 + 3))
-sleep ${wait_time}
+wait_for_starvation_detected "${STALLD_LOG}"
 
 # Extract first starvation duration
 if grep -q "starved.*for [0-9]" "${STALLD_LOG}"; then
@@ -144,15 +143,14 @@ log "Merging occurs when: PID matches AND context switches unchanged"
 
 threshold=5
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
 # Create starvation
 log "Creating starvation"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
-# Wait for detection (threshold + granularity + buffer)
-wait_time=$((threshold + 1 + 3))
-sleep ${wait_time}
+# Wait for starvation detection
+wait_for_starvation_detected "${STALLD_LOG}"
 
 # Find the starved task PID
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
@@ -218,15 +216,14 @@ log "When context switches change, timestamp should reset"
 
 threshold=5
 rm -f "${STALLD_LOG}"
-start_stalld -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d 2 > "${STALLD_LOG}" 2>&1
+start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d 2
 
 # Create starvation that will get boosted (allowing progress)
 log "Creating starvation that will be boosted"
 start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d 20
 
-# Wait for first detection and boost (threshold + granularity + buffer)
-wait_time=$((threshold + 1 + 3))
-sleep ${wait_time}
+# Wait for starvation detection
+wait_for_starvation_detected "${STALLD_LOG}"
 
 # Find tracked task
 STARVE_CHILDREN=$(pgrep -P ${STARVE_PID} 2>/dev/null)
@@ -286,7 +283,7 @@ else
     log "Testing task merging on CPU ${CPU0} and CPU ${CPU1} independently"
 
     rm -f "${STALLD_LOG}"
-    start_stalld -f -v -g 1 -l -t $threshold -c ${CPU0},${CPU1} -a ${STALLD_CPU} > "${STALLD_LOG}" 2>&1
+    start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -l -t $threshold -c ${CPU0},${CPU1} -a ${STALLD_CPU}
 
     # Create starvation on both CPUs
     log "Creating starvation on CPU ${CPU0}"
@@ -297,9 +294,8 @@ else
     start_starvation_gen -c ${CPU1} -p 80 -n 1 -d 15
     STARVE_PID1=${STARVE_PID}
 
-    # Wait for multiple detection cycles (threshold + granularity + buffer, then more)
-    wait_time=$((threshold + 1 + 3))
-    sleep ${wait_time}
+    # Wait for starvation detection on both CPUs
+    wait_for_starvation_detected "${STALLD_LOG}"
     sleep 4
 
     # Check CPU0 starvation accumulation
-- 
2.53.0

[PATCH stalld 20/36] tests/functional: Remove duplicated -a flag in test_fifo_priority_starvation

[Wander Lairson Costa]

The start_stalld_with_log call in Test 2 passes -a ${STALLD_CPU}
twice, which is a copy-paste error. Remove the duplicate.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_fifo_priority_starvation.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/functional/test_fifo_priority_starvation.sh b/tests/functional/test_fifo_priority_starvation.sh
index adb7b44..886357e 100755
--- a/tests/functional/test_fifo_priority_starvation.sh
+++ b/tests/functional/test_fifo_priority_starvation.sh
@@ -136,7 +136,7 @@ if [ -n "${blockee_pid}" ]; then
 fi
 
 log "Starting stalld with boosting enabled"
-start_stalld_with_log "${STALLD_LOG}" -f -v -N -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -a ${STALLD_CPU} -d ${boost_duration}
+start_stalld_with_log "${STALLD_LOG}" -f -v -N -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU} -d ${boost_duration}
 
 # Wait for detection and boosting
 sleep $((threshold + boost_duration + 1))
-- 
2.53.0

[PATCH stalld 21/36] tests/functional: Add missing -a flag in test_starvation_detection

[Wander Lairson Costa]

The start_stalld_with_log call in Test 3 is missing the -a flag to
pin stalld to a separate CPU, unlike all other invocations in the
same file. Add it for consistency.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_starvation_detection.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/functional/test_starvation_detection.sh b/tests/functional/test_starvation_detection.sh
index 6b7d330..3c6fd6e 100755
--- a/tests/functional/test_starvation_detection.sh
+++ b/tests/functional/test_starvation_detection.sh
@@ -180,7 +180,7 @@ start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 
 log "Starting stalld with ${threshold}s threshold (log-only mode)"
 log "Will monitor for multiple detection cycles to verify timestamp preservation"
-start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU}
+start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU} -a ${STALLD_CPU}
 
 # Wait for first detection cycle
 log "Waiting for first detection cycle..."
-- 
2.53.0

[PATCH stalld 22/36] tests/functional: Use start_stalld_with_log() in test_log_only

[Wander Lairson Costa]

Replace manual stalld invocation with start_stalld_with_log(),
which handles backend selection, PID tracking, and event-driven
readiness detection internally.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_log_only.sh | 23 +----------------------
 1 file changed, 1 insertion(+), 22 deletions(-)

diff --git a/tests/functional/test_log_only.sh b/tests/functional/test_log_only.sh
index 655560c..ba72b8e 100755
--- a/tests/functional/test_log_only.sh
+++ b/tests/functional/test_log_only.sh
@@ -52,28 +52,7 @@ STARVGEN_PID=${STARVE_PID}
 
 # Start stalld in log-only mode with verbose output to capture logs
 echo "Starting stalld in log-only mode with 5 second threshold"
-
-# Build stalld command with backend option if specified
-STALLD_ARGS="-f -v -l -t 5 -c ${TEST_CPU} -a ${STALLD_CPU}"
-if [ -n "${STALLD_TEST_BACKEND}" ]; then
-	STALLD_ARGS="-b ${STALLD_TEST_BACKEND} ${STALLD_ARGS}"
-	echo "Using backend: ${STALLD_TEST_BACKEND}"
-fi
-
-${TEST_ROOT}/../stalld ${STALLD_ARGS} > "${LOG_FILE}" 2>&1 &
-STALLD_PID=$!
-CLEANUP_PIDS+=("${STALLD_PID}")
-sleep 2
-
-# Verify stalld is running
-if ! assert_process_running "${STALLD_PID}" "stalld should be running"; then
-	echo "Failed to start stalld, aborting test"
-	echo "Log contents:"
-	cat "${LOG_FILE}"
-	end_test
-	exit 1
-fi
-echo "stalld started with PID ${STALLD_PID}"
+start_stalld_with_log "${LOG_FILE}" -f -v -l -t 5 -c ${TEST_CPU} -a ${STALLD_CPU}
 
 echo "Starvation generator started (PID ${STARVGEN_PID})"
 echo "Waiting for starvation detection..."
-- 
2.53.0

[PATCH stalld 23/36] tests/functional: Use start_stalld_with_log() in test_logging_destinations

[Wander Lairson Costa]

Replace direct stalld invocations in Tests 1 and 4 with
start_stalld_with_log(), which handles backend selection, PID
tracking, and readiness detection internally. This also removes
the now-unnecessary BACKEND_FLAG variable.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_logging_destinations.sh | 17 ++---------------
 1 file changed, 2 insertions(+), 15 deletions(-)

diff --git a/tests/functional/test_logging_destinations.sh b/tests/functional/test_logging_destinations.sh
index 171be8b..b44c08d 100755
--- a/tests/functional/test_logging_destinations.sh
+++ b/tests/functional/test_logging_destinations.sh
@@ -33,16 +33,7 @@ echo "Test 1: Verbose mode (-v) logs to stdout"
 LOG_FILE="/tmp/stalld_test_verbose_$$.log"
 CLEANUP_FILES+=("${LOG_FILE}")
 
-# Start stalld directly (not using start_stalld helper) to capture output
-# Add backend flag if specified via test runner
-BACKEND_FLAG=""
-if [ -n "${STALLD_TEST_BACKEND}" ]; then
-	BACKEND_FLAG="-b ${STALLD_TEST_BACKEND}"
-fi
-"${TEST_ROOT}/../stalld" -f -v ${BACKEND_FLAG} -l -t 5 > "${LOG_FILE}" 2>&1 &
-STALLD_PID=$!
-CLEANUP_PIDS+=("${STALLD_PID}")
-sleep 2
+start_stalld_with_log "${LOG_FILE}" -f -v -l -t 5
 
 if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 	# Check that output was written to our log file
@@ -160,11 +151,7 @@ echo "Test 4: Combined logging modes"
 LOG_FILE="/tmp/stalld_test_combined_$$.log"
 CLEANUP_FILES+=("${LOG_FILE}")
 
-# Start stalld directly (not using start_stalld helper) to capture output
-"${TEST_ROOT}/../stalld" -f -v -k -s ${BACKEND_FLAG} -l -t 5 > "${LOG_FILE}" 2>&1 &
-STALLD_PID=$!
-CLEANUP_PIDS+=("${STALLD_PID}")
-sleep 2
+start_stalld_with_log "${LOG_FILE}" -f -v -k -s -l -t 5
 
 if assert_process_running "${STALLD_PID}" "stalld with combined logging should be running"; then
 	# Verify verbose output
-- 
2.53.0

[PATCH stalld 24/36] tests/functional: Use start_stalld_with_log() in test_cpu_selection

[Wander Lairson Costa]

Replace the local start_stalld_with_output() helper, which
duplicated start_stalld_with_log() functionality, with direct
calls to start_stalld_with_log(). For the invalid CPU test, run
stalld synchronously under timeout instead of backgrounding and
sleeping, and treat unexpected success or timeout as test failures
rather than warnings.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_cpu_selection.sh | 55 ++++++--------------------
 1 file changed, 11 insertions(+), 44 deletions(-)

diff --git a/tests/functional/test_cpu_selection.sh b/tests/functional/test_cpu_selection.sh
index 9d8ac48..7960f6c 100755
--- a/tests/functional/test_cpu_selection.sh
+++ b/tests/functional/test_cpu_selection.sh
@@ -25,24 +25,6 @@ check_rt_throttling
 STALLD_LOG="/tmp/stalld_cpu_selection_$$.log"
 CLEANUP_FILES+=("${STALLD_LOG}")
 
-# Helper to start stalld with output capture
-start_stalld_with_output() {
-    local args="$@"
-
-    # Add backend option if specified
-    if [ -n "${STALLD_TEST_BACKEND}" ]; then
-        args="-b ${STALLD_TEST_BACKEND} ${args}"
-        echo "Using backend: ${STALLD_TEST_BACKEND}"
-    fi
-
-    # Start stalld with output redirected
-    ${TEST_ROOT}/../stalld ${args} > "${STALLD_LOG}" 2>&1 &
-    STALLD_PID=$!
-    CLEANUP_PIDS+=("${STALLD_PID}")
-    sleep 1
-    echo "stalld started with PID ${STALLD_PID}"
-}
-
 # Get available CPUs
 num_cpus=$(nproc)
 if [ "$num_cpus" -lt 2 ]; then
@@ -56,8 +38,7 @@ echo "System has $num_cpus CPUs"
 echo ""
 echo "Test 1: Single CPU monitoring (-c 0)"
 rm -f "${STALLD_LOG}"
-start_stalld_with_output -f -v -c 0 -l -t 5
-sleep 2
+start_stalld_with_log "${STALLD_LOG}" -f -v -c 0 -l -t 5
 
 # Check that stalld mentions CPU 0
 if grep -q "cpu 0" "$STALLD_LOG"; then
@@ -74,8 +55,7 @@ if [ "$num_cpus" -ge 4 ]; then
     echo ""
     echo "Test 2: CPU list monitoring (-c 0,2)"
     rm -f "${STALLD_LOG}"
-    start_stalld_with_output -f -v -c 0,2 -l -t 5
-    sleep 2
+    start_stalld_with_log "${STALLD_LOG}" -f -v -c 0,2 -l -t 5
 
     # Check for CPU 0 and CPU 2 in output
     cpu0_found=0
@@ -104,8 +84,7 @@ if [ "$num_cpus" -ge 4 ]; then
     echo ""
     echo "Test 3: CPU range monitoring (-c 0-2)"
     rm -f "${STALLD_LOG}"
-    start_stalld_with_output -f -v -c 0-2 -l -t 5
-    sleep 2
+    start_stalld_with_log "${STALLD_LOG}" -f -v -c 0-2 -l -t 5
 
     # Check for CPUs 0, 1, 2 in output
     cpu0_found=0
@@ -138,8 +117,7 @@ if [ "$num_cpus" -ge 6 ]; then
     echo ""
     echo "Test 4: Combined format (-c 0,2-4)"
     rm -f "${STALLD_LOG}"
-    start_stalld_with_output -f -v -c 0,2-4 -l -t 5
-    sleep 2
+    start_stalld_with_log "${STALLD_LOG}" -f -v -c 0,2-4 -l -t 5
 
     # Should monitor CPUs 0, 2, 3, 4
     monitored_cpus=0
@@ -171,24 +149,14 @@ INVALID_LOG="/tmp/stalld_invalid_cpu_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG}")
 
 # Run stalld with invalid CPU and capture output
-"${TEST_ROOT}/../stalld" -f -v -c $invalid_cpu -l -t 5 > "${INVALID_LOG}" 2>&1 &
-INVALID_PID=$!
-
-# Wait a bit to see if it exits or produces error
-sleep 2
+timeout 5 "${TEST_ROOT}/../stalld" -f -v -c $invalid_cpu -l -t 5 > "${INVALID_LOG}" 2>&1
+ret=$?
 
-if ! kill -0 "$INVALID_PID" 2>/dev/null; then
-    # Process exited - check for error message
-    if grep -qi "error\|invalid\|failed" "$INVALID_LOG"; then
-        assert_equals "1" "1" "stalld rejected invalid CPU number with error"
-    else
-        assert_equals "1" "1" "stalld exited when given invalid CPU"
-    fi
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "stalld rejected invalid CPU number"
 else
-    # Process still running - it might have ignored the invalid CPU
-    echo -e "  ${YELLOW}WARNING${NC}: stalld still running with invalid CPU (may have ignored it)"
-    kill -TERM "$INVALID_PID" 2>/dev/null
-    wait "$INVALID_PID" 2>/dev/null
+    log "✗ FAIL: stalld did not reject invalid CPU"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Test 6: Verify non-selected CPUs are NOT monitored
@@ -196,8 +164,7 @@ if [ "$num_cpus" -ge 2 ]; then
     echo ""
     echo "Test 6: Verify non-selected CPUs not monitored (-c 0)"
     rm -f "${STALLD_LOG}"
-    start_stalld_with_output -f -v -c 0 -l -t 5
-    sleep 2
+    start_stalld_with_log "${STALLD_LOG}" -f -v -c 0 -l -t 5
 
     # Check that CPU 1 is NOT mentioned (or mentioned as "not monitoring")
     if ! grep -q "cpu 1" "$STALLD_LOG" || grep -q "not monitoring.*cpu 1" "$STALLD_LOG"; then
-- 
2.53.0

[PATCH stalld 25/36] tests/functional: Use wait_for_stalld_ready() in test_backend_selection

[Wander Lairson Costa]

Replace the post-startup sleep with wait_for_stalld_ready() for
event-driven readiness detection. Add -g 1 for consistency with
start_stalld_with_log().

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_backend_selection.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/functional/test_backend_selection.sh b/tests/functional/test_backend_selection.sh
index 5d60072..c274025 100755
--- a/tests/functional/test_backend_selection.sh
+++ b/tests/functional/test_backend_selection.sh
@@ -33,15 +33,15 @@ test_backend_flag() {
 
 	CLEANUP_FILES+=("${log_file}")
 
-	"${TEST_ROOT}/../stalld" -v -f -l -b "${backend_flag}" -t 60 \
+	"${TEST_ROOT}/../stalld" -v -f -l -g 1 -b "${backend_flag}" -t 60 \
 		> "${log_file}" 2>&1 &
 	STALLD_PID=$!
 	CLEANUP_PIDS+=("${STALLD_PID}")
-	sleep 1
 
-	if ! kill -0 ${STALLD_PID} 2>/dev/null; then
+	if ! wait_for_stalld_ready "${log_file}" 15; then
 		TEST_FAILED=$((TEST_FAILED + 1))
 		echo -e "  ${RED}FAIL${NC}: stalld failed to start (${description})"
+		stop_stalld
 		return 1
 	fi
 
-- 
2.53.0

[PATCH stalld 26/36] tests/functional: Use timeout for error path in test_force_fifo

[Wander Lairson Costa]

The single-threaded FIFO rejection test backgrounds stalld and
sleeps 3 seconds to check if it exited. Run it synchronously
under timeout instead, and treat unexpected success or timeout as
test failures rather than warnings.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_force_fifo.sh | 22 +++++++---------------
 1 file changed, 7 insertions(+), 15 deletions(-)

diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index 17bc66e..2c21ddd 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -214,22 +214,14 @@ FIFO_SINGLE_LOG="/tmp/stalld_test_force_fifo_single_$$.log"
 CLEANUP_FILES+=("${FIFO_SINGLE_LOG}")
 
 log "Testing single-threaded mode with -F (should exit)"
-${TEST_ROOT}/../stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F > "${FIFO_SINGLE_LOG}" 2>&1 &
-fifo_pid=$!
-sleep 3
-
-if ! kill -0 "${fifo_pid}" 2>/dev/null; then
-    # Process exited - this is expected
-    if grep -qi "error\|single.*thread\|not.*support" "${FIFO_SINGLE_LOG}"; then
-        log "✓ PASS: Single-threaded mode rejected FIFO with error message"
-    else
-        log "✓ PASS: Single-threaded mode with FIFO caused exit (as expected)"
-    fi
+timeout 5 ${TEST_ROOT}/../stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F > "${FIFO_SINGLE_LOG}" 2>&1
+ret=$?
+
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "single-threaded mode rejected FIFO"
 else
-    # Process still running - unexpected
-    log "⚠ WARNING: Single-threaded mode accepted FIFO (may have switched to multi-threaded)"
-    kill -TERM "${fifo_pid}" 2>/dev/null
-    wait "${fifo_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject -F in single-threaded mode"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 #=============================================================================
-- 
2.53.0

[PATCH stalld 27/36] tests/functional: Use timeout for invalid argument tests

[Wander Lairson Costa]

Invalid argument tests background stalld and sleep 2-3 seconds to
check whether it exited, wasting time on every run since stalld
rejects bad arguments almost instantly. Run stalld synchronously
under timeout instead, and treat unexpected success or timeout as
test failures rather than warnings.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_affinity.sh             | 20 ++----
 tests/functional/test_boost_duration.sh       | 38 ++++--------
 tests/functional/test_boost_period.sh         | 38 ++++--------
 tests/functional/test_boost_runtime.sh        | 62 +++++++------------
 tests/functional/test_fifo_boosting.sh        | 17 ++---
 tests/functional/test_pidfile.sh              | 21 ++-----
 tests/functional/test_starvation_threshold.sh | 36 ++++-------
 7 files changed, 77 insertions(+), 155 deletions(-)

diff --git a/tests/functional/test_affinity.sh b/tests/functional/test_affinity.sh
index c3888ba..eac61f5 100755
--- a/tests/functional/test_affinity.sh
+++ b/tests/functional/test_affinity.sh
@@ -264,22 +264,14 @@ if [ -n "${STALLD_TEST_BACKEND}" ]; then
     BACKEND_FLAG="-b ${STALLD_TEST_BACKEND}"
 fi
 
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -l -t 5 -a ${invalid_cpu} > "${INVALID_LOG}" 2>&1 &
-invalid_pid=$!
-sleep 2
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -l -t 5 -a ${invalid_cpu} > "${INVALID_LOG}" 2>&1
+ret=$?
 
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    # Process exited
-    if grep -qi "error\|invalid\|failed" "${INVALID_LOG}"; then
-        log "✓ PASS: Invalid CPU affinity rejected with error"
-    else
-        log "ℹ INFO: Invalid CPU affinity caused exit"
-    fi
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Invalid CPU affinity rejected with error"
 else
-    # Process still running - might have been ignored
-    log "⚠ WARNING: stalld accepted invalid CPU affinity"
-    kill -TERM "${invalid_pid}" 2>/dev/null || true
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid CPU affinity"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 #=============================================================================
diff --git a/tests/functional/test_boost_duration.sh b/tests/functional/test_boost_duration.sh
index af657db..ff43df3 100755
--- a/tests/functional/test_boost_duration.sh
+++ b/tests/functional/test_boost_duration.sh
@@ -196,20 +196,14 @@ if [ -n "${STALLD_TEST_BACKEND}" ]; then
     BACKEND_FLAG="-b ${STALLD_TEST_BACKEND}"
 fi
 
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -d 0 > "${INVALID_LOG}" 2>&1 &
-invalid_pid=$!
-sleep 2
-
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG}"; then
-        log "✓ PASS: Zero duration rejected with error"
-    else
-        log "ℹ INFO: Zero duration caused exit (may have been rejected)"
-    fi
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -d 0 > "${INVALID_LOG}" 2>&1
+ret=$?
+
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Zero duration rejected with error"
 else
-    log "⚠ WARNING: stalld accepted zero duration"
-    kill -TERM "${invalid_pid}" 2>/dev/null || true
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid duration value 0"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Test 6: Negative duration
@@ -217,20 +211,14 @@ log "Testing with duration = -5"
 INVALID_LOG2="/tmp/stalld_test_boost_duration_invalid2_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG2}")
 
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -d -5 > "${INVALID_LOG2}" 2>&1 &
-invalid_pid=$!
-sleep 2
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -d -5 > "${INVALID_LOG2}" 2>&1
+ret=$?
 
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG2}"; then
-        log "✓ PASS: Negative duration rejected with error"
-    else
-        log "ℹ INFO: Negative duration caused exit"
-    fi
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Negative duration rejected with error"
 else
-    log "⚠ WARNING: stalld accepted negative duration"
-    kill -TERM "${invalid_pid}" 2>/dev/null || true
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid negative duration"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 log ""
diff --git a/tests/functional/test_boost_period.sh b/tests/functional/test_boost_period.sh
index 4838672..5b0ef2c 100755
--- a/tests/functional/test_boost_period.sh
+++ b/tests/functional/test_boost_period.sh
@@ -189,20 +189,14 @@ if [ -n "${STALLD_TEST_BACKEND}" ]; then
     BACKEND_FLAG="-b ${STALLD_TEST_BACKEND}"
 fi
 
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t $threshold -p 0 > "${INVALID_LOG}" 2>&1 &
-invalid_pid=$!
-sleep 2
-
-if ! kill -0 "$invalid_pid" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG}"; then
-        log "✓ PASS: Zero period rejected with error"
-    else
-        log "ℹ INFO: Zero period caused exit"
-    fi
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t $threshold -p 0 > "${INVALID_LOG}" 2>&1
+ret=$?
+
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Zero period rejected with error"
 else
-    log "⚠ WARNING: stalld accepted zero period"
-    kill -TERM "$invalid_pid" 2>/dev/null || true
-    wait "$invalid_pid" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid period value 0"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 #=============================================================================
@@ -216,20 +210,14 @@ log "=========================================="
 INVALID_LOG2="/tmp/stalld_test_boost_period_invalid2_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG2}")
 
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t $threshold -p -1000000 > "${INVALID_LOG2}" 2>&1 &
-invalid_pid=$!
-sleep 2
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t $threshold -p -1000000 > "${INVALID_LOG2}" 2>&1
+ret=$?
 
-if ! kill -0 "$invalid_pid" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG2}"; then
-        log "✓ PASS: Negative period rejected with error"
-    else
-        log "ℹ INFO: Negative period caused exit"
-    fi
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Negative period rejected with error"
 else
-    log "⚠ WARNING: stalld accepted negative period"
-    kill -TERM "$invalid_pid" 2>/dev/null || true
-    wait "$invalid_pid" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid negative period"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 #=============================================================================
diff --git a/tests/functional/test_boost_runtime.sh b/tests/functional/test_boost_runtime.sh
index 734a6f8..83e8fc5 100755
--- a/tests/functional/test_boost_runtime.sh
+++ b/tests/functional/test_boost_runtime.sh
@@ -196,22 +196,14 @@ if [ -n "${STALLD_TEST_BACKEND}" ]; then
 fi
 
 log "Testing with runtime ${invalid_runtime}ns > period ${period}ns"
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r ${invalid_runtime} -p ${period} > "${INVALID_LOG}" 2>&1 &
-invalid_pid=$!
-sleep 2
-
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    # Process exited - this is expected behavior
-    if grep -qi "error\|invalid\|failed" "${INVALID_LOG}"; then
-        log "✓ PASS: Runtime > period rejected with error"
-    else
-        log "ℹ INFO: Runtime > period caused exit"
-    fi
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r ${invalid_runtime} -p ${period} > "${INVALID_LOG}" 2>&1
+ret=$?
+
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Runtime > period rejected with error"
 else
-    # Process still running - might be accepted or might fail later
-    log "⚠ WARNING: stalld accepted runtime > period"
-    kill -TERM "${invalid_pid}" 2>/dev/null || true
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid runtime > period"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 #=============================================================================
@@ -226,20 +218,14 @@ INVALID_LOG2="/tmp/stalld_test_boost_runtime_invalid2_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG2}")
 
 log "Testing with runtime = 0"
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r 0 > "${INVALID_LOG2}" 2>&1 &
-invalid_pid=$!
-sleep 2
-
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG2}"; then
-        log "✓ PASS: Zero runtime rejected with error"
-    else
-        log "ℹ INFO: Zero runtime caused exit"
-    fi
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r 0 > "${INVALID_LOG2}" 2>&1
+ret=$?
+
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Zero runtime rejected with error"
 else
-    log "⚠ WARNING: stalld accepted zero runtime"
-    kill -TERM "${invalid_pid}" 2>/dev/null || true
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid runtime value 0"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 #=============================================================================
@@ -254,20 +240,14 @@ INVALID_LOG3="/tmp/stalld_test_boost_runtime_invalid3_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG3}")
 
 log "Testing with runtime = -5000"
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r -5000 > "${INVALID_LOG3}" 2>&1 &
-invalid_pid=$!
-sleep 2
-
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG3}"; then
-        log "✓ PASS: Negative runtime rejected with error"
-    else
-        log "ℹ INFO: Negative runtime caused exit"
-    fi
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r -5000 > "${INVALID_LOG3}" 2>&1
+ret=$?
+
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Negative runtime rejected with error"
 else
-    log "⚠ WARNING: stalld accepted negative runtime"
-    kill -TERM "${invalid_pid}" 2>/dev/null || true
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid negative runtime"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 log ""
diff --git a/tests/functional/test_fifo_boosting.sh b/tests/functional/test_fifo_boosting.sh
index 3088141..dee3250 100755
--- a/tests/functional/test_fifo_boosting.sh
+++ b/tests/functional/test_fifo_boosting.sh
@@ -331,19 +331,14 @@ CLEANUP_FILES+=("${STALLD_LOG_FAIL}")
 
 # Try to start stalld with -F but without -A (single-threaded mode)
 # This should fail because single-threaded mode only works with DEADLINE
-${TEST_ROOT}/../stalld -f -v -F -t 5 -c ${TEST_CPU} > "${STALLD_LOG_FAIL}" 2>&1 &
-FAIL_PID=$!
+timeout 5 ${TEST_ROOT}/../stalld -f -v -F -t 5 -c ${TEST_CPU} > "${STALLD_LOG_FAIL}" 2>&1
+ret=$?
 
-# Wait a bit for it to fail
-sleep 3
-
-# Check if it's still running (it shouldn't be)
-if ps -p ${FAIL_PID} > /dev/null 2>&1; then
-    log "⚠ WARNING: stalld is still running (should have exited)"
-    kill -TERM ${FAIL_PID} 2>/dev/null
-    wait ${FAIL_PID} 2>/dev/null
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "stalld exited as expected"
 else
-    log "✓ PASS: stalld exited as expected"
+    log "✗ FAIL: stalld did not reject FIFO in single-threaded mode"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Check for error message in log
diff --git a/tests/functional/test_pidfile.sh b/tests/functional/test_pidfile.sh
index 155855e..6964c0c 100755
--- a/tests/functional/test_pidfile.sh
+++ b/tests/functional/test_pidfile.sh
@@ -215,23 +215,14 @@ if [ -n "${STALLD_TEST_BACKEND}" ]; then
 fi
 
 log "Testing invalid pidfile path: ${invalid_pidfile}"
-${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -l -t 5 --pidfile "${invalid_pidfile}" > "${INVALID_LOG}" 2>&1 &
-invalid_pid=$!
-sleep 2
+timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -l -t 5 --pidfile "${invalid_pidfile}" > "${INVALID_LOG}" 2>&1
+ret=$?
 
-# Check if stalld is still running
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    # Process exited
-    if grep -qi "error\|permission\|denied\|failed" "${INVALID_LOG}"; then
-        log "✓ PASS: Invalid pidfile path rejected with error"
-    else
-        log "ℹ INFO: Invalid pidfile path caused exit"
-    fi
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Invalid pidfile path rejected with error"
 else
-    # Process still running - might have accepted it or created elsewhere
-    log "⚠ WARNING: stalld running despite potentially invalid pidfile path"
-    kill -TERM "${invalid_pid}" 2>/dev/null || true
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid pidfile path"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
diff --git a/tests/functional/test_starvation_threshold.sh b/tests/functional/test_starvation_threshold.sh
index 9b14009..709f565 100755
--- a/tests/functional/test_starvation_threshold.sh
+++ b/tests/functional/test_starvation_threshold.sh
@@ -187,20 +187,14 @@ log "Testing with threshold = 0"
 INVALID_LOG="/tmp/stalld_test_threshold_invalid_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG}")
 
-${TEST_ROOT}/../stalld -f -v -t 0 > "${INVALID_LOG}" 2>&1 &
-invalid_pid=$!
-sleep 2
+timeout 5 ${TEST_ROOT}/../stalld -f -v -t 0 > "${INVALID_LOG}" 2>&1
+ret=$?
 
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG}"; then
-        log "✓ PASS: Zero threshold rejected with error"
-    else
-        log "ℹ INFO: Zero threshold caused exit (may have been rejected)"
-    fi
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Zero threshold rejected with error"
 else
-    log "⚠ WARNING: stalld accepted zero threshold"
-    kill -TERM "${invalid_pid}" 2>/dev/null
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid threshold value 0"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Test with negative threshold
@@ -208,20 +202,14 @@ log "Testing with threshold = -5"
 INVALID_LOG2="/tmp/stalld_test_threshold_invalid2_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG2}")
 
-${TEST_ROOT}/../stalld -f -v -t -5 > "${INVALID_LOG2}" 2>&1 &
-invalid_pid=$!
-sleep 2
+timeout 5 ${TEST_ROOT}/../stalld -f -v -t -5 > "${INVALID_LOG2}" 2>&1
+ret=$?
 
-if ! kill -0 "${invalid_pid}" 2>/dev/null; then
-    if grep -qi "error\|invalid" "${INVALID_LOG2}"; then
-        log "✓ PASS: Negative threshold rejected with error"
-    else
-        log "ℹ INFO: Negative threshold caused exit"
-    fi
+if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
+    assert_equals "1" "1" "Negative threshold rejected with error"
 else
-    log "⚠ WARNING: stalld accepted negative threshold"
-    kill -TERM "${invalid_pid}" 2>/dev/null
-    wait "${invalid_pid}" 2>/dev/null || true
+    log "✗ FAIL: stalld did not reject invalid negative threshold"
+    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 log ""
-- 
2.53.0

[PATCH stalld 28/36] tests: Add pass() helper and replace assert_equals hack

[Wander Lairson Costa]

Tests record pass results by calling assert_equals with identical
expected and actual values, which is non-obvious and misleading.
Add a pass() function that directly increments the pass counter
and logs the result, then replace all 33 occurrences of the hack
across 13 test files.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_affinity.sh             |  2 +-
 tests/functional/test_backend_selection.sh    |  2 +-
 tests/functional/test_boost_duration.sh       |  4 ++--
 tests/functional/test_boost_period.sh         |  4 ++--
 tests/functional/test_boost_runtime.sh        |  6 +++---
 tests/functional/test_cpu_selection.sh        | 12 ++++++------
 tests/functional/test_fifo_boosting.sh        |  2 +-
 tests/functional/test_force_fifo.sh           |  2 +-
 tests/functional/test_foreground.sh           |  8 ++++----
 tests/functional/test_log_only.sh             |  4 ++--
 tests/functional/test_logging_destinations.sh | 14 +++++++-------
 tests/functional/test_pidfile.sh              |  2 +-
 tests/functional/test_starvation_threshold.sh |  4 ++--
 tests/helpers/test_helpers.sh                 | 11 ++++++++++-
 14 files changed, 43 insertions(+), 34 deletions(-)

diff --git a/tests/functional/test_affinity.sh b/tests/functional/test_affinity.sh
index eac61f5..738822e 100755
--- a/tests/functional/test_affinity.sh
+++ b/tests/functional/test_affinity.sh
@@ -268,7 +268,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -l -t 5 -a ${invalid_cpu}
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Invalid CPU affinity rejected with error"
+    pass "Invalid CPU affinity rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid CPU affinity"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_backend_selection.sh b/tests/functional/test_backend_selection.sh
index c274025..8a22e0a 100755
--- a/tests/functional/test_backend_selection.sh
+++ b/tests/functional/test_backend_selection.sh
@@ -46,7 +46,7 @@ test_backend_flag() {
 	fi
 
 	if grep -q "${expected_msg}" "${log_file}"; then
-		assert_equals "0" "0" "${description}"
+		pass "${description}"
 	else
 		TEST_FAILED=$((TEST_FAILED + 1))
 		echo -e "  ${RED}FAIL${NC}: Backend message not found (${description})"
diff --git a/tests/functional/test_boost_duration.sh b/tests/functional/test_boost_duration.sh
index ff43df3..104f298 100755
--- a/tests/functional/test_boost_duration.sh
+++ b/tests/functional/test_boost_duration.sh
@@ -200,7 +200,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -d 0 > "$
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Zero duration rejected with error"
+    pass "Zero duration rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid duration value 0"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -215,7 +215,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -d -5 > "
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Negative duration rejected with error"
+    pass "Negative duration rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid negative duration"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_boost_period.sh b/tests/functional/test_boost_period.sh
index 5b0ef2c..b516350 100755
--- a/tests/functional/test_boost_period.sh
+++ b/tests/functional/test_boost_period.sh
@@ -193,7 +193,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t $threshold -p 0 > "${I
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Zero period rejected with error"
+    pass "Zero period rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid period value 0"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -214,7 +214,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t $threshold -p -1000000
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Negative period rejected with error"
+    pass "Negative period rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid negative period"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_boost_runtime.sh b/tests/functional/test_boost_runtime.sh
index 83e8fc5..d516604 100755
--- a/tests/functional/test_boost_runtime.sh
+++ b/tests/functional/test_boost_runtime.sh
@@ -200,7 +200,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r ${inva
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Runtime > period rejected with error"
+    pass "Runtime > period rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid runtime > period"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -222,7 +222,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r 0 > "$
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Zero runtime rejected with error"
+    pass "Zero runtime rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid runtime value 0"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -244,7 +244,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -t ${threshold} -r -5000
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Negative runtime rejected with error"
+    pass "Negative runtime rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid negative runtime"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_cpu_selection.sh b/tests/functional/test_cpu_selection.sh
index 7960f6c..2f9875d 100755
--- a/tests/functional/test_cpu_selection.sh
+++ b/tests/functional/test_cpu_selection.sh
@@ -42,7 +42,7 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -c 0 -l -t 5
 
 # Check that stalld mentions CPU 0
 if grep -q "cpu 0" "$STALLD_LOG"; then
-    assert_equals "1" "1" "stalld monitoring CPU 0"
+    pass "stalld monitoring CPU 0"
 else
     TEST_FAILED=$((TEST_FAILED + 1))
     echo -e "  ${RED}FAIL${NC}: stalld not monitoring CPU 0"
@@ -68,7 +68,7 @@ if [ "$num_cpus" -ge 4 ]; then
     fi
 
     if [ "$cpu0_found" -eq 1 ] && [ "$cpu2_found" -eq 1 ]; then
-        assert_equals "1" "1" "stalld monitoring CPUs 0 and 2"
+        pass "stalld monitoring CPUs 0 and 2"
     else
         TEST_FAILED=$((TEST_FAILED + 1))
         echo -e "  ${RED}FAIL${NC}: stalld not monitoring specified CPUs (0: $cpu0_found, 2: $cpu2_found)"
@@ -101,7 +101,7 @@ if [ "$num_cpus" -ge 4 ]; then
     fi
 
     if [ "$cpu0_found" -eq 1 ] && [ "$cpu1_found" -eq 1 ] && [ "$cpu2_found" -eq 1 ]; then
-        assert_equals "1" "1" "stalld monitoring CPUs 0-2"
+        pass "stalld monitoring CPUs 0-2"
     else
         TEST_FAILED=$((TEST_FAILED + 1))
         echo -e "  ${RED}FAIL${NC}: stalld not monitoring specified CPU range (0: $cpu0_found, 1: $cpu1_found, 2: $cpu2_found)"
@@ -128,7 +128,7 @@ if [ "$num_cpus" -ge 6 ]; then
     done
 
     if [ "$monitored_cpus" -eq 4 ]; then
-        assert_equals "1" "1" "stalld monitoring combined CPU specification (0,2-4)"
+        pass "stalld monitoring combined CPU specification (0,2-4)"
     else
         TEST_FAILED=$((TEST_FAILED + 1))
         echo -e "  ${RED}FAIL${NC}: stalld not monitoring all specified CPUs (found $monitored_cpus/4)"
@@ -153,7 +153,7 @@ timeout 5 "${TEST_ROOT}/../stalld" -f -v -c $invalid_cpu -l -t 5 > "${INVALID_LO
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "stalld rejected invalid CPU number"
+    pass "stalld rejected invalid CPU number"
 else
     log "✗ FAIL: stalld did not reject invalid CPU"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -168,7 +168,7 @@ if [ "$num_cpus" -ge 2 ]; then
 
     # Check that CPU 1 is NOT mentioned (or mentioned as "not monitoring")
     if ! grep -q "cpu 1" "$STALLD_LOG" || grep -q "not monitoring.*cpu 1" "$STALLD_LOG"; then
-        assert_equals "1" "1" "stalld not monitoring non-selected CPU 1"
+        pass "stalld not monitoring non-selected CPU 1"
     else
         TEST_FAILED=$((TEST_FAILED + 1))
         echo -e "  ${RED}FAIL${NC}: stalld appears to be monitoring CPU 1 when only CPU 0 selected"
diff --git a/tests/functional/test_fifo_boosting.sh b/tests/functional/test_fifo_boosting.sh
index dee3250..1da1c09 100755
--- a/tests/functional/test_fifo_boosting.sh
+++ b/tests/functional/test_fifo_boosting.sh
@@ -335,7 +335,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v -F -t 5 -c ${TEST_CPU} > "${STALLD_LOG_FA
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "stalld exited as expected"
+    pass "stalld exited as expected"
 else
     log "✗ FAIL: stalld did not reject FIFO in single-threaded mode"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index 2c21ddd..f8f7b4c 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -218,7 +218,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v -c "${TEST_CPU}" -t ${threshold} -F > "${
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "single-threaded mode rejected FIFO"
+    pass "single-threaded mode rejected FIFO"
 else
     log "✗ FAIL: stalld did not reject -F in single-threaded mode"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_foreground.sh b/tests/functional/test_foreground.sh
index c5afbba..2bb0282 100755
--- a/tests/functional/test_foreground.sh
+++ b/tests/functional/test_foreground.sh
@@ -31,12 +31,12 @@ if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 	# Check parent process - should be init (PID 1) or systemd
 	PARENT_PID=$(ps -o ppid= -p ${STALLD_PID} 2>/dev/null | tr -d ' ')
 	if [ "${PARENT_PID}" == "1" ] || [ "${PARENT_PID}" == "2" ]; then
-		assert_equals "1" "1" "stalld daemonized (parent is init/kthreadd)"
+		pass "stalld daemonized (parent is init/kthreadd)"
 	else
 		# On modern systems with session leaders, ppid might not be 1
 		# Just verify it's not our shell's PID
 		if [ "${PARENT_PID}" != "$$" ]; then
-			assert_equals "1" "1" "stalld daemonized (parent is not test shell)"
+			pass "stalld daemonized (parent is not test shell)"
 		else
 			TEST_FAILED=$((TEST_FAILED + 1))
 			echo -e "  ${RED}FAIL${NC}: stalld did not daemonize (parent is test shell)"
@@ -62,7 +62,7 @@ if assert_process_running "${STALLD_PID}" "stalld should be running with -f"; th
 	# The parent might be the subshell from start_stalld, not directly our shell
 	# So we just verify it's not PID 1
 	if [ "${PARENT_PID}" != "1" ]; then
-		assert_equals "1" "1" "stalld did not daemonize with -f (parent is not init)"
+		pass "stalld did not daemonize with -f (parent is not init)"
 	else
 		TEST_FAILED=$((TEST_FAILED + 1))
 		echo -e "  ${RED}FAIL${NC}: stalld daemonized even with -f flag"
@@ -82,7 +82,7 @@ if assert_process_running "${STALLD_PID}" "stalld should be running with -v"; th
 	PARENT_PID=$(ps -o ppid= -p ${STALLD_PID} 2>/dev/null | tr -d ' ')
 
 	if [ "${PARENT_PID}" != "1" ]; then
-		assert_equals "1" "1" "-v implies foreground mode"
+		pass "-v implies foreground mode"
 	else
 		TEST_FAILED=$((TEST_FAILED + 1))
 		echo -e "  ${RED}FAIL${NC}: -v should imply foreground mode"
diff --git a/tests/functional/test_log_only.sh b/tests/functional/test_log_only.sh
index ba72b8e..d6257cf 100755
--- a/tests/functional/test_log_only.sh
+++ b/tests/functional/test_log_only.sh
@@ -59,7 +59,7 @@ echo "Waiting for starvation detection..."
 
 # Check if stalld detected the starvation (should log it)
 if wait_for_starvation_detected "${LOG_FILE}"; then
-	assert_equals "1" "1" "stalld detected and logged starvation"
+	pass "stalld detected and logged starvation"
 else
 	TEST_FAILED=$((TEST_FAILED + 1))
 	echo -e "  ${RED}FAIL${NC}: stalld did not detect starvation"
@@ -69,7 +69,7 @@ fi
 
 # Check that stalld did NOT boost (should not see "boosted" message with -l)
 if ! grep -q "boosted" "${LOG_FILE}"; then
-	assert_equals "1" "1" "stalld did not boost in log-only mode"
+	pass "stalld did not boost in log-only mode"
 else
 	TEST_FAILED=$((TEST_FAILED + 1))
 	echo -e "  ${RED}FAIL${NC}: stalld boosted despite -l flag"
diff --git a/tests/functional/test_logging_destinations.sh b/tests/functional/test_logging_destinations.sh
index b44c08d..35470c0 100755
--- a/tests/functional/test_logging_destinations.sh
+++ b/tests/functional/test_logging_destinations.sh
@@ -38,11 +38,11 @@ start_stalld_with_log "${LOG_FILE}" -f -v -l -t 5
 if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 	# Check that output was written to our log file
 	if [ -s "${LOG_FILE}" ]; then
-		assert_equals "1" "1" "verbose mode produces output"
+		pass "verbose mode produces output"
 
 		# Should contain initialization messages
 		if grep -q -E "(stalld|version|monitoring)" "${LOG_FILE}"; then
-			assert_equals "1" "1" "output contains expected messages"
+			pass "output contains expected messages"
 		fi
 	else
 		TEST_FAILED=$((TEST_FAILED + 1))
@@ -72,7 +72,7 @@ if command -v dmesg >/dev/null 2>&1; then
 		if [ ${DMESG_AFTER} -gt ${DMESG_BEFORE} ]; then
 			# Check if recent dmesg contains stalld messages
 			if dmesg | tail -10 | has_stalld_log; then
-				assert_equals "1" "1" "stalld messages in kernel log"
+				pass "stalld messages in kernel log"
 			else
 				echo -e "  ${YELLOW}SKIP${NC}: cannot verify kernel log messages"
 			fi
@@ -111,7 +111,7 @@ if [ -n "${SYSLOG_FILE}" ]; then
 		if [ ${SYSLOG_AFTER} -gt ${SYSLOG_BEFORE} ]; then
 			# Check for stalld messages in recent syslog
 			if tail -20 "${SYSLOG_FILE}" | has_stalld_log; then
-				assert_equals "1" "1" "stalld messages in syslog"
+				pass "stalld messages in syslog"
 			else
 				echo -e "  ${YELLOW}SKIP${NC}: no stalld messages found in syslog"
 			fi
@@ -131,9 +131,9 @@ elif command -v journalctl >/dev/null 2>&1; then
 	if assert_process_running "${STALLD_PID}" "stalld with -s should be running"; then
 		# Check journalctl for stalld messages
 		if journalctl -u stalld --since "1 minute ago" 2>/dev/null | has_stalld_log; then
-			assert_equals "1" "1" "stalld messages in journalctl"
+			pass "stalld messages in journalctl"
 		elif journalctl --since "1 minute ago" 2>/dev/null | has_stalld_log; then
-			assert_equals "1" "1" "stalld messages in system journal"
+			pass "stalld messages in system journal"
 		else
 			echo -e "  ${YELLOW}SKIP${NC}: no stalld messages in journal (may take time to appear)"
 		fi
@@ -156,7 +156,7 @@ start_stalld_with_log "${LOG_FILE}" -f -v -k -s -l -t 5
 if assert_process_running "${STALLD_PID}" "stalld with combined logging should be running"; then
 	# Verify verbose output
 	if [ -s "${LOG_FILE}" ]; then
-		assert_equals "1" "1" "combined logging produces output"
+		pass "combined logging produces output"
 	else
 		TEST_FAILED=$((TEST_FAILED + 1))
 		echo -e "  ${RED}FAIL${NC}: no output with combined logging"
diff --git a/tests/functional/test_pidfile.sh b/tests/functional/test_pidfile.sh
index 6964c0c..fbdc9fe 100755
--- a/tests/functional/test_pidfile.sh
+++ b/tests/functional/test_pidfile.sh
@@ -219,7 +219,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v ${BACKEND_FLAG} -l -t 5 --pidfile "${inva
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Invalid pidfile path rejected with error"
+    pass "Invalid pidfile path rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid pidfile path"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_starvation_threshold.sh b/tests/functional/test_starvation_threshold.sh
index 709f565..b904867 100755
--- a/tests/functional/test_starvation_threshold.sh
+++ b/tests/functional/test_starvation_threshold.sh
@@ -191,7 +191,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v -t 0 > "${INVALID_LOG}" 2>&1
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Zero threshold rejected with error"
+    pass "Zero threshold rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid threshold value 0"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -206,7 +206,7 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v -t -5 > "${INVALID_LOG2}" 2>&1
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    assert_equals "1" "1" "Negative threshold rejected with error"
+    pass "Negative threshold rejected with error"
 else
     log "✗ FAIL: stalld did not reject invalid negative threshold"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 864036d..144114c 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -105,6 +105,15 @@ end_test() {
 	fi
 }
 
+# Record a test pass with a description message.
+#
+# Usage: pass "description"
+pass() {
+	local message=${1:-""}
+	log "✓ PASS: ${message}"
+	TEST_PASSED=$((TEST_PASSED + 1))
+}
+
 # Assert functions
 assert_equals() {
 	local expected=$1
@@ -1121,7 +1130,7 @@ start_starvation_gen() {
 
 # Export functions for use in tests
 export -f start_test end_test
-export -f assert_equals assert_contains assert_not_contains
+export -f pass assert_equals assert_contains assert_not_contains
 export -f assert_file_exists assert_file_not_exists
 export -f assert_process_running assert_process_not_running
 export -f start_stalld stop_stalld kill_existing_stalld cleanup
-- 
2.53.0

[PATCH stalld 29/36] tests/functional: Use pass() for all test pass reporting

[Wander Lairson Costa]

Replace inline log statements that manually format pass messages
with calls to the pass() helper introduced in the previous commit.
This ensures all pass results are consistently logged with the
journal and counted by the test framework.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_affinity.sh             | 14 ++++----
 tests/functional/test_boost_duration.sh       |  8 ++---
 tests/functional/test_boost_period.sh         |  8 ++---
 tests/functional/test_boost_restoration.sh    | 24 ++++++-------
 tests/functional/test_boost_runtime.sh        |  8 ++---
 tests/functional/test_deadline_boosting.sh    | 24 ++++++-------
 tests/functional/test_fifo_boosting.sh        | 14 ++++----
 .../test_fifo_priority_starvation.sh          | 18 +++++-----
 tests/functional/test_force_fifo.sh           |  8 ++---
 tests/functional/test_idle_detection.sh       | 12 +++----
 tests/functional/test_pidfile.sh              | 18 +++++-----
 tests/functional/test_runqueue_parsing.sh     | 36 +++++++++----------
 tests/functional/test_starvation_detection.sh | 22 ++++++------
 tests/functional/test_starvation_threshold.sh |  6 ++--
 tests/functional/test_task_merging.sh         | 16 ++++-----
 15 files changed, 118 insertions(+), 118 deletions(-)

diff --git a/tests/functional/test_affinity.sh b/tests/functional/test_affinity.sh
index 738822e..b74886d 100755
--- a/tests/functional/test_affinity.sh
+++ b/tests/functional/test_affinity.sh
@@ -83,7 +83,7 @@ log "ℹ INFO: Default affinity: $default_affinity"
 
 # Typically should be all CPUs
 if [ -n "$default_affinity" ]; then
-    log "✓ PASS: stalld has default affinity: $default_affinity"
+    pass "stalld has default affinity: $default_affinity"
 else
     log "⚠ WARNING: Could not determine default affinity"
 fi
@@ -107,7 +107,7 @@ sleep 2
 affinity=$(check_affinity "${STALLD_PID}")
 
 if [ "$affinity" = "0" ]; then
-    log "✓ PASS: stalld restricted to CPU 0"
+    pass "stalld restricted to CPU 0"
 else
     log "✗ FAIL: stalld affinity ($affinity) doesn't match requested (0)"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -134,7 +134,7 @@ if [ "$num_cpus" -ge 4 ]; then
 
     # Accept either "0,2" or "0-2" or "2,0" (different systems may report differently)
     if echo "$affinity" | grep -qE '^0,2$|^0-2$|^2,0$'; then
-        log "✓ PASS: stalld restricted to CPUs 0,2 (affinity: $affinity)"
+        pass "stalld restricted to CPUs 0,2 (affinity: $affinity)"
     else
         log "⚠ WARNING: stalld affinity ($affinity) may not match requested (0,2) - format may vary"
         # Not failing as different systems may report differently
@@ -164,7 +164,7 @@ if [ "$num_cpus" -ge 4 ]; then
 
     # Accept various formats: "0-2", "0,1,2", etc.
     if echo "$affinity" | grep -qE '0.*1.*2|0-2'; then
-        log "✓ PASS: stalld restricted to CPU range 0-2 (affinity: $affinity)"
+        pass "stalld restricted to CPU range 0-2 (affinity: $affinity)"
     else
         log "⚠ WARNING: stalld affinity ($affinity) may not match requested (0-2) - format may vary"
     fi
@@ -200,7 +200,7 @@ if [ "$num_cpus" -ge 2 ]; then
     fi
 
     if [ "$affinity" = "$test_cpu" ]; then
-        log "✓ PASS: stalld process affinity set to CPU $test_cpu"
+        pass "stalld process affinity set to CPU $test_cpu"
     else
         log "⚠ WARNING: stalld affinity ($affinity) doesn't exactly match CPU $test_cpu"
     fi
@@ -229,7 +229,7 @@ if [ "$num_cpus" -ge 2 ]; then
     affinity=$(check_affinity "${STALLD_PID}")
 
     if [ "$affinity" = "0" ]; then
-        log "✓ PASS: stalld affinity to CPU 0 while monitoring CPU 1"
+        pass "stalld affinity to CPU 0 while monitoring CPU 1"
     else
         log "⚠ WARNING: stalld affinity ($affinity) doesn't match requested (0)"
     fi
@@ -298,7 +298,7 @@ affinity_end=$(check_affinity "${STALLD_PID}")
 log "ℹ INFO: Affinity after 3s: $affinity_end"
 
 if [ "$affinity_start" = "$affinity_end" ]; then
-    log "✓ PASS: CPU affinity persisted over time"
+    pass "CPU affinity persisted over time"
 else
     log "⚠ WARNING: CPU affinity changed (start: $affinity_start, end: $affinity_end)"
 fi
diff --git a/tests/functional/test_boost_duration.sh b/tests/functional/test_boost_duration.sh
index 104f298..61218e9 100755
--- a/tests/functional/test_boost_duration.sh
+++ b/tests/functional/test_boost_duration.sh
@@ -67,7 +67,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for starvation detection
 if wait_for_starvation_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Starvation detection occurred with default duration"
+    pass "Starvation detection occurred with default duration"
 else
     log "✗ FAIL: No starvation detection"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -99,7 +99,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for starvation detection
 if wait_for_starvation_detected "${STALLD_LOG2}"; then
-    log "✓ PASS: Starvation detection with ${short_duration}s duration"
+    pass "Starvation detection with ${short_duration}s duration"
 else
     log "✗ FAIL: No starvation detection with short duration"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -133,7 +133,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${long_starvation}
 
 # Wait for starvation detection
 if wait_for_starvation_detected "${STALLD_LOG3}"; then
-    log "✓ PASS: Starvation detection with ${long_duration}s duration"
+    pass "Starvation detection with ${long_duration}s duration"
 else
     log "✗ FAIL: No starvation detection with long duration"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -166,7 +166,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 1 -d 15
 
 # Wait for starvation detection
 if wait_for_starvation_detected "${STALLD_LOG4}"; then
-    log "✓ PASS: Starvation detection with ${duration}s boost duration"
+    pass "Starvation detection with ${duration}s boost duration"
 else
     log "✗ FAIL: No starvation detection"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_boost_period.sh b/tests/functional/test_boost_period.sh
index b516350..671f7cb 100755
--- a/tests/functional/test_boost_period.sh
+++ b/tests/functional/test_boost_period.sh
@@ -64,7 +64,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for starvation detection and boosting
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boosting occurred with default period"
+    pass "Boosting occurred with default period"
 
     # Try to find period value in logs
     if grep -qi "period" "${STALLD_LOG}"; then
@@ -101,7 +101,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for starvation detection and boosting
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boosting occurred with custom period ${custom_period} ns"
+    pass "Boosting occurred with custom period ${custom_period} ns"
 else
     log "✗ FAIL: No boosting with custom period"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -131,7 +131,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for starvation detection and boosting
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boosting occurred with short period ${short_period} ns"
+    pass "Boosting occurred with short period ${short_period} ns"
 else
     log "✗ FAIL: No boosting with short period"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -161,7 +161,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for starvation detection and boosting
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boosting occurred with long period ${long_period} ns"
+    pass "Boosting occurred with long period ${long_period} ns"
 else
     log "✗ FAIL: No boosting with long period"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_boost_restoration.sh b/tests/functional/test_boost_restoration.sh
index fc5a25a..024d869 100755
--- a/tests/functional/test_boost_restoration.sh
+++ b/tests/functional/test_boost_restoration.sh
@@ -85,7 +85,7 @@ if [ -n "${tracked_pid}" ]; then
     log "Initial policy: ${initial_policy} (expected: 1=SCHED_FIFO), prio: ${initial_prio}"
 
     if [ "$initial_policy" = "1" ]; then
-        log "✓ PASS: Initial policy is SCHED_FIFO"
+        pass "Initial policy is SCHED_FIFO"
     else
         log "⚠ WARNING: Initial policy is ${initial_policy}, not SCHED_FIFO (1)"
     fi
@@ -100,7 +100,7 @@ if [ -n "${tracked_pid}" ]; then
         log "Policy during boost: ${boosted_policy}"
 
         if [ "$boosted_policy" = "6" ]; then
-            log "✓ PASS: Task boosted to SCHED_DEADLINE (6)"
+            pass "Task boosted to SCHED_DEADLINE (6)"
         else
             log "ℹ INFO: Policy is ${boosted_policy} (may be between boost cycles)"
         fi
@@ -117,7 +117,7 @@ if [ -n "${tracked_pid}" ]; then
         log "Policy after boost: ${post_boost_policy}, prio: ${post_boost_prio}"
 
         if [ "$post_boost_policy" = "1" ]; then
-            log "✓ PASS: Policy restored to SCHED_FIFO (1) during boost cycle"
+            pass "Policy restored to SCHED_FIFO (1) during boost cycle"
         elif [ "$post_boost_policy" = "6" ]; then
             log "ℹ INFO: Still in boost (DEADLINE), will check final restoration"
         else
@@ -139,9 +139,9 @@ if [ -n "${tracked_pid}" ] && [ -f "/proc/${tracked_pid}/sched" ]; then
     log "Final policy: ${final_policy}, prio: ${final_prio}"
 
     if [ "$final_policy" = "1" ]; then
-        log "✓ PASS: Policy restored to SCHED_FIFO (1)"
+        pass "Policy restored to SCHED_FIFO (1)"
         if [ "$final_prio" = "$initial_prio" ]; then
-            log "✓ PASS: Priority restored to ${initial_prio}"
+            pass "Priority restored to ${initial_prio}"
         else
             log "⚠ INFO: Priority is ${final_prio} (initial was ${initial_prio})"
         fi
@@ -219,7 +219,7 @@ if chrt -f -p 10 ${FIFO_TASK_PID} 2>/dev/null; then
     log "Initial: policy=${initial_policy} (1=FIFO), prio=${initial_prio}"
 
     if [ "$initial_policy" = "1" ]; then
-        log "✓ PASS: Initial policy is SCHED_FIFO (1)"
+        pass "Initial policy is SCHED_FIFO (1)"
     else
         log "⚠ WARNING: Could not set FIFO policy (got ${initial_policy})"
     fi
@@ -234,7 +234,7 @@ if chrt -f -p 10 ${FIFO_TASK_PID} 2>/dev/null; then
         log "Policy during detection window: ${boosted_policy}"
 
         if [ "$boosted_policy" = "6" ]; then
-            log "✓ PASS: Task boosted to SCHED_DEADLINE (6)"
+            pass "Task boosted to SCHED_DEADLINE (6)"
         elif [ "$boosted_policy" = "1" ]; then
             log "ℹ INFO: Still SCHED_FIFO (may not have starved yet)"
         fi
@@ -251,7 +251,7 @@ if chrt -f -p 10 ${FIFO_TASK_PID} 2>/dev/null; then
         log "Final: policy=${final_policy}, prio=${final_prio}"
 
         if [ "$final_policy" = "1" ]; then
-            log "✓ PASS: Policy restored to SCHED_FIFO (1)"
+            pass "Policy restored to SCHED_FIFO (1)"
             log "ℹ INFO: Priority after restoration: ${final_prio}"
         else
             log "⚠ INFO: Final policy is ${final_policy}"
@@ -340,7 +340,7 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
     log "Time difference: ${time_diff}s"
 
     if [ ${time_diff} -le 2 ]; then
-        log "✓ PASS: Restoration timing within acceptable margin (±2s)"
+        pass "Restoration timing within acceptable margin (±2s)"
     else
         log "ℹ INFO: Restoration timing difference: ${time_diff}s"
         log "        (may be acceptable depending on system load)"
@@ -380,7 +380,7 @@ start_starvation_gen -c ${TEST_CPU} -p 80 -n 1 -d ${short_duration}
 
 # Wait for starvation detection and boosting
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boost occurred"
+    pass "Boost occurred"
 
     # At this point (12s), starvation_gen has exited (at 8s) during the boost
     # stalld should still be running despite the task exiting during boost
@@ -389,7 +389,7 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
 
     # Verify stalld is still running and didn't crash after task exit
     if assert_process_running "${STALLD_PID}" "stalld still running after task exit"; then
-        log "✓ PASS: stalld handled task exit during boost gracefully"
+        pass "stalld handled task exit during boost gracefully"
     else
         log "✗ FAIL: stalld crashed or exited after task died during boost"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -401,7 +401,7 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
         grep -iE "error.*restor|fail.*restor" "${STALLD_LOG}"
         log "        These errors are normal when tasks exit during boost"
     else
-        log "✓ PASS: No restoration errors (clean handling)"
+        pass "No restoration errors (clean handling)"
     fi
 else
     log "⚠ WARNING: No boost detected in this test run"
diff --git a/tests/functional/test_boost_runtime.sh b/tests/functional/test_boost_runtime.sh
index d516604..b624fc9 100755
--- a/tests/functional/test_boost_runtime.sh
+++ b/tests/functional/test_boost_runtime.sh
@@ -67,7 +67,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 if wait_for_starvation_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Starvation detection with default runtime"
+    pass "Starvation detection with default runtime"
 else
     log "✗ FAIL: No starvation detection"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -99,7 +99,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 if wait_for_starvation_detected "${STALLD_LOG2}"; then
-    log "✓ PASS: Starvation detection with custom runtime ${custom_runtime}ns"
+    pass "Starvation detection with custom runtime ${custom_runtime}ns"
 else
     log "✗ FAIL: No starvation detection with custom runtime"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -131,7 +131,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 if wait_for_starvation_detected "${STALLD_LOG3}"; then
-    log "✓ PASS: Starvation detection with large runtime ${large_runtime}ns"
+    pass "Starvation detection with large runtime ${large_runtime}ns"
 else
     log "✗ FAIL: No starvation detection with large runtime"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -165,7 +165,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 
 # Wait for detection and boosting
 if wait_for_starvation_detected "${STALLD_LOG4}"; then
-    log "✓ PASS: Starvation detection with runtime < period"
+    pass "Starvation detection with runtime < period"
 else
     log "✗ FAIL: No starvation detection when runtime < period"
     TEST_FAILED=$((TEST_FAILED + 1))
diff --git a/tests/functional/test_deadline_boosting.sh b/tests/functional/test_deadline_boosting.sh
index a5727da..86ca0f1 100755
--- a/tests/functional/test_deadline_boosting.sh
+++ b/tests/functional/test_deadline_boosting.sh
@@ -68,11 +68,11 @@ start_starvation_gen -c ${TEST_CPU} -p 80 -n 2 -d ${starvation_duration}
 # Wait for boosting
 log "Waiting for boost detection..."
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boosting occurred"
+    pass "Boosting occurred"
 
     # Verify SCHED_DEADLINE was used
     if grep -q "SCHED_DEADLINE" "${STALLD_LOG}"; then
-        log "✓ PASS: SCHED_DEADLINE boosting used (default)"
+        pass "SCHED_DEADLINE boosting used (default)"
     else
         log "✗ FAIL: SCHED_DEADLINE not mentioned in boost message"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -81,7 +81,7 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
     # Verify boost happened after threshold
     # (starvation logged, then boosting)
     if grep -q "starved" "${STALLD_LOG}"; then
-        log "✓ PASS: Starvation detected before boosting"
+        pass "Starvation detected before boosting"
     else
         log "⚠ WARNING: No starvation message before boost"
     fi
@@ -140,7 +140,7 @@ for child_pid in ${STARVE_CHILDREN}; do
 
         # Policy 6 = SCHED_DEADLINE
         if [ "$policy" = "6" ]; then
-            log "✓ PASS: Task PID ${child_pid} boosted to SCHED_DEADLINE (policy 6)"
+            pass "Task PID ${child_pid} boosted to SCHED_DEADLINE (policy 6)"
             boosted_task_found=1
             break
         fi
@@ -151,7 +151,7 @@ if [ ${boosted_task_found} -eq 0 ]; then
     log "⚠ INFO: Could not verify DEADLINE policy in /proc (timing issue or boost already expired)"
     # Still check if boost happened in logs
     if grep -q "boosted.*SCHED_DEADLINE" "${STALLD_LOG}"; then
-        log "✓ PASS: SCHED_DEADLINE boost confirmed in logs"
+        pass "SCHED_DEADLINE boost confirmed in logs"
     else
         log "✗ FAIL: No SCHED_DEADLINE boost detected"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -213,7 +213,7 @@ if [ -n "${tracked_pid}" ]; then
     # Verify task made progress (context switches increased)
     ctxt_delta=$((ctxt_after - ctxt_before))
     if [ ${ctxt_delta} -gt 5 ]; then
-        log "✓ PASS: Task made progress during boost (${ctxt_delta} context switches)"
+        pass "Task made progress during boost (${ctxt_delta} context switches)"
     else
         log "⚠ INFO: Limited progress detected (${ctxt_delta} context switches)"
         log "        This may be acceptable depending on boost parameters"
@@ -224,7 +224,7 @@ fi
 
 # Verify boost happened
 if grep -q "boosted" "${STALLD_LOG}"; then
-    log "✓ PASS: Boost occurred as expected"
+    pass "Boost occurred as expected"
 else
     log "✗ FAIL: No boost detected"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -284,7 +284,7 @@ if [ -n "${tracked_pid}" ]; then
     log "Policy during boost window: ${boosted_policy} (6=SCHED_DEADLINE)"
 
     if [ "$boosted_policy" = "6" ]; then
-        log "✓ PASS: Policy changed to SCHED_DEADLINE during boost"
+        pass "Policy changed to SCHED_DEADLINE during boost"
     else
         log "⚠ INFO: Policy is ${boosted_policy} (may have already restored or not yet boosted)"
     fi
@@ -299,7 +299,7 @@ if [ -n "${tracked_pid}" ]; then
         log "Policy after boost: ${restored_policy}"
 
         if [ "$restored_policy" = "0" ]; then
-            log "✓ PASS: Policy restored to SCHED_OTHER (0)"
+            pass "Policy restored to SCHED_OTHER (0)"
         else
             log "⚠ INFO: Policy is ${restored_policy} after boost"
             log "        (task may have exited or restoration timing differs)"
@@ -360,18 +360,18 @@ else
     log "Number of boost events: ${boost_count}"
 
     if [ ${boost_count} -ge 2 ]; then
-        log "✓ PASS: Multiple boost events detected (${boost_count})"
+        pass "Multiple boost events detected (${boost_count})"
 
         # Verify both CPUs mentioned
         if grep -q "CPU ${CPU0}" "${STALLD_LOG}" && grep -q "CPU ${CPU1}" "${STALLD_LOG}"; then
-            log "✓ PASS: Boosts occurred on both CPUs"
+            pass "Boosts occurred on both CPUs"
         else
             log "⚠ INFO: Could not verify boosts on both specific CPUs"
         fi
 
         # Verify independent boost cycles
         if [ ${boost_count} -gt 2 ]; then
-            log "✓ PASS: Multiple boost cycles (${boost_count} total), showing independent operation"
+            pass "Multiple boost cycles (${boost_count} total), showing independent operation"
         fi
     else
         log "⚠ INFO: Only ${boost_count} boost event(s) detected"
diff --git a/tests/functional/test_fifo_boosting.sh b/tests/functional/test_fifo_boosting.sh
index 1da1c09..df26483 100755
--- a/tests/functional/test_fifo_boosting.sh
+++ b/tests/functional/test_fifo_boosting.sh
@@ -66,11 +66,11 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -g 1 -N -F -A -t $threshold -c ${TES
 # Wait for boosting
 log "Waiting for boost detection..."
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boosting occurred with -F flag"
+    pass "Boosting occurred with -F flag"
 
     # Verify SCHED_FIFO was used
     if grep -q "SCHED_FIFO" "${STALLD_LOG}"; then
-        log "✓ PASS: SCHED_FIFO boosting used (as requested by -F)"
+        pass "SCHED_FIFO boosting used (as requested by -F)"
     else
         log "✗ FAIL: SCHED_FIFO not mentioned in boost message"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -120,7 +120,7 @@ for child_pid in ${STARVE_CHILDREN}; do
         # Policy 1 = SCHED_FIFO
         if [ "$policy" = "1" ]; then
             priority=$(get_sched_priority ${child_pid})
-            log "✓ PASS: Task PID ${child_pid} boosted to SCHED_FIFO (policy 1)"
+            pass "Task PID ${child_pid} boosted to SCHED_FIFO (policy 1)"
             log "        Priority: ${priority}"
             fifo_task_found=1
             break
@@ -132,7 +132,7 @@ if [ ${fifo_task_found} -eq 0 ]; then
     log "⚠ INFO: Could not verify FIFO policy in /proc (timing issue or boost already expired)"
     # FIFO emulation cycles between FIFO and OTHER, so we may catch it in OTHER state
     if grep -q "boosted.*SCHED_FIFO" "${STALLD_LOG}"; then
-        log "✓ PASS: SCHED_FIFO boost confirmed in logs"
+        pass "SCHED_FIFO boost confirmed in logs"
     else
         log "✗ FAIL: No SCHED_FIFO boost detected"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -186,7 +186,7 @@ boost_count=$(grep -c "boosted.*SCHED_FIFO" "${STALLD_LOG}")
 log "Number of FIFO boost events: ${boost_count}"
 
 if [ ${boost_count} -gt 1 ]; then
-    log "✓ PASS: Multiple FIFO boost events (${boost_count}) - emulation cycling detected"
+    pass "Multiple FIFO boost events (${boost_count}) - emulation cycling detected"
     log "        (FIFO emulation boosts, sleeps, restores, repeats)"
 else
     log "⚠ INFO: Only ${boost_count} FIFO boost event(s)"
@@ -303,7 +303,7 @@ log "  DEADLINE: ${deadline_progress} context switches"
 log "  FIFO: ${fifo_progress} context switches"
 
 if [ ${deadline_progress} -gt 0 ] && [ ${fifo_progress} -gt 0 ]; then
-    log "✓ PASS: Both DEADLINE and FIFO allowed tasks to make progress"
+    pass "Both DEADLINE and FIFO allowed tasks to make progress"
 
     # Both should be effective, but exact numbers may vary
     if [ ${deadline_progress} -gt ${fifo_progress} ]; then
@@ -343,7 +343,7 @@ fi
 
 # Check for error message in log
 if grep -qiE "single.*thread.*fifo|fifo.*single.*thread|can.*only.*deadline" "${STALLD_LOG_FAIL}"; then
-    log "✓ PASS: Error message about FIFO+single-threaded incompatibility found"
+    pass "Error message about FIFO+single-threaded incompatibility found"
 else
     log "ℹ INFO: Checking exit status or error messages..."
     if [ -s "${STALLD_LOG_FAIL}" ]; then
diff --git a/tests/functional/test_fifo_priority_starvation.sh b/tests/functional/test_fifo_priority_starvation.sh
index 886357e..3776e6b 100755
--- a/tests/functional/test_fifo_priority_starvation.sh
+++ b/tests/functional/test_fifo_priority_starvation.sh
@@ -72,11 +72,11 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a $
 # Wait for starvation detection
 log "Waiting for starvation detection..."
 if wait_for_starvation_detected "${STALLD_LOG}"; then
-    log "✓ PASS: FIFO-on-FIFO starvation detected"
+    pass "FIFO-on-FIFO starvation detected"
 
     # Verify correct CPU is logged
     if grep "starved on CPU ${TEST_CPU}" "${STALLD_LOG}"; then
-        log "✓ PASS: Correct CPU ID logged (CPU ${TEST_CPU})"
+        pass "Correct CPU ID logged (CPU ${TEST_CPU})"
     else
         log "✗ FAIL: Wrong CPU ID in log"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -84,7 +84,7 @@ if wait_for_starvation_detected "${STALLD_LOG}"; then
 
     # Verify duration is logged
     if grep -E "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}"; then
-        log "✓ PASS: Starvation duration logged"
+        pass "Starvation duration logged"
     else
         log "✗ FAIL: Starvation duration not logged"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -152,7 +152,7 @@ ctxt_delta=$((ctxt_after - ctxt_before))
 log "Context switch delta: ${ctxt_delta}"
 
 if [ ${ctxt_delta} -gt 0 ]; then
-    log "✓ PASS: Blockee task made progress (${ctxt_delta} context switches)"
+    pass "Blockee task made progress (${ctxt_delta} context switches)"
 else
     log "⚠ WARNING: Could not verify progress (timing issue or blockee not found)"
     # Check if boosting occurred at least
@@ -204,7 +204,7 @@ log "Third detection cycle should have occurred"
 # Check if we see accumulating starvation time in logs
 # Task merging means the timestamp is preserved, so duration increases
 if grep -E "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}" | wc -l | grep -q "[2-9]"; then
-    log "✓ PASS: Multiple starvation reports found"
+    pass "Multiple starvation reports found"
 
     # Extract starvation durations from log
     durations=$(grep -oE "starved on CPU ${TEST_CPU} for [0-9]+" "${STALLD_LOG}" | grep -oE "[0-9]+$")
@@ -215,7 +215,7 @@ if grep -E "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}" | wc
     last_duration=$(echo "$durations" | tail -1)
 
     if [ ${last_duration} -gt ${first_duration} ]; then
-        log "✓ PASS: Starvation duration increased (${first_duration}s -> ${last_duration}s)"
+        pass "Starvation duration increased (${first_duration}s -> ${last_duration}s)"
         log "        This confirms task merging preserved the timestamp"
     else
         log "✗ FAIL: Starvation duration did not increase (timestamp may have been reset)"
@@ -258,7 +258,7 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -l -t $threshold -c ${TEST_CPU} -a $
 # Wait for starvation detection
 log "Waiting for starvation detection..."
 if wait_for_starvation_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Starvation detected even with close priority gap (6 vs 5)"
+    pass "Starvation detected even with close priority gap (6 vs 5)"
 else
     log "⚠ WARNING: Starvation not detected with close priority gap"
     log "        (May be due to queue_track backend limitation)"
@@ -297,12 +297,12 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -N -t $threshold -c ${TEST_CPU} -a $
 # Wait for boosting
 log "Waiting for boost detection..."
 if wait_for_boost_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Boosting occurred"
+    pass "Boosting occurred"
 
     # Try to verify the correct task was boosted
     # stalld logs should show the blockee task name (starvation_gen thread)
     if grep "boosted.*starvation_gen" "${STALLD_LOG}"; then
-        log "✓ PASS: starvation_gen task was boosted (likely the blockee)"
+        pass "starvation_gen task was boosted (likely the blockee)"
     else
         log "ℹ INFO: Could not verify specific task from logs"
     fi
diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index f8f7b4c..fec2e00 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -64,7 +64,7 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
 
     # Look for SCHED_DEADLINE indicators
     if grep -qi "deadline\|SCHED_DEADLINE" "${STALLD_LOG}"; then
-        log "✓ PASS: SCHED_DEADLINE used by default"
+        pass "SCHED_DEADLINE used by default"
     elif grep -qi "fifo\|SCHED_FIFO" "${STALLD_LOG}"; then
         log "⚠ WARNING: SCHED_FIFO used instead of SCHED_DEADLINE"
     else
@@ -105,7 +105,7 @@ if wait_for_boost_detected "${STALLD_LOG2}"; then
 
     # Look for SCHED_FIFO indicators
     if grep -qi "fifo\|SCHED_FIFO" "${STALLD_LOG2}"; then
-        log "✓ PASS: SCHED_FIFO used with -F flag"
+        pass "SCHED_FIFO used with -F flag"
     elif grep -qi "deadline\|SCHED_DEADLINE" "${STALLD_LOG2}"; then
         log "✗ FAIL: SCHED_DEADLINE used despite -F flag"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -148,7 +148,7 @@ if grep -qi "priority\|prio" "${STALLD_LOG3}"; then
 fi
 
 if grep -q "boost" "${STALLD_LOG3}"; then
-    log "✓ PASS: FIFO boosting with priority setting completed"
+    pass "FIFO boosting with priority setting completed"
 else
     log "⚠ WARNING: No boosting detected"
 fi
@@ -187,7 +187,7 @@ if wait_for_boost_detected "${STALLD_LOG4}"; then
 
     # Check for restoration messages (part of FIFO emulation)
     if grep -qi "restor\|unboosted\|normal\|original" "${STALLD_LOG4}"; then
-        log "✓ PASS: FIFO emulation with restoration detected"
+        pass "FIFO emulation with restoration detected"
     else
         log "ℹ INFO: FIFO boosting completed (restoration may be implicit)"
     fi
diff --git a/tests/functional/test_idle_detection.sh b/tests/functional/test_idle_detection.sh
index ef1c572..af4d7a4 100755
--- a/tests/functional/test_idle_detection.sh
+++ b/tests/functional/test_idle_detection.sh
@@ -100,7 +100,7 @@ fi
 
 # Verify CPU is actually idle
 if is_cpu_idle ${TEST_CPU}; then
-    log "✓ PASS: CPU ${TEST_CPU} is currently idle (idle time increasing)"
+    pass "CPU ${TEST_CPU} is currently idle (idle time increasing)"
 else
     log "⚠ INFO: CPU ${TEST_CPU} appears busy (background activity)"
 fi
@@ -130,10 +130,10 @@ if [ -n "${idle_time1}" ] && [ -n "${idle_time2}" ]; then
     log "Idle time delta: ${delta}"
 
     if [ ${delta} -gt 0 ]; then
-        log "✓ PASS: Idle time increased (CPU is idle)"
+        pass "Idle time increased (CPU is idle)"
         log "        stalld would skip this CPU"
     else
-        log "✓ PASS: Idle time unchanged (CPU is busy)"
+        pass "Idle time unchanged (CPU is busy)"
         log "        stalld would parse this CPU"
     fi
 else
@@ -167,7 +167,7 @@ log "Waiting for stalld to detect busy CPU and starvation..."
 
 # Verify stalld detected starvation (meaning it resumed monitoring)
 if wait_for_starvation_detected "${STALLD_LOG}"; then
-    log "✓ PASS: stalld detected starvation on now-busy CPU"
+    pass "stalld detected starvation on now-busy CPU"
     log "        Monitoring resumed when CPU became busy"
 else
     log "⚠ INFO: No starvation detected"
@@ -241,9 +241,9 @@ else
     log "CPU ${CPU1} detections: ${cpu1_detections} (should be >0, it's busy)"
 
     if [ ${cpu0_detections} -eq 0 ] && [ ${cpu1_detections} -gt 0 ]; then
-        log "✓ PASS: Idle CPU skipped, busy CPU monitored"
+        pass "Idle CPU skipped, busy CPU monitored"
     elif [ ${cpu1_detections} -gt 0 ]; then
-        log "✓ PASS: Busy CPU ${CPU1} monitored"
+        pass "Busy CPU ${CPU1} monitored"
         if [ ${cpu0_detections} -gt 0 ]; then
             log "⚠ INFO: CPU ${CPU0} also had detections (may have background activity)"
         fi
diff --git a/tests/functional/test_pidfile.sh b/tests/functional/test_pidfile.sh
index fbdc9fe..fd4fedc 100755
--- a/tests/functional/test_pidfile.sh
+++ b/tests/functional/test_pidfile.sh
@@ -54,7 +54,7 @@ for pidfile in /var/run/stalld.pid /run/stalld.pid; do
         # Verify PID matches
         pid_from_file=$(cat "$pidfile")
         if [ "$pid_from_file" = "${STALLD_PID}" ]; then
-            log "✓ PASS: Default pidfile contains correct PID"
+            pass "Default pidfile contains correct PID"
         else
             log "✗ FAIL: Default pidfile PID ($pid_from_file) doesn't match stalld PID (${STALLD_PID})"
             TEST_FAILED=$((TEST_FAILED + 1))
@@ -92,12 +92,12 @@ sleep 2
 
 # Verify pidfile was created
 if [ -f "${custom_pidfile}" ]; then
-    log "✓ PASS: Custom pidfile created at ${custom_pidfile}"
+    pass "Custom pidfile created at ${custom_pidfile}"
 
     # Verify content
     pid_from_file=$(cat "${custom_pidfile}")
     if [ "$pid_from_file" = "${STALLD_PID}" ]; then
-        log "✓ PASS: Custom pidfile contains correct PID ($pid_from_file)"
+        pass "Custom pidfile contains correct PID ($pid_from_file)"
     else
         log "✗ FAIL: Custom pidfile PID ($pid_from_file) doesn't match stalld PID (${STALLD_PID})"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -113,7 +113,7 @@ log "Test 3: Verify pidfile removed on clean shutdown"
 stop_stalld
 
 if [ ! -f "${custom_pidfile}" ]; then
-    log "✓ PASS: Pidfile removed on clean shutdown"
+    pass "Pidfile removed on clean shutdown"
 else
     log "⚠ WARNING: Pidfile still exists after shutdown (may be expected)"
     # Not failing - some implementations keep pidfile
@@ -139,11 +139,11 @@ start_stalld -l -t 5 --pidfile "${tmp_pidfile}"
 sleep 2
 
 if [ -f "${tmp_pidfile}" ]; then
-    log "✓ PASS: Pidfile created in /tmp directory"
+    pass "Pidfile created in /tmp directory"
 
     pid_from_file=$(cat "${tmp_pidfile}")
     if [ "$pid_from_file" = "${STALLD_PID}" ]; then
-        log "✓ PASS: /tmp pidfile contains correct PID"
+        pass "/tmp pidfile contains correct PID"
     else
         log "✗ FAIL: /tmp pidfile has incorrect PID"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -175,11 +175,11 @@ start_stalld -f -v -l -t 5 --pidfile "${fg_pidfile}"
 sleep 2
 
 if [ -f "${fg_pidfile}" ]; then
-    log "✓ PASS: Pidfile created in foreground mode"
+    pass "Pidfile created in foreground mode"
 
     pid_from_file=$(cat "${fg_pidfile}")
     if [ "$pid_from_file" = "${STALLD_PID}" ]; then
-        log "✓ PASS: Foreground mode pidfile contains correct PID"
+        pass "Foreground mode pidfile contains correct PID"
     else
         log "✗ FAIL: Foreground mode pidfile has incorrect PID"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -250,7 +250,7 @@ sleep 2
 if [ -f "${readable_pidfile}" ]; then
     # Try to read the pidfile as a regular user would
     if cat "${readable_pidfile}" > /dev/null 2>&1; then
-        log "✓ PASS: Pidfile is readable"
+        pass "Pidfile is readable"
 
         # Check permissions
         perms=$(stat -c "%a" "${readable_pidfile}" 2>/dev/null || stat -f "%Lp" "${readable_pidfile}" 2>/dev/null)
diff --git a/tests/functional/test_runqueue_parsing.sh b/tests/functional/test_runqueue_parsing.sh
index ccb1982..fc69fa0 100755
--- a/tests/functional/test_runqueue_parsing.sh
+++ b/tests/functional/test_runqueue_parsing.sh
@@ -102,11 +102,11 @@ if [ ${BPF_AVAILABLE} -eq 1 ]; then
 
     # Wait for starvation detection
     if wait_for_starvation_detected "${STALLD_LOG_BPF}"; then
-        log "✓ PASS: eBPF backend detected starving tasks"
+        pass "eBPF backend detected starving tasks"
 
         # Verify task info is present (PID, comm)
         if grep -E "starvation_gen.*starved on CPU ${TEST_CPU}" "${STALLD_LOG_BPF}"; then
-            log "✓ PASS: Task name (comm) correctly extracted"
+            pass "Task name (comm) correctly extracted"
         else
             log "✗ FAIL: Task name not found in eBPF backend output"
             TEST_FAILED=$((TEST_FAILED + 1))
@@ -114,7 +114,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ]; then
 
         # Verify PID is logged
         if grep -E "\[[0-9]+\].*starved on CPU" "${STALLD_LOG_BPF}"; then
-            log "✓ PASS: Task PID correctly extracted"
+            pass "Task PID correctly extracted"
         else
             log "⚠ INFO: PID format may have changed"
         fi
@@ -157,11 +157,11 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
 
     # Wait for starvation detection
     if wait_for_starvation_detected "${STALLD_LOG_SCHED}"; then
-        log "✓ PASS: sched_debug backend detected starving tasks"
+        pass "sched_debug backend detected starving tasks"
 
         # Verify task info is present
         if grep -E "starvation_gen.*starved on CPU ${TEST_CPU}" "${STALLD_LOG_SCHED}"; then
-            log "✓ PASS: Task name (comm) correctly extracted"
+            pass "Task name (comm) correctly extracted"
         else
             log "✗ FAIL: Task name not found in sched_debug backend output"
             TEST_FAILED=$((TEST_FAILED + 1))
@@ -169,7 +169,7 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
 
         # Verify PID is logged
         if grep -E "\[[0-9]+\].*starved on CPU" "${STALLD_LOG_SCHED}"; then
-            log "✓ PASS: Task PID correctly extracted"
+            pass "Task PID correctly extracted"
         else
             log "⚠ INFO: PID format may have changed"
         fi
@@ -249,14 +249,14 @@ if [ ${BPF_AVAILABLE} -eq 1 ] && [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     # Compare results
     log ""
     if [ ${bpf_detections} -gt 0 ] && [ ${sched_detections} -gt 0 ]; then
-        log "✓ PASS: Both backends detected starvation"
+        pass "Both backends detected starvation"
 
         # Check if detection counts are similar (within reasonable variance)
         diff=$((bpf_detections - sched_detections))
         diff=${diff#-}  # absolute value
 
         if [ ${diff} -le 2 ]; then
-            log "✓ PASS: Detection counts are consistent (eBPF: ${bpf_detections}, sched_debug: ${sched_detections})"
+            pass "Detection counts are consistent (eBPF: ${bpf_detections}, sched_debug: ${sched_detections})"
         else
             log "⚠ INFO: Detection counts differ (eBPF: ${bpf_detections}, sched_debug: ${sched_detections})"
             log "        This may be due to timing differences between backends"
@@ -313,7 +313,7 @@ if [ -n "$test_backend" ]; then
 
     # Check for task name (comm field)
     if grep -q "starvation_gen" "${log_file}"; then
-        log "✓ PASS: Task name (comm) field extracted"
+        pass "Task name (comm) field extracted"
     else
         log "✗ FAIL: Task name (comm) field not found"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -321,7 +321,7 @@ if [ -n "$test_backend" ]; then
 
     # Check for PID field (format: name-PID or [PID])
     if grep -qE "(starvation_gen-[0-9]+|\[[0-9]+\])" "${log_file}"; then
-        log "✓ PASS: PID field extracted"
+        pass "PID field extracted"
     else
         log "✗ FAIL: PID field not found"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -329,7 +329,7 @@ if [ -n "$test_backend" ]; then
 
     # Check for CPU ID
     if grep -q "CPU ${TEST_CPU}" "${log_file}"; then
-        log "✓ PASS: CPU ID field extracted"
+        pass "CPU ID field extracted"
     else
         log "✗ FAIL: CPU ID field not found"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -337,7 +337,7 @@ if [ -n "$test_backend" ]; then
 
     # Check for starvation duration
     if grep -qE "for [0-9]+ seconds" "${log_file}"; then
-        log "✓ PASS: Starvation duration calculated from context switches/time"
+        pass "Starvation duration calculated from context switches/time"
     else
         log "✗ FAIL: Starvation duration not found"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -371,21 +371,21 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
     # Check for format detection messages
     if grep -q "detect_task_format" "${STALLD_LOG_SCHED}"; then
         detected_format=$(grep "detect_task_format" "${STALLD_LOG_SCHED}" | grep "detected" | tail -1)
-        log "✓ PASS: Kernel format auto-detection occurred"
+        pass "Kernel format auto-detection occurred"
         log "ℹ INFO: ${detected_format}"
 
         # Check if field offsets were detected
         if grep -q "found 'task' at word" "${STALLD_LOG_SCHED}"; then
-            log "✓ PASS: Task field offset detected"
+            pass "Task field offset detected"
         fi
         if grep -q "found 'PID' at word" "${STALLD_LOG_SCHED}"; then
-            log "✓ PASS: PID field offset detected"
+            pass "PID field offset detected"
         fi
         if grep -q "found 'switches' at word" "${STALLD_LOG_SCHED}"; then
-            log "✓ PASS: Switches field offset detected"
+            pass "Switches field offset detected"
         fi
         if grep -q "found 'prio' at word" "${STALLD_LOG_SCHED}"; then
-            log "✓ PASS: Priority field offset detected"
+            pass "Priority field offset detected"
         fi
     else
         log "⚠ INFO: Format detection messages not in log (may not be verbose enough)"
@@ -393,7 +393,7 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
 
     # Verify the backend still works despite format
     if grep -q "starved on CPU" "${STALLD_LOG_SCHED}"; then
-        log "✓ PASS: Backend successfully parsed tasks despite kernel format"
+        pass "Backend successfully parsed tasks despite kernel format"
     else
         log "⚠ INFO: No starvation detected in this test run"
     fi
diff --git a/tests/functional/test_starvation_detection.sh b/tests/functional/test_starvation_detection.sh
index 3c6fd6e..6d4d7c5 100755
--- a/tests/functional/test_starvation_detection.sh
+++ b/tests/functional/test_starvation_detection.sh
@@ -75,11 +75,11 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${TEST_CPU} -
 # Wait for starvation detection
 log "Waiting for starvation detection..."
 if wait_for_starvation_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Starvation detected"
+    pass "Starvation detected"
 
     # Verify correct CPU is logged
     if grep "starved on CPU ${TEST_CPU}" "${STALLD_LOG}"; then
-        log "✓ PASS: Correct CPU ID logged (CPU ${TEST_CPU})"
+        pass "Correct CPU ID logged (CPU ${TEST_CPU})"
     else
         log "✗ FAIL: Wrong CPU ID in log"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -87,7 +87,7 @@ if wait_for_starvation_detected "${STALLD_LOG}"; then
 
     # Verify duration is logged
     if grep -E "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}"; then
-        log "✓ PASS: Starvation duration logged"
+        pass "Starvation duration logged"
     else
         log "✗ FAIL: Starvation duration not logged"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -145,7 +145,7 @@ if [ -n "${STARVE_CHILDREN}" ]; then
 
             ctxt_delta=$((ctxt_after - ctxt_before))
             if [ ${ctxt_delta} -lt 5 ]; then
-                log "✓ PASS: Context switch count remained low (delta: ${ctxt_delta})"
+                pass "Context switch count remained low (delta: ${ctxt_delta})"
             else
                 log "✗ FAIL: Context switches increased significantly (delta: ${ctxt_delta})"
                 TEST_FAILED=$((TEST_FAILED + 1))
@@ -199,7 +199,7 @@ stop_stalld
 # Task merging means the timestamp is preserved, so duration increases
 report_count=$(grep -cE "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}")
 if [ "${report_count}" -ge 2 ]; then
-    log "✓ PASS: Multiple starvation reports found (${report_count} reports)"
+    pass "Multiple starvation reports found (${report_count} reports)"
 
     # Extract starvation durations from log
     durations=$(grep -oE "starved on CPU ${TEST_CPU} for [0-9]+" "${STALLD_LOG}" | grep -oE "[0-9]+$")
@@ -210,7 +210,7 @@ if [ "${report_count}" -ge 2 ]; then
     last_duration=$(echo "$durations" | tail -1)
 
     if [ ${last_duration} -gt ${first_duration} ]; then
-        log "✓ PASS: Starvation duration increased (${first_duration}s -> ${last_duration}s)"
+        pass "Starvation duration increased (${first_duration}s -> ${last_duration}s)"
         log "        This confirms task merging preserved the timestamp"
     else
         log "✗ FAIL: Starvation duration did not increase (timestamp may have been reset)"
@@ -283,14 +283,14 @@ else
 
     # Check both CPUs detected - specifically look for starvation_gen tasks
     if grep -qE "starvation_gen.*starved on CPU ${CPU0}|starved on CPU ${CPU0}.*starvation_gen" "${STALLD_LOG}"; then
-        log "✓ PASS: Starvation detected on CPU ${CPU0}"
+        pass "Starvation detected on CPU ${CPU0}"
     else
         log "✗ FAIL: Starvation not detected on CPU ${CPU0}"
         TEST_FAILED=$((TEST_FAILED + 1))
     fi
 
     if grep -qE "starvation_gen.*starved on CPU ${CPU1}|starved on CPU ${CPU1}.*starvation_gen" "${STALLD_LOG}"; then
-        log "✓ PASS: Starvation detected on CPU ${CPU1}"
+        pass "Starvation detected on CPU ${CPU1}"
     else
         log "✗ FAIL: Starvation not detected on CPU ${CPU1}"
         TEST_FAILED=$((TEST_FAILED + 1))
@@ -330,7 +330,7 @@ log "Creating a busy task that should NOT be starved"
     # Verify this task was NOT reported as starved
     # Since it's making progress, stalld shouldn't detect it
     if ! grep "starved" "${STALLD_LOG}"; then
-        log "✓ PASS: No false positive - task making progress not reported as starved"
+        pass "No false positive - task making progress not reported as starved"
     else
         # Check if our specific task was reported
         log "Log shows starvation, checking if it's our progress-making task..."
@@ -370,7 +370,7 @@ sleep 5
 
 # Verify stalld is still running (didn't crash)
 if assert_process_running "${STALLD_PID}" "stalld still running after task exit"; then
-    log "✓ PASS: stalld handled task exit gracefully"
+    pass "stalld handled task exit gracefully"
 else
     log "✗ FAIL: stalld crashed or exited unexpectedly"
     TEST_FAILED=$((TEST_FAILED + 1))
@@ -383,7 +383,7 @@ if grep -iE "error|segfault|crash" "${STALLD_LOG}"; then
     grep -iE "error|segfault|crash" "${STALLD_LOG}"
     TEST_FAILED=$((TEST_FAILED + 1))
 else
-    log "✓ PASS: No error messages in log"
+    pass "No error messages in log"
 fi
 
 stop_stalld
diff --git a/tests/functional/test_starvation_threshold.sh b/tests/functional/test_starvation_threshold.sh
index b904867..a35f898 100755
--- a/tests/functional/test_starvation_threshold.sh
+++ b/tests/functional/test_starvation_threshold.sh
@@ -76,7 +76,7 @@ log "Waiting for detection (threshold: ${threshold}s)"
 
 # Check if starvation was detected - specifically look for starvation_gen tasks
 if wait_for_starvation_detected "${STALLD_LOG}"; then
-    log "✓ PASS: Starvation detected after ${threshold}s threshold"
+    pass "Starvation detected after ${threshold}s threshold"
 else
     log "✗ FAIL: Starvation not detected after ${threshold}s threshold"
     log "Log contents:"
@@ -121,7 +121,7 @@ sleep 2
 
 # Check that starvation_gen was NOT detected (duration less than threshold)
 if ! grep -qE "starvation_gen.*starved on CPU ${TEST_CPU}|starved on CPU ${TEST_CPU}.*starvation_gen" "${STALLD_LOG2}"; then
-    log "✓ PASS: No starvation detected for duration less than threshold"
+    pass "No starvation detected for duration less than threshold"
 else
     log "✗ FAIL: Starvation detected before threshold"
     log "Found starvation_gen task in logs:"
@@ -161,7 +161,7 @@ log "Waiting for detection (threshold: ${threshold}s)"
 
 # Check if starvation_gen was detected
 if wait_for_starvation_detected "${STALLD_LOG3}"; then
-    log "✓ PASS: Starvation detected with ${threshold}s threshold"
+    pass "Starvation detected with ${threshold}s threshold"
 else
     log "✗ FAIL: Starvation not detected with ${threshold}s threshold"
     log "Log contents:"
diff --git a/tests/functional/test_task_merging.sh b/tests/functional/test_task_merging.sh
index 5cd6406..b89230e 100755
--- a/tests/functional/test_task_merging.sh
+++ b/tests/functional/test_task_merging.sh
@@ -103,7 +103,7 @@ log "Second detection: task starved for ${second_duration}s"
 # Verify timestamp was preserved (duration increased)
 if [ "${second_duration}" -gt "${first_duration}" ]; then
     delta=$((second_duration - first_duration))
-    log "✓ PASS: Starvation duration increased by ${delta}s"
+    pass "Starvation duration increased by ${delta}s"
     log "        Timestamp preserved across monitoring cycles"
 else
     log "✗ FAIL: Duration did not increase (${first_duration}s -> ${second_duration}s)"
@@ -122,7 +122,7 @@ fi
 log "Third detection: task starved for ${third_duration}s"
 
 if [ "${third_duration}" -gt "${second_duration}" ]; then
-    log "✓ PASS: Duration continues to accumulate (${third_duration}s total)"
+    pass "Duration continues to accumulate (${third_duration}s total)"
 else
     log "⚠ INFO: Duration did not increase in third cycle"
 fi
@@ -179,7 +179,7 @@ if [ -n "${tracked_pid}" ]; then
 
         delta=$((ctxsw_after - ctxsw_before))
         if [ ${delta} -lt 5 ]; then
-            log "✓ PASS: Context switches remained low (delta: ${delta})"
+            pass "Context switches remained low (delta: ${delta})"
             log "        Task meeting merge criteria (same PID, same ctxsw)"
         else
             log "⚠ INFO: Context switches increased by ${delta}"
@@ -191,7 +191,7 @@ if [ -n "${tracked_pid}" ]; then
         log "Total starvation detections: ${detections}"
 
         if [ ${detections} -ge 2 ]; then
-            log "✓ PASS: Multiple detections indicate task merging across cycles"
+            pass "Multiple detections indicate task merging across cycles"
         fi
     else
         log "⚠ WARNING: Task exited before second check"
@@ -242,7 +242,7 @@ if [ -n "${tracked_pid}" ]; then
     # If task was boosted, context switches should have changed
     # meaning timestamp should reset for next starvation period
     if grep -q "boosted" "${STALLD_LOG}"; then
-        log "✓ PASS: Task was boosted (made progress)"
+        pass "Task was boosted (made progress)"
 
         # Check if we see a new starvation period starting
         # (This is harder to verify, but context switches changing = no merge)
@@ -311,7 +311,7 @@ else
         log "CPU ${CPU0}: ${cpu0_first}s -> ${cpu0_last}s"
 
         if [ "${cpu0_last}" -gt "${cpu0_first}" ]; then
-            log "✓ PASS: CPU ${CPU0} task merging working (timestamp preserved)"
+            pass "CPU ${CPU0} task merging working (timestamp preserved)"
         fi
     fi
 
@@ -327,12 +327,12 @@ else
         log "CPU ${CPU1}: ${cpu1_first}s -> ${cpu1_last}s"
 
         if [ "${cpu1_last}" -gt "${cpu1_first}" ]; then
-            log "✓ PASS: CPU ${CPU1} task merging working (timestamp preserved)"
+            pass "CPU ${CPU1} task merging working (timestamp preserved)"
         fi
     fi
 
     if [ ${cpu0_detections} -ge 2 ] && [ ${cpu1_detections} -ge 2 ]; then
-        log "✓ PASS: Independent task merging on both CPUs"
+        pass "Independent task merging on both CPUs"
     fi
 
     # Cleanup
-- 
2.53.0

[PATCH stalld 30/36] tests: Add fail() helper and use for all test failures

[Wander Lairson Costa]

Add a fail() function that mirrors pass(), logging a failure
message and incrementing the failure counter. Replace all manual
fail patterns across the test suite, including both the log-based
and echo-based variants.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_affinity.sh             |  6 ++--
 tests/functional/test_backend_selection.sh    |  6 ++--
 tests/functional/test_boost_duration.sh       | 18 ++++-------
 tests/functional/test_boost_period.sh         | 18 ++++-------
 tests/functional/test_boost_restoration.sh    |  3 +-
 tests/functional/test_boost_runtime.sh        | 21 +++++--------
 tests/functional/test_cpu_selection.sh        | 18 ++++-------
 tests/functional/test_deadline_boosting.sh    | 12 +++-----
 tests/functional/test_fifo_boosting.sh        | 12 +++-----
 .../test_fifo_priority_starvation.sh          | 15 ++++------
 tests/functional/test_force_fifo.sh           |  6 ++--
 tests/functional/test_foreground.sh           |  9 ++----
 tests/functional/test_idle_detection.sh       |  3 +-
 tests/functional/test_log_only.sh             |  6 ++--
 tests/functional/test_logging_destinations.sh |  6 ++--
 tests/functional/test_pidfile.sh              | 27 ++++++-----------
 tests/functional/test_runqueue_parsing.sh     | 27 ++++++-----------
 tests/functional/test_starvation_detection.sh | 30 +++++++------------
 tests/functional/test_starvation_threshold.sh | 15 ++++------
 tests/functional/test_task_merging.sh         |  3 +-
 tests/helpers/test_helpers.sh                 | 11 ++++++-
 21 files changed, 97 insertions(+), 175 deletions(-)

diff --git a/tests/functional/test_affinity.sh b/tests/functional/test_affinity.sh
index b74886d..7ecd85f 100755
--- a/tests/functional/test_affinity.sh
+++ b/tests/functional/test_affinity.sh
@@ -109,8 +109,7 @@ affinity=$(check_affinity "${STALLD_PID}")
 if [ "$affinity" = "0" ]; then
     pass "stalld restricted to CPU 0"
 else
-    log "✗ FAIL: stalld affinity ($affinity) doesn't match requested (0)"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld affinity ($affinity) doesn't match requested (0)"
 fi
 
 stop_stalld
@@ -270,8 +269,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Invalid CPU affinity rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid CPU affinity"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid CPU affinity"
 fi
 
 #=============================================================================
diff --git a/tests/functional/test_backend_selection.sh b/tests/functional/test_backend_selection.sh
index 8a22e0a..2db8353 100755
--- a/tests/functional/test_backend_selection.sh
+++ b/tests/functional/test_backend_selection.sh
@@ -39,8 +39,7 @@ test_backend_flag() {
 	CLEANUP_PIDS+=("${STALLD_PID}")
 
 	if ! wait_for_stalld_ready "${log_file}" 15; then
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: stalld failed to start (${description})"
+		fail "stalld failed to start (${description})"
 		stop_stalld
 		return 1
 	fi
@@ -48,8 +47,7 @@ test_backend_flag() {
 	if grep -q "${expected_msg}" "${log_file}"; then
 		pass "${description}"
 	else
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: Backend message not found (${description})"
+		fail "Backend message not found (${description})"
 		echo "  Expected: ${expected_msg}"
 		echo "  Log contents:"
 		cat "${log_file}"
diff --git a/tests/functional/test_boost_duration.sh b/tests/functional/test_boost_duration.sh
index 61218e9..83d8355 100755
--- a/tests/functional/test_boost_duration.sh
+++ b/tests/functional/test_boost_duration.sh
@@ -69,8 +69,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_starvation_detected "${STALLD_LOG}"; then
     pass "Starvation detection occurred with default duration"
 else
-    log "✗ FAIL: No starvation detection"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection"
 fi
 
 # Cleanup
@@ -101,8 +100,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_starvation_detected "${STALLD_LOG2}"; then
     pass "Starvation detection with ${short_duration}s duration"
 else
-    log "✗ FAIL: No starvation detection with short duration"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection with short duration"
 fi
 
 # Cleanup
@@ -135,8 +133,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${long_starvation}
 if wait_for_starvation_detected "${STALLD_LOG3}"; then
     pass "Starvation detection with ${long_duration}s duration"
 else
-    log "✗ FAIL: No starvation detection with long duration"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection with long duration"
 fi
 
 # Cleanup
@@ -168,8 +165,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 1 -d 15
 if wait_for_starvation_detected "${STALLD_LOG4}"; then
     pass "Starvation detection with ${duration}s boost duration"
 else
-    log "✗ FAIL: No starvation detection"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection"
 fi
 
 # Cleanup
@@ -202,8 +198,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Zero duration rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid duration value 0"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid duration value 0"
 fi
 
 # Test 6: Negative duration
@@ -217,8 +212,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Negative duration rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid negative duration"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid negative duration"
 fi
 
 log ""
diff --git a/tests/functional/test_boost_period.sh b/tests/functional/test_boost_period.sh
index 671f7cb..3f3ef46 100755
--- a/tests/functional/test_boost_period.sh
+++ b/tests/functional/test_boost_period.sh
@@ -71,10 +71,9 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
         log "ℹ INFO: Period information found in logs"
     fi
 else
-    log "✗ FAIL: No boosting detected"
+    fail "No boosting detected"
     log "Log contents:"
     cat "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -103,8 +102,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_boost_detected "${STALLD_LOG}"; then
     pass "Boosting occurred with custom period ${custom_period} ns"
 else
-    log "✗ FAIL: No boosting with custom period"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No boosting with custom period"
 fi
 
 # Cleanup
@@ -133,8 +131,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_boost_detected "${STALLD_LOG}"; then
     pass "Boosting occurred with short period ${short_period} ns"
 else
-    log "✗ FAIL: No boosting with short period"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No boosting with short period"
 fi
 
 # Cleanup
@@ -163,8 +160,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_boost_detected "${STALLD_LOG}"; then
     pass "Boosting occurred with long period ${long_period} ns"
 else
-    log "✗ FAIL: No boosting with long period"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No boosting with long period"
 fi
 
 # Cleanup
@@ -195,8 +191,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Zero period rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid period value 0"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid period value 0"
 fi
 
 #=============================================================================
@@ -216,8 +211,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Negative period rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid negative period"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid negative period"
 fi
 
 #=============================================================================
diff --git a/tests/functional/test_boost_restoration.sh b/tests/functional/test_boost_restoration.sh
index 024d869..bac73ba 100755
--- a/tests/functional/test_boost_restoration.sh
+++ b/tests/functional/test_boost_restoration.sh
@@ -391,8 +391,7 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
     if assert_process_running "${STALLD_PID}" "stalld still running after task exit"; then
         pass "stalld handled task exit during boost gracefully"
     else
-        log "✗ FAIL: stalld crashed or exited after task died during boost"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "stalld crashed or exited after task died during boost"
     fi
 
     # Check for error messages
diff --git a/tests/functional/test_boost_runtime.sh b/tests/functional/test_boost_runtime.sh
index b624fc9..71e25f1 100755
--- a/tests/functional/test_boost_runtime.sh
+++ b/tests/functional/test_boost_runtime.sh
@@ -69,8 +69,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_starvation_detected "${STALLD_LOG}"; then
     pass "Starvation detection with default runtime"
 else
-    log "✗ FAIL: No starvation detection"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection"
 fi
 
 # Cleanup
@@ -101,8 +100,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_starvation_detected "${STALLD_LOG2}"; then
     pass "Starvation detection with custom runtime ${custom_runtime}ns"
 else
-    log "✗ FAIL: No starvation detection with custom runtime"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection with custom runtime"
 fi
 
 # Cleanup
@@ -133,8 +131,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_starvation_detected "${STALLD_LOG3}"; then
     pass "Starvation detection with large runtime ${large_runtime}ns"
 else
-    log "✗ FAIL: No starvation detection with large runtime"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection with large runtime"
 fi
 
 # Cleanup
@@ -167,8 +164,7 @@ start_starvation_gen -c "${TEST_CPU}" -p 80 -n 2 -d ${starvation_duration}
 if wait_for_starvation_detected "${STALLD_LOG4}"; then
     pass "Starvation detection with runtime < period"
 else
-    log "✗ FAIL: No starvation detection when runtime < period"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No starvation detection when runtime < period"
 fi
 
 # Cleanup
@@ -202,8 +198,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Runtime > period rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid runtime > period"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid runtime > period"
 fi
 
 #=============================================================================
@@ -224,8 +219,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Zero runtime rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid runtime value 0"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid runtime value 0"
 fi
 
 #=============================================================================
@@ -246,8 +240,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Negative runtime rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid negative runtime"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid negative runtime"
 fi
 
 log ""
diff --git a/tests/functional/test_cpu_selection.sh b/tests/functional/test_cpu_selection.sh
index 2f9875d..99eac44 100755
--- a/tests/functional/test_cpu_selection.sh
+++ b/tests/functional/test_cpu_selection.sh
@@ -44,8 +44,7 @@ start_stalld_with_log "${STALLD_LOG}" -f -v -c 0 -l -t 5
 if grep -q "cpu 0" "$STALLD_LOG"; then
     pass "stalld monitoring CPU 0"
 else
-    TEST_FAILED=$((TEST_FAILED + 1))
-    echo -e "  ${RED}FAIL${NC}: stalld not monitoring CPU 0"
+    fail "stalld not monitoring CPU 0"
 fi
 
 stop_stalld
@@ -70,8 +69,7 @@ if [ "$num_cpus" -ge 4 ]; then
     if [ "$cpu0_found" -eq 1 ] && [ "$cpu2_found" -eq 1 ]; then
         pass "stalld monitoring CPUs 0 and 2"
     else
-        TEST_FAILED=$((TEST_FAILED + 1))
-        echo -e "  ${RED}FAIL${NC}: stalld not monitoring specified CPUs (0: $cpu0_found, 2: $cpu2_found)"
+        fail "stalld not monitoring specified CPUs (0: $cpu0_found, 2: $cpu2_found)"
     fi
 
     stop_stalld
@@ -103,8 +101,7 @@ if [ "$num_cpus" -ge 4 ]; then
     if [ "$cpu0_found" -eq 1 ] && [ "$cpu1_found" -eq 1 ] && [ "$cpu2_found" -eq 1 ]; then
         pass "stalld monitoring CPUs 0-2"
     else
-        TEST_FAILED=$((TEST_FAILED + 1))
-        echo -e "  ${RED}FAIL${NC}: stalld not monitoring specified CPU range (0: $cpu0_found, 1: $cpu1_found, 2: $cpu2_found)"
+        fail "stalld not monitoring specified CPU range (0: $cpu0_found, 1: $cpu1_found, 2: $cpu2_found)"
     fi
 
     stop_stalld
@@ -130,8 +127,7 @@ if [ "$num_cpus" -ge 6 ]; then
     if [ "$monitored_cpus" -eq 4 ]; then
         pass "stalld monitoring combined CPU specification (0,2-4)"
     else
-        TEST_FAILED=$((TEST_FAILED + 1))
-        echo -e "  ${RED}FAIL${NC}: stalld not monitoring all specified CPUs (found $monitored_cpus/4)"
+        fail "stalld not monitoring all specified CPUs (found $monitored_cpus/4)"
     fi
 
     stop_stalld
@@ -155,8 +151,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "stalld rejected invalid CPU number"
 else
-    log "✗ FAIL: stalld did not reject invalid CPU"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid CPU"
 fi
 
 # Test 6: Verify non-selected CPUs are NOT monitored
@@ -170,8 +165,7 @@ if [ "$num_cpus" -ge 2 ]; then
     if ! grep -q "cpu 1" "$STALLD_LOG" || grep -q "not monitoring.*cpu 1" "$STALLD_LOG"; then
         pass "stalld not monitoring non-selected CPU 1"
     else
-        TEST_FAILED=$((TEST_FAILED + 1))
-        echo -e "  ${RED}FAIL${NC}: stalld appears to be monitoring CPU 1 when only CPU 0 selected"
+        fail "stalld appears to be monitoring CPU 1 when only CPU 0 selected"
     fi
 
     stop_stalld
diff --git a/tests/functional/test_deadline_boosting.sh b/tests/functional/test_deadline_boosting.sh
index 86ca0f1..544222a 100755
--- a/tests/functional/test_deadline_boosting.sh
+++ b/tests/functional/test_deadline_boosting.sh
@@ -74,8 +74,7 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
     if grep -q "SCHED_DEADLINE" "${STALLD_LOG}"; then
         pass "SCHED_DEADLINE boosting used (default)"
     else
-        log "✗ FAIL: SCHED_DEADLINE not mentioned in boost message"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "SCHED_DEADLINE not mentioned in boost message"
     fi
 
     # Verify boost happened after threshold
@@ -86,10 +85,9 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
         log "⚠ WARNING: No starvation message before boost"
     fi
 else
-    log "✗ FAIL: No boosting detected"
+    fail "No boosting detected"
     log "Log contents:"
     cat "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -153,8 +151,7 @@ if [ ${boosted_task_found} -eq 0 ]; then
     if grep -q "boosted.*SCHED_DEADLINE" "${STALLD_LOG}"; then
         pass "SCHED_DEADLINE boost confirmed in logs"
     else
-        log "✗ FAIL: No SCHED_DEADLINE boost detected"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "No SCHED_DEADLINE boost detected"
     fi
 fi
 
@@ -226,8 +223,7 @@ fi
 if grep -q "boosted" "${STALLD_LOG}"; then
     pass "Boost occurred as expected"
 else
-    log "✗ FAIL: No boost detected"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "No boost detected"
 fi
 
 # Cleanup
diff --git a/tests/functional/test_fifo_boosting.sh b/tests/functional/test_fifo_boosting.sh
index df26483..50c8d14 100755
--- a/tests/functional/test_fifo_boosting.sh
+++ b/tests/functional/test_fifo_boosting.sh
@@ -72,14 +72,12 @@ if wait_for_boost_detected "${STALLD_LOG}"; then
     if grep -q "SCHED_FIFO" "${STALLD_LOG}"; then
         pass "SCHED_FIFO boosting used (as requested by -F)"
     else
-        log "✗ FAIL: SCHED_FIFO not mentioned in boost message"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "SCHED_FIFO not mentioned in boost message"
     fi
 else
-    log "✗ FAIL: No boosting detected with -F flag"
+    fail "No boosting detected with -F flag"
     log "Log contents:"
     cat "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -134,8 +132,7 @@ if [ ${fifo_task_found} -eq 0 ]; then
     if grep -q "boosted.*SCHED_FIFO" "${STALLD_LOG}"; then
         pass "SCHED_FIFO boost confirmed in logs"
     else
-        log "✗ FAIL: No SCHED_FIFO boost detected"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "No SCHED_FIFO boost detected"
     fi
 fi
 
@@ -337,8 +334,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "stalld exited as expected"
 else
-    log "✗ FAIL: stalld did not reject FIFO in single-threaded mode"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject FIFO in single-threaded mode"
 fi
 
 # Check for error message in log
diff --git a/tests/functional/test_fifo_priority_starvation.sh b/tests/functional/test_fifo_priority_starvation.sh
index 3776e6b..9554d49 100755
--- a/tests/functional/test_fifo_priority_starvation.sh
+++ b/tests/functional/test_fifo_priority_starvation.sh
@@ -78,22 +78,19 @@ if wait_for_starvation_detected "${STALLD_LOG}"; then
     if grep "starved on CPU ${TEST_CPU}" "${STALLD_LOG}"; then
         pass "Correct CPU ID logged (CPU ${TEST_CPU})"
     else
-        log "✗ FAIL: Wrong CPU ID in log"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Wrong CPU ID in log"
     fi
 
     # Verify duration is logged
     if grep -E "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}"; then
         pass "Starvation duration logged"
     else
-        log "✗ FAIL: Starvation duration not logged"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Starvation duration not logged"
     fi
 else
-    log "✗ FAIL: FIFO-on-FIFO starvation not detected"
+    fail "FIFO-on-FIFO starvation not detected"
     log "Log contents:"
     cat "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -159,8 +156,7 @@ else
     if grep -q "boosted" "${STALLD_LOG}"; then
         log "ℹ INFO: Boosting did occur according to logs"
     else
-        log "✗ FAIL: No boosting detected"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "No boosting detected"
     fi
 fi
 
@@ -218,8 +214,7 @@ if grep -E "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}" | wc
         pass "Starvation duration increased (${first_duration}s -> ${last_duration}s)"
         log "        This confirms task merging preserved the timestamp"
     else
-        log "✗ FAIL: Starvation duration did not increase (timestamp may have been reset)"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Starvation duration did not increase (timestamp may have been reset)"
     fi
 else
     log "⚠ WARNING: Not enough starvation reports to verify task merging"
diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index fec2e00..ecaa7ac 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -107,8 +107,7 @@ if wait_for_boost_detected "${STALLD_LOG2}"; then
     if grep -qi "fifo\|SCHED_FIFO" "${STALLD_LOG2}"; then
         pass "SCHED_FIFO used with -F flag"
     elif grep -qi "deadline\|SCHED_DEADLINE" "${STALLD_LOG2}"; then
-        log "✗ FAIL: SCHED_DEADLINE used despite -F flag"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "SCHED_DEADLINE used despite -F flag"
     else
         log "⚠ WARNING: Scheduling policy not explicitly mentioned in logs"
     fi
@@ -220,8 +219,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "single-threaded mode rejected FIFO"
 else
-    log "✗ FAIL: stalld did not reject -F in single-threaded mode"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject -F in single-threaded mode"
 fi
 
 #=============================================================================
diff --git a/tests/functional/test_foreground.sh b/tests/functional/test_foreground.sh
index 2bb0282..8b89d02 100755
--- a/tests/functional/test_foreground.sh
+++ b/tests/functional/test_foreground.sh
@@ -38,8 +38,7 @@ if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 		if [ "${PARENT_PID}" != "$$" ]; then
 			pass "stalld daemonized (parent is not test shell)"
 		else
-			TEST_FAILED=$((TEST_FAILED + 1))
-			echo -e "  ${RED}FAIL${NC}: stalld did not daemonize (parent is test shell)"
+			fail "stalld did not daemonize (parent is test shell)"
 		fi
 	fi
 fi
@@ -64,8 +63,7 @@ if assert_process_running "${STALLD_PID}" "stalld should be running with -f"; th
 	if [ "${PARENT_PID}" != "1" ]; then
 		pass "stalld did not daemonize with -f (parent is not init)"
 	else
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: stalld daemonized even with -f flag"
+		fail "stalld daemonized even with -f flag"
 	fi
 fi
 
@@ -84,8 +82,7 @@ if assert_process_running "${STALLD_PID}" "stalld should be running with -v"; th
 	if [ "${PARENT_PID}" != "1" ]; then
 		pass "-v implies foreground mode"
 	else
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: -v should imply foreground mode"
+		fail "-v should imply foreground mode"
 	fi
 fi
 
diff --git a/tests/functional/test_idle_detection.sh b/tests/functional/test_idle_detection.sh
index af4d7a4..8757d74 100755
--- a/tests/functional/test_idle_detection.sh
+++ b/tests/functional/test_idle_detection.sh
@@ -137,8 +137,7 @@ if [ -n "${idle_time1}" ] && [ -n "${idle_time2}" ]; then
         log "        stalld would parse this CPU"
     fi
 else
-    log "✗ FAIL: Could not read idle time from /proc/stat"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "Could not read idle time from /proc/stat"
 fi
 
 #=============================================================================
diff --git a/tests/functional/test_log_only.sh b/tests/functional/test_log_only.sh
index d6257cf..21518a4 100755
--- a/tests/functional/test_log_only.sh
+++ b/tests/functional/test_log_only.sh
@@ -61,8 +61,7 @@ echo "Waiting for starvation detection..."
 if wait_for_starvation_detected "${LOG_FILE}"; then
 	pass "stalld detected and logged starvation"
 else
-	TEST_FAILED=$((TEST_FAILED + 1))
-	echo -e "  ${RED}FAIL${NC}: stalld did not detect starvation"
+	fail "stalld did not detect starvation"
 	echo "Log contents:"
 	cat "${LOG_FILE}"
 fi
@@ -71,8 +70,7 @@ fi
 if ! grep -q "boosted" "${LOG_FILE}"; then
 	pass "stalld did not boost in log-only mode"
 else
-	TEST_FAILED=$((TEST_FAILED + 1))
-	echo -e "  ${RED}FAIL${NC}: stalld boosted despite -l flag"
+	fail "stalld boosted despite -l flag"
 	echo "Log contents:"
 	cat "${LOG_FILE}"
 fi
diff --git a/tests/functional/test_logging_destinations.sh b/tests/functional/test_logging_destinations.sh
index 35470c0..f05cd00 100755
--- a/tests/functional/test_logging_destinations.sh
+++ b/tests/functional/test_logging_destinations.sh
@@ -45,8 +45,7 @@ if assert_process_running "${STALLD_PID}" "stalld should be running"; then
 			pass "output contains expected messages"
 		fi
 	else
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: no output in verbose mode"
+		fail "no output in verbose mode"
 	fi
 fi
 
@@ -158,8 +157,7 @@ if assert_process_running "${STALLD_PID}" "stalld with combined logging should b
 	if [ -s "${LOG_FILE}" ]; then
 		pass "combined logging produces output"
 	else
-		TEST_FAILED=$((TEST_FAILED + 1))
-		echo -e "  ${RED}FAIL${NC}: no output with combined logging"
+		fail "no output with combined logging"
 	fi
 fi
 
diff --git a/tests/functional/test_pidfile.sh b/tests/functional/test_pidfile.sh
index fd4fedc..0e4c623 100755
--- a/tests/functional/test_pidfile.sh
+++ b/tests/functional/test_pidfile.sh
@@ -56,8 +56,7 @@ for pidfile in /var/run/stalld.pid /run/stalld.pid; do
         if [ "$pid_from_file" = "${STALLD_PID}" ]; then
             pass "Default pidfile contains correct PID"
         else
-            log "✗ FAIL: Default pidfile PID ($pid_from_file) doesn't match stalld PID (${STALLD_PID})"
-            TEST_FAILED=$((TEST_FAILED + 1))
+            fail "Default pidfile PID ($pid_from_file) doesn't match stalld PID (${STALLD_PID})"
         fi
         break
     fi
@@ -99,12 +98,10 @@ if [ -f "${custom_pidfile}" ]; then
     if [ "$pid_from_file" = "${STALLD_PID}" ]; then
         pass "Custom pidfile contains correct PID ($pid_from_file)"
     else
-        log "✗ FAIL: Custom pidfile PID ($pid_from_file) doesn't match stalld PID (${STALLD_PID})"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Custom pidfile PID ($pid_from_file) doesn't match stalld PID (${STALLD_PID})"
     fi
 else
-    log "✗ FAIL: Custom pidfile not created at ${custom_pidfile}"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "Custom pidfile not created at ${custom_pidfile}"
 fi
 
 # Test 3: Verify pidfile removed on clean shutdown
@@ -145,12 +142,10 @@ if [ -f "${tmp_pidfile}" ]; then
     if [ "$pid_from_file" = "${STALLD_PID}" ]; then
         pass "/tmp pidfile contains correct PID"
     else
-        log "✗ FAIL: /tmp pidfile has incorrect PID"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "/tmp pidfile has incorrect PID"
     fi
 else
-    log "✗ FAIL: Pidfile not created in /tmp"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "Pidfile not created in /tmp"
 fi
 
 stop_stalld
@@ -181,8 +176,7 @@ if [ -f "${fg_pidfile}" ]; then
     if [ "$pid_from_file" = "${STALLD_PID}" ]; then
         pass "Foreground mode pidfile contains correct PID"
     else
-        log "✗ FAIL: Foreground mode pidfile has incorrect PID"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Foreground mode pidfile has incorrect PID"
     fi
 else
     log "⚠ WARNING: Pidfile not created in foreground mode (may be expected)"
@@ -221,8 +215,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Invalid pidfile path rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid pidfile path"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid pidfile path"
 fi
 
 # Cleanup
@@ -256,12 +249,10 @@ if [ -f "${readable_pidfile}" ]; then
         perms=$(stat -c "%a" "${readable_pidfile}" 2>/dev/null || stat -f "%Lp" "${readable_pidfile}" 2>/dev/null)
         log "ℹ INFO: Pidfile permissions: $perms"
     else
-        log "✗ FAIL: Pidfile not readable"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Pidfile not readable"
     fi
 else
-    log "✗ FAIL: Pidfile not created"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "Pidfile not created"
 fi
 
 stop_stalld
diff --git a/tests/functional/test_runqueue_parsing.sh b/tests/functional/test_runqueue_parsing.sh
index fc69fa0..ac5190e 100755
--- a/tests/functional/test_runqueue_parsing.sh
+++ b/tests/functional/test_runqueue_parsing.sh
@@ -108,8 +108,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ]; then
         if grep -E "starvation_gen.*starved on CPU ${TEST_CPU}" "${STALLD_LOG_BPF}"; then
             pass "Task name (comm) correctly extracted"
         else
-            log "✗ FAIL: Task name not found in eBPF backend output"
-            TEST_FAILED=$((TEST_FAILED + 1))
+            fail "Task name not found in eBPF backend output"
         fi
 
         # Verify PID is logged
@@ -119,10 +118,9 @@ if [ ${BPF_AVAILABLE} -eq 1 ]; then
             log "⚠ INFO: PID format may have changed"
         fi
     else
-        log "✗ FAIL: eBPF backend did not detect starvation"
+        fail "eBPF backend did not detect starvation"
         log "Log contents:"
         cat "${STALLD_LOG_BPF}"
-        TEST_FAILED=$((TEST_FAILED + 1))
     fi
 
     # Cleanup
@@ -163,8 +161,7 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
         if grep -E "starvation_gen.*starved on CPU ${TEST_CPU}" "${STALLD_LOG_SCHED}"; then
             pass "Task name (comm) correctly extracted"
         else
-            log "✗ FAIL: Task name not found in sched_debug backend output"
-            TEST_FAILED=$((TEST_FAILED + 1))
+            fail "Task name not found in sched_debug backend output"
         fi
 
         # Verify PID is logged
@@ -180,10 +177,9 @@ if [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
             log "ℹ INFO: Kernel format detected: $format"
         fi
     else
-        log "✗ FAIL: sched_debug backend did not detect starvation"
+        fail "sched_debug backend did not detect starvation"
         log "Log contents:"
         cat "${STALLD_LOG_SCHED}"
-        TEST_FAILED=$((TEST_FAILED + 1))
     fi
 
     # Cleanup
@@ -262,8 +258,7 @@ if [ ${BPF_AVAILABLE} -eq 1 ] && [ ${SCHED_DEBUG_AVAILABLE} -eq 1 ]; then
             log "        This may be due to timing differences between backends"
         fi
     else
-        log "✗ FAIL: One or both backends failed to detect starvation"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "One or both backends failed to detect starvation"
     fi
 else
     log ""
@@ -315,32 +310,28 @@ if [ -n "$test_backend" ]; then
     if grep -q "starvation_gen" "${log_file}"; then
         pass "Task name (comm) field extracted"
     else
-        log "✗ FAIL: Task name (comm) field not found"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Task name (comm) field not found"
     fi
 
     # Check for PID field (format: name-PID or [PID])
     if grep -qE "(starvation_gen-[0-9]+|\[[0-9]+\])" "${log_file}"; then
         pass "PID field extracted"
     else
-        log "✗ FAIL: PID field not found"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "PID field not found"
     fi
 
     # Check for CPU ID
     if grep -q "CPU ${TEST_CPU}" "${log_file}"; then
         pass "CPU ID field extracted"
     else
-        log "✗ FAIL: CPU ID field not found"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "CPU ID field not found"
     fi
 
     # Check for starvation duration
     if grep -qE "for [0-9]+ seconds" "${log_file}"; then
         pass "Starvation duration calculated from context switches/time"
     else
-        log "✗ FAIL: Starvation duration not found"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Starvation duration not found"
     fi
 
     # Cleanup
diff --git a/tests/functional/test_starvation_detection.sh b/tests/functional/test_starvation_detection.sh
index 6d4d7c5..680cf39 100755
--- a/tests/functional/test_starvation_detection.sh
+++ b/tests/functional/test_starvation_detection.sh
@@ -81,22 +81,19 @@ if wait_for_starvation_detected "${STALLD_LOG}"; then
     if grep "starved on CPU ${TEST_CPU}" "${STALLD_LOG}"; then
         pass "Correct CPU ID logged (CPU ${TEST_CPU})"
     else
-        log "✗ FAIL: Wrong CPU ID in log"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Wrong CPU ID in log"
     fi
 
     # Verify duration is logged
     if grep -E "starved on CPU ${TEST_CPU} for [0-9]+ seconds" "${STALLD_LOG}"; then
         pass "Starvation duration logged"
     else
-        log "✗ FAIL: Starvation duration not logged"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Starvation duration not logged"
     fi
 else
-    log "✗ FAIL: Starvation not detected"
+    fail "Starvation not detected"
     log "Log contents:"
     cat "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -147,8 +144,7 @@ if [ -n "${STARVE_CHILDREN}" ]; then
             if [ ${ctxt_delta} -lt 5 ]; then
                 pass "Context switch count remained low (delta: ${ctxt_delta})"
             else
-                log "✗ FAIL: Context switches increased significantly (delta: ${ctxt_delta})"
-                TEST_FAILED=$((TEST_FAILED + 1))
+                fail "Context switches increased significantly (delta: ${ctxt_delta})"
             fi
             break
         fi
@@ -213,14 +209,12 @@ if [ "${report_count}" -ge 2 ]; then
         pass "Starvation duration increased (${first_duration}s -> ${last_duration}s)"
         log "        This confirms task merging preserved the timestamp"
     else
-        log "✗ FAIL: Starvation duration did not increase (timestamp may have been reset)"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Starvation duration did not increase (timestamp may have been reset)"
     fi
 else
-    log "✗ FAIL: Not enough starvation reports to verify task merging (found ${report_count}, need >= 2)"
+    fail "Not enough starvation reports to verify task merging (found ${report_count}, need >= 2)"
     log "Log contents:"
     cat "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup starvation generator
@@ -285,15 +279,13 @@ else
     if grep -qE "starvation_gen.*starved on CPU ${CPU0}|starved on CPU ${CPU0}.*starvation_gen" "${STALLD_LOG}"; then
         pass "Starvation detected on CPU ${CPU0}"
     else
-        log "✗ FAIL: Starvation not detected on CPU ${CPU0}"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Starvation not detected on CPU ${CPU0}"
     fi
 
     if grep -qE "starvation_gen.*starved on CPU ${CPU1}|starved on CPU ${CPU1}.*starvation_gen" "${STALLD_LOG}"; then
         pass "Starvation detected on CPU ${CPU1}"
     else
-        log "✗ FAIL: Starvation not detected on CPU ${CPU1}"
-        TEST_FAILED=$((TEST_FAILED + 1))
+        fail "Starvation not detected on CPU ${CPU1}"
     fi
 
     # Cleanup
@@ -372,16 +364,14 @@ sleep 5
 if assert_process_running "${STALLD_PID}" "stalld still running after task exit"; then
     pass "stalld handled task exit gracefully"
 else
-    log "✗ FAIL: stalld crashed or exited unexpectedly"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld crashed or exited unexpectedly"
 fi
 
 # Check for error messages
 if grep -iE "error|segfault|crash" "${STALLD_LOG}"; then
-    log "✗ FAIL: Error messages found in log"
+    fail "Error messages found in log"
     log "Errors:"
     grep -iE "error|segfault|crash" "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 else
     pass "No error messages in log"
 fi
diff --git a/tests/functional/test_starvation_threshold.sh b/tests/functional/test_starvation_threshold.sh
index a35f898..a2cd420 100755
--- a/tests/functional/test_starvation_threshold.sh
+++ b/tests/functional/test_starvation_threshold.sh
@@ -78,10 +78,9 @@ log "Waiting for detection (threshold: ${threshold}s)"
 if wait_for_starvation_detected "${STALLD_LOG}"; then
     pass "Starvation detected after ${threshold}s threshold"
 else
-    log "✗ FAIL: Starvation not detected after ${threshold}s threshold"
+    fail "Starvation not detected after ${threshold}s threshold"
     log "Log contents:"
     cat "${STALLD_LOG}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -123,10 +122,9 @@ sleep 2
 if ! grep -qE "starvation_gen.*starved on CPU ${TEST_CPU}|starved on CPU ${TEST_CPU}.*starvation_gen" "${STALLD_LOG2}"; then
     pass "No starvation detected for duration less than threshold"
 else
-    log "✗ FAIL: Starvation detected before threshold"
+    fail "Starvation detected before threshold"
     log "Found starvation_gen task in logs:"
     grep -E "starvation_gen.*starved on CPU|starved on CPU.*starvation_gen" "${STALLD_LOG2}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -163,10 +161,9 @@ log "Waiting for detection (threshold: ${threshold}s)"
 if wait_for_starvation_detected "${STALLD_LOG3}"; then
     pass "Starvation detected with ${threshold}s threshold"
 else
-    log "✗ FAIL: Starvation not detected with ${threshold}s threshold"
+    fail "Starvation not detected with ${threshold}s threshold"
     log "Log contents:"
     cat "${STALLD_LOG3}"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Cleanup
@@ -193,8 +190,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Zero threshold rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid threshold value 0"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid threshold value 0"
 fi
 
 # Test with negative threshold
@@ -208,8 +204,7 @@ ret=$?
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "Negative threshold rejected with error"
 else
-    log "✗ FAIL: stalld did not reject invalid negative threshold"
-    TEST_FAILED=$((TEST_FAILED + 1))
+    fail "stalld did not reject invalid negative threshold"
 fi
 
 log ""
diff --git a/tests/functional/test_task_merging.sh b/tests/functional/test_task_merging.sh
index b89230e..e1a4cc1 100755
--- a/tests/functional/test_task_merging.sh
+++ b/tests/functional/test_task_merging.sh
@@ -106,9 +106,8 @@ if [ "${second_duration}" -gt "${first_duration}" ]; then
     pass "Starvation duration increased by ${delta}s"
     log "        Timestamp preserved across monitoring cycles"
 else
-    log "✗ FAIL: Duration did not increase (${first_duration}s -> ${second_duration}s)"
+    fail "Duration did not increase (${first_duration}s -> ${second_duration}s)"
     log "        Timestamp may have been reset (task merging failed)"
-    TEST_FAILED=$((TEST_FAILED + 1))
 fi
 
 # Wait for third detection to confirm continued accumulation
diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 144114c..20f8a3c 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -114,6 +114,15 @@ pass() {
 	TEST_PASSED=$((TEST_PASSED + 1))
 }
 
+# Record a test failure with a description message.
+#
+# Usage: fail "description"
+fail() {
+	local message=${1:-""}
+	log "✗ FAIL: ${message}"
+	TEST_FAILED=$((TEST_FAILED + 1))
+}
+
 # Assert functions
 assert_equals() {
 	local expected=$1
@@ -1130,7 +1139,7 @@ start_starvation_gen() {
 
 # Export functions for use in tests
 export -f start_test end_test
-export -f pass assert_equals assert_contains assert_not_contains
+export -f pass fail assert_equals assert_contains assert_not_contains
 export -f assert_file_exists assert_file_not_exists
 export -f assert_process_running assert_process_not_running
 export -f start_stalld stop_stalld kill_existing_stalld cleanup
-- 
2.53.0

[PATCH stalld 31/36] tests/helpers: Use pass()/fail() in assert functions

[Wander Lairson Costa]

Refactor all assert functions to delegate to pass() and fail()
instead of duplicating the echo formatting and counter increment
logic. Diagnostic context lines in assert_equals, assert_contains,
and assert_not_contains are changed from echo to log() so they
appear in the journal.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/helpers/test_helpers.sh | 50 +++++++++++++----------------------
 1 file changed, 18 insertions(+), 32 deletions(-)

diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 20f8a3c..3a7164d 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -130,14 +130,12 @@ assert_equals() {
 	local message=${3:-""}
 
 	if [ "${expected}" == "${actual}" ]; then
-		echo -e "  ${GREEN}PASS${NC}: ${message}"
-		TEST_PASSED=$((TEST_PASSED + 1))
+		pass "${message}"
 		return 0
 	else
-		echo -e "  ${RED}FAIL${NC}: ${message}"
-		echo "    Expected: ${expected}"
-		echo "    Actual:   ${actual}"
-		TEST_FAILED=$((TEST_FAILED + 1))
+		fail "${message}"
+		log "    Expected: ${expected}"
+		log "    Actual:   ${actual}"
 		return 1
 	fi
 }
@@ -148,13 +146,11 @@ assert_contains() {
 	local message=${3:-""}
 
 	if echo "${haystack}" | grep -q "${needle}"; then
-		echo -e "  ${GREEN}PASS${NC}: ${message}"
-		TEST_PASSED=$((TEST_PASSED + 1))
+		pass "${message}"
 		return 0
 	else
-		echo -e "  ${RED}FAIL${NC}: ${message}"
-		echo "    String '${needle}' not found"
-		TEST_FAILED=$((TEST_FAILED + 1))
+		fail "${message}"
+		log "    String '${needle}' not found"
 		return 1
 	fi
 }
@@ -165,13 +161,11 @@ assert_not_contains() {
 	local message=${3:-""}
 
 	if ! echo "${haystack}" | grep -q "${needle}"; then
-		echo -e "  ${GREEN}PASS${NC}: ${message}"
-		TEST_PASSED=$((TEST_PASSED + 1))
+		pass "${message}"
 		return 0
 	else
-		echo -e "  ${RED}FAIL${NC}: ${message}"
-		echo "    String '${needle}' found but should not be present"
-		TEST_FAILED=$((TEST_FAILED + 1))
+		fail "${message}"
+		log "    String '${needle}' found but should not be present"
 		return 1
 	fi
 }
@@ -181,12 +175,10 @@ assert_file_exists() {
 	local message=${2:-"File should exist: ${file}"}
 
 	if [ -f "${file}" ]; then
-		echo -e "  ${GREEN}PASS${NC}: ${message}"
-		TEST_PASSED=$((TEST_PASSED + 1))
+		pass "${message}"
 		return 0
 	else
-		echo -e "  ${RED}FAIL${NC}: ${message}"
-		TEST_FAILED=$((TEST_FAILED + 1))
+		fail "${message}"
 		return 1
 	fi
 }
@@ -196,12 +188,10 @@ assert_file_not_exists() {
 	local message=${2:-"File should not exist: ${file}"}
 
 	if [ ! -f "${file}" ]; then
-		echo -e "  ${GREEN}PASS${NC}: ${message}"
-		TEST_PASSED=$((TEST_PASSED + 1))
+		pass "${message}"
 		return 0
 	else
-		echo -e "  ${RED}FAIL${NC}: ${message}"
-		TEST_FAILED=$((TEST_FAILED + 1))
+		fail "${message}"
 		return 1
 	fi
 }
@@ -211,12 +201,10 @@ assert_process_running() {
 	local message=${2:-"Process ${pid} should be running"}
 
 	if kill -0 ${pid} 2>/dev/null; then
-		echo -e "  ${GREEN}PASS${NC}: ${message}"
-		TEST_PASSED=$((TEST_PASSED + 1))
+		pass "${message}"
 		return 0
 	else
-		echo -e "  ${RED}FAIL${NC}: ${message}"
-		TEST_FAILED=$((TEST_FAILED + 1))
+		fail "${message}"
 		return 1
 	fi
 }
@@ -226,12 +214,10 @@ assert_process_not_running() {
 	local message=${2:-"Process ${pid} should not be running"}
 
 	if ! kill -0 ${pid} 2>/dev/null; then
-		echo -e "  ${GREEN}PASS${NC}: ${message}"
-		TEST_PASSED=$((TEST_PASSED + 1))
+		pass "${message}"
 		return 0
 	else
-		echo -e "  ${RED}FAIL${NC}: ${message}"
-		TEST_FAILED=$((TEST_FAILED + 1))
+		fail "${message}"
 		return 1
 	fi
 }
-- 
2.53.0

[PATCH stalld 32/36] tests/functional: Fix multi-CPU detection in test_starvation_detection

[Wander Lairson Costa]

The multi-CPU test greps for "starvation_gen" in the starvation log,
but stalld might detect other starving tasks on the target CPU, such
as kworker threads. Relax the grep to match any task starving on
the expected CPU. To support this, add an optional cpu parameter to
wait_for_starvation_detected() so the test waits for each CPU
independently.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_starvation_detection.sh | 12 +++++-------
 tests/helpers/test_helpers.sh                 |  9 +++++++--
 2 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/tests/functional/test_starvation_detection.sh b/tests/functional/test_starvation_detection.sh
index 680cf39..6ca2121 100755
--- a/tests/functional/test_starvation_detection.sh
+++ b/tests/functional/test_starvation_detection.sh
@@ -271,18 +271,16 @@ else
 
     start_stalld_with_log "${STALLD_LOG}" -f -v -N -l -t $threshold -c ${CPU0},${CPU1} -a ${STALLD_CPU_MULTI}
 
-    # Wait for starvation detection
-    log "Waiting for starvation detection..."
-    wait_for_starvation_detected "${STALLD_LOG}"
-
-    # Check both CPUs detected - specifically look for starvation_gen tasks
-    if grep -qE "starvation_gen.*starved on CPU ${CPU0}|starved on CPU ${CPU0}.*starvation_gen" "${STALLD_LOG}"; then
+    # Wait for starvation detection on both CPUs
+    log "Waiting for starvation detection on CPU ${CPU0}..."
+    if wait_for_starvation_detected "${STALLD_LOG}" 30 "${CPU0}"; then
         pass "Starvation detected on CPU ${CPU0}"
     else
         fail "Starvation not detected on CPU ${CPU0}"
     fi
 
-    if grep -qE "starvation_gen.*starved on CPU ${CPU1}|starved on CPU ${CPU1}.*starvation_gen" "${STALLD_LOG}"; then
+    log "Waiting for starvation detection on CPU ${CPU1}..."
+    if wait_for_starvation_detected "${STALLD_LOG}" 30 "${CPU1}"; then
         pass "Starvation detected on CPU ${CPU1}"
     else
         fail "Starvation not detected on CPU ${CPU1}"
diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 3a7164d..6e29a3b 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -542,11 +542,16 @@ wait_for_stalld_ready() {
 
 # Wait for stalld to detect a starving task.
 #
-# Usage: wait_for_starvation_detected <log_file> [timeout]
+# Usage: wait_for_starvation_detected <log_file> [timeout] [cpu]
 wait_for_starvation_detected() {
 	local log_file=$1
 	local timeout=${2:-30}
-	wait_for_log_message "starved on CPU" "${timeout}" "${log_file}"
+	local cpu=${3:-}
+	local pattern="starved on CPU"
+	if [ -n "${cpu}" ]; then
+		pattern="starved on CPU ${cpu}"
+	fi
+	wait_for_log_message "${pattern}" "${timeout}" "${log_file}"
 }
 
 # Wait for stalld to boost a starving task.
-- 
2.53.0

[PATCH stalld 33/36] tests/functional: Accept FIFO fallback in test_fifo_boosting

[Wander Lairson Costa]

Test 5 expects stalld to exit when given -F without -A, but stalld
prints a warning and falls back to adaptive mode instead. The test
had contradictory results: a FAIL for not exiting and a PASS for
finding the warning message. Accept the fallback as valid behavior
and only fail if stalld silently ignores the incompatibility.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_fifo_boosting.sh | 17 ++++-------------
 1 file changed, 4 insertions(+), 13 deletions(-)

diff --git a/tests/functional/test_fifo_boosting.sh b/tests/functional/test_fifo_boosting.sh
index 50c8d14..a003deb 100755
--- a/tests/functional/test_fifo_boosting.sh
+++ b/tests/functional/test_fifo_boosting.sh
@@ -332,20 +332,11 @@ timeout 5 ${TEST_ROOT}/../stalld -f -v -F -t 5 -c ${TEST_CPU} > "${STALLD_LOG_FA
 ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
-    pass "stalld exited as expected"
+    pass "stalld rejected FIFO in single-threaded mode"
+elif grep -qiE "single.*thread|falling back|adaptive" "${STALLD_LOG_FAIL}"; then
+    pass "stalld detected incompatibility and fell back to adaptive mode"
 else
-    fail "stalld did not reject FIFO in single-threaded mode"
-fi
-
-# Check for error message in log
-if grep -qiE "single.*thread.*fifo|fifo.*single.*thread|can.*only.*deadline" "${STALLD_LOG_FAIL}"; then
-    pass "Error message about FIFO+single-threaded incompatibility found"
-else
-    log "ℹ INFO: Checking exit status or error messages..."
-    if [ -s "${STALLD_LOG_FAIL}" ]; then
-        log "Log contents:"
-        cat "${STALLD_LOG_FAIL}"
-    fi
+    fail "stalld silently accepted FIFO in single-threaded mode"
 fi
 
 #=============================================================================
-- 
2.53.0

[PATCH stalld 34/36] tests/functional: Fix readiness detection and FIFO fallback in test_force_fifo

[Wander Lairson Costa]

Add "skipping" to the wait_for_stalld_ready() pattern to handle
aggressive mode with idle detection, where stalld prints "skipping
next phase" instead of "checking cpu" or "waiting tasks". Also
accept stalld's fallback to adaptive mode when given -F without -A,
matching the fix already applied to test_fifo_boosting.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_force_fifo.sh | 4 +++-
 tests/helpers/test_helpers.sh       | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/functional/test_force_fifo.sh b/tests/functional/test_force_fifo.sh
index ecaa7ac..239bc37 100755
--- a/tests/functional/test_force_fifo.sh
+++ b/tests/functional/test_force_fifo.sh
@@ -218,8 +218,10 @@ ret=$?
 
 if [ $ret -ne 0 ] && [ $ret -ne 124 ]; then
     pass "single-threaded mode rejected FIFO"
+elif grep -qiE "single.*thread|falling back|adaptive" "${FIFO_SINGLE_LOG}"; then
+    pass "stalld detected incompatibility and fell back to adaptive mode"
 else
-    fail "stalld did not reject -F in single-threaded mode"
+    fail "stalld silently accepted FIFO in single-threaded mode"
 fi
 
 #=============================================================================
diff --git a/tests/helpers/test_helpers.sh b/tests/helpers/test_helpers.sh
index 6e29a3b..9d49b55 100755
--- a/tests/helpers/test_helpers.sh
+++ b/tests/helpers/test_helpers.sh
@@ -537,7 +537,7 @@ wait_for_log_message() {
 wait_for_stalld_ready() {
 	local log_file=$1
 	local timeout=${2:-15}
-	wait_for_log_message "checking cpu\|waiting tasks" "${timeout}" "${log_file}"
+	wait_for_log_message "checking cpu\|waiting tasks\|skipping" "${timeout}" "${log_file}"
 }
 
 # Wait for stalld to detect a starving task.
-- 
2.53.0

[PATCH stalld 35/36] tests/functional: Fix invalid pidfile test in test_pidfile

[Wander Lairson Costa]

The test created a directory with no write permission and expected
stalld to fail writing the pidfile. Since the tests run as root,
the permission restriction has no effect and stalld succeeds. Use
a non-existent parent directory instead so fopen() fails regardless
of privileges.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 tests/functional/test_pidfile.sh | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/tests/functional/test_pidfile.sh b/tests/functional/test_pidfile.sh
index 0e4c623..76be14a 100755
--- a/tests/functional/test_pidfile.sh
+++ b/tests/functional/test_pidfile.sh
@@ -192,12 +192,8 @@ log "=========================================="
 log "Test 6: Invalid pidfile path (permission denied)"
 log "=========================================="
 
-# Create a directory with no write permissions
-test_dir="/tmp/stalld_test_no_write_$$"
-mkdir -p "${test_dir}"
-chmod 555 "${test_dir}"
-invalid_pidfile="${test_dir}/stalld.pid"
-CLEANUP_FILES+=("${test_dir}")
+# Use a non-existent parent directory so fopen() fails even as root
+invalid_pidfile="/nonexistent_${$}/stalld.pid"
 
 INVALID_LOG="/tmp/stalld_test_pidfile_invalid_$$.log"
 CLEANUP_FILES+=("${INVALID_LOG}")
-- 
2.53.0

[PATCH stalld 36/36] stalld: die on invalid CPU affinity

[Wander Lairson Costa]

set_cpu_affinity() validates the CPU list and returns -1 on error,
but main() ignored the return value, allowing stalld to run with
an invalid or empty affinity mask.

Signed-off-by: Wander Lairson Costa <wander@redhat.com>
---
 src/stalld.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/src/stalld.c b/src/stalld.c
index b9372db..d95b48f 100644
--- a/src/stalld.c
+++ b/src/stalld.c
@@ -1166,11 +1166,9 @@ int main(int argc, char **argv)
 
 	parse_args(argc, argv);
 
-	/*
-	 * it will not die...
-	 */
 	if (config_affinity_cpus)
-		set_cpu_affinity(config_affinity_cpus);
+		if (set_cpu_affinity(config_affinity_cpus))
+			die("invalid CPU affinity: %s\n", config_affinity_cpus);
 
 	if (!check_dl_server_dir_exists())
 		log_msg("DL-server detected.\n");
-- 
2.53.0