[LTP] [PATCH v2 0/6] sched_football: Re-add crazy fans and other cleanups

[LTP] [PATCH v2 0/6] sched_football: Re-add crazy fans and other cleanups

[John Stultz via ltp]

Just the other day (~18 years ago), I had implemented the
sched_football test to validate the RT scheduling invariant that
we always run the top NR_CPU priority tasks at any time.

That early version of the test included crazy fans, which
disrupted the scheduling of the defensive threads. This was
later removed from the test, likely due to occasional failures
seen with it.

Recently, in testing the behavior of a scheduler patch I was
working on I started tinkering with this test again, and found
when I re-added the crazy fans, I could reproduce an actual
scheduling problem with the vanilla kernel.

So I wanted to submit these changes to re-add the crazy fans
along with some other cleanups and improvements to make it
easier to use with tracing, so that the test can be used by
others to reproduce this problem.

New in v2 (thank you for the feedback!):
* Rework to use tst_atomic* methods instead of atomic_t
  suggested by Cyril
* Remove unused variables
* Drop changelog comment as suggested by Cyril
* Rework to use tst_atomic_* functions as suggested by Cyril
* Rework to use tst_timespec* functions as suggested by Cyril
* Pulled trace marker writing out into librttest helper functions
  as suggested by Cyril

thanks
-john

Cc: kernel-team@android.com
Cc: Cyril Hrubis <chrubis@suse.cz>
Cc: Darren Hart <darren@os.amperecomputing.com>


John Stultz (6):
  sched_football: Drop use of sched_yeild()
  sched_football: Use atomic operations for ball
  sched_football: Re-add the crazy fans to interrupt everyone
  sched_football: Add a sleep before the game begins to get into steady
    state
  sched_football: Add prctrl calls to set thread comms
  sched_football: Add trace_marker messages if we're tracing

 .../func/sched_football/sched_football.c      | 95 +++++++++++++++----
 testcases/realtime/include/librttest.h        | 13 +++
 testcases/realtime/lib/librttest.c            | 32 +++++++
 3 files changed, 119 insertions(+), 21 deletions(-)

-- 
2.45.2.741.gdbec12cfda-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 1/6] sched_football: Drop use of sched_yeild()

[John Stultz via ltp]

sched_yeild() just causes the SCHED_FIFO tasks to behave like
SCHED_RR which makes analysis of the behavior more difficult to
follow. So lets drop the calls.

Also fixes up some whitespace inconsistencies in the header comment.

Cc: kernel-team@android.com
Cc: Cyril Hrubis <chrubis@suse.cz>
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 .../realtime/func/sched_football/sched_football.c      | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 6f075aea3..1e205219d 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -32,12 +32,12 @@
  *       - Create a fixed number of offense threads (lower priority)
  *       - Create a referee thread (highest priority)
  *       - Once everyone is on the field, the offense thread increments the
- *	 value of 'the_ball' and yields. The defense thread tries to block
- *	 the ball by never letting the offense players get the CPU (it just
- * 	   does a sched_yield).
+ *         value of 'the_ball'. The defense thread tries to block
+ *         the ball by never letting the offense players get the CPU (it just
+ *         spins).
  *       - The refree threads wakes up regularly to check if the game is over :)
  *       - In the end, if the value of 'the_ball' is >0, the test is considered
- *	 to have failed.
+ *         to have failed.
  *
  * USAGE:
  *      Use run_auto.sh script in current directory to build and run test.
@@ -113,7 +113,6 @@ void *thread_defense(void *arg)
 	atomic_inc(&players_ready);
 	/*keep the ball from being moved */
 	while (1) {
-		sched_yield();	/* let other defenders run */
 	}
 	return NULL;
 }
@@ -124,7 +123,6 @@ void *thread_offense(void *arg)
 	atomic_inc(&players_ready);
 	while (1) {
 		the_ball++;	/* move the ball ahead one yard */
-		sched_yield();	/* let other offensive players run */
 	}
 	return NULL;
 }
-- 
2.45.2.741.gdbec12cfda-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 2/6] sched_football: Use atomic operations for ball

[John Stultz via ltp]

Use the tst_atomic* methods for the player count and ball values,
as we don't have any locking going on.

Cc: kernel-team@android.com
Cc: Cyril Hrubis <chrubis@suse.cz>
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
v2: Rework to use tst_atomic* methods instead of atomic_t, as
    suggested by Cyril
---
 .../func/sched_football/sched_football.c      | 21 ++++++++++---------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 1e205219d..1a1bc5780 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -67,15 +67,16 @@
 #include <unistd.h>
 #include <sys/time.h>
 #include <librttest.h>
+#include <tst_atomic.h>
 
 #define DEF_GAME_LENGTH 5
 
 /* Here's the position of the ball */
-volatile int the_ball;
+static int the_ball;
 
 static int players_per_team = 0;
 static int game_length = DEF_GAME_LENGTH;
-static atomic_t players_ready;
+static int players_ready;
 
 void usage(void)
 {
@@ -110,7 +111,7 @@ int parse_args(int c, char *v)
 /* This is the defensive team. They're trying to block the offense */
 void *thread_defense(void *arg)
 {
-	atomic_inc(&players_ready);
+	tst_atomic_add_return(1, &players_ready);
 	/*keep the ball from being moved */
 	while (1) {
 	}
@@ -120,9 +121,9 @@ void *thread_defense(void *arg)
 /* This is the offensive team. They're trying to move the ball */
 void *thread_offense(void *arg)
 {
-	atomic_inc(&players_ready);
+	tst_atomic_add_return(1, &players_ready);
 	while (1) {
-		the_ball++;	/* move the ball ahead one yard */
+		tst_atomic_add_return(1, &the_ball); /* move the ball ahead one yard */
 	}
 	return NULL;
 }
@@ -138,16 +139,16 @@ int referee(int game_length)
 	now = start;
 
 	/* Start the game! */
-	the_ball = 0;
+	tst_atomic_store(0, &the_ball);
 
 	/* Watch the game */
 	while ((now.tv_sec - start.tv_sec) < game_length) {
 		sleep(1);
 		gettimeofday(&now, NULL);
 	}
+	final_ball = tst_atomic_load(&the_ball);
 	/* Blow the whistle */
 	printf("Game Over!\n");
-	final_ball = the_ball;
 	printf("Final ball position: %d\n", final_ball);
 	return final_ball != 0;
 }
@@ -165,7 +166,7 @@ int main(int argc, char *argv[])
 	if (players_per_team == 0)
 		players_per_team = sysconf(_SC_NPROCESSORS_ONLN);
 
-	atomic_set(0, &players_ready);
+	tst_atomic_store(0, &players_ready);
 
 	printf("Running with: players_per_team=%d game_length=%d\n",
 	       players_per_team, game_length);
@@ -185,7 +186,7 @@ int main(int argc, char *argv[])
 		create_fifo_thread(thread_offense, NULL, priority);
 
 	/* Wait for the offense threads to start */
-	while (atomic_get(&players_ready) < players_per_team)
+	while (tst_atomic_load(&players_ready) < players_per_team)
 		usleep(100);
 
 	/* Start the defense */
@@ -196,7 +197,7 @@ int main(int argc, char *argv[])
 		create_fifo_thread(thread_defense, NULL, priority);
 
 	/* Wait for the defense threads to start */
-	while (atomic_get(&players_ready) < players_per_team * 2)
+	while (tst_atomic_load(&players_ready) < players_per_team * 2)
 		usleep(100);
 
 	/* Ok, everyone is on the field, bring out the ref */
-- 
2.45.2.741.gdbec12cfda-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 3/6] sched_football: Re-add the crazy fans to interrupt everyone

[John Stultz via ltp]

An idea that I had included long ago in this test case, but was
later dropped was the "crazy fans", which added extra disruption
to the scheduling behavior.

In my recent work, I've found that including them uncovers some
unexpected behavior with current Linux kernels.

So re-add the high priroity tasks that wake up and run to disrupt
the scheduling of defense threads. We run 2*NR_CPU "fan" threads
and they spin for awhile, to force migrations.

Cc: kernel-team@android.com
Cc: Cyril Hrubis <chrubis@suse.cz>
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
v2:
* Remove unused variable
* Drop changelog comment as suggested by Cyril
* Reuse existing ts-to-ns helpers
* Rework to use tst_atomic_* functions as suggested by Cyril
* Rework to use tst_timespec* functions as suggested by Cyril
---
 .../func/sched_football/sched_football.c      | 58 ++++++++++++++++---
 1 file changed, 50 insertions(+), 8 deletions(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 1a1bc5780..37cff515c 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -22,19 +22,23 @@
  * DESCRIPTION
  *      This is a scheduler test that uses a football analogy.
  *      The premise is that we want to make sure that lower priority threads
- *      (defensive team). The offense is trying to increment the balls position,
- *      while the defense is trying to block that from happening.
+ *      don't run while we have runnable higher priority threads.
+ *      The offense is trying to increment the balls position, while the
+ *      defense is trying to block that from happening.
  *      And the ref (highest priority thread) will blow the wistle if the
  *      ball moves. Finally, we have crazy fans (higer prority) that try to
  *      distract the defense by occasionally running onto the field.
  *
  *      Steps:
- *       - Create a fixed number of offense threads (lower priority)
+ *       - Create NR_CPU offense threads (lower priority)
+ *       - Create NR_CPU defense threads (mid priority)
+ *       - Create 2*NR_CPU fan threads (high priority)
  *       - Create a referee thread (highest priority)
- *       - Once everyone is on the field, the offense thread increments the
- *         value of 'the_ball'. The defense thread tries to block
- *         the ball by never letting the offense players get the CPU (it just
- *         spins).
+ *       - Once everyone is on the field, the offense thread spins incrementing
+ *         the value of 'the_ball'. The defense thread tries to block the ball
+ *         by never letting the offense players get the CPU (it just spins).
+ *         The crazy fans sleep a bit, then jump the rail and run across the
+ *         field, disrupting the players on the field.
  *       - The refree threads wakes up regularly to check if the game is over :)
  *       - In the end, if the value of 'the_ball' is >0, the test is considered
  *         to have failed.
@@ -52,7 +56,6 @@
  *		bugfixes and cleanups. -- Josh Triplett
  *     2009-06-23 Simplified atomic startup mechanism, avoiding thundering herd
  *		scheduling at the beginning of the game. -- Darren Hart
- *
  *****************************************************************************/
 
 #include <stdio.h>
@@ -68,6 +71,9 @@
 #include <sys/time.h>
 #include <librttest.h>
 #include <tst_atomic.h>
+#define TST_NO_DEFAULT_MAIN
+#include <tst_timer.h>
+
 
 #define DEF_GAME_LENGTH 5
 
@@ -108,6 +114,31 @@ int parse_args(int c, char *v)
 	return handled;
 }
 
+#define SPIN_TIME_NS 200000000ULL
+#define SLEEP_TIME_NS 50000000ULL
+/* These are fans running across the field. They're trying to interrupt/distract everyone */
+void *thread_fan(void *arg)
+{
+	tst_atomic_add_return(1, &players_ready);
+	/*occasionally wake up and run across the field */
+	while (1) {
+		struct timespec start, stop;
+		nsec_t nsec;
+
+		start.tv_sec = 0;
+		start.tv_nsec = SLEEP_TIME_NS;
+		clock_nanosleep(CLOCK_MONOTONIC, 0, &start, NULL);
+		clock_gettime(CLOCK_MONOTONIC, &start);
+		clock_gettime(CLOCK_MONOTONIC, &stop);
+		nsec = tst_timespec_diff_ns(stop, start);
+		while (nsec < SPIN_TIME_NS) {
+			clock_gettime(CLOCK_MONOTONIC, &stop);
+			nsec = tst_timespec_diff_ns(stop, start);
+		}
+	}
+	return NULL;
+}
+
 /* This is the defensive team. They're trying to block the offense */
 void *thread_defense(void *arg)
 {
@@ -200,6 +231,17 @@ int main(int argc, char *argv[])
 	while (tst_atomic_load(&players_ready) < players_per_team * 2)
 		usleep(100);
 
+	/* Start the crazy fans*/
+	priority = 50;
+	printf("Starting %d fan threads at priority %d\n",
+	       players_per_team, priority);
+	for (i = 0; i < players_per_team*2; i++)
+		create_fifo_thread(thread_fan, NULL, priority);
+
+	/* Wait for the crazy fan threads to start */
+	while (tst_atomic_load(&players_ready) < players_per_team * 4)
+		usleep(100);
+
 	/* Ok, everyone is on the field, bring out the ref */
 	printf("Starting referee thread\n");
 	result = referee(game_length);
-- 
2.45.2.741.gdbec12cfda-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 4/6] sched_football: Add a sleep before the game begins to get into steady state

[John Stultz via ltp]

Before we begin the game, lets give a little extra time for any
odd initial placement to get sorted before we start testing the
RT scheduling invariant.

Cc: kernel-team@android.com
Cc: Cyril Hrubis <chrubis@suse.cz>
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 testcases/realtime/func/sched_football/sched_football.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 37cff515c..9d12f0193 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -242,6 +242,8 @@ int main(int argc, char *argv[])
 	while (tst_atomic_load(&players_ready) < players_per_team * 4)
 		usleep(100);
 
+	/* let things get into steady state */
+	sleep(2);
 	/* Ok, everyone is on the field, bring out the ref */
 	printf("Starting referee thread\n");
 	result = referee(game_length);
-- 
2.45.2.741.gdbec12cfda-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 5/6] sched_football: Add prctrl calls to set thread comms

[John Stultz via ltp]

Make it easier to follow whats happening in traces by setting
the thread comms to be more descriptive.

Cc: kernel-team@android.com
Cc: Cyril Hrubis <chrubis@suse.cz>
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 testcases/realtime/func/sched_football/sched_football.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 9d12f0193..40496cc22 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -68,6 +68,7 @@
 #include <errno.h>
 #include <sys/syscall.h>
 #include <unistd.h>
+#include <sys/prctl.h>
 #include <sys/time.h>
 #include <librttest.h>
 #include <tst_atomic.h>
@@ -119,6 +120,7 @@ int parse_args(int c, char *v)
 /* These are fans running across the field. They're trying to interrupt/distract everyone */
 void *thread_fan(void *arg)
 {
+	prctl(PR_SET_NAME, "crazy_fan", 0, 0, 0);
 	tst_atomic_add_return(1, &players_ready);
 	/*occasionally wake up and run across the field */
 	while (1) {
@@ -142,6 +144,7 @@ void *thread_fan(void *arg)
 /* This is the defensive team. They're trying to block the offense */
 void *thread_defense(void *arg)
 {
+	prctl(PR_SET_NAME, "defense", 0, 0, 0);
 	tst_atomic_add_return(1, &players_ready);
 	/*keep the ball from being moved */
 	while (1) {
@@ -152,6 +155,7 @@ void *thread_defense(void *arg)
 /* This is the offensive team. They're trying to move the ball */
 void *thread_offense(void *arg)
 {
+	prctl(PR_SET_NAME, "offense", 0, 0, 0);
 	tst_atomic_add_return(1, &players_ready);
 	while (1) {
 		tst_atomic_add_return(1, &the_ball); /* move the ball ahead one yard */
@@ -164,6 +168,7 @@ int referee(int game_length)
 	struct timeval start, now;
 	int final_ball;
 
+	prctl(PR_SET_NAME, "referee", 0, 0, 0);
 	printf("Game On (%d seconds)!\n", game_length);
 
 	gettimeofday(&start, NULL);
-- 
2.45.2.741.gdbec12cfda-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 6/6] sched_football: Add trace_marker messages if we're tracing

[John Stultz via ltp]

To further help with tracing, add trace_marker messages so we
can see exactly when the game starts and ends in the tracelog.

Cc: kernel-team@android.com
Cc: Cyril Hrubis <chrubis@suse.cz>
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
v2:
* Pulled trace marker writing out into librttest helper functions
  as suggested by Cyril
---
 .../func/sched_football/sched_football.c      |  5 +++
 testcases/realtime/include/librttest.h        | 13 ++++++++
 testcases/realtime/lib/librttest.c            | 32 +++++++++++++++++++
 3 files changed, 50 insertions(+)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 40496cc22..b6ae692af 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -66,6 +66,7 @@
 #include <pthread.h>
 #include <sched.h>
 #include <errno.h>
+#include <fcntl.h>
 #include <sys/syscall.h>
 #include <unistd.h>
 #include <sys/prctl.h>
@@ -171,17 +172,21 @@ int referee(int game_length)
 	prctl(PR_SET_NAME, "referee", 0, 0, 0);
 	printf("Game On (%d seconds)!\n", game_length);
 
+	/* open trace marker early to avoid latency with the first message */
+	trace_marker_prep();
 	gettimeofday(&start, NULL);
 	now = start;
 
 	/* Start the game! */
 	tst_atomic_store(0, &the_ball);
+	atrace_marker_write("sched_football", "Game_started!");
 
 	/* Watch the game */
 	while ((now.tv_sec - start.tv_sec) < game_length) {
 		sleep(1);
 		gettimeofday(&now, NULL);
 	}
+	atrace_marker_write("sched_football", "Game_Over!");
 	final_ball = tst_atomic_load(&the_ball);
 	/* Blow the whistle */
 	printf("Game Over!\n");
diff --git a/testcases/realtime/include/librttest.h b/testcases/realtime/include/librttest.h
index 8733479e7..0a1bb0540 100644
--- a/testcases/realtime/include/librttest.h
+++ b/testcases/realtime/include/librttest.h
@@ -342,4 +342,17 @@ void latency_trace_stop(void);
  */
 void latency_trace_print(void);
 
+/* trace_marker_prep: open trace_marker file (optional)
+ */
+void trace_marker_prep(void);
+
+/* trace_marker_write: write buf to trace_marker.
+ * Will open trace_marker file if not already open
+ */
+int trace_marker_write(char *buf, int len);
+
+/* atrace_marker_write: write atrace format message to trace_marker
+ */
+int atrace_marker_write(char *tag, char *msg);
+
 #endif /* LIBRTTEST_H */
diff --git a/testcases/realtime/lib/librttest.c b/testcases/realtime/lib/librttest.c
index eaa623b72..191c667a1 100644
--- a/testcases/realtime/lib/librttest.c
+++ b/testcases/realtime/lib/librttest.c
@@ -732,3 +732,35 @@ void latency_trace_print(void)
 {
 	read_and_print("/proc/latency_trace", STDOUT_FILENO);
 }
+
+static int trace_marker_fd = -1;
+
+void trace_marker_prep(void)
+{
+	if (trace_marker_fd != -1)
+		return;
+	trace_marker_fd = open("/sys/kernel/tracing/trace_marker", O_RDWR, 0);
+}
+
+int trace_marker_write(char *buf, int len)
+{
+	if (trace_marker_fd == -1)
+		trace_marker_prep();
+
+	if (trace_marker_fd < 0)
+		return -1;
+
+	return write(trace_marker_fd, buf, len);
+}
+
+#define TRACE_BUF_LEN 256
+static char trace_buf[TRACE_BUF_LEN];
+
+int atrace_marker_write(char *tag, char *msg)
+{
+	/* Uses atrace format perfetto can visualize */
+	snprintf(trace_buf, TRACE_BUF_LEN, "I|%i|%s: %s\n", getpid(), tag, msg);
+	return trace_marker_write(trace_buf,
+				  strnlen(trace_buf, TRACE_BUF_LEN));
+}
+
-- 
2.45.2.741.gdbec12cfda-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 0/6] sched_football: Re-add crazy fans and other cleanups

[Cyril Hrubis]

Hi!
Patchset pushed, thanks.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp