[LTP] [PATCH 0/6] sched_football: Re-add crazy fans and other cleanups

[LTP] [PATCH 0/6] sched_football: Re-add crazy fans and other cleanups

[John Stultz via ltp]

Just the other day (~18 years ago), I had implemented the
sched_football test to validate the RT scheduling invariant that
we always run the top NR_CPU priority tasks at any time. 

That early version of the test included crazy fans, which
disrupted the scheduling of the defensive threads. This was
later removed from the test, likely due to occasional failures
seen with it.

Recently, in testing the behavior of a scheduler patch I was
working on I started tinkering with this test again, and found
when I re-added the crazy fans, I could reproduce an actual
scheduling problem with the vanilla kernel.

So I wanted to submit these changes to re-add the crazy fans
along with some other cleanups and improvements to make it
easier to use with tracing, so that the test can be used by
others to reproduce this problem.

Feedback would be appreciated!

thanks
-john

Cc: kernel-team@android.com
Cc: Darren Hart <darren@os.amperecomputing.com>

John Stultz (6):
  sched_football: Drop use of sched_yeild()
  sched_football: Use atomic for ball
  sched_football: Re-add the crazy fans to interrupt everyone
  sched_football: Add a sleep before the game begins to get into steady
    state
  sched_football: Add prctrl calls to set thread comms
  sched_football: Add trace_marker messages if we're tracing

 .../func/sched_football/sched_football.c      | 94 +++++++++++++++----
 1 file changed, 78 insertions(+), 16 deletions(-)

-- 
2.44.0.769.g3c40516874-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 1/6] sched_football: Drop use of sched_yeild()

[John Stultz via ltp]

sched_yeild() just causes the SCHED_FIFO tasks to behave like
SCHED_RR which makes analysis of the behavior more difficult to
follow. So lets drop the calls.

Also fixes up some whitespace inconsistencies in the header comment.

Cc: kernel-team@android.com
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 .../realtime/func/sched_football/sched_football.c      | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 6f075aea3..1e205219d 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -32,12 +32,12 @@
  *       - Create a fixed number of offense threads (lower priority)
  *       - Create a referee thread (highest priority)
  *       - Once everyone is on the field, the offense thread increments the
- *	 value of 'the_ball' and yields. The defense thread tries to block
- *	 the ball by never letting the offense players get the CPU (it just
- * 	   does a sched_yield).
+ *         value of 'the_ball'. The defense thread tries to block
+ *         the ball by never letting the offense players get the CPU (it just
+ *         spins).
  *       - The refree threads wakes up regularly to check if the game is over :)
  *       - In the end, if the value of 'the_ball' is >0, the test is considered
- *	 to have failed.
+ *         to have failed.
  *
  * USAGE:
  *      Use run_auto.sh script in current directory to build and run test.
@@ -113,7 +113,6 @@ void *thread_defense(void *arg)
 	atomic_inc(&players_ready);
 	/*keep the ball from being moved */
 	while (1) {
-		sched_yield();	/* let other defenders run */
 	}
 	return NULL;
 }
@@ -124,7 +123,6 @@ void *thread_offense(void *arg)
 	atomic_inc(&players_ready);
 	while (1) {
 		the_ball++;	/* move the ball ahead one yard */
-		sched_yield();	/* let other offensive players run */
 	}
 	return NULL;
 }
-- 
2.44.0.769.g3c40516874-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

Use atomic type for the ball value, as we don't have any locking
going on.

Cc: kernel-team@android.com
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 testcases/realtime/func/sched_football/sched_football.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 1e205219d..359c96703 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -71,7 +71,7 @@
 #define DEF_GAME_LENGTH 5
 
 /* Here's the position of the ball */
-volatile int the_ball;
+static atomic_t the_ball;
 
 static int players_per_team = 0;
 static int game_length = DEF_GAME_LENGTH;
@@ -122,7 +122,7 @@ void *thread_offense(void *arg)
 {
 	atomic_inc(&players_ready);
 	while (1) {
-		the_ball++;	/* move the ball ahead one yard */
+		atomic_inc(&the_ball);	/* move the ball ahead one yard */
 	}
 	return NULL;
 }
@@ -138,16 +138,16 @@ int referee(int game_length)
 	now = start;
 
 	/* Start the game! */
-	the_ball = 0;
+	atomic_set(0, &the_ball);
 
 	/* Watch the game */
 	while ((now.tv_sec - start.tv_sec) < game_length) {
 		sleep(1);
 		gettimeofday(&now, NULL);
 	}
+	final_ball = atomic_get(&the_ball);
 	/* Blow the whistle */
 	printf("Game Over!\n");
-	final_ball = the_ball;
 	printf("Final ball position: %d\n", final_ball);
 	return final_ball != 0;
 }
-- 
2.44.0.769.g3c40516874-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 3/6] sched_football: Re-add the crazy fans to interrupt everyone

[John Stultz via ltp]

An idea that I had included long ago in this test case, but was
later dropped was the "crazy fans", which added extra disruption
to the scheduling behavior.

In my recent work, I've found that including them uncovers some
unexpected behavior with current Linux kernels.

So re-add the high priroity tasks that wake up and run to disrupt
the scheduling of defense threads. We run 2*NR_CPU "fan" threads
and they spin for awhile, to force migrations.

Cc: kernel-team@android.com
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 .../func/sched_football/sched_football.c      | 63 ++++++++++++++++---
 1 file changed, 55 insertions(+), 8 deletions(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 359c96703..ed3c5cadf 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -22,19 +22,23 @@
  * DESCRIPTION
  *      This is a scheduler test that uses a football analogy.
  *      The premise is that we want to make sure that lower priority threads
- *      (defensive team). The offense is trying to increment the balls position,
- *      while the defense is trying to block that from happening.
+ *      don't run while we have runnable higher priority threads.
+ *      The offense is trying to increment the balls position, while the
+ *      defense is trying to block that from happening.
  *      And the ref (highest priority thread) will blow the wistle if the
  *      ball moves. Finally, we have crazy fans (higer prority) that try to
  *      distract the defense by occasionally running onto the field.
  *
  *      Steps:
- *       - Create a fixed number of offense threads (lower priority)
+ *       - Create NR_CPU offense threads (lower priority)
+ *       - Create NR_CPU defense threads (mid priority)
+ *       - Create 2*NR_CPU fan threads (high priority)
  *       - Create a referee thread (highest priority)
- *       - Once everyone is on the field, the offense thread increments the
- *         value of 'the_ball'. The defense thread tries to block
- *         the ball by never letting the offense players get the CPU (it just
- *         spins).
+ *       - Once everyone is on the field, the offense thread spins incrementing
+ *         the value of 'the_ball'. The defense thread tries to block the ball
+ *         by never letting the offense players get the CPU (it just spins).
+ *         The crazy fans sleep a bit, then jump the rail and run across the
+ *         field, disrupting the players on the field.
  *       - The refree threads wakes up regularly to check if the game is over :)
  *       - In the end, if the value of 'the_ball' is >0, the test is considered
  *         to have failed.
@@ -52,7 +56,7 @@
  *		bugfixes and cleanups. -- Josh Triplett
  *     2009-06-23 Simplified atomic startup mechanism, avoiding thundering herd
  *		scheduling at the beginning of the game. -- Darren Hart
- *
+ *     2024-04-23 Re-add crazy fans! Along with minor cleanups -- John Stultz
  *****************************************************************************/
 
 #include <stdio.h>
@@ -107,6 +111,38 @@ int parse_args(int c, char *v)
 	return handled;
 }
 
+#define NSEC_PER_SEC 1000000000ULL
+unsigned long long ts_delta(struct timespec *start, struct timespec *stop)
+{
+	unsigned long long a, b;
+
+	a = start->tv_sec * NSEC_PER_SEC + start->tv_nsec;
+	b = stop->tv_sec * NSEC_PER_SEC + stop->tv_nsec;
+	return b - a;
+}
+
+#define SPIN_TIME_NS 200000000
+#define SLEEP_TIME_NS 50000000
+/* These are fans running across the field. They're trying to interrupt/distract everyone */
+void *thread_fan(void *arg)
+{
+	atomic_inc(&players_ready);
+	/*occasionally wake up and run across the field */
+	while (1) {
+		int i;
+		struct timespec start, stop;
+
+		start.tv_sec = 0;
+		start.tv_nsec = SLEEP_TIME_NS;
+		clock_nanosleep(CLOCK_MONOTONIC, 0, &start, NULL);
+		clock_gettime(CLOCK_MONOTONIC, &start);
+		clock_gettime(CLOCK_MONOTONIC, &stop);
+		while (ts_delta(&start, &stop) < SPIN_TIME_NS)
+			clock_gettime(CLOCK_MONOTONIC, &stop);
+	}
+	return NULL;
+}
+
 /* This is the defensive team. They're trying to block the offense */
 void *thread_defense(void *arg)
 {
@@ -199,6 +235,17 @@ int main(int argc, char *argv[])
 	while (atomic_get(&players_ready) < players_per_team * 2)
 		usleep(100);
 
+	/* Start the crazy fans*/
+	priority = 50;
+	printf("Starting %d fan threads at priority %d\n",
+	       players_per_team, priority);
+	for (i = 0; i < players_per_team*2; i++)
+		create_fifo_thread(thread_fan, NULL, priority);
+
+	/* Wait for the crazy fan threads to start */
+	while (atomic_get(&players_ready) < players_per_team * 4)
+		usleep(100);
+
 	/* Ok, everyone is on the field, bring out the ref */
 	printf("Starting referee thread\n");
 	result = referee(game_length);
-- 
2.44.0.769.g3c40516874-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 4/6] sched_football: Add a sleep before the game begins to get into steady state

[John Stultz via ltp]

Before we begin the game, lets give a little extra time for any
odd initial placement to get sorted before we start testing the
RT scheduling invariant.

Cc: kernel-team@android.com
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 testcases/realtime/func/sched_football/sched_football.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index ed3c5cadf..7686ce3e6 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -246,6 +246,8 @@ int main(int argc, char *argv[])
 	while (atomic_get(&players_ready) < players_per_team * 4)
 		usleep(100);
 
+	/* let things get into steady state */
+	sleep(2);
 	/* Ok, everyone is on the field, bring out the ref */
 	printf("Starting referee thread\n");
 	result = referee(game_length);
-- 
2.44.0.769.g3c40516874-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 5/6] sched_football: Add prctrl calls to set thread comms

[John Stultz via ltp]

Make it easier to follow whats happening in traces by setting
the thread comms to be more descriptive.

Cc: kernel-team@android.com
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 testcases/realtime/func/sched_football/sched_football.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 7686ce3e6..45fbf6766 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -69,6 +69,7 @@
 #include <errno.h>
 #include <sys/syscall.h>
 #include <unistd.h>
+#include <sys/prctl.h>
 #include <sys/time.h>
 #include <librttest.h>
 
@@ -126,6 +127,7 @@ unsigned long long ts_delta(struct timespec *start, struct timespec *stop)
 /* These are fans running across the field. They're trying to interrupt/distract everyone */
 void *thread_fan(void *arg)
 {
+	prctl(PR_SET_NAME, "crazy_fan", 0, 0, 0);
 	atomic_inc(&players_ready);
 	/*occasionally wake up and run across the field */
 	while (1) {
@@ -146,6 +148,7 @@ void *thread_fan(void *arg)
 /* This is the defensive team. They're trying to block the offense */
 void *thread_defense(void *arg)
 {
+	prctl(PR_SET_NAME, "defense", 0, 0, 0);
 	atomic_inc(&players_ready);
 	/*keep the ball from being moved */
 	while (1) {
@@ -156,6 +159,7 @@ void *thread_defense(void *arg)
 /* This is the offensive team. They're trying to move the ball */
 void *thread_offense(void *arg)
 {
+	prctl(PR_SET_NAME, "offense", 0, 0, 0);
 	atomic_inc(&players_ready);
 	while (1) {
 		atomic_inc(&the_ball);	/* move the ball ahead one yard */
@@ -168,6 +172,7 @@ int referee(int game_length)
 	struct timeval start, now;
 	int final_ball;
 
+	prctl(PR_SET_NAME, "referee", 0, 0, 0);
 	printf("Game On (%d seconds)!\n", game_length);
 
 	gettimeofday(&start, NULL);
-- 
2.44.0.769.g3c40516874-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 6/6] sched_football: Add trace_marker messages if we're tracing

[John Stultz via ltp]

To further help with tracing, add trace_marker messages so we
can see exactly when the game starts and ends in the tracelog.

Cc: kernel-team@android.com
Cc: Darren Hart <darren@os.amperecomputing.com>
Signed-off-by: John Stultz <jstultz@google.com>
---
 .../realtime/func/sched_football/sched_football.c    | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 45fbf6766..ca44584b0 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -67,6 +67,7 @@
 #include <pthread.h>
 #include <sched.h>
 #include <errno.h>
+#include <fcntl.h>
 #include <sys/syscall.h>
 #include <unistd.h>
 #include <sys/prctl.h>
@@ -167,10 +168,14 @@ void *thread_offense(void *arg)
 	return NULL;
 }
 
+#define BUF_LEN 256
 int referee(int game_length)
 {
 	struct timeval start, now;
+	char buf[BUF_LEN];
 	int final_ball;
+	int fd = open("/sys/kernel/tracing/trace_marker", O_RDWR, 0);
+	int ret;
 
 	prctl(PR_SET_NAME, "referee", 0, 0, 0);
 	printf("Game On (%d seconds)!\n", game_length);
@@ -178,14 +183,19 @@ int referee(int game_length)
 	gettimeofday(&start, NULL);
 	now = start;
 
+	sprintf(buf, "I|%i|Game_Started!\n", getpid());
 	/* Start the game! */
 	atomic_set(0, &the_ball);
-
+	if (fd > 0)
+		ret = write(fd, buf, strnlen(buf, BUF_LEN));
 	/* Watch the game */
 	while ((now.tv_sec - start.tv_sec) < game_length) {
 		sleep(1);
 		gettimeofday(&now, NULL);
 	}
+	sprintf(buf, "I|%i|Game_Over!\n", getpid());
+	if (fd > 0)
+		ret = write(fd, buf, strnlen(buf, BUF_LEN));
 	final_ball = atomic_get(&the_ball);
 	/* Blow the whistle */
 	printf("Game Over!\n");
-- 
2.44.0.769.g3c40516874-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Cyril Hrubis]

Hi!
Wouldn't it make more sense to use the include/tst_atomic.h now instead
of the functions from librttest.h. At least the version in the
tst_atomic.h have proper memory barries for load, fallback in the case
that compiler does not support __atomic_foo(), etc.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 3/6] sched_football: Re-add the crazy fans to interrupt everyone

[Cyril Hrubis]

Hi!
>   *      This is a scheduler test that uses a football analogy.
>   *      The premise is that we want to make sure that lower priority threads
> - *      (defensive team). The offense is trying to increment the balls position,
> - *      while the defense is trying to block that from happening.
> + *      don't run while we have runnable higher priority threads.
> + *      The offense is trying to increment the balls position, while the
> + *      defense is trying to block that from happening.
>   *      And the ref (highest priority thread) will blow the wistle if the
>   *      ball moves. Finally, we have crazy fans (higer prority) that try to
>   *      distract the defense by occasionally running onto the field.
>   *
>   *      Steps:
> - *       - Create a fixed number of offense threads (lower priority)
> + *       - Create NR_CPU offense threads (lower priority)
> + *       - Create NR_CPU defense threads (mid priority)
> + *       - Create 2*NR_CPU fan threads (high priority)
>   *       - Create a referee thread (highest priority)
> - *       - Once everyone is on the field, the offense thread increments the
> - *         value of 'the_ball'. The defense thread tries to block
> - *         the ball by never letting the offense players get the CPU (it just
> - *         spins).
> + *       - Once everyone is on the field, the offense thread spins incrementing
> + *         the value of 'the_ball'. The defense thread tries to block the ball
> + *         by never letting the offense players get the CPU (it just spins).
> + *         The crazy fans sleep a bit, then jump the rail and run across the
> + *         field, disrupting the players on the field.
>   *       - The refree threads wakes up regularly to check if the game is over :)
>   *       - In the end, if the value of 'the_ball' is >0, the test is considered
>   *         to have failed.
> @@ -52,7 +56,7 @@
>   *		bugfixes and cleanups. -- Josh Triplett
>   *     2009-06-23 Simplified atomic startup mechanism, avoiding thundering herd
>   *		scheduling at the beginning of the game. -- Darren Hart
> - *
> + *     2024-04-23 Re-add crazy fans! Along with minor cleanups -- John Stultz

We use git now, so it's kind of pointless to add more changelog lines
into the file itself.

>   *****************************************************************************/
>  
>  #include <stdio.h>
> @@ -107,6 +111,38 @@ int parse_args(int c, char *v)
>  	return handled;
>  }
>  
> +#define NSEC_PER_SEC 1000000000ULL
> +unsigned long long ts_delta(struct timespec *start, struct timespec *stop)
> +{
> +	unsigned long long a, b;
> +
> +	a = start->tv_sec * NSEC_PER_SEC + start->tv_nsec;
> +	b = stop->tv_sec * NSEC_PER_SEC + stop->tv_nsec;
> +	return b - a;
> +}

This is tst_timespec_diff() from include/tst_timer.h

> +#define SPIN_TIME_NS 200000000
> +#define SLEEP_TIME_NS 50000000
> +/* These are fans running across the field. They're trying to interrupt/distract everyone */
> +void *thread_fan(void *arg)
> +{
> +	atomic_inc(&players_ready);
> +	/*occasionally wake up and run across the field */
> +	while (1) {
> +		int i;
> +		struct timespec start, stop;
> +
> +		start.tv_sec = 0;
> +		start.tv_nsec = SLEEP_TIME_NS;
> +		clock_nanosleep(CLOCK_MONOTONIC, 0, &start, NULL);
> +		clock_gettime(CLOCK_MONOTONIC, &start);
> +		clock_gettime(CLOCK_MONOTONIC, &stop);
> +		while (ts_delta(&start, &stop) < SPIN_TIME_NS)
> +			clock_gettime(CLOCK_MONOTONIC, &stop);
> +	}
> +	return NULL;
> +}
> +
>  /* This is the defensive team. They're trying to block the offense */
>  void *thread_defense(void *arg)
>  {
> @@ -199,6 +235,17 @@ int main(int argc, char *argv[])
>  	while (atomic_get(&players_ready) < players_per_team * 2)
>  		usleep(100);
>  
> +	/* Start the crazy fans*/
> +	priority = 50;
> +	printf("Starting %d fan threads at priority %d\n",
> +	       players_per_team, priority);
> +	for (i = 0; i < players_per_team*2; i++)
> +		create_fifo_thread(thread_fan, NULL, priority);
> +
> +	/* Wait for the crazy fan threads to start */
> +	while (atomic_get(&players_ready) < players_per_team * 4)
> +		usleep(100);
> +
>  	/* Ok, everyone is on the field, bring out the ref */
>  	printf("Starting referee thread\n");
>  	result = referee(game_length);
> -- 
> 2.44.0.769.g3c40516874-goog
> 
> 
> -- 
> Mailing list info: https://lists.linux.it/listinfo/ltp

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 6/6] sched_football: Add trace_marker messages if we're tracing

[Cyril Hrubis]

Hi!
> Cc: kernel-team@android.com
> Cc: Darren Hart <darren@os.amperecomputing.com>
> Signed-off-by: John Stultz <jstultz@google.com>
> ---
>  .../realtime/func/sched_football/sched_football.c    | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
> index 45fbf6766..ca44584b0 100644
> --- a/testcases/realtime/func/sched_football/sched_football.c
> +++ b/testcases/realtime/func/sched_football/sched_football.c
> @@ -67,6 +67,7 @@
>  #include <pthread.h>
>  #include <sched.h>
>  #include <errno.h>
> +#include <fcntl.h>
>  #include <sys/syscall.h>
>  #include <unistd.h>
>  #include <sys/prctl.h>
> @@ -167,10 +168,14 @@ void *thread_offense(void *arg)
>  	return NULL;
>  }
>  
> +#define BUF_LEN 256
>  int referee(int game_length)
>  {
>  	struct timeval start, now;
> +	char buf[BUF_LEN];
>  	int final_ball;
> +	int fd = open("/sys/kernel/tracing/trace_marker", O_RDWR, 0);
> +	int ret;
>  
>  	prctl(PR_SET_NAME, "referee", 0, 0, 0);
>  	printf("Game On (%d seconds)!\n", game_length);
> @@ -178,14 +183,19 @@ int referee(int game_length)
>  	gettimeofday(&start, NULL);
>  	now = start;
>  
> +	sprintf(buf, "I|%i|Game_Started!\n", getpid());
>  	/* Start the game! */
>  	atomic_set(0, &the_ball);
> -
> +	if (fd > 0)
> +		ret = write(fd, buf, strnlen(buf, BUF_LEN));
>  	/* Watch the game */
>  	while ((now.tv_sec - start.tv_sec) < game_length) {
>  		sleep(1);
>  		gettimeofday(&now, NULL);
>  	}
> +	sprintf(buf, "I|%i|Game_Over!\n", getpid());
> +	if (fd > 0)
> +		ret = write(fd, buf, strnlen(buf, BUF_LEN));

I know that we havre only two usages now, but it would be a bit cleaner
and readable to have a function to print the messages, e.g.

static void tracer_write(const char *msg);

And we would do:

	tracer_write("Game_Started!");

It would be a good idea to add the test name to the message as well,
maybe as:

	sprintf(buf, "sched_football|%i|%s\n", getpid(), msg);

>  	final_ball = atomic_get(&the_ball);
>  	/* Blow the whistle */
>  	printf("Game Over!\n");
> -- 
> 2.44.0.769.g3c40516874-goog
> 
> 
> -- 
> Mailing list info: https://lists.linux.it/listinfo/ltp

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

On Mon, Apr 29, 2024 at 2:07 AM Cyril Hrubis <chrubis@suse.cz> wrote:
>
> Hi!
> Wouldn't it make more sense to use the include/tst_atomic.h now instead
> of the functions from librttest.h. At least the version in the
> tst_atomic.h have proper memory barries for load, fallback in the case
> that compiler does not support __atomic_foo(), etc.

I don't object, but trying to include tst_atomic.h causes lots of
redefinition warnings for PACKAGE, PACKAGE_*, and VERSION which are
also set from librttest.h -> realtime_config.h

Suggestions on how to proceed there?

thanks
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 3/6] sched_football: Re-add the crazy fans to interrupt everyone

[John Stultz via ltp]

On Mon, Apr 29, 2024 at 2:11 AM Cyril Hrubis <chrubis@suse.cz> wrote:
>
> Hi!
> >   *      This is a scheduler test that uses a football analogy.
> >   *      The premise is that we want to make sure that lower priority threads
> > - *      (defensive team). The offense is trying to increment the balls position,
> > - *      while the defense is trying to block that from happening.
> > + *      don't run while we have runnable higher priority threads.
> > + *      The offense is trying to increment the balls position, while the
> > + *      defense is trying to block that from happening.
> >   *      And the ref (highest priority thread) will blow the wistle if the
> >   *      ball moves. Finally, we have crazy fans (higer prority) that try to
> >   *      distract the defense by occasionally running onto the field.
> >   *
> >   *      Steps:
> > - *       - Create a fixed number of offense threads (lower priority)
> > + *       - Create NR_CPU offense threads (lower priority)
> > + *       - Create NR_CPU defense threads (mid priority)
> > + *       - Create 2*NR_CPU fan threads (high priority)
> >   *       - Create a referee thread (highest priority)
> > - *       - Once everyone is on the field, the offense thread increments the
> > - *         value of 'the_ball'. The defense thread tries to block
> > - *         the ball by never letting the offense players get the CPU (it just
> > - *         spins).
> > + *       - Once everyone is on the field, the offense thread spins incrementing
> > + *         the value of 'the_ball'. The defense thread tries to block the ball
> > + *         by never letting the offense players get the CPU (it just spins).
> > + *         The crazy fans sleep a bit, then jump the rail and run across the
> > + *         field, disrupting the players on the field.
> >   *       - The refree threads wakes up regularly to check if the game is over :)
> >   *       - In the end, if the value of 'the_ball' is >0, the test is considered
> >   *         to have failed.
> > @@ -52,7 +56,7 @@
> >   *           bugfixes and cleanups. -- Josh Triplett
> >   *     2009-06-23 Simplified atomic startup mechanism, avoiding thundering herd
> >   *           scheduling at the beginning of the game. -- Darren Hart
> > - *
> > + *     2024-04-23 Re-add crazy fans! Along with minor cleanups -- John Stultz
>
> We use git now, so it's kind of pointless to add more changelog lines
> into the file itself.

Ack. Sounds good. Will drop.

> >
> > +#define NSEC_PER_SEC 1000000000ULL
> > +unsigned long long ts_delta(struct timespec *start, struct timespec *stop)
> > +{
> > +     unsigned long long a, b;
> > +
> > +     a = start->tv_sec * NSEC_PER_SEC + start->tv_nsec;
> > +     b = stop->tv_sec * NSEC_PER_SEC + stop->tv_nsec;
> > +     return b - a;
> > +}
>
> This is tst_timespec_diff() from include/tst_timer.h
>

Ah. I didn't even think to look, it's such a habit to hand write it in
my own small tests.

Though this runs into the same issue of conflicting defines from the
librttest.h, so your suggestions there would be appreciated.

I could probably utilize the ts_minus/ts_to_nsec logic as well in
librttest.h as an alternative.

Thanks for the review feedback!
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 6/6] sched_football: Add trace_marker messages if we're tracing

[John Stultz via ltp]

On Mon, Apr 29, 2024 at 2:18 AM Cyril Hrubis <chrubis@suse.cz> wrote:
> > +     sprintf(buf, "I|%i|Game_Over!\n", getpid());
> > +     if (fd > 0)
> > +             ret = write(fd, buf, strnlen(buf, BUF_LEN));
>
> I know that we havre only two usages now, but it would be a bit cleaner
> and readable to have a function to print the messages, e.g.
>
> static void tracer_write(const char *msg);
>
> And we would do:
>
>         tracer_write("Game_Started!");

Sure. For now I'll add it to the librttest code, but I'm happy to pull
it out to more generic logic once there's a good approach to the
config conflicts in the subsystem headers.

> It would be a good idea to add the test name to the message as well,
> maybe as:
>
>         sprintf(buf, "sched_football|%i|%s\n", getpid(), msg);
>

How about sprintf(buf, "I|%i|%s: %s\n", getpid(), test_tag, msg) ?

This is mostly because I'm using the trace_marker format that perfetto
understands, and can visualize.

thanks
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Cyril Hrubis]

Hi!
> > Wouldn't it make more sense to use the include/tst_atomic.h now instead
> > of the functions from librttest.h. At least the version in the
> > tst_atomic.h have proper memory barries for load, fallback in the case
> > that compiler does not support __atomic_foo(), etc.
> 
> I don't object, but trying to include tst_atomic.h causes lots of
> redefinition warnings for PACKAGE, PACKAGE_*, and VERSION which are
> also set from librttest.h -> realtime_config.h
> 
> Suggestions on how to proceed there?

The problem obviously is that we have two different config.h files that
conflicts.

Given that there isn't much in the realtime/configure.ac and these
checks does not seem to be relevant anymore it would make sense just to
get rid of it.

What we check for:

PTHREAD_PRIO_INHERIT - likely required for LinuxThreads vs NPTL
exp10 - seems to be supported in glibc since 2.1
__sync_add_and_fetch() - not needed when include/tst_atomic.h is used

Other than that the realtime/configure.ac contains stale data such as
defunct email to the ltp sourceforge mailing list which should be
removed anyways.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Cyril Hrubis]

Hi!
FYI the patch that removes the autotools from the testcases/realtime has
been merged, it should be possible to use the include/tst_atomic.h and
other headers without conflicts now.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

On Mon, Jun 24, 2024 at 3:45 AM Cyril Hrubis <chrubis@suse.cz> wrote:
>
> Hi!
> FYI the patch that removes the autotools from the testcases/realtime has
> been merged, it should be possible to use the include/tst_atomic.h and
> other headers without conflicts now.

Hey Cyril!

I went to take a stab at this, but the master branch on
https://github.com/linux-test-project/ltp.git doesn't seem to build:
  $ ./configure --with-realtime-testsuite
  $ make -j > /dev/null
make[3]: *** No rule to make target 'Makefile.am', needed by
'Makefile.in'.  Stop.
make[2]: *** [../../include/mk/generic_trunk_target.inc:108: all] Error 2
make[1]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
make: *** [Makefile:85: testcases-all] Error 2

Bisecting points to commit a05298ec4494 ("testcases: realtime: Get rid
of autotools")

I'm unfortunately not savvy enough with build systems to have much of
an idea of what is needed here.

Any suggestions?

thanks
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Li Wang]

Hi John,

On Tue, Jun 25, 2024 at 8:05 AM John Stultz via ltp <ltp@lists.linux.it>
wrote:

> On Mon, Jun 24, 2024 at 3:45 AM Cyril Hrubis <chrubis@suse.cz> wrote:
> >
> > Hi!
> > FYI the patch that removes the autotools from the testcases/realtime has
> > been merged, it should be possible to use the include/tst_atomic.h and
> > other headers without conflicts now.
>
> Hey Cyril!
>
> I went to take a stab at this, but the master branch on
> https://github.com/linux-test-project/ltp.git doesn't seem to build:
>   $ ./configure --with-realtime-testsuite
>   $ make -j > /dev/null
> make[3]: *** No rule to make target 'Makefile.am', needed by
> 'Makefile.in'.  Stop.
> make[2]: *** [../../include/mk/generic_trunk_target.inc:108: all] Error 2
> make[1]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
> make: *** [Makefile:85: testcases-all] Error 2
>
> Bisecting points to commit a05298ec4494 ("testcases: realtime: Get rid
> of autotools")
>
> I'm unfortunately not savvy enough with build systems to have much of
> an idea of what is needed here.
>
> Any suggestions?
>

Looks like you were trying to configure realtime-testsuite in LTP build.

Try this:

# cd ltp dir/
# make autotools
#  ./configure
#  make -C testcases/realtime/
See:
https://linux-test-project.readthedocs.io/en/latest/users/quick_start.html

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

On Mon, Jun 24, 2024 at 8:15 PM Li Wang <liwang@redhat.com> wrote:
> On Tue, Jun 25, 2024 at 8:05 AM John Stultz via ltp <ltp@lists.linux.it> wrote:
>> Hey Cyril!
>>
>> I went to take a stab at this, but the master branch on
>> https://github.com/linux-test-project/ltp.git doesn't seem to build:
>>   $ ./configure --with-realtime-testsuite
>>   $ make -j > /dev/null
>> make[3]: *** No rule to make target 'Makefile.am', needed by
>> 'Makefile.in'.  Stop.
>> make[2]: *** [../../include/mk/generic_trunk_target.inc:108: all] Error 2
>> make[1]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
>> make: *** [Makefile:85: testcases-all] Error 2
>>
>> Bisecting points to commit a05298ec4494 ("testcases: realtime: Get rid
>> of autotools")
>>
>> I'm unfortunately not savvy enough with build systems to have much of
>> an idea of what is needed here.
>>
>> Any suggestions?
>
>
> Looks like you were trying to configure realtime-testsuite in LTP build.

Yes, as previously that seemed to be required.


> Try this:
>
> # cd ltp dir/
> # make autotools
> #  ./configure
> #  make -C testcases/realtime/

Thank you! That does indeed work.

> See: https://linux-test-project.readthedocs.io/en/latest/users/quick_start.html

Should the old --with-realtime-testsuite option be removed then?

thanks again!
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Li Wang]

Hi John,

John Stultz <jstultz@google.com> wrote:

> Try this:
> >
> > # cd ltp dir/
> > # make autotools
> > #  ./configure
> > #  make -C testcases/realtime/
>
> Thank you! That does indeed work.
>
> > See:
> https://linux-test-project.readthedocs.io/en/latest/users/quick_start.html
>
> Should the old --with-realtime-testsuite option be removed then?
>

No, we need it.

When configuring LTP without the --with-realtime-testsuite option, it will
skip building the stuff under testcases/realtime/ by default.

Unless you build it manually like: make -C testcase/realtime/.

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

On Tue, Jun 25, 2024 at 7:26 PM Li Wang <liwang@redhat.com> wrote:
> John Stultz <jstultz@google.com> wrote:
>>
>> Should the old --with-realtime-testsuite option be removed then?
>
>
> No, we need it.
>
> When configuring LTP without the --with-realtime-testsuite option, it will
> skip building the stuff under testcases/realtime/ by default.

Right. But as I mentioned earlier, building with it seems to be broken
currently?

> Unless you build it manually like: make -C testcase/realtime/.

Should maybe the realtime tests just be included in the normal build
now? Or is it still useful to keep it separate due to some of the
additional dependencies it needs?

thanks
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Li Wang]

Hi John,

On Thu, Jun 27, 2024 at 1:01 AM John Stultz <jstultz@google.com> wrote:

> On Tue, Jun 25, 2024 at 7:26 PM Li Wang <liwang@redhat.com> wrote:
> > John Stultz <jstultz@google.com> wrote:
> >>
> >> Should the old --with-realtime-testsuite option be removed then?
> >
> >
> > No, we need it.
> >
> > When configuring LTP without the --with-realtime-testsuite option, it
> will
> > skip building the stuff under testcases/realtime/ by default.
>
> Right. But as I mentioned earlier, building with it seems to be broken
> currently?
>

Not broken, that's because you didn't build in correct way.
'make autotools' is the first necessary step for the build.


> > Unless you build it manually like: make -C testcase/realtime/.
>
> Should maybe the realtime tests just be included in the normal build
> now? Or is it still useful to keep it separate due to some of the
> additional dependencies it needs?
>

Hmm, at least in the build.sh script, it enables in the configuration phase,
if you build by that (./build.sh) it should be included.

But I'm not very sure if it's worth to enable by default.  @Cryil, @Petr.

See: https://github.com/linux-test-project/ltp/blob/master/build.sh#L14

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Cyril Hrubis]

Hi!
> Hmm, at least in the build.sh script, it enables in the configuration phase,
> if you build by that (./build.sh) it should be included.
> 
> But I'm not very sure if it's worth to enable by default.  @Cryil, @Petr.

I do not think that it actually makes a big sense to have the configure
switch for something so small as the realtime testsuite. It's ~30 tests
that are compiled in a few seconds and it does not add too much space to
the installed LTP as well.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 3/6] sched_football: Re-add the crazy fans to interrupt everyone

[Martin Doucha]

On 29. 04. 24 11:10, Cyril Hrubis wrote:
> Hi!
>> @@ -107,6 +111,38 @@ int parse_args(int c, char *v)
>>   	return handled;
>>   }
>>   
>> +#define NSEC_PER_SEC 1000000000ULL
>> +unsigned long long ts_delta(struct timespec *start, struct timespec *stop)
>> +{
>> +	unsigned long long a, b;
>> +
>> +	a = start->tv_sec * NSEC_PER_SEC + start->tv_nsec;
>> +	b = stop->tv_sec * NSEC_PER_SEC + stop->tv_nsec;
>> +	return b - a;
>> +}
> 
> This is tst_timespec_diff() from include/tst_timer.h

... which cannot be used in this test without porting it to new LTP API 
first. Otherwise you'll get undefined reference to tst_brk_().

-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 3/6] sched_football: Re-add the crazy fans to interrupt everyone

[Cyril Hrubis]

Hi!
> > This is tst_timespec_diff() from include/tst_timer.h
> 
> ... which cannot be used in this test without porting it to new LTP API 
> first. Otherwise you'll get undefined reference to tst_brk_().

Already send a patch to enable this. And I'm going to send v2 right now.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

On Wed, Jun 26, 2024 at 8:32 PM Li Wang <liwang@redhat.com> wrote:
> On Thu, Jun 27, 2024 at 1:01 AM John Stultz <jstultz@google.com> wrote:
>> On Tue, Jun 25, 2024 at 7:26 PM Li Wang <liwang@redhat.com> wrote:
>> > John Stultz <jstultz@google.com> wrote:
>> >>
>> >> Should the old --with-realtime-testsuite option be removed then?
>> >
>> >
>> > No, we need it.
>> >
>> > When configuring LTP without the --with-realtime-testsuite option, it will
>> > skip building the stuff under testcases/realtime/ by default.
>>
>> Right. But as I mentioned earlier, building with it seems to be broken
>> currently?
>
>
> Not broken, that's because you didn't build in correct way.
> 'make autotools' is the first necessary step for the build.
>

Ok, but even after doing that as you suggested, building with the
following fails for me:

make autotools
./configure -C testcases/realtime/
make -j
...
make[3]: *** No rule to make target 'Makefile.am', needed by
'Makefile.in'.  Stop.
make[2]: *** [../../include/mk/generic_trunk_target.inc:108: all] Error 2
make[1]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
make: *** [Makefile:85: testcases-all] Error 2

So it does *seem* like something is broken. :)

thanks
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

On Thu, Jun 27, 2024 at 11:01 AM John Stultz <jstultz@google.com> wrote:
>
> On Wed, Jun 26, 2024 at 8:32 PM Li Wang <liwang@redhat.com> wrote:
> > On Thu, Jun 27, 2024 at 1:01 AM John Stultz <jstultz@google.com> wrote:
> >> On Tue, Jun 25, 2024 at 7:26 PM Li Wang <liwang@redhat.com> wrote:
> >> > John Stultz <jstultz@google.com> wrote:
> >> >>
> >> >> Should the old --with-realtime-testsuite option be removed then?
> >> >
> >> >
> >> > No, we need it.
> >> >
> >> > When configuring LTP without the --with-realtime-testsuite option, it will
> >> > skip building the stuff under testcases/realtime/ by default.
> >>
> >> Right. But as I mentioned earlier, building with it seems to be broken
> >> currently?
> >
> >
> > Not broken, that's because you didn't build in correct way.
> > 'make autotools' is the first necessary step for the build.
> >
>
> Ok, but even after doing that as you suggested, building with the
> following fails for me:
>
> make autotools
> ./configure -C testcases/realtime/

Oh, sorry, I cross wires typing this out. It was
./configure --with-realtime-testsuite

> make -j
> ...
> make[3]: *** No rule to make target 'Makefile.am', needed by
> 'Makefile.in'.  Stop.
> make[2]: *** [../../include/mk/generic_trunk_target.inc:108: all] Error 2
> make[1]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
> make: *** [Makefile:85: testcases-all] Error 2
>
> So it does *seem* like something is broken. :)
>
> thanks
> -john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[Cyril Hrubis]

Hi!
> Ok, but even after doing that as you suggested, building with the
> following fails for me:
> 
> make autotools
> ./configure -C testcases/realtime/
> make -j
> ...
> make[3]: *** No rule to make target 'Makefile.am', needed by
> 'Makefile.in'.  Stop.
> make[2]: *** [../../include/mk/generic_trunk_target.inc:108: all] Error 2
> make[1]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
> make: *** [Makefile:85: testcases-all] Error 2
> 
> So it does *seem* like something is broken. :)

I can't reproduce that and it didn't show up on our CI that builds LTP
on 16 different distributions.

And the error message you get does not make much sense either. The
Makefile.am file is only referenced in the automake target. It shouldn't
be referenced from the testcases-all target at all.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 2/6] sched_football: Use atomic for ball

[John Stultz via ltp]

On Fri, Jun 28, 2024 at 1:12 AM Cyril Hrubis <chrubis@suse.cz> wrote:
> > Ok, but even after doing that as you suggested, building with the
> > following fails for me:
> >
> > make autotools
> > ./configure -C testcases/realtime/
> > make -j
> > ...
> > make[3]: *** No rule to make target 'Makefile.am', needed by
> > 'Makefile.in'.  Stop.
> > make[2]: *** [../../include/mk/generic_trunk_target.inc:108: all] Error 2
> > make[1]: *** [../include/mk/generic_trunk_target.inc:108: all] Error 2
> > make: *** [Makefile:85: testcases-all] Error 2
> >
> > So it does *seem* like something is broken. :)
>
> I can't reproduce that and it didn't show up on our CI that builds LTP
> on 16 different distributions.

Interesting. Thanks for checking on this! I appreciate your time in
trying to reproduce it.

> And the error message you get does not make much sense either. The
> Makefile.am file is only referenced in the automake target. It shouldn't
> be referenced from the testcases-all target at all.

So digging further, I think this has to do with some sort of state
from prior to commit a05298ec4494 sticking around and causing problems
in the build.

I had been using both `make clean` and `make distclean` and even `git
clean -f -d` (as I noticed there were often `git status` leftover bits
after the distclean), but even then I would trip the problem above.

But re-checking out a clean repo, I indeed don't hit the problem.

After distcleaning and git clean'ing both, the diffstat looks like:

* A bunch of confXXXXXX directories littering the old tree.
 include/config.h.in~                                                   |  656
 testcases/kernel/syscalls/delete_module/.dummy_del_mod.o.d             |    2
 testcases/open_posix_testsuite/autom4te.cache/requests                 |  102
 testcases/open_posix_testsuite/conformance/definitions/mqueue_h/null.o |binary
 testcases/open_posix_testsuite/functional/timers/timers/Makefile       |   94
 testcases/realtime/aclocal.m4                                          | 1165 -
 testcases/realtime/autom4te.cache/output.0
 | 7058 ------
 testcases/realtime/autom4te.cache/output.1
 | 7059 ------
 testcases/realtime/autom4te.cache/output.2
 | 7058 ------
 testcases/realtime/autom4te.cache/requests                             |  238
 testcases/realtime/autom4te.cache/traces.0                             | 1020 -
 testcases/realtime/autom4te.cache/traces.1                             |  491
 testcases/realtime/autom4te.cache/traces.2                             |  491
 testcases/realtime/compile                                             |  348
 testcases/realtime/config.guess                                        | 1754 -
 testcases/realtime/config.log                                          |  474
 testcases/realtime/config.status                                       | 1197 -
 testcases/realtime/config.sub                                          | 1890 -
 testcases/realtime/configure
 | 7059 ------
 testcases/realtime/include/realtime_config.h                           |   70
 testcases/realtime/include/realtime_config.h.in                        |   69
 testcases/realtime/include/stamp-h1                                    |    1
 testcases/realtime/install-sh                                          |  541
 testcases/realtime/m4/Makefile.in                                      |  411
 testcases/realtime/missing                                             |  215

So I'm guessing that leftover testcases/realtime/m4/Makefile.in is the issue.
And I'm guessing the change where I saw things break removed the
realtime dir from AUTOCONFED_SUBDIRS so it wouldn't get cleaned
afterwards if it hadn't been before.

I guess in my bisection I should have done the distclean prior to
switching branches, not first thing after.  Or I guess I should have
tried with git clean -x?

Well, apologies for the noise. I greatly appreciate the help resolving this!

thanks
-john

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp