[LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

Previously, thread synchronization in sched_football only relied on a
thread_barrier. This ensured that all threads were created before the
referee started the game, but did not fully prevent offense threads from
getting a scheduling opportunity before defense threads were migrated,
leading to occasional non-zero final ball positions on kvm or debug kernels.

This patch introduces an explicit `kickoff_flag`:

* All player threads (offense, defense, fans) wait at the barrier and
  then spin until the referee sets `kickoff_flag`. This reduces kernel
  scheduling skew, as threads only proceed once the referee explicitly
  signals the kickoff.

* The referee now:
  - Waits at the barrier.
  - Clears the ball position.
  - Sets `kickoff_flag` to start the ball.

* Game termination is also slightly reordered (by Cyril):
  - Final ball position is read before `game_over` is set,
    avoiding a race where the ball could still move right after
    defense threads stop.

* Only test on RT-kernels.

This makes startup sequencing more deterministic while still allowing
some nondeterminism, which is intentional for testing scheduler
behavior under load.

Signed-off-by: Li Wang <liwang@redhat.com>
Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
Tested-by: Andrea Cervesato <andrea.cervesato@suse.com>
Reviewed-by: Petr Vorel <pvorel@suse.cz>
---

Notes:
    V1 -> V2:
    	* add usleep(20000) to increase the chances of each player
    	  being active before the kickoff.

 .../func/sched_football/sched_football.c      | 28 +++++++++++++++----
 1 file changed, 22 insertions(+), 6 deletions(-)

diff --git a/testcases/realtime/func/sched_football/sched_football.c b/testcases/realtime/func/sched_football/sched_football.c
index 0617bdb87..4a72874ce 100644
--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -44,6 +44,7 @@
 static tst_atomic_t the_ball;
 static int players_per_team = 0;
 static int game_length = DEF_GAME_LENGTH;
+static tst_atomic_t kickoff_flag;
 static tst_atomic_t game_over;
 
 static char *str_game_length;
@@ -55,6 +56,9 @@ void *thread_fan(void *arg LTP_ATTRIBUTE_UNUSED)
 {
 	prctl(PR_SET_NAME, "crazy_fan", 0, 0, 0);
 	pthread_barrier_wait(&start_barrier);
+	while (!tst_atomic_load(&kickoff_flag))
+		;
+
 	/*occasionally wake up and run across the field */
 	while (!tst_atomic_load(&game_over)) {
 		struct timespec start, stop;
@@ -80,6 +84,9 @@ void *thread_defense(void *arg LTP_ATTRIBUTE_UNUSED)
 {
 	prctl(PR_SET_NAME, "defense", 0, 0, 0);
 	pthread_barrier_wait(&start_barrier);
+	while (!tst_atomic_load(&kickoff_flag))
+		;
+
 	/*keep the ball from being moved */
 	while (!tst_atomic_load(&game_over)) {
 	}
@@ -92,6 +99,9 @@ void *thread_offense(void *arg LTP_ATTRIBUTE_UNUSED)
 {
 	prctl(PR_SET_NAME, "offense", 0, 0, 0);
 	pthread_barrier_wait(&start_barrier);
+	while (!tst_atomic_load(&kickoff_flag))
+		;
+
 	while (!tst_atomic_load(&game_over)) {
 		tst_atomic_add_return(1, &the_ball); /* move the ball ahead one yard */
 	}
@@ -115,9 +125,11 @@ void referee(int game_length)
 	now = start;
 
 	/* Start the game! */
-	tst_atomic_store(0, &the_ball);
-	pthread_barrier_wait(&start_barrier);
 	atrace_marker_write("sched_football", "Game_started!");
+	pthread_barrier_wait(&start_barrier);
+	tst_atomic_store(0, &the_ball);
+	tst_atomic_store(1, &kickoff_flag);
+	usleep(20000);
 
 	/* Watch the game */
 	while ((now.tv_sec - start.tv_sec) < game_length) {
@@ -125,14 +137,14 @@ void referee(int game_length)
 		gettimeofday(&now, NULL);
 	}
 
-	/* Stop the game! */
-	tst_atomic_store(1, &game_over);
-	atrace_marker_write("sched_football", "Game_Over!");
-
 	/* Blow the whistle */
 	final_ball = tst_atomic_load(&the_ball);
 	tst_res(TINFO, "Final ball position: %d", final_ball);
 
+	/* Stop the game! */
+	tst_atomic_store(1, &game_over);
+	atrace_marker_write("sched_football", "Game_Over!");
+
 	TST_EXP_EXPR(final_ball == 0);
 }
 
@@ -154,6 +166,7 @@ static void do_test(void)
 	/* We're the ref, so set our priority right */
 	param.sched_priority = sched_get_priority_min(SCHED_FIFO) + 80;
 	sched_setscheduler(0, SCHED_FIFO, &param);
+	tst_atomic_store(0, &kickoff_flag);
 
 	/*
 	 * Start the offense
@@ -186,6 +199,9 @@ static void do_test(void)
 
 static void do_setup(void)
 {
+	if (!tst_check_preempt_rt())
+		tst_brk(TCONF, "Test requires real-time kernel");
+
 	if (tst_parse_int(str_game_length, &game_length, 1, INT_MAX))
 		tst_brk(TBROK, "Invalid game length '%s'", str_game_length);
 
-- 
2.51.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Petr Vorel]

Hi Li,

...
>  static void do_setup(void)
>  {
> +	if (!tst_check_preempt_rt())
> +		tst_brk(TCONF, "Test requires real-time kernel");

I understood Cyril is really suggesting to keep it [1]. I would also vote to
keep it (we still have some time to see if it got fixed before release).

I know we had this discussion in the past (some of your colleague suggesting it
should not be run on non-RT kernel), so I'm not pushing for it.

Kind regards,
Petr

[1] https://lore.kernel.org/ltp/aLllCF-dioIEJftH@yuki.lan/


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Cyril Hrubis]

Hi!
> >  static void do_setup(void)
> >  {
> > +	if (!tst_check_preempt_rt())
> > +		tst_brk(TCONF, "Test requires real-time kernel");
> 
> I understood Cyril is really suggesting to keep it [1]. I would also vote to
> keep it (we still have some time to see if it got fixed before release).
> 
> I know we had this discussion in the past (some of your colleague suggesting it
> should not be run on non-RT kernel), so I'm not pushing for it.

I stil do not understand reasons for disabling the test. The POSIX
realtime schedulling classes have to work properly regardless of the
kernel flavor. Why should we turn the test off on non-rt kernel then?

It may make sense to prolong the settling period for non-rt something
as:

...
	if (tst_check_preempt_rt()
		settling_period = 20000;
	else
		settling_period = 200000;


...
	usleep(settling_period);
...


In order to make sure non-rt scheduller has enough time to shuffle the
processes around the available CPUs. But that should be the only
difference.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Thu, Sep 4, 2025 at 7:42 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > >  static void do_setup(void)
> > >  {
> > > +   if (!tst_check_preempt_rt())
> > > +           tst_brk(TCONF, "Test requires real-time kernel");
> >
> > I understood Cyril is really suggesting to keep it [1]. I would also
> vote to
> > keep it (we still have some time to see if it got fixed before release).
> >
> > I know we had this discussion in the past (some of your colleague
> suggesting it
> > should not be run on non-RT kernel), so I'm not pushing for it.
>
> I stil do not understand reasons for disabling the test. The POSIX
> realtime schedulling classes have to work properly regardless of the
> kernel flavor. Why should we turn the test off on non-rt kernel then?
>

No special reasons. I still can sporadically catch the failure on non-RT
kernel with even sleep 2 seconds.

Thus, I took this very extreme approach, because on non-RT kernel, sleep
may not have a particularly perfect effect, I guess the stock kernel with
sched_setscheduler(, SCHED_FIFO, ) is still has scheduling skew with
workload.

But the RT kernel is more aggressive in scheduling, and it is almost
impossible
to encounter false positives with that enhancement.


--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -130,6 +130,8 @@ void referee(int game_length)
        pthread_barrier_wait(&start_barrier);
        tst_atomic_store(0, &the_ball);
        tst_atomic_store(1, &kickoff_flag);
+       usleep(2000000);
+

        /* Watch the game */
        while ((now.tv_sec - start.tv_sec) < game_length) {
@@ -199,9 +201,6 @@ static void do_test(void)

 static void do_setup(void)
 {
-       if (!tst_check_preempt_rt())
-               tst_brk(TCONF, "Test requires real-time kernel");
-
        if (tst_parse_int(str_game_length, &game_length, 1, INT_MAX))
                tst_brk(TBROK, "Invalid game length '%s'", str_game_length);

# ./sched_football
tst_test.c:2004: TINFO: LTP version: 20250530
tst_test.c:2007: TINFO: Tested kernel: 6.12.0-55.29.1.el10_0.x86_64
                                  #1 SMP PREEMPT_DYNAMIC Mon Aug 18
05:37:24 EDT 2025 x86_64
tst_kconfig.c:88: TINFO: Parsing kernel config
'/lib/modules/6.12.0-55.29.1.el10_0.x86_64/config'
tst_test.c:1825: TINFO: Overall timeout per run is 0h 00m 30s
sched_football.c:162: TINFO: players_per_team: 32 game_length: 5
sched_football.c:178: TINFO: Starting 32 offense threads at priority 15
sched_football.c:185: TINFO: Starting 32 defense threads at priority 30
sched_football.c:192: TINFO: Starting 64 crazy-fan threads at priority 50
sched_football.c:118: TINFO: Starting referee thread
sched_football.c:121: TINFO: Starting the game (5 sec)
sched_football.c:144: TINFO: Final ball position: 16761
sched_football.c:150: TFAIL: Expect: final_ball == 0



>
> It may make sense to prolong the settling period for non-rt something
> as:
>
> ...
>         if (tst_check_preempt_rt()
>                 settling_period = 20000;
>         else
>                 settling_period = 200000;
>
>
> ...
>         usleep(settling_period);
> ...
>
>
> In order to make sure non-rt scheduller has enough time to shuffle the
> processes around the available CPUs. But that should be the only
> difference.
>
> --
> Cyril Hrubis
> chrubis@suse.cz
>
>

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Cyril Hrubis]

Hi!
> > > >  static void do_setup(void)
> > > >  {
> > > > +   if (!tst_check_preempt_rt())
> > > > +           tst_brk(TCONF, "Test requires real-time kernel");
> > >
> > > I understood Cyril is really suggesting to keep it [1]. I would also
> > vote to
> > > keep it (we still have some time to see if it got fixed before release).
> > >
> > > I know we had this discussion in the past (some of your colleague
> > suggesting it
> > > should not be run on non-RT kernel), so I'm not pushing for it.
> >
> > I stil do not understand reasons for disabling the test. The POSIX
> > realtime schedulling classes have to work properly regardless of the
> > kernel flavor. Why should we turn the test off on non-rt kernel then?
> >
> 
> No special reasons. I still can sporadically catch the failure on non-RT
> kernel with even sleep 2 seconds.

That is very strange. The SCHED_FIFO threads should preempty any lower
prio thread as long as they become runable and should stay running until
they finish or yield. Two seconds should be more than enough for that to
happen.

> Thus, I took this very extreme approach, because on non-RT kernel, sleep
> may not have a particularly perfect effect, I guess the stock kernel with
> sched_setscheduler(, SCHED_FIFO, ) is still has scheduling skew with
> workload.

Does this happen on vanilla Linux as well or only on RedHat kernels?

Because AFAIK the test started to fail on SUSE only after the change
that introduced barriers.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Petr Vorel]

Hi,


> On Thu, Sep 4, 2025 at 7:42 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> > Hi!
> > > >  static void do_setup(void)
> > > >  {
> > > > +   if (!tst_check_preempt_rt())
> > > > +           tst_brk(TCONF, "Test requires real-time kernel");

> > > I understood Cyril is really suggesting to keep it [1]. I would also
> > vote to
> > > keep it (we still have some time to see if it got fixed before release).

> > > I know we had this discussion in the past (some of your colleague
> > suggesting it
> > > should not be run on non-RT kernel), so I'm not pushing for it.

> > I stil do not understand reasons for disabling the test. The POSIX
> > realtime schedulling classes have to work properly regardless of the
> > kernel flavor. Why should we turn the test off on non-rt kernel then?


> No special reasons. I still can sporadically catch the failure on non-RT
> kernel with even sleep 2 seconds.

> Thus, I took this very extreme approach, because on non-RT kernel, sleep
> may not have a particularly perfect effect, I guess the stock kernel with
> sched_setscheduler(, SCHED_FIFO, ) is still has scheduling skew with
> workload.

> But the RT kernel is more aggressive in scheduling, and it is almost
> impossible
> to encounter false positives with that enhancement.


> --- a/testcases/realtime/func/sched_football/sched_football.c
> +++ b/testcases/realtime/func/sched_football/sched_football.c
> @@ -130,6 +130,8 @@ void referee(int game_length)
>         pthread_barrier_wait(&start_barrier);
>         tst_atomic_store(0, &the_ball);
>         tst_atomic_store(1, &kickoff_flag);
> +       usleep(2000000);
	
> +

>         /* Watch the game */
>         while ((now.tv_sec - start.tv_sec) < game_length) {
> @@ -199,9 +201,6 @@ static void do_test(void)

>  static void do_setup(void)
>  {
> -       if (!tst_check_preempt_rt())
> -               tst_brk(TCONF, "Test requires real-time kernel");
> -
>         if (tst_parse_int(str_game_length, &game_length, 1, INT_MAX))
>                 tst_brk(TBROK, "Invalid game length '%s'", str_game_length);

> # ./sched_football
> tst_test.c:2004: TINFO: LTP version: 20250530
> tst_test.c:2007: TINFO: Tested kernel: 6.12.0-55.29.1.el10_0.x86_64
>                                   #1 SMP PREEMPT_DYNAMIC Mon Aug 18
> 05:37:24 EDT 2025 x86_64
> tst_kconfig.c:88: TINFO: Parsing kernel config
> '/lib/modules/6.12.0-55.29.1.el10_0.x86_64/config'
> tst_test.c:1825: TINFO: Overall timeout per run is 0h 00m 30s
> sched_football.c:162: TINFO: players_per_team: 32 game_length: 5
> sched_football.c:178: TINFO: Starting 32 offense threads at priority 15
> sched_football.c:185: TINFO: Starting 32 defense threads at priority 30
> sched_football.c:192: TINFO: Starting 64 crazy-fan threads at priority 50
> sched_football.c:118: TINFO: Starting referee thread
> sched_football.c:121: TINFO: Starting the game (5 sec)
> sched_football.c:144: TINFO: Final ball position: 16761
> sched_football.c:150: TFAIL: Expect: final_ball == 0

OK even 2 s sleep does not work. I'm out of ideas what in the test is wrong.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Thu, Sep 4, 2025 at 11:28 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > > > >  static void do_setup(void)
> > > > >  {
> > > > > +   if (!tst_check_preempt_rt())
> > > > > +           tst_brk(TCONF, "Test requires real-time kernel");
> > > >
> > > > I understood Cyril is really suggesting to keep it [1]. I would also
> > > vote to
> > > > keep it (we still have some time to see if it got fixed before
> release).
> > > >
> > > > I know we had this discussion in the past (some of your colleague
> > > suggesting it
> > > > should not be run on non-RT kernel), so I'm not pushing for it.
> > >
> > > I stil do not understand reasons for disabling the test. The POSIX
> > > realtime schedulling classes have to work properly regardless of the
> > > kernel flavor. Why should we turn the test off on non-rt kernel then?
> > >
> >
> > No special reasons. I still can sporadically catch the failure on non-RT
> > kernel with even sleep 2 seconds.
>
> That is very strange. The SCHED_FIFO threads should preempty any lower
> prio thread as long as they become runable and should stay running until
> they finish or yield. Two seconds should be more than enough for that to
> happen.
>
> > Thus, I took this very extreme approach, because on non-RT kernel, sleep
> > may not have a particularly perfect effect, I guess the stock kernel with
> > sched_setscheduler(, SCHED_FIFO, ) is still has scheduling skew with
> > workload.
>
> Does this happen on vanilla Linux as well or only on RedHat kernels?
>

Yes, both vanilla Linux and CentOS kernels.

more CI test history for sched_football:

   Without barrier patch:
           Fails on both RT and non-RT CentOS Stream 9/10 kernels
           Fails on non-RT mainline v6.17-rc4 kernels. (not build v6.17 RT
kernel)

  (^ that's why we started to look into the failure and submit the commit
e523ba88dd9b)

   With barrier patch:
           Fails on both RT and non-RT CentOS kernels, but the final ball
position is noticeably lower.
           Fails on non-RT mainline v6.17-rc4 kernels. (not build v6.17 RT
kernel)

   With barrier patch + kickoff flag enhancement:
           Fails on the non-RT CentOS Stream 10 kernel
           Fails on non-RT mainline v6.17-rc4 kernel. (not build v6.17
RT-kernel)
           Passed on RT CentOS stream kernel

(^ here I started to suspect the SCHED_FIFO threads can not perform well
like RT-kernel)


[root@dell-per7625-01 sched_football]# uname -r
6.17.0-rc4.liwang

[root@dell-per7625-01 sched_football]# ./sched_football
tst_test.c:2004: TINFO: LTP version: 20250530
tst_test.c:2007: TINFO: Tested kernel: 6.17.0-rc4.liwang #1 SMP
PREEMPT_DYNAMIC Thu Sep  4 20:07:20 EDT 2025 x86_64
tst_kconfig.c:88: TINFO: Parsing kernel config
'/lib/modules/6.17.0-rc4.liwang/build/.config'
tst_kconfig.c:676: TINFO: CONFIG_FAULT_INJECTION kernel option detected
which might slow the execution
tst_test.c:1825: TINFO: Overall timeout per run is 0h 02m 00s
sched_football.c:162: TINFO: players_per_team: 32 game_length: 5
sched_football.c:178: TINFO: Starting 32 offense threads at priority 15
sched_football.c:185: TINFO: Starting 32 defense threads at priority 30
sched_football.c:192: TINFO: Starting 64 crazy-fan threads at priority 50
sched_football.c:118: TINFO: Starting referee thread
sched_football.c:121: TINFO: Starting the game (5 sec)
sched_football.c:144: TINFO: Final ball position: 20205
sched_football.c:150: TFAIL: Expect: final_ball == 0

Summary:
passed   0
failed   1
broken   0
skipped  0
warnings 0

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Fri, Sep 5, 2025 at 8:54 AM Li Wang <liwang@redhat.com> wrote:

>
>
> On Thu, Sep 4, 2025 at 11:28 PM Cyril Hrubis <chrubis@suse.cz> wrote:
>
>> Hi!
>> > > > >  static void do_setup(void)
>> > > > >  {
>> > > > > +   if (!tst_check_preempt_rt())
>> > > > > +           tst_brk(TCONF, "Test requires real-time kernel");
>> > > >
>> > > > I understood Cyril is really suggesting to keep it [1]. I would also
>> > > vote to
>> > > > keep it (we still have some time to see if it got fixed before
>> release).
>> > > >
>> > > > I know we had this discussion in the past (some of your colleague
>> > > suggesting it
>> > > > should not be run on non-RT kernel), so I'm not pushing for it.
>> > >
>> > > I stil do not understand reasons for disabling the test. The POSIX
>> > > realtime schedulling classes have to work properly regardless of the
>> > > kernel flavor. Why should we turn the test off on non-rt kernel then?
>> > >
>> >
>> > No special reasons. I still can sporadically catch the failure on non-RT
>> > kernel with even sleep 2 seconds.
>>
>> That is very strange. The SCHED_FIFO threads should preempty any lower
>> prio thread as long as they become runable and should stay running until
>> they finish or yield. Two seconds should be more than enough for that to
>> happen.
>>
>> > Thus, I took this very extreme approach, because on non-RT kernel, sleep
>> > may not have a particularly perfect effect, I guess the stock kernel
>> with
>> > sched_setscheduler(, SCHED_FIFO, ) is still has scheduling skew with
>> > workload.
>>
>> Does this happen on vanilla Linux as well or only on RedHat kernels?
>>
>
> Yes, both vanilla Linux and CentOS kernels.
>
> more CI test history for sched_football:
>
>    Without barrier patch:
>            Fails on both RT and non-RT CentOS Stream 9/10 kernels
>            Fails on non-RT mainline v6.17-rc4 kernels. (not build v6.17
> RT kernel)
>
>   (^ that's why we started to look into the failure and submit the commit
> e523ba88dd9b)
>
>    With barrier patch:
>            Fails on both RT and non-RT CentOS kernels, but the final ball
> position is noticeably lower.
>            Fails on non-RT mainline v6.17-rc4 kernels. (not build v6.17
> RT kernel)
>
>    With barrier patch + kickoff flag enhancement:
>            Fails on the non-RT CentOS Stream 10 kernel
>            Fails on non-RT mainline v6.17-rc4 kernel. (not build v6.17
> RT-kernel)
>            Passed on RT CentOS stream kernel
>
> (^ here I started to suspect the SCHED_FIFO threads can not perform well
> like RT-kernel)
>
>
> [root@dell-per7625-01 sched_football]# uname -r
> 6.17.0-rc4.liwang
>
> [root@dell-per7625-01 sched_football]# ./sched_football
> tst_test.c:2004: TINFO: LTP version: 20250530
> tst_test.c:2007: TINFO: Tested kernel: 6.17.0-rc4.liwang #1 SMP
> PREEMPT_DYNAMIC Thu Sep  4 20:07:20 EDT 2025 x86_64
> tst_kconfig.c:88: TINFO: Parsing kernel config
> '/lib/modules/6.17.0-rc4.liwang/build/.config'
> tst_kconfig.c:676: TINFO: CONFIG_FAULT_INJECTION kernel option detected
> which might slow the execution
> tst_test.c:1825: TINFO: Overall timeout per run is 0h 02m 00s
> sched_football.c:162: TINFO: players_per_team: 32 game_length: 5
> sched_football.c:178: TINFO: Starting 32 offense threads at priority 15
> sched_football.c:185: TINFO: Starting 32 defense threads at priority 30
> sched_football.c:192: TINFO: Starting 64 crazy-fan threads at priority 50
> sched_football.c:118: TINFO: Starting referee thread
> sched_football.c:121: TINFO: Starting the game (5 sec)
> sched_football.c:144: TINFO: Final ball position: 20205
> sched_football.c:150: TFAIL: Expect: final_ball == 0
>


Checking the configurations of the stock kernel and the real-time
kernel, the stock kernel uses "CONFIG_PREEMPT_VOLUNTARY=y,"
which only provides voluntary preemption.

This preemption model is designed to strike a balance between throughput
and latency. It only allows the kernel to be preempted at specific, well
defined
"safe points," potentially resulting in long, unbounded latencies.

However, the sched_football test was most likely designed to measure or
stress-test the deterministic, low-latency scheduling behavior that is
characteristic of real-time (RT) kernel.

So, I tend to believe the test's failure on the stock kernel is acceptable.
And, by the way, what does the SUSE kernel configuration look like?


# grep CONFIG_PREEMPT /boot/config-6.12.0-55.29.1.el10_0.x86_64
CONFIG_PREEMPT_BUILD=y
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y

(^ I built my v6.17-rc4 with this config too)


# grep CONFIG_PREEMPT /boot/config-6.12.0-55.31.1.el10_0.x86_64+rt
CONFIG_PREEMPT_BUILD=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
# CONFIG_PREEMPT_DYNAMIC is not set
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y


-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Fri, Sep 5, 2025 at 12:03 PM Li Wang <liwang@redhat.com> wrote:

>
>
> On Fri, Sep 5, 2025 at 8:54 AM Li Wang <liwang@redhat.com> wrote:
>
>>
>>
>> On Thu, Sep 4, 2025 at 11:28 PM Cyril Hrubis <chrubis@suse.cz> wrote:
>>
>>> Hi!
>>> > > > >  static void do_setup(void)
>>> > > > >  {
>>> > > > > +   if (!tst_check_preempt_rt())
>>> > > > > +           tst_brk(TCONF, "Test requires real-time kernel");
>>> > > >
>>> > > > I understood Cyril is really suggesting to keep it [1]. I would
>>> also
>>> > > vote to
>>> > > > keep it (we still have some time to see if it got fixed before
>>> release).
>>> > > >
>>> > > > I know we had this discussion in the past (some of your colleague
>>> > > suggesting it
>>> > > > should not be run on non-RT kernel), so I'm not pushing for it.
>>> > >
>>> > > I stil do not understand reasons for disabling the test. The POSIX
>>> > > realtime schedulling classes have to work properly regardless of the
>>> > > kernel flavor. Why should we turn the test off on non-rt kernel then?
>>> > >
>>> >
>>> > No special reasons. I still can sporadically catch the failure on
>>> non-RT
>>> > kernel with even sleep 2 seconds.
>>>
>>> That is very strange. The SCHED_FIFO threads should preempty any lower
>>> prio thread as long as they become runable and should stay running until
>>> they finish or yield. Two seconds should be more than enough for that to
>>> happen.
>>>
>>> > Thus, I took this very extreme approach, because on non-RT kernel,
>>> sleep
>>> > may not have a particularly perfect effect, I guess the stock kernel
>>> with
>>> > sched_setscheduler(, SCHED_FIFO, ) is still has scheduling skew with
>>> > workload.
>>>
>>> Does this happen on vanilla Linux as well or only on RedHat kernels?
>>>
>>
>> Yes, both vanilla Linux and CentOS kernels.
>>
>> more CI test history for sched_football:
>>
>>    Without barrier patch:
>>            Fails on both RT and non-RT CentOS Stream 9/10 kernels
>>            Fails on non-RT mainline v6.17-rc4 kernels. (not build v6.17
>> RT kernel)
>>
>>   (^ that's why we started to look into the failure and submit the commit
>> e523ba88dd9b)
>>
>>    With barrier patch:
>>            Fails on both RT and non-RT CentOS kernels, but the final
>> ball position is noticeably lower.
>>            Fails on non-RT mainline v6.17-rc4 kernels. (not build v6.17
>> RT kernel)
>>
>>    With barrier patch + kickoff flag enhancement:
>>            Fails on the non-RT CentOS Stream 10 kernel
>>            Fails on non-RT mainline v6.17-rc4 kernel. (not build v6.17
>> RT-kernel)
>>            Passed on RT CentOS stream kernel
>>
>> (^ here I started to suspect the SCHED_FIFO threads can not perform well
>> like RT-kernel)
>>
>>
>> [root@dell-per7625-01 sched_football]# uname -r
>> 6.17.0-rc4.liwang
>>
>> [root@dell-per7625-01 sched_football]# ./sched_football
>> tst_test.c:2004: TINFO: LTP version: 20250530
>> tst_test.c:2007: TINFO: Tested kernel: 6.17.0-rc4.liwang #1 SMP
>> PREEMPT_DYNAMIC Thu Sep  4 20:07:20 EDT 2025 x86_64
>> tst_kconfig.c:88: TINFO: Parsing kernel config
>> '/lib/modules/6.17.0-rc4.liwang/build/.config'
>> tst_kconfig.c:676: TINFO: CONFIG_FAULT_INJECTION kernel option detected
>> which might slow the execution
>> tst_test.c:1825: TINFO: Overall timeout per run is 0h 02m 00s
>> sched_football.c:162: TINFO: players_per_team: 32 game_length: 5
>> sched_football.c:178: TINFO: Starting 32 offense threads at priority 15
>> sched_football.c:185: TINFO: Starting 32 defense threads at priority 30
>> sched_football.c:192: TINFO: Starting 64 crazy-fan threads at priority 50
>> sched_football.c:118: TINFO: Starting referee thread
>> sched_football.c:121: TINFO: Starting the game (5 sec)
>> sched_football.c:144: TINFO: Final ball position: 20205
>> sched_football.c:150: TFAIL: Expect: final_ball == 0
>>
>
>
> Checking the configurations of the stock kernel and the real-time
> kernel, the stock kernel uses "CONFIG_PREEMPT_VOLUNTARY=y,"
> which only provides voluntary preemption.
>
> This preemption model is designed to strike a balance between throughput
> and latency. It only allows the kernel to be preempted at specific, well
> defined
> "safe points," potentially resulting in long, unbounded latencies.
>
> However, the sched_football test was most likely designed to measure or
> stress-test the deterministic, low-latency scheduling behavior that is
> characteristic of real-time (RT) kernel.
>
> So, I tend to believe the test's failure on the stock kernel is
> acceptable.
> And, by the way, what does the SUSE kernel configuration look like?
>
>
> # grep CONFIG_PREEMPT /boot/config-6.12.0-55.29.1.el10_0.x86_64
> CONFIG_PREEMPT_BUILD=y
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> # CONFIG_PREEMPT_RT is not set
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
>


> CONFIG_PREEMPT_DYNAMIC=y
>

As Chunyu pointed out to me (on Slack) he saw that the stock kernel
enables dynamical preemption switch.

The default value of CentOS/RHEL is 'voluntary':

# cat /sys/kernel/debug/sched/preempt
none (voluntary) full lazy

Once after switching preempt to the 'full', sched_football wouldn't fail on
the v6.17-rc4 stock kernels.

So I guess we probably need to achieve a separate function like
switch_preempt_model("full"); to switch to the full preempt
before doing the test on the stock kernel.

(And, the CentOS stock kernel still catches fails, Chunyu also pointed
there may be related to the load balance, and perhaps another thing
is that we create CPU-number defense threads, but unless they are
CPU-pinned, the scheduler might not keep them 1:1 against offense
threads. Some CPUs may momentarily run offense if their defense
partner isn’t scheduled yet.
So, I will try to test bind the thread to each CPU and retest it on CentOS)



> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y
>
> (^ I built my v6.17-rc4 with this config too)
>
>
> # grep CONFIG_PREEMPT /boot/config-6.12.0-55.31.1.el10_0.x86_64+rt
> CONFIG_PREEMPT_BUILD=y
> CONFIG_PREEMPT=y
> CONFIG_PREEMPT_RT=y
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
> # CONFIG_PREEMPT_DYNAMIC is not set
> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y
>
>
> --
> Regards,
> Li Wang
>


-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Petr Vorel]

Hi all,

...
> Checking the configurations of the stock kernel and the real-time
> kernel, the stock kernel uses "CONFIG_PREEMPT_VOLUNTARY=y,"
> which only provides voluntary preemption.

Yes, CONFIG_PREEMPT_VOLUNTARY=y is in all mainline defconfigs
(arch/*/configs/*_defconfig). Therefore we have it in our Tumbleweed (6.16.3-1)
and in upcoming SLES16 (6.12 based). SLE15-SP7 (6.4 based) use
CONFIG_PREEMPT_NONE=y instead.

Looking at the history It works well on all archs, we just recently had a
problem with ppc64le hmc on SLE16 (regular ppc64le works). Later I'll check
history of older SLES versions.

And most importantly I'll try to run manually with -i (more stress test).

> This preemption model is designed to strike a balance between throughput
> and latency. It only allows the kernel to be preempted at specific, well
> defined
> "safe points," potentially resulting in long, unbounded latencies.

> However, the sched_football test was most likely designed to measure or
> stress-test the deterministic, low-latency scheduling behavior that is
> characteristic of real-time (RT) kernel.

> So, I tend to believe the test's failure on the stock kernel is acceptable.
> And, by the way, what does the SUSE kernel configuration look like?


> # grep CONFIG_PREEMPT /boot/config-6.12.0-55.29.1.el10_0.x86_64
> CONFIG_PREEMPT_BUILD=y
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> # CONFIG_PREEMPT_RT is not set
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
> CONFIG_PREEMPT_DYNAMIC=y
> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y

> (^ I built my v6.17-rc4 with this config too)

/boot/config-6.12.0-160000.4-default # Tumbleweed x68_64
CONFIG_PREEMPT_BUILD=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_LAZY is not set
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_TRACER is not set
CONFIG_PREEMPTIRQ_DELAY_TEST=m

Upcoming SLE16 is similar to Tumbleweed.

/boot/config-6.4.0-150700.714.ga8e7017-default # SLE15-SP7 x86_64 RT
CONFIG_PREEMPT_BUILD=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_RT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_TRACER is not set
CONFIG_PREEMPTIRQ_DELAY_TEST=m

/boot/config-6.4.0-150700.5-rt # SLE15-SP7 x86_64 RT
CONFIG_PREEMPT_LAZY=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_TRACER is not set
CONFIG_PREEMPTIRQ_DELAY_TEST=m

Kind regards,
Petr

> # grep CONFIG_PREEMPT /boot/config-6.12.0-55.31.1.el10_0.x86_64+rt
> CONFIG_PREEMPT_BUILD=y
> CONFIG_PREEMPT=y
> CONFIG_PREEMPT_RT=y
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
> # CONFIG_PREEMPT_DYNAMIC is not set
> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Petr Vorel]

> Hi all,

> ...
> > Checking the configurations of the stock kernel and the real-time
> > kernel, the stock kernel uses "CONFIG_PREEMPT_VOLUNTARY=y,"
> > which only provides voluntary preemption.

> Yes, CONFIG_PREEMPT_VOLUNTARY=y is in all mainline defconfigs
> (arch/*/configs/*_defconfig). Therefore we have it in our Tumbleweed (6.16.3-1)
> and in upcoming SLES16 (6.12 based). SLE15-SP7 (6.4 based) use
> CONFIG_PREEMPT_NONE=y instead.

> Looking at the history It works well on all archs, we just recently had a
> problem with ppc64le hmc on SLE16 (regular ppc64le works). Later I'll check
> history of older SLES versions.

> And most importantly I'll try to run manually with -i (more stress test).

> > This preemption model is designed to strike a balance between throughput
> > and latency. It only allows the kernel to be preempted at specific, well
> > defined
> > "safe points," potentially resulting in long, unbounded latencies.

> > However, the sched_football test was most likely designed to measure or
> > stress-test the deterministic, low-latency scheduling behavior that is
> > characteristic of real-time (RT) kernel.

> > So, I tend to believe the test's failure on the stock kernel is acceptable.
> > And, by the way, what does the SUSE kernel configuration look like?


> > # grep CONFIG_PREEMPT /boot/config-6.12.0-55.29.1.el10_0.x86_64
> > CONFIG_PREEMPT_BUILD=y
> > CONFIG_PREEMPT_VOLUNTARY=y
> > # CONFIG_PREEMPT is not set
> > # CONFIG_PREEMPT_RT is not set
> > CONFIG_PREEMPT_COUNT=y
> > CONFIG_PREEMPTION=y
> > CONFIG_PREEMPT_DYNAMIC=y
> > CONFIG_PREEMPT_RCU=y
> > CONFIG_PREEMPT_NOTIFIERS=y

> > (^ I built my v6.17-rc4 with this config too)

> /boot/config-6.12.0-160000.4-default # Tumbleweed x68_64
> CONFIG_PREEMPT_BUILD=y
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set
> # CONFIG_PREEMPT_LAZY is not set
> # CONFIG_PREEMPT_RT is not set
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
> CONFIG_PREEMPT_DYNAMIC=y
> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y
> # CONFIG_PREEMPT_TRACER is not set
> CONFIG_PREEMPTIRQ_DELAY_TEST=m

> Upcoming SLE16 is similar to Tumbleweed.

> /boot/config-6.4.0-150700.714.ga8e7017-default # SLE15-SP7 x86_64 RT
> CONFIG_PREEMPT_BUILD=y
> CONFIG_PREEMPT_NONE=y
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT is not set
> # CONFIG_PREEMPT_RT is not set
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
> CONFIG_PREEMPT_DYNAMIC=y
> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y
> # CONFIG_PREEMPT_TRACER is not set
> CONFIG_PREEMPTIRQ_DELAY_TEST=m

> /boot/config-6.4.0-150700.5-rt # SLE15-SP7 x86_64 RT
> CONFIG_PREEMPT_LAZY=y
> # CONFIG_PREEMPT_NONE is not set
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT is not set
> CONFIG_PREEMPT_RT=y
> CONFIG_PREEMPT_COUNT=y
> CONFIG_PREEMPTION=y
> CONFIG_PREEMPT_RCU=y
> CONFIG_PREEMPT_NOTIFIERS=y
> # CONFIG_PREEMPT_TRACER is not set
> CONFIG_PREEMPTIRQ_DELAY_TEST=m

And looking on some other distros, e.g. Debian had CONFIG_PREEMPT_VOLUNTARY=y on
6.12.38+deb13 (from recently released stable Debian 13 trixie), but they changed
it to CONFIG_PREEMPT_LAZY=y in 6.16.3+deb14 (for upcoming Debian forky (testing)
- in devel for now). I would expect that setup for RT kernel.

> Kind regards,
> Petr

> > # grep CONFIG_PREEMPT /boot/config-6.12.0-55.31.1.el10_0.x86_64+rt
> > CONFIG_PREEMPT_BUILD=y
> > CONFIG_PREEMPT=y
> > CONFIG_PREEMPT_RT=y
> > CONFIG_PREEMPT_COUNT=y
> > CONFIG_PREEMPTION=y
> > # CONFIG_PREEMPT_DYNAMIC is not set
> > CONFIG_PREEMPT_RCU=y
> > CONFIG_PREEMPT_NOTIFIERS=y

I suppose this RT kernel has CONFIG_PREEMPT_LAZY=y, right?

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Fri, Sep 5, 2025 at 3:31 PM Petr Vorel <pvorel@suse.cz> wrote:

> > Hi all,
>
> > ...
> > > Checking the configurations of the stock kernel and the real-time
> > > kernel, the stock kernel uses "CONFIG_PREEMPT_VOLUNTARY=y,"
> > > which only provides voluntary preemption.
>
> > Yes, CONFIG_PREEMPT_VOLUNTARY=y is in all mainline defconfigs
> > (arch/*/configs/*_defconfig). Therefore we have it in our Tumbleweed
> (6.16.3-1)
> > and in upcoming SLES16 (6.12 based). SLE15-SP7 (6.4 based) use
> > CONFIG_PREEMPT_NONE=y instead.
>
> > Looking at the history It works well on all archs, we just recently had a
> > problem with ppc64le hmc on SLE16 (regular ppc64le works). Later I'll
> check
> > history of older SLES versions.
>
> > And most importantly I'll try to run manually with -i (more stress test).
>
> > > This preemption model is designed to strike a balance between
> throughput
> > > and latency. It only allows the kernel to be preempted at specific,
> well
> > > defined
> > > "safe points," potentially resulting in long, unbounded latencies.
>
> > > However, the sched_football test was most likely designed to measure or
> > > stress-test the deterministic, low-latency scheduling behavior that is
> > > characteristic of real-time (RT) kernel.
>
> > > So, I tend to believe the test's failure on the stock kernel is
> acceptable.
> > > And, by the way, what does the SUSE kernel configuration look like?
>
>
> > > # grep CONFIG_PREEMPT /boot/config-6.12.0-55.29.1.el10_0.x86_64
> > > CONFIG_PREEMPT_BUILD=y
> > > CONFIG_PREEMPT_VOLUNTARY=y
> > > # CONFIG_PREEMPT is not set
> > > # CONFIG_PREEMPT_RT is not set
> > > CONFIG_PREEMPT_COUNT=y
> > > CONFIG_PREEMPTION=y
> > > CONFIG_PREEMPT_DYNAMIC=y
> > > CONFIG_PREEMPT_RCU=y
> > > CONFIG_PREEMPT_NOTIFIERS=y
>
> > > (^ I built my v6.17-rc4 with this config too)
>
> > /boot/config-6.12.0-160000.4-default # Tumbleweed x68_64
> > CONFIG_PREEMPT_BUILD=y
> > # CONFIG_PREEMPT_NONE is not set
> > CONFIG_PREEMPT_VOLUNTARY=y
> > # CONFIG_PREEMPT is not set
> > # CONFIG_PREEMPT_LAZY is not set
> > # CONFIG_PREEMPT_RT is not set
> > CONFIG_PREEMPT_COUNT=y
> > CONFIG_PREEMPTION=y
> > CONFIG_PREEMPT_DYNAMIC=y
> > CONFIG_PREEMPT_RCU=y
> > CONFIG_PREEMPT_NOTIFIERS=y
> > # CONFIG_PREEMPT_TRACER is not set
> > CONFIG_PREEMPTIRQ_DELAY_TEST=m
>
> > Upcoming SLE16 is similar to Tumbleweed.
>
> > /boot/config-6.4.0-150700.714.ga8e7017-default # SLE15-SP7 x86_64 RT
> > CONFIG_PREEMPT_BUILD=y
> > CONFIG_PREEMPT_NONE=y
> > # CONFIG_PREEMPT_VOLUNTARY is not set
> > # CONFIG_PREEMPT is not set
> > # CONFIG_PREEMPT_RT is not set
> > CONFIG_PREEMPT_COUNT=y
> > CONFIG_PREEMPTION=y
> > CONFIG_PREEMPT_DYNAMIC=y
> > CONFIG_PREEMPT_RCU=y
> > CONFIG_PREEMPT_NOTIFIERS=y
> > # CONFIG_PREEMPT_TRACER is not set
> > CONFIG_PREEMPTIRQ_DELAY_TEST=m
>
> > /boot/config-6.4.0-150700.5-rt # SLE15-SP7 x86_64 RT
> > CONFIG_PREEMPT_LAZY=y
> > # CONFIG_PREEMPT_NONE is not set
> > # CONFIG_PREEMPT_VOLUNTARY is not set
> > # CONFIG_PREEMPT is not set
> > CONFIG_PREEMPT_RT=y
> > CONFIG_PREEMPT_COUNT=y
> > CONFIG_PREEMPTION=y
> > CONFIG_PREEMPT_RCU=y
> > CONFIG_PREEMPT_NOTIFIERS=y
> > # CONFIG_PREEMPT_TRACER is not set
> > CONFIG_PREEMPTIRQ_DELAY_TEST=m
>
> And looking on some other distros, e.g. Debian had
> CONFIG_PREEMPT_VOLUNTARY=y on
> 6.12.38+deb13 (from recently released stable Debian 13 trixie), but they
> changed
> it to CONFIG_PREEMPT_LAZY=y in 6.16.3+deb14 (for upcoming Debian forky
> (testing)
> - in devel for now). I would expect that setup for RT kernel.
>
> > Kind regards,
> > Petr
>
> > > # grep CONFIG_PREEMPT /boot/config-6.12.0-55.31.1.el10_0.x86_64+rt
> > > CONFIG_PREEMPT_BUILD=y
> > > CONFIG_PREEMPT=y
> > > CONFIG_PREEMPT_RT=y
> > > CONFIG_PREEMPT_COUNT=y
> > > CONFIG_PREEMPTION=y
> > > # CONFIG_PREEMPT_DYNAMIC is not set
> > > CONFIG_PREEMPT_RCU=y
> > > CONFIG_PREEMPT_NOTIFIERS=y
>
> I suppose this RT kernel has CONFIG_PREEMPT_LAZY=y, right?
>

Not exactly, I removed some unenabling options when posting.
It has 'CONFIG_ARCH_HAS_PREEMPT_LAZY'.

The full preempt config looks like:

# grep PREEMPT /boot/config-6.12.0-55.31.1.el10_0.x86_64+rt
CONFIG_PREEMPT_BUILD=y
CONFIG_ARCH_HAS_PREEMPT_LAZY=y
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_LAZY is not set
# CONFIG_PREEMPT_LAZIEST is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
# CONFIG_PREEMPT_DYNAMIC is not set
CONFIG_PREEMPT_RCU=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_DRM_I915_PREEMPT_TIMEOUT=640
CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE=7500
CONFIG_DRM_XE_PREEMPT_TIMEOUT=640000
CONFIG_DRM_XE_PREEMPT_TIMEOUT_MAX=10000000
CONFIG_DRM_XE_PREEMPT_TIMEOUT_MIN=1
CONFIG_DEBUG_PREEMPT=y
# CONFIG_PREEMPT_TRACER is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set


> Kind regards,
> Petr
>
>

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Cyril Hrubis]

Hi!
> Checking the configurations of the stock kernel and the real-time
> kernel, the stock kernel uses "CONFIG_PREEMPT_VOLUNTARY=y,"
> which only provides voluntary preemption.
>
> This preemption model is designed to strike a balance between throughput
> and latency. It only allows the kernel to be preempted at specific, well
> defined
> "safe points," potentially resulting in long, unbounded latencies.
> 
> However, the sched_football test was most likely designed to measure or
> stress-test the deterministic, low-latency scheduling behavior that is
> characteristic of real-time (RT) kernel.
> 
> So, I tend to believe the test's failure on the stock kernel is acceptable.

I still find it a bit unexpected though. The preeption models apply only
to kernel code. The user space code can be stil preempted at any point,
so the offense threads should be preempted and replaced by high priority
tasks and never executed again since we do not call to the kernel there
at all, we just run a loop that increments an integer there. I guess
that one possibility is that we saturate the machine with real-time
tasks to the extend that scheduller code in kernel does not get to
distribute the processes. If that is a problem we need to give kernel
chance to shuffle the processes when we wait for the kickoff flag.

Does things start to work if we change the loops that wait for the final
kickoff to:

       while (!tst_atomic_load(&kickoff_flag))
               sched_yield();

This should trigger the scheduller code to be executed so that it has
chance to distribute the processes around.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Fri, Sep 5, 2025 at 5:18 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > Checking the configurations of the stock kernel and the real-time
> > kernel, the stock kernel uses "CONFIG_PREEMPT_VOLUNTARY=y,"
> > which only provides voluntary preemption.
> >
> > This preemption model is designed to strike a balance between throughput
> > and latency. It only allows the kernel to be preempted at specific, well
> > defined
> > "safe points," potentially resulting in long, unbounded latencies.
> >
> > However, the sched_football test was most likely designed to measure or
> > stress-test the deterministic, low-latency scheduling behavior that is
> > characteristic of real-time (RT) kernel.
> >
> > So, I tend to believe the test's failure on the stock kernel is
> acceptable.
>
> I still find it a bit unexpected though. The preeption models apply only
> to kernel code. The user space code can be stil preempted at any point,
> so the offense threads should be preempted and replaced by high priority
> tasks and never executed again since we do not call to the kernel there
> at all, we just run a loop that increments an integer there. I guess
> that one possibility is that we saturate the machine with real-time
> tasks to the extend that scheduller code in kernel does not get to
> distribute the processes. If that is a problem we need to give kernel
> chance to shuffle the processes when we wait for the kickoff flag.
>
> Does things start to work if we change the loops that wait for the final
> kickoff to:
>
>        while (!tst_atomic_load(&kickoff_flag))
>                sched_yield();
>

This has improved something, but still observes sporadic fail on RHEL stock
kernel.
However, both the RHEL RT-kernel and the mainline v6.17-rc4 stock kernel
pass.

Below is my complete test patch, can you guys try it on SUSE stock kernel?

--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -44,6 +44,7 @@
 static tst_atomic_t the_ball;
 static int players_per_team = 0;
 static int game_length = DEF_GAME_LENGTH;
+static tst_atomic_t kickoff_flag;
 static tst_atomic_t game_over;

 static char *str_game_length;
@@ -55,6 +56,9 @@ void *thread_fan(void *arg LTP_ATTRIBUTE_UNUSED)
 {
        prctl(PR_SET_NAME, "crazy_fan", 0, 0, 0);
        pthread_barrier_wait(&start_barrier);
+       while (!tst_atomic_load(&kickoff_flag))
+               sched_yield();
+
        /*occasionally wake up and run across the field */
        while (!tst_atomic_load(&game_over)) {
                struct timespec start, stop;
@@ -80,6 +84,9 @@ void *thread_defense(void *arg LTP_ATTRIBUTE_UNUSED)
 {
        prctl(PR_SET_NAME, "defense", 0, 0, 0);
        pthread_barrier_wait(&start_barrier);
+       while (!tst_atomic_load(&kickoff_flag))
+               sched_yield();
+
        /*keep the ball from being moved */
        while (!tst_atomic_load(&game_over)) {
        }
@@ -92,6 +99,9 @@ void *thread_offense(void *arg LTP_ATTRIBUTE_UNUSED)
 {
        prctl(PR_SET_NAME, "offense", 0, 0, 0);
        pthread_barrier_wait(&start_barrier);
+       while (!tst_atomic_load(&kickoff_flag))
+               sched_yield();
+
        while (!tst_atomic_load(&game_over)) {
                tst_atomic_add_return(1, &the_ball); /* move the ball ahead
one yard */
        }
@@ -115,9 +125,15 @@ void referee(int game_length)
        now = start;

        /* Start the game! */
-       tst_atomic_store(0, &the_ball);
-       pthread_barrier_wait(&start_barrier);
        atrace_marker_write("sched_football", "Game_started!");
+       pthread_barrier_wait(&start_barrier);
+       tst_atomic_store(0, &the_ball);
+       tst_atomic_store(1, &kickoff_flag);
+
+       if (tst_check_preempt_rt())
+               usleep(20000);
+       else
+               usleep(2000000);

        /* Watch the game */
        while ((now.tv_sec - start.tv_sec) < game_length) {
@@ -125,14 +141,14 @@ void referee(int game_length)
                gettimeofday(&now, NULL);
        }

-       /* Stop the game! */
-       tst_atomic_store(1, &game_over);
-       atrace_marker_write("sched_football", "Game_Over!");
-
        /* Blow the whistle */
        final_ball = tst_atomic_load(&the_ball);
        tst_res(TINFO, "Final ball position: %d", final_ball);

+       /* Stop the game! */
+       tst_atomic_store(1, &game_over);
+       atrace_marker_write("sched_football", "Game_Over!");
+
        TST_EXP_EXPR(final_ball == 0);
 }

@@ -154,6 +170,7 @@ static void do_test(void)
        /* We're the ref, so set our priority right */
        param.sched_priority = sched_get_priority_min(SCHED_FIFO) + 80;
        sched_setscheduler(0, SCHED_FIFO, &param);
+       tst_atomic_store(0, &kickoff_flag);

        /*
         * Start the offense


> This should trigger the scheduller code to be executed so that it has
> chance to distribute the processes around.
>
> --
> Cyril Hrubis
> chrubis@suse.cz
>
>

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Cyril Hrubis]

Hi!
> This has improved something, but still observes sporadic fail on RHEL stock
> kernel.
> However, both the RHEL RT-kernel and the mainline v6.17-rc4 stock kernel
> pass.

Maybe add a short usleep() before the referee sets the kickoff_flag so
that the system has more time to shuffle processes?

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Petr Vorel]

> Hi!
> > This has improved something, but still observes sporadic fail on RHEL stock
> > kernel.
> > However, both the RHEL RT-kernel and the mainline v6.17-rc4 stock kernel
> > pass.

FYI I run tests all SLES and Tumbleweed versions on non-RT kernels, with -i50,
for both master and your v2 (with enabled non-RT kernels).  All are passing.


* 6.16.3-1 (Tumbleweed, very close to kernel stable - it has very few patches unlike the others)
CONFIG_PREEMPT_VOLUNTARY=y
$ cat /sys/kernel/debug/sched/preempt
none (voluntary) full lazy

SLES (these are heavily modified)

* 6.4
* 5.14
CONFIG_PREEMPT_VOLUNTARY=y
$ cat /sys/kernel/debug/sched/preempt
(none) voluntary full

* 5.3
* 4.12
* 4.4
CONFIG_PREEMPT_NONE=y
(/sys/kernel/debug/sched/preempt not enabled)

Later I recheck what was your config which breaks it.

> Maybe add a short usleep() before the referee sets the kickoff_flag so
> that the system has more time to shuffle processes?

+1

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Fri, Sep 5, 2025 at 8:31 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > This has improved something, but still observes sporadic fail on RHEL
> stock
> > kernel.
> > However, both the RHEL RT-kernel and the mainline v6.17-rc4 stock kernel
> > pass.
>
> Maybe add a short usleep() before the referee sets the kickoff_flag so
> that the system has more time to shuffle processes?
>

Thanks, I do manually test and now almost 1/20 times fail.
And the worth mentioning that this is not a new failure in RHEL kernels.
Those patches since the barrier have already much improved the test
result from RHEL side. At least from RHEL RT-kernel I didn't observe
fail any more.

So ,as long as the patch works on the SUSE kernel to resolve it's NEW
problem,
I think we can merge.

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Cyril Hrubis]

Hi!
> Thanks, I do manually test and now almost 1/20 times fail.
> And the worth mentioning that this is not a new failure in RHEL kernels.
> Those patches since the barrier have already much improved the test
> result from RHEL side. At least from RHEL RT-kernel I didn't observe
> fail any more.
> 
> So ,as long as the patch works on the SUSE kernel to resolve it's NEW
> problem,
> I think we can merge.

I got another idea meanwhile, what happens if we keep bussy loop for the
defense and add the sched_yield() only to offense and crazy fans? That
should ensure that before the kickoff the defense threads should be
hogging the available CPUs.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

On Fri, Sep 5, 2025 at 9:44 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > Thanks, I do manually test and now almost 1/20 times fail.
> > And the worth mentioning that this is not a new failure in RHEL kernels.
> > Those patches since the barrier have already much improved the test
> > result from RHEL side. At least from RHEL RT-kernel I didn't observe
> > fail any more.
> >
> > So ,as long as the patch works on the SUSE kernel to resolve it's NEW
> > problem,
> > I think we can merge.
>
> I got another idea meanwhile, what happens if we keep bussy loop for the
> defense and add the sched_yield() only to offense and crazy fans? That
> should ensure that before the kickoff the defense threads should be
> hogging the available CPUs.
>

Sounds great, or something we can do more, maybe remove the busy-loop
from the crazy-fan threads, since the fan threads are the distraction
during the
game so we don't need to have them waking up so early in the first place.

With those refined, I would not get fails on RHEL non-RT kernel anymore,
I will keep the sched_football test run more than 10000 times tonight.
    ` for i in {1..10000}; do ./sched_football ; done`

If everything goes well, I will sign off patch v3 as below:

--- a/testcases/realtime/func/sched_football/sched_football.c
+++ b/testcases/realtime/func/sched_football/sched_football.c
@@ -44,6 +44,7 @@
 static tst_atomic_t the_ball;
 static int players_per_team = 0;
 static int game_length = DEF_GAME_LENGTH;
+static tst_atomic_t kickoff_flag;
 static tst_atomic_t game_over;

 static char *str_game_length;
@@ -80,6 +81,9 @@ void *thread_defense(void *arg LTP_ATTRIBUTE_UNUSED)
 {
        prctl(PR_SET_NAME, "defense", 0, 0, 0);
        pthread_barrier_wait(&start_barrier);
+       while (!tst_atomic_load(&kickoff_flag))
+               ;
+
        /*keep the ball from being moved */
        while (!tst_atomic_load(&game_over)) {
        }
@@ -92,6 +96,9 @@ void *thread_offense(void *arg LTP_ATTRIBUTE_UNUSED)
 {
        prctl(PR_SET_NAME, "offense", 0, 0, 0);
        pthread_barrier_wait(&start_barrier);
+       while (!tst_atomic_load(&kickoff_flag))
+               sched_yield();
+
        while (!tst_atomic_load(&game_over)) {
                tst_atomic_add_return(1, &the_ball); /* move the ball ahead
one yard */
        }
@@ -115,9 +122,16 @@ void referee(int game_length)
        now = start;

        /* Start the game! */
-       tst_atomic_store(0, &the_ball);
-       pthread_barrier_wait(&start_barrier);
        atrace_marker_write("sched_football", "Game_started!");
+       pthread_barrier_wait(&start_barrier);
+       usleep(200000);
+
+       tst_atomic_store(0, &the_ball);
+       tst_atomic_store(1, &kickoff_flag);
+       if (tst_check_preempt_rt())
+               usleep(20000);
+       else
+               usleep(2000000);

        /* Watch the game */
        while ((now.tv_sec - start.tv_sec) < game_length) {
@@ -125,14 +139,14 @@ void referee(int game_length)
                gettimeofday(&now, NULL);
        }

-       /* Stop the game! */
-       tst_atomic_store(1, &game_over);
-       atrace_marker_write("sched_football", "Game_Over!");
-
        /* Blow the whistle */
        final_ball = tst_atomic_load(&the_ball);
        tst_res(TINFO, "Final ball position: %d", final_ball);

+       /* Stop the game! */
+       tst_atomic_store(1, &game_over);
+       atrace_marker_write("sched_football", "Game_Over!");
+
        TST_EXP_EXPR(final_ball == 0);
 }

@@ -154,6 +168,7 @@ static void do_test(void)
        /* We're the ref, so set our priority right */
        param.sched_priority = sched_get_priority_min(SCHED_FIFO) + 80;
        sched_setscheduler(0, SCHED_FIFO, &param);
+       tst_atomic_store(0, &kickoff_flag);

        /*
         * Start the offense




-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] sched_football: synchronize with kickoff flag to reduce skew

[Li Wang via ltp]

Petr Vorel <pvorel@suse.cz> wrote:

> Hi!
> > > This has improved something, but still observes sporadic fail on RHEL
> stock
> > > kernel.
> > > However, both the RHEL RT-kernel and the mainline v6.17-rc4 stock
> kernel
> > > pass.
>
> FYI I run tests all SLES and Tumbleweed versions on non-RT kernels, with
> -i50,
> for both master and your v2 (with enabled non-RT kernels).  All are
> passing.
>

Thank you so much Petr, it's valuable to me.

As patch v3 has just been sent out, it would be great to get SUSE tests
against that.
(so far v3 passed on all my RHEL + stock/RT/mainline kernels :)


-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp