[PATCH] sched: fix RLIMIT_RTTIME when PI-boosting to RT

[PATCH] sched: fix RLIMIT_RTTIME when PI-boosting to RT

[brian]

From: Brian Silverman <brian@peloton-tech.com>

When non-realtime tasks get priority-inheritance boosted to a realtime
scheduling class, RLIMIT_RTTIME starts to apply to them. However, the
counter used for checking this (the same one used for SCHED_RR
timeslices) was not getting reset. This meant that tasks running with a
non-realtime scheduling class which are repeatedly boosted to a realtime
one, but never block while they are running realtime, eventually hit the
timeout without ever running for a time over the limit. This patch
resets the realtime timeslice counter when un-PI-boosting from an RT to
a non-RT scheduling class.

I have some test code with two threads and a shared PTHREAD_PRIO_INHERIT
mutex which induces priority boosting and spins while boosted that gets
killed by a SIGXCPU on non-fixed kernels but doesn't with this patch
applied. It happens much faster with a CONFIG_PREEMPT_RT kernel, and
does happen eventually with PREEMPT_VOLUNTARY kernels.

Signed-off-by: Brian Silverman <brian@peloton-tech.com>
---
I am not subscribed to the list so please CC me on any responses.

 kernel/sched/core.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 87b9814..16ad0ed 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3192,6 +3192,8 @@ void rt_mutex_setprio(struct task_struct *p, int prio)
 	} else {
 		if (dl_prio(oldprio))
 			p->dl.dl_boosted = 0;
+		if (rt_prio(oldprio))
+			p->rt.timeout = 0;
 		p->sched_class = &fair_sched_class;
 	}
 
-- 
1.7.10.4

Re: [PATCH] sched: fix RLIMIT_RTTIME when PI-boosting to RT

[Brian Silverman]

Here's my test code. Compile with `gcc -pthread -lrt test_pi.c`. It
requires permission to set a realtime scheduling policy of 2 when
running.

#define _GNU_SOURCE

#include <pthread.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <sched.h>
#include <assert.h>
#include <sys/resource.h>
#include <string.h>
#include <signal.h>
#include <stdlib.h>

static const struct timespec kSleepTime = {0, 10000};
static pthread_mutex_t mutex;

extern void nop() {}

void *nonrealtime(void *ignored_param) {
  while (1) {
    assert(pthread_mutex_lock(&mutex) == 0);
    assert(pthread_mutex_unlock(&mutex) == 0);
    assert(clock_nanosleep(CLOCK_MONOTONIC, 0, &kSleepTime, NULL) == 0);
  }
}

void *realtime(void *ignored_param) {
  struct sched_param param;
  memset(&param, 0, sizeof(param));
  param.sched_priority = 2;
  assert(sched_setscheduler(0, SCHED_FIFO, &param) == 0);
  while (1) {
    assert(pthread_mutex_lock(&mutex) == 0);
    assert(clock_nanosleep(CLOCK_MONOTONIC, 0, &kSleepTime, NULL) == 0);
    assert(pthread_mutex_unlock(&mutex) == 0);
  }
}

void signal_handler(int number) {
  printf("got signal %d, SIGXCPU=%d\n", number, SIGXCPU);
  exit(0);
}

int main() {
  struct sigaction action;
  memset(&action, 0, sizeof(action));
  action.sa_handler = signal_handler;
  assert(sigaction(SIGXCPU, &action, NULL) == 0);

  struct rlimit rlim;
  rlim.rlim_cur = 500;
  rlim.rlim_max = 5000;
  assert(prlimit(0, RLIMIT_RTTIME, &rlim, NULL) == 0);

  pthread_mutexattr_t mutexattr;
  assert(pthread_mutexattr_init(&mutexattr) == 0);
  assert(pthread_mutexattr_setprotocol(&mutexattr, PTHREAD_PRIO_INHERIT) == 0);
  assert(pthread_mutex_init(&mutex, &mutexattr) == 0);
  assert(pthread_mutexattr_destroy(&mutexattr) == 0);

  pthread_t nrt, rt;
  assert(pthread_create(&nrt, NULL, nonrealtime, NULL) == 0);
  assert(pthread_create(&rt, NULL, realtime, NULL) == 0);
  assert(pthread_join(nrt, NULL) == 0);
  assert(pthread_join(rt, NULL) == 0);
  return 0;
}


On Wed, Feb 18, 2015 at 7:23 PM,  <brian@peloton-tech.com> wrote:
> From: Brian Silverman <brian@peloton-tech.com>
>
> When non-realtime tasks get priority-inheritance boosted to a realtime
> scheduling class, RLIMIT_RTTIME starts to apply to them. However, the
> counter used for checking this (the same one used for SCHED_RR
> timeslices) was not getting reset. This meant that tasks running with a
> non-realtime scheduling class which are repeatedly boosted to a realtime
> one, but never block while they are running realtime, eventually hit the
> timeout without ever running for a time over the limit. This patch
> resets the realtime timeslice counter when un-PI-boosting from an RT to
> a non-RT scheduling class.
>
> I have some test code with two threads and a shared PTHREAD_PRIO_INHERIT
> mutex which induces priority boosting and spins while boosted that gets
> killed by a SIGXCPU on non-fixed kernels but doesn't with this patch
> applied. It happens much faster with a CONFIG_PREEMPT_RT kernel, and
> does happen eventually with PREEMPT_VOLUNTARY kernels.
>
> Signed-off-by: Brian Silverman <brian@peloton-tech.com>
> ---
> I am not subscribed to the list so please CC me on any responses.
>
>  kernel/sched/core.c |    2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 87b9814..16ad0ed 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3192,6 +3192,8 @@ void rt_mutex_setprio(struct task_struct *p, int prio)
>         } else {
>                 if (dl_prio(oldprio))
>                         p->dl.dl_boosted = 0;
> +               if (rt_prio(oldprio))
> +                       p->rt.timeout = 0;
>                 p->sched_class = &fair_sched_class;
>         }
>
> --
> 1.7.10.4
>

Re: [PATCH] sched: fix RLIMIT_RTTIME when PI-boosting to RT

[Ingo Molnar]

* Brian Silverman <brian@peloton-tech.com> wrote:

> Here's my test code. Compile with `gcc -pthread -lrt test_pi.c`. It 
> requires permission to set a realtime scheduling policy of 2 when 
> running.

Mind sending a patch that sticks this testcase into 
tools/testing/selftests/sched/ or so, with the new 'sched' directory 
and new Makefile to be created by you as well?

I've reformatted the testcase below, to kernel coding style. Note that 
I added a minimal license notification, you might want to add your 
copyright.

Thanks,

	Ingo

==========================>


/*
 * RLIMIT_RTTIME test code. Compile with:
 *
 *     gcc -pthread -lrt test_pi.c
 *
 * It requires permission to set a realtime scheduling policy of 2 when running.
 *
 * License: GPLv2
 */
#define _GNU_SOURCE

#include <pthread.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <sched.h>
#include <assert.h>
#include <sys/resource.h>
#include <string.h>
#include <signal.h>
#include <stdlib.h>

static const struct timespec kSleepTime = { 0, 10000 };

static pthread_mutex_t mutex;

extern void nop()
{
}

void *nonrealtime(void *ignored_param)
{
	while (1) {
		assert(pthread_mutex_lock(&mutex) == 0);
		assert(pthread_mutex_unlock(&mutex) == 0);
		assert(clock_nanosleep(CLOCK_MONOTONIC, 0, &kSleepTime, NULL) == 0);
	}
}

void *realtime(void *ignored_param)
{
	struct sched_param param;

	memset(&param, 0, sizeof(param));
	param.sched_priority = 2;
	assert(sched_setscheduler(0, SCHED_FIFO, &param) == 0);

	while (1) {
		assert(pthread_mutex_lock(&mutex) == 0);
		assert(clock_nanosleep(CLOCK_MONOTONIC, 0, &kSleepTime, NULL) == 0);
		assert(pthread_mutex_unlock(&mutex) == 0);
	}
}

void signal_handler(int number)
{
	printf("got signal %d, SIGXCPU=%d\n", number, SIGXCPU);

	exit(0);
}

int main()
{
	struct sigaction action;
	memset(&action, 0, sizeof(action));
	action.sa_handler = signal_handler;
	assert(sigaction(SIGXCPU, &action, NULL) == 0);

	struct rlimit rlim;
	rlim.rlim_cur = 500;
	rlim.rlim_max = 5000;
	assert(prlimit(0, RLIMIT_RTTIME, &rlim, NULL) == 0);

	pthread_mutexattr_t mutexattr;
	assert(pthread_mutexattr_init(&mutexattr) == 0);
	assert(pthread_mutexattr_setprotocol(&mutexattr, PTHREAD_PRIO_INHERIT) == 0);
	assert(pthread_mutex_init(&mutex, &mutexattr) == 0);
	assert(pthread_mutexattr_destroy(&mutexattr) == 0);

	pthread_t nrt, rt;
	assert(pthread_create(&nrt, NULL, nonrealtime, NULL) == 0);
	assert(pthread_create(&rt, NULL, realtime, NULL) == 0);
	assert(pthread_join(nrt, NULL) == 0);
	assert(pthread_join(rt, NULL) == 0);

	return 0;
}

Re: [PATCH] sched: fix RLIMIT_RTTIME when PI-boosting to RT

[Austin Schuh]

ping?

On Wed, Feb 18, 2015 at 4:23 PM,  <brian@peloton-tech.com> wrote:
> From: Brian Silverman <brian@peloton-tech.com>
>
> When non-realtime tasks get priority-inheritance boosted to a realtime
> scheduling class, RLIMIT_RTTIME starts to apply to them. However, the
> counter used for checking this (the same one used for SCHED_RR
> timeslices) was not getting reset. This meant that tasks running with a
> non-realtime scheduling class which are repeatedly boosted to a realtime
> one, but never block while they are running realtime, eventually hit the
> timeout without ever running for a time over the limit. This patch
> resets the realtime timeslice counter when un-PI-boosting from an RT to
> a non-RT scheduling class.
>
> I have some test code with two threads and a shared PTHREAD_PRIO_INHERIT
> mutex which induces priority boosting and spins while boosted that gets
> killed by a SIGXCPU on non-fixed kernels but doesn't with this patch
> applied. It happens much faster with a CONFIG_PREEMPT_RT kernel, and
> does happen eventually with PREEMPT_VOLUNTARY kernels.
>
> Signed-off-by: Brian Silverman <brian@peloton-tech.com>
> ---
> I am not subscribed to the list so please CC me on any responses.
>
>  kernel/sched/core.c |    2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 87b9814..16ad0ed 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3192,6 +3192,8 @@ void rt_mutex_setprio(struct task_struct *p, int prio)
>         } else {
>                 if (dl_prio(oldprio))
>                         p->dl.dl_boosted = 0;
> +               if (rt_prio(oldprio))
> +                       p->rt.timeout = 0;
>                 p->sched_class = &fair_sched_class;
>         }
>
> --
> 1.7.10.4
>

Re: [PATCH] sched: fix RLIMIT_RTTIME when PI-boosting to RT

[Sebastian Andrzej Siewior]

* Austin Schuh | 2015-03-05 09:10:47 [-0800]:

>ping?

Why is this a ping? I haven't seen this in my rt nor in my lkml inbox.
Please repost it properly including lkml.
>From what I can tell not beeing a sched guy is that the patch looks
reasonable since the timeout gets only set to zero on enqueue_task_rt().
Is there something special you do to trigger this?

Sebastian

Re: [PATCH] sched: fix RLIMIT_RTTIME when PI-boosting to RT

[Brian Silverman]

On Mon, Mar 9, 2015 at 1:34 PM, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:

> From what I can tell not beeing a sched guy is that the patch looks
> reasonable since the timeout gets only set to zero on enqueue_task_rt().
> Is there something special you do to trigger this?

I posted some test code with two threads and a shared PTHREAD_PRIO_INHERIT
mutex. It forces repeated priority boosting from SCHED_OTHER to SCHED_RR and
then spins for a bit while boosted. It eventually receives a SIGXCPU
on non-fixed kernels.
The SIGXCPU happens much faster with a CONFIG_PREEMPT_RT kernel, and does
happen eventually with CONFIG_PREEMPT_VOLUNTARY kernels.

>
> Sebastian

Thanks,
Brian

[tip:sched/core] sched: Fix RLIMIT_RTTIME when PI-boosting to RT

[tip-bot for Brian Silverman]

Commit-ID:  746db9443ea57fd9c059f62c4bfbf41cf224fe13
Gitweb:     http://git.kernel.org/tip/746db9443ea57fd9c059f62c4bfbf41cf224fe13
Author:     Brian Silverman <brian@peloton-tech.com>
AuthorDate: Wed, 18 Feb 2015 16:23:56 -0800
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 23 Mar 2015 10:47:55 +0100

sched: Fix RLIMIT_RTTIME when PI-boosting to RT

When non-realtime tasks get priority-inheritance boosted to a realtime
scheduling class, RLIMIT_RTTIME starts to apply to them. However, the
counter used for checking this (the same one used for SCHED_RR
timeslices) was not getting reset. This meant that tasks running with a
non-realtime scheduling class which are repeatedly boosted to a realtime
one, but never block while they are running realtime, eventually hit the
timeout without ever running for a time over the limit. This patch
resets the realtime timeslice counter when un-PI-boosting from an RT to
a non-RT scheduling class.

I have some test code with two threads and a shared PTHREAD_PRIO_INHERIT
mutex which induces priority boosting and spins while boosted that gets
killed by a SIGXCPU on non-fixed kernels but doesn't with this patch
applied. It happens much faster with a CONFIG_PREEMPT_RT kernel, and
does happen eventually with PREEMPT_VOLUNTARY kernels.

Signed-off-by: Brian Silverman <brian@peloton-tech.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: austin@peloton-tech.com
Cc: <stable@vger.kernel.org>
Link: http://lkml.kernel.org/r/1424305436-6716-1-git-send-email-brian@peloton-tech.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f0f831e..62671f5 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -3034,6 +3034,8 @@ void rt_mutex_setprio(struct task_struct *p, int prio)
 	} else {
 		if (dl_prio(oldprio))
 			p->dl.dl_boosted = 0;
+		if (rt_prio(oldprio))
+			p->rt.timeout = 0;
 		p->sched_class = &fair_sched_class;
 	}