[PATCH RT] time/hrtimer: Use softirq based wakeups for non-RT threads

[PATCH RT] time/hrtimer: Use softirq based wakeups for non-RT threads

[Sebastian Andrzej Siewior]

Normal wake ups (like clock_nanosleep()) which are performed by normal
users can easily lead to 2ms latency spikes if (enough) hrtimer wakeups
are synchronized.
This patch moves all hrtimers wakeups to the softirq queue unless the
caller has a RT priority.

Reported-by: Gratian Crisan <gratian.crisan@ni.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 kernel/time/hrtimer.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 3aa823fcab70..8ea2825a50af 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1622,14 +1622,31 @@ static enum hrtimer_restart hrtimer_wakeup(struct hrtimer *timer)
 	return HRTIMER_NORESTART;
 }
 
+#ifdef CONFIG_PREEMPT_RT_FULL
+static bool task_is_elevated(struct task_struct *tsk)
+{
+	int policy = tsk->policy;
+
+	if (policy == SCHED_FIFO || policy == SCHED_RR)
+		return true;
+	if (policy == SCHED_DEADLINE)
+		return true;
+	return false;
+}
+#endif
+
 static void __hrtimer_init_sleeper(struct hrtimer_sleeper *sl,
 				   clockid_t clock_id,
 				   enum hrtimer_mode mode,
 				   struct task_struct *task)
 {
 #ifdef CONFIG_PREEMPT_RT_FULL
-	if (!(clock_id & HRTIMER_BASE_SOFT_MASK))
-		clock_id |= HRTIMER_BASE_HARD_MASK;
+	if (!(clock_id & (HRTIMER_BASE_HARD_MASK | HRTIMER_BASE_SOFT_MASK))) {
+		if (task_is_elevated(current) || system_state != SYSTEM_RUNNING)
+			clock_id |= HRTIMER_BASE_HARD_MASK;
+		else
+			clock_id |= HRTIMER_BASE_SOFT_MASK;
+	}
 #endif
 	__hrtimer_init(&sl->timer, clock_id, mode);
 	sl->timer.function = hrtimer_wakeup;
-- 
2.14.2

Re: [PATCH RT] time/hrtimer: Use softirq based wakeups for non-RT threads

[Gratian Crisan]

Sebastian Andrzej Siewior writes:

> Normal wake ups (like clock_nanosleep()) which are performed by normal
> users can easily lead to 2ms latency spikes if (enough) hrtimer wakeups
> are synchronized.
> This patch moves all hrtimers wakeups to the softirq queue unless the
> caller has a RT priority.
>
> Reported-by: Gratian Crisan <gratian.crisan@ni.com>

I can confirm this patch fixes the original problem reported.

I ran an overnight test (about 30 hours total) on two platforms using
cyclictest + hrtimer stress load: configurable number of SCHED_OTHER
threads doing random clock_nanosleep() in up to 1 mS intervals.

Without this patch the max latency recorded was:
  * on dual core, Intel Atom E3825 @ 1.33GHz: 107 uS
  * on dual core, Zynq 7020 @ 667MHz: 261 uS

With this patch the max numbers drop to:
  * on dual core, Intel Atom E3825 @ 1.33GHz: 66 uS
  * on dual core, Zynq 7020 @ 667MHz: 90 uS

The max latency numbers with this patch are in-line with expectations
for these platforms.

Thank you so much,
-Gratian