[PATCH 1/5] sched/core: Clear stale rq->dl_server when core-sched replaces a pick

[Fernand Sieber <sieberf@amazon.com>]

With core scheduling and the fair server, the forced-idle accounting
warns from an idle core:

  WARNING: kernel/sched/core_sched.c:263 at __sched_core_account_forceidle+0x1d6/0x1f0, CPU#0: swapper/0/0

The core is in forced idle with no sibling running a task. The reason
is a stale rq->dl_server causing the idle task to incorrectly be
considered as carried by the deadline server. The idle task then wins
over a runnable sibling task and the whole core is idle despite having
runnable tasks.

The timeline below describes the issue:

  Setup. CPU0 and CPU1 are SMT siblings.
  - CPU1 runs R, a SCHED_FIFO task with cookie X.
  - CPU0 has C, a CFS task with no cookie, which has been starved.
    So CPU0's fair server is due to step in.

  t0, CPU0 picks for the core.
  - pick_task(rq0) goes through the deadline class, which picks the
    fair server, which picks C. As a side effect it sets
    rq0->dl_server = &rq0->fair_server.
  - pick_task(rq1) picks R.

  t1, ranking. C->dl_server is only set once C is switched in, so C
  ranks as CFS. R (RT) beats C (CFS), so the core cookie becomes X.

  t2, matching. CPU0's pick C has cookie 0, not X, so it is replaced.
  There is no cookie-X task on CPU0, so the replacement is idle, and
  CPU0 is forced idle.
  - The code sets rq0->core_pick = idle and rq0->core_dl_server = NULL.
  - The bug: it leaves rq0->dl_server pointing at CPU0's fair server,
    even though idle didn't come through any server.

  t3, CPU0 switches. CPU0 is the selecting CPU, so next = idle, and
  put_prev_set_next_task() hands the stale rq0->dl_server to it. Now
  idle->dl_server is CPU0's fair server.

  t4, C leaves. Load balancing moves C to another core, so CPU0 has
  nothing runnable. The stale link stays on idle.

  t5, the next core-wide selection (when CPU1 reschedules, say on a
  wakeup).
  - CPU0's pick is idle, and the ranking sees idle->dl_server set, so
    it ranks idle as a deadline task.
  - idle now beats R, and the core cookie becomes idle's cookie, 0.
  - CPU1's R has cookie X, not 0, so CPU1 is forced idle.

  t6, result. The whole core is idle. R is runnable but can't run, and
  nothing runs on either CPU. That's the WARN in
  __sched_core_account_forceidle(): forced idle with no sibling running
  a task. It lasts until CPU0 goes through a full schedule that resets
  its server link.

With this patch, step t2 also does rq0->dl_server = NULL. At t3 idle
gets no server, and at t5 it ranks as plain idle, so R wins and keeps
running on CPU1.

Fixes: bd9bbc96e835 ("sched: Rework dl_server")
Signed-off-by: Fernand Sieber <sieberf@amazon.com>
---
 kernel/sched/core.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 175b52a7ea793..f7ac13186ee59 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6453,6 +6453,12 @@ pick_next_task(struct rq *rq, struct rq_flags *rf)
 				p = sched_core_find(rq_i, cookie);
 			if (!p)
 				p = idle_sched_class.pick_task(rq_i, rf);
+
+			/*
+			 * Neither a core tree pick nor idle comes through a
+			 * DL server; drop the one the replaced pick left.
+			 */
+			rq_i->dl_server = NULL;
 		}
 
 		rq_i->core_pick = p;
-- 
2.47.3




Amazon Development Centre (South Africa) (Proprietary) Limited
29 Gogosoa Street, Observatory, Cape Town, Western Cape, 7925, South Africa
Registration Number: 2004 / 034463 / 07