[PATCH] sched/deadline: Fix lock pinning warning during cpu hotplug

[PATCH] sched/deadline: Fix lock pinning warning during cpu hotplug

[Wanpeng Li]

From: Wanpeng Li <wanpeng.li@hotmail.com>

WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:3531 lock_release+0x690/0x6a0
releasing a pinned lock
Call Trace:
 dump_stack+0x99/0xd0
 __warn+0xd1/0xf0
 ? dl_task_timer+0x1a1/0x2b0
 warn_slowpath_fmt+0x4f/0x60
 ? sched_clock+0x13/0x20
 lock_release+0x690/0x6a0
 ? enqueue_pushable_dl_task+0x9b/0xa0
 ? enqueue_task_dl+0x1ca/0x480
 _raw_spin_unlock+0x1f/0x40
 dl_task_timer+0x1a1/0x2b0
 ? push_dl_task.part.31+0x190/0x190
WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:3649 lock_unpin_lock+0x181/0x1a0
unpinning an unpinned lock
Call Trace:
 dump_stack+0x99/0xd0
 __warn+0xd1/0xf0
 warn_slowpath_fmt+0x4f/0x60
 lock_unpin_lock+0x181/0x1a0
 dl_task_timer+0x127/0x2b0
 ? push_dl_task.part.31+0x190/0x190

This can be triggered by hot-unplug the cpu which dl task is running on.

DL task will be migrated to a most suitable later deadline rq or fallback 
to any eligible online CPU after dl timer fires if current rq is offline. 
We need to hold rq lock of both rqs to confirm there is nothing changed 
and compare the task deadline and the earliest_dl deadline of the destination 
cpu in the progress of finding suitable destination cpu.

The rq lock of offline cpu will be unlocked in _double_lock_balance and then 
reacquire both rq locks in a fair way, the rq lock of offline cpu is held and 
lockdep pin before, however, it will be unlocked in _double_lock_balance directly 
w/o lockdep unpin, which triggers the "releasing a pinned lock" warning above.

Queue the task to the destination cpu might have overloaded rq, so the current 
implementation will check if we need to kick some away, rq lock will be lockdep 
unpin since push_dl_task() will release the rq lock and reacquire it. However, 
if there is cpu hot-unplug and dl task migration, the rq lock is lockdep unpin, 
so it will trigger the "unpinning an unpinned lock" above.

This patch fix it by lockdep unpin the offline cpu before task migration during 
cpu hotplug handling and lockdep pin destination cpu after this handling since 
the rq lock is held during this handling.

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@arm.com>
Cc: Luca Abeni <luca.abeni@unitn.it>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
 kernel/sched/deadline.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index fcb7f02..1ce8867 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -658,8 +658,11 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 	 *
 	 * XXX figure out if select_task_rq_dl() deals with offline cpus.
 	 */
-	if (unlikely(!rq->online))
+	if (unlikely(!rq->online)) {
+		lockdep_unpin_lock(&rq->lock, rf.cookie);
 		rq = dl_task_offline_migration(rq, p);
+		rf.cookie = lockdep_pin_lock(&rq->lock);
+	}
 
 	/*
 	 * Queueing this task back might have overloaded rq, check if we need
-- 
1.9.1

[tip:sched/core] sched/deadline: Fix lock pinning warning during CPU hotplug

[tip-bot for Wanpeng Li]

Commit-ID:  c0c8c9fa210c9a042060435f17e40ba4a76d6d6f
Gitweb:     http://git.kernel.org/tip/c0c8c9fa210c9a042060435f17e40ba4a76d6d6f
Author:     Wanpeng Li <wanpeng.li@hotmail.com>
AuthorDate: Thu, 4 Aug 2016 09:42:20 +0800
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Wed, 10 Aug 2016 14:02:55 +0200

sched/deadline: Fix lock pinning warning during CPU hotplug

The following warning can be triggered by hot-unplugging the CPU
on which an active SCHED_DEADLINE task is running on:

  WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:3531 lock_release+0x690/0x6a0
  releasing a pinned lock
  Call Trace:
   dump_stack+0x99/0xd0
   __warn+0xd1/0xf0
   ? dl_task_timer+0x1a1/0x2b0
   warn_slowpath_fmt+0x4f/0x60
   ? sched_clock+0x13/0x20
   lock_release+0x690/0x6a0
   ? enqueue_pushable_dl_task+0x9b/0xa0
   ? enqueue_task_dl+0x1ca/0x480
   _raw_spin_unlock+0x1f/0x40
   dl_task_timer+0x1a1/0x2b0
   ? push_dl_task.part.31+0x190/0x190
  WARNING: CPU: 0 PID: 0 at kernel/locking/lockdep.c:3649 lock_unpin_lock+0x181/0x1a0
  unpinning an unpinned lock
  Call Trace:
   dump_stack+0x99/0xd0
   __warn+0xd1/0xf0
   warn_slowpath_fmt+0x4f/0x60
   lock_unpin_lock+0x181/0x1a0
   dl_task_timer+0x127/0x2b0
   ? push_dl_task.part.31+0x190/0x190

As per the comment before this code, its safe to drop the RQ lock
here, and since we (potentially) change rq, unpin and repin to avoid
the splat.

Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
[ Rewrote changelog. ]
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@arm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Luca Abeni <luca.abeni@unitn.it>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/1470274940-17976-1-git-send-email-wanpeng.li@hotmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/deadline.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index fcb7f02..1ce8867 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -658,8 +658,11 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 	 *
 	 * XXX figure out if select_task_rq_dl() deals with offline cpus.
 	 */
-	if (unlikely(!rq->online))
+	if (unlikely(!rq->online)) {
+		lockdep_unpin_lock(&rq->lock, rf.cookie);
 		rq = dl_task_offline_migration(rq, p);
+		rf.cookie = lockdep_pin_lock(&rq->lock);
+	}
 
 	/*
 	 * Queueing this task back might have overloaded rq, check if we need