From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760945AbbCDJxZ (ORCPT ); Wed, 4 Mar 2015 04:53:25 -0500 Received: from service87.mimecast.com ([91.220.42.44]:43024 "EHLO service87.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758935AbbCDJxV convert rfc822-to-8bit (ORCPT ); Wed, 4 Mar 2015 04:53:21 -0500 Message-ID: <54F6D60F.8020502@arm.com> Date: Wed, 04 Mar 2015 09:53:19 +0000 From: Juri Lelli User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Wanpeng Li , Ingo Molnar , Peter Zijlstra CC: "linux-kernel@vger.kernel.org" Subject: Re: [PATCH v9] sched/deadline: support dl task migration during cpu hotplug References: <1425339349-3321-1-git-send-email-wanpeng.li@linux.intel.com> In-Reply-To: <1425339349-3321-1-git-send-email-wanpeng.li@linux.intel.com> X-OriginalArrivalTime: 04 Mar 2015 09:53:19.0496 (UTC) FILETIME=[0B4A0080:01D05661] X-MC-Unique: 115030409531912701 Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, I think we are still missing a corner case: no admission control, a task with an affinity mask of a single cpu, the cpu goes off. In this case we could try to let it run just somewhere else, as we don't guarantee anything from start. This applies on top of your patch, comments? Thanks, - Juri --- kernel/sched/deadline.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 467ec5d..8dec157 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -579,10 +579,24 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer) * online cpu. */ fallback = true; - cpu = cpumask_any_and(cpu_active_mask, tsk_cpus_allowed(p)); + cpu = cpumask_any_and(cpu_active_mask, + tsk_cpus_allowed(p)); if (cpu >= nr_cpu_ids) { - WARN_ON(1); - goto unlock; + if (dl_bandwidth_enabled()) { + /* + * Fail to find any suitable cpu. + * The task will never come back! + */ + WARN_ON(1); + goto unlock; + } else { + /* + * If admission control is disabled we + * try a little harder to let the task + * run. + */ + cpu = cpumask_any(cpu_active_mask); + } } later_rq = cpu_rq(cpu); double_lock_balance(rq, later_rq); -- 2.3.0 On 02/03/2015 23:35, Wanpeng Li wrote: > I observe that dl task can't be migrated to other cpus during cpu hotplug, > in addition, task may/may not be running again if cpu is added back. The > root cause which I found is that dl task will be throtted and removed from > dl rq after comsuming all budget, which leads to stop task can't pick it up > from dl rq and migrate to other cpus during hotplug. > > The method to reproduce: > schedtool -E -t 50000:100000 -e ./test > Actually test is just a simple for loop. Then observe which cpu the test > task is on. > echo 0 > /sys/devices/system/cpu/cpuN/online > > This patch adds the dl task migration during cpu hotplug by finding a most > suitable later deadline rq after dl timer fire if current rq is offline, > if fail to find a suitable later deadline rq then fallback to any eligible > online cpu in order that the deadline task will come back to us, and the > push/pull mechanism should then move it around properly. > > Signed-off-by: Wanpeng Li > --- > v8 -> v9: > * align tsk_cpus_allowed(p) to cpu_active_mask > * add WARN_ON(1) > * don't resched_curr if later_rq come from the cpumask_any_and() > v7 -> v8: > * remove rd->span related modification since Pang's commit 16b269436b72 > (sched/deadline: Modify cpudl::free_cpus to reflect rd->online) merged > upstream, which Juri pointed out can handle the exclusive cpusets. > * rebase > v6 -> v7: > * rebase > v5 -> v6: > * add double_lock_balance in the fallback path > v4 -> v5: > * remove raw_spin_unlock(&rq->lock) > * cleanup codes, spotted by Peterz > * cleanup patch description > v3 -> v4: > * use tsk_cpus_allowed wrapper > * fix compile error > v2 -> v3: > * don't get_task_struct > * if cannot preempt any rq, fallback to pick any online cpus > * use cpu_active_mask as original later_mask if cpu is offline > v1 -> v2: > * push the task to another cpu in dl_task_timer() if rq is offline. > > kernel/sched/deadline.c | 40 ++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 40 insertions(+) > > diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c > index 08766a3..d5b1b16 100644 > --- a/kernel/sched/deadline.c > +++ b/kernel/sched/deadline.c > @@ -492,6 +492,7 @@ static int start_dl_timer(struct sched_dl_entity *dl_se, bool boosted) > return hrtimer_active(&dl_se->dl_timer); > } > > +static struct rq *find_lock_later_rq(struct task_struct *task, struct rq *rq); > /* > * This is the bandwidth enforcement timer callback. If here, we know > * a task is not on its dl_rq, since the fact that the timer was running > @@ -537,6 +538,45 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer) > update_rq_clock(rq); > > /* > + * So if we find that the rq the task was on is no longer > + * available, we need to select a new rq. > + */ > + if (unlikely(!rq->online)) { > + struct rq *later_rq = NULL; > + bool fallback = false; > + > + later_rq = find_lock_later_rq(p, rq); > + > + if (!later_rq) { > + int cpu; > + > + /* > + * If cannot preempt any rq, fallback to pick any > + * online cpu. > + */ > + fallback = true; > + cpu = cpumask_any_and(cpu_active_mask, tsk_cpus_allowed(p)); > + if (cpu >= nr_cpu_ids) { > + WARN_ON(1); > + goto unlock; > + } > + later_rq = cpu_rq(cpu); > + double_lock_balance(rq, later_rq); > + } > + > + deactivate_task(rq, p, 0); > + set_task_cpu(p, later_rq->cpu); > + activate_task(later_rq, p, ENQUEUE_REPLENISH); > + > + if (!fallback) > + resched_curr(later_rq); > + > + double_unlock_balance(rq, later_rq); > + > + goto unlock; > + } > + > + /* > * If the throttle happened during sched-out; like: > * > * schedule() >