* [patch 0/5] scheduler fixlets
@ 2011-01-22 4:44 Paul Turner
2011-01-22 4:44 ` [patch 1/5] sched: fix sign under-flows in wake_affine Paul Turner
` (5 more replies)
0 siblings, 6 replies; 15+ messages in thread
From: Paul Turner @ 2011-01-22 4:44 UTC (permalink / raw)
To: linux-kernel; +Cc: Peter Zijlstra, Ingo Molnar, Mike Galbraith
A small set of scheduler patches for the CFS side of things.
Mostly buglet fixes with one or two associated clean-ups. Each patch should
be (modulo any merge fuzz) independent.
Thanks,
- Paul
^ permalink raw reply [flat|nested] 15+ messages in thread* [patch 1/5] sched: fix sign under-flows in wake_affine 2011-01-22 4:44 [patch 0/5] scheduler fixlets Paul Turner @ 2011-01-22 4:44 ` Paul Turner 2011-01-26 12:09 ` [tip:sched/core] sched: Fix " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 2/5] sched: (cleanup) remove redundant cfs_rq checks Paul Turner ` (4 subsequent siblings) 5 siblings, 1 reply; 15+ messages in thread From: Paul Turner @ 2011-01-22 4:44 UTC (permalink / raw) To: linux-kernel; +Cc: Peter Zijlstra, Ingo Molnar, Mike Galbraith [-- Attachment #1: sched-signed_wake_affine.patch --] [-- Type: text/plain, Size: 1246 bytes --] While care is taken around the zero-point in effective_load to not exceed the instantaneous rq->weight, it's still possible (e.g. using wake_idx != 0) for (load + effective_load) to underflow. In this case the comparing the unsigned values can result in incorrect balanced decisions. Signed-off-by: Paul Turner <pjt@google.com> --- kernel/sched_fair.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) Index: tip3/kernel/sched_fair.c =================================================================== --- tip3.orig/kernel/sched_fair.c +++ tip3/kernel/sched_fair.c @@ -1404,7 +1404,7 @@ static inline unsigned long effective_lo static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) { - unsigned long this_load, load; + s64 this_load, load; int idx, this_cpu, prev_cpu; unsigned long tl_per_task; struct task_group *tg; @@ -1443,8 +1443,8 @@ static int wake_affine(struct sched_doma * Otherwise check if either cpus are near enough in load to allow this * task to be woken on this_cpu. */ - if (this_load) { - unsigned long this_eff_load, prev_eff_load; + if (this_load > 0) { + s64 this_eff_load, prev_eff_load; this_eff_load = 100; this_eff_load *= power_of(prev_cpu); ^ permalink raw reply [flat|nested] 15+ messages in thread
* [tip:sched/core] sched: Fix sign under-flows in wake_affine 2011-01-22 4:44 ` [patch 1/5] sched: fix sign under-flows in wake_affine Paul Turner @ 2011-01-26 12:09 ` tip-bot for Paul Turner 0 siblings, 0 replies; 15+ messages in thread From: tip-bot for Paul Turner @ 2011-01-26 12:09 UTC (permalink / raw) To: linux-tip-commits Cc: linux-kernel, hpa, mingo, a.p.zijlstra, pjt, tglx, mingo Commit-ID: e37b6a7b27b400c3aa488db8c6629a05095bc79c Gitweb: http://git.kernel.org/tip/e37b6a7b27b400c3aa488db8c6629a05095bc79c Author: Paul Turner <pjt@google.com> AuthorDate: Fri, 21 Jan 2011 20:44:59 -0800 Committer: Ingo Molnar <mingo@elte.hu> CommitDate: Wed, 26 Jan 2011 12:31:01 +0100 sched: Fix sign under-flows in wake_affine While care is taken around the zero-point in effective_load to not exceed the instantaneous rq->weight, it's still possible (e.g. using wake_idx != 0) for (load + effective_load) to underflow. In this case the comparing the unsigned values can result in incorrect balanced decisions. Signed-off-by: Paul Turner <pjt@google.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> LKML-Reference: <20110122044851.734245014@google.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> --- kernel/sched_fair.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index 3547699..ccecfec 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1432,7 +1432,7 @@ static inline unsigned long effective_load(struct task_group *tg, int cpu, static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) { - unsigned long this_load, load; + s64 this_load, load; int idx, this_cpu, prev_cpu; unsigned long tl_per_task; struct task_group *tg; @@ -1471,8 +1471,8 @@ static int wake_affine(struct sched_domain *sd, struct task_struct *p, int sync) * Otherwise check if either cpus are near enough in load to allow this * task to be woken on this_cpu. */ - if (this_load) { - unsigned long this_eff_load, prev_eff_load; + if (this_load > 0) { + s64 this_eff_load, prev_eff_load; this_eff_load = 100; this_eff_load *= power_of(prev_cpu); ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [patch 2/5] sched: (cleanup) remove redundant cfs_rq checks 2011-01-22 4:44 [patch 0/5] scheduler fixlets Paul Turner 2011-01-22 4:44 ` [patch 1/5] sched: fix sign under-flows in wake_affine Paul Turner @ 2011-01-22 4:45 ` Paul Turner 2011-01-26 12:10 ` [tip:sched/core] sched: Fix/remove " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 3/5] sched: simplify update_cfs_shares parameters Paul Turner ` (3 subsequent siblings) 5 siblings, 1 reply; 15+ messages in thread From: Paul Turner @ 2011-01-22 4:45 UTC (permalink / raw) To: linux-kernel; +Cc: Peter Zijlstra, Ingo Molnar, Mike Galbraith, Dan Carpenter [-- Attachment #1: sched-remove_cfs_rq_checks.patch --] [-- Type: text/plain, Size: 1252 bytes --] Since updates are against an entity's queuing cfs_rq it's not possible to enter update_cfs_{shares,load} with a NULL cfs_rq. (Indeed, update_cfs_load would crash prior to the check if we did anyway since we load is examined during the initializers). Also, in the update_cfs_load case there's no point in maintaining averages for rq->cfs_rq since we don't perform shares distribution at that level -- NULL check is replaced accordingly. Thanks to Dan Carpenter for pointing out the deference before NULL check. Signed-off-by: Paul Turner <pjt@google.com> --- kernel/sched_fair.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) Index: tip3/kernel/sched_fair.c =================================================================== --- tip3.orig/kernel/sched_fair.c +++ tip3/kernel/sched_fair.c @@ -721,7 +721,7 @@ static void update_cfs_load(struct cfs_r u64 now, delta; unsigned long load = cfs_rq->load.weight; - if (!cfs_rq) + if (cfs_rq->tg == &root_task_group) return; now = rq_of(cfs_rq)->clock; @@ -784,9 +784,6 @@ static void update_cfs_shares(struct cfs struct sched_entity *se; long load_weight, load, shares; - if (!cfs_rq) - return; - tg = cfs_rq->tg; se = tg->se[cpu_of(rq_of(cfs_rq))]; if (!se) ^ permalink raw reply [flat|nested] 15+ messages in thread
* [tip:sched/core] sched: Fix/remove redundant cfs_rq checks 2011-01-22 4:45 ` [patch 2/5] sched: (cleanup) remove redundant cfs_rq checks Paul Turner @ 2011-01-26 12:10 ` tip-bot for Paul Turner 0 siblings, 0 replies; 15+ messages in thread From: tip-bot for Paul Turner @ 2011-01-26 12:10 UTC (permalink / raw) To: linux-tip-commits Cc: linux-kernel, hpa, mingo, a.p.zijlstra, pjt, tglx, mingo Commit-ID: b815f1963e47b9b69bb17e0588bd5af5b1114ae0 Gitweb: http://git.kernel.org/tip/b815f1963e47b9b69bb17e0588bd5af5b1114ae0 Author: Paul Turner <pjt@google.com> AuthorDate: Fri, 21 Jan 2011 20:45:00 -0800 Committer: Ingo Molnar <mingo@elte.hu> CommitDate: Wed, 26 Jan 2011 12:31:02 +0100 sched: Fix/remove redundant cfs_rq checks Since updates are against an entity's queuing cfs_rq it's not possible to enter update_cfs_{shares,load} with a NULL cfs_rq. (Indeed, update_cfs_load would crash prior to the check if we did anyway since we load is examined during the initializers). Also, in the update_cfs_load case there's no point in maintaining averages for rq->cfs_rq since we don't perform shares distribution at that level -- NULL check is replaced accordingly. Thanks to Dan Carpenter for pointing out the deference before NULL check. Signed-off-by: Paul Turner <pjt@google.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> LKML-Reference: <20110122044851.825284940@google.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> --- kernel/sched_fair.c | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index ccecfec..1997383 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -722,7 +722,7 @@ static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update) u64 now, delta; unsigned long load = cfs_rq->load.weight; - if (!cfs_rq) + if (cfs_rq->tg == &root_task_group) return; now = rq_of(cfs_rq)->clock; @@ -830,9 +830,6 @@ static void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta) struct sched_entity *se; long shares; - if (!cfs_rq) - return; - tg = cfs_rq->tg; se = tg->se[cpu_of(rq_of(cfs_rq))]; if (!se) ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [patch 3/5] sched: simplify update_cfs_shares parameters 2011-01-22 4:44 [patch 0/5] scheduler fixlets Paul Turner 2011-01-22 4:44 ` [patch 1/5] sched: fix sign under-flows in wake_affine Paul Turner 2011-01-22 4:45 ` [patch 2/5] sched: (cleanup) remove redundant cfs_rq checks Paul Turner @ 2011-01-22 4:45 ` Paul Turner 2011-01-26 12:11 ` [tip:sched/core] sched: Simplify " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 4/5] sched: use rq->clock_task instead of rq->clock for maintaining load averages Paul Turner ` (2 subsequent siblings) 5 siblings, 1 reply; 15+ messages in thread From: Paul Turner @ 2011-01-22 4:45 UTC (permalink / raw) To: linux-kernel; +Cc: Peter Zijlstra, Ingo Molnar, Mike Galbraith [-- Attachment #1: sched-clean_update_shares.patch --] [-- Type: text/plain, Size: 3992 bytes --] Re-visiting this: Since update_cfs_shares will now only ever re-weight an entity that is a relative parent of the current entity in enqueue_entity; we can safely issue the account_entity_enqueue relative to that cfs_rq and avoid the requirement for special handling of the enqueue case in update_cfs_shares. Signed-off-by: Paul Turner <pjt@google.com> --- kernel/sched.c | 2 +- kernel/sched_fair.c | 22 +++++++++++----------- 2 files changed, 12 insertions(+), 12 deletions(-) Index: tip3/kernel/sched_fair.c =================================================================== --- tip3.orig/kernel/sched_fair.c +++ tip3/kernel/sched_fair.c @@ -540,7 +540,7 @@ static u64 sched_vslice(struct cfs_rq *c } static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update); -static void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta); +static void update_cfs_shares(struct cfs_rq *cfs_rq); /* * Update the current task's runtime statistics. Skip current tasks that @@ -778,7 +778,7 @@ static void reweight_entity(struct cfs_r account_entity_enqueue(cfs_rq, se); } -static void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta) +static void update_cfs_shares(struct cfs_rq *cfs_rq) { struct task_group *tg; struct sched_entity *se; @@ -789,11 +789,11 @@ static void update_cfs_shares(struct cfs if (!se) return; - load = cfs_rq->load.weight + weight_delta; + load = cfs_rq->load.weight; load_weight = atomic_read(&tg->load_weight); - load_weight -= cfs_rq->load_contribution; load_weight += load; + load_weight -= cfs_rq->load_contribution; shares = (tg->shares * load); if (load_weight) @@ -811,7 +811,7 @@ static void update_entity_shares_tick(st { if (cfs_rq->load_unacc_exec_time > sysctl_sched_shares_window) { update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); } } #else /* CONFIG_FAIR_GROUP_SCHED */ @@ -819,7 +819,7 @@ static void update_cfs_load(struct cfs_r { } -static inline void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta) +static inline void update_cfs_shares(struct cfs_rq *cfs_rq) { } @@ -950,8 +950,8 @@ enqueue_entity(struct cfs_rq *cfs_rq, st */ update_curr(cfs_rq); update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, se->load.weight); account_entity_enqueue(cfs_rq, se); + update_cfs_shares(cfs_rq); if (flags & ENQUEUE_WAKEUP) { place_entity(cfs_rq, se, 0); @@ -1013,7 +1013,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, st update_cfs_load(cfs_rq, 0); account_entity_dequeue(cfs_rq, se); update_min_vruntime(cfs_rq); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); /* * Normalize the entity after updating the min_vruntime because the @@ -1254,7 +1254,7 @@ enqueue_task_fair(struct rq *rq, struct struct cfs_rq *cfs_rq = cfs_rq_of(se); update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); } hrtick_update(rq); @@ -1284,7 +1284,7 @@ static void dequeue_task_fair(struct rq struct cfs_rq *cfs_rq = cfs_rq_of(se); update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); } hrtick_update(rq); @@ -2095,7 +2095,7 @@ static int update_shares_cpu(struct task * We need to update shares after updating tg->load_weight in * order to adjust the weight of groups with long running tasks. */ - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); raw_spin_unlock_irqrestore(&rq->lock, flags); Index: tip3/kernel/sched.c =================================================================== --- tip3.orig/kernel/sched.c +++ tip3/kernel/sched.c @@ -8510,7 +8510,7 @@ int sched_group_set_shares(struct task_g /* Propagate contribution to hierarchy */ raw_spin_lock_irqsave(&rq->lock, flags); for_each_sched_entity(se) - update_cfs_shares(group_cfs_rq(se), 0); + update_cfs_shares(group_cfs_rq(se)); raw_spin_unlock_irqrestore(&rq->lock, flags); } ^ permalink raw reply [flat|nested] 15+ messages in thread
* [tip:sched/core] sched: Simplify update_cfs_shares parameters 2011-01-22 4:45 ` [patch 3/5] sched: simplify update_cfs_shares parameters Paul Turner @ 2011-01-26 12:11 ` tip-bot for Paul Turner 0 siblings, 0 replies; 15+ messages in thread From: tip-bot for Paul Turner @ 2011-01-26 12:11 UTC (permalink / raw) To: linux-tip-commits Cc: linux-kernel, hpa, mingo, a.p.zijlstra, pjt, tglx, mingo Commit-ID: 6d5ab2932a21ea54406ab95c43ecff90a3eddfda Gitweb: http://git.kernel.org/tip/6d5ab2932a21ea54406ab95c43ecff90a3eddfda Author: Paul Turner <pjt@google.com> AuthorDate: Fri, 21 Jan 2011 20:45:01 -0800 Committer: Ingo Molnar <mingo@elte.hu> CommitDate: Wed, 26 Jan 2011 12:33:19 +0100 sched: Simplify update_cfs_shares parameters Re-visiting this: Since update_cfs_shares will now only ever re-weight an entity that is a relative parent of the current entity in enqueue_entity; we can safely issue the account_entity_enqueue relative to that cfs_rq and avoid the requirement for special handling of the enqueue case in update_cfs_shares. Signed-off-by: Paul Turner <pjt@google.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> LKML-Reference: <20110122044851.915214637@google.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> --- kernel/sched.c | 2 +- kernel/sched_fair.c | 30 ++++++++++++++---------------- 2 files changed, 15 insertions(+), 17 deletions(-) diff --git a/kernel/sched.c b/kernel/sched.c index 18d38e4..e0fa3ff 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -8510,7 +8510,7 @@ int sched_group_set_shares(struct task_group *tg, unsigned long shares) /* Propagate contribution to hierarchy */ raw_spin_lock_irqsave(&rq->lock, flags); for_each_sched_entity(se) - update_cfs_shares(group_cfs_rq(se), 0); + update_cfs_shares(group_cfs_rq(se)); raw_spin_unlock_irqrestore(&rq->lock, flags); } diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index 0c26e2d..0c550c8 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -540,7 +540,7 @@ static u64 sched_vslice(struct cfs_rq *cfs_rq, struct sched_entity *se) } static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update); -static void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta); +static void update_cfs_shares(struct cfs_rq *cfs_rq); /* * Update the current task's runtime statistics. Skip current tasks that @@ -763,16 +763,15 @@ static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update) list_del_leaf_cfs_rq(cfs_rq); } -static long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg, - long weight_delta) +static long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg) { long load_weight, load, shares; - load = cfs_rq->load.weight + weight_delta; + load = cfs_rq->load.weight; load_weight = atomic_read(&tg->load_weight); - load_weight -= cfs_rq->load_contribution; load_weight += load; + load_weight -= cfs_rq->load_contribution; shares = (tg->shares * load); if (load_weight) @@ -790,7 +789,7 @@ static void update_entity_shares_tick(struct cfs_rq *cfs_rq) { if (cfs_rq->load_unacc_exec_time > sysctl_sched_shares_window) { update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); } } # else /* CONFIG_SMP */ @@ -798,8 +797,7 @@ static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update) { } -static inline long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg, - long weight_delta) +static inline long calc_cfs_shares(struct cfs_rq *cfs_rq, struct task_group *tg) { return tg->shares; } @@ -824,7 +822,7 @@ static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, account_entity_enqueue(cfs_rq, se); } -static void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta) +static void update_cfs_shares(struct cfs_rq *cfs_rq) { struct task_group *tg; struct sched_entity *se; @@ -838,7 +836,7 @@ static void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta) if (likely(se->load.weight == tg->shares)) return; #endif - shares = calc_cfs_shares(cfs_rq, tg, weight_delta); + shares = calc_cfs_shares(cfs_rq, tg); reweight_entity(cfs_rq_of(se), se, shares); } @@ -847,7 +845,7 @@ static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update) { } -static inline void update_cfs_shares(struct cfs_rq *cfs_rq, long weight_delta) +static inline void update_cfs_shares(struct cfs_rq *cfs_rq) { } @@ -978,8 +976,8 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) */ update_curr(cfs_rq); update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, se->load.weight); account_entity_enqueue(cfs_rq, se); + update_cfs_shares(cfs_rq); if (flags & ENQUEUE_WAKEUP) { place_entity(cfs_rq, se, 0); @@ -1041,7 +1039,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) update_cfs_load(cfs_rq, 0); account_entity_dequeue(cfs_rq, se); update_min_vruntime(cfs_rq); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); /* * Normalize the entity after updating the min_vruntime because the @@ -1282,7 +1280,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) struct cfs_rq *cfs_rq = cfs_rq_of(se); update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); } hrtick_update(rq); @@ -1312,7 +1310,7 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) struct cfs_rq *cfs_rq = cfs_rq_of(se); update_cfs_load(cfs_rq, 0); - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); } hrtick_update(rq); @@ -2123,7 +2121,7 @@ static int update_shares_cpu(struct task_group *tg, int cpu) * We need to update shares after updating tg->load_weight in * order to adjust the weight of groups with long running tasks. */ - update_cfs_shares(cfs_rq, 0); + update_cfs_shares(cfs_rq); raw_spin_unlock_irqrestore(&rq->lock, flags); ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [patch 4/5] sched: use rq->clock_task instead of rq->clock for maintaining load averages 2011-01-22 4:44 [patch 0/5] scheduler fixlets Paul Turner ` (2 preceding siblings ...) 2011-01-22 4:45 ` [patch 3/5] sched: simplify update_cfs_shares parameters Paul Turner @ 2011-01-22 4:45 ` Paul Turner 2011-01-26 12:10 ` [tip:sched/core] sched: Use rq->clock_task instead of rq->clock for correctly " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 5/5] sched: avoid expensive initial update_cfs_load() Paul Turner 2011-01-24 10:17 ` [patch 0/5] scheduler fixlets Peter Zijlstra 5 siblings, 1 reply; 15+ messages in thread From: Paul Turner @ 2011-01-22 4:45 UTC (permalink / raw) To: linux-kernel; +Cc: Peter Zijlstra, Ingo Molnar, Mike Galbraith [-- Attachment #1: sched-use_clock_task.patch --] [-- Type: text/plain, Size: 836 bytes --] The delta in clock_task is a more fair attribution of how much time a tg has been contributing load to the current cpu. While not really important it also means we're more in sync (by magnitude) with respect to periodic updates (since __update_curr deltas are clock_task based). Signed-off-by: Paul Turner <pjt@google.com> --- kernel/sched_fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Index: tip3/kernel/sched_fair.c =================================================================== --- tip3.orig/kernel/sched_fair.c +++ tip3/kernel/sched_fair.c @@ -724,7 +724,7 @@ static void update_cfs_load(struct cfs_r if (cfs_rq->tg == root_task_group) return; - now = rq_of(cfs_rq)->clock; + now = rq_of(cfs_rq)->clock_task; delta = now - cfs_rq->load_stamp; /* truncate load history at 4 idle periods */ ^ permalink raw reply [flat|nested] 15+ messages in thread
* [tip:sched/core] sched: Use rq->clock_task instead of rq->clock for correctly maintaining load averages 2011-01-22 4:45 ` [patch 4/5] sched: use rq->clock_task instead of rq->clock for maintaining load averages Paul Turner @ 2011-01-26 12:10 ` tip-bot for Paul Turner 0 siblings, 0 replies; 15+ messages in thread From: tip-bot for Paul Turner @ 2011-01-26 12:10 UTC (permalink / raw) To: linux-tip-commits Cc: linux-kernel, hpa, mingo, a.p.zijlstra, pjt, tglx, mingo Commit-ID: 05ca62c6ca17f39b88fa956d5ebc1fa6e93ad5e3 Gitweb: http://git.kernel.org/tip/05ca62c6ca17f39b88fa956d5ebc1fa6e93ad5e3 Author: Paul Turner <pjt@google.com> AuthorDate: Fri, 21 Jan 2011 20:45:02 -0800 Committer: Ingo Molnar <mingo@elte.hu> CommitDate: Wed, 26 Jan 2011 12:31:03 +0100 sched: Use rq->clock_task instead of rq->clock for correctly maintaining load averages The delta in clock_task is a more fair attribution of how much time a tg has been contributing load to the current cpu. While not really important it also means we're more in sync (by magnitude) with respect to periodic updates (since __update_curr deltas are clock_task based). Signed-off-by: Paul Turner <pjt@google.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> LKML-Reference: <20110122044852.007092349@google.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> --- kernel/sched_fair.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index 1997383..0c26e2d 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -725,7 +725,7 @@ static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update) if (cfs_rq->tg == &root_task_group) return; - now = rq_of(cfs_rq)->clock; + now = rq_of(cfs_rq)->clock_task; delta = now - cfs_rq->load_stamp; /* truncate load history at 4 idle periods */ ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [patch 5/5] sched: avoid expensive initial update_cfs_load() 2011-01-22 4:44 [patch 0/5] scheduler fixlets Paul Turner ` (3 preceding siblings ...) 2011-01-22 4:45 ` [patch 4/5] sched: use rq->clock_task instead of rq->clock for maintaining load averages Paul Turner @ 2011-01-22 4:45 ` Paul Turner 2011-01-26 12:11 ` [tip:sched/core] sched: Avoid " tip-bot for Paul Turner ` (2 more replies) 2011-01-24 10:17 ` [patch 0/5] scheduler fixlets Peter Zijlstra 5 siblings, 3 replies; 15+ messages in thread From: Paul Turner @ 2011-01-22 4:45 UTC (permalink / raw) To: linux-kernel; +Cc: Peter Zijlstra, Ingo Molnar, Mike Galbraith [-- Attachment #1: sched-fix_first_update_load.patch --] [-- Type: text/plain, Size: 1648 bytes --] Since cfs->{load_stamp,load_last} are zero-initalized the initial load update will consider the delta to be 'since the beginning of time'. This results in a lot of pointless divisions to bring this large period to be within the sysctl_sched_shares_window. Fix this by initializing load_stamp to be 1 at cfs_rq initialization, this allows for an initial load_stamp > load_last which then lets standard idle truncation proceed. We avoid spinning (and slightly improve consistency) by fixing delta to be [period - 1] in this path resulting in a slightly more predictable shares ramp. (Previously the amount of idle time preserved by the overflow would range between [period/2,period-1].) Signed-off-by: Paul Turner <pjt@google.com> --- kernel/sched.c | 2 ++ kernel/sched_fair.c | 1 + 2 files changed, 3 insertions(+) Index: tip3/kernel/sched.c =================================================================== --- tip3.orig/kernel/sched.c +++ tip3/kernel/sched.c @@ -7796,6 +7796,8 @@ static void init_cfs_rq(struct cfs_rq *c INIT_LIST_HEAD(&cfs_rq->tasks); #ifdef CONFIG_FAIR_GROUP_SCHED cfs_rq->rq = rq; + /* allow initial update_cfs_load() to truncate */ + cfs_rq->load_stamp = 1; #endif cfs_rq->min_vruntime = (u64)(-(1LL << 20)); } Index: tip3/kernel/sched_fair.c =================================================================== --- tip3.orig/kernel/sched_fair.c +++ tip3/kernel/sched_fair.c @@ -732,6 +732,7 @@ static void update_cfs_load(struct cfs_r now - cfs_rq->load_last > 4 * period) { cfs_rq->load_period = 0; cfs_rq->load_avg = 0; + delta = period - 1; } cfs_rq->load_stamp = now; ^ permalink raw reply [flat|nested] 15+ messages in thread
* [tip:sched/core] sched: Avoid expensive initial update_cfs_load() 2011-01-22 4:45 ` [patch 5/5] sched: avoid expensive initial update_cfs_load() Paul Turner @ 2011-01-26 12:11 ` tip-bot for Paul Turner 2011-01-26 12:36 ` Peter Zijlstra 2011-01-26 12:45 ` [tip:sched/core] sched: Avoid expensive initial update_cfs_load(), on UP too tip-bot for Peter Zijlstra 2011-01-27 11:58 ` tip-bot for Peter Zijlstra 2 siblings, 1 reply; 15+ messages in thread From: tip-bot for Paul Turner @ 2011-01-26 12:11 UTC (permalink / raw) To: linux-tip-commits Cc: linux-kernel, hpa, mingo, a.p.zijlstra, pjt, tglx, mingo Commit-ID: f07333bf6ee66d9b49286cec4371cf375e745b7a Gitweb: http://git.kernel.org/tip/f07333bf6ee66d9b49286cec4371cf375e745b7a Author: Paul Turner <pjt@google.com> AuthorDate: Fri, 21 Jan 2011 20:45:03 -0800 Committer: Ingo Molnar <mingo@elte.hu> CommitDate: Wed, 26 Jan 2011 12:33:19 +0100 sched: Avoid expensive initial update_cfs_load() Since cfs->{load_stamp,load_last} are zero-initalized the initial load update will consider the delta to be 'since the beginning of time'. This results in a lot of pointless divisions to bring this large period to be within the sysctl_sched_shares_window. Fix this by initializing load_stamp to be 1 at cfs_rq initialization, this allows for an initial load_stamp > load_last which then lets standard idle truncation proceed. We avoid spinning (and slightly improve consistency) by fixing delta to be [period - 1] in this path resulting in a slightly more predictable shares ramp. (Previously the amount of idle time preserved by the overflow would range between [period/2,period-1].) Signed-off-by: Paul Turner <pjt@google.com> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> LKML-Reference: <20110122044852.102126037@google.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> --- kernel/sched.c | 2 ++ kernel/sched_fair.c | 1 + 2 files changed, 3 insertions(+), 0 deletions(-) diff --git a/kernel/sched.c b/kernel/sched.c index e0fa3ff..6820b5b 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -7796,6 +7796,8 @@ static void init_cfs_rq(struct cfs_rq *cfs_rq, struct rq *rq) INIT_LIST_HEAD(&cfs_rq->tasks); #ifdef CONFIG_FAIR_GROUP_SCHED cfs_rq->rq = rq; + /* allow initial update_cfs_load() to truncate */ + cfs_rq->load_stamp = 1; #endif cfs_rq->min_vruntime = (u64)(-(1LL << 20)); } diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index 0c550c8..4cbc912 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -733,6 +733,7 @@ static void update_cfs_load(struct cfs_rq *cfs_rq, int global_update) now - cfs_rq->load_last > 4 * period) { cfs_rq->load_period = 0; cfs_rq->load_avg = 0; + delta = period - 1; } cfs_rq->load_stamp = now; ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [tip:sched/core] sched: Avoid expensive initial update_cfs_load() 2011-01-26 12:11 ` [tip:sched/core] sched: Avoid " tip-bot for Paul Turner @ 2011-01-26 12:36 ` Peter Zijlstra 0 siblings, 0 replies; 15+ messages in thread From: Peter Zijlstra @ 2011-01-26 12:36 UTC (permalink / raw) To: mingo, hpa, linux-kernel, pjt, tglx, mingo; +Cc: linux-tip-commits On Wed, 2011-01-26 at 12:11 +0000, tip-bot for Paul Turner wrote: > index e0fa3ff..6820b5b 100644 > --- a/kernel/sched.c > +++ b/kernel/sched.c > @@ -7796,6 +7796,8 @@ static void init_cfs_rq(struct cfs_rq *cfs_rq, > struct rq *rq) > INIT_LIST_HEAD(&cfs_rq->tasks); > #ifdef CONFIG_FAIR_GROUP_SCHED > cfs_rq->rq = rq; > + /* allow initial update_cfs_load() to truncate */ > + cfs_rq->load_stamp = 1; > #endif > cfs_rq->min_vruntime = (u64)(-(1LL << 20)); > } That wants a fix to build on UP, --- Index: linux-2.6/kernel/sched.c =================================================================== --- linux-2.6.orig/kernel/sched.c +++ linux-2.6/kernel/sched.c @@ -7797,8 +7797,10 @@ static void init_cfs_rq(struct cfs_rq *c #ifdef CONFIG_FAIR_GROUP_SCHED cfs_rq->rq = rq; /* allow initial update_cfs_load() to truncate */ +#ifdef CONFIG_SMP cfs_rq->load_stamp = 1; #endif +#endif cfs_rq->min_vruntime = (u64)(-(1LL << 20)); } ^ permalink raw reply [flat|nested] 15+ messages in thread
* [tip:sched/core] sched: Avoid expensive initial update_cfs_load(), on UP too 2011-01-22 4:45 ` [patch 5/5] sched: avoid expensive initial update_cfs_load() Paul Turner 2011-01-26 12:11 ` [tip:sched/core] sched: Avoid " tip-bot for Paul Turner @ 2011-01-26 12:45 ` tip-bot for Peter Zijlstra 2011-01-27 11:58 ` tip-bot for Peter Zijlstra 2 siblings, 0 replies; 15+ messages in thread From: tip-bot for Peter Zijlstra @ 2011-01-26 12:45 UTC (permalink / raw) To: linux-tip-commits Cc: linux-kernel, hpa, mingo, a.p.zijlstra, pjt, tglx, mingo Commit-ID: 2d65333ab09b9a11722d822231c85b9f251cfe9d Gitweb: http://git.kernel.org/tip/2d65333ab09b9a11722d822231c85b9f251cfe9d Author: Peter Zijlstra <a.p.zijlstra@chello.nl> AuthorDate: Wed, 26 Jan 2011 13:36:03 +0100 Committer: Ingo Molnar <mingo@elte.hu> CommitDate: Wed, 26 Jan 2011 13:43:55 +0100 sched: Avoid expensive initial update_cfs_load(), on UP too Fix the build on UP. Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Turner <pjt@google.com> LKML-Reference: <20110122044852.102126037@google.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> --- kernel/sched.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/kernel/sched.c b/kernel/sched.c index 3fc5749..1f6ca89 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -7931,8 +7931,10 @@ static void init_cfs_rq(struct cfs_rq *cfs_rq, struct rq *rq) #ifdef CONFIG_FAIR_GROUP_SCHED cfs_rq->rq = rq; /* allow initial update_cfs_load() to truncate */ +#ifdef CONFIG_SMP cfs_rq->load_stamp = 1; #endif +#endif cfs_rq->min_vruntime = (u64)(-(1LL << 20)); } ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [tip:sched/core] sched: Avoid expensive initial update_cfs_load(), on UP too 2011-01-22 4:45 ` [patch 5/5] sched: avoid expensive initial update_cfs_load() Paul Turner 2011-01-26 12:11 ` [tip:sched/core] sched: Avoid " tip-bot for Paul Turner 2011-01-26 12:45 ` [tip:sched/core] sched: Avoid expensive initial update_cfs_load(), on UP too tip-bot for Peter Zijlstra @ 2011-01-27 11:58 ` tip-bot for Peter Zijlstra 2 siblings, 0 replies; 15+ messages in thread From: tip-bot for Peter Zijlstra @ 2011-01-27 11:58 UTC (permalink / raw) To: linux-tip-commits Cc: linux-kernel, hpa, mingo, a.p.zijlstra, pjt, tglx, mingo Commit-ID: 6ea72f12069306b235151c5b05ac0cca7e1dedfa Gitweb: http://git.kernel.org/tip/6ea72f12069306b235151c5b05ac0cca7e1dedfa Author: Peter Zijlstra <a.p.zijlstra@chello.nl> AuthorDate: Wed, 26 Jan 2011 13:36:03 +0100 Committer: Ingo Molnar <mingo@elte.hu> CommitDate: Thu, 27 Jan 2011 12:48:14 +0100 sched: Avoid expensive initial update_cfs_load(), on UP too Fix the build on UP. Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Paul Turner <pjt@google.com> LKML-Reference: <20110122044852.102126037@google.com> Signed-off-by: Ingo Molnar <mingo@elte.hu> --- kernel/sched.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/kernel/sched.c b/kernel/sched.c index 78fa753..477e1bc 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -7922,8 +7922,10 @@ static void init_cfs_rq(struct cfs_rq *cfs_rq, struct rq *rq) #ifdef CONFIG_FAIR_GROUP_SCHED cfs_rq->rq = rq; /* allow initial update_cfs_load() to truncate */ +#ifdef CONFIG_SMP cfs_rq->load_stamp = 1; #endif +#endif cfs_rq->min_vruntime = (u64)(-(1LL << 20)); } ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [patch 0/5] scheduler fixlets 2011-01-22 4:44 [patch 0/5] scheduler fixlets Paul Turner ` (4 preceding siblings ...) 2011-01-22 4:45 ` [patch 5/5] sched: avoid expensive initial update_cfs_load() Paul Turner @ 2011-01-24 10:17 ` Peter Zijlstra 5 siblings, 0 replies; 15+ messages in thread From: Peter Zijlstra @ 2011-01-24 10:17 UTC (permalink / raw) To: Paul Turner; +Cc: linux-kernel, Ingo Molnar, Mike Galbraith On Fri, 2011-01-21 at 20:44 -0800, Paul Turner wrote: > A small set of scheduler patches for the CFS side of things. > > Mostly buglet fixes with one or two associated clean-ups. Each patch should > be (modulo any merge fuzz) independent. Thanks Paul, these all look good, applied! ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2011-01-27 11:59 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-01-22 4:44 [patch 0/5] scheduler fixlets Paul Turner 2011-01-22 4:44 ` [patch 1/5] sched: fix sign under-flows in wake_affine Paul Turner 2011-01-26 12:09 ` [tip:sched/core] sched: Fix " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 2/5] sched: (cleanup) remove redundant cfs_rq checks Paul Turner 2011-01-26 12:10 ` [tip:sched/core] sched: Fix/remove " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 3/5] sched: simplify update_cfs_shares parameters Paul Turner 2011-01-26 12:11 ` [tip:sched/core] sched: Simplify " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 4/5] sched: use rq->clock_task instead of rq->clock for maintaining load averages Paul Turner 2011-01-26 12:10 ` [tip:sched/core] sched: Use rq->clock_task instead of rq->clock for correctly " tip-bot for Paul Turner 2011-01-22 4:45 ` [patch 5/5] sched: avoid expensive initial update_cfs_load() Paul Turner 2011-01-26 12:11 ` [tip:sched/core] sched: Avoid " tip-bot for Paul Turner 2011-01-26 12:36 ` Peter Zijlstra 2011-01-26 12:45 ` [tip:sched/core] sched: Avoid expensive initial update_cfs_load(), on UP too tip-bot for Peter Zijlstra 2011-01-27 11:58 ` tip-bot for Peter Zijlstra 2011-01-24 10:17 ` [patch 0/5] scheduler fixlets Peter Zijlstra
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox