[Patch] sched: remove unnecessary sched domains

[Patch] sched: remove unnecessary sched domains

[Siddha, Suresh B]

Appended patch removes the unnecessary scheduler domains(containing
only one sched group) setup during the sched-domain init.

For example on x86_64, we always have NUMA configured in. On Intel EM64T
systems, top most sched domain will be of NUMA and with only one sched_group in
it. 

With fork/exec balances(recent Nick's fixes in -mm tree), we always endup 
taking wrong decisions because of this topmost domain (as it contains only 
one group and find_idlest_group always returns NULL). We will endup loading 
HT package completely first, letting active load balance kickin and correct it.

In general, this patch also makes sense with out recent Nick's fixes
in -mm.

Signed-off-by: Suresh Siddha <suresh.b.siddha@intel.com>


--- linux-2.6.12-rc1-mm4/kernel/sched.c	2005-04-01 10:07:24.183605296 -0800
+++ linux/kernel/sched.c	2005-04-01 15:29:28.274896888 -0800
@@ -4748,6 +4748,21 @@
 	unsigned long flags;
 	runqueue_t *rq = cpu_rq(cpu);
 	int local = 1;
+	struct sched_domain *tmp = sd, *tmp1;
+
+	/* Remove the sched domains which has only one group
+	 */
+	while (tmp) {
+		tmp1 = tmp->parent;
+		if (!tmp1)
+			break;
+		if (tmp1->groups == tmp1->groups->next)
+			tmp->parent = tmp1->parent;
+		tmp = tmp->parent;
+	}
+
+	if (sd->parent && sd->groups && sd->groups == sd->groups->next)
+		sd = sd->parent;
 
 	sched_domain_debug(sd, cpu);
 

Re: [Patch] sched: remove unnecessary sched domains

[Nick Piggin]

Siddha, Suresh B wrote:
> Appended patch removes the unnecessary scheduler domains(containing
> only one sched group) setup during the sched-domain init.
> 
> For example on x86_64, we always have NUMA configured in. On Intel EM64T
> systems, top most sched domain will be of NUMA and with only one sched_group in
> it. 
> 
> With fork/exec balances(recent Nick's fixes in -mm tree), we always endup 
> taking wrong decisions because of this topmost domain (as it contains only 
> one group and find_idlest_group always returns NULL). We will endup loading 
> HT package completely first, letting active load balance kickin and correct it.
> 
> In general, this patch also makes sense with out recent Nick's fixes
> in -mm.
> 

Yeah, this makes sense. We may want to add some other criteria on the
removal of a domain as well (because some of the domain flags do things
that don't use groups).

I don't like so much that we'd rely on it to fix the above problem.
There are a general class of problems with the fork/exec balancing in
that it only works on the top most domain, so it may not spread load over
lower domains very well.

I was thinking we could fix that by running balance on fork/exec multiple
times from top to bottom level domains. I'll have to measure the cost of
doing that, because it may be worthwhile.

Thanks

-- 
SUSE Labs, Novell Inc.

Re: [Patch] sched: remove unnecessary sched domains

[Siddha, Suresh B]

On Sat, Apr 02, 2005 at 12:07:27PM +1000, Nick Piggin wrote:
> Siddha, Suresh B wrote:
> > Appended patch removes the unnecessary scheduler domains(containing
> > only one sched group) setup during the sched-domain init.
> > 
> > For example on x86_64, we always have NUMA configured in. On Intel EM64T
> > systems, top most sched domain will be of NUMA and with only one sched_group in
> > it. 
> > 
> > With fork/exec balances(recent Nick's fixes in -mm tree), we always endup 
> > taking wrong decisions because of this topmost domain (as it contains only 
> > one group and find_idlest_group always returns NULL). We will endup loading 
> > HT package completely first, letting active load balance kickin and correct it.
> > 
> > In general, this patch also makes sense with out recent Nick's fixes
> > in -mm.
> > 
> 
> Yeah, this makes sense. We may want to add some other criteria on the
> removal of a domain as well (because some of the domain flags do things
> that don't use groups).
> 
> I don't like so much that we'd rely on it to fix the above problem.
> There are a general class of problems with the fork/exec balancing in
> that it only works on the top most domain, so it may not spread load over
> lower domains very well.
> 
> I was thinking we could fix that by running balance on fork/exec multiple
> times from top to bottom level domains. I'll have to measure the cost of
> doing that, because it may be worthwhile.

Agreed.

BTW, why are we setting SD_BALANCE_FORK flag for NUMA domain on i386, ia64.
This should be set only on x86_64 and that too not for Intel systems.

thanks,
suresh

Re: [Patch] sched: remove unnecessary sched domains

[Nick Piggin]

Siddha, Suresh B wrote:
> On Sat, Apr 02, 2005 at 12:07:27PM +1000, Nick Piggin wrote:
> 
>>I was thinking we could fix that by running balance on fork/exec multiple
>>times from top to bottom level domains. I'll have to measure the cost of
>>doing that, because it may be worthwhile.
> 
> 
> Agreed.
> 
> BTW, why are we setting SD_BALANCE_FORK flag for NUMA domain on i386, ia64.
> This should be set only on x86_64 and that too not for Intel systems.
> 

Mainly to see whether anyone reports regressions, and to exercise
the code a bit.

-- 
SUSE Labs, Novell Inc.