Problems with O(1) scheduler on non-x86 arch's

Problems with O(1) scheduler on non-x86 arch's

[James Bottomley]

[-- Attachment #1: Type: text/plain, Size: 2870 bytes --]

I've been on a bug hunting expedition to try to make the new scheduler work with the voyager architecture port in kernel 2.4.2.  I'm not sure if this is a side effect of the port or an intentional architecture shift.  Some of the architecture ports (mine included) store the physical cpu number in p->processor (now p->cpu) rather than the logical one.  I'm not sure if shifting to logical cpu numbering was behind the name change or not, but anyway, in case this is merely an oversight, the attached patch fixes the problem for me.

James Bottomley
Content-Type: multipart/mixed ;
	boundary="==_Exmh_20997329790"
--==_Exmh_20997329790--
This is a multipart MIME message.

--==_Exmh_20997329790


--==_Exmh_20997329790
Content-Type: text/plain ; name="sched.diff"; charset=us-ascii
Content-Description: sched.diff
Content-Disposition: attachment; filename="sched.diff"

Index: kernel/fork.c
===================================================================
RCS file: /home/jejb/CVSROOT/linux/2.5/kernel/fork.c,v
retrieving revision 1.1.1.5
diff -u -r1.1.1.5 fork.c
--- kernel/fork.c	16 Jan 2002 17:25:33 -0000	1.1.1.5
+++ kernel/fork.c	17 Jan 2002 02:18:29 -0000
@@ -650,7 +650,8 @@
 
 		/* ?? should we just memset this ?? */
 		for(i = 0; i < smp_num_cpus; i++)
-			p->per_cpu_utime[i] = p->per_cpu_stime[i] = 0;
+			p->per_cpu_utime[cpu_logical_map(i)] =
+				p->per_cpu_stime[cpu_logical_map(i)] = 0;
 		spin_lock_init(&p->sigmask_lock);
 	}
 #endif
Index: kernel/sched.c
===================================================================
RCS file: /home/jejb/CVSROOT/linux/2.5/kernel/sched.c,v
retrieving revision 1.1.1.4
diff -u -r1.1.1.4 sched.c
--- kernel/sched.c	16 Jan 2002 17:25:33 -0000	1.1.1.4
+++ kernel/sched.c	17 Jan 2002 02:18:29 -0000
@@ -267,7 +267,7 @@
 	unsigned long i, sum = 0;
 
 	for (i = 0; i < smp_num_cpus; i++)
-		sum += cpu_rq(i)->nr_running;
+		sum += cpu_rq(cpu_logical_map(i))->nr_running;
 
 	return sum;
 }
@@ -277,7 +277,7 @@
 	unsigned long i, sum = 0;
 
 	for (i = 0; i < smp_num_cpus; i++)
-		sum += cpu_rq(i)->nr_switches;
+		sum += cpu_rq(cpu_logical_map(i))->nr_switches;
 
 	return sum;
 }
@@ -287,7 +287,7 @@
 	unsigned long i, curr, max = 0;
 
 	for (i = 0; i < smp_num_cpus; i++) {
-		curr = cpu_rq(i)->nr_running;
+		curr = cpu_rq(cpu_logical_map(i))->nr_running;
 		if (curr > max)
 			max = curr;
 	}
@@ -327,7 +327,7 @@
 	busiest = NULL;
 	max_load = 0;
 	for (i = 0; i < smp_num_cpus; i++) {
-		rq_tmp = cpu_rq(i);
+		rq_tmp = cpu_rq(cpu_logical_map(i));
 		load = rq_tmp->nr_running;
 		if ((load > max_load) && (load < prev_max_load) &&
 						(rq_tmp != this_rq)) {




-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Problems with O(1) scheduler on non-x86 arch's

[Ingo Molnar]

On Wed, 16 Jan 2002, James Bottomley wrote:

> I've been on a bug hunting expedition to try to make the new scheduler
> work with the voyager architecture port in kernel 2.4.2.  I'm not sure
> if this is a side effect of the port or an intentional architecture
> shift.  Some of the architecture ports (mine included) store the
> physical cpu number in p->processor (now p->cpu) rather than the
> logical one.  I'm not sure if shifting to logical cpu numbering was
> behind the name change or not, but anyway, in case this is merely an
> oversight, the attached patch fixes the problem for me.

it's merely an oversight.

in the long term i think the correct approach would be to always store the
logical CPU number in p->cpu. Architectures that have some strange
physical numbering can always do a mapping themselves. This way we could
remove tons of cpu-id conversions from the generic code. And we are always
going to have these problems, now that the x86 SMP boot code renumbers
physical CPU ids to match the logical order.

i've checked the architectures, and besides your architecture it appears
that only Alpha does a non-identity ID conversion. So it would be much
cleaner for the generic kernel if we stored the logical CPU id in p->cpu,
and all SMP interfaces towards lowlevel SMP code used the logical CPU ID.

	Ingo

Re: Problems with O(1) scheduler on non-x86 arch's

[James Bottomley]

mingo@elte.hu said:
> in the long term i think the correct approach would be to always store
> the logical CPU number in p->cpu. Architectures that have some strange
> physical numbering can always do a mapping themselves. This way we
> could remove tons of cpu-id conversions from the generic code. And we
> are always going to have these problems, now that the x86 SMP boot
> code renumbers physical CPU ids to match the logical order. 

I can see for something like the x86, where the apic id bears no relationship 
to where the CPU is and there's no support for hot CPU removal, this approach 
may be the correct one.  For Voyager, each CPU has an activity light. So, from 
the physical CPU number, I can tell what the processor is doing, thus it does 
make more sense for me to keep a physical number in p->cpu rather than using 
the logical one.  The voyager architecture is less constrained about ASMP as 
well (I currently run an 8 way box with 4x66MHz and 4x33MHz CPUs).  If I'm to 
use the process or interrupt affinity features, I really need to know 
physically which CPU I want.

Think about the problems logical numbering will cause for hot processor 
removal or failure (if we get around to implementing the feature).  If p->cpu 
is physical, all I have to do is remove the mapping and decrement 
smp_num_cpus, quiesce the processor and redistribute the run queue.  In a 
logically mapped system, I'm going to have to renumber p->cpu in at least one 
other CPUs runqueue as well since the logical mapping code assumes no gaps in 
the 0...smp_num_cpus-1 sequence.

> i've checked the architectures, and besides your architecture it
> appears that only Alpha does a non-identity ID conversion. So it would
> be much cleaner for the generic kernel if we stored the logical CPU id
> in p->cpu, and all SMP interfaces towards lowlevel SMP code used the
> logical CPU ID. 

Actually, the parisc SMP code (on cvs.parisc-linux.org) uses physical 
numbering and it looks to me like the sparc SMP code does as well.

James

Re: Problems with O(1) scheduler on non-x86 arch's

[Rusty Russell]

On Thu, 17 Jan 2002 10:50:58 +0100 (CET)
Ingo Molnar <mingo@elte.hu> wrote:

> in the long term i think the correct approach would be to always store the
> logical CPU number in p->cpu.

The hotplug CPU patch gets rid of the whole concept of "logical CPU number".
This is even cleaner, and avoids these mistakes which bite us all the time.

	http://www.kernel.org/pub/linux/kernel/people/rusty/patches/Hotcpu

Rusty.
-- 
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.