Processor IDs on the Niagara

Processor IDs on the Niagara

[Elad Lahav]

I have a question regarding the meaning of cpu_data.proc_id.
My T1000 contains a single T1 processor, with 8 cores and 4 threads per core. The core_id 
value for each hardware thread is set up properly (though starting from 1 rather than 0). 
The proc_id value, however, does not make sense: it maps to core_id - 1. I would have 
expected one of two values:
1. The virtual processor ID, as described in the Machine Description block ("cpu"->"id");
2. A physical processor ID, shared by all 32 threads (and differs among chips on the T2+).
While the former is probably the same as the kernel's CPU id, the latter, so I believe, 
should be part of cpu_data.

A side effect of this issue is that cpu_sibling_map and cpu_core_map are the same (though 
this may be the expected result, I'm not sure).

Here are the per-virtual-processor values on my machine. The format is
CPUID proc_id core_id (sibling map) (core map)

00 00 01 (00,01,02,03) (00,01,02,03)
01 00 01 (00,01,02,03) (00,01,02,03)
02 00 01 (00,01,02,03) (00,01,02,03)
03 00 01 (00,01,02,03) (00,01,02,03)
04 01 02 (04,05,06,07) (04,05,06,07)
05 01 02 (04,05,06,07) (04,05,06,07)
06 01 02 (04,05,06,07) (04,05,06,07)
07 01 02 (04,05,06,07) (04,05,06,07)
08 02 03 (08,09,10,11) (08,09,10,11)
09 02 03 (08,09,10,11) (08,09,10,11)
10 02 03 (08,09,10,11) (08,09,10,11)
11 02 03 (08,09,10,11) (08,09,10,11)
12 03 04 (12,13,14,15) (12,13,14,15)
13 03 04 (12,13,14,15) (12,13,14,15)
14 03 04 (12,13,14,15) (12,13,14,15)
15 03 04 (12,13,14,15) (12,13,14,15)
16 04 05 (16,17,18,19) (16,17,18,19)
17 04 05 (16,17,18,19) (16,17,18,19)
18 04 05 (16,17,18,19) (16,17,18,19)
19 04 05 (16,17,18,19) (16,17,18,19)
20 05 06 (20,21,22,23) (20,21,22,23)
21 05 06 (20,21,22,23) (20,21,22,23)
22 05 06 (20,21,22,23) (20,21,22,23)
23 05 06 (20,21,22,23) (20,21,22,23)
24 06 07 (24,25,26,27) (24,25,26,27)
25 06 07 (24,25,26,27) (24,25,26,27)
26 06 07 (24,25,26,27) (24,25,26,27)
27 06 07 (24,25,26,27) (24,25,26,27)
28 07 08 (28,29,30,31) (28,29,30,31)
29 07 08 (28,29,30,31) (28,29,30,31)
30 07 08 (28,29,30,31) (28,29,30,31)
31 07 08 (28,29,30,31) (28,29,30,31)

Elad

Re: Processor IDs on the Niagara

[David Miller]

From: Elad Lahav <elahav@uwaterloo.ca>
Date: Fri, 12 Sep 2008 17:10:50 -0400

> I have a question regarding the meaning of cpu_data.proc_id.  My
> T1000 contains a single T1 processor, with 8 cores and 4 threads per
> core. The core_id value for each hardware thread is set up properly
> (though starting from 1 rather than 0). The proc_id value, however,
> does not make sense: it maps to core_id - 1.

This is the correct setting.

The kernel treats core_id=0 as special, therefore we start the
indexes there at "1".

The kernel treats proc_id=-1 as special, therefore we start those
indexes at "0".

This is why we initialize core_id=0 and prof_id=-1 before we are
able to figure out what the values should be.

This is what other ports do.

Re: Processor IDs on the Niagara

[Elad Lahav]

> The kernel treats core_id=0 as special, therefore we start the
> indexes there at "1".
> 
> The kernel treats proc_id=-1 as special, therefore we start those
> indexes at "0".

My problem was not with the indexing, but rather with the meaning of 
proc_id. It's neither the physical processor ID, nor the virtual 
processor ID. It's just a one-to-one mapping of the core ID. As the 
virtual processor ID is the same as the kernel's CPU ID, I think that 
proc_id should hold the physical processor ID. If not, than this 
information should be kept in some other field of cpu_data.

Elad

Re: Processor IDs on the Niagara

[Elad Lahav]

> This is what other ports do.

Here's the information from my dual-core athlon processor. Again, the 
format is CPUID proc_id core_id (sibling mask) (core mask):

CPUID  proc_id  core_id  sibling_mask  core_mask
00     00       00       (00)          (00,01)
01     00       01       (01)          (00,01)

As you can see, the processor ID refers to a single package (with ID 0), 
and the core ID is different for each CPU, as expected.

Elad

Re: Processor IDs on the Niagara

[David Miller]

From: Elad Lahav <elahav@uwaterloo.ca>
Date: Fri, 12 Sep 2008 20:47:24 -0400

> > The kernel treats core_id=0 as special, therefore we start the
> > indexes there at "1".
> > The kernel treats proc_id=-1 as special, therefore we start those
> > indexes at "0".
>
> My problem was not with the indexing, but rather with the meaning of
> proc_id. It's neither the physical processor ID, nor the virtual
> processor ID. It's just a one-to-one mapping of the core ID. As the
> virtual processor ID is the same as the kernel's CPU ID, I think
> that proc_id should hold the physical processor ID. If not, than
> this information should be kept in some other field of cpu_data.

For a cpu such as Niagara T1, the proc_id and core_id will represent
the same thing.  Niagara T1 comes only in single socket variants.

For Niagara T2, which can be multisocket, core_id and proc_id can be different.

Look at how this information is obtained from the machine description in
arch/sparc64/kernel/mdesc.c, and then look at how it is used to build the
sibling and core cpu maps, which are used to build the scheduling domain
datastructures in kernel/sched.c

If afterwards you think there is still some problem, thene please
describe a discrete issue you thing arises because of how these
values are set.

Re: Processor IDs on the Niagara

[David Miller]

From: Elad Lahav <elahav@uwaterloo.ca>
Date: Fri, 12 Sep 2008 20:53:11 -0400

> > This is what other ports do.
> 
> Here's the information from my dual-core athlon processor. Again, the format is CPUID proc_id core_id (sibling mask) (core mask):
> 
> CPUID  proc_id  core_id  sibling_mask  core_mask
> 00     00       00       (00)          (00,01)
> 01     00       01       (01)          (00,01)
> 
> As you can see, the processor ID refers to a single package (with ID 0), and the core ID is different for each CPU, as expected.

Niagara T1 is a single "package", with 8 "cores".

Look at a dump from a multi-socket Niagara T2 system.

Re: Processor IDs on the Niagara

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Fri, 12 Sep 2008 18:02:17 -0700 (PDT)

> Niagara T1 is a single "package", with 8 "cores".
> 
> Look at a dump from a multi-socket Niagara T2 system.

BTW, I think I see where your confusion comes from.

We're using core_id and proc_id to represent the physical
internal layout of the cpu in order to guide the scheduling
domain construction.

It doesn't really parallel the conceptual things you think
these values should represent.

On Niagara-T1, the cpu internally has equal resources available
for all virtual cpus within a core.  That's why core_id and
proc_id follow eachother in sequence.

But on Niagara-T2 there are two integer units available amongst
the 8 per-core virtual cpus.

And _that_ is what these values are meant to represent.

For example, on Niagara-T2:

	core_id		proc_id
cpu0:	1		0
cpu1:	1		0
cpu2:	1		0
cpu3:	1		0
cpu4:	1		1
cpu5:	1		1
cpu6:	1		1
cpu7:	1		1

This tells the scheduler to evenly distribute processes inside of this
core between the first 4 virtual cpus and the last 4 virtual cpus.

I hope this clarifies things.

Re: Processor IDs on the Niagara

[Elad Lahav]

> But on Niagara-T2 there are two integer units available amongst
> the 8 per-core virtual cpus.
> 
> And _that_ is what these values are meant to represent.
> 
> For example, on Niagara-T2:
> 
> 	core_id		proc_id
> cpu0:	1		0
> cpu1:	1		0
> cpu2:	1		0
> cpu3:	1		0
> cpu4:	1		1
> cpu5:	1		1
> cpu6:	1		1
> cpu7:	1		1

But what happens on the T2+? How is a physical package represented? 
Clearly the scheduler needs to be aware of the fact that two hardware 
threads are on different physical processors. As far as I know, the 
scheduler supports 4 levels: threads, cores, physical processors and 
NUMA nodes. I don't see how the current scheme falls into these categories.

Elad

Re: Processor IDs on the Niagara

[David Miller]

From: Elad Lahav <elahav@uwaterloo.ca>
Date: Fri, 12 Sep 2008 21:27:20 -0400

> > But on Niagara-T2 there are two integer units available amongst
> > the 8 per-core virtual cpus.
> > And _that_ is what these values are meant to represent.
> > For example, on Niagara-T2:
> > 	core_id		proc_id
> > cpu0:	1		0
> > cpu1:	1		0
> > cpu2:	1		0
> > cpu3:	1		0
> > cpu4:	1		1
> > cpu5:	1		1
> > cpu6:	1		1
> > cpu7:	1		1
> 
> But what happens on the T2+? How is a physical package represented? 

It is represented with the NUMA scheduler domain.  On Niagara-T2+ each
physical socket is a NUMA node.

> Clearly the scheduler needs to be aware of the fact that two
> hardware threads are on different physical processors. As far as I
> know, the scheduler supports 4 levels: threads, cores, physical
> processors and NUMA nodes. I don't see how the current scheme falls
> into these categories.

cpu ID                   = thread
core                     = core
Integer unit within core = physical processor
physical socket          = NUMA node

Re: Processor IDs on the Niagara

[Elad Lahav]

Sorry to be nagging about this...
Maybe I should have provided some motivation in the first place:
Knowing the processor topology may be of importance to a certain class of programmes. 
Specifically, I/O-intensive applications (such as a web server), may want to ensure that 
interrupts are serviced close to the process/thread executing the service. I have run some 
tests that show that servicing interrupts on a different physical package than the one 
responsible for the synchronous part of the service can result in very poor performance 
(for obvious reasons).

> Niagara T1 is a single "package", with 8 "cores".
Exactly my point. So why isn't there a single physical package ID for all CPUs under 
/sys/devices/system/cpu/cpuX/topology/physical_package_id ?

Elad

Re: Processor IDs on the Niagara

[David Miller]

From: Elad Lahav <elahav@uwaterloo.ca>
Date: Mon, 15 Sep 2008 17:26:38 -0400

> Sorry to be nagging about this...
> Maybe I should have provided some motivation in the first place:
> Knowing the processor topology may be of importance to a certain class of programmes. Specifically, I/O-intensive applications (such as a web server), may want to ensure that interrupts are serviced close to the process/thread executing the service. I have run some tests that show that servicing interrupts on a different physical package than the one responsible for the synchronous part of the service can result in very poor performance (for obvious reasons).

The interrupts will arrive at a certain processor, will wake up
the process, and if this happens enough the process will migrate
to that cpu.

We don't migrate interrupts dynamically on sparc64 like other
platforms do (it's too expensive), they stay fixed at their
choosen locations.

> > Niagara T1 is a single "package", with 8 "cores".
> Exactly my point. So why isn't there a single physical package ID for all CPUs under /sys/devices/system/cpu/cpuX/topology/physical_package_id ?

I misspoke here, this is not how I use the topology.

Re: Processor IDs on the Niagara

[Elad Lahav]

> The interrupts will arrive at a certain processor, will wake up
> the process, and if this happens enough the process will migrate
> to that cpu.
True, except that on the Niagara this may not be the desired configuration, as it would
result in both the process and the interrupt handling executing on the same strand.
I've just tested it, and this seems to be the case. I don't know how the scheduler is
tuned for inter-strand load-balancing, but it seems like having the process on a different
strand on the same core would be preferable.

>>> Niagara T1 is a single "package", with 8 "cores".
>> Exactly my point. So why isn't there a single physical package ID for all CPUs under /sys/devices/system/cpu/cpuX/topology/physical_package_id ?
> 
> I misspoke here, this is not how I use the topology.
If proc_id is used by the scheduler and is supposed to indicate the internal core
arrangement, then cpu_data should have an extra phys_proc_id field, similar to other
architectures (which would then be exported via physical_package_id).

Elad

Re: Processor IDs on the Niagara

[David Miller]

From: Elad Lahav <elahav@uwaterloo.ca>
Date: Tue, 16 Sep 2008 14:47:39 -0400

> If proc_id is used by the scheduler and is supposed to indicate the
> internal core arrangement, then cpu_data should have an extra
> phys_proc_id field, similar to other architectures (which would then
> be exported via physical_package_id).

The cpumasks are used by the scheduler to build the scheduler
to build the scheduler domain.

But, yes, those are built based upon the proc_id and core_id
values.

Re: Processor IDs on the Niagara

[Elad Lahav]

>>> Niagara T1 is a single "package", with 8 "cores".
>> Exactly my point. So why isn't there a single physical package ID for all CPUs under /sys/devices/system/cpu/cpuX/topology/physical_package_id ?
> 
> I misspoke here, this is not how I use the topology.

Just in case you decide to fix the physical_package_id numbers, here's a possible patch. 
It uses the serial number of the processor to distinguish between physical CPUs, and 
assigns an ID based on the number of distinct serial numbers found.
The only caveat (I know of) is that it increases the size of cpu_data beyond 2 cache lines 
(not sure if that means much in practice, though).

--Elad

diff -r -u linux-2.6.26.5-org/arch/sparc64/kernel/mdesc.c 
linux-2.6.26.5/arch/sparc64/kernel/mdesc.c
--- linux-2.6.26.5-org/arch/sparc64/kernel/mdesc.c      2008-09-08 13:40:20.000000000 -0400
+++ linux-2.6.26.5/arch/sparc64/kernel/mdesc.c  2008-09-17 14:17:32.000000000 -0400
@@ -762,10 +762,44 @@
         get_one_mondo_bits(val, &tb->nonresum_qmask, 2);
  }

+static void set_phys_proc_id(struct mdesc_handle *hp, u64 mp, int cpuid,
+                             int* new_phys_id)
+{
+       const u64 *sernum = mdesc_get_property(hp, mp, "serial#", NULL);
+       int i;
+
+       cpu_data(cpuid).phys_proc_id = -1;
+
+       if (sernum = NULL) {
+               printk(KERN_INFO "CPU %d: no serial# property\n", cpuid);
+               return;
+       }
+
+       cpu_data(cpuid).serial_num = *sernum;
+
+#ifdef CONFIG_SMP
+       for_each_cpu_mask(i, cpu_present_map) {
+               if (cpu_data(i).serial_num = *sernum) {
+                       cpu_data(cpuid).phys_proc_id = cpu_data(i).phys_proc_id;
+                       break;
+               }
+       }
+
+       if (cpu_data(cpuid).phys_proc_id = -1)
+               cpu_data(cpuid).phys_proc_id = (*new_phys_id)++;
+#else
+       cpu_data(cpuid).phys_proc_id = new_phys_id;
+#endif
+       printk(KERN_INFO "CPU %d: serial#=%lu phys_proc_id=%d\n", cpuid,
+              cpu_data(cpuid).serial_num,
+              cpu_data(cpuid).phys_proc_id);
+}
+
  void __cpuinit mdesc_fill_in_cpu_data(cpumask_t mask)
  {
         struct mdesc_handle *hp = mdesc_grab();
         u64 mp;
+       int phys_id = 0;

         ncpus_probed = 0;
         mdesc_for_each_node_by_name(hp, mp, "cpu") {
@@ -825,6 +859,8 @@
                         }
                 }

+               set_phys_proc_id(hp, mp, cpuid, &phys_id);
+
  #ifdef CONFIG_SMP
                 cpu_set(cpuid, cpu_present_map);
  #endif
diff -r -u linux-2.6.26.5-org/include/asm-sparc64/cpudata.h 
linux-2.6.26.5/include/asm-sparc64/cpudata.h
--- linux-2.6.26.5-org/include/asm-sparc64/cpudata.h    2008-09-08 13:40:20.000000000 -0400
+++ linux-2.6.26.5/include/asm-sparc64/cpudata.h        2008-09-17 13:32:22.000000000 -0400
@@ -32,6 +32,10 @@
         unsigned int    ecache_line_size;
         int             core_id;
         int             proc_id;
+
+       /* Dcache line 3, rarely used */
+       u64             serial_num;
+       int             phys_proc_id;
  } cpuinfo_sparc;

  DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
diff -r -u linux-2.6.26.5-org/include/asm-sparc64/topology.h 
linux-2.6.26.5/include/asm-sparc64/topology.h
--- linux-2.6.26.5-org/include/asm-sparc64/topology.h   2008-09-08 13:40:20.000000000 -0400
+++ linux-2.6.26.5/include/asm-sparc64/topology.h       2008-09-17 13:31:42.000000000 -0400
@@ -73,7 +73,7 @@
  #endif /* !(CONFIG_NUMA) */

  #ifdef CONFIG_SMP
-#define topology_physical_package_id(cpu)      (cpu_data(cpu).proc_id)
+#define topology_physical_package_id(cpu)      (cpu_data(cpu).phys_proc_id)
  #define topology_core_id(cpu)                  (cpu_data(cpu).core_id)
  #define topology_core_siblings(cpu)            (cpu_core_map[cpu])
  #define topology_thread_siblings(cpu)          (per_cpu(cpu_sibling_map, cpu))

Re: Processor IDs on the Niagara

[David Miller]

From: Elad Lahav <elahav@uwaterloo.ca>
Date: Wed, 17 Sep 2008 14:45:58 -0400

> >>> Niagara T1 is a single "package", with 8 "cores".
> >> Exactly my point. So why isn't there a single physical package ID for all CPUs under /sys/devices/system/cpu/cpuX/topology/physical_package_id ?
> > I misspoke here, this is not how I use the topology.
> 
> Just in case you decide to fix the physical_package_id numbers, here's a possible patch. It uses the serial number of the processor to distinguish between physical CPUs, and assigns an ID based on the number of distinct serial numbers found.
> The only caveat (I know of) is that it increases the size of cpu_data beyond 2 cache lines (not sure if that means much in practice, though).

You didn't even test boot this, you can't change the size of cpuinfo_sparc
without fixing all the hardcoded assembler indexing of that structure.

First, you're need to pad it out to the next power of two, then you'd
need to update the shift constant macro that all the assembler uses.

But I'm really not interested in changing this anyways.

Re: Processor IDs on the Niagara

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Sun, 05 Oct 2008 13:47:11 -0700 (PDT)

> You didn't even test boot this, you can't change the size of cpuinfo_sparc
> without fixing all the hardcoded assembler indexing of that structure.
> 
> First, you're need to pad it out to the next power of two, then you'd
> need to update the shift constant macro that all the assembler uses.

Sorry, my bad, I was thinking of the trap_info.  That structure
needs such careful handling, this one doesn't.

Re: Processor IDs on the Niagara

[Elad Lahav]

> You didn't even test boot this
Of course I did. I've been running with a patched kernel for the past couple of weeks, 
without any problems.

> But I'm really not interested in changing this anyways.
Up to you.

--Elad