Dubious MIPS kernel SMP Structures

Dubious MIPS kernel SMP Structures

[Kevin D. Kissell]

In arch/mips/kerenl/smp.c, there are two tables defined, __cpu_number_map[]
and __cpu_logical_map[], which would appear to provide forward and backward
mapping between a set of unique but arbitrary CPU numbers and a monotonically
increasing number 0..n of indices into per-CPU data.   As near as I can tell, the
only use of this is in the sb1250 code for setting up interrupt hardware.  Is there
a reason why it's defined at the mips/kernel level, and not down in the SiByte
platform subtree?  Is there a generic, architectural definition of how these mappings
should and should not be set up and used?

            Regards,

            Kevin K.

Re: Dubious MIPS kernel SMP Structures

[Ralf Baechle]

On Wed, Nov 17, 2004 at 04:29:21PM +0100, Kevin D. Kissell wrote:

> In arch/mips/kerenl/smp.c, there are two tables defined, __cpu_number_map[]
> and __cpu_logical_map[], which would appear to provide forward and backward
> mapping between a set of unique but arbitrary CPU numbers and a monotonically
> increasing number 0..n of indices into per-CPU data.   As near as I can tell, the
> only use of this is in the sb1250 code for setting up interrupt hardware.  Is there
> a reason why it's defined at the mips/kernel level, and not down in the SiByte
> platform subtree?  Is there a generic, architectural definition of how these mappings
> should and should not be set up and used?

The Linux kernel is living in the assumption of having a dense CPU number
space and being started by cpu 0 - something that isn't necessarily
reflected by the underlying hardware.  The system for which this concept
was introduced into Linux is SGI's IP27.  IP27 permits dividing of large
systems into multiple independant partitions.  Assume a two module
Origin 2000; let's assume it's fully configured with 16 processors.
Parititioning results in two domains.  The first consists of processors
0 - 7; the second 8 - 15.  Just for the kicks let's assume physical
processors 11 and 13 are broken and therfor were disabled.  From point of
view of the generic Linux kernel this leaves a 6 processor system.  So
when we boot the kernel it will initialize these two mappings to the
following mapping:

  logical     physical
   CPU #  ->    CPU #
    0            8
    1            9
    2           10
    3           12
    4           14
    5           15

  physical     logical  
   CPU #   ->   CPU #
     8           0
     9           1
    10           2
    11	  			# broken CPU, we're skipping this CPU number
    12           3
    13	  			# broken CPU, we're skipping this CPU number
    14           4
    15           5

As you found the IP27 code doesn't properly setup these mappings anymore;
partly because it's SMP initialization code is twisted to the point
where nobody understands it anymore.  Partly also because the systems
we used as SGI were too large to leave CPU numbers unused :)

Honestly no idea why the Sibyte code is using that mapping stuff.  The
Sibyte firmware is always launching the kernel on CPU 0 anyway so we have
the case of either only CPU 0 or both CPU 0 and CPU 1 which means the
mapping would always be a 1:1 mapping.

For most simple SMP or ccNUMA configurations assuming a 1:1 mapping is
reasonable.  For some uniprocessor configurations where a uniprocessor
kernel is running on a single processor other than processor number 0 on
a multiprocessor platform this also may be useful.

  Ralf

Re: Dubious MIPS kernel SMP Structures

[Kevin D. Kissell]

Ralf Baechle wrote:

[snip of lots of cool historical explanations]

 > As you found the IP27 code doesn't properly setup these mappings anymore;
 > partly because it's SMP initialization code is twisted to the point
 > where nobody understands it anymore.  Partly also because the systems
 > we used as SGI were too large to leave CPU numbers unused :)
 >
 > Honestly no idea why the Sibyte code is using that mapping stuff.  The
 > Sibyte firmware is always launching the kernel on CPU 0 anyway so we have
 > the case of either only CPU 0 or both CPU 0 and CPU 1 which means the
 > mapping would always be a 1:1 mapping.
 >
 > For most simple SMP or ccNUMA configurations assuming a 1:1 mapping is
 > reasonable.  For some uniprocessor configurations where a uniprocessor
 > kernel is running on a single processor other than processor number 0 on
 > a multiprocessor platform this also may be useful.

But my question is really one of why it is that the platform-independent
MIPS kernel code needs/needed to know anything about physical CPU numbers?
Naively, I would have thought that any such mapping would be burried
in the platform code, and that the architectural kernel code would
simply invoke (possibly null) platform-level functions that do whatever
mapping of logical 0...N CPU numbers to bizarre mesh node numbers might
be necessary. As it stands, people who have no need to do mapping (SiByte,
PMC-Sierra, and stuff we're doing at MIPS around MIPS MT) are mindlessly
replicating code to set up 1:1 mappings that will never (in the PMC-Sierra
and MIPS cases) be referenced.

		Regards,

		Kevin K.