[PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[Martin J. Bligh]

oops --- the original of this went to Linus, but not LKML, sorry.

------------------------------------

The following sequence of patches finishes the move of NUMA-Q into
subarch, and cleans up some of the mess I made when I put it in
originally. clustered_apic_mode, CONFIG_CLUSTERED_APIC and smpboot.h
all die by the end of the sequence.

There are some topology changes in here, which are needed as a precursor
to the cpu<->apicid mapping cleanups, which are needed to make Summit work.
I've tried hard to break everything out into small readable chunks, and
it's designed to be a complete no-op on standard machines.

Tested on UP, standard SMP, Crusader, NUMA-Q, and Summit (x440).
Test-compiled on UP+IO/APIC. No new compiler warnings are introduced.
Removes more lines of code than it adds, according to diffstat.
This is the last of the stuff for moving NUMA-Q into subarch.

Please apply,

Thanks,

Martin.

RE: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[Protasevich, Natalie]

Martin,

I ran into a couple places where CONFIG_X86_NUMA is still there (replaced
now with CONFIG_X86_NUMAQ), which disables following defines:

in asm-i386/apicdef.h:

...
#ifdef CONFIG_X86_NUMA
 #define MAX_IO_APICS 32
...


and in asm-i386/mpspec.h:
...
/*
 * a maximum of 16 APICs with the current APIC ID architecture.
 */
#ifdef CONFIG_X86_NUMA
#define MAX_APICS 256
#else /* !CONFIG_X86_NUMA */
#define MAX_APICS 16
#endif /* CONFIG_X86_NUMA */
...


--Natalie

RE: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[Martin J. Bligh]

> I ran into a couple places where CONFIG_X86_NUMA is still there (replaced
> now with CONFIG_X86_NUMAQ), which disables following defines:

Yeah, I saw those the other day. Should probably be replaced with 
CONFIG_NUMA ... I'll tweak them once I get the last dribbles of Summit
pushed out to Linus.

Thanks,

M.

RE: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[Pallipadi, Venkatesh]

Can these (MAX_IO_APICS, MAX_APICS) be moved to sub-arch too, instead of
replacing CONFIG NUMA by CONFIG NUMAQ?

Thanks,
-Venkatesh

> -----Original Message-----
> From: Martin J. Bligh [mailto:mbligh@aracnet.com] 
> Sent: Monday, January 13, 2003 10:30 AM
> To: Protasevich, Natalie
> Cc: Pallipadi, Venkatesh; 'William Lee Irwin III'; Nakajima, 
> Jun; 'Christoph Hellwig'; 'James Cleverdon'; 'Linux Kernel'
> Subject: RE: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup
> 
> 
> > I ran into a couple places where CONFIG_X86_NUMA is still 
> there (replaced
> > now with CONFIG_X86_NUMAQ), which disables following defines:
> 
> Yeah, I saw those the other day. Should probably be replaced with 
> CONFIG_NUMA ... I'll tweak them once I get the last dribbles of Summit
> pushed out to Linus.
> 
> Thanks,
> 
> M.
> 
> 

RE: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[Martin J. Bligh]

> Can these (MAX_IO_APICS, MAX_APICS) be moved to sub-arch too, instead of

Not easily, at least not without creating a circular dependency problem.
Try it ;-)

I guess we chould create another subarch header file just for these if we 
really have to, but that seem like overkill.

If you can come up with a clean patch (check it compiles on uniproc with
and without IO/APIC turned on, and standard SMP as well), I'd really 
be interested to see it ... would be most helpful

> replacing CONFIG NUMA by CONFIG NUMAQ?

Actually replacing CONFIG_X86_NUMA with CONFIG_NUMA ... and we could
do (CONFIG_NUMA || CONFIG_BIGSMP) instead. But you're right, subarch
would be much better if you can find a way.

M.

Re: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[William Lee Irwin III]

On Wed, Jan 15, 2003 at 10:29:27AM -0800, Pallipadi, Venkatesh wrote:
> Can these (MAX_IO_APICS, MAX_APICS) be moved to sub-arch too, instead of
> replacing CONFIG NUMA by CONFIG NUMAQ?

Actually, I've also been feeling pain about MAX_IRQ_SOURCES, NR_IRQS,
and HARDIRQ_BITS in addition to MAX_IO_APICS and MAX_APICS. I'll bet
some ppl will have trouble with MAX_MP_BUSSES also, though I don't
have any heavily-populated systems to stress that with.

There are also somewhat deeper issues with vector assignments to
interrupt sources that prevent elevating any of the above to useful
levels and utilizing them. The assumptions based on the vector assignment
algorithm appear to be widely distributed enough to discourage me after
an initial attempt or two to get any kind of useful interrupt routing
for a number of IRQ sources larger than the number of vectors.

I pretty much reprogrammed the IDT to push only the vector and then still
got interrupts on the wrong node(s) despite the physical broadcast RTE's
plus (node, vector) <-> irq accounting and irq number lookup in do_IRQ().


Bill

RE: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[Protasevich, Natalie]

>> Can these (MAX_IO_APICS, MAX_APICS) be moved to sub-arch too, instead of

Yes, pleeese! Without CLUSTERED_APIC I would have to re-define it in some
ugly way in subarch.

>Actually replacing CONFIG_X86_NUMA with CONFIG_NUMA ... and we could
>do (CONFIG_NUMA || CONFIG_BIGSMP) instead. But you're right, subarch
>would be much better if you can find a way.

With BIGSMP, we are still only allowed 16, whereus es7000 needs 256 of
each...

RE: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[Protasevich, Natalie]

>Actually, I've also been feeling pain about MAX_IRQ_SOURCES, NR_IRQS,
>and HARDIRQ_BITS in addition to MAX_IO_APICS and MAX_APICS. I'll bet
>some ppl will have trouble with MAX_MP_BUSSES also, though I don't
>have any heavily-populated systems to stress that with.

... and one more from me: isn't it time to let IO-APIC id be 8 bit in the
asm/io_apic.h (make it a union fot both?..)?
Look what I have to do in io_apic.h to get around it and ... "mister, have a
heart":
=========================================
struct IO_APIC_reg_00 {
	__u32	__reserved_2	: 24,
		ID		:  4,
		__reserved_1	:  4;
} __attribute__ ((packed));
#ifdef CONFIG_ES7000
struct IO_APIC_reg_00_1 {
        __u32   __reserved_2    : 24,
                ID              :  8;
} __attribute__ ((packed));
#endif /* CONFIG_ES7000 */
=========================================

and then in io_apic.c:

=========================================
#ifdef CONFIG_ES7000
	struct IO_APIC_reg_00_1 reg_00_1;
	unsigned long *reg_00_p;
	
	if (es7000_plat)
		reg_00_p = (unsigned long *)&reg_00_1;
	else
		reg_00_p = (unsigned long *)&reg_00;
#endif /*CONFIG_ES7000*/
.......
#ifndef CONFIG_ES7000
		*(int *)&reg_00 = io_apic_read(apic, 0);
#else
		*(int *)reg_00_p = io_apic_read(apic, 0);
#endif /*CONFIG_ES7000*/
.....
====================================================

>There are also somewhat deeper issues with vector assignments to
>interrupt sources that prevent elevating any of the above to useful
>levels and utilizing them. The assumptions based on the vector assignment
>algorithm appear to be widely distributed enough to discourage me after
>an initial attempt or two to get any kind of useful interrupt routing
>for a number of IRQ sources larger than the number of vectors.

I strongly suggest to take a look in IA64 implementation. 
They have 1:1 correspondence between IRQ and vector and don't seem to be
able to run out of vectors or IRQs.

>I pretty much reprogrammed the IDT to push only the vector and then still
>got interrupts on the wrong node(s) despite the physical broadcast RTE's
>plus (node, vector) <-> irq accounting and irq number lookup in do_IRQ().

Re: [PATCH] (0/7) Finish moving NUMA-Q into subarch, cleanup

[William Lee Irwin III]

On Wed, Jan 15, 2003 at 01:30:37PM -0600, Protasevich, Natalie wrote:
> ... and one more from me: isn't it time to let IO-APIC id be 8 bit in the
> asm/io_apic.h (make it a union fot both?..)?
> Look what I have to do in io_apic.h to get around it and ... "mister, have a
> heart":

Point taken; it doesn't burn NUMA-Q, but this probably hits Summit (with
its much more recent APIC and IO-APIC revisions). I don't see why not.
The hardware is there, time to drop in the code to handle it.


At some point in the past, I wrote:
>> There are also somewhat deeper issues with vector assignments to
>> interrupt sources that prevent elevating any of the above to useful
>> levels and utilizing them. The assumptions based on the vector assignment
>> algorithm appear to be widely distributed enough to discourage me after
>> an initial attempt or two to get any kind of useful interrupt routing
>> for a number of IRQ sources larger than the number of vectors.

On Wed, Jan 15, 2003 at 01:30:37PM -0600, Protasevich, Natalie wrote:
> I strongly suggest to take a look in IA64 implementation. 
> They have 1:1 correspondence between IRQ and vector and don't seem to be
> able to run out of vectors or IRQs.

Given that almost nothing actually cares what the irq numbers are, it
sounds like a really good idea to encode the node ID in the upper bits
and the vector in the lower bits (SN2 uses cpuid). I'll try it out.


Thanks!
Bill