[2.4.21-pre3] APIC routing broken on ASUS P2B-DS

[2.4.21-pre3] APIC routing broken on ASUS P2B-DS

[Simon Kirby]

Something broke between 2.4.20 and 2.4.21-pre3 which is causing
interrupts to not be routed the second CPU.  I saw the problem on one box
and copied the kernel to another which then had the same problem (both
ASUS P2B-DS boards, one with PIII CPUs, one with PII CPUs).  

[sroot@devel:/]# cat /proc/interrupts
           CPU0       CPU1       
  0:     114480          0    IO-APIC-edge  timer
  1:          2          0    IO-APIC-edge  keyboard
  2:          0          0          XT-PIC  cascade
  8:          3          0    IO-APIC-edge  rtc
 16:      30707          0   IO-APIC-level  eth0
 19:       3260          0   IO-APIC-level  aic7xxx
NMI:          0          0 
LOC:     114405     114404 
ERR:          0
MIS:          0

With 2.4.20:

[sroot@devel:/root]# cat /proc/interrupts 
           CPU0       CPU1       
  0:      23627      27340    IO-APIC-edge  timer
  1:          0          2    IO-APIC-edge  keyboard
  2:          0          0          XT-PIC  cascade
  8:          2          1    IO-APIC-edge  rtc
 16:       1671          2   IO-APIC-level  eth0
 19:        965        979   IO-APIC-level  aic7xxx
NMI:          0          0 
LOC:      50893      50891 
ERR:          0
MIS:          0

...If nobody knows what happened immediately, I'll try narrowing it down
to the patch.

Simon-

[        Simon Kirby        ][        Network Operations        ]
[     sim@netnation.com     ][     NetNation Communications     ]
[  Opinions expressed are not necessarily those of my employer. ]

Re: [2.4.21-pre3] APIC routing broken on ASUS P2B-DS

[Willy Tarreau]

On Mon, Jan 27, 2003 at 04:49:06PM -0800, Simon Kirby wrote:
> Something broke between 2.4.20 and 2.4.21-pre3 which is causing
> interrupts to not be routed the second CPU.  I saw the problem on one box
> and copied the kernel to another which then had the same problem (both
> ASUS P2B-DS boards, one with PIII CPUs, one with PII CPUs).  
> 
> [sroot@devel:/]# cat /proc/interrupts
>            CPU0       CPU1       
>   0:     114480          0    IO-APIC-edge  timer
>   1:          2          0    IO-APIC-edge  keyboard
>   2:          0          0          XT-PIC  cascade
>   8:          3          0    IO-APIC-edge  rtc
>  16:      30707          0   IO-APIC-level  eth0
>  19:       3260          0   IO-APIC-level  aic7xxx
> NMI:          0          0 
> LOC:     114405     114404 
> ERR:          0
> MIS:          0

Hi !

Well, yesterday I've noticed the same problem here on an Asus A7M266-D. I
haven't had time to test every previous version, but I've searched through the
archives, and other similar reports have been posted since December. I believe
that some people did have it with 2.4.21pre2 too.

Your report narrows the problem down to fewer releases than what I saw first,
and if I have time I'll test 2.4.20 and 2.4.21pre1 here.

BTW, I don't know if -ac exhibits the same problem. Alan, would you check your
SMP box, please ?

Cheers,
Willy
 

[Nearly Solved]: APIC routing broken on ASUS P2B-DS

[Willy Tarreau]

On Tue, Jan 28, 2003 at 07:06:29AM +0100, Willy Tarreau wrote:
> On Mon, Jan 27, 2003 at 04:49:06PM -0800, Simon Kirby wrote:
> > Something broke between 2.4.20 and 2.4.21-pre3 which is causing
> > interrupts to not be routed the second CPU.  I saw the problem on one box
> > and copied the kernel to another which then had the same problem (both
> > ASUS P2B-DS boards, one with PIII CPUs, one with PII CPUs).  

Hi !

I noticed that 2.4.21-pre4 had the same problem whereas -ac1 and -aa1 worked
fine. But unfortunately, this was unrelated since both use irq_balance which
seems to work around or fix the problem. So I searched back the earlier
versions, and finally narrowed this problem down to the introduction of
CONFIG_X86_NUMA and associated code in 2.4.21pre1.

If I compile my kernel for an SMP K7, only CPU0 gets the interrupts. But if
I enable CONFIG_X86_CLUSTERED_APIC by enabling either CONFIG_X86_NUMAQ or
CONFIG_X86_SUMMIT (CONFIG_X86_NUMA alone isn't enough), then I get my interrupts
distributed across both CPUs. This is on an Asus A7M266D with 2 Athlon XP 1800+.
I don't know if this option can affect performance or stability.
BTW, the system runs in Flat APIC mode, as reported at boot time. I can provide
dmesg on request, but didn't want to pollute the list.

I looked through the code but since I don't know much about APIC, I didn't
understand the changes nor how they would affect what I observed.

Anyone has any clue ?

Cheers,
Willy

Re: [Nearly Solved]: APIC routing broken on ASUS P2B-DS

[Jeff Garzik]

Willy Tarreau wrote:
> If I compile my kernel for an SMP K7, only CPU0 gets the interrupts. But if
> I enable CONFIG_X86_CLUSTERED_APIC by enabling either CONFIG_X86_NUMAQ or
> CONFIG_X86_SUMMIT (CONFIG_X86_NUMA alone isn't enough), then I get my interrupts
> distributed across both CPUs. This is on an Asus A7M266D with 2 Athlon XP 1800+.


did you ever get a response on this?

The answer is a big fat "don't do that" ;-)

Summit and Numa are two things your box definitely does not have... 
don't enable those options.  If you still have problems with those 
options disabled, please re-post...

	Jeff

Re: [Nearly Solved]: APIC routing broken on ASUS P2B-DS

[Willy Tarreau]

On Tue, Feb 18, 2003 at 02:11:12AM -0500, Jeff Garzik wrote:
> Willy Tarreau wrote:
> >If I compile my kernel for an SMP K7, only CPU0 gets the interrupts. But if
> >I enable CONFIG_X86_CLUSTERED_APIC by enabling either CONFIG_X86_NUMAQ or
> >CONFIG_X86_SUMMIT (CONFIG_X86_NUMA alone isn't enough), then I get my 
> >interrupts
> >distributed across both CPUs. This is on an Asus A7M266D with 2 Athlon XP 
> >1800+.
> 
> 
> did you ever get a response on this?

Well, Maciej Rozycki replied to me privately saying that he would look at the
problem though he didn't have time right now. I can't blame him, I didn't
either...
 
> The answer is a big fat "don't do that" ;-)
> 
> Summit and Numa are two things your box definitely does not have... 
> don't enable those options.  If you still have problems with those 
> options disabled, please re-post...

If I manage to get a few hours, I might compare the initialization code between
these different modes and try to determine what should be added or removed in
the classical code to make it work. Unfortunately, I'm fairly busy right now,
and trying to understand this unknown area will take some time :-(

Willy