Why is "noapic" with some Dual P3 boards necessary?

Why is "noapic" with some Dual P3 boards necessary?

[Matthias Kleine]

Dear kernel developers,

I am using an ASUS CUV4X-D mainboard since the days this
board came out (about beginning of 2001). Right from the
beginning, it was necessary to start the kernel with the "noapic"
option. This is, of course, a limitation of an SMP system, because
there are not all the IRQ's of both CPU's available when booted
"noapic". So I guess it must have some strong technical reason
why this kind of board cannot be used in apic mode. Can anybody
explain me the very reason why this is the case?
-- 
   ______ ______
  /_   _//_  __/
Ma / /    / / hias
  /_/    /_/

Re: Why is "noapic" with some Dual P3 boards necessary?

[Alan Cox]

On Sun, 2002-10-27 at 11:08, Matthias Kleine wrote:
> Dear kernel developers,
> 
> I am using an ASUS CUV4X-D mainboard since the days this
> board came out (about beginning of 2001). Right from the
> beginning, it was necessary to start the kernel with the "noapic"
> option. This is, of course, a limitation of an SMP system, because
> there are not all the IRQ's of both CPU's available when booted
> "noapic". So I guess it must have some strong technical reason
> why this kind of board cannot be used in apic mode. Can anybody
> explain me the very reason why this is the case?

Three reasons depending on the actual board/bios.

1. Some boards don't have the APIC stuff properly wired so APIC won't
work. Not something you should ever see on SMP boards
2. The APIC apparently has bugs (eg 440BX) and so interrupts
occasionally "vanish" with it enabled
3. BIOS APIC IRQ routing tables are wrong so the hardware may work but
the actual connections don't match up 

Re: Why is "noapic" with some Dual P3 boards necessary?

[Vladimir G. Ivanovic]

It is not necessary to boot a CUV4X-DLS board with the "noapic" option.

I have been running a CUV4X-DLS board (== CUV4X-D + LAN + SCSI) for over
a year now with APIC. You must however fall back to MPS 1.1 from the
default MPS 1.4. (It's a BIOS setting.) 

I suspect that a CUV4X-D board will have the same behavior.

--- Vladimir

------------------------------------------------------------------------
Vladimir G. Ivanovic                        http://leonora.org/~vladimir
2770 Cowper St.                                         vladimir@acm.org
Palo Alto, CA 94306-2447                                 +1 650 678 8014

"MK" == Matthias Kleine <kleine_matthias@gmx.de> writes:

  MK> I am using an ASUS CUV4X-D mainboard since the days this
  MK> board came out (about beginning of 2001). Right from the
  MK> beginning, it was necessary to start the kernel with the "noapic"
  MK> option. This is, of course, a limitation of an SMP system, because
  MK> there are not all the IRQ's of both CPU's available when booted
  MK> "noapic". So I guess it must have some strong technical reason
  MK> why this kind of board cannot be used in apic mode. Can anybody
  MK> explain me the very reason why this is the case?

Re: Why is "noapic" with some Dual P3 boards necessary?

[Matthias Kleine]

On Tuesday 29 October 2002 18:03, Vladimir G. Ivanovic wrote:
> It is not necessary to boot a CUV4X-DLS board with the "noapic" option.
>
> I have been running a CUV4X-DLS board (== CUV4X-D + LAN + SCSI) for over
> a year now with APIC. You must however fall back to MPS 1.1 from the
> default MPS 1.4. (It's a BIOS setting.)
>
> I suspect that a CUV4X-D board will have the same behavior.

Without noapic as kernel option

- my eth0 and ippp0 devices do not work
- my NVIDIA driver does not work

With noapic (no other changes) everything works quite
fine. I think you need a _very strong_ reason to convince
me that noapic is not necessary with the CUV4X-D.
-- 
   ______ ______
  /_   _//_   _/
Ma / /    / / hias
  /_/    /_/

Re: Why is "noapic" with some Dual P3 boards necessary?

[Dr. Michael Weller]

On Wed, 30 Oct 2002, Matthias Kleine wrote:

> On Tuesday 29 October 2002 18:03, Vladimir G. Ivanovic wrote:
[...]
> > a year now with APIC. You must however fall back to MPS 1.1 from the
> > default MPS 1.4. (It's a BIOS setting.)
[...]
> Without noapic as kernel option
[...] do not work
> - my NVIDIA driver does not work

Experience with other boards (some compaq) on my side and also what was
written in this thread already shows, that BIOS settings modify how
the apic is setup and hence if linux needs noapic.

You should check if you can modify the bios setting as was described.

If not, or if a modification does not help, it seem your BIOS is the
culprit, it just can't set up the apic in a way usable for linux, while
the one on the DLS board can. You should compare BIOS versions/release
dates; maybe you can upgrade the BIOS. It might of course be that the BIOS
of the DLS board is generally better (since the DLS board has more
features)

If it's not a general hardware issue, the (no)apic issue is generally a
BIOS issue (since linux relies on a proper preconfigured apic/chipset).

Michael.

--

Michael Weller: eowmob@exp-math.uni-essen.de, eowmob@ms.exp-math.uni-essen.de,
or even mat42b@spi.power.uni-essen.de. If you encounter an eowmob account on
any machine in the net, it's very likely it's me.

Re: Why is "noapic" with some Dual P3 boards necessary?

[Bill Davidsen]

On Tue, 29 Oct 2002, Dr. Michael Weller wrote:

> On Wed, 30 Oct 2002, Matthias Kleine wrote:
> 
> > On Tuesday 29 October 2002 18:03, Vladimir G. Ivanovic wrote:
> [...]
> > > a year now with APIC. You must however fall back to MPS 1.1 from the
> > > default MPS 1.4. (It's a BIOS setting.)
> [...]
> > Without noapic as kernel option
> [...] do not work
> > - my NVIDIA driver does not work
> 
> Experience with other boards (some compaq) on my side and also what was
> written in this thread already shows, that BIOS settings modify how
> the apic is setup and hence if linux needs noapic.

What isn't clear is why anyone should make the big effort. I changed a
bunch of SMP machines, IBM Netfinitiy type, in hopes of a big improvement
in stability. I didn't see that, nor did I see any measurable change in
performance. If it works with noapic, why make a big effort to change?
 
> You should check if you can modify the bios setting as was described.
> 
> If not, or if a modification does not help, it seem your BIOS is the
> culprit, it just can't set up the apic in a way usable for linux, while
> the one on the DLS board can. You should compare BIOS versions/release
> dates; maybe you can upgrade the BIOS. It might of course be that the BIOS
> of the DLS board is generally better (since the DLS board has more
> features)
> 
> If it's not a general hardware issue, the (no)apic issue is generally a
> BIOS issue (since linux relies on a proper preconfigured apic/chipset).

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: Why is "noapic" with some Dual P3 boards necessary?

[Matthias Kleine]

On Wednesday 30 October 2002 20:20, Bill Davidsen wrote:

> What isn't clear is why anyone should make the big effort. I changed a
> bunch of SMP machines, IBM Netfinitiy type, in hopes of a big improvement
> in stability. I didn't see that, nor did I see any measurable change in
> performance. If it works with noapic, why make a big effort to change?

What I do not understand is that APIC seems to work with the
W* kind of systems. However, I am not able to say if this better tolerance
results in instability or other disadvantages. 

In theory, APIC should take some load from a CPU that has to
handle lots of interrupts (as far as I understand it). I.e. the conditions 
where APIC is a noticable advantage are probably very special.
-- 
   ______ ______
  /_   _//_   _/
Ma / /    / / hias
  /_/    /_/

Re: Why is "noapic" with some Dual P3 boards necessary?

[Alan Cox]

On Thu, 2002-10-31 at 22:20, Matthias Kleine wrote:
> What I do not understand is that APIC seems to work with the
> W* kind of systems. However, I am not able to say if this better tolerance
> results in instability or other disadvantages. 
> 
> In theory, APIC should take some load from a CPU that has to
> handle lots of interrupts (as far as I understand it). I.e. the conditions 
> where APIC is a noticable advantage are probably very special.

APIC helps some loads and hinders others, although its normally a
benefit. One thing that has been a problem in a few cases is that BIOS
vendors have several times shipped systems with bogus APIC MP1.4 tables
but correct ACPI tables. As we get better ACPI (eg in 2.5) hopefully we
will seen much compatibility with broken bioses and the like.

There are hardware APIC problems but those are not the majority of APIC
compatibility things we seem to see

Re: Why is "noapic" with some Dual P3 boards necessary?

[vladimir]

"BD" == Bill Davidsen <davidsen@tmr.com> writes:

  BD> What isn't clear is why anyone should make the big effort. I changed a
  BD> bunch of SMP machines, IBM Netfinitiy type, in hopes of a big improvement
  BD> in stability. I didn't see that, nor did I see any measurable change in
  BD> performance. If it works with noapic, why make a big effort to change?

I have been told that balancing the interrupts across CPUs makes a
difference on servers.

--- Vladimir

------------------------------------------------------------------------
Vladimir G. Ivanovic                        http://leonora.org/~vladimir
2770 Cowper St.                                         vladimir@acm.org
Palo Alto, CA 94306-2447                                 +1 650 678 8014

RE: Why is "noapic" with some Dual P3 boards necessary?

[Eff Norwood]

> I have been told that balancing the interrupts across CPUs makes a
> difference on servers.

And how is this done?

Thanks,

Eff Norwood

Re: Why is "noapic" with some Dual P3 boards necessary?

[Vladimir G. Ivanovic]

"EN" == Eff Norwood <enorwood@effrem.com> writes:

  >> I have been told that balancing the interrupts across CPUs makes a
  >> difference on servers.

  EN> And how is this done?

   Intel Architecture Software Developer s Manual Volume 3: System
   Programming (1999), p 7-13:

   The I/O APIC is responsible for receiving interrupts generated by I/O
   devices and distributing them among the local APICs by means of the APIC
   Bus. The I/O APIC manages interrupts using either static or dynamic
   distribution schemes. Dynamic distribution of interrupts allows routing
   of interrupts to the lowest priority processors. It also handles the
   distribution of interprocessor interrupts and system-wide control
   functions such as NMI, INIT, SMI and start-up-interprocessor interrupts.
   Individual pins on the I/O APIC can be programmed to generate a
   specific, prioritized interrupt vector when asserted. The I/O APIC also
   has a virtual wire mode that allows it to cooperate with an external
   8259A in the system.

See also the kernel docs

    .../Documentation/i386/IO-APIC.txt

and Randy Dunlap's write-up of how Linux differs from the MPS 1.4 spec

   http://www.xenotime.net/linux/mp-spec-1x.txt

--- Vladimir

------------------------------------------------------------------------
Vladimir G. Ivanovic                        http://leonora.org/~vladimir
2770 Cowper St.                                         vladimir@acm.org
Palo Alto, CA 94306-2447                                 +1 650 678 8014

Re: Why is "noapic" with some Dual P3 boards necessary?

[Bill Davidsen]

On Wed, 30 Oct 2002 vladimir@acm.org wrote:

> "BD" == Bill Davidsen <davidsen@tmr.com> writes:
> 
>   BD> What isn't clear is why anyone should make the big effort. I changed a
>   BD> bunch of SMP machines, IBM Netfinitiy type, in hopes of a big improvement
>   BD> in stability. I didn't see that, nor did I see any measurable change in
>   BD> performance. If it works with noapic, why make a big effort to change?
> 
> I have been told that balancing the interrupts across CPUs makes a
> difference on servers.

These are the news (reader) servers at a national ISP, in six states. 

vmstat2 says:
 load free buffs swap pgin pgou dk0 dk1 dk2 dk3 ipkt opkt  int  ctx   usr  sys idl  i_netK  o_netK
 0.61  5.0  1796  0.0  730 2076  29  55  50   0 1814 1155 3069 5876     2    6  92  1978.7   395.5
 0.49  5.0  1796  0.0 1225 2146  29 102 110   0 1969 1264 3433 6143     2    6  92  2179.5   445.0
 0.39  4.8  1795  0.0  789 2039  26  56  56   0 1842 1175 3127 5854     2    7  91  2071.0   436.3
 0.37  5.0  1796  0.0  742 2007  26  57  58   0 1899 1245 3252 6339     2    7  91  2074.8   453.5
 0.37  5.0  1795  0.0  699 1757  24  52  51   0 1672 1090 2861 4865     2    5  93  1804.8   409.8

I can measure no difference between noapic and apic except by looking at
/proc/interrupts. I'm not sure what "makes a difference" would means, in
any case.

Alan: you suggested this to me, any comments?

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.