Request: increase in PCI bus limit

Request: increase in PCI bus limit

[Christopher Neufeld]

   Hello,

   I'm working at a customer site with custom hardware.  The 2.4.0 series
kernel almost works out of the box, but the machine has 52 PCI busses.
Plans are to produce a 4-way box which would have over 80 PCI busses.  The
file include/asm-i386/mpspec.h allocates space for 32 busses in the
definition of the macro MAX_MP_BUSSES.  When 52 busses are probed, some
arrays are filled out past their ends (there is no bounds checking
performed on the array filling), and the kernel oopses out.  The only patch
which has to be applied to make Linux run stably on these systems is to
increase that limit.  Would it be possible to bump it up to 128, or even
256, in later 2.4.* kernel releases?  That would allow this customer to
work with an unpatched kernel, at the cost of an additional 3.5 kB of
variables in the kernel.

   Thank you for any help.
   For completeness, here's the patch (against 2.4.0):

--- linux-2.4.0/include/asm-i386/mpspec.h.orig  Tue Jan 30 16:06:08 2001
+++ linux-2.4.0/include/asm-i386/mpspec.h       Tue Jan 30 16:06:21 2001
@@ -157,7 +157,7 @@
  */
 
 #define MAX_IRQ_SOURCES 128
-#define MAX_MP_BUSSES 32
+#define MAX_MP_BUSSES 256
 enum mp_bustype {
        MP_BUS_ISA = 1,
        MP_BUS_EISA,



-- 
 Christopher Neufeld		 		 neufeld@linuxcare.com
 Home page:  http://caliban.physics.utoronto.ca/neufeld/Intro.html
 "Don't edit reality for the sake of simplicity"
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Timur Tabi]

** Reply to message from Christopher Neufeld <neufeld@linuxcare.com> on Tue, 30
Jan 2001 16:08:32 -0800


> Would it be possible to bump it up to 128, or even
> 256, in later 2.4.* kernel releases?  That would allow this customer to
> work with an unpatched kernel, at the cost of an additional 3.5 kB of
> variables in the kernel.

I don't think that's going to happen.  If we did this for your obscure system,
then we'd have to do it for every obscure system, and before you know it, the
kernel is 200KB larger.

Besides, why is your client afraid of patched kernels?  It sounds like a very
odd request from someone with a linuxcare.com email address.  I would think that
you'd WANT to provide patched kernels so that the customer can keep paying you
(until they learn how to use a text editor, at which point they can patch the
kernel themselves!!!)


-- 
Timur Tabi - ttabi@interactivesi.com
Interactive Silicon - http://www.interactivesi.com

When replying to a mailing-list message, please direct the reply to the mailing list only.  Don't send another copy to me.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Udo A. Steinberg]

Christopher Neufeld wrote:

>  The only patch
> which has to be applied to make Linux run stably on these systems is to
> increase that limit.  Would it be possible to bump it up to 128, or even
> 256, in later 2.4.* kernel releases?  That would allow this customer to
> work with an unpatched kernel, at the cost of an additional 3.5 kB of
> variables in the kernel.

I guess the cleanest solution would be to allow variable setting of the
maximum number of PCI busses in the config file, similar to the
CONFIG_UNIX98_PTY_COUNT setting, so that "exotic" users with 32+ PCI
busses can boost the standard value according to their needs, without
having to increase kernel size for the normal users.

Regards,
Udo.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Bernd Eckenfels]

In article <200101310008.f0V08Wv23250@localhost.localdomain> you wrote:
> 256, in later 2.4.* kernel releases?  That would allow this customer to
> work with an unpatched kernel, at the cost of an additional 3.5 kB of
> variables in the kernel.

Don't think this is fairly common. So especially since I consider that kind of
hardware (what is it) to require finetuning ("enterprise kernel") anyway,
there is no real gain out of it, as long as the structure is not dynamically.

Is that some kind of file server or masspar system? Intel?

Greetings
Bernd
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Bernd Eckenfels]

In article <3A775FEB.D7614D98@Hell.WH8.TU-Dresden.De> you wrote:
> I guess the cleanest solution would be to allow variable setting of the
> maximum number of PCI busses in the config file, similar to the
> CONFIG_UNIX98_PTY_COUNT setting, so that "exotic" users with 32+ PCI
> busses can boost the standard value according to their needs, without
> having to increase kernel size for the normal users.

May even decrease the kernel for systems < 4 busses.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[List User]

He's probably sending it to the list the same reason why most in tech
circles do.
To cut down on the amount of work required.  Personally I would love to have
all my custom changes put into a 'standard' distribution that way I wouldn't
need
to keep as many custom notes for x,y,z platforms or what-else.  Also for the
fact that it gives something back (generally if I run into a situation there
is a good
chance that someone else will run into it as well, so if I can make _his_
life
easier if he doesn't have the same knowledge it's worth it).

Steve
----- Original Message -----
From: "Timur Tabi" <ttabi@interactivesi.com>
To: <linux-kernel@vger.kernel.org>
Sent: Tuesday, January 30, 2001 18:36
Subject: Re: Request: increase in PCI bus limit


> ** Reply to message from Christopher Neufeld <neufeld@linuxcare.com> on
Tue, 30
> Jan 2001 16:08:32 -0800
>
>
> > Would it be possible to bump it up to 128, or even
> > 256, in later 2.4.* kernel releases?  That would allow this customer to
> > work with an unpatched kernel, at the cost of an additional 3.5 kB of
> > variables in the kernel.
>
> I don't think that's going to happen.  If we did this for your obscure
system,
> then we'd have to do it for every obscure system, and before you know it,
the
> kernel is 200KB larger.
>
> Besides, why is your client afraid of patched kernels?  It sounds like a
very
> odd request from someone with a linuxcare.com email address.  I would
think that
> you'd WANT to provide patched kernels so that the customer can keep paying
you
> (until they learn how to use a text editor, at which point they can patch
the
> kernel themselves!!!)
>
>
> --
> Timur Tabi - ttabi@interactivesi.com
> Interactive Silicon - http://www.interactivesi.com
>
> When replying to a mailing-list message, please direct the reply to the
mailing list only.  Don't send another copy to me.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Peter Samuelson]

[Bernd Eckenfels]
> May even decrease the kernel for systems < 4 busses.

Be careful, though.  Users may set this thinking "I have a generic
system with only one PCI bus" without realizing that AGP, cardbus and
some motherboard devices are all counted.  Pad the CONFIG option by
about 4 busses and we'll be OK..

Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Helge Hafting]

"Udo A. Steinberg" wrote:
> 
> Christopher Neufeld wrote:
> 
> >  The only patch
> > which has to be applied to make Linux run stably on these systems is to
> > increase that limit.  Would it be possible to bump it up to 128, or even
> > 256, in later 2.4.* kernel releases?  That would allow this customer to
> > work with an unpatched kernel, at the cost of an additional 3.5 kB of
> > variables in the kernel.
> 
> I guess the cleanest solution would be to allow variable setting of the
> maximum number of PCI busses in the config file, similar to the
> CONFIG_UNIX98_PTY_COUNT setting, so that "exotic" users with 32+ PCI
> busses can boost the standard value according to their needs, without
> having to increase kernel size for the normal users.
> 
Wouldn't autodetection be the better solution?  Seems to me the kernel
has
no problem detecting all the buses, so why not break the probing into a
two-
pass thing - first pass simply counts buses, second pass allocate memory
and fills in the arrays.  
(One-pass is possible too, if dynamic allocation is possible at this
time.)
No config option to worry about, no excess room for 32 buses on ordinary
pc's, no worry for the 100-bus user.


Helge Hafting
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[James Sutherland]

On Tue, 30 Jan 2001, Timur Tabi wrote:

> ** Reply to message from Christopher Neufeld <neufeld@linuxcare.com> on Tue, 30
> Jan 2001 16:08:32 -0800
> 
> 
> > Would it be possible to bump it up to 128, or even
> > 256, in later 2.4.* kernel releases?  That would allow this customer to
> > work with an unpatched kernel, at the cost of an additional 3.5 kB of
> > variables in the kernel.
> 
> I don't think that's going to happen.  If we did this for your obscure system,
> then we'd have to do it for every obscure system, and before you know it, the
> kernel is 200KB larger.
> 
> Besides, why is your client afraid of patched kernels?  It sounds like a very
> odd request from someone with a linuxcare.com email address.  I would think that
> you'd WANT to provide patched kernels so that the customer can keep paying you
> (until they learn how to use a text editor, at which point they can patch the
> kernel themselves!!!)

Should there not at least be some bounds checking on this table, though?!

If it's only built at boot time, it's not performance critical. Maybe at a
later date it could even expand (or shrink, on small PCs??) the table as
needed?


James.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Manfred Spraul]

> 
>    I'm working at a customer site with custom hardware. The 2.4.0 series 
> kernel almost works out of the box, but the machine has 52 PCI busses. 
> Plans are to produce a 4-way box which would have over 80 PCI busses. The 
> file include/asm-i386/mpspec.h allocates space for 32 busses in the 
> definition of the macro MAX_MP_BUSSES. 
>

How long is the MP structure?
smp_scan_config() reserves only 4 kB:

	reserve_bootmem(mpf->mpf_physptr, PAGE_SIZE);

reserving the actual size (mpf->mpf_physptr->mpc_length) could be
tricky.

It should be possible to dynamically allocate the memory for the busses:
It's not yet possible (smp_read_mpc() is called at a very early stage,
before kmalloc is initialized), but we must move it to a later stage
anyway:
some Compaq bios version need ioremap() in smp_read_mpc(), and we should
parse the ACPI tables for APIC descriptors (MADT, ia64 does that
already).

I'll add it to my TODO list.

--
	Manfred
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Christopher Neufeld]

From: Timur Tabi <ttabi@interactivesi.com>
>
>Besides, why is your client afraid of patched kernels?  It sounds like a very
>odd request from someone with a linuxcare.com email address.  I would think that
>you'd WANT to provide patched kernels so that the customer can keep paying you
>(until they learn how to use a text editor, at which point they can patch the
>kernel themselves!!!)
>
   Well, to be honest, this kernel patch was developed by the client long
before I showed up, I only heard about it yesterday evening when they asked
me to pass it on to linux-kernel. I see your point, but I think it would be
a very strange and magical world where customers line up to pay a company
to apply one-line patches to a header file and type "make" every time a new
kernel comes along.


   There are a few reasons that it would be nice to have such changes in a
standard kernel:

>From the company's point of view:  ease of support.  If the box runs on a
standard Linux kernel it makes it much easier on people who purchase the
boxes.  The company doesn't have to provide patched kernels whenever a new
kernel release comes out, and purchasers can be confident that they won't
find themselves trapped if the company becomes unable or unwilling to
continue to provide those patches.

>From the Linux community's point of view:  flexibility.  Maybe this is the
first computer to pass the 32 PCI bus limit, maybe not.  I really doubt
it's the last, though.  At some point, this will rear up again, and a
kernel which handles the condition gracefully would probably be appreciated
by the people working on this hypothetical future system.

>From the company's point of view:  verification.  I've been told in email
that the correct solution is more subtle than simply increasing this one
value.  The fact that this patch works properly in house is probably
somewhat fortuitous.  Remember the fun when people started increasing the
open file descriptor limits a couple of years ago, and almost getting it
right?  By bringing this patch to the attention of linux-kernel, we point
out a possible need for this functionality to the people who developed the
code in question, and who might be able to fix some subtler gotchas which
were missed when the change was developed.


-- 
 Christopher Neufeld		 		 neufeld@linuxcare.com
 Home page:  http://caliban.physics.utoronto.ca/neufeld/Intro.html
 "Don't edit reality for the sake of simplicity"
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[George]

On Wed, 31 Jan 2001, Peter Samuelson wrote:

>[Bernd Eckenfels]
>> May even decrease the kernel for systems < 4 busses.
>
>Be careful, though.  Users may set this thinking "I have a generic
>system with only one PCI bus" without realizing that AGP, cardbus and
>some motherboard devices are all counted.  Pad the CONFIG option by
>about 4 busses and we'll be OK..

If someone says 1 bus, give them one bus.

Just make the description say:
  Add 1 for every PCI
  Add 1 for every AGP
  Add 1 for every CardBus
  Also account for anything else funny in the system.

Then panic on boot if they're wrong (sort of like processor type).


It's somewhat annoying that by choosing SMP NR_CPUS goes to 32 when I know
I only have (and ever will have) 2 in this machine.  Don't make busses have
the same assumptions that just waste memory.

-George Greer

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Scott Laird]

On Wed, 31 Jan 2001, George wrote:
>
> If someone says 1 bus, give them one bus.
>
> Just make the description say:
>   Add 1 for every PCI
>   Add 1 for every AGP
>   Add 1 for every CardBus
>   Also account for anything else funny in the system.
>
> Then panic on boot if they're wrong (sort of like processor type).

Where do cards with PCI-PCI bridges, like multiport PCI ethernet cards,
fit into this?  I can easily add 3 or 4 extra busses into a box just by
grabbing a couple extra Intel dual-port Ethernet cards.


Scott

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[nick]

depends on the network card.  The adaptecs have one.  the Obsidion X16 and
X24 have them.  Many dual channel SCSI cards use them, and even some nicer
motherboards have them onboard.  I would like to recommend autodetection
at boot, but I'm assuming that someone has looked into it and determined
it impractical?
	Nick

On Wed, 31 Jan 2001, Scott Laird wrote:

> 
> 
> On Wed, 31 Jan 2001, George wrote:
> >
> > If someone says 1 bus, give them one bus.
> >
> > Just make the description say:
> >   Add 1 for every PCI
> >   Add 1 for every AGP
> >   Add 1 for every CardBus
> >   Also account for anything else funny in the system.
> >
> > Then panic on boot if they're wrong (sort of like processor type).
> 
> Where do cards with PCI-PCI bridges, like multiport PCI ethernet cards,
> fit into this?  I can easily add 3 or 4 extra busses into a box just by
> grabbing a couple extra Intel dual-port Ethernet cards.
> 
> 
> Scott
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
> 

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Dan Hollis]

On Wed, 31 Jan 2001, Scott Laird wrote:
> Where do cards with PCI-PCI bridges, like multiport PCI ethernet cards,
> fit into this?  I can easily add 3 or 4 extra busses into a box just by
> grabbing a couple extra Intel dual-port Ethernet cards.

I loaded a PC with quad-tulip cards once and the result was not pretty.
IRQs all over the place, and /proc/pci shat all over itself too

-Dan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Dr. Kelsey Hudson]

On Wed, 31 Jan 2001, Scott Laird wrote:
> On Wed, 31 Jan 2001, George wrote:
> >   Also account for anything else funny in the system.
> >
> > Then panic on boot if they're wrong (sort of like processor type).
> 
> Where do cards with PCI-PCI bridges, like multiport PCI ethernet cards,
> fit into this?  I can easily add 3 or 4 extra busses into a box just by
> grabbing a couple extra Intel dual-port Ethernet cards.

Notice the 'also account for anything funny' line. Multiport ethernet
cards IMO fall under this category.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Request: increase in PCI bus limit

[Ralf Baechle]

On Wed, Jan 31, 2001 at 03:38:55PM -0500, George wrote:

> It's somewhat annoying that by choosing SMP NR_CPUS goes to 32 when I know
> I only have (and ever will have) 2 in this machine.  Don't make busses have
> the same assumptions that just waste memory.

It's just sad that bumping to NR_CPUS to 128 won't work that easily ;-)

  Ralf  (fixing that so he gets some machine time again ...)