Re: Possible PCI subsystem bug in 2.4

Re: Possible PCI subsystem bug in 2.4

[Linus Torvalds]

On Thu, 3 May 2001, Edward Spidre wrote:
> 
> Note: a diff between booting with mem and without it
> yield the same results (the user-defined phys ram map
> is identical to the bios provided one)

Interesting. Your BIOS-provided memory map is buggy:

> BIOS-provided physical RAM map:
>  BIOS-e820: 000000000009fc00 @ 0000000000000000 (usable)
>  BIOS-e820: 0000000000000400 @ 000000000009fc00 (reserved)
>  BIOS-e820: 000000000000c000 @ 00000000000c0000 (reserved)
>  BIOS-e820: 0000000013eec000 @ 0000000000100000 (usable)
>  BIOS-e820: 0000000000004000 @ 0000000013fec000 (reserved)
>  BIOS-e820: 0000000000200000 @ 00000000ffe00000 (reserved)

Note how it says that you have usable RAM from

	0000000000100000 - 0000000013fec000

(the thing is hard to read and the output was changed in later kernels: it
really says that you have 0000000013eec000 bytes of ram starting at
0000000000100000, which obviously doing the math means that it goes up to
0000000013fec000).

Now, it then says that you have reserved memory (ie probably the BIOS has
reserved 1kB at high memory) from

	0000000013fec000 - 0000000013ff0000

In particular, notice how it does NOT mention the memory region from

	0000000013ff0000 - 0000000014000000

at ALL. Which means that Linux thinks that it is free... And Linux will
place PCI devices there. Even though there certainly is memory there.

I'll have to work around the BIOS bug some way. Will you be willing to
try out patches?

		Linus

Re: Possible PCI subsystem bug in 2.4

[Alan Cox]

> at ALL. Which means that Linux thinks that it is free... And Linux will
> place PCI devices there. Even though there certainly is memory there.

Ahah.. that explains the Dell inspiron 8000 cardbus + 256Mb bug as well.

> I'll have to work around the BIOS bug some way. Will you be willing to
> try out patches?

Obvious one is to go to the next power of two clear. 

Semi related question: To do I2O properly I need to grab PCI bus space and
'loan' it to the controller when I configure it. Im wondering what the
preferred approach there is.

Alan

Re: Possible PCI subsystem bug in 2.4

[Linus Torvalds]

On Thu, 3 May 2001, Alan Cox wrote:
> 
> Obvious one is to go to the next power of two clear. 

The question is mainly _which_ power of two. 

I don't think we can round up infinitely, as that might just end up
causing us to not have any PCI space at all. Or we could end up deciding
that real PCI space is memory, and then getting a clash when a real device
tries to register its bios-allocated area that clashes with our extreme
rounding.

I suspect it would be safe to round up to the next megabyte, possibly up
to 64MB or so. But much more would make me nervous.

Any suggestions? 

> Semi related question: To do I2O properly I need to grab PCI bus space and
> 'loan' it to the controller when I configure it. Im wondering what the
> preferred approach there is.

Do the same thing that the yenta driver does, just do a 

	root = pci_find_parent_resource(dev, res);
	allocate_resource(root, res, size, min, max, align, NULL, NULL);

and keep it allocated (and then the i2o driver can do sub-allocations
within that resource by doing "allocate_resource(res, ...)").

		Linus

Re: Possible PCI subsystem bug in 2.4

[Jeff Garzik]

Linus Torvalds wrote:
> The question is mainly _which_ power of two.
> 
> I don't think we can round up infinitely, as that might just end up
> causing us to not have any PCI space at all. Or we could end up deciding
> that real PCI space is memory, and then getting a clash when a real device
> tries to register its bios-allocated area that clashes with our extreme
> rounding.
> 
> I suspect it would be safe to round up to the next megabyte, possibly up
> to 64MB or so. But much more would make me nervous.
> 
> Any suggestions?

Is there any chance you could simply test the bottom of PCI address
space?  If you could set up the x86 to trap non-DRAM read/writes
temporarily, you could tell where useable DRAM area stops.

-- 
Jeff Garzik      | Game called on account of naked chick
Building 1024    |
MandrakeSoft     |

Re: Possible PCI subsystem bug in 2.4

[Alan Cox]

> I suspect it would be safe to round up to the next megabyte, possibly up
> to 64MB or so. But much more would make me nervous.
> Any suggestions? 

I'd go for 1MByte simply because I've not seen an EBDA/NVRAM area that large
stuck at the top of RAM. 1Mb would fix the Dell. (It was only when I saw
your email it suddenely clicked and I grabbed the bootup log)

> > Semi related question: To do I2O properly I need to grab PCI bus space and
> > 'loan' it to the controller when I configure it. Im wondering what the
> > preferred approach there is.
> 
> Do the same thing that the yenta driver does, just do a 
> 
> 	root = pci_find_parent_resource(dev, res);
> 	allocate_resource(root, res, size, min, max, align, NULL, NULL);
> 
> and keep it allocated (and then the i2o driver can do sub-allocations
> within that resource by doing "allocate_resource(res, ...)").

Thanks.

Alan

Re: Possible PCI subsystem bug in 2.4

[Linus Torvalds]

On Thu, 3 May 2001, Edward Spidre wrote:
>
> Sure, I'm more than willing to test out patches. Just
> let me know which version of source to apply it
> against.

Ok, this is not a patch per se, just a description of what I think should
work..

In include/asm-i386/pci.h, change the line

	#define PCIBIOS_MIN_MEM		0x10000000

into

	extern unsigned long pci_mem_start;
	#define PCIBIOS_MIN_MEM		(pci_mem_start)

In arch/i386/kernel/setup.c, at the very end of the
"setup_arch()" function, add

	/* Start PCI allocations at max_low_memory rounded up to 1MB */
	pci_mem_start = ((max_low_pfn << PAGE_SHIFT) + 0xfffff) & ~0xfffff;

and somewhere in the same file, add the actual variable:

	unsigned long pci_mem_start = 0x10000000;

and you're done. Does this work for you?

Alan, do you have access to that Dell laptop to test? The thing looks
obvious, but I'd rather not apply it to my tree until somebody sends me
the above back as a tested patch.. Call me a sissy.

		Linus

Re: Possible PCI subsystem bug in 2.4

[Rogier Wolff]

Linus Torvalds wrote:
> 
> On Thu, 3 May 2001, Alan Cox wrote:
> > 
> > Obvious one is to go to the next power of two clear. 
> 
> The question is mainly _which_ power of two. 
> 
> I don't think we can round up infinitely, as that might just end up
> causing us to not have any PCI space at all. Or we could end up deciding
> that real PCI space is memory, and then getting a clash when a real device
> tries to register its bios-allocated area that clashes with our extreme
> rounding.
> 
> I suspect it would be safe to round up to the next megabyte, possibly up
> to 64MB or so. But much more would make me nervous.
> 
> Any suggestions? 

The amount of RAM in a machine almost always has just one or two "1"
bits. 

8, 16, 20, 24, 32, 36, 40, 48, 64, 80 Mb were the numbers that you'd
see in the early Pentium times, right?

So rounding up would mean: Add one until the number of 1-bits in the
address is less than 3. People with 3 or more differently sized DIMMS
in their machine will experience a slightly ineffcient round-up. 

Speed this up with: Find-lowest-1-bit, add that. 

Example you quoted:
                                                nr of 1 bits. 
BIOS-proclaimed end-of-ram: 	0x13fff0000         15
lowest 1-bit: 			0x000010000          1
add:                            0x140000000          2


long round_highmem (long val)
{
	while (hweight32 (val) > 2)
		val += 1 << ffs (val); 
	return val;
}

		Roger. 


-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* There are old pilots, and there are bold pilots. 
* There are also old, bald pilots. 

Re: Possible PCI subsystem bug in 2.4

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > I suspect it would be safe to round up to the next megabyte, possibly up
> > to 64MB or so. But much more would make me nervous.
> > Any suggestions? 
> 
> I'd go for 1MByte simply because I've not seen an EBDA/NVRAM area that large
> stuck at the top of RAM. 1Mb would fix the Dell. (It was only when I saw
> your email it suddenely clicked and I grabbed the bootup log)

There are a couple of options here.
1) read the MTRRs unless the BIOS is braindead it will set up that area as
   write-back.  At any rate we shouldn't ever try to allocate a pci region
   that is write-back cached.

2) read the memory locations from the northbridge.  It's not possible
   on every chipset (lack of documentation) but with the linuxBIOS
   project we code for a couple of them, and we are working on more
   all of the time.

Eric

Re: Possible PCI subsystem bug in 2.4

[Alan Cox]

> There are a couple of options here.
> 1) read the MTRRs unless the BIOS is braindead it will set up that area as
>    write-back.  At any rate we shouldn't ever try to allocate a pci region
>    that is write-back cached.

'unless the BIOS is braindead'. Right. We only got into this problem because
the BIOS _was_ braindead.

Alan

Re: Possible PCI subsystem bug in 2.4

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > There are a couple of options here.
> > 1) read the MTRRs unless the BIOS is braindead it will set up that area as
> >    write-back.  At any rate we shouldn't ever try to allocate a pci region
> >    that is write-back cached.
> 
> 'unless the BIOS is braindead'. Right. We only got into this problem because
> the BIOS _was_ braindead.

Well I did provide a suggestion so you don't have to second guess...
Usually it's actually easier to read the memory size from the northbridge
than to parse the E820 map.

However since it is different kinds of braindamage to mess up the MTRRs,
and the E820 memory map, it is worth a shot.  Personally I think MTRRs
are much easier to get right, because you don't need to take into
account what the BIOS is going to do just where your ram is.

As for braindead BIOS's in general any comments on totally nuking
them?  

Seriously.  With the general attitude of distrusting BIOS's I have
been amazed at the number of things linux expects the BIOS to get
right.  In practice windows seem to trust the BIOS much less than
linux does.

Eric

Re: Possible PCI subsystem bug in 2.4

[Alan Cox]

> Seriously.  With the general attitude of distrusting BIOS's I have
> been amazed at the number of things linux expects the BIOS to get
> right.  In practice windows seem to trust the BIOS much less than
> linux does.

It becomes more and more obvious over time exactly why. One problem however
is that windows gets away with this because many vendors ship random extra
gunge for their box with the system. We dont yet have that power

Alan

Re: Possible PCI subsystem bug in 2.4

[Linus Torvalds]

On 4 May 2001, Eric W. Biederman wrote:
> 
> There are a couple of options here.
> 1) read the MTRRs unless the BIOS is braindead it will set up that area as
>    write-back.  At any rate we shouldn't ever try to allocate a pci region
>    that is write-back cached.

This one I'd really hesitate to use. We do _not_ want to trust the BIOS
any more than necessary (obviously trusting even the e820 was too much),
and especially wrt MTRR's we know that there are too many buggy bioses
already out there.

> 2) read the memory locations from the northbridge.  It's not possible
>    on every chipset (lack of documentation) but with the linuxBIOS
>    project we code for a couple of them, and we are working on more
>    all of the time.

This will be easy.

In fact, we can easily "mix" different heuristics. Ie we'd do the simple
thing I suggested in setup_arch(), and use that as a "base guess", and
then we can have incremental improvements on that guess that might be
chipset-specific or might depend on other information that is not
necessarily generic (things like existing PCI programming etc).

		Linus

Re: Possible PCI subsystem bug in 2.4

[Eric W. Biederman]

Linus Torvalds <torvalds@transmeta.com> writes:

> On 4 May 2001, Eric W. Biederman wrote:
> > 
> > There are a couple of options here.
> > 1) read the MTRRs unless the BIOS is braindead it will set up that area as
> >    write-back.  At any rate we shouldn't ever try to allocate a pci region
> >    that is write-back cached.
> 
> This one I'd really hesitate to use. We do _not_ want to trust the BIOS
> any more than necessary (obviously trusting even the e820 was too much),
> and especially wrt MTRR's we know that there are too many buggy bioses
> already out there.

O.k.  I was simply thinking that if we weren't reprogramming the mtrrs, it
would be a good idea to make certain we didn't map any PCI drivers
into a write back area. 
 
> > 2) read the memory locations from the northbridge.  It's not possible
> >    on every chipset (lack of documentation) but with the linuxBIOS
> >    project we code for a couple of them, and we are working on more
> >    all of the time.
> 
> This will be easy.

O.k.  If it easy.  

Currently I can find information for some via chipsets.  One
serverworks chipset, practically every intel chipset, the AMD 750
& 760 chipsets, current SIS chipsets, and the alpha tsunami chipset.
There is probably more that is just what I have actually seen.

There are complications of memory reserved for AGP, and memory
holes.  But they shouldn't be too bad.

My only problem is I don't have the bandwidth to code it up at the
moment, but if it's the kind of patch that if done well you are
likely to accept I'll look at it.  

Since it is only likely to be 30-40 lines per chipset it should
only take an hour or two per chipset.  And after the first couple are
in the kernel others can clone the work. 

This has the nice side effect that once we know where the ram is on a
chipset we can reprogram the MTRRs correctly ourselves, correcting bad
BIOS's.

> In fact, we can easily "mix" different heuristics. Ie we'd do the simple
> thing I suggested in setup_arch(), and use that as a "base guess", and
> then we can have incremental improvements on that guess that might be
> chipset-specific or might depend on other information that is not
> necessarily generic (things like existing PCI programming etc).

That sounds reasonable.  Since we don't always have all the
documentation we need heuristics so we can run on boards where we are
just guessing.

On the not trusting the BIOS, would you accept patches to do more
of the lowlevel setup?  

There is a pnp-os spec that basically says you should only need to
initialize devices necessary for booting.  With linuxBIOS we are
working to take this to the extreme, and define devices necessary for
booting as:  RAM, the bootstrap CPU, the Flash ROM chip, and possibly
a serial port for debugging.  Currently I need to a little more setup
than this because of limitations in linux. 

The more of the lowlevel set we can get into linux, the less you need
to trust the BIOS and the less I need to code to get linuxBIOS up on
an individual motherboard.

Eric

Re: Possible PCI subsystem bug in 2.4

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > Seriously.  With the general attitude of distrusting BIOS's I have
> > been amazed at the number of things linux expects the BIOS to get
> > right.  In practice windows seem to trust the BIOS much less than
> > linux does.
> 
> It becomes more and more obvious over time exactly why. One problem however
> is that windows gets away with this because many vendors ship random extra
> gunge for their box with the system. We dont yet have that power

Right.  So we always need to keep heuristics in our toolbox to fallback on,
so we can run on boards with incomplete information.  However there is a lot
of things we can do that we aren't currently doing.

The example that sticks out in my head is we rely on the MP table to
tell us if the local apic is in pic_mode or in virtual wire mode.
When all we really have to do is ask it.

Eric

Re: Possible PCI subsystem bug in 2.4

[Alan Cox]

> O.k.  I was simply thinking that if we weren't reprogramming the mtrrs, it
> would be a good idea to make certain we didn't map any PCI drivers
> into a write back area. 

What if the BIOS set up an mtrr for a device ?

Re: Possible PCI subsystem bug in 2.4

[Maciej W. Rozycki]

On 4 May 2001, Eric W. Biederman wrote:

> The example that sticks out in my head is we rely on the MP table to
> tell us if the local apic is in pic_mode or in virtual wire mode.
> When all we really have to do is ask it.

 You can't.  IMCR is write-only and may involve chipset-specific
side-effects.  Then even if IMCR exists, a system's firmware might have
chosen the virtual wire mode for whatever reason (e.g. broken hardware). 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Possible PCI subsystem bug in 2.4

[Eric W. Biederman]

"Maciej W. Rozycki" <macro@ds2.pg.gda.pl> writes:

> On 4 May 2001, Eric W. Biederman wrote:
> 
> > The example that sticks out in my head is we rely on the MP table to
> > tell us if the local apic is in pic_mode or in virtual wire mode.
> > When all we really have to do is ask it.
> 
>  You can't.  IMCR is write-only and may involve chipset-specific
> side-effects.  Then even if IMCR exists, a system's firmware might have
> chosen the virtual wire mode for whatever reason (e.g. broken hardware). 

Admittedly you can't detect directly detect IMCR state.  But
triggering an interrupt on the bootstrap processor local apic, and
failing to receive it should be proof the IMCR is at work.
Alternatively if I'm wrong about the wiring disabling all interrupts
at the apic level and receiving one is a second proof that IMCR is at
work.  Further I don't think a processor with an onboard apic, works
with an IMCR register. 

What I was thinking of earlier is that you can detect an apic or
ioapic in virtual wire mode, which the current code and the intel MP
spec treats as the opposite possibility.

Eric