no more MTRRs available ?

no more MTRRs available ?

[Stephan von Krawczynski]

Hello all,

what exactly does

	mtrr: no more MTRRs available
	mtrr: no more MTRRs available

during boot mean? What can I do against this? This comes up while booting a
system with 6GB and P-III 1.4 GHz (Serverworks chipset). Kernel is 2.4.20.

Thanks for any hints.

-- 
Regards,
Stephan

Re: no more MTRRs available ?

[Stephan von Krawczynski]

On Wed, 29 Jan 2003 10:25:57 -0500 (EST)
Mark Hahn <hahn@physics.mcmaster.ca> wrote:

> > what exactly does
> > 
> > 	mtrr: no more MTRRs available
> > 	mtrr: no more MTRRs available
> > 
> > during boot mean? What can I do against this? This comes up while booting a
> > system with 6GB and P-III 1.4 GHz (Serverworks chipset). Kernel is 2.4.20.
> 
> you need to look at /proc/mtrr.

Thanks for your hint, but what does this tell me?

# cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
reg05: base=0xf7000000 (3952MB), size=  16MB: uncachable, count=1
reg06: base=0x100000000 (4096MB), size=4096MB: write-back, count=1
reg07: base=0x200000000 (8192MB), size=8192MB: write-back, count=1
 
-- 
Regards,
Stephan

Re: no more MTRRs available ?

[Stephan von Krawczynski]

On Wed, 29 Jan 2003 10:51:03 -0500 (EST)
Mark Hahn <hahn@physics.mcmaster.ca> wrote:

> > > > what exactly does
> > > > 
> > > > 	mtrr: no more MTRRs available
> > > > 	mtrr: no more MTRRs available
> > > > 
> > > > during boot mean? What can I do against this? This comes up while
> > > > booting a system with 6GB and P-III 1.4 GHz (Serverworks chipset).
> > > > Kernel is 2.4.20.
> > > 
> > > you need to look at /proc/mtrr.
> > 
> > Thanks for your hint, but what does this tell me?
> 
> that your bios is stupid, I think.  mtrr's handle areas that are 
> powers of two in size (and >= 1M, I think).  the problem here is that
> the bios is trying to represent 4G of write-back ram and a 16M of 
> uncachable IO area (AGP aperture, I'm guessing).  the correct way
> to do this is a single 4G mtrr with an overlapping 16M one.

Ah, I see. So getting the above two messages would simply mean that there is no
table space left to add yet another two entries kernel gets from bios?

> do you have >4G ram?  that would explain the latter two.

Yes, this box has 6GB.

> note that you can fairly freely add/delete mtrr's from userspace.
> as long as you do it infrequently, I don't see why there would be 
> any risk or performance problem.

So the conclusion is that it does not look nice currently, but does not do any
harm (performance loss) either. I can live with that.
Thanks for your comments.

> > # cat /proc/mtrr
> > reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> > reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> > reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
> > reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
> > reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
> > reg05: base=0xf7000000 (3952MB), size=  16MB: uncachable, count=1
> > reg06: base=0x100000000 (4096MB), size=4096MB: write-back, count=1
> > reg07: base=0x200000000 (8192MB), size=8192MB: write-back, count=1
-- 
Regards,
Stephan

Re: no more MTRRs available ?

[Dave Jones]

On Wed, Jan 29, 2003 at 05:05:54PM +0100, Stephan von Krawczynski wrote:

 > Mark Hahn <hahn@physics.mcmaster.ca> wrote:

(odd, I'm not getting Mark's mails to l-k).
 
 > > that your bios is stupid, I think.  mtrr's handle areas that are 
 > > powers of two in size (and >= 1M, I think).  the problem here is that
 > > the bios is trying to represent 4G of write-back ram and a 16M of 
 > > uncachable IO area (AGP aperture, I'm guessing).  the correct way
 > > to do this is a single 4G mtrr with an overlapping 16M one.

That does seem to make more sense. Perhaps too much sense.
ISTR there were ordering rules in how you layer MTRRs on top of each
other.
 
		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: no more MTRRs available ?

[Stephan von Krawczynski]

On Wed, 29 Jan 2003 16:20:13 +0000
Dave Jones <davej@codemonkey.org.uk> wrote:

> On Wed, Jan 29, 2003 at 05:05:54PM +0100, Stephan von Krawczynski wrote:
> 
>  > Mark Hahn <hahn@physics.mcmaster.ca> wrote:
> 
> (odd, I'm not getting Mark's mails to l-k).

He does not send them there. I know it is a no-no to forward his writings to
the list, but I thought it may be of use to others, too (and he did not shoot
me for it so far ;-)

>  > > that your bios is stupid, I think.  mtrr's handle areas that are 
>  > > powers of two in size (and >= 1M, I think).  the problem here is that
>  > > the bios is trying to represent 4G of write-back ram and a 16M of 
>  > > uncachable IO area (AGP aperture, I'm guessing).  the correct way
>  > > to do this is a single 4G mtrr with an overlapping 16M one.
> 
> That does seem to make more sense. Perhaps too much sense.
> ISTR there were ordering rules in how you layer MTRRs on top of each
> other.

Ok, after reading your thoughts and re-reading the bios stuff I found and "DRAM
alignment" option, and voila:

reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=4096MB: write-back, count=1
reg02: base=0x200000000 (8192MB), size=8192MB: write-back, count=1
reg03: base=0x400000000 (16384MB), size=16384MB: write-back, count=1
reg04: base=0x800000000 (32768MB), size=32768MB: write-back, count=1
reg05: base=0xfb000000 (4016MB), size=   8MB: write-combining, count=1

The table got obviously shorter and significantly "better". It seems the
onboard VGA stuff is now mapped out of the normal DRAM range. 

I must admit though that some of this bios looks fishy, because now I cannot
find the original DRAM caching entries any longer, no matter what I switch on
or off (you know those "write-back, write-through" stuff).

Now I think one can really be content with this setup. Thanks a lot to Mark and
Dave.
In case anyone is interested, this is an Asus TRL-DLS mb.

-- 
Regards,
Stephan

Re: no more MTRRs available ?

[Dave Jones]

On Wed, Jan 29, 2003 at 04:45:52PM +0100, Stephan von Krawczynski wrote:
 
 > # cat /proc/mtrr
 > reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
 > reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
 > reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
 > reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
 > reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
 > reg05: base=0xf7000000 (3952MB), size=  16MB: uncachable, count=1

Due to this 16MB hole, your BIOS has to set up a write-back range
covering 2048-16 (2032MB). Due to limitations on the way MTRRs work,
you can't do this all in 1, so you have to split it up over several.
The result is that you use up 5 MTRRs covering 1 range + 1 for the
unmapped 16MB. Leaving just two for..
 
 > reg06: base=0x100000000 (4096MB), size=4096MB: write-back, count=1
 > reg07: base=0x200000000 (8192MB), size=8192MB: write-back, count=1

So you really are out of MTRRs (There are only 8 on this CPU).
Does your BIOS have any option to disable this hole in low memory ?
I've seen similar things on laptops that use shared memory between
CPU & Graphics card, but I was unaware of anything like this on
desktop/server motherboards.

If this has onboard VGA, and theres a way to disable it, perhaps that
will work. (Even if it means plugging in a PCI/AGP video card)

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: no more MTRRs available ?

[Dave Jones]

On Wed, Jan 29, 2003 at 04:14:46PM +0000, Dave Jones wrote:

 >  > # cat /proc/mtrr
 >  > reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
 >  > reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
 >  > reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
 >  > reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
 >  > reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
 >  > reg05: base=0xf7000000 (3952MB), size=  16MB: uncachable, count=1
 > Due to this 16MB hole, your BIOS has to set up a write-back range
 > covering 2048-16 (2032MB).

Dummies guide to maths. This should be 4096-16=4080 of course

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: no more MTRRs available ?

[William Lee Irwin III]

On Wed, Jan 29, 2003 at 04:45:52PM +0100, Stephan von Krawczynski wrote:
> # cat /proc/mtrr
> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
> reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
> reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
> reg05: base=0xf7000000 (3952MB), size=  16MB: uncachable, count=1
> reg06: base=0x100000000 (4096MB), size=4096MB: write-back, count=1
> reg07: base=0x200000000 (8192MB), size=8192MB: write-back, count=1

Looks better than what I'm getting on 2.5.59:

curly:~# cat /proc/mtrr
reg00: base=0xc0000000 (49152MB), size=16384MB: uncachable, count=1
reg01: base=0x00000000 (   0MB), size=524288MB: write-back, count=1
reg02: base=0x800000000 (524288MB), size=262144MB: write-back, count=1

Yes, this is standard ia32 (P-III/Coppermine cpus), and hence the
numbers here are utter garbage.


-- wli

Re: no more MTRRs available ?

[Dave Jones]

On Wed, Jan 29, 2003 at 09:20:01AM -0800, William Lee Irwin III wrote:

 > Looks better than what I'm getting on 2.5.59:
 > 
 > curly:~# cat /proc/mtrr
 > reg00: base=0xc0000000 (49152MB), size=16384MB: uncachable, count=1
 > reg01: base=0x00000000 (   0MB), size=524288MB: write-back, count=1
 > reg02: base=0x800000000 (524288MB), size=262144MB: write-back, count=1
 > Yes, this is standard ia32 (P-III/Coppermine cpus), and hence the
 > numbers here are utter garbage.

Bizarre. The size field isn't being shifted, and your base is somewhere
off in 64bit land.
See Andi's "RED-PEN" comments in various parts of arch/i386/kernel/cpu/mtrr/
They need fixing at some point, and could be the cause of your problems.

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: no more MTRRs available ?

[William Lee Irwin III]

On Wed, Jan 29, 2003 at 09:20:01AM -0800, William Lee Irwin III wrote:
>> Looks better than what I'm getting on 2.5.59:
>> curly:~# cat /proc/mtrr
>> reg00: base=0xc0000000 (49152MB), size=16384MB: uncachable, count=1
>> reg01: base=0x00000000 (   0MB), size=524288MB: write-back, count=1
>> reg02: base=0x800000000 (524288MB), size=262144MB: write-back, count=1
>> Yes, this is standard ia32 (P-III/Coppermine cpus), and hence the
>> numbers here are utter garbage.

On Wed, Jan 29, 2003 at 05:48:42PM +0000, Dave Jones wrote:
> Bizarre. The size field isn't being shifted, and your base is somewhere
> off in 64bit land.
> See Andi's "RED-PEN" comments in various parts of arch/i386/kernel/cpu/mtrr/
> They need fixing at some point, and could be the cause of your problems.

OTOH since 0-512GB are in there this explains why the (massive) perf.
decrease only happens sometimes. The MTRR corruption issues are only
visible with 48GB atm. I haven't been focusing on MTRR's but I may
arrange to trace the codepaths etc. in the eventual future to find
where the bits are going bad esp. as benchmark time approaches.

-- wli

Re: no more MTRRs available ?

[Dave Jones]

On Wed, Jan 29, 2003 at 10:00:11AM -0800, William Lee Irwin III wrote:

 > >> reg00: base=0xc0000000 (49152MB), size=16384MB: uncachable, count=1
 > >> reg01: base=0x00000000 (   0MB), size=524288MB: write-back, count=1
 > >> reg02: base=0x800000000 (524288MB), size=262144MB: write-back, count=1
 > >> Yes, this is standard ia32 (P-III/Coppermine cpus), and hence the
 > >> numbers here are utter garbage.
 > 
 > On Wed, Jan 29, 2003 at 05:48:42PM +0000, Dave Jones wrote:
 > > Bizarre. The size field isn't being shifted, and your base is somewhere
 > > off in 64bit land.
 > > See Andi's "RED-PEN" comments in various parts of arch/i386/kernel/cpu/mtrr/
 > > They need fixing at some point, and could be the cause of your problems.
 > 
 > OTOH since 0-512GB are in there this explains why the (massive) perf.
 > decrease only happens sometimes. The MTRR corruption issues are only
 > visible with 48GB atm. I haven't been focusing on MTRR's but I may
 > arrange to trace the codepaths etc. in the eventual future to find
 > where the bits are going bad esp. as benchmark time approaches.

ohhh, 48GB. I forgot you were doing the silly-amounts-of-mem thing.
Its extremely likely you'll have to fix up those comments Andi made,
and possibly some other parts too.  That code should be 64bit clean
now (due to x86-64 sharing it), but there may still be some gotchas,
especially on weird-ass systems like what you've been playing with 8)

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs