MTRR initialization

MTRR initialization

[Howard Chu]

Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of 
RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By 
default, my system boots up with these MTRR settings:

reg00: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

The X server and various other programs try to add a mapping for my video 
card's buffer, at 0xd0000000, size=256MB, type=write-combining, and this always 
fails with a type mismatch error (old type is write-back). Apparently it's 
conflicting with mapping register 0. I can't just disable the existing settings 
and re-add them; the system hangs soon after disabling reg01.

I guess the kernel must be getting the initial setup from the BIOS. I've hacked 
around this in mtrr/generic.c by explicitly changing the MTRR state in 
get_mtrr_state to split the first mapping into two; one at base 0 size 2048M 
and one at base 2048M size 1024M. So now I have this, which is pretty much what 
I wanted:

reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1

So the question is - was there an easier/correct way to do this?

It might have been nice if the MTRR ioctls allowed the register number to be 
specified on the Set commands, though I'm not sure that would have helped in 
this case.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Re: MTRR initialization

[Yinghai Lu]

On 9/14/07, Howard Chu <hyc@symas.com> wrote:
> Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
> default, my system boots up with these MTRR settings:
>
> reg00: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

BIOS should have another setup option about MTRR (continous?)
then you could get

> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> reg01: base=0x100000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
===> if you have rev E...instead of rev F.

YH

Re: MTRR initialization

[Howard Chu]

[-- Attachment #1: Type: text/plain, Size: 1262 bytes --]

Yinghai Lu wrote:
> On 9/14/07, Howard Chu <hyc@symas.com> wrote:
>> Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
>> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
>> default, my system boots up with these MTRR settings:
>>
>> reg00: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
>> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
>> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
> 
> BIOS should have another setup option about MTRR (continous?)
> then you could get
> 
>> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
>> reg01: base=0x100000000 (2048MB), size=1024MB: write-back, count=1
> reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> ===> if you have rev E...instead of rev F.

Thanks for the reply. Unfortunately this BIOS doesn't seem to have any other options related to the 
MTRRs. I guess I'll just live with this hack.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

[-- Attachment #2: dif.txt --]
[-- Type: text/plain, Size: 1262 bytes --]

--- generic.c.O	2007-08-30 23:21:01.000000000 -0700
+++ generic.c	2007-09-16 09:09:34.000000000 -0700
@@ -78,6 +78,8 @@
 			base, base + step - 1, mtrr_attrib_to_str(*types));
 }
 
+static int set_mtrr_var_ranges(unsigned int index, struct mtrr_var_range *vr);
+
 /*  Grab all of the MTRR state for this CPU into *state  */
 void get_mtrr_state(void)
 {
@@ -105,6 +107,21 @@
 	mtrr_state.def_type = (lo & 0xff);
 	mtrr_state.enabled = (lo & 0xc00) >> 10;
 
+	/* Check for initial 4GB range, it should only be 3GB */
+	if (!mtrr_state.var_ranges[0].base_hi && !(mtrr_state.var_ranges[0].base_lo & 0xffffff00) &&
+		!(mtrr_state.var_ranges[0].mask_lo & 0x80000000)) {
+		/* split initial 4GB range into 2GB and 1GB */
+		for (i = num_var_ranges-1; i>0; i--)
+			mtrr_state.var_ranges[i] = mtrr_state.var_ranges[i-1];
+		mtrr_state.var_ranges[0].mask_lo |= 0x80000000;
+		mtrr_state.var_ranges[1].base_hi = 0;
+		mtrr_state.var_ranges[1].base_lo = 0x80000000 | MTRR_TYPE_WRBACK ;
+		mtrr_state.var_ranges[1].mask_hi = mtrr_state.var_ranges[2].mask_hi;
+		mtrr_state.var_ranges[1].mask_lo = mtrr_state.var_ranges[2].mask_lo;
+		for (i = 0; i < num_var_ranges; i++)
+			set_mtrr_var_ranges(i, &mtrr_state.var_ranges[i]);
+	}
+
 	if (mtrr_show) {
 		int high_width;
 

Re: MTRR initialization

[Yinghai Lu]

On 9/16/07, Howard Chu <hyc@symas.com> wrote:
> Yinghai Lu wrote:
> > On 9/14/07, Howard Chu <hyc@symas.com> wrote:
> >> Hi, was wondering if anyone else has been tripped up by this... I've got 4GB of
> >> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
> >> default, my system boots up with these MTRR settings:
> >>
> >> reg00: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
> >> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be
> >> reg00: base=0x00000000 (   0MB), size=8192MB: write-back, count=1
> >> reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
and you will have problem...

So good way is get the ram top, and hole size...and reset the whole
set ot var mtrr.

you could refer the code in LinuxBIOS about setting that...
http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
esp the part about CONFIG_VAR_MTRR_HOLE...

but Andi and Eric said resetting mtrr is not good... when someone from
intel try to trim the MTRR for intel CPU.

YH

Re: MTRR initialization

[Eric W. Biederman]

"Yinghai Lu" <yhlu.kernel@gmail.com> writes:

> On 9/16/07, Howard Chu <hyc@symas.com> wrote:
>> Yinghai Lu wrote:
>> > On 9/14/07, Howard Chu <hyc@symas.com> wrote:
>> >> Hi, was wondering if anyone else has been tripped up by this... I've got
> 4GB of
>> >> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
>> >> default, my system boots up with these MTRR settings:
>> >>
>> >> reg00: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
>> >> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
>> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
>
> if rev F before cpu aka Rev E is installed, and 8G RAM installed, it will be
>> >> reg00: base=0x00000000 (   0MB), size=8192MB: write-back, count=1
>> >> reg01: base=0x100000000 (8192MB), size=1024MB: write-back, count=1
>> >> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
>> >> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
> and you will have problem...
>
> So good way is get the ram top, and hole size...and reset the whole
> set ot var mtrr.
>
> you could refer the code in LinuxBIOS about setting that...
> http://www.openbios.org/viewvc/trunk/LinuxBIOSv2/src/cpu/x86/mtrr/mtrr.c?revision=2616&view=markup
> esp the part about CONFIG_VAR_MTRR_HOLE...
>
> but Andi and Eric said resetting mtrr is not good... when someone from
> intel try to trim the MTRR for intel CPU.

There are a couple issues with changing the MTRR configuration.
- You may not have perfect information on the cpu, the AMD revF is a good
  example.
- Code in SMM mode may actually depend on the current mtrr configuration.
- The BIOS's need to fixed to setup MTRRs properly.

So the sanest approach appears to be.
- In linux only use ram that is mapped by a write-back mtrr.
  This preserves performance and is always safe.
- If you need write-combining set it up in the page tables with PAT.

There is some difficulty there but software can always do those things
safely.

Eric

Re: MTRR initialization

[Howard Chu]

Eric W. Biederman wrote:
>> but Andi and Eric said resetting mtrr is not good... when someone from
>> intel try to trim the MTRR for intel CPU.
> 
> There are a couple issues with changing the MTRR configuration.
> - You may not have perfect information on the cpu, the AMD revF is a good
>   example.
> - Code in SMM mode may actually depend on the current mtrr configuration.
> - The BIOS's need to fixed to setup MTRRs properly.

Well the BIOS is definitely doing it wrong here. As I mentioned before, it was 
setting up
	0-0x100000000 WB
	0xc0000000 - 0x100000000 UC
	0xc0000000 - 0xd0000000 WC

But the Intel Architecture Software Developer's Manual states that whenever 
any variable MTRR range overlaps with an UC MTRR range, the range remains UC. 
(Section 9.12.2.3). So in fact what I needed to set was
	0-2GB WB
	2-3GB WB
	3-3.25GB WC
and delete the 3-4GB UC range to get the behavior that the BIOS seems to have 
been intending to set up. (Relying on the default of UC for the unspecified 
ranges.)

> So the sanest approach appears to be.
> - In linux only use ram that is mapped by a write-back mtrr.
>   This preserves performance and is always safe.
> - If you need write-combining set it up in the page tables with PAT.
> 
> There is some difficulty there but software can always do those things
> safely.

Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is 
always UC regardless of the bits in the page table. So with the MTRR setup 
that the BIOS left me with, this is still a no-go. There's no way to get the 
desired effect without completely reinitializing the MTRRs.

Of course, this isn't the only problem with these Asus BIOSs...
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Re: MTRR initialization

[Eric W. Biederman]

Howard Chu <hyc@symas.com> writes:

> Eric W. Biederman wrote:
>>> but Andi and Eric said resetting mtrr is not good... when someone from
>>> intel try to trim the MTRR for intel CPU.
>>
>> There are a couple issues with changing the MTRR configuration.
>> - You may not have perfect information on the cpu, the AMD revF is a good
>>   example.
>> - Code in SMM mode may actually depend on the current mtrr configuration.
>> - The BIOS's need to fixed to setup MTRRs properly.
>
> Well the BIOS is definitely doing it wrong here. As I mentioned before, it was
> setting up
> 	0-0x100000000 WB
> 	0xc0000000 - 0x100000000 UC
> 	0xc0000000 - 0xd0000000 WC

Ok.  I do agree that is wrong, and really weird to.

> But the Intel Architecture Software Developer's Manual states that whenever any
> variable MTRR range overlaps with an UC MTRR range, the range remains
> UC. (Section 9.12.2.3). So in fact what I needed to set was
> 	0-2GB WB
> 	2-3GB WB
> 	3-3.25GB WC
> and delete the 3-4GB UC range to get the behavior that the BIOS seems to have
> been intending to set up. (Relying on the default of UC for the unspecified
> ranges.)

Which is certainly a good way to go if your hole sizes allow and you
have enough MTRRs.

>> So the sanest approach appears to be.
>> - In linux only use ram that is mapped by a write-back mtrr.
>>   This preserves performance and is always safe.
>> - If you need write-combining set it up in the page tables with PAT.
>>
>> There is some difficulty there but software can always do those things
>> safely.
>
> Hm. Section 9.5.1 of the doc (table 9-5) says that anything marked UC is always
> UC regardless of the bits in the page table. So with the MTRR setup that the
> BIOS left me with, this is still a no-go. There's no way to get the desired
> effect without completely reinitializing the MTRRs.

Please look a little more closely.  WC is a flavor of UC.  There is a
specific exception that allows page tables to promote UC to WC.

I don't seem to have the same version of this document so I can't
refer you to specific sections.  But I have looked it up several
times and every time I have looked I have found the exception that
allows PAT to promote UC to WC.

> Of course, this isn't the only problem with these Asus BIOSs...

Sure.

My point was that when we try to solve the general problem rather then
a specific case there are some very practical limits on what can be
done.

Now it should be mentioned that you can go in with /proc/mtrr and fix
things manually on a specific machine.


Eric

Re: MTRR initialization

[Jesse Barnes]

On Friday, September 14, 2007 9:33 am Howard Chu wrote:
> So the question is - was there an easier/correct way to do this?
>
> It might have been nice if the MTRR ioctls allowed the register
> number to be specified on the Set commands, though I'm not sure that
> would have helped in this case.

To do this in a nicer way (and be less vulnerable to similar BIOS 
funkiness) the kernel really needs full PAT support.  That should allow 
WC over WB and WC over UC mappings to occur, at least if I'm 
remembering the docs right...

Jesse

Re: MTRR initialization

[Andi Kleen]

Jesse Barnes <jesse.barnes@intel.com> writes:
> 
> To do this in a nicer way (and be less vulnerable to similar BIOS 
> funkiness) the kernel really needs full PAT support.  That should allow 
> WC over WB and WC over UC mappings to occur, at least if I'm 
> remembering the docs right...

PAT only really helps for device driver performance optimizations.
But if the basic WB MTRRs are wrong PAT cannot really salvage it. 

-Andi

Re: MTRR initialization

[Avi Kivity]

Andi Kleen wrote:
> Jesse Barnes <jesse.barnes@intel.com> writes:
>   
>> To do this in a nicer way (and be less vulnerable to similar BIOS 
>> funkiness) the kernel really needs full PAT support.  That should allow 
>> WC over WB and WC over UC mappings to occur, at least if I'm 
>> remembering the docs right...
>>     
>
> PAT only really helps for device driver performance optimizations.
> But if the basic WB MTRRs are wrong PAT cannot really salvage it. 
>   

Is there any reason not to set the MTRRs to define the entire memory as 
write back, and use PAT exclusively for setting cacheability?

On my home machine for instance, the BIOS uses all 8 MTRRs leaving none 
for X.  I hacked it by merging a couple of MTRRs but this isn't a 
generic solution.

-- 
error compiling committee.c: too many arguments to function

Re: MTRR initialization

[Andi Kleen]

> Is there any reason not to set the MTRRs to define the entire memory as 
> write back, and use PAT exclusively for setting cacheability?

That risks breaking SMM or BIOS code.

-Andi

Re: MTRR initialization

[Avi Kivity]

Andi Kleen wrote:
>> Is there any reason not to set the MTRRs to define the entire memory as 
>> write back, and use PAT exclusively for setting cacheability?
>>     
>
> That risks breaking SMM or BIOS code.
>
>   


Yes.  Too bad.

-- 
error compiling committee.c: too many arguments to function

Re: MTRR initialization

[Jesse Barnes]

On Wednesday, September 19, 2007 11:50:23 pm Andi Kleen wrote:
> Jesse Barnes <jesse.barnes@intel.com> writes:
> > To do this in a nicer way (and be less vulnerable to similar BIOS
> > funkiness) the kernel really needs full PAT support.  That should allow
> > WC over WB and WC over UC mappings to occur, at least if I'm
> > remembering the docs right...
>
> PAT only really helps for device driver performance optimizations.
> But if the basic WB MTRRs are wrong PAT cannot really salvage it.

Look at the original bug report.  There's a bug WB region so any subsequent WC 
mappings via the MTRRs won't work, so you need PAT to do WC (unless you want 
to rewrite all your MTRR settings as Howard has to do now).

Seems like more and more machines are being setup this way, and until we have 
PAT users will have to jump through hoops to get good performance.

Jesse

Re: MTRR initialization

[Jesse Barnes]

On Thursday, September 20, 2007, Jesse Barnes wrote:
> On Wednesday, September 19, 2007 11:50:23 pm Andi Kleen wrote:
> > Jesse Barnes <jesse.barnes@intel.com> writes:
> > > To do this in a nicer way (and be less vulnerable to similar BIOS
> > > funkiness) the kernel really needs full PAT support.  That should
> > > allow WC over WB and WC over UC mappings to occur, at least if
> > > I'm remembering the docs right...
> >
> > PAT only really helps for device driver performance optimizations.
> > But if the basic WB MTRRs are wrong PAT cannot really salvage it.
>
> Look at the original bug report.  There's a bug WB region so any
                                              ^^^
                                          err big

Re: MTRR initialization

[Siddha, Suresh B]

On Fri, Sep 14, 2007 at 09:33:30AM -0700, Howard Chu wrote:
> Hi, was wondering if anyone else has been tripped up by this... I've got
> 4GB of
> RAM in my Asus A8V Deluxe and memory hole mapping enabled in the BIOS. By
> default, my system boots up with these MTRR settings:
> 
> reg00: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> reg03: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
> 
> The X server and various other programs try to add a mapping for my video
> card's buffer, at 0xd0000000, size=256MB, type=write-combining, and this
> always
> fails with a type mismatch error (old type is write-back). Apparently it's
> conflicting with mapping register 0. I can't just disable the existing
> settings
> and re-add them; the system hangs soon after disabling reg01.
> 
> I guess the kernel must be getting the initial setup from the BIOS. I've
> hacked
> around this in mtrr/generic.c by explicitly changing the MTRR state in
> get_mtrr_state to split the first mapping into two; one at base 0 size
> 2048M
> and one at base 2048M size 1024M. So now I have this, which is pretty much
> what
> I wanted:
> 
> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
> reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
> reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1

BTW, having overlapping WC, UC regions make the end result UC. So in this
case, you may not be getting the desired performance.

> 
> So the question is - was there an easier/correct way to do this?
> 
> It might have been nice if the MTRR ioctls allowed the register number to
> be
> specified on the Set commands, though I'm not sure that would have helped
> in
> this case.

Re: MTRR initialization

[Howard Chu]

Siddha, Suresh B wrote:
> On Fri, Sep 14, 2007 at 09:33:30AM -0700, Howard Chu wrote:
>> So now I have this, which is pretty much
>> what
>> I wanted:
>>
>> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
>> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
>> reg02: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
>> reg03: base=0xc0000000 (3072MB), size=1024MB: uncachable, count=1
>> reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1
>> reg05: base=0xd0000000 (3328MB), size= 256MB: write-combining, count=1
> 
> BTW, having overlapping WC, UC regions make the end result UC. So in this
> case, you may not be getting the desired performance.

Thanks, I noticed that later. I simply deleted the UC mapping since it was no 
longer needed.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/