Re: Terrible IO performance when using 4GB of RAM on a 32 bit machine

Re: Terrible IO performance when using 4GB of RAM on a 32 bit machine

[Robert Hancock]

Peter Rabbitson wrote:
> H. Peter Anvin wrote:
>> Peter Rabbitson wrote:
>>> I have captured dmesg output without mem[5], with mem=3900M[6] and
>>> mem=2048M[7].
>>>
>>
>> What does /proc/mtrr look like in the two cases?
>>
> 
> Identical for mem=3900 and without it.
> 
> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
> reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
> reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
> reg05: base=0xf8000000 (3968MB), size=  32MB: write-back, count=1

Looks like another case of bad MTRRs on an Intel motherboard? The BIOS 
is marking only memory up to 4000MB as cacheable, but the actual memory 
extends up to about 4031MB. Therefore anything that accesses the top 
31MB of memory will run very slow.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Terrible IO performance when using 4GB of RAM on a 32 bit machine

[Peter Rabbitson]

Robert Hancock wrote:
> Peter Rabbitson wrote:
>> H. Peter Anvin wrote:
>>>
>>> What does /proc/mtrr look like in the two cases?
>>>
>>
>> Identical for mem=3900 and without it.
>>
>> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
>> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
>> reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
>> reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
>> reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
>> reg05: base=0xf8000000 (3968MB), size=  32MB: write-back, count=1
> 
> Looks like another case of bad MTRRs on an Intel motherboard? The BIOS 
> is marking only memory up to 4000MB as cacheable, but the actual memory 
> extends up to about 4031MB. Therefore anything that accesses the top 
> 31MB of memory will run very slow.
> 

Ah, it all makes sense now. In this case I assume mem=4000 is perfectly 
safe and usable for the time being. In the beginning I tried with 
mem=4g, which obviously did not work. If anyone is interested in adding 
an exception/workaround for this particular motherboard, I'd be happy to 
help with testing. I have added more information about the system: 
current kernel config [1], output of `lspci -vv`[2], dmesg with mem=4000[3].

Thank you!

Peter

[1] http://rabbit.us/pool/4g/config-2.6.21.5.arzamas.6.txt
[2] http://rabbit.us/pool/4g/lspci_4000.txt
[3] http://rabbit.us/pool/4g/dmesg_4000.txt

Re: Terrible IO performance when using 4GB of RAM on a 32 bit machine

[Robert Hancock]

Peter Rabbitson wrote:
> Robert Hancock wrote:
>> Peter Rabbitson wrote:
>>> H. Peter Anvin wrote:
>>>>
>>>> What does /proc/mtrr look like in the two cases?
>>>>
>>>
>>> Identical for mem=3900 and without it.
>>>
>>> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
>>> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
>>> reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
>>> reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
>>> reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
>>> reg05: base=0xf8000000 (3968MB), size=  32MB: write-back, count=1
>>
>> Looks like another case of bad MTRRs on an Intel motherboard? The BIOS 
>> is marking only memory up to 4000MB as cacheable, but the actual 
>> memory extends up to about 4031MB. Therefore anything that accesses 
>> the top 31MB of memory will run very slow.
>>
> 
> Ah, it all makes sense now. In this case I assume mem=4000 is perfectly 
> safe and usable for the time being. In the beginning I tried with 
> mem=4g, which obviously did not work. If anyone is interested in adding 
> an exception/workaround for this particular motherboard, I'd be happy to 
> help with testing. I have added more information about the system: 
> current kernel config [1], output of `lspci -vv`[2], dmesg with 
> mem=4000[3].
> 
> Thank you!
> 
> Peter

There was a patch floating around recently to detect the case where the 
MTRRs don't map all of RAM as write-back, automatically cap the memory 
used by the kernel to what is mapped and print some loud warnings..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Terrible IO performance when using 4GB of RAM on a 32 bit machine [solved]

[Peter Rabbitson]

Robert Hancock wrote:
> Peter Rabbitson wrote:
>>
>> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
>> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
>> reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
>> reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
>> reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
>> reg05: base=0xf8000000 (3968MB), size=  32MB: write-back, count=1
> 
> Looks like another case of bad MTRRs on an Intel motherboard? The BIOS 
> is marking only memory up to 4000MB as cacheable, but the actual memory 
> extends up to about 4031MB. Therefore anything that accesses the top 
> 31MB of memory will run very slow.

I sincerely apologize for not paying enough attention. Intel has fixed 
this issue 2 BIOS revisions ago[1], kudos to Suren Karapetyan for 
pointing this out. I just upgraded the BIOS and it indeed solves the 
problem. The mtrr still seems not to be going over 4000MB, but 
everything works without any visible slowdowns at all. Here is how the 
mtrr and the relevant dmesg line look like now:

reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 512MB: write-back, count=1
reg03: base=0xe0000000 (3584MB), size= 256MB: write-back, count=1
reg04: base=0xf0000000 (3840MB), size= 128MB: write-back, count=1
(there is no reg05)
...
Memory: 4024568k/4063168k available (2894k kernel code, 37456k reserved, 
930k data, 252k init, 3145664k highmem)
...

Thank you for the help!

Peter

[1] 
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=N&Inst=Yes&ProductID=1640&DwnldID=11261&strOSs=All&OSFullName=All%20Operating%20Systems&lang=eng