memory management weirdness

memory management weirdness

[Martin MOKREJŠ]

Hi,
  I have received no answer to my former question
(see http://marc.theaimsgroup.com/?l=linux-kernel&m=110827143716215&w=2).
I've spent some more time on that problem and have more or less confirmed
it's because of buggy bios. However, the linux kernel doesn't handle properly
such case. I've tested 2.4.30-pre1 kernel and latest 2.6.11-rc4 kernel.
The conclusion is, that once the machine has physically installed 4x1GB
DDR400 DIMM's (bios detects only 3556 or less memory as some buffers
are allocated by the Intel 875P chipset and AGP card), the linux 2.6.11*
runs up-to 18x slower than when only 2x1GB + 2x 512MB DDR memory is installed.

  Although I've not re-tested this today again, it used to help a bit to specify
mem=3548M to decrease memory used by linux (tested with AGP card plugged in, when
bios reported 3556MB RAM only).

  I found that removing the AGP based videoc card and using an old PCI based
video card results in bios detecting 4072MB of RAM. But still, the machine was
slow. I've tried to "cat >| /proc/mtrr" to alter the memory settings, but the
result was only a partial speedup.

  I'm not sure how to convince linux kernel to run fast again.
I suspect either the memory mapping of interrupts are the cause. Disabling
acpi did not help me initially, so I've conducted most of my tests with
acpi enabled.

  I've put dmesg, iomem, interrupt, lspci and time(1) requirements of my test
on web: http://www.natur.cuni.cz/~mmokrejs/tmp/. The differences can be seen easily
by diffing the files. All tests in http://www.natur.cuni.cz/~mmokrejs/tmp/4MB/
we carried with AGP aperture size set to 4MB, although teh video card has
128MB RAM.
  
  Later I've reverted to AGP aperture set to 128MB back and tested again:
http://www.natur.cuni.cz/~mmokrejs/tmp/128MB/.

  Finally, I put back two 512MB memory modules to have only 3GB RAM physically,
and the result is at http://www.natur.cuni.cz/~mmokrejs/tmp/128MB/only_phys_3GB/.

  About a week ago I tried to contact ASUS, but no answer so far from their
techinical support through some web robot.
http://vip.asus.com/eservice/techmailstatus.aspx?ID=WTM200502111723398547
I do not recommend their "greatest" and real "flag-ship" P4C800-E-Deluxe
motherboard for use with memory sizes above 3GB (although they claim 4GB
is possible). BIOS is the latest release 1.19, although 1.20.001 was tested
as well.


  
My questions to LKML people are:

1)  Could someone tell me what are the differences in
2.4.30-pre1 kernel'd dmesg and 2.6.11-rc4* dmesg outputs? For example, memory
areas "reserved twice" reported by 2.4.30-pre1. Also, differences in /proc/mtrr
under both kernels differ.

2)  How about the /proc/interrupts outputs? Aren't they too high? How about the
level/edge interrupt mappings? Would they help?

Please Cc: me in replies. Many thanks for any response, I have wasted seemingly
a lot of money on 2GB RAM. :(
martin

P.S.:

1GB DDR400 modules are Micron CL2.5 2bank 512M chip modules (64Mx8),
non-ecc, unbuffered

512MB DDR 500 modules (yes, PC4000, not PC3200 as is the max supported
by the motherboard) are Kingston HyperX modules, non-ecc, unbuffered.

Re: memory management weirdness

[Parag Warudkar]

> Hi,
>   I have received no answer to my former question
> (see http://marc.theaimsgroup.com/?l=linux-kernel&m=110827143716215&w=2).
> I've spent some more time on that problem and have more or less confirmed
> it's because of buggy bios. However, the linux kernel doesn't handle properly
> such case. I've tested 2.4.30-pre1 kernel and latest 2.6.11-rc4 kernel.
> The conclusion is, that once the machine has physically installed 4x1GB
> DDR400 DIMM's (bios detects only 3556 or less memory as some buffers
> are allocated by the Intel 875P chipset and AGP card), the linux 2.6.11*
> runs up-to 18x slower than when only 2x1GB + 2x 512MB DDR memory is installed.
> 
Can you enable profiling and then post the profile info for various cases - slow and fast? Check out Documentation/basic_profiling.txt in the kernel source for understanding how to do this. This might help narrow down the issue.

Parag

Re: memory management weirdness

[Andi Kleen]

Martin MOKREJ© <mmokrejs@ribosome.natur.cuni.cz> writes:

> Hi,
>   I have received no answer to my former question
> (see http://marc.theaimsgroup.com/?l=linux-kernel&m=110827143716215&w=2).

That's because it's a BIOS problem.

There are limits on how much Linux can work around BIOS breakage.


>   Although I've not re-tested this today again, it used to help a bit to specify
> mem=3548M to decrease memory used by linux (tested with AGP card plugged in, when
> bios reported 3556MB RAM only).
>
>   I found that removing the AGP based videoc card and using an old PCI based
> video card results in bios detecting 4072MB of RAM. But still, the machine was
> slow. I've tried to "cat >| /proc/mtrr" to alter the memory settings, but the
> result was only a partial speedup.
>
>   I'm not sure how to convince linux kernel to run fast again.

It's most likely a MTRR problem. Play more with them.


>   Finally, I put back two 512MB memory modules to have only 3GB RAM physically,
> and the result is at http://www.natur.cuni.cz/~mmokrejs/tmp/128MB/only_phys_3GB/.


The cheaper Intel chipsets don't support >4GB at all, and you always
need some space below 4GB for PCI memory mappings/AGP aperture etc.


>   About a week ago I tried to contact ASUS, but no answer so far from their
> techinical support through some web robot.
> http://vip.asus.com/eservice/techmailstatus.aspx?ID=WTM200502111723398547
> I do not recommend their "greatest" and real "flag-ship" P4C800-E-Deluxe
> motherboard for use with memory sizes above 3GB (although they claim 4GB
> is possible). BIOS is the latest release 1.19, although 1.20.001 was tested
> as well.

In general non server boards tend to be not very well or not at all
tested with a lot of memory ("a lot" is defined as >2GB for higher end
desktop boards, or >1GB on very cheap desktop boards). That is a
common problem on other motherboards too; Asus is not alone with this.

-Andi

Re: memory management weirdness

[Ingo Molnar]

* Andi Kleen <ak@muc.de> wrote:

> >   Although I've not re-tested this today again, it used to help a bit to specify
> > mem=3548M to decrease memory used by linux (tested with AGP card plugged in, when
> > bios reported 3556MB RAM only).
> >
> >   I found that removing the AGP based videoc card and using an old PCI based
> > video card results in bios detecting 4072MB of RAM. But still, the machine was
> > slow. I've tried to "cat >| /proc/mtrr" to alter the memory settings, but the
> > result was only a partial speedup.
> >
> >   I'm not sure how to convince linux kernel to run fast again.
> 
> It's most likely a MTRR problem. Play more with them.

in particular, try to create two small tables in the same format: one
showing the e820 memory map as reported in your kernel log, and one
showing the mtrr areas. If there is any e820 area that is not write-back
cached via the mtrr mappings then that's the problem. You can also use
"mem=exactmap,..." to fix up the memory map that the BIOS provides to
Linux. Slowdowns are very often such MTRR problems. (perhaps the kernel
should report RAM areas that are not covered by MTRR write-back?)

	Ingo

Re: memory management weirdness

[Martin MOKREJŠ]

Parag Warudkar wrote:
>>Hi,
>>  I have received no answer to my former question
>>(see http://marc.theaimsgroup.com/?l=linux-kernel&m=110827143716215&w=2).
>>I've spent some more time on that problem and have more or less confirmed
>>it's because of buggy bios. However, the linux kernel doesn't handle properly
>>such case. I've tested 2.4.30-pre1 kernel and latest 2.6.11-rc4 kernel.
>>The conclusion is, that once the machine has physically installed 4x1GB
>>DDR400 DIMM's (bios detects only 3556 or less memory as some buffers
>>are allocated by the Intel 875P chipset and AGP card), the linux 2.6.11*
>>runs up-to 18x slower than when only 2x1GB + 2x 512MB DDR memory is installed.
>>
> 
> Can you enable profiling and then post the profile info for various cases
> - slow and fast? Check out Documentation/basic_profiling.txt in the kernel
> source for understanding how to do this. This might help narrow down the issue.

http://www.natur.cuni.cz/~mmokrejs/tmp/profile-2.6.11-rc4-bk7-(3|4)GB.txt

The 3GB labeled file corresponds to fast case, 4GB is ugly slow.
What can you gather from those files? I've used readprofile but also oprofile
was enabled in kernel. I've left on the web also /proc/profile snapshots along with
System.map file. Maybe oprofile can also be used later to extract info from them.
Many thanks for help!
Martin

Re: memory management weirdness

[Martin MOKREJŠ]

Ingo Molnar wrote:
> * Andi Kleen <ak@muc.de> wrote:
> 
> 
>>>  Although I've not re-tested this today again, it used to help a bit to specify
>>>mem=3548M to decrease memory used by linux (tested with AGP card plugged in, when
>>>bios reported 3556MB RAM only).
>>>
>>>  I found that removing the AGP based videoc card and using an old PCI based
>>>video card results in bios detecting 4072MB of RAM. But still, the machine was
>>>slow. I've tried to "cat >| /proc/mtrr" to alter the memory settings, but the
>>>result was only a partial speedup.
>>>
>>>  I'm not sure how to convince linux kernel to run fast again.
>>
>>It's most likely a MTRR problem. Play more with them.
> 
> 
> in particular, try to create two small tables in the same format: one
> showing the e820 memory map as reported in your kernel log, and one
> showing the mtrr areas. If there is any e820 area that is not write-back
> cached via the mtrr mappings then that's the problem. You can also use
> "mem=exactmap,..." to fix up the memory map that the BIOS provides to
> Linux. Slowdowns are very often such MTRR problems. (perhaps the kernel
> should report RAM areas that are not covered by MTRR write-back?)

I've just extracted the requested info from the files I've put on web.
Here it is:

                                               2.4.30-pre1-bk5    2.6.11-rc4-bk7
0000000000000000 - 000000000009fc00 (usable)       +                    +
000000000009fc00 - 00000000000a0000 (reserved)     +                    +
00000000000e8000 - 0000000000100000 (reserved)     +                    +
0000000000100000 - 00000000de330000 (usable)       +                    +
00000000de330000 - 00000000de340000 (ACPI data)    +                    +
00000000de340000 - 00000000de3f0000 (ACPI NVS)     +                    +
00000000de3f0000 - 00000000de400000 (reserved)     +                    +
00000000ffb80000 - 0000000100000000 (reserved)     +                    +
found SMP MP-table at 000ff780                     +                    +
hm, page 000ff000 reserved twice.                  +                    - ???
hm, page 00100000 reserved twice.                  +                    - ???
hm, page 000f1000 reserved twice.                  +                    - ???
hm, page 000f2000 reserved twice.                  +                    - ???




                                                                    2.4.30-pre1-bk5    2.6.11-rc4-bk7
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1       +                      +
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1       +                      +
reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1       +                      +
reg03: base=0xd0000000 (3328MB), size= 128MB: write-back, count=1       +                      +
reg04: base=0xd8000000 (3456MB), size=  64MB: write-back, count=1       +                      +
reg05: base=0xdc000000 (3520MB), size=  32MB: write-back, count=1       +                      +
reg06: base=0xfe800000 (4072MB), size=   4MB: write-combining, count=1  +                      - !!!
reg06: base=0xf0000000 (3840MB), size= 128MB: write-combining, count=1  +                      +
The 4MB area should be AGP aperture, as it was set in BIOS to 4MB only

The files on web contain concatened infor from dmes, iomem, interrupts, mtrr, lspci:
http://www.natur.cuni.cz/~mmokrejs/tmp/4MB/2.4.30-pre1-bk5
http://www.natur.cuni.cz/~mmokrejs/tmp/4MB/2.6.11-rc4-bk7


So, 2.6 kernel does not see AGP aperture area. What to do next? ;)
Martin

Re: memory management weirdness

[Parag Warudkar]

On Tuesday 22 February 2005 04:57 am, Martin MOKREJŠ wrote:
> The 3GB labeled file corresponds to fast case, 4GB is ugly slow.
> What can you gather from those files?
I did take a look and didn't analyze it further since Andi Mentioned it is a 
known BIOS bug.
Sorry about the trouble - didn't imagine it might be  BIOS related. Generally 
speaking it helps to have profile available when things are going slow.

Parag

Re: memory management weirdness

[Martin MOKREJS(]

Hi,
  I'm continuing an old thread on this topic (see bottom of this mail fro previous discussion).
I've received some answer from ASUS and their German support says that although
their BIOS cannot re-map the memory the operating system can access the whole
memory (they mention Win200 Advanced Server OS)

0    - 3327 MB -> 3327 MB Mainmemory
3327 - 4096 MB -> PCI ROM Reserved
4096 - 4865 MB -> 769 MB Mainmemory (Rest of the 4 GB Memory)

At the moment I have all four 1gB ddr modules in the slots and BIOS has stopped
memory checks at 3327 MB and started to boot. I get sloow machine and:

Linux version 2.6.12-rc5-git6 (root@aquarius) (gcc version 3.4.3-20050110 (Gentoo Linux 3.4.3.20050110-r2, ssp-3.4.3.20050110-0, pie-8.7.7)) #2 Thu Jun 2 11:49:00 CEST 2005
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000cfe30000 (usable)
 BIOS-e820: 00000000cfe30000 - 00000000cfe40000 (ACPI data)
 BIOS-e820: 00000000cfe40000 - 00000000cfef0000 (ACPI NVS)
 BIOS-e820: 00000000cfef0000 - 00000000cff00000 (reserved)
 BIOS-e820: 00000000ffb80000 - 0000000100000000 (reserved)
2430MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000ff780
On node 0 totalpages: 851504
  DMA zone: 4096 pages, LIFO batch:1
  Normal zone: 225280 pages, LIFO batch:31
  HighMem zone: 622128 pages, LIFO batch:31
DMI 2.3 present.
ACPI: RSDP (v002 ACPIAM                                ) @ 0x000f9e30
ACPI: XSDT (v001 A M I  OEMXSDT  0x10000426 MSFT 0x00000097) @ 0xcfe30100
ACPI: FADT (v003 A M I  OEMFACP  0x10000426 MSFT 0x00000097) @ 0xcfe30290
ACPI: MADT (v001 A M I  OEMAPIC  0x10000426 MSFT 0x00000097) @ 0xcfe30390
ACPI: OEMB (v001 A M I  OEMBIOS  0x10000426 MSFT 0x00000097) @ 0xcfe40040
ACPI: DSDT (v001  P4CED P4CED106 0x00000106 INTL 0x02002026) @ 0x00000000
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] disabled)
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at cff00000 (gap: cff00000:2fc80000)
Built 1 zonelists
Kernel command line: root=/dev/sda2 ide=reverse agp=try_unsupported console=ttyS0,57600n8 console=tty0 vga=792 idebus=66

# cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 128MB: write-back, count=1
reg03: base=0xc8000000 (3200MB), size=  64MB: write-back, count=1
reg04: base=0xcc000000 (3264MB), size=  32MB: write-back, count=1
reg05: base=0xce000000 (3296MB), size=  16MB: write-back, count=1
reg06: base=0xe0000000 (3584MB), size= 128MB: write-combining, count=3
reg07: base=0xf0000000 (3840MB), size= 128MB: write-combining, count=1
#


My questions are:
how can I make linux kernel to use rest of the memory? Below is their original answer.
Maybe there's someone interrested in this problem while living in Germany and speaking
much better that I do ... AND understanding the kernel stuff (which I do not).
Martin



From: tsd_germany at asuscom dot de
Hallo,

Das Bios kann kein Remapping bei uns. Sie haben ja schon selber gemerkt, wenn eine etwas höhere Karte eingesetzt wird, geht auch der Speicher runter, das ist normal, weil manche AGP Karten mehr Rom brauchen. 

...

dies ist kein Fehler, da 32bit System standardmäßig nur 4GB adressieren können.
Da alle PCI ROMs (VGA, LAN Controller etc) unterhalb von 4GB adressiert werden müssen, reservieren diese sich Bereiche unterhalb von 4GB.
Dies können mehrere hundert MB sein.

Der eigentliche Arbeitsspeicher liegt dann oberhalb von 4GB und kann durch Betriebssysteme angesprochen werden. (Win2000 Advanced Server)

0    - 3327 MB -> 3327 MB Hauptspeicher
3327 - 4096 MB -> PCI ROM Reservierung
	 4096 MB -> 4GB Grenze
4096 - 4865 MB -> 769 MB Hauptspeicher (Rest der 4 GB Memory)

- Bitte fügen Sie einer Antwort immer den gesamten Schriftverkehr bei !

- Please always attach all previous mails !

Mit freundlichen Grüssen 

 

Technical Support Division  [M07M]
Customer Service Center
ASUS Computer Germany

Homepage: http://www.asuscom.de/
FTP-Server: ftp://ftp.asuscom.de/pub/ASUSCOM

Tel : 02102/9599 - 0 ( Mo. - Fr. 10-17Uhr )
Fax: 02102/959911 ( 24h ) 

ASUS Computer GmbH
Harkortstr. 25
40880 Ratingen











Andi Kleen wrote:
> Martin MOKREJ© <mmokrejs@ribosome.natur.cuni.cz> writes:
> 
> 
>>Hi,
>>  I have received no answer to my former question
>>(see http://marc.theaimsgroup.com/?l=linux-kernel&m=110827143716215&w=2).
> 
> 
> That's because it's a BIOS problem.
> 
> There are limits on how much Linux can work around BIOS breakage.
> 
> 
> 
>>  Although I've not re-tested this today again, it used to help a bit to specify
>>mem=3548M to decrease memory used by linux (tested with AGP card plugged in, when
>>bios reported 3556MB RAM only).
>>
>>  I found that removing the AGP based videoc card and using an old PCI based
>>video card results in bios detecting 4072MB of RAM. But still, the machine was
>>slow. I've tried to "cat >| /proc/mtrr" to alter the memory settings, but the
>>result was only a partial speedup.
>>
>>  I'm not sure how to convince linux kernel to run fast again.
> 
> 
> It's most likely a MTRR problem. Play more with them.
> 
> 
> 
>>  Finally, I put back two 512MB memory modules to have only 3GB RAM physically,
>>and the result is at http://www.natur.cuni.cz/~mmokrejs/tmp/128MB/only_phys_3GB/.
> 
> 
> 
> The cheaper Intel chipsets don't support >4GB at all, and you always
> need some space below 4GB for PCI memory mappings/AGP aperture etc.
> 
> 
> 
>>  About a week ago I tried to contact ASUS, but no answer so far from their
>>techinical support through some web robot.
>>http://vip.asus.com/eservice/techmailstatus.aspx?ID=WTM200502111723398547
>>I do not recommend their "greatest" and real "flag-ship" P4C800-E-Deluxe
>>motherboard for use with memory sizes above 3GB (although they claim 4GB
>>is possible). BIOS is the latest release 1.19, although 1.20.001 was tested
>>as well.
> 
> 
> In general non server boards tend to be not very well or not at all
> tested with a lot of memory ("a lot" is defined as >2GB for higher end
> desktop boards, or >1GB on very cheap desktop boards). That is a
> common problem on other motherboards too; Asus is not alone with this.
> 
> -Andi
> 
>