Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Robert Hancock]

Rajkumar S wrote:
> On 10/24/07, Robert Hancock <hancockr@shaw.ca> wrote:
>> Rajkumar S wrote:
>>> Hello,
>>>
>>> I am using a Core 2 Duo E6750 CPU on an intel DG33FB mother board with
>>> 4GB Ram, running Debian Lenny.
>>>
>>> Since the box has 4 GB ram I compiled a big mem kernel, but the
>>> machine is very slow while running big mem kernel. It takes about 37
>>> minutes to compile the intel e1000 driver  (e1000-7.6.5.tar.gz) from
>>> intel site. But it's performing normally when using a non big mem
>>> kernel. The diff of the .config between working and non working is as
>>> follows.
>> Post your contents of /proc/mtrr. Likely a BIOS bug which has been seen
>> on a number of Intel boards, which doesn't mark all of RAM as cachable.
> 
> I have upgraded the bios to latest  (v. 0293 October 02, 2007)
> Previously the /proc/mtrr was:
> 
> ravanan:~# cat /proc/mtrr
> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1
> reg03: base=0xcf800000 (3320MB), size=   8MB: uncachable, count=1
> reg04: base=0xcf600000 (3318MB), size=   2MB: uncachable, count=1
> reg05: base=0xcf500000 (3317MB), size=   1MB: uncachable, count=1
> reg06: base=0x100000000 (4096MB), size= 512MB: write-back, count=1
> reg07: base=0x120000000 (4608MB), size= 128MB: write-back, count=1
> 
> Now after upgrading the bios it's
> 
> reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
> reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
> reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1
> reg03: base=0xcf800000 (3320MB), size=   8MB: uncachable, count=1
> reg04: base=0xcf400000 (3316MB), size=   4MB: uncachable, count=1
> reg05: base=0x100000000 (4096MB), size= 512MB: write-back, count=1
> reg06: base=0x120000000 (4608MB), size= 128MB: write-back, count=1

Yup, it's a BIOS bug. Your BIOS only marks ram up to physical address of 
4736MB as cacheable, while the actual RAM reported by the BIOS goes up 
to physical address 4800MB.

I think we had a patch in -mm to detect this case and disable the extra 
memory (64MB in this case) to keep the kernel from using it.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Robert Hancock]

Rajkumar S wrote:
> Hello,
> 
> I am using a Core 2 Duo E6750 CPU on an intel DG33FB mother board with
> 4GB Ram, running Debian Lenny.
> 
> Since the box has 4 GB ram I compiled a big mem kernel, but the
> machine is very slow while running big mem kernel. It takes about 37
> minutes to compile the intel e1000 driver  (e1000-7.6.5.tar.gz) from
> intel site. But it's performing normally when using a non big mem
> kernel. The diff of the .config between working and non working is as
> follows.

Post your contents of /proc/mtrr. Likely a BIOS bug which has been seen 
on a number of Intel boards, which doesn't mark all of RAM as cachable. 
When the top memory starts being used with the bigmem kernel it causes a 
major slowdown. Check for a BIOS update from Intel, first.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Rajkumar S]

On 10/24/07, Robert Hancock <hancockr@shaw.ca> wrote:
> Rajkumar S wrote:
> > Hello,
> >
> > I am using a Core 2 Duo E6750 CPU on an intel DG33FB mother board with
> > 4GB Ram, running Debian Lenny.
> >
> > Since the box has 4 GB ram I compiled a big mem kernel, but the
> > machine is very slow while running big mem kernel. It takes about 37
> > minutes to compile the intel e1000 driver  (e1000-7.6.5.tar.gz) from
> > intel site. But it's performing normally when using a non big mem
> > kernel. The diff of the .config between working and non working is as
> > follows.
>
> Post your contents of /proc/mtrr. Likely a BIOS bug which has been seen
> on a number of Intel boards, which doesn't mark all of RAM as cachable.

I have upgraded the bios to latest  (v. 0293 October 02, 2007)
Previously the /proc/mtrr was:

ravanan:~# cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1
reg03: base=0xcf800000 (3320MB), size=   8MB: uncachable, count=1
reg04: base=0xcf600000 (3318MB), size=   2MB: uncachable, count=1
reg05: base=0xcf500000 (3317MB), size=   1MB: uncachable, count=1
reg06: base=0x100000000 (4096MB), size= 512MB: write-back, count=1
reg07: base=0x120000000 (4608MB), size= 128MB: write-back, count=1

Now after upgrading the bios it's

reg00: base=0x00000000 (   0MB), size=2048MB: write-back, count=1
reg01: base=0x80000000 (2048MB), size=1024MB: write-back, count=1
reg02: base=0xc0000000 (3072MB), size= 256MB: write-back, count=1
reg03: base=0xcf800000 (3320MB), size=   8MB: uncachable, count=1
reg04: base=0xcf400000 (3316MB), size=   4MB: uncachable, count=1
reg05: base=0x100000000 (4096MB), size= 512MB: write-back, count=1
reg06: base=0x120000000 (4608MB), size= 128MB: write-back, count=1

raj

HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Rajkumar S]

Hello,

I am using a Core 2 Duo E6750 CPU on an intel DG33FB mother board with
4GB Ram, running Debian Lenny.

Since the box has 4 GB ram I compiled a big mem kernel, but the
machine is very slow while running big mem kernel. It takes about 37
minutes to compile the intel e1000 driver  (e1000-7.6.5.tar.gz) from
intel site. But it's performing normally when using a non big mem
kernel. The diff of the .config between working and non working is as
follows.

--- config-nobigmem     2007-10-24 12:51:42.000000000 +0530
+++ config-bigmem       2007-10-24 12:48:56.000000000 +0530
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.23.1
-# Wed Oct 24 12:50:21 2007
+# Tue Oct 23 18:08:30 2007
 #
 CONFIG_X86_32=y
 CONFIG_GENERIC_TIME=y
@@ -189,10 +189,11 @@
 CONFIG_DELL_RBU=m
 CONFIG_DCDBAS=m
 CONFIG_DMIID=y
-CONFIG_NOHIGHMEM=y
+# CONFIG_NOHIGHMEM is not set
 # CONFIG_HIGHMEM4G is not set
-# CONFIG_HIGHMEM64G is not set
+CONFIG_HIGHMEM64G=y
 CONFIG_PAGE_OFFSET=0xC0000000
+CONFIG_HIGHMEM=y
 CONFIG_X86_PAE=y
 CONFIG_ARCH_FLATMEM_ENABLE=y
 CONFIG_ARCH_SPARSEMEM_ENABLE=y
@@ -211,6 +212,7 @@
 CONFIG_BOUNCE=y
 CONFIG_NR_QUICK=1
 CONFIG_VIRT_TO_BUS=y
+# CONFIG_HIGHPTE is not set
 # CONFIG_MATH_EMULATION is not set
 CONFIG_MTRR=y
 CONFIG_EFI=y
@@ -223,11 +225,13 @@
 # CONFIG_HZ_1000 is not set
 CONFIG_HZ=250
 CONFIG_KEXEC=y
+# CONFIG_CRASH_DUMP is not set
 CONFIG_PHYSICAL_START=0x100000
 # CONFIG_RELOCATABLE is not set
 CONFIG_PHYSICAL_ALIGN=0x100000
 CONFIG_HOTPLUG_CPU=y
 CONFIG_COMPAT_VDSO=y
+CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

 #
 # Power management options (ACPI, APM)
@@ -1283,6 +1287,7 @@
 CONFIG_I2O=m
 CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
 CONFIG_I2O_EXT_ADAPTEC=y
+CONFIG_I2O_EXT_ADAPTEC_DMA64=y
 CONFIG_I2O_CONFIG=m
 CONFIG_I2O_CONFIG_OLD_IOCTL=y
 CONFIG_I2O_BUS=m
@@ -3323,6 +3328,7 @@
 # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
 # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
 # CONFIG_DEBUG_KOBJECT is not set
+# CONFIG_DEBUG_HIGHMEM is not set
 CONFIG_DEBUG_BUGVERBOSE=y
 # CONFIG_DEBUG_INFO is not set
 # CONFIG_DEBUG_VM is not set

I was having this same issue with debian kernel also (2.6.22-2-686),
ie stock kernel will work, but big mem is slow. Which is why I tried
latest kernel.

I am posting the full .config at http://pastebin.ca/747758 dmesg while
booting big mem kernel is at http://pastebin.ca/747766

raj

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Boaz Harrosh]

On Wed, Oct 24 2007 at 12:26 +0200, "Rajkumar S" <rajkumars@gmail.com> wrote:
> Hello,
> 
> I am using a Core 2 Duo E6750 CPU on an intel DG33FB mother board with
> 4GB Ram, running Debian Lenny.
> 
> Since the box has 4 GB ram I compiled a big mem kernel, but the
> machine is very slow while running big mem kernel. It takes about 37
> minutes to compile the intel e1000 driver  (e1000-7.6.5.tar.gz) from
> intel site. But it's performing normally when using a non big mem
> kernel. The diff of the .config between working and non working is as
> follows.
> 
> --- config-nobigmem     2007-10-24 12:51:42.000000000 +0530
> +++ config-bigmem       2007-10-24 12:48:56.000000000 +0530
> @@ -1,7 +1,7 @@
>  #
>  # Automatically generated make config: don't edit
>  # Linux kernel version: 2.6.23.1
> -# Wed Oct 24 12:50:21 2007
> +# Tue Oct 23 18:08:30 2007
>  #
>  CONFIG_X86_32=y
>  CONFIG_GENERIC_TIME=y
> @@ -189,10 +189,11 @@
>  CONFIG_DELL_RBU=m
>  CONFIG_DCDBAS=m
>  CONFIG_DMIID=y
> -CONFIG_NOHIGHMEM=y
> +# CONFIG_NOHIGHMEM is not set
>  # CONFIG_HIGHMEM4G is not set
> -# CONFIG_HIGHMEM64G is not set
> +CONFIG_HIGHMEM64G=y
>  CONFIG_PAGE_OFFSET=0xC0000000
> +CONFIG_HIGHMEM=y
>  CONFIG_X86_PAE=y
>  CONFIG_ARCH_FLATMEM_ENABLE=y
>  CONFIG_ARCH_SPARSEMEM_ENABLE=y
> @@ -211,6 +212,7 @@
>  CONFIG_BOUNCE=y
>  CONFIG_NR_QUICK=1
>  CONFIG_VIRT_TO_BUS=y
> +# CONFIG_HIGHPTE is not set
>  # CONFIG_MATH_EMULATION is not set
>  CONFIG_MTRR=y
>  CONFIG_EFI=y
> @@ -223,11 +225,13 @@
>  # CONFIG_HZ_1000 is not set
>  CONFIG_HZ=250
>  CONFIG_KEXEC=y
> +# CONFIG_CRASH_DUMP is not set
>  CONFIG_PHYSICAL_START=0x100000
>  # CONFIG_RELOCATABLE is not set
>  CONFIG_PHYSICAL_ALIGN=0x100000
>  CONFIG_HOTPLUG_CPU=y
>  CONFIG_COMPAT_VDSO=y
> +CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> 
>  #
>  # Power management options (ACPI, APM)
> @@ -1283,6 +1287,7 @@
>  CONFIG_I2O=m
>  CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
>  CONFIG_I2O_EXT_ADAPTEC=y
> +CONFIG_I2O_EXT_ADAPTEC_DMA64=y
>  CONFIG_I2O_CONFIG=m
>  CONFIG_I2O_CONFIG_OLD_IOCTL=y
>  CONFIG_I2O_BUS=m
> @@ -3323,6 +3328,7 @@
>  # CONFIG_DEBUG_SPINLOCK_SLEEP is not set
>  # CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
>  # CONFIG_DEBUG_KOBJECT is not set
> +# CONFIG_DEBUG_HIGHMEM is not set
>  CONFIG_DEBUG_BUGVERBOSE=y
>  # CONFIG_DEBUG_INFO is not set
>  # CONFIG_DEBUG_VM is not set
> 
> I was having this same issue with debian kernel also (2.6.22-2-686),
> ie stock kernel will work, but big mem is slow. Which is why I tried
> latest kernel.
> 
> I am posting the full .config at http://pastebin.ca/747758 dmesg while
> booting big mem kernel is at http://pastebin.ca/747766
> 
> raj

I thought that for 4GB of memory the CONFIG_HIGHMEM4G=y CONFIG_HIGHMEM64G is not set
would be enough.

>From looking in source code the CONFIG_HIGHMEM64G is broken in many respects,
in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even
run on None 64-bit machines? Not many! 

You have a magnificent 64-bit Machine, why?

Boaz

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Lennart Sorensen]

On Wed, Oct 24, 2007 at 04:41:16PM +0200, Boaz Harrosh wrote:
> I thought that for 4GB of memory the CONFIG_HIGHMEM4G=y CONFIG_HIGHMEM64G is not set
> would be enough.
> 
> From looking in source code the CONFIG_HIGHMEM64G is broken in many respects,
> in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even
> run on None 64-bit machines? Not many! 
> 
> You have a magnificent 64-bit Machine, why?

CONFIG_HIGHMEM4G means 4GB address space.  Since PCI and BIOS use some
of the first 4GB address space, a machine with 4GB of actual RAM must
map some of that RAM above the 4GB address space, so you access it all
you need to enable PAE (address space extensions intel invented to give
36bit addressing on 32bit machines), which lets the OS map memory above
4GB (up to 64GB), to let applications use it.  Applications are still
limited to a 32bit address space.  64bit machines of course just use all
memory directly without any silly extensions.

The common intel BIOS bug relating to configuring the MTRR and such for
caching of memory makes the top portion of memory uncached and hence
very slow, which means any code placed in the top part of memory gets
slow.  The kernel likes to place itself at the top of memory which means
the kernel gets slow and hence the whole system gets slow.  Holefully
intel will learn to make proper BIOSes real soon so that people with
intel boards can stop having such slow machines when they have lots of
ram.

--
Len Sorensen

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Boaz Harrosh]

On Wed, Oct 24 2007 at 18:35 +0200, lsorense@csclub.uwaterloo.ca (Lennart Sorensen) wrote:
> On Wed, Oct 24, 2007 at 04:41:16PM +0200, Boaz Harrosh wrote:
>> I thought that for 4GB of memory the CONFIG_HIGHMEM4G=y CONFIG_HIGHMEM64G is not set
>> would be enough.
>>
>> From looking in source code the CONFIG_HIGHMEM64G is broken in many respects,
>> in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even
>> run on None 64-bit machines? Not many! 
>>
>> You have a magnificent 64-bit Machine, why?
> 
> CONFIG_HIGHMEM4G means 4GB address space.  Since PCI and BIOS use some
> of the first 4GB address space, a machine with 4GB of actual RAM must
> map some of that RAM above the 4GB address space, so you access it all
> you need to enable PAE (address space extensions intel invented to give
> 36bit addressing on 32bit machines), which lets the OS map memory above
> 4GB (up to 64GB), to let applications use it.  Applications are still
> limited to a 32bit address space.  64bit machines of course just use all
> memory directly without any silly extensions.
> 
> The common intel BIOS bug relating to configuring the MTRR and such for
> caching of memory makes the top portion of memory uncached and hence
> very slow, which means any code placed in the top part of memory gets
> slow.  The kernel likes to place itself at the top of memory which means
> the kernel gets slow and hence the whole system gets slow.  Holefully
> intel will learn to make proper BIOSes real soon so that people with
> intel boards can stop having such slow machines when they have lots of
> ram.
> 
> --
> Len Sorensen

So one thing I don't understand is the difference between CONFIG_HIGHMEM4G
and regular 32-bit. I always thought that 32 bit means 4GB address space
and that the HIGHMEM4G is using your above trick but with 32-bit dma_addr_t
since there is only actual 4GB of real memory and the rest is mapping tricks.
But I guess I was wrong.

The other point I was making is why use 36 bit trickery when you have a machine
that is capable of the full 64-bit, and runs much faster at that? Just a waste of
a good machine, wont you say? Perhaps we learn to use x86_64 ARCHs before Intel
fixes their BIOS, and they where right all along?

Boaz

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[Lennart Sorensen]

On Wed, Oct 24, 2007 at 07:56:12PM +0200, Boaz Harrosh wrote:
> So one thing I don't understand is the difference between CONFIG_HIGHMEM4G
> and regular 32-bit. I always thought that 32 bit means 4GB address space
> and that the HIGHMEM4G is using your above trick but with 32-bit dma_addr_t
> since there is only actual 4GB of real memory and the rest is mapping tricks.
> But I guess I was wrong.

For efficiency reasons the kernel only supports about 900MB of ram on
i386, unless you enable the support for a full 4GB address space which
is what CONFIG_HIGHMEM4G does.  What exactly changes in the memory
management and mapping I am not sure of, but it made some difference.
It only gives 4GB of address space and does not enable PAE, which means
if the system has more ram than it can map in the first 4GB of address
space, then the remaining ram is simply invisible to the system.  So you
might have 4GB ram, and only see 3GB or 2.5GB of it with a 4GB kernel.
If you want the kernel to have access to the rest of the ram then you
need a kernel capable of addressing physical ram at addresses higher
than 4GB, which means either PAE (CONFIG_HIGHMEM64G) or a 64bit kernel.
I believe the comments for the CONFIG_HIGHMEM* options was recently
updated to me correct rather than the rather misleading messages they
have had for years.  Unfortunately it would appear those nice updates
have not gone tp Linus' tree so we are still stuck with the old awful
descriptions.  I wonder what happened to them.

> The other point I was making is why use 36 bit trickery when you have a machine
> that is capable of the full 64-bit, and runs much faster at that? Just a waste of
> a good machine, wont you say? Perhaps we learn to use x86_64 ARCHs before Intel
> fixes their BIOS, and they where right all along?

Because the 36bit trickery works with an i386 system and kernel.  Not
everyone wants to run 64bit and deal with the (few) programs that don't
yet work on 64bit systems.

Personally I run a 64bit kernel with a 32bit user space on any 64bit
capable machines I deal with.  That seems to work fine, and it lets me
play with 64bit programs in a 64bit chroot when I want to.

--
Len Sorensen

Re: HIGHMEM64G Kernel (2.6.23.1) makes system crawl

[H. Peter Anvin]

Boaz Harrosh wrote:
> 
> From looking in source code the CONFIG_HIGHMEM64G is broken in many respects,
> in my opinion. It's really a 64-bit with 32-bit quirks enabled. Does it even
> run on None 64-bit machines? Not many! 
> 

Total and utter bullsh*t.  It runs on virtually all modern x86's since 
the P6/K7 (certain Pentium M chips being an exception).

Also, it is required for NX protection and for several of the paravirt 
environments to work.

	-hpa