General slowness with using 64Gb HIGHMEM option on 32-bit kernel

General slowness with using 64Gb HIGHMEM option on 32-bit kernel

[Sergey Popov]

Short description: after recompiling 32-bit kernel with 64Gb highmem
support and rebooting into it, OS started working very much slower
then before.

Specifications: Intel Core 2 Duo E6600@2,4Ghz on Intel i965Q-based
motherboard. 6Gb of DDR2 RAM, 2x1Gb+2x2Gb, dual-channel.
Vector Linux 5.8 (Slackware Linux 11-based), 2.6.20.3 kernel.

Long description: after recompiling 32-bit 2.6.20.3 kernel with 64Gb
highmem option and rebooting into it, booting process froze on udevd.
After booting up from CD and disabling rc.udev, system started, but
it's work after rc.M script sarted was 5-10 times slower then usual.
top showed, that processes are taking much more CPU% time then usual.
I tried downloading the newest stable kernel - 2.6.22.9 - hoping, that
it would solve the problem. I downloaded it, booted from CD, and
compiled it with 64Gb highmem support. I felt some speed improvement,
but if on 2.6.20.3 machine performed like P166, now it is working like
P2-350.
I've found a similar problem, described on lkml -
http://lkml.org/lkml/2007/5/30/5 . But, unfortunately, it is x86_64
related and contains no clues for me.

Moving to 64-bit distro is rather unpleasant - as there is an option
in 32-bit kernel, it should work ;) .

What tests should I run to help investigate this problem?

Re: General slowness with using 64Gb HIGHMEM option on 32-bit kernel

[Nick Piggin]

On Saturday 29 September 2007 03:15, Sergey Popov wrote:
> Short description: after recompiling 32-bit kernel with 64Gb highmem
> support and rebooting into it, OS started working very much slower
> then before.
>
> Specifications: Intel Core 2 Duo E6600@2,4Ghz on Intel i965Q-based
> motherboard. 6Gb of DDR2 RAM, 2x1Gb+2x2Gb, dual-channel.
> Vector Linux 5.8 (Slackware Linux 11-based), 2.6.20.3 kernel.
>
> Long description: after recompiling 32-bit 2.6.20.3 kernel with 64Gb
> highmem option and rebooting into it, booting process froze on udevd.
> After booting up from CD and disabling rc.udev, system started, but
> it's work after rc.M script sarted was 5-10 times slower then usual.
> top showed, that processes are taking much more CPU% time then usual.
> I tried downloading the newest stable kernel - 2.6.22.9 - hoping, that
> it would solve the problem. I downloaded it, booted from CD, and
> compiled it with 64Gb highmem support. I felt some speed improvement,
> but if on 2.6.20.3 machine performed like P166, now it is working like
> P2-350.
> I've found a similar problem, described on lkml -
> http://lkml.org/lkml/2007/5/30/5 . But, unfortunately, it is x86_64
> related and contains no clues for me.

It's most likely the MTRR problem that Rik mentioned.


> Moving to 64-bit distro is rather unpleasant - as there is an option
> in 32-bit kernel, it should work ;) .

If you have a gcc that can compile 64-bit code, then I found it was as
simple as doing a 'make ARCH=x86_64' and building the kernel with 32-bit
program support.

Re: General slowness with using 64Gb HIGHMEM option on 32-bit kernel

[Chuck Ebbert]

On 09/28/2007 01:15 PM, Sergey Popov wrote:
> Short description: after recompiling 32-bit kernel with 64Gb highmem
> support and rebooting into it, OS started working very much slower
> then before.
> 
> Specifications: Intel Core 2 Duo E6600@2,4Ghz on Intel i965Q-based
> motherboard. 6Gb of DDR2 RAM, 2x1Gb+2x2Gb, dual-channel.
> Vector Linux 5.8 (Slackware Linux 11-based), 2.6.20.3 kernel.
> 
> Long description: after recompiling 32-bit 2.6.20.3 kernel with 64Gb
> highmem option and rebooting into it, booting process froze on udevd.
> After booting up from CD and disabling rc.udev, system started, but
> it's work after rc.M script sarted was 5-10 times slower then usual.
> top showed, that processes are taking much more CPU% time then usual.
> I tried downloading the newest stable kernel - 2.6.22.9 - hoping, that
> it would solve the problem. I downloaded it, booted from CD, and
> compiled it with 64Gb highmem support. I felt some speed improvement,
> but if on 2.6.20.3 machine performed like P166, now it is working like
> P2-350.
> I've found a similar problem, described on lkml -
> http://lkml.org/lkml/2007/5/30/5 . But, unfortunately, it is x86_64
> related and contains no clues for me.
> 
> Moving to 64-bit distro is rather unpleasant - as there is an option
> in 32-bit kernel, it should work ;) .
> 
> What tests should I run to help investigate this problem?

Boot with option "mem=3900M".

Re: General slowness with using 64Gb HIGHMEM option on 32-bit kernel

[Sergey Popov]

Ok, I found a palliative.

Now I'm working with "mem=6770M" option, that gives me 6072MB RAM (of
6144 total).
Interesting, that if you choose too much, like "mem=6800M", system
behaviour is exactly the same as without any options.

Can it be that kernel incorrectly sets amount of memory to use so such
a situation occurs?

Re: General slowness with using 64Gb HIGHMEM option on 32-bit kernel

[Rik van Riel]

On Fri, 28 Sep 2007 21:57:55 +0400
"Sergey Popov" <faijeya@gmail.com> wrote:

> Ok, I found a palliative.
> 
> Now I'm working with "mem=6770M" option, that gives me 6072MB RAM (of
> 6144 total).
> Interesting, that if you choose too much, like "mem=6800M", system
> behaviour is exactly the same as without any options.
> 
> Can it be that kernel incorrectly sets amount of memory to use so such
> a situation occurs?

Maybe the BIOS marks the top megabytes of memory as
uncacheable, which makes accesses ridiculously slow.

Cat /proc/mtrr to check.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: General slowness with using 64Gb HIGHMEM option on 32-bit kernel

[Robert Hancock]

Sergey Popov wrote:
> Short description: after recompiling 32-bit kernel with 64Gb highmem
> support and rebooting into it, OS started working very much slower
> then before.
> 
> Specifications: Intel Core 2 Duo E6600@2,4Ghz on Intel i965Q-based
> motherboard. 6Gb of DDR2 RAM, 2x1Gb+2x2Gb, dual-channel.
> Vector Linux 5.8 (Slackware Linux 11-based), 2.6.20.3 kernel.
> 
> Long description: after recompiling 32-bit 2.6.20.3 kernel with 64Gb
> highmem option and rebooting into it, booting process froze on udevd.
> After booting up from CD and disabling rc.udev, system started, but
> it's work after rc.M script sarted was 5-10 times slower then usual.
> top showed, that processes are taking much more CPU% time then usual.
> I tried downloading the newest stable kernel - 2.6.22.9 - hoping, that
> it would solve the problem. I downloaded it, booted from CD, and
> compiled it with 64Gb highmem support. I felt some speed improvement,
> but if on 2.6.20.3 machine performed like P166, now it is working like
> P2-350.
> I've found a similar problem, described on lkml -
> http://lkml.org/lkml/2007/5/30/5 . But, unfortunately, it is x86_64
> related and contains no clues for me.
> 
> Moving to 64-bit distro is rather unpleasant - as there is an option
> in 32-bit kernel, it should work ;) .
> 
> What tests should I run to help investigate this problem?

Make sure your BIOS is up to date. If so, post your dmesg output and the 
contents of /proc/mtrr. There are a number of Intel boards which 
have/had BIOS bugs where the BIOS didn't set up the MTRRs to mark all 
RAM as write-back. This results in the last bit of memory being 
uncacheable and causes a major performance drop.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/