Maximum Physical Memory on 2.4 and ia32

Maximum Physical Memory on 2.4 and ia32

[Stephen Rothwell]

Hi all,

This may be a FAQ (but I did search).

Given this statement by RedHat:

"RAM Limitations on IA32

Red Hat Linux releases based on the 2.4 kernel -- including Red Hat Linux
7.1, 7.2, 7.3 and Red Hat Linux Advanced Server 2.1 -- support a maximum
of 16GB of RAM. Previous product announcements from Red Hat suggested that
Red Hat Linux 7.1 (and by extension, other releases based on the 2.4
kernel) supported up to 64GB of RAM. A more accurate statement is the
2.4-based kernels included in Red Hat Linux 7.1, 7.2, 7.3 and Red Hat
Linux Advanced Server 2.1 support the hardware extensions that support up
to 64GB of RAM. This is an important distinction: while the hardware will
indeed support up to 64GB of physical memory, the operating system design
limits the supported physical memory to approximately 16GB."

(http://www.redhat.com/services/techsupport/production/GSS_caveat.html)
What are the "operating system design limits" that restrict the amount of
supported memory to 16GB?
-- 
Cheers, Stephen Rothwell                   sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

Re: Maximum Physical Memory on 2.4 and ia32

[Andrew Morton]

Stephen Rothwell wrote:
> 
> Hi all,
> 
> This may be a FAQ (but I did search).
> 
> Given this statement by RedHat:
> 
> "RAM Limitations on IA32
> 
> Red Hat Linux releases based on the 2.4 kernel -- including Red Hat Linux
> 7.1, 7.2, 7.3 and Red Hat Linux Advanced Server 2.1 -- support a maximum
> of 16GB of RAM. Previous product announcements from Red Hat suggested that
> Red Hat Linux 7.1 (and by extension, other releases based on the 2.4
> kernel) supported up to 64GB of RAM. A more accurate statement is the
> 2.4-based kernels included in Red Hat Linux 7.1, 7.2, 7.3 and Red Hat
> Linux Advanced Server 2.1 support the hardware extensions that support up
> to 64GB of RAM. This is an important distinction: while the hardware will
> indeed support up to 64GB of physical memory, the operating system design
> limits the supported physical memory to approximately 16GB."
> 
> (http://www.redhat.com/services/techsupport/production/GSS_caveat.html)
> What are the "operating system design limits" that restrict the amount of
> supported memory to 16GB?

It's a practical limit.  The mem_map array alone would consume 720 megabytes,
so you have no normal-zone memory left.

At 16G, mem_map[] consumes 180 megabytes and there's 540ish megabytes
of normal zone left for general use.

Even at this 20:1 highmem:lowmem ratio, the system will be struggling.
Any time you have normal-zone data structures which are pinned by
pages, the maths gets you in the end.

buffer_heads, pagetable pages, radix-tree nodes, pte_chains and inodes
are normal-zone data structures which, depending on the kernel version,
may be pinned into the normal zone by highmem pages.

In 2.5, with ext2's no-buffer-head option, shared pagetables, highpte,
with your fingers crossed and the wind in the south east, 32G might
be practical.

Re: Maximum Physical Memory on 2.4 and ia32

[Martin J. Bligh]

> It's a practical limit.  The mem_map array alone would consume 720 megabytes,
> so you have no normal-zone memory left.
> 
> At 16G, mem_map[] consumes 180 megabytes and there's 540ish megabytes
> of normal zone left for general use.
> 
> Even at this 20:1 highmem:lowmem ratio, the system will be struggling.
> Any time you have normal-zone data structures which are pinned by
> pages, the maths gets you in the end.
> 
> buffer_heads, pagetable pages, radix-tree nodes, pte_chains and inodes
> are normal-zone data structures which, depending on the kernel version,
> may be pinned into the normal zone by highmem pages.
> 
> In 2.5, with ext2's no-buffer-head option, shared pagetables, highpte,
> with your fingers crossed and the wind in the south east, 32G might
> be practical.

32Gb was indeed what we've been working towards for 2.6, and we've 
been running that on some workloads.

However, if you're willing to run with a 2:2 or even a 1:3 user:kernel
split instead of the default 3:1, you can get away with all sorts of
things, probably including 64Gb. I've got the hardware to build such
a beast, but haven't bothered yet (we have enough problems already ;-)).
Big databases won't like it, but other workloads without huge individual
processes (or shared mappings) will be fine.

M.

Re: Maximum Physical Memory on 2.4 and ia32

[William Lee Irwin III]

On Mon, Dec 02, 2002 at 12:08:35PM +1100, Stephen Rothwell wrote:
> (http://www.redhat.com/services/techsupport/production/GSS_caveat.html)
> What are the "operating system design limits" that restrict the amount of
> supported memory to 16GB?

I've relayed this privately in less organized private discussions:

The design limitation is that a PTE is required to map a PAGE_SIZE
-sized region. The kernel allocation unit needs to be enlarged without
breaking ABI to reduce the fraction of physical memory reserved for
metadata i.e. mem_map[]. This is called page clustering.

Hugh Dickins implemented page clustering by creating a distinction
between PAGE_SIZE and MMUPAGE_SIZE then introducing an API for dealing
with vectors of PTE's. He implemented this for 2.4.6 and 2.4.7. His
patches are at ftp://ftp.veritas.com/linux/ In other environments this
is an optimization, not a requirement to be able to run, and so it
benefits all platforms.

There are, of course, other kinds of metadata generated at runtime that
may impose limitations on feasibility of workloads on large highmem boxen.

Bill