HIMEM calculation

HIMEM calculation

[James C. Georgas]

I'm not sure I understand how the kernel calculates the amount of
physical RAM it can map during the boot process.

I've quoted two blocks of kernel messages below, one for a kernel with
NOHIGHMEM and another for a kernel with HIGHMEM4G.

If I do the math on the BIOS provided physical RAM map, there is less
than 5MiB of the address space reserved. Since I only have 1GiB of
physical RAM in the board, I figured that it would still be possible to
physically map 1019MiB, even with the 3GiB/1GiB split between user space
and kernel space that occurs with NOHIGHMEM.

However, What actually happens is that I'm 127MiB short of a full GiB.

What am I missing here? Why does that last 127MiB have to go in HIGHMEM?

Message log for a NOHIGHMEM kernel:

Sep  3 16:56:50 Tachyon kernel: Linux version 2.6.22-gentoo-r5 (root@Tachyon) (gcc version 4.1.2 (Gentoo 4.1.2 p1.0.1)) #2 PREEMPT Mon Sep 3 16:01:08 EDT 2007
Sep  3 16:56:50 Tachyon kernel: BIOS-provided physical RAM map:
Sep  3 16:56:50 Tachyon kernel:  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Sep  3 16:56:50 Tachyon kernel:  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Sep  3 16:56:50 Tachyon kernel:  BIOS-e820: 0000000000100000 - 000000003ffd3000 (usable)
Sep  3 16:56:50 Tachyon kernel:  BIOS-e820: 000000003ffd3000 - 0000000040000000 (reserved)
Sep  3 16:56:50 Tachyon kernel:  BIOS-e820: 00000000feda0000 - 00000000fee00000 (reserved)
Sep  3 16:56:50 Tachyon kernel:  BIOS-e820: 00000000ffb80000 - 0000000100000000 (reserved)
Sep  3 16:56:50 Tachyon kernel: Warning only 896MB will be used.
Sep  3 16:56:50 Tachyon kernel: Use a HIGHMEM enabled kernel.
Sep  3 16:56:50 Tachyon kernel: 896MB LOWMEM available.
Sep  3 16:56:50 Tachyon kernel: Zone PFN ranges:
Sep  3 16:56:50 Tachyon kernel:   DMA             0 ->     4096
Sep  3 16:56:50 Tachyon kernel:   Normal       4096 ->   229376
Sep  3 16:56:50 Tachyon kernel: early_node_map[1] active PFN ranges
Sep  3 16:56:50 Tachyon kernel:     0:        0 ->   229376
Sep  3 16:56:50 Tachyon kernel: DMI 2.3 present.




Message log for a HIGHMEM4G kernel:

Sep  3 17:40:53 Tachyon kernel: BIOS-provided physical RAM map:
Sep  3 17:40:53 Tachyon kernel:  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Sep  3 17:40:53 Tachyon kernel:  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Sep  3 17:40:53 Tachyon kernel:  BIOS-e820: 0000000000100000 - 000000003ffd3000 (usable)
Sep  3 17:40:53 Tachyon kernel:  BIOS-e820: 000000003ffd3000 - 0000000040000000 (reserved)
Sep  3 17:40:53 Tachyon kernel:  BIOS-e820: 00000000feda0000 - 00000000fee00000 (reserved)
Sep  3 17:40:53 Tachyon kernel:  BIOS-e820: 00000000ffb80000 - 0000000100000000 (reserved)
Sep  3 17:40:53 Tachyon kernel: 127MB HIGHMEM available.
Sep  3 17:40:53 Tachyon kernel: 896MB LOWMEM available.
Sep  3 17:40:53 Tachyon kernel: Zone PFN ranges:
Sep  3 17:40:53 Tachyon kernel:   DMA             0 ->     4096
Sep  3 17:40:53 Tachyon kernel:   Normal       4096 ->   229376
Sep  3 17:40:53 Tachyon kernel:   HighMem    229376 ->   262099
Sep  3 17:40:53 Tachyon kernel: early_node_map[1] active PFN ranges
Sep  3 17:40:53 Tachyon kernel:     0:        0 ->   262099
Sep  3 17:40:53 Tachyon kernel: DMI 2.3 present.

Re: HIMEM calculation

[Chris Snook]

James C. Georgas wrote:
> I'm not sure I understand how the kernel calculates the amount of
> physical RAM it can map during the boot process.
> 
> I've quoted two blocks of kernel messages below, one for a kernel with
> NOHIGHMEM and another for a kernel with HIGHMEM4G.
> 
> If I do the math on the BIOS provided physical RAM map, there is less
> than 5MiB of the address space reserved. Since I only have 1GiB of
> physical RAM in the board, I figured that it would still be possible to
> physically map 1019MiB, even with the 3GiB/1GiB split between user space
> and kernel space that occurs with NOHIGHMEM.
> 
> However, What actually happens is that I'm 127MiB short of a full GiB.
> 
> What am I missing here? Why does that last 127MiB have to go in HIGHMEM?

That's the vmalloc address space.  You only get 896 MB in the NORMAL 
zone on i386, to leave room for vmalloc.  If you don't like it, go 64-bit.

	-- Chris

Re: HIMEM calculation

[Chris Snook]

James Georgas wrote:
>  > That's the vmalloc address space.  You only get 896 MB in the NORMAL
>  > zone on i386, to leave room for vmalloc.  If you don't like it, go 
> 64-bit.
>  >
>  >       -- Chris
> 
> I like it fine. I just didn't understand it. Thanks for answering.
> 
> So, basically, the vmalloc address space is not backed by physical RAM,
> right? Rather, the virtual address space associated with vmalloc is
> mapped to physical pages by page tables?

Basically, yes, but that's an oversimplification.  We actually use page tables 
everywhere, but the conversion is simply +/- 0xC0000000 for the NORMAL zone, so 
we can skip most of the fancy VM work and just use a trivial macro.  vmalloc can 
allocate large chunks of virtually contiguous memory even when the physical 
memory is heavily fragmented, and since we've set aside address space for it, 
it's visible in all process contexts.

vmalloc is handy sometimes because it can complete even if there's no memory 
free when it's called, since the VM will swap out user pages and then return 
those remapped into the vmalloc address space.  Unfortunately, we can't use 
vmalloc anywhere we want to use DMA because it will be accessed without the MMU. 
  Worse, we also can't use it in any path that could be called while trying to 
free memory, due to recursion issues, which substantially limits its utility in 
the kernel.  Some people *cough*OpenAFS*cough* use it carelessly and get all 
kinds of exciting panics under rare and difficult-to-reproduce load conditions.

	-- Chris