Allocating more than 890MB in the kernel?

Allocating more than 890MB in the kernel?

[Timur Tabi]

vmalloc() fails after about 890MB because the kernel memory map is only for 
about 1GB.  I know there are some hacks and work-arounds to get more than 
that, but instead of reinventing the wheel, I was hoping some kind soul would 
tell me how (a few hints would be nice!)

The reason we use vmalloc() is because we need to apply memory pressure during 
the allocating: memory should be swapped out to make room for our allocation.

We're trying to allocate up to 3GB on a 4GB machine.  Thanks in advance!

Re: Allocating more than 890MB in the kernel?

[John Tyner]

Isn't this solved by just recompiling the kernel with HIGHMEM support?

John

On Fri, 19 Oct 2001, Timur Tabi wrote:

> vmalloc() fails after about 890MB because the kernel memory map is only for
> about 1GB.  I know there are some hacks and work-arounds to get more than
> that, but instead of reinventing the wheel, I was hoping some kind soul would
> tell me how (a few hints would be nice!)
>
> The reason we use vmalloc() is because we need to apply memory pressure during
> the allocating: memory should be swapped out to make room for our allocation.
>
> We're trying to allocate up to 3GB on a 4GB machine.  Thanks in advance!
>
> -
> Kernelnewbies: Help each other learn about the Linux kernel.
> Archive:       http://mail.nl.linux.org/kernelnewbies/
> IRC Channel:   irc.openprojects.net / #kernelnewbies
> Web Page:      http://www.kernelnewbies.org/
>

-- 
John Tyner
jtyner@cs.ucr.edu

Re: Allocating more than 890MB in the kernel?

[Timur Tabi]

John Tyner wrote:

> Isn't this solved by just recompiling the kernel with HIGHMEM support?


I don't think so.  The Red Hat 7.1 kernel is compiled with "4GB" support, 
which apparently is the same as HIGHMEM.  We see the 890MB kernel vmalloc 
limit still.

Re: Allocating more than 890MB in the kernel?

[H. Peter Anvin]

Followup to:  <3BD08207.7090807@interactivesi.com>
By author:    Timur Tabi <ttabi@interactivesi.com>
In newsgroup: linux.dev.kernel
> 
> > Isn't this solved by just recompiling the kernel with HIGHMEM support?
> 
> 
> I don't think so.  The Red Hat 7.1 kernel is compiled with "4GB" support, 
> which apparently is the same as HIGHMEM.  We see the 890MB kernel vmalloc 
> limit still.
> 

That's because you're running out of address space, not memory.
HIGHMEM doesn't do anything for the latter -- it can't.  You start
running into a lot of fundamental problems when your memory size gets
in the same (or higher) ballpark than your address space.

The best solution is go buy a 64-bit CPU.  There isn't much else you
can do about it.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Allocating more than 890MB in the kernel?

[Timur Tabi]

H. Peter Anvin wrote:

> That's because you're running out of address space, not memory.
> HIGHMEM doesn't do anything for the latter -- it can't.  You start
> running into a lot of fundamental problems when your memory size gets
> in the same (or higher) ballpark than your address space.
> 
> The best solution is go buy a 64-bit CPU.  There isn't much else you
> can do about it.


That's completely missing the point of my request (which, I admit, I didn't 
make clear).  I need to allocate about 3/4 of available memory in the kernel. 
  If I had 2GB of RAM, I'd need to allocate 1.5GB.  If I had 8 GB of RAM, I'd 
need to allocate 6GB.  I just used 3GB/4GB because it's our current test platform.

Re: Allocating more than 890MB in the kernel?

[H. Peter Anvin]

Timur Tabi wrote:

> 
> That's completely missing the point of my request (which, I admit, I
> didn't make clear).  I need to allocate about 3/4 of available memory in
> the kernel.  If I had 2GB of RAM, I'd need to allocate 1.5GB.  If I had
> 8 GB of RAM, I'd need to allocate 6GB.  I just used 3GB/4GB because it's
> our current test platform.
> 


There is no way you can allocate 6 GB of address space when you have 4 GB
to dole out -- and that includes to userspace.  Linux tends to allocate
most of the address space (usually 3 GB) to userspace, because it can be
re-used between processes and it matches the needs of more users.

This puts fundamental limits on how much space is addressible in the
kernel.  What you can do if your application permits is allocate HIGHMEM
pages, and use kmap()/kmap_atomic()/kunmap() to selectively bring them
into the address space on an as-need basis.  THIS IS EXPENSIVE, have no
illusions about it, and doesn't give you space that is contiguous in
either linear nor physical space.

Obviously, on a 64-bit CPU these limitations utterly vanish, since address
space is no longer limited.

	-hpa

Re: Allocating more than 890MB in the kernel?

[Rob Landley]

On Friday 19 October 2001 16:21, Timur Tabi wrote:
> H. Peter Anvin wrote:
> > That's because you're running out of address space, not memory.
> > HIGHMEM doesn't do anything for the latter -- it can't.  You start
> > running into a lot of fundamental problems when your memory size gets
> > in the same (or higher) ballpark than your address space.
> >
> > The best solution is go buy a 64-bit CPU.  There isn't much else you
> > can do about it.
>
> That's completely missing the point of my request (which, I admit, I didn't
> make clear).  I need to allocate about 3/4 of available memory in the
> kernel. If I had 2GB of RAM, I'd need to allocate 1.5GB.  If I had 8 GB of
> RAM, I'd need to allocate 6GB.  I just used 3GB/4GB because it's our
> current test platform.

Each user process has 32 bit pointers for memory.  This means they only have 
4 gigabytes of virtual address space, regardless of how many physical pages 
the machine has.  The kernel doesn't use segment:offset addressing.  It just 
uses the offset.  Flat memory model.

Now your page tables can shuffle memory around (using physical pages, swap, 
etc), but that's irrelevant.  You've still only got 4 gigs of virtual space 
per process to work with on a 32 bit system.  The register you're ultimately 
dereferencing to access memory isn't big enough to hold a number larger than 
that.

To simplify the page tables, the kernel is in shared memory.  The first 
gigabyte of every process's address space is the kernel.  The kernel's page 
table is sort of spliced into everybody's page table.  And there was much 
rejoicing.

The problem is, you're trying to allocate kernel memory, meaning you hit the 
1 gigabyte limit.  And that includes the kernel itself.  You can recompile so 
the kernel has more than 1 gig of everybody's virtual address range reserved 
to it, but that reduces the amount of virtual memory user space processes 
have (4 gigs - 1 gig) because the kernel space (shared memory) is mapped into 
each and every process so the process can access kernel resources easily.  
(You do NOT want to change page tables for every system call.  We're not 
going there.)

To allocate more memory than that, you need to allocate pages in user space.  
To allocate more than 4 gigabytes of memory, you MUST use more than one 
process (you only have 4 gigs of virtual address space per process, so two 
processes must use overlapping virtual address ranges to point to different 
pages.  Perhaps they could communicate with each other through a shared mmap. 
 Congratulations, you've just reinvented expanded memory.)

Or, we could get back to the "buy a real computer" answer everybody's been 
giving you: buy a 64 bit computer that has more than 4 gigabytes of virtual 
address space per process.  Then you have 64 bit pointers for your virtual 
addresses, and can directly address virtual petabytes or exabytes or whatever 
comes next...

Rob

Re: Allocating more than 890MB in the kernel?

[H. Peter Anvin]

Rob Landley wrote:

> 
> Each user process has 32 bit pointers for memory.  This means they only have 
> 4 gigabytes of virtual address space, regardless of how many physical pages 
> the machine has.  The kernel doesn't use segment:offset addressing.  It just 
> uses the offset.  Flat memory model.
> 


And even if it did (on i386) it wouldn't help... the segment:offset is 
folded into a single 32-bit space before paging.

	-hpa

Re: Allocating more than 890MB in the kernel?

[Albert D. Cahalan]

> I need to allocate about 3/4 of available memory in the kernel. 
> If I had 2GB of RAM, I'd need to allocate 1.5GB.  If I had 8 GB
> of RAM, I'd need to allocate 6GB.  I just used 3GB/4GB because
> it's our current test platform.

The best you can do, IMHO:

1. reserve a 3 GB chunk of memory at boot
2. create a regular user process
3. have that process make a system call which will never return
4. in that system call, wipe out all memory mappings in the process
5. hand-craft a 3 GB memory mapping (0 GB virt --> 1 GB phys)
6. call your desired code, remembering to schedule by hand

Re: Allocating more than 890MB in the kernel?

[Timur Tabi]

Albert D. Cahalan wrote:

> 1. reserve a 3 GB chunk of memory at boot
> 2. create a regular user process
> 3. have that process make a system call which will never return
> 4. in that system call, wipe out all memory mappings in the process
> 5. hand-craft a 3 GB memory mapping (0 GB virt --> 1 GB phys)
> 6. call your desired code, remembering to schedule by hand


Thanks, that's the most useful idea I've gotten.  It's crazy, but it just 
might work!

Too bad no one else on this list can think outside of the box like you just did.