increasing the TASK_SIZE

increasing the TASK_SIZE

[Ernest N. Mamikonyan]

I was wondering how I can increase the process address space, TASK_SIZE
(PAGE_OFFSET), in the current kernel. It looks like the 3 GB value is
hardcoded in a couple of places and is thus not trivial to alter. Is
there any good reason to limit this value at all, why not just have it
be the same as the max addressable space (64 GB)? We have an ix86 SMP
box with 4 GB of RAM and want to be able to allocate all of it to a
single program (physics simulation). I would greatly appreciate any help
on this.


Thanks a great deal,
Ernie

PS. Please `CC' me the answer!

/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
Ernest N. Mamikonyan     E-Mail: ernest@newton.physics.drexel.edu
Department of Physics,   Phone: (215) 895-1544
Drexel University        Fax: (215) 895-5934
Philadelphia, PA  19104  Web: www.physics.drexel.edu/research/astro
/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

Re: increasing the TASK_SIZE

[Matti Aarnio]

On Thu, Jul 05, 2001 at 05:21:21PM -0400, Ernest N. Mamikonyan wrote:
> I was wondering how I can increase the process address space, TASK_SIZE
> (PAGE_OFFSET), in the current kernel. It looks like the 3 GB value is
> hardcoded in a couple of places and is thus not trivial to alter. Is
> there any good reason to limit this value at all, why not just have it
> be the same as the max addressable space (64 GB)? We have an ix86 SMP
> box with 4 GB of RAM and want to be able to allocate all of it to a
> single program (physics simulation). I would greatly appreciate any help
> on this.

	It is marginally possible to increase that up so much
	that you get about 3.8-3.9 GB for usermode process.
	(I use k=1024, M=k*k, G=k*k*k)

	It is absolutely impossible to get it into anything above
	the 4.0 GB limit.   This hard limit is buried inside the i386
	(and all of its successors) memory addressing, and mapping
	hardware.  There is a choke-point of 32 address bits along
	the way, which prevents going above 4.0 GB most effectively.

	With considerable infrastructural work(*) it MIGHT be possible
	to go very near the 4.0 GB limit for userspace, but I am not
	an expert here.   The crux is at the supervisor/interrupt mode
	stack memory mapping.  As far as I understand, in i386 we
	must have the supervisor stack (and 'struct task') mapped
	into the same address space as the usermode.  Only the memory
	protection prevents the usermode to access that data.
	Also parts of kernel code must be in that address space + parts
	of kernel data related into MMU control.

	(*) Supervisor (kernel) mode must have the stack, and switch-
	around code + some datasets in its access space when transition
	into the kernel space is done (and reversed).  Accessing user-
	space from kernel can then be done via kmap() (-like) windows.
	Of course this is considerably much slower than the current method
	where each user-space has 1/4 of its total address space allocated
	for kernel internal use.

	To get most out of your box, you need to run your problem as much
	as possible at separate processors and with separate contexts.
	That way you will get most out of your SMP setup.
	(Consider your box as a small Beowulf-cluster.)

	Of course problems where you run e.g. PVM, you will need fast
	communication in between processes, and nothing would beat single
	shared memory space.   You might be able to get that by having
	e.g. SHM segments used for PVM's IPC task.
	Linux doesn't support user semaphores in SHM in scheduling sense,
	though.  You can, of course, do CPU burning spin-locks for shared
	memory area access.  The best would, IMO, be a hybride of using
	SHM for transfering large amounts of data in between processes,
	and something alike PF_UNIX sockets for signaling that there is
	some new data available.

	In _usual_ case you can ignore such details, and use your favourite
	clustering library, like PVM.

> Thanks a great deal,
> Ernie
> 
> PS. Please `CC' me the answer!
> Ernest N. Mamikonyan     E-Mail: ernest@newton.physics.drexel.edu
> Philadelphia, PA  19104  Web: www.physics.drexel.edu/research/astro

/Matti Aarnio

Re: increasing the TASK_SIZE

[Albert D. Cahalan]

Matti Aarnio writes:

> It is absolutely impossible to get it into anything above
> the 4.0 GB limit.   This hard limit is buried inside the i386
> (and all of its successors) memory addressing, and mapping
> hardware.  There is a choke-point of 32 address bits along
> the way, which prevents going above 4.0 GB most effectively.

Is is possible. It just won't perform well. You have to do
a paging-like trick on that choke point. You also need a
modified compiler that does LP64 on x86.

Use 13 bits of the segment register for the top of your address
space. Segments are 512 MB. Use the low 3 GB of the choke point
for greater compatibility with regular apps and kernel code.
Keep all but 6 segments invalid, so that faults can be used to
change the 512-MB windows through the choke point.

So that gets you 4 TB of address space. (8192 segments, each of
which is 512 MB) You may shrink the segments a bit to reduce your
fault rate, but giving up some of that nice address space.
Yes I'm aware that one must move page table chunks and invalidate
stuff while adjusting the segments -- that is part of the fun!

If you actually want arrays larger than 512 MB, well then your
performance gets even worse. The compiler has to do nasty stuff to
normalize your pointers after every pointer arithmetic operation.

Compiler: write a new back end
Libc: redo much of the assembly
Kernel: new fault handler, copy_to_user, copy_from_user...

You might need to use an IP32L64 compiler for the kernel. That
won't be nice for performance, but not too much of a disaster.
Maybe pointers get padding to carry bits through casts.

BTW Ernest, nobody does this. It would likely be just a toy
due to the bad performance. You really ought to get 64-bit
hardware or find some way to use multiple processes. Toys can
be fun though, and this one is nice for cleaning up 32-bit code.

Re: increasing the TASK_SIZE

[Gareth Hughes]

"Ernest N. Mamikonyan" wrote:
> 
> I was wondering how I can increase the process address space, TASK_SIZE
> (PAGE_OFFSET), in the current kernel. It looks like the 3 GB value is
> hardcoded in a couple of places and is thus not trivial to alter. Is
> there any good reason to limit this value at all, why not just have it
> be the same as the max addressable space (64 GB)? We have an ix86 SMP
> box with 4 GB of RAM and want to be able to allocate all of it to a
> single program (physics simulation). I would greatly appreciate any help
> on this.

Sounds like you just need to enable highmem.  Check the help for "High
Memory Support" in "Processor type and features".

-- Gareth

Re: increasing the TASK_SIZE

[Richard B. Johnson]

On Tue, 10 Jul 2001, Gareth Hughes wrote:

> "Ernest N. Mamikonyan" wrote:
> > 
> > I was wondering how I can increase the process address space, TASK_SIZE
> > (PAGE_OFFSET), in the current kernel. It looks like the 3 GB value is
> > hardcoded in a couple of places and is thus not trivial to alter. Is
> > there any good reason to limit this value at all, why not just have it
> > be the same as the max addressable space (64 GB)? We have an ix86 SMP
> > box with 4 GB of RAM and want to be able to allocate all of it to a
> > single program (physics simulation). I would greatly appreciate any help
> > on this.
> 
> Sounds like you just need to enable highmem.  Check the help for "High
> Memory Support" in "Processor type and features".
> 
> -- Gareth

Also, additional memory on an ix86, as specified, can only be accessed
via page registers (like the old DOS himem.sys). This is because
the Intel machines have 32 bits of address-space. That's around
4 GB, not 64 GB. So, if you intend to do conventional, user-space
programming,i (like using malloc) you will not be able to get anything
like 4 GB.


Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

    I was going to compile a list of innovations that could be
    attributed to Microsoft. Once I realized that Ctrl-Alt-Del
    was handled in the BIOS, I found that there aren't any.

Re: increasing the TASK_SIZE

[Rik van Riel]

On Thu, 5 Jul 2001, Ernest N. Mamikonyan wrote:

> I was wondering how I can increase the process address space, TASK_SIZE
> (PAGE_OFFSET), in the current kernel. It looks like the 3 GB value is
> hardcoded in a couple of places and is thus not trivial to alter. Is
> there any good reason to limit this value at all, why not just have it
> be the same as the max addressable space (64 GB)? We have an ix86 SMP
> box with 4 GB of RAM and want to be able to allocate all of it to a
> single program (physics simulation). I would greatly appreciate any help
> on this.

The only way to achieve your goal would be to modify the
hardware. What you want is not possible due to hardware
limitations of the x86 platform.

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: increasing the TASK_SIZE

[Gareth Hughes]

Rik van Riel wrote:
> 
> On Thu, 5 Jul 2001, Ernest N. Mamikonyan wrote:
> 
> > I was wondering how I can increase the process address space, TASK_SIZE
> > (PAGE_OFFSET), in the current kernel. It looks like the 3 GB value is
> > hardcoded in a couple of places and is thus not trivial to alter. Is
> > there any good reason to limit this value at all, why not just have it
> > be the same as the max addressable space (64 GB)? We have an ix86 SMP
> > box with 4 GB of RAM and want to be able to allocate all of it to a
> > single program (physics simulation). I would greatly appreciate any help
> > on this.
> 
> The only way to achieve your goal would be to modify the
> hardware. What you want is not possible due to hardware
> limitations of the x86 platform.

I had a sneaking suspicion I didn't know what I was talking about. 
Should have read his original email a little closer :-)

-- Gareth