[uml-devel] memory

[uml-devel] memory

[Vaibhav Sharma, Noida]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

Hi,
I have to build a simulator, to check the scalability limits in memory
management of linux. For this, the simulator  will provide a large amount of
RAM (in GBs), even if the amout of physical memory is only 512MB. 

Can UML be used for this purpose? I had seen the "mem" option in the kernel
switches option of UML. Can UML simulate such a case? If it does, how UML
handles this? Specifically, can the UML kernel allocate all this amount of
memory (leaving aside the page faults which will be generated in case of
read/write requests to the allocated pages, for it has to use the actual
RAM)? Say, for example if I allocate 4 GB, will the UML kernel allocate all
the amount in the simulated RAM. (maybe it does it in the some swap, or the
/tmp, as the documentation says for the "mem" usage
<http://user-mode-linux.sourceforge.net/switches.html
<http://user-mode-linux.sourceforge.net/switches.html> > .  

If I have wrong interpretation of the "mem" option in UML, please do correct
me. 

TIA, 
Vaibhav 

[-- Attachment #2: Type: text/html, Size: 1674 bytes --]

Re: [uml-devel] memory

[Doug Dumitru]

Vaibhav Sharma, Noida wrote:
> Hi,
> I have to build a simulator, to check the scalability limits in memory 
> management of linux. For this, the simulator  will provide a large 
> amount of RAM (in GBs), even if the amout of physical memory is only 512MB.
> 
> Can UML be used for this purpose? I had seen the "mem" option in the 
> kernel switches option of UML. Can UML simulate such a case? If it does, 
> how UML handles this? Specifically, can the UML kernel allocate all this 
> amount of memory (leaving aside the page faults which will be generated 
> in case of read/write requests to the allocated pages, for it has to use 
> the actual RAM)? Say, for example if I allocate 4 GB, will the UML 
> kernel allocate all the amount in the simulated RAM. (maybe it does it 
> in the some swap, or the /tmp, as the documentation says for the "mem" 
> usage <_http://user-mode-linux.sourceforge.net/switches.html_> . 
> 
> If I have wrong interpretation of the "mem" option in UML, please do 
> correct me.
> 
> TIA,
> Vaibhav

Mr. Sharma,

What you are trying to do will work, but not for large amounts of 
memory.  UML runs the client using a single user mode memory block as 
the entire client's core.  Thus the clients core size is limited to what 
a single task can allocate as plain memory.  With x86 architechure cpus, 
this is 2G or 3G depending on how you build a kernel (but usually 2G). 
There are no paging mechanism here so there is no way to make it "gigabytes"

Others can correct me if I have the exact numbers wrong.



-- 

--------------------------------------------------------------------
Doug Dumitru     800-470-2756     (610-237-2000)
EasyCo LLC       doug@easyco.com  http://easyco.com
--------------------------------------------------------------------


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Rob Landley]

On Sunday 23 January 2005 03:28 am, Doug Dumitru wrote:

> Mr. Sharma,
>
> What you are trying to do will work, but not for large amounts of
> memory.  UML runs the client using a single user mode memory block as
> the entire client's core.  Thus the clients core size is limited to what
> a single task can allocate as plain memory.  With x86 architechure cpus,
> this is 2G or 3G depending on how you build a kernel (but usually 2G).
> There are no paging mechanism here so there is no way to make it
> "gigabytes"
>
> Others can correct me if I have the exact numbers wrong.

UML uses mmap (generally of a /tmp file) as physical memory.  This can be way 
bigger than physical memory on the machine, sure, and the parent kernel will 
transparently use the mmaped file as backing store, demand paged and all 
that.  (Just like it would for any other application.)

You could run out of virtual address space, though.  Trying to map more than a 
few gigabytes on a 32 bit machine is unlikely to work.

Might I suggest buying a cheap x86-64 system (you can get 'em at Fry's a 
couple hundred bucks), feed it a cheap 300 gigabyte IDE drive, and have fun.

Rob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Blaisorblade]

On Sunday 23 January 2005 10:51, Rob Landley wrote:
> On Sunday 23 January 2005 03:28 am, Doug Dumitru wrote:
> > Mr. Sharma,
> >
> > What you are trying to do will work, but not for large amounts of
> > memory.  UML runs the client using a single user mode memory block as
> > the entire client's core.  Thus the clients core size is limited to what
> > a single task can allocate as plain memory.  With x86 architechure cpus,
> > this is 2G or 3G depending on how you build a kernel (but usually 2G).
> > There are no paging mechanism here so there is no way to make it
> > "gigabytes"
> >
> > Others can correct me if I have the exact numbers wrong.
>
> UML uses mmap (generally of a /tmp file) as physical memory.  This can be
> way bigger than physical memory on the machine, sure, and the parent kernel
> will transparently use the mmaped file as backing store, demand paged and
> all that.  (Just like it would for any other application.)
>
> You could run out of virtual address space, though.  Trying to map more
> than a few gigabytes on a 32 bit machine is unlikely to work.
>
> Might I suggest buying a cheap x86-64 system (you can get 'em at Fry's a
> couple hundred bucks), feed it a cheap 300 gigabyte IDE drive, and have
> fun.

This is not (yet) exactly supported... a lot of work is going on this, and 
somebody was even able to compile and run (successfully?) some x86_64 umls... 
However, consider it still as a developer-only tool for now. 2.6.11 could 
effectively be a good x86_64 tree, but do not yet use it for anything 
critical.

Clearly, 32-bit UMLs run well on x86_64 boxes, but don't have the 64-bit 
advantages...
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Henrik Nordstrom]

On Sun, 23 Jan 2005, Doug Dumitru wrote:

> What you are trying to do will work, but not for large amounts of memory. 
> UML runs the client using a single user mode memory block as the entire 
> client's core.  Thus the clients core size is limited to what a single task 
> can allocate as plain memory.

It should be possible to work around this using the kernel HIGHMEM 
support. Would also be a good environment to verify the HIGHMEM support in 
the kernel in general.

In HIGHMEM mode the kernel doen't have the whole physical RAM mapped in 
it's address space, remapping memory as required. This doesn't increase 
the per-process limitations, but allows for the total sum or processes + 
kernel memory to be much greater.

Regards
Henrik


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Rob Landley]

On Sunday 23 January 2005 01:27 pm, Henrik Nordstrom wrote:
> On Sun, 23 Jan 2005, Doug Dumitru wrote:
> > What you are trying to do will work, but not for large amounts of memory.
> > UML runs the client using a single user mode memory block as the entire
> > client's core.  Thus the clients core size is limited to what a single
> > task can allocate as plain memory.
>
> It should be possible to work around this using the kernel HIGHMEM
> support. Would also be a good environment to verify the HIGHMEM support in
> the kernel in general.

The client kernel's highmem suport is unlikely to do much, I'd think.  Not 
unless it's unmapping and remapping multiple mmaps.  (There's large file 
support, but trying to mmap a 5 gig chunk out of a large file can't work: 
what would that mean?  How could you generate an offset into the last meg?  
What would the pointer _be_?)

The parent kernel's highmem support still doesn't provide more than 4 
gigabytes per application, and the UML kernel is one application.  (And 
that's with the 4 gig kernel/4 gig user patches from...  Ingo Molnar, I 
think.)  Highmem lets the system as a whole use PAE (up to 64 gigs physical 
memory), but the kernel is basically doing something very like the old DOS 
expanded memory behind the scenes, swapping around page tables so each 
process sees a different set of physical memory. pages  (The overhead isn't 
quite so bad since it's doing that anyway as part of protecting process's 
memory from each other...)

Hence the x86-64 system.  They really are getting cheap these days.  And when 
the PS3 comes out, that's going to be a 64 bit (mutant) Power PC for 
somewhere around $300 fully loaded, which runs Linux internally.  (Probably 
with binary only drivers out the wazoo, but still...)

Rob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Henrik Nordstrom]

On Sun, 23 Jan 2005, Rob Landley wrote:

> The client kernel's highmem suport is unlikely to do much, I'd think. 
> Not unless it's unmapping and remapping multiple mmaps. (There's large 
> file support, but trying to mmap a 5 gig chunk out of a large file can't 
> work: what would that mean?  How could you generate an offset into the 
> last meg? What would the pointer _be_?)

off_t is 64 bits in 64-bit file pointer mode, so there is no problem to 
mmap portions of an multi-gigabyte "ram" file within a 32-bit address 
space.

> The parent kernel's highmem support still doesn't provide more than 4
> gigabytes per application, and the UML kernel is one application.

Yes, but with HIGHMEM UML in theory can provide as as much ram to each 
single process running below it as it can for a single process. So yes, 
each process in the UML would still be limited, but the total RAM usage of 
UML would be practically unlimited. And in addition a very good 
environment for debugging HIGHMEM in general as you could set up things to 
only provide a very limited directly addressible address space with the 
rest remapped by HIGHMEM.

With HIGHMEM the host process size limitations has an direct effect on the 
maximum process size possible within the UML, but not on the RAM size of 
UML. In addition running an UML with HIGHMEM support does not depend on 
the host HIGHMEM support, but performance will obviously suffer if UML 
needs to swap..

> (And that's with the 4 gig kernel/4 gig user patches from...  Ingo 
> Molnar, I think.)  Highmem lets the system as a whole use PAE (up to 64 
> gigs physical memory), but the kernel is basically doing something very 
> like the old DOS expanded memory behind the scenes, swapping around page 
> tables so each process sees a different set of physical memory. pages 
> (The overhead isn't quite so bad since it's doing that anyway as part of 
> protecting process's memory from each other...)

Exctaly, and UML can do something very similar with the help of mmap().

Regards
Henrik


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Blaisorblade]

On Monday 24 January 2005 11:02, Henrik Nordstrom wrote:
> On Sun, 23 Jan 2005, Rob Landley wrote:
> > The client kernel's highmem suport is unlikely to do much, I'd think.
> > Not unless it's unmapping and remapping multiple mmaps.
It does this, indeed and sadly (that's why it's so slow).
> > (There's large 
> > file support, but trying to mmap a 5 gig chunk out of a large file can't
> > work: what would that mean?  How could you generate an offset into the
> > last meg? What would the pointer _be_?)

> off_t is 64 bits in 64-bit file pointer mode, so there is no problem to
> mmap portions of an multi-gigabyte "ram" file within a 32-bit address
> space.

Offset can sure be 64-bit wide, but he refers to mmaping a 5G-wide chunk of 
memory, and to a "pointer offset"... you cannot mmap more than 4G in memory 
even with 4G/4G. Yes, you can do it like he says (i.e. mmapping back and 
forth).

> > The parent kernel's highmem support still doesn't provide more than 4
> > gigabytes per application, and the UML kernel is one application.

> Yes, but with HIGHMEM UML in theory can provide as as much ram to each
> single process running below it as it can for a single process. So yes,
> each process in the UML would still be limited, but the total RAM usage of
> UML would be practically unlimited. And in addition a very good
> environment for debugging HIGHMEM in general as you could set up things to
> only provide a very limited directly addressible address space with the
> rest remapped by HIGHMEM.

This point holds actually.

I think however that Henrik will agree that this is not yet implemented, and 
(maybe) nor in sight... about this, I would add somebody suggested 
using /proc/mm to this aim: since a single address space is not enough to map 
all the 64 Giga on a PAE system, he suggested using multiple address spaces 
and switching between them to use memory as cache for MySQL (IIRC)... this 
was also posted on this list.

Also, since x86_64 systems are so cheap (I'm writing from one of them), x86_64 
UML is going to be a reality soon, and we're in short of manpower, the 
logical conclusion is that this is unlikely to be implemented... unless it's 
very simple; do you think that making sure the offset inside /tmp/vm_file-*** 
is 64-bit wide would be enough?

> With HIGHMEM the host process size limitations has an direct effect on the
> maximum process size possible within the UML, but not on the RAM size of
> UML. In addition running an UML with HIGHMEM support does not depend on
> the host HIGHMEM support, but performance will obviously suffer if UML
> needs to swap..

> > (And that's with the 4 gig kernel/4 gig user patches from...  Ingo
> > Molnar, I think.)
I confirm
> > Highmem lets the system as a whole use PAE (up to 64 
> > gigs physical memory), but the kernel is basically doing something very
> > like the old DOS expanded memory behind the scenes, swapping around page
> > tables so each process sees a different set of physical memory. pages
> > (The overhead isn't quite so bad since it's doing that anyway as part of
> > protecting process's memory from each other...)

> Exctaly, and UML can do something very similar with the help of mmap().
Which is too slow, though...
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Jeff Dike]

Vaibhavs@noida.hcltech.com said:
> I have to build a simulator, to check the scalability limits in memory
> management of linux. For this, the simulator  will provide a large
> amount of RAM (in GBs), even if the amout of physical memory is only
> 512MB. 
>
> Can UML be used for this purpose? I had seen the "mem" option in the
> kernel switches option of UML. Can UML simulate such a case? 

Yes, it should be fine for what you want.

> If it
> does, how UML handles this? Specifically, can the UML kernel allocate
> all this amount of memory

It'll allocate a temp file of the appropriate size.  If it's larger than what
it can use for physical memory, the rest will be allocated as highmem.  If
you leave 2-level pagetables on, then you're limited to 4G. If you turn on
3-level pagetables, then you get up to 128G.  

doug@easyco.com said:
> UML runs the client using a single user mode memory block as  the
> entire client's core.  Thus the clients core size is limited to what
> a single task can allocate as plain memory.

No, it only maps the physical memory as a block.  The highmem part is mapped
on demand, as with i386.

rob@landley.net said:
> You could run out of virtual address space, though.  Trying to map
> more than a  few gigabytes on a 32 bit machine is unlikely to work. 

With highmem, it's not all mapped at once.

rob@landley.net said:
> Not  unless it's unmapping and remapping multiple mmaps.  

It is.

> (There's
> large file  support, but trying to mmap a 5 gig chunk out of a large
> file can't work:  

It does it a page at a time.

> what would that mean?  How could you generate an
> offset into the last meg?   

The LFS interfaces take 64-bit offsets.

> What would the pointer _be_?)

What pointer?

> The parent kernel's highmem support still doesn't provide more than 4
> gigabytes per application, and the UML kernel is one application.

The host kernel's highmem support is totally irrelevant.  There doesn't even
need to be highmem support on the host.

				Jeff



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Rob Landley]

On Sunday 23 January 2005 08:11 pm, Jeff Dike wrote:
> It'll allocate a temp file of the appropriate size.  If it's larger than
> what it can use for physical memory, the rest will be allocated as highmem.
>  If you leave 2-level pagetables on, then you're limited to 4G. If you turn
> on 3-level pagetables, then you get up to 128G.

Interesting.  I wonder how that works?  (PAE on x86 only lets you have 64G.)

> rob@landley.net said:
> > Not  unless it's unmapping and remapping multiple mmaps.
>
> It is.

But an individual process running under UML can still only have 4 gigabytes.

> > (There's
> > large file  support, but trying to mmap a 5 gig chunk out of a large
> > file can't work:
>
> It does it a page at a time.
>
> > what would that mean?  How could you generate an
> > offset into the last meg?
>
> The LFS interfaces take 64-bit offsets.

Yeah.  (Ouch.  But yeah, I can see that working...)

> > What would the pointer _be_?)
>
> What pointer?

The virtual or physical address used to access memory.  (I'm guessing 
userspace programs running under UML are limited to 4 gigs, and UML is using 
page indexes and is thus limited to 4 billion pages, not 4 billion bytes.)

> > The parent kernel's highmem support still doesn't provide more than 4
> > gigabytes per application, and the UML kernel is one application.
>
> The host kernel's highmem support is totally irrelevant.  There doesn't
> even need to be highmem support on the host.

Just large file support, and the ability to mmap up to 4 gigs of memory at a 
time (with a starting offset potentially above 4 gigabytes), and unmap it and 
map a different 4 gigs when you switch to the next process...

Makes my brain hurt just thinking about it, but that could be caffeine 
withdrawl...

> 				Jeff

Rob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Henrik Nordstrom]

On Mon, 24 Jan 2005, Rob Landley wrote:

> Interesting.  I wonder how that works?  (PAE on x86 only lets you have 64G.)

Thats only an limitation of the CPU support for PAE. As UML is using 
mmap() other limits apply and these limits is mainly set by the UML 
pagetable structures.

> But an individual process running under UML can still only have 4 gigabytes.

Not even that as we do not have a 4GB/4GB split in UML.

>>> What would the pointer _be_?)
>>
>> What pointer?
>
> The virtual or physical address used to access memory.  (I'm guessing
> userspace programs running under UML are limited to 4 gigs, and UML is using
> page indexes and is thus limited to 4 billion pages, not 4 billion bytes.)

In theory UML HIGHMEM could be made to support even more pages, but it's 
besides the point.

> Just large file support, and the ability to mmap up to 4 gigs of memory at a
> time (with a starting offset potentially above 4 gigabytes), and unmap it and
> map a different 4 gigs when you switch to the next process...
>
> Makes my brain hurt just thinking about it, but that could be caffeine
> withdrawl...

Thinking of HIGHMEM does hurt by design as you no longer have the 
equivalence of pointers to userspace in kernel land and must explicitly 
ask to have the needed userspace areas mapped/unmapped as needed, giving 
you the required pointers..

Most of this is dealt with automatically in copy_to/from_user, but some 
parts of the kernel uses other means to access userspace data and thus 
need to be very careful.

Regards
Henrik



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Bodo Stroesser]

Henrik Nordstrom wrote:
> On Mon, 24 Jan 2005, Rob Landley wrote:
> 
>> Interesting.  I wonder how that works?  (PAE on x86 only lets you have 
>> 64G.)
> 
> 
> Thats only an limitation of the CPU support for PAE. As UML is using 
> mmap() other limits apply and these limits is mainly set by the UML 
> pagetable structures.
> 
>> But an individual process running under UML can still only have 4 
>> gigabytes.
> 
> 
> Not even that as we do not have a 4GB/4GB split in UML.
> 
>>>> What would the pointer _be_?)
>>>
>>>
>>> What pointer?
>>
>>
>> The virtual or physical address used to access memory.  (I'm guessing
>> userspace programs running under UML are limited to 4 gigs, and UML is 
>> using
>> page indexes and is thus limited to 4 billion pages, not 4 billion 
>> bytes.)
> 
> 
> In theory UML HIGHMEM could be made to support even more pages, but it's 
> besides the point.
I agree with nearly all of your good explanations, but have to write my
objection to this one.
Yes, using 3-level pagetables and mmap64, uml can access very much pagetables.
But that's not all, what's needed to have *support* for big memory.
Linux needs to have one struct page for each physical mem page, which in the
simple case of UML are placed in one memmap array.
This array needs to be accessible permanently to the kernel, i.e. it
has to be in low-mem. AFAIK, one struct page is 44 bytes in size, thus the
array is about 1% of the entire physical mem in size.
Assume we are using skas3 on a 3GB/1GB host that provides 3GB address space
for user processes (i.e. 3GB address space for UML-Kernel, too). Let the UML
kernel need about 1GB for code, vmalloc-area and data, that also need to reside
in lowmem. Then, there remain 2GB space for the page-structures, which limits
supportable memory to about 200GB.
That still is very much compared to i386, where on a 3GB/1GB host kernels
address space is 1GB only. Since memmap shouldn't use more than about one third
of this, i386 can support max. about 32GB of physical memory (some people talk
about 16GB only).
BTW: This is the reason for having the 2GB/2GB option in i386.

      Bodo
> 
>> Just large file support, and the ability to mmap up to 4 gigs of 
>> memory at a
>> time (with a starting offset potentially above 4 gigabytes), and unmap 
>> it and
>> map a different 4 gigs when you switch to the next process...
>>
>> Makes my brain hurt just thinking about it, but that could be caffeine
>> withdrawl...
> 
> 
> Thinking of HIGHMEM does hurt by design as you no longer have the 
> equivalence of pointers to userspace in kernel land and must explicitly 
> ask to have the needed userspace areas mapped/unmapped as needed, giving 
> you the required pointers..
> 
> Most of this is dealt with automatically in copy_to/from_user, but some 
> parts of the kernel uses other means to access userspace data and thus 
> need to be very careful.
> 
> Regards
> Henrik
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> User-mode-linux-devel mailing list
> User-mode-linux-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[stian]

> Linux needs to have one struct page for each physical mem page, which in
> the
> simple case of UML are placed in one memmap array.
> This array needs to be accessible permanently to the kernel, i.e. it
> has to be in low-mem. AFAIK, one struct page is 44 bytes in size, thus the
> array is about 1% of the entire physical mem in size.

You can mangle with the page-size as-well. You will break some few
programs by increasing the page-size too 16K instead of 4K, but most
software should respect the output mmap gives it, etc.


Stian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Bodo Stroesser]

Jeff Dike wrote:
> bstroesser@fujitsu-siemens.com said:
> 
>>But that's not all, what's needed to have *support* for big memory.
>>Linux needs to have one struct page for each physical mem page, which
>>in the simple case of UML are placed in one memmap array. 
> 
> 
> The original poster is interested in the memory scalability of Linux, so
> presumably he'll be looking into things like this :-)
> 
> And it's not only this.  It's the kernel stacks and page tables you'll have
> for all the processes you'll be running on this large machine, inodes and other
> file system data for all the files they have open, etc.
Yes. That's the reason, why I estimated 1GB of memory for kernels internal use,
additionally to the memmap.

> 
> 				Jeff


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

RE: [uml-devel] memory

[Vaibhav Sharma, Noida]

Thank you all for ur valuable discussion (temp dir) and suggestions related
to highmem/PAE and x86_64. 

I have another thing to ask related to debugging...i want to check the
routines which will be involved in the memory management.  

According to the design of UML, it has its own VM system. If I have to check
the scalability limits of the "parent kernel" for memory management, will
UML be useful..? 

When I debug the UML kernel when a process requests a large amount of
memory, I will only get the flow used by UML for its VM. Is the code/design
of UML very different from the parent kernel..?

Is there a way in I can get to code of parent kernel when 2G /3G RAM size is
used?

(I may be totally wrong in this mail, please do correct me...)
--

vaibhav

-----Original Message-----
From: Rob Landley [mailto:rob@landley.net] 
Sent: Monday, January 24, 2005 10:40 AM
To: user-mode-linux-devel@lists.sourceforge.net
Cc: Jeff Dike; Vaibhav Sharma, Noida; Doug Dumitru
Subject: Re: [uml-devel] memory

On Sunday 23 January 2005 08:11 pm, Jeff Dike wrote:
> It'll allocate a temp file of the appropriate size.  If it's larger 
> than what it can use for physical memory, the rest will be allocated as
highmem.
>  If you leave 2-level pagetables on, then you're limited to 4G. If you 
> turn on 3-level pagetables, then you get up to 128G.

Interesting.  I wonder how that works?  (PAE on x86 only lets you have 64G.)

> rob@landley.net said:
> > Not  unless it's unmapping and remapping multiple mmaps.
>
> It is.

But an individual process running under UML can still only have 4 gigabytes.

> > (There's
> > large file  support, but trying to mmap a 5 gig chunk out of a large 
> > file can't work:
>
> It does it a page at a time.
>
> > what would that mean?  How could you generate an offset into the 
> > last meg?
>
> The LFS interfaces take 64-bit offsets.

Yeah.  (Ouch.  But yeah, I can see that working...)

> > What would the pointer _be_?)
>
> What pointer?

The virtual or physical address used to access memory.  (I'm guessing
userspace programs running under UML are limited to 4 gigs, and UML is using
page indexes and is thus limited to 4 billion pages, not 4 billion bytes.)

> > The parent kernel's highmem support still doesn't provide more than 
> > 4 gigabytes per application, and the UML kernel is one application.
>
> The host kernel's highmem support is totally irrelevant.  There 
> doesn't even need to be highmem support on the host.

Just large file support, and the ability to mmap up to 4 gigs of memory at a
time (with a starting offset potentially above 4 gigabytes), and unmap it
and map a different 4 gigs when you switch to the next process...

Makes my brain hurt just thinking about it, but that could be caffeine
withdrawl...

> 				Jeff

Rob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Rob Landley]

On Monday 24 January 2005 03:13 am, Vaibhav  Sharma, Noida wrote:
> According to the design of UML, it has its own VM system. If I have to
> check the scalability limits of the "parent kernel" for memory management,
> will UML be useful..?

Jeff Dike already answered this.  (He's the UML maintainer.)  UML _does_ remap 
a different chunk out of the physical memory backing file when it schedules a 
different process.  UML uses mmap to get its memory, so as long as the parent 
kernel has Large File Support (which lets a 32 bit machine use 64 bit file 
offsets, this is pretty widely deployed because people need a single file to 
be larger than INT_MAX, which is 2 gigabytes since file offsets are signed 
for some reason), and the UML instance is configured to use highmem (which is 
what unmaps and remaps the memory on a schedule) and at least 3 level page 
tables (so it doesn't run out of PTEs), UML shouldn't be limited to 32 bits 
of (simulated) physical memory.  Not that I've tried this. :)

An individual process running under a 32-bit UML would still only be able to 
access 4 gigabytes or less.  (It has a 32 bit virtual address space, after 
all...)  And since I'm unaware of any version of UML using the 4 gig/4 gig 
patch, I suspect UML might fill up its kernel memory area with page table 
entries eventually...  (Was Dave Jones's shared page table patch ever 
integrated?  Are linux page tables swappable?)

> When I debug the UML kernel when a process requests a large amount of
> memory, I will only get the flow used by UML for its VM. Is the code/design
> of UML very different from the parent kernel..?

UML uses mmap on a file to simulate an MMU.  It shouldn't care what kind of 
physical memory the parent kernel has.  But Jeff's the expert here.

However running 'find arch/um -name "*.c" | xargs grep mmap' in your linux 
source directory should get you started.

> (I may be totally wrong in this mail, please do correct me...)

I'm not the world's greatest source of info here either, I've only been 
playing with UML for a little while...

Rob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Blaisorblade]

On Monday 24 January 2005 09:13, Vaibhav  Sharma, Noida wrote:
> Thank you all for ur valuable discussion (temp dir) and suggestions related
> to highmem/PAE and x86_64.
>
> I have another thing to ask related to debugging...i want to check the
> routines which will be involved in the memory management.
>
> According to the design of UML, it has its own VM system. If I have to
> check the scalability limits of the "parent kernel" for memory management,
> will UML be useful..?

Well, yes, indeed. UML, indeed, has no special way to request memory on the 
host*, so the host kernel must be efficient in managing and swapping UML 
memory. On top of that, using the very same algorithms as any Linux system 
(or better, as that release compiled for another arch), UML manages his own 
memory and splits it between processes, swaps part of it, and so on.

* (if something is thinking to SKAS, it does not account in the discussion - 
that is a different question, which is about implementation details and not 
about the conceptual model, as shown by skas0, i.e. UML running in a 
(limited) SKAS mode without host SKAS support...).

> When I debug the UML kernel when a process requests a large amount of
> memory, I will only get the flow used by UML for its VM. Is the code/design
> of UML very different from the parent kernel..?
The code is the same, but the data they use for their decision are 
different... RAM is passed by the host to UML and by UML to the guest, as 
said above.

> Is there a way in I can get to code of parent kernel when 2G /3G RAM size
> is used?
??? Do you want to understand where the host is limited to 3G for its 
applications? I don't think you want this... the exact point is the TASK_SIZE 
macro, which is configurable somehow... however you must be careful...

If you leave 512 M to the kernel and 3,5G to the userspace, all the memory 
above 512M will be slower... Actually, it's 512 - 128 M at least (the kernel 
part of the 4G of virtual address space must map physical RAM, AND host some 
virtual mappings, i.e. 128M of vmalloc()ed memory).

You'd probably prefer to use the 4G/4G patch from Ingo Molnar, integrated into 
Fedora kernels, to allow processes to use 4G of virtual address space.
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Henrik Nordstrom]

On Mon, 24 Jan 2005, Blaisorblade wrote:

> You'd probably prefer to use the 4G/4G patch from Ingo Molnar, integrated into
> Fedora kernels, to allow processes to use 4G of virtual address space.

Side note: Fedora apparently dropped this patch some time ago, reason not 
known to me.

* Thu Dec 09 2004 Dave Jones <davej@redhat.com>

- Drop the 4g/4g hugemem kernel completely.

Regards
Henrik


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] memory

[Rob Landley]

On Monday 24 January 2005 05:41 pm, Henrik Nordstrom wrote:
> On Mon, 24 Jan 2005, Blaisorblade wrote:
> > You'd probably prefer to use the 4G/4G patch from Ingo Molnar, integrated
> > into Fedora kernels, to allow processes to use 4G of virtual address
> > space.
>
> Side note: Fedora apparently dropped this patch some time ago, reason not
> known to me.

Dell started shipping em64t systems, making it unanimous that their 
"enterprise" customers (only ones who ever cared) could use buy x86-64 
hardware.

Rob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel