RAM allocation and miscellaneous questions

RAM allocation and miscellaneous questions

[Paul Dorman]

Hi everyone,

I'm running 10 VMs now quite happily, using LVM2 with distinct storage areas 
for each VM filesystem.

I suspect I'll be able to run around 30 VMs on the dual xeon 2.8GHz machines 
successfully, but I'm wondering how to manage the RAM allocations for each. I 
calculate that each VM will have around 110MB RAM if I run that many, 
including a healthy amount of RAM for Dom-0. I am wondering how this can be 
enough for LAMP VMs. Some of the systems I'm running now don't seem to 
perform very well with less than 400MB of RAM, and I can only really do that 
if some kind soul fixes Xen so I can use the full 12GB capacity of my Xen 
servers (they have 4GB now -- no good wasting money!).

This is probably a stupid question, but can I allocate more RAM to each VM 
than arithmetic would suggest? Is there some kind of compression or COW that 
goes on in the Xen memory management code?

Would it be best to run one replicated MySQL server (or several with different 
versions) loaded into a VM on each physical machine, and have all the other 
machines use that over virtual network connections? I suppose this would be 
perfectly okay, and I could do the same with Apache too I guess. But as I am 
going to be migrating the sites from many real machines into Xen, splitting 
them up into individual site machines as I go (at the moment our servers can 
run many sites), it would take a lot of work to change their configuration so 
that they connected to a networked copy of MySQL or Apache rather than a 
local one.

Another option, and I'm thinking you are all going to say 'go for that one!', 
would be to use 64bit machines for our main cluster, perhaps just using the 
two dual Xeon machines I have now for development purposes. That way I could 
use lots of RAM and presumably load up even more VMs into each. If I go down 
this road, I'm wondering what kind of hardware I should get. Does Xen work 
well with Opterons? I'm an AMD fan and would be very happy to use them if I 
went for this option. I'm after inexpensive commodity servers, so if anyone 
has any recommendations (especially if you are in NZ or Australia), I'm all 
ears.

May I ask how much RAM I could use with a dual Opteron system in a sensible 
Xen configuration? What's been tested?

One final question: if I were to ask the directors of the company if they 
would be interested in sponsoring big memory on Xeon-based machines, what 
kind of commitment would we be talking about? It could be cheaper to do this 
than to buy a cluster of dual Opteron machines! I do realise though that the 
true impediment will probably be the amount of time you developers have, so 
I'll brace myself for disappointment! :o) 

Thanks everyone for your valuable time.

Regards,
Paul


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Mark A. Williamson]

> I'm running 10 VMs now quite happily, using LVM2 with distinct storage
> areas for each VM filesystem.

Cool!

> can I allocate more RAM to each VM than arithmetic would suggest? 

Yes and no ;-)

Xen does not do paging to disk, so you can't (for instance) tell a domain it 
has 256MB but only give it 128MB and then fake out the extra capacity using 
disk.  The solution under Xen would be to just get the domain to do its own 
paging - this has the same end result and avoids certain performance and 
implementation nasties associated with paging in the VMM.

Using the balloon driver, you can have a setup where you add and remove memory 
to / from domains according to their need, so that you can shrink the memory 
footprints of domains that don't (currently) need as much in core.  This is 
done manually.

Of course, you can also use suspend / resume and live migration to move 
domains around in order to balance memory load.

> Is there some kind of compression or COW 
> that goes on in the Xen memory management code?

Not right now.  There is a plan to implement a shared buffer cache, which 
would allow domains to share unmodified pages of data that have been read 
from disk.  This is planned as a performance optimisation not a space 
optimisation, however.

> Would it be best to run one replicated MySQL server (or several with
> different versions) loaded into a VM on each physical machine, and have all
> the other machines use that over virtual network connections?

It probably depends.  I guess you'd save some memory footprint that way 
(compared to running many copies of the servers) but I really don't know how 
much.

> [64 bit machines]
> Does Xen work well with Opterons?

In 32-bit mode, it works.  In 64-bit mode, not quite yet.  x86_64 support is 
under development at the moment and is scheduled as a post-2.0 feature.  
However that will be the preferred path to large memory configurations.

HTH,
Mark


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Paul Dorman]

It certainly does Mark (help, that is). I would like to learn more about this 
balloon driver, which could make things very managable. Is there any 
documentation or tutorial-like info about?

Also, though my domains are running happily, I am seeing plenty of 
memory-related errors. I've just hooked up a serial cable between my 
development servers and my desktop (I live with fans) so I'll be able to 
furnish the list with more details.

I have been wondering why my Domain-0 machine hasn't been using swap! :o) So I 
should just dump that partition? I suppose I'll have to be very rigorous when 
it comes to keeping that domain lean yes? (Especially as I only have 4GB to 
play with!). Out side of basic OS functionality, are there any Xen or LVM2 
related processes I have to account for when setting the amount of RAM I can 
use for Domain-0?

I don't suppose I could use the ballon driver for Domain-0 too!?!

Thanks for your reply.

Regards,
Paul

On Wednesday 29 September 2004 03:30 pm, Mark A. Williamson wrote:
> > I'm running 10 VMs now quite happily, using LVM2 with distinct storage
> > areas for each VM filesystem.
>
> Cool!
>
> > can I allocate more RAM to each VM than arithmetic would suggest?
>
> Yes and no ;-)
>
> Xen does not do paging to disk, so you can't (for instance) tell a domain
> it has 256MB but only give it 128MB and then fake out the extra capacity
> using disk.  The solution under Xen would be to just get the domain to do
> its own paging - this has the same end result and avoids certain
> performance and implementation nasties associated with paging in the VMM.
>
> Using the balloon driver, you can have a setup where you add and remove
> memory to / from domains according to their need, so that you can shrink
> the memory footprints of domains that don't (currently) need as much in
> core.  This is done manually.
>
> Of course, you can also use suspend / resume and live migration to move
> domains around in order to balance memory load.
>
> > Is there some kind of compression or COW
> > that goes on in the Xen memory management code?
>
> Not right now.  There is a plan to implement a shared buffer cache, which
> would allow domains to share unmodified pages of data that have been read
> from disk.  This is planned as a performance optimisation not a space
> optimisation, however.
>
> > Would it be best to run one replicated MySQL server (or several with
> > different versions) loaded into a VM on each physical machine, and have
> > all the other machines use that over virtual network connections?
>
> It probably depends.  I guess you'd save some memory footprint that way
> (compared to running many copies of the servers) but I really don't know
> how much.
>
> > [64 bit machines]
> > Does Xen work well with Opterons?
>
> In 32-bit mode, it works.  In 64-bit mode, not quite yet.  x86_64 support
> is under development at the moment and is scheduled as a post-2.0 feature.
> However that will be the preferred path to large memory configurations.
>
> HTH,
> Mark


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Ian Pratt]

> It certainly does Mark (help, that is). I would like to learn more about this 
> balloon driver, which could make things very managable. Is there any 
> documentation or tutorial-like info about?

The balloon driver hasn't been forward ported from 2.4 to 2.6 --
it's on the todo list (it'll probably work unmodified modulo
changes to the way proc fs works).
 
> I have been wondering why my Domain-0 machine hasn't been using swap! :o) So I 
> should just dump that partition? 

If you've configured swap, domain 0 should be using just like any
normal linux kernel. The balloon driver works in dom0 just like
any other domain.

> I suppose I'll have to be very rigorous when 
> it comes to keeping that domain lean yes? (Especially as I only have 4GB to 
> play with!). Out side of basic OS functionality, are there any Xen or LVM2 
> related processes I have to account for when setting the amount of RAM I can 
> use for Domain-0?

-xen linux kernels don't have any special processes that take any
noticeable extra resources. 

In domain 0 you'll have to worry about the footprint of xend, but
it's not that big.

Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Mark A. Williamson]

> It certainly does Mark (help, that is). I would like to learn more about
> this balloon driver, which could make things very managable. Is there any
> documentation or tutorial-like info about?

AFAIK, there's a file in /proc/xen called memory_target or something similar.  
You just write your memory target into there as ASCII text.
e.g. echo target > /proc/xen/memory_target

The idea is that there'll (at some stage) be control messages from dom0 to do 
this so that you don't have to log into a domain to change it's memory 
target.

As Ian said, the driver isn't yet ported to 2.6 (I forgot that) but it 
shouldn't take long when someone gets around to it :-)

> Also, though my domains are running happily, I am seeing plenty of
> memory-related errors. I've just hooked up a serial cable between my
> development servers and my desktop (I live with fans) so I'll be able to
> furnish the list with more details.

I can relate to living with fans :-)  It's useful to have a serial console 
available in case of machine crashes / hangs.  When the machine is still 
operational, xm dmesg will retrive Xen's boot output.

> I have been wondering why my Domain-0 machine hasn't been using swap! :o)

I should have explained this better: domains can swap their own memory to / 
from disk as much as they like.  In contrast, Xen itself does not do any 
swapping to provide the illusion of bigger physical memory.  If domains want 
a bigger memory than you gave them, it's up to them to do all their own 
swapping.

If you want dom0 to swap, you should be able to set up swap in the usual way 
(if it worked on vanilla linux, it should just keep working).  If you want 
other domains to swap, you can give them VBDs to swap to - you just have to 
mkswap on them and configure it in the fstab, same as usual.

> So I should just dump that partition? I suppose I'll have to be very
> rigorous when it comes to keeping that domain lean yes? (Especially as I
> only have 4GB to play with!). Out side of basic OS functionality, are there
> any Xen or LVM2 related processes I have to account for when setting the
> amount of RAM I can use for Domain-0?
>
> I don't suppose I could use the ballon driver for Domain-0 too!?!
>
> Thanks for your reply.
>
> Regards,
> Paul
>
> On Wednesday 29 September 2004 03:30 pm, Mark A. Williamson wrote:
> > > I'm running 10 VMs now quite happily, using LVM2 with distinct storage
> > > areas for each VM filesystem.
> >
> > Cool!
> >
> > > can I allocate more RAM to each VM than arithmetic would suggest?
> >
> > Yes and no ;-)
> >
> > Xen does not do paging to disk, so you can't (for instance) tell a domain
> > it has 256MB but only give it 128MB and then fake out the extra capacity
> > using disk.  The solution under Xen would be to just get the domain to do
> > its own paging - this has the same end result and avoids certain
> > performance and implementation nasties associated with paging in the VMM.
> >
> > Using the balloon driver, you can have a setup where you add and remove
> > memory to / from domains according to their need, so that you can shrink
> > the memory footprints of domains that don't (currently) need as much in
> > core.  This is done manually.
> >
> > Of course, you can also use suspend / resume and live migration to move
> > domains around in order to balance memory load.
> >
> > > Is there some kind of compression or COW
> > > that goes on in the Xen memory management code?
> >
> > Not right now.  There is a plan to implement a shared buffer cache, which
> > would allow domains to share unmodified pages of data that have been read
> > from disk.  This is planned as a performance optimisation not a space
> > optimisation, however.
> >
> > > Would it be best to run one replicated MySQL server (or several with
> > > different versions) loaded into a VM on each physical machine, and have
> > > all the other machines use that over virtual network connections?
> >
> > It probably depends.  I guess you'd save some memory footprint that way
> > (compared to running many copies of the servers) but I really don't know
> > how much.
> >
> > > [64 bit machines]
> > > Does Xen work well with Opterons?
> >
> > In 32-bit mode, it works.  In 64-bit mode, not quite yet.  x86_64 support
> > is under development at the moment and is scheduled as a post-2.0
> > feature. However that will be the preferred path to large memory
> > configurations.
> >
> > HTH,
> > Mark


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Keir Fraser]

> Another option, and I'm thinking you are all going to say 'go for that one!', 
> would be to use 64bit machines for our main cluster, perhaps just using the 
> two dual Xeon machines I have now for development purposes. That way I could 
> use lots of RAM and presumably load up even more VMs into each. If I go down 
> this road, I'm wondering what kind of hardware I should get. Does Xen work 
> well with Opterons? I'm an AMD fan and would be very happy to use them if I 
> went for this option. I'm after inexpensive commodity servers, so if anyone 
> has any recommendations (especially if you are in NZ or Australia), I'm all 
> ears.

Opteron isn't supported yet in 64-bit mode, but it /is/ on the todo
list to get it finished off. As opposed to PAE36 mode, which it is
highly likley that we will never support.

> May I ask how much RAM I could use with a dual Opteron system in a sensible 
> Xen configuration? What's been tested?

You will be able to stuff in as much RAM as the machine will take. :-)

> One final question: if I were to ask the directors of the company if they 
> would be interested in sponsoring big memory on Xeon-based machines, what 
> kind of commitment would we be talking about? It could be cheaper to do this 
> than to buy a cluster of dual Opteron machines! I do realise though that the 
> true impediment will probably be the amount of time you developers have, so 
> I'll brace myself for disappointment! :o) 

So you already have an existing 32-bit cluster that you would have to
sideline?

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Paul Dorman]

Hmmmmmm. Lots of RAM :o)

Yes, I do have a 'cluster', but only two machines so far. The problem for me 
is that multi-processor Opteron 1u machines (or 2u or 4u for that matter) are 
hard to come by here in New Zealand, unless we pay through the nose for an HP 
or other big-name machine. I can get dual Xeon machines pretty cheap though. 
If I look at the Opteron option, I'll have to compare it to just buying more 
Xeons. Either way, it's going to cost more for our initial cluster than it 
would if I could just use 12GB in a cheap Xeon unit. Don't get me wrong 
though, if Opteron units were available here I would have got them from the 
outset.

Personally I have no issue with sourcing machines from Australia or somewhere, 
but my bosses will be reluctant due to support issues, etc.

Regards,
Paul

On Wednesday 29 September 2004 08:37 pm, Keir Fraser wrote:
> > Another option, and I'm thinking you are all going to say 'go for that
> > one!', would be to use 64bit machines for our main cluster, perhaps just
> > using the two dual Xeon machines I have now for development purposes.
> > That way I could use lots of RAM and presumably load up even more VMs
> > into each. If I go down this road, I'm wondering what kind of hardware I
> > should get. Does Xen work well with Opterons? I'm an AMD fan and would be
> > very happy to use them if I went for this option. I'm after inexpensive
> > commodity servers, so if anyone has any recommendations (especially if
> > you are in NZ or Australia), I'm all ears.
>
> Opteron isn't supported yet in 64-bit mode, but it /is/ on the todo
> list to get it finished off. As opposed to PAE36 mode, which it is
> highly likley that we will never support.
>
> > May I ask how much RAM I could use with a dual Opteron system in a
> > sensible Xen configuration? What's been tested?
>
> You will be able to stuff in as much RAM as the machine will take. :-)
>
> > One final question: if I were to ask the directors of the company if they
> > would be interested in sponsoring big memory on Xeon-based machines, what
> > kind of commitment would we be talking about? It could be cheaper to do
> > this than to buy a cluster of dual Opteron machines! I do realise though
> > that the true impediment will probably be the amount of time you
> > developers have, so I'll brace myself for disappointment! :o)
>
> So you already have an existing 32-bit cluster that you would have to
> sideline?
>
>  -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Ronald G. Minnich]

> On Wednesday 29 September 2004 08:37 pm, Keir Fraser wrote:
> > Opteron isn't supported yet in 64-bit mode, but it /is/ on the todo
> > list to get it finished off. As opposed to PAE36 mode, which it is
> > highly likley that we will never support.
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

a truly excellent decision. I mean it. PAE36 is just too ugly to deal 
with, and we don't need it now with Opterons out there as cheap or cheaper 
than Pentiums.

ron


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Keir Fraser]

How odd -- I expected that in most places there would be multiple
white-box vendors selling Opteron-based systems. It's not like the
component parts are in short supply. Except maybe in NZ. :-)

 -- Keir

> Hmmmmmm. Lots of RAM :o)
> 
> Yes, I do have a 'cluster', but only two machines so far. The problem for me 
> is that multi-processor Opteron 1u machines (or 2u or 4u for that matter) are 
> hard to come by here in New Zealand, unless we pay through the nose for an HP 
> or other big-name machine. I can get dual Xeon machines pretty cheap though. 
> If I look at the Opteron option, I'll have to compare it to just buying more 
> Xeons. Either way, it's going to cost more for our initial cluster than it 
> would if I could just use 12GB in a cheap Xeon unit. Don't get me wrong 
> though, if Opteron units were available here I would have got them from the 
> outset.
> 
> Personally I have no issue with sourcing machines from Australia or somewhere, 
> but my bosses will be reluctant due to support issues, etc.
> 
> Regards,
> Paul


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Mike Brady]

The parts are readily available in NZ and I would be surprised if you couldn't 
find a local supplier in Wellington to put a system together for you.  You 
would be getting support for the parts rather than support for the system in 
this case though.  Which is what I assume the issue would be.

On Thursday 30 September 2004 21:02, Keir Fraser wrote:
> How odd -- I expected that in most places there would be multiple
> white-box vendors selling Opteron-based systems. It's not like the
> component parts are in short supply. Except maybe in NZ. :-)
>
>  -- Keir
>
> > Hmmmmmm. Lots of RAM :o)
> >
> > Yes, I do have a 'cluster', but only two machines so far. The problem for
> > me is that multi-processor Opteron 1u machines (or 2u or 4u for that
> > matter) are hard to come by here in New Zealand, unless we pay through
> > the nose for an HP or other big-name machine. I can get dual Xeon
> > machines pretty cheap though. If I look at the Opteron option, I'll have
> > to compare it to just buying more Xeons. Either way, it's going to cost
> > more for our initial cluster than it would if I could just use 12GB in a
> > cheap Xeon unit. Don't get me wrong though, if Opteron units were
> > available here I would have got them from the outset.
> >
> > Personally I have no issue with sourcing machines from Australia or
> > somewhere, but my bosses will be reluctant due to support issues, etc.
> >
> > Regards,
> > Paul
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Mat]

On September 29, 9:55 am Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
> >  It certainly does Mark (help, that is). I would like to learn more
> >  about this balloon driver, which could make things very managable. Is
> >  there any documentation or tutorial-like info about?
>
> The balloon driver hasn't been forward ported from 2.4 to 2.6 --
> it's on the todo list (it'll probably work unmodified modulo
> changes to the way proc fs works).
>
For people that still use linux 2.4 (like me) is there a tutorial or some
documentation about the ballon driver ?
> >  I have been wondering why my Domain-0 machine hasn't been using swap!
> >  :o) So I should just dump that partition?
>
> If you've configured swap, domain 0 should be using just like any
> normal linux kernel. The balloon driver works in dom0 just like
> any other domain.
It seems logical, but due to the fact that only a few process are running
ind dom0, swap file don't need to be huge is it ?

>
> >  I suppose I'll have to be very rigorous when
> >  it comes to keeping that domain lean yes? (Especially as I only have
> >  4GB to play with!). Out side of basic OS functionality, are there any
> >  Xen or LVM2 related processes I have to account for when setting the
> >  amount of RAM I can use for Domain-0?
>
> -xen linux kernels don't have any special processes that take any
> noticeable extra resources.
>
> In domain 0 you'll have to worry about the footprint of xend, but
> it's not that big.
>
When starting with xen one or two month ago, i've created a dom0 with only
32 Mo of ram, and when i launched xend, it was so slow that i tought i had
a problem.
I didn't take time to investigate and didn't remember how much memory was
used and if the system was swaping or not.
I just have done a test now with my current xen putting 32768 Mo on xen
command line (not the kernel line in grub).
And the system booted a bit slower and i suspect that durring the launch of
xend it has done some swap.
free return this just after the boot :
mat@zeus:/$ free
             total       used       free     shared    buffers     cached
Mem:         32768      28572       4196          0          0       7172
-/+ buffers/cache:      21400      11368
Swap:       506008        312     505696

Having this process:
  207 ?        Ss     0:00 dhclient -e -pf /var/run/dhclient.eth0.pid -lf
/var/
  253 ?        Ss     0:00 /sbin/syslogd
  256 ?        Ss     0:00 /sbin/klogd
  261 ?        Ss     0:00 /usr/sbin/inetd
  268 ?        Ss     0:00 /usr/sbin/sshd
  274 ?        S      0:00 xfrd
  275 ?        S      0:00 python /usr/sbin/xend start
  288 ?        Ss     0:00 /usr/sbin/cron
  327 tty1     Ss+    0:00 /sbin/getty 38400 tty1
  328 tty2     Ss+    0:00 /sbin/getty 38400 tty2
  329 tty3     Ss+    0:00 /sbin/getty 38400 tty3
  330 tty4     Ss+    0:00 /sbin/getty 38400 tty4
  331 tty5     Ss+    0:00 /sbin/getty 38400 tty5
  332 tty6     Ss+    0:00 /sbin/getty 38400 tty6
  333 ?        Ss     0:00 sshd: mat [priv]
  335 ?        R      0:00 sshd: mat@pts/0
  336 pts/0    Ss     0:00 -bash
  344 pts/0    R+     0:00 ps ax

i guess i could reduce the number of getty and get rid of dhclient (the
adress of dom0 is a static one ...).

It was my 2cents thoughts ...
Matthieu


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Ian Pratt]

> On September 29, 9:55 am Ian Pratt <Ian.Pratt@cl.cam.ac.uk> wrote:
> > >  It certainly does Mark (help, that is). I would like to learn more
> > >  about this balloon driver, which could make things very managable. Is
> > >  there any documentation or tutorial-like info about?
> >
> > The balloon driver hasn't been forward ported from 2.4 to 2.6 --
> > it's on the todo list (it'll probably work unmodified modulo
> > changes to the way proc fs works).
> >
> For people that still use linux 2.4 (like me) is there a tutorial or some
> documentation about the ballon driver ?

Mark is currently forward porting it to 2.6, and we're planing on
adding a couple of features to enable the memory footprint of a
domain to be controlled from xm/xend rather than just from just
within a domain.

> > If you've configured swap, domain 0 should be using just like any
> > normal linux kernel. The balloon driver works in dom0 just like
> > any other domain.
> It seems logical, but due to the fact that only a few process are running
> ind dom0, swap file don't need to be huge is it ?

Correct.
 
> When starting with xen one or two month ago, i've created a dom0 with only
> 32 Mo of ram, and when i launched xend, it was so slow that i tought i had
> a problem.
> I didn't take time to investigate and didn't remember how much memory was
> used and if the system was swaping or not.
> I just have done a test now with my current xen putting 32768 Mo on xen
> command line (not the kernel line in grub).
> And the system booted a bit slower and i suspect that durring the launch of
> xend it has done some swap.
> free return this just after the boot :
> mat@zeus:/$ free
>              total       used       free     shared    buffers     cached
> Mem:         32768      28572       4196          0          0       7172
> -/+ buffers/cache:      21400      11368
> Swap:       506008        312     505696


The resident working set of xend once it's started isn't too bad,
but all the library loading (particularly twisted) does lead to
quite a large virtual memory footprint. 

Perhaps one day, when xend has all the features we want, it'll be
time to re-write it in C, or OCAML...

Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Mark A. Williamson]

> For people that still use linux 2.4 (like me) is there a tutorial or some
> documentation about the ballon driver ?

Documentation, 2.6 support and facilities to control other domain's balloons 
using xm in domain 0 are cooking and should be ready shortly.

In the meantime, do:
# echo target_megabytes > /proc/xen/memory_target
To set a new memory footprint.  There's a Linux kernel param (mem=..., I 
think) for setting the maximum amount of memory a domain can handle, if 
larger than it's initial allocation (i.e. if you want to start with x 
megabytes but grow it later).

Trying to grow the domain's memory beyond what it's expecting (the initial 
allocation or the kernel param if you specified that) will not work.

Use:
# cat /proc/xen/memory_target
To get the current footprint.

HTH,
Mark

> > >  I have been wondering why my Domain-0 machine hasn't been using swap!
> > >
> > >  :o) So I should just dump that partition?
> >
> > If you've configured swap, domain 0 should be using just like any
> > normal linux kernel. The balloon driver works in dom0 just like
> > any other domain.
>
> It seems logical, but due to the fact that only a few process are running
> ind dom0, swap file don't need to be huge is it ?
>
> > >  I suppose I'll have to be very rigorous when
> > >  it comes to keeping that domain lean yes? (Especially as I only have
> > >  4GB to play with!). Out side of basic OS functionality, are there any
> > >  Xen or LVM2 related processes I have to account for when setting the
> > >  amount of RAM I can use for Domain-0?
> >
> > -xen linux kernels don't have any special processes that take any
> > noticeable extra resources.
> >
> > In domain 0 you'll have to worry about the footprint of xend, but
> > it's not that big.
>
> When starting with xen one or two month ago, i've created a dom0 with only
> 32 Mo of ram, and when i launched xend, it was so slow that i tought i had
> a problem.
> I didn't take time to investigate and didn't remember how much memory was
> used and if the system was swaping or not.
> I just have done a test now with my current xen putting 32768 Mo on xen
> command line (not the kernel line in grub).
> And the system booted a bit slower and i suspect that durring the launch of
> xend it has done some swap.
> free return this just after the boot :
> mat@zeus:/$ free
>              total       used       free     shared    buffers     cached
> Mem:         32768      28572       4196          0          0       7172
> -/+ buffers/cache:      21400      11368
> Swap:       506008        312     505696
>
> Having this process:
>   207 ?        Ss     0:00 dhclient -e -pf /var/run/dhclient.eth0.pid -lf
> /var/
>   253 ?        Ss     0:00 /sbin/syslogd
>   256 ?        Ss     0:00 /sbin/klogd
>   261 ?        Ss     0:00 /usr/sbin/inetd
>   268 ?        Ss     0:00 /usr/sbin/sshd
>   274 ?        S      0:00 xfrd
>   275 ?        S      0:00 python /usr/sbin/xend start
>   288 ?        Ss     0:00 /usr/sbin/cron
>   327 tty1     Ss+    0:00 /sbin/getty 38400 tty1
>   328 tty2     Ss+    0:00 /sbin/getty 38400 tty2
>   329 tty3     Ss+    0:00 /sbin/getty 38400 tty3
>   330 tty4     Ss+    0:00 /sbin/getty 38400 tty4
>   331 tty5     Ss+    0:00 /sbin/getty 38400 tty5
>   332 tty6     Ss+    0:00 /sbin/getty 38400 tty6
>   333 ?        Ss     0:00 sshd: mat [priv]
>   335 ?        R      0:00 sshd: mat@pts/0
>   336 pts/0    Ss     0:00 -bash
>   344 pts/0    R+     0:00 ps ax
>
> i guess i could reduce the number of getty and get rid of dhclient (the
> adress of dom0 is a static one ...).
>
> It was my 2cents thoughts ...
> Matthieu
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Brian Wolfe]

Question from a security standpoint...

How exactly does this proc control relate to the memory parameter
specified when you star tthe domain? Can an unprivileged domain increase
it's memory footprint to MORE than the originally specified mem_size? Or
is this just a local control of up to the originally specified mem_size?

If it's arbitrary (not constrained by mem_size) then I'd have to point
out that some of us do NOT want this control to be allowed to go beyond
what mem_size originally sets the domain at in unprivileged domains.
Privileged domains would be exempted from this max limit of course. ;)

If i'm just not paying attention and this was already explained, feel
free to smack me. :)

On Thu, 2004-09-30 at 08:26, Mark A. Williamson wrote:
> > For people that still use linux 2.4 (like me) is there a tutorial or some
> > documentation about the ballon driver ?
> 
> Documentation, 2.6 support and facilities to control other domain's balloons 
> using xm in domain 0 are cooking and should be ready shortly.
> 
> In the meantime, do:
> # echo target_megabytes > /proc/xen/memory_target
> To set a new memory footprint.  There's a Linux kernel param (mem=..., I 
> think) for setting the maximum amount of memory a domain can handle, if 
> larger than it's initial allocation (i.e. if you want to start with x 
> megabytes but grow it later).
> 
> Trying to grow the domain's memory beyond what it's expecting (the initial 
> allocation or the kernel param if you specified that) will not work.
> 
> Use:
> # cat /proc/xen/memory_target
> To get the current footprint.
> 
> HTH,
> Mark
> 
> > > >  I have been wondering why my Domain-0 machine hasn't been using swap!
> > > >
> > > >  :o) So I should just dump that partition?
> > >
> > > If you've configured swap, domain 0 should be using just like any
> > > normal linux kernel. The balloon driver works in dom0 just like
> > > any other domain.
> >
> > It seems logical, but due to the fact that only a few process are running
> > ind dom0, swap file don't need to be huge is it ?
> >
> > > >  I suppose I'll have to be very rigorous when
> > > >  it comes to keeping that domain lean yes? (Especially as I only have
> > > >  4GB to play with!). Out side of basic OS functionality, are there any
> > > >  Xen or LVM2 related processes I have to account for when setting the
> > > >  amount of RAM I can use for Domain-0?
> > >
> > > -xen linux kernels don't have any special processes that take any
> > > noticeable extra resources.
> > >
> > > In domain 0 you'll have to worry about the footprint of xend, but
> > > it's not that big.
> >
> > When starting with xen one or two month ago, i've created a dom0 with only
> > 32 Mo of ram, and when i launched xend, it was so slow that i tought i had
> > a problem.
> > I didn't take time to investigate and didn't remember how much memory was
> > used and if the system was swaping or not.
> > I just have done a test now with my current xen putting 32768 Mo on xen
> > command line (not the kernel line in grub).
> > And the system booted a bit slower and i suspect that durring the launch of
> > xend it has done some swap.
> > free return this just after the boot :
> > mat@zeus:/$ free
> >              total       used       free     shared    buffers     cached
> > Mem:         32768      28572       4196          0          0       7172
> > -/+ buffers/cache:      21400      11368
> > Swap:       506008        312     505696
> >
> > Having this process:
> >   207 ?        Ss     0:00 dhclient -e -pf /var/run/dhclient.eth0.pid -lf
> > /var/
> >   253 ?        Ss     0:00 /sbin/syslogd
> >   256 ?        Ss     0:00 /sbin/klogd
> >   261 ?        Ss     0:00 /usr/sbin/inetd
> >   268 ?        Ss     0:00 /usr/sbin/sshd
> >   274 ?        S      0:00 xfrd
> >   275 ?        S      0:00 python /usr/sbin/xend start
> >   288 ?        Ss     0:00 /usr/sbin/cron
> >   327 tty1     Ss+    0:00 /sbin/getty 38400 tty1
> >   328 tty2     Ss+    0:00 /sbin/getty 38400 tty2
> >   329 tty3     Ss+    0:00 /sbin/getty 38400 tty3
> >   330 tty4     Ss+    0:00 /sbin/getty 38400 tty4
> >   331 tty5     Ss+    0:00 /sbin/getty 38400 tty5
> >   332 tty6     Ss+    0:00 /sbin/getty 38400 tty6
> >   333 ?        Ss     0:00 sshd: mat [priv]
> >   335 ?        R      0:00 sshd: mat@pts/0
> >   336 pts/0    Ss     0:00 -bash
> >   344 pts/0    R+     0:00 ps ax
> >
> > i guess i could reduce the number of getty and get rid of dhclient (the
> > adress of dom0 is a static one ...).
> >
> > It was my 2cents thoughts ...
> > Matthieu
> >
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > Use IT products in your business? Tell us what you think of them. Give us
> > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: RAM allocation and miscellaneous questions

[Mark A. Williamson]

Domains can't allocate more than you set using the 'xm maxmem' command.  By 
default AFAIK they can only shrink themselves.

I'm not sure this has been mentioned particularly explicitly alhtough the area 
got discussed a while back.

Cheers,
Mark

On Thursday 30 September 2004 20:21, Brian Wolfe wrote:
> Question from a security standpoint...
>
> How exactly does this proc control relate to the memory parameter
> specified when you star tthe domain? Can an unprivileged domain increase
> it's memory footprint to MORE than the originally specified mem_size? Or
> is this just a local control of up to the originally specified mem_size?
>
> If it's arbitrary (not constrained by mem_size) then I'd have to point
> out that some of us do NOT want this control to be allowed to go beyond
> what mem_size originally sets the domain at in unprivileged domains.
> Privileged domains would be exempted from this max limit of course. ;)
>
> If i'm just not paying attention and this was already explained, feel
> free to smack me. :)
>
> On Thu, 2004-09-30 at 08:26, Mark A. Williamson wrote:
> > > For people that still use linux 2.4 (like me) is there a tutorial or
> > > some documentation about the ballon driver ?
> >
> > Documentation, 2.6 support and facilities to control other domain's
> > balloons using xm in domain 0 are cooking and should be ready shortly.
> >
> > In the meantime, do:
> > # echo target_megabytes > /proc/xen/memory_target
> > To set a new memory footprint.  There's a Linux kernel param (mem=..., I
> > think) for setting the maximum amount of memory a domain can handle, if
> > larger than it's initial allocation (i.e. if you want to start with x
> > megabytes but grow it later).
> >
> > Trying to grow the domain's memory beyond what it's expecting (the
> > initial allocation or the kernel param if you specified that) will not
> > work.
> >
> > Use:
> > # cat /proc/xen/memory_target
> > To get the current footprint.
> >
> > HTH,
> > Mark
> >
> > > > >  I have been wondering why my Domain-0 machine hasn't been using
> > > > > swap!
> > > > >
> > > > >  :o) So I should just dump that partition?
> > > >
> > > > If you've configured swap, domain 0 should be using just like any
> > > > normal linux kernel. The balloon driver works in dom0 just like
> > > > any other domain.
> > >
> > > It seems logical, but due to the fact that only a few process are
> > > running ind dom0, swap file don't need to be huge is it ?
> > >
> > > > >  I suppose I'll have to be very rigorous when
> > > > >  it comes to keeping that domain lean yes? (Especially as I only
> > > > > have 4GB to play with!). Out side of basic OS functionality, are
> > > > > there any Xen or LVM2 related processes I have to account for when
> > > > > setting the amount of RAM I can use for Domain-0?
> > > >
> > > > -xen linux kernels don't have any special processes that take any
> > > > noticeable extra resources.
> > > >
> > > > In domain 0 you'll have to worry about the footprint of xend, but
> > > > it's not that big.
> > >
> > > When starting with xen one or two month ago, i've created a dom0 with
> > > only 32 Mo of ram, and when i launched xend, it was so slow that i
> > > tought i had a problem.
> > > I didn't take time to investigate and didn't remember how much memory
> > > was used and if the system was swaping or not.
> > > I just have done a test now with my current xen putting 32768 Mo on xen
> > > command line (not the kernel line in grub).
> > > And the system booted a bit slower and i suspect that durring the
> > > launch of xend it has done some swap.
> > > free return this just after the boot :
> > > mat@zeus:/$ free
> > >              total       used       free     shared    buffers    
> > > cached Mem:         32768      28572       4196          0          0  
> > >     7172 -/+ buffers/cache:      21400      11368
> > > Swap:       506008        312     505696
> > >
> > > Having this process:
> > >   207 ?        Ss     0:00 dhclient -e -pf /var/run/dhclient.eth0.pid
> > > -lf /var/
> > >   253 ?        Ss     0:00 /sbin/syslogd
> > >   256 ?        Ss     0:00 /sbin/klogd
> > >   261 ?        Ss     0:00 /usr/sbin/inetd
> > >   268 ?        Ss     0:00 /usr/sbin/sshd
> > >   274 ?        S      0:00 xfrd
> > >   275 ?        S      0:00 python /usr/sbin/xend start
> > >   288 ?        Ss     0:00 /usr/sbin/cron
> > >   327 tty1     Ss+    0:00 /sbin/getty 38400 tty1
> > >   328 tty2     Ss+    0:00 /sbin/getty 38400 tty2
> > >   329 tty3     Ss+    0:00 /sbin/getty 38400 tty3
> > >   330 tty4     Ss+    0:00 /sbin/getty 38400 tty4
> > >   331 tty5     Ss+    0:00 /sbin/getty 38400 tty5
> > >   332 tty6     Ss+    0:00 /sbin/getty 38400 tty6
> > >   333 ?        Ss     0:00 sshd: mat [priv]
> > >   335 ?        R      0:00 sshd: mat@pts/0
> > >   336 pts/0    Ss     0:00 -bash
> > >   344 pts/0    R+     0:00 ps ax
> > >
> > > i guess i could reduce the number of getty and get rid of dhclient (the
> > > adress of dom0 is a static one ...).
> > >
> > > It was my 2cents thoughts ...
> > > Matthieu
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.net email is sponsored by: IT Product Guide on
> > > ITManagersJournal Use IT products in your business? Tell us what you
> > > think of them. Give us Your Opinions, Get Free ThinkGeek Gift
> > > Certificates! Click to find out more
> > > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > Use IT products in your business? Tell us what you think of them. Give us
> > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
> > more http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl