memory hotplug for guests?

memory hotplug for guests?

[Tomasz Chmielewski]

Does KVM allow something like "memory hotplug" for its guests?


For example, lets says you are running several guests, and would like to 
start yet another one for a while - but have no free memory left.


Obviously, your guests are so important that you don't want to stop them 
- so you simply "hotplug remove" memory from a guest that has a lot of 
free memory left - and start a new guest.

When that new guest is no longer needed, and is stopped, you can 
"hotplug add" memory to the guest it was previously removed from.

Guest's kernel would of course need to support memory hotplugging, too.


Is it possible with KVM? If not, is such a feature planned?

I noticed it was only mentioned once or twice on the list:

http://article.gmane.org/gmane.comp.emulators.kvm.devel/712/

"Well, the _interface_ supports removing, the implementation does not :)

Everything was written in mind to allow memory hotplug."



-- 
Tomasz Chmielewski
http://wpkg.org

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Dor Laor]

>Does KVM allow something like "memory hotplug" for its guests?

It does not support.

>
>
>For example, lets says you are running several guests, and would like
to
>start yet another one for a while - but have no free memory left.
>

We have another solution for it that will soon be pushed into the
kernel:
It is the balloon driver solution.
Each guest runs a balloon driver, when the host needs to free up memory
a daemon with certain policy asks some of the guests to inflate their
balloon,
KVM frees their ballooned pages and the host free memory increases.
When the memory pressure relives, the balloons get deflate command.


>
>Obviously, your guests are so important that you don't want to stop
them
>- so you simply "hotplug remove" memory from a guest that has a lot of
>free memory left - and start a new guest.
>
>When that new guest is no longer needed, and is stopped, you can
>"hotplug add" memory to the guest it was previously removed from.
>
>Guest's kernel would of course need to support memory hotplugging, too.
>
>
>Is it possible with KVM? If not, is such a feature planned?
>
>I noticed it was only mentioned once or twice on the list:
>
>http://article.gmane.org/gmane.comp.emulators.kvm.devel/712/
>
>"Well, the _interface_ supports removing, the implementation does not
:)
>
>Everything was written in mind to allow memory hotplug."
>
>
>
>--
>Tomasz Chmielewski
>http://wpkg.org
>
>-----------------------------------------------------------------------
--
>Take Surveys. Earn Cash. Influence the Future of IT
>Join SourceForge.net's Techsay panel and you'll get the chance to share
>your
>opinions on IT & business topics through brief surveys-and earn cash
>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVD
EV
>_______________________________________________
>kvm-devel mailing list
>kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>https://lists.sourceforge.net/lists/listinfo/kvm-devel

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Heiko Carstens]

> >For example, lets says you are running several guests, and would like
> to
> >start yet another one for a while - but have no free memory left.
> >
> 
> We have another solution for it that will soon be pushed into the
> kernel:
> It is the balloon driver solution.
> Each guest runs a balloon driver, when the host needs to free up memory
> a daemon with certain policy asks some of the guests to inflate their
> balloon,
> KVM frees their ballooned pages and the host free memory increases.
> When the memory pressure relives, the balloons get deflate command.

You probably want to have a look at arch/s390/mm/cmm.c which is
exactly doing what you want. Of course the message interface to the
hypervisor is different to what you want to do.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Dor Laor]

>> >For example, lets says you are running several guests, and would
like
>> to
>> >start yet another one for a while - but have no free memory left.
>> >
>>
>> We have another solution for it that will soon be pushed into the
>> kernel:
>> It is the balloon driver solution.
>> Each guest runs a balloon driver, when the host needs to free up
memory
>> a daemon with certain policy asks some of the guests to inflate their
>> balloon,
>> KVM frees their ballooned pages and the host free memory increases.
>> When the memory pressure relives, the balloons get deflate command.
>
>You probably want to have a look at arch/s390/mm/cmm.c which is
>exactly doing what you want. Of course the message interface to the
>hypervisor is different to what you want to do.

Gee, thanks, I wasn't aware of it. I wrote the Linux driver, not yet
ready for mainline. This might change things and I'll might use/change
the cmm.
It will take a while because my queue is jammed with too much stuff.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Arnd Bergmann]

On Thursday 05 April 2007, Dor Laor wrote:
> >> >For example, lets says you are running several guests, and would
> like
> >> to
> >> >start yet another one for a while - but have no free memory left.
> >> >
> >>
> >> We have another solution for it that will soon be pushed into the
> >> kernel:
> >> It is the balloon driver solution.
> >> Each guest runs a balloon driver, when the host needs to free up
> memory
> >> a daemon with certain policy asks some of the guests to inflate their
> >> balloon,
> >> KVM frees their ballooned pages and the host free memory increases.
> >> When the memory pressure relives, the balloons get deflate command.
> >
> >You probably want to have a look at arch/s390/mm/cmm.c which is
> >exactly doing what you want. Of course the message interface to the
> >hypervisor is different to what you want to do.
> 
> Gee, thanks, I wasn't aware of it. I wrote the Linux driver, not yet
> ready for mainline. This might change things and I'll might use/change
> the cmm.
> It will take a while because my queue is jammed with too much stuff.

Another driver you might want to look at is the Xen balloon driver:
http://81.161.245.2/lxr/http/source/linux-2.6-xen-sparse/drivers/xen/balloon/balloon.c

The s390 driver is probably a better example though. One thing the
Xen one got wrong is that its interface requires you do an hcall in
order to get back a page that you gave up to the host. On s390, it
simply maps the host empty_zero_page as COW into the guest like
the initial memory, so you get a new page back with the regular
method whenever you write to the guest real page.

	Arnd <><

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Avi Kivity]

Arnd Bergmann wrote:
>
> Another driver you might want to look at is the Xen balloon driver:
> http://81.161.245.2/lxr/http/source/linux-2.6-xen-sparse/drivers/xen/balloon/balloon.c
>
> The s390 driver is probably a better example though. One thing the
> Xen one got wrong is that its interface requires you do an hcall in
> order to get back a page that you gave up to the host. 

Er, that's what we do :)

> On s390, it
> simply maps the host empty_zero_page as COW into the guest like
> the initial memory, so you get a new page back with the regular
> method whenever you write to the guest real page.
>   

This has the advantage of surviving a guest reboot, but the disadvantage 
of not allowing the host to nack the request.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Dor Laor]

>Arnd Bergmann wrote:
>>
>> Another driver you might want to look at is the Xen balloon driver:
>> http://81.161.245.2/lxr/http/source/linux-2.6-xen-
>sparse/drivers/xen/balloon/balloon.c
>>
>> The s390 driver is probably a better example though. One thing the
>> Xen one got wrong is that its interface requires you do an hcall in
>> order to get back a page that you gave up to the host.
>
>Er, that's what we do :)

I don't think it a disadvantage since it's happens rarely.
It's not performance sensitive thing.
Never the less it's worth learning/copying from the alternatives.

Currently the only memory-over-commit mechanism is the balloon.
In the future we will add all the wise spectrum of host demand pages,
shared pages, etc.

>
>> On s390, it
>> simply maps the host empty_zero_page as COW into the guest like
>> the initial memory, so you get a new page back with the regular
>> method whenever you write to the guest real page.
>>
>
>This has the advantage of surviving a guest reboot, but the
disadvantage
>of not allowing the host to nack the request.
>
>--
>error compiling committee.c: too many arguments to function
>
>
>-----------------------------------------------------------------------
--
>Take Surveys. Earn Cash. Influence the Future of IT
>Join SourceForge.net's Techsay panel and you'll get the chance to share
>your
>opinions on IT & business topics through brief surveys-and earn cash
>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVD
EV
>_______________________________________________
>kvm-devel mailing list
>kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>https://lists.sourceforge.net/lists/listinfo/kvm-devel

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Arnd Bergmann]

On Thursday 05 April 2007, Dor Laor wrote:
> Currently the only memory-over-commit mechanism is the balloon.
> In the future we will add all the wise spectrum of host demand pages,
> shared pages, etc.

Ok, just as another hint: you should definitely take a look at
pfault_interrupt() in arch/s390/mm/fault.c, which manages notifications
about host memory manangement, so that the guest can put a task to
sleep that accesses a page on the host block device, instead of suspending
the entire guest, while waiting for the page to come back.

arch/s390/mm/extmem.c has another very interesting concept, though the
hcall interfaces used there are not as flexible as they should be in
kvm. It's basically about mapping host files into the guest real address
space, e.g. for shared memory between guests.

	Arnd <><

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Avi Kivity]

Arnd Bergmann wrote:
> On Thursday 05 April 2007, Dor Laor wrote:
>   
>> Currently the only memory-over-commit mechanism is the balloon.
>> In the future we will add all the wise spectrum of host demand pages,
>> shared pages, etc.
>>     
>
> Ok, just as another hint: you should definitely take a look at
> pfault_interrupt() in arch/s390/mm/fault.c, which manages notifications
> about host memory manangement, so that the guest can put a task to
> sleep that accesses a page on the host block device, instead of suspending
> the entire guest, while waiting for the page to come back.
>   

It's a lot simpler than I would have guessed... this could be quite 
easily ported to kvm, once we have guest swap.


> arch/s390/mm/extmem.c has another very interesting concept, though the
> hcall interfaces used there are not as flexible as they should be in
> kvm. It's basically about mapping host files into the guest real address
> space, e.g. for shared memory between guests.
>   

This, too, is interesting, though it means guest apps need to be written 
to take advantage of it.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Arnd Bergmann]

On Thursday 05 April 2007, Avi Kivity wrote:
> > arch/s390/mm/extmem.c has another very interesting concept, though the
> > hcall interfaces used there are not as flexible as they should be in
> > kvm. It's basically about mapping host files into the guest real address
> > space, e.g. for shared memory between guests.
> >   
> 
> This, too, is interesting, though it means guest apps need to be written 
> to take advantage of it.

The primary user of this interface is the ext2 file system with the -oxip
mount option that can easily share read-only file systems across guests
without actually copying files into the page cache of each guest.

Another potential (future) user are cluster file systems like ocfs2 and
gfs2, but to be really efficient, these also need a specialized distributed
lock manager that communicates through hcalls instead of TCP/IP networking.

Simply mapping the shared memory as a character device is similar enough
that applications written for posix shared memory can easily be adapted,
but I'm not aware of anyone doing that currently.

	Arnd <><

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: memory hotplug for guests?

[Dor Laor]

>On Thursday 05 April 2007, Avi Kivity wrote:
>> > arch/s390/mm/extmem.c has another very interesting concept, though
the
>> > hcall interfaces used there are not as flexible as they should be
in
>> > kvm. It's basically about mapping host files into the guest real
>address
>> > space, e.g. for shared memory between guests.
>> >
>>
>> This, too, is interesting, though it means guest apps need to be
written
>> to take advantage of it.
>
>The primary user of this interface is the ext2 file system with the
-oxip
>mount option that can easily share read-only file systems across guests
>without actually copying files into the page cache of each guest.

That's nice. Thanks for the tip.

>
>Another potential (future) user are cluster file systems like ocfs2 and
>gfs2, but to be really efficient, these also need a specialized
distributed
>lock manager that communicates through hcalls instead of TCP/IP
networking.
>
>Simply mapping the shared memory as a character device is similar
enough
>that applications written for posix shared memory can easily be
adapted,
>but I'm not aware of anyone doing that currently.
>
>	Arnd <><

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV