Balloon driver for Linux/HVM

Balloon driver for Linux/HVM

[Rui Chu]

[-- Attachment #1.1: Type: text/plain, Size: 347 bytes --]

Hi,
I have noticed that, in the code of linux/drivers/xen/balloon.c, there exists the snippet as this:

static int __init balloon_init(void)
{
unsigned long pfn;
struct page *page;
if (!xen_pv_domain())
return -ENODEV;
.....
}

Does it means the driver will not work in HVM? If so, where is the HVN-enabled code for that?

2010-11-16 



Rui Chu 

[-- Attachment #1.2: Type: text/html, Size: 1394 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Balloon driver for Linux/HVM

[Rui Chu]

[-- Attachment #1.1: Type: text/plain, Size: 347 bytes --]

Hi,
I have noticed that, in the code of linux/drivers/xen/balloon.c, there exists the snippet as this:

static int __init balloon_init(void)
{
unsigned long pfn;
struct page *page;
if (!xen_pv_domain())
return -ENODEV;
.....
}

Does it means the driver will not work in HVM? If so, where is the HVN-enabled code for that?

2010-11-16 



Rui Chu 

[-- Attachment #1.2: Type: text/html, Size: 1394 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Balloon driver for Linux/HVM

[Chu Rui]

[-- Attachment #1.1: Type: text/plain, Size: 375 bytes --]

Hi,
I have noticed that, in the code of linux/drivers/xen/balloon.c, there
exists the snippet as this:

 static int __init balloon_init(void)
{
unsigned long pfn;
struct page *page;
 if (!xen_pv_domain())
return -ENODEV;
.....
}

Does it means the driver will not work in HVM? If so, where is the
HVN-enabled code for that?

2010-11-16
------------------------------
Rui Chu

[-- Attachment #1.2: Type: text/html, Size: 997 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Balloon driver for Linux/HVM

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 550 bytes --]

On Tue, 16 Nov 2010, Chu Rui wrote:
> Hi,
> I have noticed that, in the code of linux/drivers/xen/balloon.c, there exists the snippet as this:
>  
> static int __init balloon_init(void)
> {
> unsigned long pfn;
> struct page *page;
> if (!xen_pv_domain())
> return -ENODEV;
> .....
> }
>  
> Does it means the driver will not work in HVM? If so, where is the HVN-enabled code for that?

not yet, even though I have a patch ready to enable it:

git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git 2.6.36-rc7-pvhvm-v1

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re:Re: Balloon driver for Linux/HVM

[牛立新]

[-- Attachment #1.1: Type: text/plain, Size: 689 bytes --]

o, it's strange, the old version have no this limitation.




At 2010-11-16 19:35:50，"Stefano Stabellini" <stefano.stabellini@eu.citrix.com> wrote:

>On Tue, 16 Nov 2010, Chu Rui wrote:
>> Hi,
>> I have noticed that, in the code of linux/drivers/xen/balloon.c, there exists the snippet as this:
>>  
>> static int __init balloon_init(void)
>> {
>> unsigned long pfn;
>> struct page *page;
>> if (!xen_pv_domain())
>> return -ENODEV;
>> .....
>> }
>>  
>> Does it means the driver will not work in HVM? If so, where is the HVN-enabled code for that?
>
>not yet, even though I have a patch ready to enable it:
>
>git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git 2.6.36-rc7-pvhvm-v1

[-- Attachment #1.2: Type: text/html, Size: 1437 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Balloon driver for Linux/HVM

[George Dunlap]

2010/11/16 牛立新 <topperxin@126.com>:
> o, it's strange, the old version have no this limitation.

No; unfortunately a great deal of functionality present in "classic
xen" has been lost in the process of getting the core dom0 support
into the pvops kernel.  I think the plan is, once we have the
necessary changes to non-xen code pushed up stream, we can start
working on getting feature parity with classic xen.

>
>
> At 2010-11-16 19:35:50，"Stefano Stabellini" <stefano.stabellini@eu.citrix.com> wrote:
>
>>On Tue, 16 Nov 2010, Chu Rui wrote:
>>> Hi,
>>> I have noticed that, in the code of linux/drivers/xen/balloon.c, there exists the snippet as this:
>>>
>>> static int __init balloon_init(void)
>>> {
>>> unsigned long pfn;
>>> struct page *page;
>>> if (!xen_pv_domain())
>>> return -ENODEV;
>>> .....
>>> }
>>>
>>> Does it means the driver will not work in HVM? If so, where is the HVN-enabled code for that?
>>
>>not yet, even though I have a patch ready to enable it:
>>
>>git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git 2.6.36-rc7-pvhvm-v1
>
>
> ________________________________
> 网易163/126邮箱百分百兼容iphone ipad邮件收发
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>

Re: Re: Balloon driver for Linux/HVM

[Chu Rui]

[-- Attachment #1.1: Type: text/plain, Size: 2250 bytes --]

Thank you for your kind reply, George.

I am interested on the PoD memory. In my perspective, PoD mainly works in
the system initialization stage. Before the balloon driver begins to work,
it can limit the  memory consumption of the guests. However, after a while
the guest OS will commit more memory, but PoD cannot reclaim any more at
that time even when the committed pages is IO cache. While the balloon keeps
work all of the time.

Would you please tell me whether my thought is correct?

Actually, in my opinion, the guest IO cache is mostly useless, since the
Dom0 will cache the IO operations. Such a double-cache wastes the memory
resources. Is there any good idea for that like Transcendent Memory while
works with HVM?

在 2010年11月16日 下午8:56，George Dunlap <dunlapg@umich.edu>写道：

> 2010/11/16 牛立新 <topperxin@126.com>:
> > o, it's strange, the old version have no this limitation.
>
> No; unfortunately a great deal of functionality present in "classic
> xen" has been lost in the process of getting the core dom0 support
> into the pvops kernel.  I think the plan is, once we have the
> necessary changes to non-xen code pushed up stream, we can start
> working on getting feature parity with classic xen.
>
> >
> >
> > At 2010-11-16 19:35:50，"Stefano Stabellini" <
> stefano.stabellini@eu.citrix.com> wrote:
> >
> >>On Tue, 16 Nov 2010, Chu Rui wrote:
> >>> Hi,
> >>> I have noticed that, in the code of linux/drivers/xen/balloon.c, there
> exists the snippet as this:
> >>>
> >>> static int __init balloon_init(void)
> >>> {
> >>> unsigned long pfn;
> >>> struct page *page;
> >>> if (!xen_pv_domain())
> >>> return -ENODEV;
> >>> .....
> >>> }
> >>>
> >>> Does it means the driver will not work in HVM? If so, where is the
> HVN-enabled code for that?
> >>
> >>not yet, even though I have a patch ready to enable it:
> >>
> >>git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git2.6.36-rc7-pvhvm-v1
> >
> >
> > ________________________________
> > 网易163/126邮箱百分百兼容iphone ipad邮件收发
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> >
>

[-- Attachment #1.2: Type: text/html, Size: 3149 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: Balloon driver for Linux/HVM

[George Dunlap]

PoD is a mechanism designed for exactly one purpose: to allow a VM to
"boot ballooned".  It's designed to allow the guest to run on less
than the amount of memory it thinks it has until the balloon driver
loads.  After that, its job is done.  So you're right, it is designed
to work for the system initialization stage.

Regarding disk caching: I disagree about the guest IO cache.  I'd say
if one cache is to go, it should be the dom0 cache.  There are lots of
reasons for this:
* It's more fair: if you did all caching in dom0, then VM A might be
able to use almost the entire cache, leaving VM B without.  If each
guest does its own caching, then it's using its own resources and not
impacting someone else.
* I think the guest OS has a better idea what blocks need to be cached
and which don't.  It's much better to let that decision happen
locally, than to try to guess it from dom0, where we don't know
anything about processes, disk layout, &c.
* As Dan said, for write caching there's a consistency issue; better
to let the guest decide when it's safe not to write a page.
* If dom0 memory isn't being used for something else, it doesn't hurt
to have duplicate copies of things in memory.  But ideally guest disk
caching shouldn't take away from anything else on the system.

My $0.02. :-)

 -George

2010/11/16 Chu Rui <ruichu@gmail.com>:
> Thank you for your kind reply, George.
>
> I am interested on the PoD memory. In my perspective, PoD mainly works in
> the system initialization stage. Before the balloon driver begins to work,
> it can limit the  memory consumption of the guests. However, after a while
> the guest OS will commit more memory, but PoD cannot reclaim any more at
> that time even when the committed pages is IO cache. While the balloon keeps
> work all of the time.
>
> Would you please tell me whether my thought is correct?
>
> Actually, in my opinion, the guest IO cache is mostly useless, since the
> Dom0 will cache the IO operations. Such a double-cache wastes the memory
> resources. Is there any good idea for that like Transcendent Memory while
> works with HVM?
>
> 在 2010年11月16日 下午8:56，George Dunlap <dunlapg@umich.edu>写道：
>>
>> 2010/11/16 牛立新 <topperxin@126.com>:
>> > o, it's strange, the old version have no this limitation.
>>
>> No; unfortunately a great deal of functionality present in "classic
>> xen" has been lost in the process of getting the core dom0 support
>> into the pvops kernel.  I think the plan is, once we have the
>> necessary changes to non-xen code pushed up stream, we can start
>> working on getting feature parity with classic xen.
>>
>> >
>> >
>> > At 2010-11-16 19:35:50，"Stefano Stabellini"
>> > <stefano.stabellini@eu.citrix.com> wrote:
>> >
>> >>On Tue, 16 Nov 2010, Chu Rui wrote:
>> >>> Hi,
>> >>> I have noticed that, in the code of linux/drivers/xen/balloon.c, there
>> >>> exists the snippet as this:
>> >>>
>> >>> static int __init balloon_init(void)
>> >>> {
>> >>> unsigned long pfn;
>> >>> struct page *page;
>> >>> if (!xen_pv_domain())
>> >>> return -ENODEV;
>> >>> .....
>> >>> }
>> >>>
>> >>> Does it means the driver will not work in HVM? If so, where is the
>> >>> HVN-enabled code for that?
>> >>
>> >>not yet, even though I have a patch ready to enable it:
>> >>
>> >>git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git
>> >> 2.6.36-rc7-pvhvm-v1
>> >
>> >
>> > ________________________________
>> > 网易163/126邮箱百分百兼容iphone ipad邮件收发
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xensource.com
>> > http://lists.xensource.com/xen-devel
>> >
>> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>

Re: Re: Balloon driver for Linux/HVM

[George Dunlap]

I should point out also, that the balloon driver will most likely
(indirectly) pull memory from the guest's IO cache.  The balloon
driver asks the guest OS for a page, and the guest OS decides which
page is the least useful at this point.  If it doesn't have any free
pages, it will most likely choose either a page from the buffer cache,
or page out a not-recently-used application memory page.  The guest is
really in the best position to know which will have the least impact
on performance at that point.

Also, making dom0's buffer cache tiny and giving all the memory to the
guests allows the guests to use memory the way they see fit as well.
If the guest OS thinks having a larger buffer cache will be
advantageous, it can do that; OTOH, if it thinks giving almost all the
memory to processes will be more advantageous, it can do that too.
Having memory set aside for a dom0 guest-disk cache doesn't give the
guest that choice.

 -George

2010/11/17 George Dunlap <George.Dunlap@eu.citrix.com>:
> PoD is a mechanism designed for exactly one purpose: to allow a VM to
> "boot ballooned".  It's designed to allow the guest to run on less
> than the amount of memory it thinks it has until the balloon driver
> loads.  After that, its job is done.  So you're right, it is designed
> to work for the system initialization stage.
>
> Regarding disk caching: I disagree about the guest IO cache.  I'd say
> if one cache is to go, it should be the dom0 cache.  There are lots of
> reasons for this:
> * It's more fair: if you did all caching in dom0, then VM A might be
> able to use almost the entire cache, leaving VM B without.  If each
> guest does its own caching, then it's using its own resources and not
> impacting someone else.
> * I think the guest OS has a better idea what blocks need to be cached
> and which don't.  It's much better to let that decision happen
> locally, than to try to guess it from dom0, where we don't know
> anything about processes, disk layout, &c.
> * As Dan said, for write caching there's a consistency issue; better
> to let the guest decide when it's safe not to write a page.
> * If dom0 memory isn't being used for something else, it doesn't hurt
> to have duplicate copies of things in memory.  But ideally guest disk
> caching shouldn't take away from anything else on the system.
>
> My $0.02. :-)
>
>  -George
>
> 2010/11/16 Chu Rui <ruichu@gmail.com>:
>> Thank you for your kind reply, George.
>>
>> I am interested on the PoD memory. In my perspective, PoD mainly works in
>> the system initialization stage. Before the balloon driver begins to work,
>> it can limit the  memory consumption of the guests. However, after a while
>> the guest OS will commit more memory, but PoD cannot reclaim any more at
>> that time even when the committed pages is IO cache. While the balloon keeps
>> work all of the time.
>>
>> Would you please tell me whether my thought is correct?
>>
>> Actually, in my opinion, the guest IO cache is mostly useless, since the
>> Dom0 will cache the IO operations. Such a double-cache wastes the memory
>> resources. Is there any good idea for that like Transcendent Memory while
>> works with HVM?
>>
>> 在 2010年11月16日 下午8:56，George Dunlap <dunlapg@umich.edu>写道：
>>>
>>> 2010/11/16 牛立新 <topperxin@126.com>:
>>> > o, it's strange, the old version have no this limitation.
>>>
>>> No; unfortunately a great deal of functionality present in "classic
>>> xen" has been lost in the process of getting the core dom0 support
>>> into the pvops kernel.  I think the plan is, once we have the
>>> necessary changes to non-xen code pushed up stream, we can start
>>> working on getting feature parity with classic xen.
>>>
>>> >
>>> >
>>> > At 2010-11-16 19:35:50，"Stefano Stabellini"
>>> > <stefano.stabellini@eu.citrix.com> wrote:
>>> >
>>> >>On Tue, 16 Nov 2010, Chu Rui wrote:
>>> >>> Hi,
>>> >>> I have noticed that, in the code of linux/drivers/xen/balloon.c, there
>>> >>> exists the snippet as this:
>>> >>>
>>> >>> static int __init balloon_init(void)
>>> >>> {
>>> >>> unsigned long pfn;
>>> >>> struct page *page;
>>> >>> if (!xen_pv_domain())
>>> >>> return -ENODEV;
>>> >>> .....
>>> >>> }
>>> >>>
>>> >>> Does it means the driver will not work in HVM? If so, where is the
>>> >>> HVN-enabled code for that?
>>> >>
>>> >>not yet, even though I have a patch ready to enable it:
>>> >>
>>> >>git://xenbits.xen.org/people/sstabellini/linux-pvhvm.git
>>> >> 2.6.36-rc7-pvhvm-v1
>>> >
>>> >
>>> > ________________________________
>>> > 网易163/126邮箱百分百兼容iphone ipad邮件收发
>>> > _______________________________________________
>>> > Xen-devel mailing list
>>> > Xen-devel@lists.xensource.com
>>> > http://lists.xensource.com/xen-devel
>>> >
>>> >
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>
>

RE: Re: Balloon driver for Linux/HVM

[Dan Magenheimer]

> As we agreed ealier sometimes it is better to return memory to the
> system "directly" than allocate it as a backend for swap. I written
> PoC (with some suggestions from you) which works and cope very good
> with high memory demands. However, it is a long way (to some extent)
> to fully working implementation containing all features (ballooning,
> tmem and memory hotplug). I could do that. However at the beginnig
> we should agree on which kernel version I start development.
> For me it looks that
> git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
> repository, xen/balloon head is the best. What do you think about that
> ???

Sorry, I've had to refocus my energy on other non-virtualization
tmem-related issues* for awhile so haven't had a chance to work
on the future balloon driver features you are proposing.

What is the difference between 2.6.36 balloon driver and Jeremy's
xen/balloon tree head?  I think ideally the kernel version for
development should be the latest (2.6.36).
 
* Search for cleancache in http://lwn.net/Articles/412687/

Re: Re: Balloon driver for Linux/HVM

[Daniel Kiper]

Hi,

On Thu, Nov 18, 2010 at 12:32:35PM -0800, Dan Magenheimer wrote:
> What is the difference between 2.6.36 balloon driver and Jeremy's
> xen/balloon tree head?  I think ideally the kernel version for
> development should be the latest (2.6.36).

I know that the best practice is to base new work on the
latest stable kernel version, however I am not sure that it is
the case there. xen/balloon head contains most of neweset fixes
and improvments which are not merged into current stable till
now. That is why I think we should start from this and later
merge it with current stable. However, maybe Jeremy may have
better view of which version would be better for the start
of development.

Daniel

Re: Balloon driver for Linux/HVM

[Jeremy Fitzhardinge]

On 11/22/2010 01:51 AM, Daniel Kiper wrote:
> Hi,
>
> On Thu, Nov 18, 2010 at 12:32:35PM -0800, Dan Magenheimer wrote:
>> What is the difference between 2.6.36 balloon driver and Jeremy's
>> xen/balloon tree head?  I think ideally the kernel version for
>> development should be the latest (2.6.36).
> I know that the best practice is to base new work on the
> latest stable kernel version, however I am not sure that it is
> the case there. xen/balloon head contains most of neweset fixes
> and improvments which are not merged into current stable till
> now. That is why I think we should start from this and later
> merge it with current stable. However, maybe Jeremy may have
> better view of which version would be better for the start
> of development.
>

The big difference between xen/balloon and current mainline is hugepage
support, which is not yet in an upstreamable state.  Aside from that,
current upstream balloon driver (in current git, post 2.6.37-rc3) is
equivalent, and would make a good base for development.

The main difference between .37-rc and .36 is support for boot-time
ballooning, but the changes to the balloon driver to support that are
small, so .36 would probably make a reasonable base as well.

    J

Re: Balloon driver for Linux/HVM

[Daniel Kiper]

Hi,

Sorry for late reply, however I am very busy now.

On Wed, Nov 24, 2010 at 10:53:58AM -0800, Jeremy Fitzhardinge wrote:
> On 11/22/2010 01:51 AM, Daniel Kiper wrote:
> > Hi,
> >
> > On Thu, Nov 18, 2010 at 12:32:35PM -0800, Dan Magenheimer wrote:
> >> What is the difference between 2.6.36 balloon driver and Jeremy's
> >> xen/balloon tree head?  I think ideally the kernel version for
> >> development should be the latest (2.6.36).
> > I know that the best practice is to base new work on the
> > latest stable kernel version, however I am not sure that it is
> > the case there. xen/balloon head contains most of neweset fixes
> > and improvments which are not merged into current stable till
> > now. That is why I think we should start from this and later
> > merge it with current stable. However, maybe Jeremy may have
> > better view of which version would be better for the start
> > of development.
> >
>
> The big difference between xen/balloon and current mainline is hugepage
> support, which is not yet in an upstreamable state.  Aside from that,
> current upstream balloon driver (in current git, post 2.6.37-rc3) is
> equivalent, and would make a good base for development.
>
> The main difference between .37-rc and .36 is support for boot-time
> ballooning, but the changes to the balloon driver to support that are
> small, so .36 would probably make a reasonable base as well.

Thx. I will prepare patches for 2.6.36.

Daniel