memory hotplug for domUs

memory hotplug for domUs

[Juergen Gross]

Recently Jim asked me why he can use "xl mem-max" to raise the allowed
memory size of a domain in the hypervisor above the configured maxmem
limit of the domain, but not use "xl mem-set" to balloon the domain up
to this value later.

I thought libxl_domain_setmaxmem() being buggy as it doesn't modify the
memory/static-max value in Xenstore and posted a patch adding the
Xenstore modification.

Later I had an IRC discussion with Ian on #xendevel with the following
outcome:

xl mem-max is defined to do things as it does without my patch. OTOH
this is documented very poorly. I'll send a patch enhancing the xl man
page in this regard.

For support of memory hotplug in a domain (adding memory above the
current defined maximum memory as in Xenstore memory/static-max) we
decided to add a new option to xl.

Basically doing such a kind of memory hotplug requires the capability
of the guest to support this feature. PV Linux kernel has support for
it if the kernel has been built with CONFIG_XEN_BALLOON_MEMORY_HOTPLUG
set. For HVM guests support is more difficult as there has to be some
reserved space in the memory map of the guest for hotplugged memory.

Ian suggested to let the guest write a Xenstore entry indicating
support of memory hotplug in order for Xen tools to decide whether such
an operation should be supported. As this entry isn't being written by
current kernels which do support hotplug there has to be some kind of
"force" flag to override the test for presence of the Xenstore entry.

We first thought to enhance "xl mem-set", but after some more thinking
about it I'd rather add a new xl command, e.g. "mem-add" (we could later
even add "mem-remove" to support memory unplug).

xl mem-add would add the specified amount of memory to a running domain
and it would raise memory/static-max accordingly. The first version
would support PV domains only, but future support for HVM could be
possible (the domain configuration would have to be enhanced to create
a memory map entry indicating hot-pluggable memory). Other future
enhancements would include the possibility to add memory to a vnuma
node of the domain, possibly taking the memory from a specified numa
node of the host.

In case this proposal is accepted I can write patches (libxl and Linux
kernel).


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Andrew Cooper]

On 20/01/17 09:26, Juergen Gross wrote:
> Recently Jim asked me why he can use "xl mem-max" to raise the allowed
> memory size of a domain in the hypervisor above the configured maxmem
> limit of the domain, but not use "xl mem-set" to balloon the domain up
> to this value later.
>
> I thought libxl_domain_setmaxmem() being buggy as it doesn't modify the
> memory/static-max value in Xenstore and posted a patch adding the
> Xenstore modification.
>
> Later I had an IRC discussion with Ian on #xendevel with the following
> outcome:
>
> xl mem-max is defined to do things as it does without my patch. OTOH
> this is documented very poorly. I'll send a patch enhancing the xl man
> page in this regard.
>
> For support of memory hotplug in a domain (adding memory above the
> current defined maximum memory as in Xenstore memory/static-max) we
> decided to add a new option to xl.
>
> Basically doing such a kind of memory hotplug requires the capability
> of the guest to support this feature. PV Linux kernel has support for
> it if the kernel has been built with CONFIG_XEN_BALLOON_MEMORY_HOTPLUG
> set. For HVM guests support is more difficult as there has to be some
> reserved space in the memory map of the guest for hotplugged memory.
>
> Ian suggested to let the guest write a Xenstore entry indicating
> support of memory hotplug in order for Xen tools to decide whether such
> an operation should be supported. As this entry isn't being written by
> current kernels which do support hotplug there has to be some kind of
> "force" flag to override the test for presence of the Xenstore entry.
>
> We first thought to enhance "xl mem-set", but after some more thinking
> about it I'd rather add a new xl command, e.g. "mem-add" (we could later
> even add "mem-remove" to support memory unplug).
>
> xl mem-add would add the specified amount of memory to a running domain
> and it would raise memory/static-max accordingly. The first version
> would support PV domains only, but future support for HVM could be
> possible (the domain configuration would have to be enhanced to create
> a memory map entry indicating hot-pluggable memory). Other future
> enhancements would include the possibility to add memory to a vnuma
> node of the domain, possibly taking the memory from a specified numa
> node of the host.
>
> In case this proposal is accepted I can write patches (libxl and Linux
> kernel).

For PV guests, this sounds plausible.

For HVM/PVH guests however, ACPI Memory Hotplug.  That way it will work
for all operating systems, not just Linux.  (Lets please not waste time
building a somewhat-square wheel which only fits on a Linux shaped axle :) )

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Konrad Rzeszutek Wilk]

On Fri, Jan 20, 2017 at 11:08:54AM +0000, Andrew Cooper wrote:
> On 20/01/17 09:26, Juergen Gross wrote:

Cc-ing Daniel as he wrote the Xen Balloon memory hotplug code.

> > Recently Jim asked me why he can use "xl mem-max" to raise the allowed
> > memory size of a domain in the hypervisor above the configured maxmem
> > limit of the domain, but not use "xl mem-set" to balloon the domain up
> > to this value later.
> >
> > I thought libxl_domain_setmaxmem() being buggy as it doesn't modify the
> > memory/static-max value in Xenstore and posted a patch adding the
> > Xenstore modification.
> >
> > Later I had an IRC discussion with Ian on #xendevel with the following
> > outcome:
> >
> > xl mem-max is defined to do things as it does without my patch. OTOH
> > this is documented very poorly. I'll send a patch enhancing the xl man
> > page in this regard.
> >
> > For support of memory hotplug in a domain (adding memory above the
> > current defined maximum memory as in Xenstore memory/static-max) we
> > decided to add a new option to xl.
> >
> > Basically doing such a kind of memory hotplug requires the capability
> > of the guest to support this feature. PV Linux kernel has support for
> > it if the kernel has been built with CONFIG_XEN_BALLOON_MEMORY_HOTPLUG
> > set. For HVM guests support is more difficult as there has to be some
> > reserved space in the memory map of the guest for hotplugged memory.
> >
> > Ian suggested to let the guest write a Xenstore entry indicating
> > support of memory hotplug in order for Xen tools to decide whether such
> > an operation should be supported. As this entry isn't being written by
> > current kernels which do support hotplug there has to be some kind of
> > "force" flag to override the test for presence of the Xenstore entry.
> >
> > We first thought to enhance "xl mem-set", but after some more thinking
> > about it I'd rather add a new xl command, e.g. "mem-add" (we could later
> > even add "mem-remove" to support memory unplug).
> >
> > xl mem-add would add the specified amount of memory to a running domain
> > and it would raise memory/static-max accordingly. The first version
> > would support PV domains only, but future support for HVM could be
> > possible (the domain configuration would have to be enhanced to create
> > a memory map entry indicating hot-pluggable memory). Other future
> > enhancements would include the possibility to add memory to a vnuma
> > node of the domain, possibly taking the memory from a specified numa
> > node of the host.
> >
> > In case this proposal is accepted I can write patches (libxl and Linux
> > kernel).
> 
> For PV guests, this sounds plausible.
> 
> For HVM/PVH guests however, ACPI Memory Hotplug.  That way it will work
> for all operating systems, not just Linux.  (Lets please not waste time
> building a somewhat-square wheel which only fits on a Linux shaped axle :) )
> 
> ~Andrew
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Boris Ostrovsky]

>> For HVM/PVH guests however, ACPI Memory Hotplug.  That way it will work
>> for all operating systems, not just Linux.  (Lets please not waste time
>> building a somewhat-square wheel which only fits on a Linux shaped axle :) )

What about PVH dom0? I suspect we will have the same problem with ACPI
memory hotplug there as we have with ACPI CPU hotplug.

Of course, we can decide not to support ACPI hotplug for dom0 (which I
think is reasonable).

-boris



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Roger Pau Monné]

On Fri, Jan 20, 2017 at 09:19:48AM -0500, Boris Ostrovsky wrote:
> 
> >> For HVM/PVH guests however, ACPI Memory Hotplug.  That way it will work
> >> for all operating systems, not just Linux.  (Lets please not waste time
> >> building a somewhat-square wheel which only fits on a Linux shaped axle :) )
> 
> What about PVH dom0? I suspect we will have the same problem with ACPI
> memory hotplug there as we have with ACPI CPU hotplug.
> 
> Of course, we can decide not to support ACPI hotplug for dom0 (which I
> think is reasonable).

But IIRC we don't support ACPI memory hotplug for any kind of guests (not even
HVM), all guest types use the PV balloon driver, that relies on xenstore.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Juergen Gross]

On 20/01/17 15:27, Roger Pau Monné wrote:
> On Fri, Jan 20, 2017 at 09:19:48AM -0500, Boris Ostrovsky wrote:
>>
>>>> For HVM/PVH guests however, ACPI Memory Hotplug.  That way it will work
>>>> for all operating systems, not just Linux.  (Lets please not waste time
>>>> building a somewhat-square wheel which only fits on a Linux shaped axle :) )
>>
>> What about PVH dom0? I suspect we will have the same problem with ACPI
>> memory hotplug there as we have with ACPI CPU hotplug.
>>
>> Of course, we can decide not to support ACPI hotplug for dom0 (which I
>> think is reasonable).
> 
> But IIRC we don't support ACPI memory hotplug for any kind of guests (not even
> HVM), all guest types use the PV balloon driver, that relies on xenstore.

Adding memory above maxmem requires additional efforts for HVM. We
could use the PV balloon driver or ACPI hotplug. I'm not so sure using
ACPI hotplug is the best choice for PVH. This should be made dependent
on the qemu functionality we can reuse for HVM.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Boris Ostrovsky]

On 01/20/2017 09:27 AM, Roger Pau Monné wrote:
> On Fri, Jan 20, 2017 at 09:19:48AM -0500, Boris Ostrovsky wrote:
>>>> For HVM/PVH guests however, ACPI Memory Hotplug.  That way it will work
>>>> for all operating systems, not just Linux.  (Lets please not waste time
>>>> building a somewhat-square wheel which only fits on a Linux shaped axle :) )
>> What about PVH dom0? I suspect we will have the same problem with ACPI
>> memory hotplug there as we have with ACPI CPU hotplug.
>>
>> Of course, we can decide not to support ACPI hotplug for dom0 (which I
>> think is reasonable).
> But IIRC we don't support ACPI memory hotplug for any kind of guests (not even
> HVM), all guest types use the PV balloon driver, that relies on xenstore.


Right. I was responding to Andrew's suggestion to do so in the future
(which is what I think what he meant).


-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Ian Jackson]

Juergen Gross writes ("memory hotplug for domUs"):
> We first thought to enhance "xl mem-set", but after some more thinking
> about it I'd rather add a new xl command, e.g. "mem-add" (we could later
> even add "mem-remove" to support memory unplug).

Why ?  Why would xl mem-set not automatically do the right thing ?

This problem with lack of support (and indeed requirementse to do
hotplug things) is a detail that in the long term should not be a
problem, and in the meantime can be dealt with by a force option.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Juergen Gross]

On 20/01/17 16:54, Ian Jackson wrote:
> Juergen Gross writes ("memory hotplug for domUs"):
>> We first thought to enhance "xl mem-set", but after some more thinking
>> about it I'd rather add a new xl command, e.g. "mem-add" (we could later
>> even add "mem-remove" to support memory unplug).
> 
> Why ?  Why would xl mem-set not automatically do the right thing ?

How would you specify the numa node to add the memory to?

Especially for removal of memory ballooning and hotplug are _very_
different.

> This problem with lack of support (and indeed requirementse to do
> hotplug things) is a detail that in the long term should not be a
> problem, and in the meantime can be dealt with by a force option.

I think it would be clearer with new commands for hotplugging. I
don't feel very strong about this, so in case everyone else is fine
with handling everything via mem-set I won't object.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Ian Jackson]

Juergen Gross writes ("Re: memory hotplug for domUs"):
> On 20/01/17 16:54, Ian Jackson wrote:
> > Why ?  Why would xl mem-set not automatically do the right thing ?
> 
> How would you specify the numa node to add the memory to?

You don't have to specify a numa node when you create a domain.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Juergen Gross]

On 20/01/17 17:37, Ian Jackson wrote:
> Juergen Gross writes ("Re: memory hotplug for domUs"):
>> On 20/01/17 16:54, Ian Jackson wrote:
>>> Why ?  Why would xl mem-set not automatically do the right thing ?
>>
>> How would you specify the numa node to add the memory to?
> 
> You don't have to specify a numa node when you create a domain.

But you can do it.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Jim Fehlig]

On 01/20/2017 09:06 AM, Juergen Gross wrote:
> On 20/01/17 16:54, Ian Jackson wrote:
>> Juergen Gross writes ("memory hotplug for domUs"):
>>> We first thought to enhance "xl mem-set", but after some more thinking
>>> about it I'd rather add a new xl command, e.g. "mem-add" (we could later
>>> even add "mem-remove" to support memory unplug).
>>
>> Why ?  Why would xl mem-set not automatically do the right thing ?
>
> How would you specify the numa node to add the memory to?

And the host numa node providing the memory?

> Especially for removal of memory ballooning and hotplug are _very_
> different.

I agree. libvirt already models ballooning and hot(un)plug differently.

For ballooning, typical VM configuration is

   <memory unit='KiB'>16777216</memory>
   <currentMemory unit='KiB'>2097152</currentMemory>
   <memballoon model='virtio|xen'>

'virsh setmaxmem' and 'virsh setmem' allow adjusting runtime and/or persistent 
configuration.

For hot(un)plug (currently only supported by the qemu driver) typical 
configuration is

   <maxMemory slots='16' unit='KiB'>16777216</maxMemory>
   <devices>
     <memory model='dimm'>
       <source>
         <nodemask>0-1</nodemask>
       </source>
       <target>
         <size unit='KiB'>1048576</size>
         <node>0</node>
       </target>
     </memory>
     ...
</devices>

which describes a VM with 16 dimm slots with total capacity of 16G. One slot is 
filled with a 1G memory module whose memory must come from host numa nodes 0 or 
1, and is provided to numa node 0 in the VM.

'virsh attach-device' and virsh detach-device' allow adding or removing memory 
modules from the dimms. AIUI, under the covers qemu uses ACPI memory hotplug.

>
>> This problem with lack of support (and indeed requirementse to do
>> hotplug things) is a detail that in the long term should not be a
>> problem, and in the meantime can be dealt with by a force option.
>
> I think it would be clearer with new commands for hotplugging. I
> don't feel very strong about this, so in case everyone else is fine
> with handling everything via mem-set I won't object.

If ACPI memory hotplug is indeed the goal, then I agree new commands should be 
used since it is quite different than ballooning.

Regards,
Jim


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2471 bytes --]

On Tue, 2017-01-24 at 16:03 -0700, Jim Fehlig wrote:
> On 01/20/2017 09:06 AM, Juergen Gross wrote:
> > On 20/01/17 16:54, Ian Jackson wrote:
> > > Juergen Gross writes ("memory hotplug for domUs"):
> > > > We first thought to enhance "xl mem-set", but after some more
> > > > thinking
> > > > about it I'd rather add a new xl command, e.g. "mem-add" (we
> > > > could later
> > > > even add "mem-remove" to support memory unplug).
> > > 
> > > Why ?  Why would xl mem-set not automatically do the right thing
> > > ?
> > 
> > How would you specify the numa node to add the memory to?
> 
> And the host numa node providing the memory?
> 
Exactly!

I've actually always been a lot confused by all these mem-max,
mem-set, set-mem-max, max-set-mem, static-set-mem-max, set-mem-static-
max, etc ( :-P ), so I'm not really comfortable giving advices.

However, I strongly agree on the fact that this new capability need to
be (v)NUMA aware (at least, potentially, for when we'll have complete
support for that in all the moving parts). Or we risk having to revisit
and potentially change again and re-document the behavior!

If a new command is not desirable, can't we add options and parameters
to `xl mem-set', and fallback to some well defined default if they're
not there ?

> > I think it would be clearer with new commands for hotplugging. I
> > don't feel very strong about this, so in case everyone else is fine
> > with handling everything via mem-set I won't object.
> 
> If ACPI memory hotplug is indeed the goal, then I agree new commands
> should be 
> used since it is quite different than ballooning.
> 
Yeah, but again, the multiplexing can also happen inside of xl or
libxl, basing on parameters, etc.

I think the biggest issue here --as in, the thing we're most in lack of
right now-- is is to come up with a well defined and well documented
behavior, much rather than any technical aspect.

FWIW, I'm a bit two minded, as I agree with Juergen and Jim that adding
new commands may be cleaner, but I wonder whether having
_yet_another_mem_foobar_thing_ would not reveal too confusing and hard
to use from a user perspective...

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Juergen Gross]

On 25/01/17 14:02, Dario Faggioli wrote:
> On Tue, 2017-01-24 at 16:03 -0700, Jim Fehlig wrote:
>> On 01/20/2017 09:06 AM, Juergen Gross wrote:
>>> On 20/01/17 16:54, Ian Jackson wrote:
>>>> Juergen Gross writes ("memory hotplug for domUs"):
>>>>> We first thought to enhance "xl mem-set", but after some more
>>>>> thinking
>>>>> about it I'd rather add a new xl command, e.g. "mem-add" (we
>>>>> could later
>>>>> even add "mem-remove" to support memory unplug).
>>>> Why ?  Why would xl mem-set not automatically do the right thing
>>>> ?
>>> How would you specify the numa node to add the memory to?
>> And the host numa node providing the memory?
>>
> Exactly!
> 
> I've actually always been a lot confused by all these mem-max,
> mem-set, set-mem-max, max-set-mem, static-set-mem-max, set-mem-static-
> max, etc ( :-P ), so I'm not really comfortable giving advices.
> 
> However, I strongly agree on the fact that this new capability need to
> be (v)NUMA aware (at least, potentially, for when we'll have complete
> support for that in all the moving parts). Or we risk having to revisit
> and potentially change again and re-document the behavior!
> 
> If a new command is not desirable, can't we add options and parameters
> to `xl mem-set', and fallback to some well defined default if they're
> not there ?

While the syntax for "xl mem-add" (or however we would call it) could
be rather clear and simple:

xl mem-add <domain> <mem-amount> [--host-node <xy>] [--guest-node <ab>]

I don't see how this would be the case for xl mem-set. Imagine a guest
ballooned down and now you are setting a memory size above its current
maxmem value. What should be done? Should the guest first be ballooned
up and then a hotplug operation is to happen? Or should all the memory
missing be added via hotplug? Should xl mem-set with numa nodes
specified be rejected in case the requested size could be achieved by
ballooning up?

In case of removing memory: should ballooning or hotplug or a
combination of both be used?

With xl mem-add/mem-remove this would be very clear. We could even
support adding pre-ballooned memory (might be interesting in case a
guest supports only hotplugging of specific memory sizes but one
wants to add less host memory to the guest).

> 
>>> I think it would be clearer with new commands for hotplugging. I
>>> don't feel very strong about this, so in case everyone else is fine
>>> with handling everything via mem-set I won't object.
>> If ACPI memory hotplug is indeed the goal, then I agree new commands
>> should be 
>> used since it is quite different than ballooning.
>>
> Yeah, but again, the multiplexing can also happen inside of xl or
> libxl, basing on parameters, etc.
> 
> I think the biggest issue here --as in, the thing we're most in lack of
> right now-- is is to come up with a well defined and well documented
> behavior, much rather than any technical aspect.

Right.

> FWIW, I'm a bit two minded, as I agree with Juergen and Jim that adding
> new commands may be cleaner, but I wonder whether having
> _yet_another_mem_foobar_thing_ would not reveal too confusing and hard
> to use from a user perspective...

I believe trying to do it all via mem-set with magic inside would be
even more confusing. A clear set of commands with each having well
defined semantics is much clearer IMO.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Ian Jackson]

Juergen Gross writes ("Re: [Xen-devel] memory hotplug for domUs"):
> While the syntax for "xl mem-add" (or however we would call it) could
> be rather clear and simple:
> 
> xl mem-add <domain> <mem-amount> [--host-node <xy>] [--guest-node <ab>]
> 
> I don't see how this would be the case for xl mem-set. Imagine a guest
> ballooned down and now you are setting a memory size above its current
> maxmem value. What should be done? Should the guest first be ballooned
> up and then a hotplug operation is to happen? Or should all the memory
> missing be added via hotplug? Should xl mem-set with numa nodes
> specified be rejected in case the requested size could be achieved by
> ballooning up?
> 
> In case of removing memory: should ballooning or hotplug or a
> combination of both be used?
> 
> With xl mem-add/mem-remove this would be very clear. We could even
> support adding pre-ballooned memory (might be interesting in case a
> guest supports only hotplugging of specific memory sizes but one
> wants to add less host memory to the guest).

The problem I have with this approach is that from most users' point
of view, static-max is not a very meaningful concept.  If the user has
to `mem-add' iff the new amount is bigger than the static-max, this
makes the whole thing very odd to use.

I do see that these numa node allocation possibilities make matters
more complicated, but the usual case is that the user will want the
default numa allocation strategy; and the next usual case is that they
specified an allocation strategy at domain creation time, in the
domain config file.  I don't think the dynamic numa special cases
should be allowed to force us into making an interface which is
unnatural in the usual cases.

Bottom line: I want `xl mem-set', without further options, when
supported by the guest, to automatically do any memory hotplug that
may be required, according to the domain's existing numa allocation
policy (whatever that may be).

For users who want to adjust the numa allocation strategy during the
life of the domain, I am content that there should be more complex
mechanisms.  So I don't think we need numa options to mem-set unless
that is the most convenient interface.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Juergen Gross]

On 03/02/17 13:19, Ian Jackson wrote:
> Juergen Gross writes ("Re: [Xen-devel] memory hotplug for domUs"):
>> While the syntax for "xl mem-add" (or however we would call it) could
>> be rather clear and simple:
>>
>> xl mem-add <domain> <mem-amount> [--host-node <xy>] [--guest-node <ab>]
>>
>> I don't see how this would be the case for xl mem-set. Imagine a guest
>> ballooned down and now you are setting a memory size above its current
>> maxmem value. What should be done? Should the guest first be ballooned
>> up and then a hotplug operation is to happen? Or should all the memory
>> missing be added via hotplug? Should xl mem-set with numa nodes
>> specified be rejected in case the requested size could be achieved by
>> ballooning up?
>>
>> In case of removing memory: should ballooning or hotplug or a
>> combination of both be used?
>>
>> With xl mem-add/mem-remove this would be very clear. We could even
>> support adding pre-ballooned memory (might be interesting in case a
>> guest supports only hotplugging of specific memory sizes but one
>> wants to add less host memory to the guest).
> 
> The problem I have with this approach is that from most users' point
> of view, static-max is not a very meaningful concept.  If the user has
> to `mem-add' iff the new amount is bigger than the static-max, this
> makes the whole thing very odd to use.
> 
> I do see that these numa node allocation possibilities make matters
> more complicated, but the usual case is that the user will want the
> default numa allocation strategy; and the next usual case is that they
> specified an allocation strategy at domain creation time, in the
> domain config file.  I don't think the dynamic numa special cases
> should be allowed to force us into making an interface which is
> unnatural in the usual cases.
> 
> Bottom line: I want `xl mem-set', without further options, when
> supported by the guest, to automatically do any memory hotplug that
> may be required, according to the domain's existing numa allocation
> policy (whatever that may be).
> 
> For users who want to adjust the numa allocation strategy during the
> life of the domain, I am content that there should be more complex
> mechanisms.  So I don't think we need numa options to mem-set unless
> that is the most convenient interface.

Okay. This means for me:

As user interface we are starting with "xl mem-set" which will use
memory hotplug if necessary and supported by the guest (guest indicates
support via a Xenstore entry). A "force" parameter can be used with
"xl mem-set" in case there is no support indicated in Xenstore. In the
beginning memory hotplug will be supported for pv guests only.

The memory hotplug functionality is offered by a libxl function which
can be used by e.g. libvirt and, of course, by "xl mem-set". Support
of memory hotplug by a guest can be tested by another libxl function.

In case it is needed we can add more xl commands in order to support
e.g. special numa functionality.

Should we already consider numa parameters at the libxl interface
(for now accepting "ANY" parameter only)?


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Ian Jackson]

Juergen Gross writes ("Re: [Xen-devel] memory hotplug for domUs"):
> As user interface we are starting with "xl mem-set" which will use
> memory hotplug if necessary and supported by the guest (guest indicates
> support via a Xenstore entry). A "force" parameter can be used with
> "xl mem-set" in case there is no support indicated in Xenstore. In the
> beginning memory hotplug will be supported for pv guests only.

Right.

> The memory hotplug functionality is offered by a libxl function which
> can be used by e.g. libvirt and, of course, by "xl mem-set". Support
> of memory hotplug by a guest can be tested by another libxl function.

I would be inclined to suggest that the automatic memory hotplug
should be implemented in libxl.  I hesistate to suggest atht the
existing libxl_set_memory_target should do this, so perhaps "xl
mem-set" should call a new function.

> In case it is needed we can add more xl commands in order to support
> e.g. special numa functionality.

Or additional options.

> Should we already consider numa parameters at the libxl interface
> (for now accepting "ANY" parameter only)?

Yes, certainly.

Mayb we should ahve a new version of libxl_set_memory_target which
takes an idl-generated parameter block rather than the existing two
paraemters, and also new optional numa and hotplus options ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: memory hotplug for domUs

[Juergen Gross]

On 03/02/17 14:23, Ian Jackson wrote:
> Juergen Gross writes ("Re: [Xen-devel] memory hotplug for domUs"):
>> As user interface we are starting with "xl mem-set" which will use
>> memory hotplug if necessary and supported by the guest (guest indicates
>> support via a Xenstore entry). A "force" parameter can be used with
>> "xl mem-set" in case there is no support indicated in Xenstore. In the
>> beginning memory hotplug will be supported for pv guests only.
> 
> Right.
> 
>> The memory hotplug functionality is offered by a libxl function which
>> can be used by e.g. libvirt and, of course, by "xl mem-set". Support
>> of memory hotplug by a guest can be tested by another libxl function.
> 
> I would be inclined to suggest that the automatic memory hotplug
> should be implemented in libxl.  I hesistate to suggest atht the
> existing libxl_set_memory_target should do this, so perhaps "xl
> mem-set" should call a new function.

Sure, why not. I'd still like to have an explicit memory hotplug
interface on libxl level. This can be used by the new libxl function
you are suggesting.

>> In case it is needed we can add more xl commands in order to support
>> e.g. special numa functionality.
> 
> Or additional options.

Right.

>> Should we already consider numa parameters at the libxl interface
>> (for now accepting "ANY" parameter only)?
> 
> Yes, certainly.
> 
> Mayb we should ahve a new version of libxl_set_memory_target which
> takes an idl-generated parameter block rather than the existing two
> paraemters, and also new optional numa and hotplus options ?

No objections from me.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel