Re: NUMA TODO-list for xen-devel

[Andre Przywara <andre.przywara@amd.com>]

On 08/03/2012 11:23 AM, Jan Beulich wrote:
>>>> On 02.08.12 at 18:36, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>> On Thu, Aug 2, 2012 at 2:34 PM, Dario Faggioli <raistlin@linux.it> wrote:
>>> On Thu, 2012-08-02 at 10:43 +0100, Jan Beulich wrote:
>>>>>>> On 01.08.12 at 18:16, Dario Faggioli <raistlin@linux.it> wrote:
>>>>>      - Virtual NUMA topology exposure to guests (a.k.a guest-numa). If a
>>>>>        guest ends up on more than one nodes, make sure it knows it's
>>>>>        running on a NUMA platform (smaller than the actual host, but
>>>>>        still NUMA). This interacts with some of the above points:
>>>>
>>>> The question is whether this is really useful beyond the (I would
>>>> suppose) relatively small set of cases where migration isn't
>>>> needed.
>>>>
>>> Mmm... Not sure I'm getting what you're saying here, sorry. Are you
>>> suggesting that exposing a virtual topology is not a good idea as it
>>> poses constraints/prevents live migration?

Honestly, what would be the problems with migration? NUMA awareness is 
actually a software optimization, so we will not really break something 
if the advertised topology isn't the real one. This is especially true 
if we lower the number of NUMA nodes. Say the guest starts with two 
nodes and then gets migrated to a machine where it can happily live in 
one node. There would be some extra effort by the guest OS to obey the 
virtual NUMA topology, but if there isn't actually a NUMA penalty 
anymore this shouldn't really hurt, right?
Even if we would need to go to a machine where we have more nodes for a 
certain guest than before, this is actually what we have today: guest 
NUMA unawareness. I am not sure if this is really a migration 
showstopper, and certainly not a NUMA guest showstopper.

But we could make it a config file option, so we leave this decision to 
the admin. I have talked to people with huge guests, they keep asking me 
about this feature.

>>>
>>> If yes, well, I mostly agree that this is an huge issue, and that's why
>>> I think wee need some bright idea on how to deal with it. I mean, it's
>>> easy to make it optional and let it automatically disable migration,
>>> giving users the choice what they prefer, but I think this is more
>>> dodging the problem than dealing with it! :-P
>>>
>>>>>         * consider this during automatic placement for
>>>>>           resuming/migrating domains (if they have a virtual topology,
>>>>>           better not to change it);
>>>>>         * consider this during memory migration (it can change the
>>>>>           actual topology, should we update it on-line or disable memory
>>>>>           migration?)
>>
>> I think we could use cpu hot-plug to change the "virtual topology" of
>> VMs, couldn't we?  We could probably even do that on a running guest
>> if we really needed to.
>
> Hmm, not sure - using hotplug behind the back of the guest might
> be possible, but you'd first need to hot-unplug the vCPU. That's
> something that I don't think you can do on HVM guests (and for
> PV guests, guest visible NUMA support makes even less sense
> than for HVM ones).

I don't think that hotplug would really work. I have checked this some 
times ago, at least the Linux NUMA code cannot be really fooled by this. 
The SRAT table is firmware defined and static by nature, so there is no 
code in Linux to change the NUMA topology at runtime. This is especially 
true for the memory layout.

But as said above, I don't really buy this as an argument against guest 
NUMA. At least provide it as an option to people who know what they do.

Regards,
Andre.


-- 
Andre Przywara
AMD-OSRC (Dresden)
Tel: x29712