Re: [PATCH] xen/pvhvm: Support more than 32 VCPUs when migrating (v3).

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-07-10 at 14:57 -0400, Konrad Rzeszutek Wilk wrote:
> On Fri, Jul 10, 2015 at 02:37:46PM -0400, Konrad Rzeszutek Wilk wrote:
> > When Xen migrates an HVM guest, by default its shared_info can
> > only hold up to 32 CPUs. As such the hypercall
> > VCPUOP_register_vcpu_info was introduced which allowed us to
> > setup per-page areas for VCPUs. This means we can boot PVHVM
> > guest with more than 32 VCPUs. During migration the per-cpu
> > structure is allocated freshly by the hypervisor (vcpu_info_mfn
> > is set to INVALID_MFN) so that the newly migrated guest
> > can make an VCPUOP_register_vcpu_info hypercall.
> > 
> > Unfortunatly we end up triggering this condition in Xen:
> > /* Run this command on yourself or on other offline VCPUS. */
> >  if ( (v != current) && !test_bit(_VPF_down, &v->pause_flags) )
> > 
> > which means we are unable to setup the per-cpu VCPU structures
> > for running vCPUS. The Linux PV code paths make this work by
> > iterating over every vCPU with:
> > 
> >  1) is target CPU up (VCPUOP_is_up hypercall?)
> >  2) if yes, then VCPUOP_down to pause it.
> >  3) VCPUOP_register_vcpu_info
> >  4) if it was down, then VCPUOP_up to bring it back up
> > 
> > But since VCPUOP_down, VCPUOP_is_up, and VCPUOP_up are
> > not allowed on HVM guests we can't do this. However with the
> > Xen git commit f80c5623a126afc31e6bb9382268d579f0324a7a
> > ("xen/x86: allow HVM guests to use hypercalls to bring up vCPUs"")
> 
> <sigh> I was in my local tree which was Roger's 'hvm_without_dm_v3'
> looking at patches and spotted this - and thought it was already in!
> 
> Sorry about this patch - and please ignore it until the VCPU_op*
> can be used by HVM guests.

FYI I just tripped over this while implementing ARM save/restore (in that I
couldn't figure out HTF HVM VCPUs > MAX_LEGACY_VCPUS were getting their
vcpu_info re-registered, which turns out to be because they aren't...).

ARM also lack the VCPU_up/down/is_up hypercalls. My plan there is simply to
use on_each_cpu to do it, I can get away with this on ARM because the
necessary infra (IPIs etc) are provided by the h/w virt platform (i.e. look
native) so there is no reliance on Xen infra being fully up.

Not sure if that is also true of x86/PVHVM but thought I would mention it
in case it seemed preferable to you.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel