Re: [PATCH] xen: schedule: allow dom0 vCPUs to be re-pinned when dom0_vcpus_pin is set

[Matt Wilson <msw@amazon.com>]

On Thu, 6 Dec 2012 10:44:04 +0000, Jan Beulich <JBeulich@suse.com> wrote:
> On Wed, 5 Dec 2012 09:06:20 -0800, Matt Wilson <msw@amazon.com> wrote:
> > On Wed, 5 Dec 2012 16:25:36 +0000, Jan Beulich <JBeulich@suse.com> wrote:
> > > On Wed, 5 Dec 2012 07:59:08 -0800, Matt Wilson <msw@amazon.com> wrote:
> > > > 
> > > > If this is true, the existing is_pinned_vcpu() test is broken:
> > > > 
> > > >    #define is_pinned_vcpu(v) ((v)->domain->is_pinned || \
> > > >                               cpumask_weight((v)->cpu_affinity) == 1)
> > > > 
> > > > It's && not ||. So if someone pins dom0 vCPUs to pCPUs 1:1 after boot,
> > > > the MSR traps will suddenly start working.
> > > > 
> > > > See commit: 
> > > >   http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=cc0854dd 
> > > 
> > > I don't see what's wrong here. Certain things merely require the
> > > pCPU that a vCPU runs on to be stable, which is what the test
> > > above is for.
> > 
> > Me either. That said, are you willing to Ack and commit my patch that
> > started this thread?
> 
> In no case without Andrew's concerns addressed. Beyond that,
> I'd be hesitant to ack it as I'm myself suspecting side effects that
> we don't want and/or aren't aware of, and in no case could I
> commit it without Keir's ack.

Jan,

So today if I boot Xen without dom0_vcpus_pin set, dom0's vCPUs will
be allowed to run on any pCPU. Xen will block attempts to write
certain MSRs (MSR_AMD64_NB_CFG, MSR_FAM10H_MMIO_CONF_BASE and
MSR_IA32_ENERGY_PERF_BIAS). The VCPUOP_get_physid subop of the vcpu_op
hypercall will not return the initial APIC ID or ACPI ID for dom0.

Also today, if I run "xl vcpu-pin 0 0", suddenly those MSR writes and
the VCPUOP_get_physid hypercall will start working for vCPU 0. For
what it's worth, only legacy XenoLinux-derived kernels appear to use
this hypercall during SMP boot. Upstream Linux does not.

I think that the real risk is in the XenoLinux SMP booting code on AMD
processors where sometimes initial APIC ID != ACPI ID. If the CPU
pinning changes, the ACPI ID to APIC ID mapping will be wrong. This
broke PowerNow! when it ran in dom0.

But PowerNow! is handled by the hypervisor now. So what's the real
danger here?

Andrew, your thoughts?

Thanks,

Matt