Re: [Qemu-devel] [BUG] Migrate failes between boards with different PMC counts

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Zhuangyanying (ann.zhuangyanying@huawei.com) wrote:
> 
> 
> > -----Original Message-----
> > From: Daniel P. Berrange [mailto:berrange@redhat.com]
> > Sent: Monday, April 24, 2017 6:34 PM
> > To: Dr. David Alan Gilbert
> > Cc: Zhuangyanying; Zhanghailiang; wangxin (U); qemu-devel@nongnu.org;
> > Gonglei (Arei); Huangzhichao; pbonzini@redhat.com
> > Subject: Re: [Qemu-devel] [BUG] Migrate failes between boards with different
> > PMC counts
> > 
> > On Mon, Apr 24, 2017 at 11:27:16AM +0100, Dr. David Alan Gilbert wrote:
> > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > On Mon, Apr 24, 2017 at 10:23:21AM +0100, Dr. David Alan Gilbert wrote:
> > > > > * Zhuangyanying (ann.zhuangyanying@huawei.com) wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > Recently, I found migration failed when enable vPMU.
> > > > > >
> > > > > > migrate vPMU state was introduced in linux-3.10 + qemu-1.7.
> > > > > >
> > > > > > As long as enable vPMU, qemu will save / load the
> > > > > > vmstate_msr_architectural_pmu(msr_global_ctrl) register during the
> > migration.
> > > > > > But global_ctrl generated based on cpuid(0xA), the number of
> > > > > > general-purpose performance monitoring counters(PMC) can vary
> > > > > > according to Intel SDN. The number of PMC presented to vm, does
> > > > > > not support configuration currently, it depend on host cpuid, and enable
> > all pmc defaultly at KVM. It cause migration to fail between boards with
> > different PMC counts.
> > > > > >
> > > > > > The return value of cpuid (0xA) is different dur to cpu, according to Intel
> > SDN，18-10 Vol. 3B:
> > > > > >
> > > > > > Note: The number of general-purpose performance monitoring
> > > > > > counters (i.e. N in Figure 18-9) can vary across processor
> > > > > > generations within a processor family, across processor
> > > > > > families, or could be different depending on the configuration
> > > > > > chosen at boot time in the BIOS regarding Intel Hyper Threading
> > > > > > Technology, (e.g. N=2 for 45 nm Intel Atom processors; N =4 for
> > processors based on the Nehalem microarchitecture; for processors based on
> > the Sandy Bridge microarchitecture, N = 4 if Intel Hyper Threading Technology
> > is active and N=8 if not active).
> > > > > >
> > > > > > Also I found, N=8 if HT is not active based on the broadwell，,
> > > > > > such as CPU E7-8890 v4 @ 2.20GHz
> > > > > >
> > > > > > # ./x86_64-softmmu/qemu-system-x86_64 --enable-kvm -smp 4 -m
> > > > > > 4096 -hda
> > > > > > /data/zyy/test_qemu.img.sles12sp1 -vnc :99 -cpu kvm64,pmu=true
> > > > > > -incoming tcp::8888 Completed 100 %
> > > > > > qemu-system-x86_64: error: failed to set MSR 0x38f to
> > > > > > 0x7000000ff
> > > > > > qemu-system-x86_64: /data/zyy/git/test/qemu/target/i386/kvm.c:1833:
> > kvm_put_msrs:
> > > > > > Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
> > > > > > Aborted
> > > > > >
> > > > > > So make number of pmc configurable to vm ? Any better idea ?
> > > > >
> > > > > Coincidentally we hit a similar problem a few days ago with -cpu
> > > > > host  - it took me quite a while to spot the difference between
> > > > > the machines was the source had hyperthreading disabled.
> > > > >
> > > > > An option to set the number of counters makes sense to me; but I
> > > > > wonder how many other options we need as well.  Also, I'm not sure
> > > > > there's any easy way for libvirt etc to figure out how many
> > > > > counters a host supports - it's not in /proc/cpuinfo.
> > > >
> > > > We actually try to avoid /proc/cpuinfo whereever possible. We do
> > > > direct CPUID asm instructions to identify features, and prefer to
> > > > use /sys/devices/system/cpu if that has suitable data
> > > >
> > > > Where do the PMC counts come from originally ? CPUID or something
> > else ?
> > >
> > > Yes, they're bits 8..15 of CPUID leaf 0xa
> > 
> > Ok, that's easy enough for libvirt to detect then. More a question of what libvirt
> > should then do this with the info....
> > 
> 
> Do you mean to do a validation at the begining of migration? in qemuMigrationBakeCookie() & qemuMigrationEatCookie(), if the PMC numbers are not equal, just quit migration?
> It maybe a good enough first edition.
> But for a further better edition, maybe it's better to support Heterogeneous migration I think, so we might need to make PMC number configrable, then we need to modify KVM/qemu as well.

Yes agreed; the only thing I wanted to check was that libvirt would have enough
information to be able to use any feature we added to QEMU.

Dave

> Regards,
> -Zhuang Yanying
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK