Re: [Qemu-devel] invtsc + migration + TSC scaling

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Radim Krčmář (rkrcmar@redhat.com) wrote:
> 2016-10-17 07:47-0200, Marcelo Tosatti:
> > On Fri, Oct 14, 2016 at 06:20:31PM -0300, Eduardo Habkost wrote:
> >> I have been wondering: should we allow live migration with the
> >> invtsc flag enabled, if TSC scaling is available on the
> >> destination?
> > 
> > TSC scaling and invtsc flag, yes.
> 
> Yes, if we have well synchronized time between hosts, then we might be
> able to migrate with a TSC shift that cannot be perceived by the guest.
> 
> Unless the VM also has a migratable assigned PCI device that uses ART,
> because we have no protocol to update the setting of ART (in CPUID), so
> we should keep migration forbidden then.
> 
> >> For reference, this is what the Intel SDM says about invtsc:
> >> 
> >>   The time stamp counter in newer processors may support an
> >>   enhancement, referred to as invariant TSC. Processor’s support
> >>   for invariant TSC is indicated by CPUID.80000007H:EDX[8].
> >> 
> >>   The invariant TSC will run at a constant rate in all ACPI P-,
> >>   C-. and T-states. This is the architectural behavior moving
> >>   forward. On processors with invariant TSC support, the OS may
> >>   use the TSC for wall clock timer services (instead of ACPI or
> >>   HPET timers). TSC reads are much more efficient and do not
> >>   incur the overhead associated with a ring transition or access
> >>   to a platform resource.
> >
> > Yes. The blockage happened for different reasons:
> > 
> > 1) Migration: to host with different TSC frequency.
> 
> We shouldn't have done this even now when emulating anything newer than
> Pentium 4, because those CPUs have constant TSC, which only lacks the
> guarantee that it doesn't stop in deep C-states:
> 
>   For [a list of processors we emulate]: the time-stamp counter
>   increments at a constant rate. That rate may be set by the maximum
>   core-clock to bus-clock ratio of the processor or may be set by the
>   maximum resolved frequency at which the processor is booted. The
>   maximum resolved frequency may differ from the processor base
>   frequency, see Section 18.18.2 for more detail. On certain processors,
>   the TSC frequency may not be the same as the frequency in the brand
>   string.
> 
>   The specific processor configuration determines the behavior. Constant
>   TSC behavior ensures that the duration of each clock tick is uniform
>   and supports the use of the TSC as a wall clock timer even if the
>   processor core changes frequency. This is the architectural behavior
>   moving forward.
> 
> Invariant TSC is more useful, though, so more applications would break
> when migrating to a different TSC frequency.
> 
> > 2) Savevm: It is not safe to use the TSC for wall clock timer
> > services.
> 
> With constant TSC, we could argue that a shift to deep C-state happened
> and paused TSC, which is not a good behavior, but somewhat defensible.
> 
> > By allowing savevm, you make a commitment to allow a feature
> > at the expense of not complying with the spec (specifically the "
> > the OS may use the TSC for wall clock timer services", because the
> > TSC stops relative to realtime for the duration of the savevm stop
> > window).
> 
> Yep, we should at least guesstimate the TSC to allow the guest to resume
> with as small TSC-shift as possible and check that hosts were somewhat
> synchronized with UTC (or something we choose for time).
> 
> > But since Linux guests use kvmclock and Windows guests use Hyper-V
> > enlightenment, it should be fine to disable 2).
> > 
> > There is a bug open for this, btw: 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1353073
> 
> These people should be happy with just live-migrations, so can't we just
> keep savevm forbidden?

Why is savevm so much harder? Is it just the difference in real time?
If so then I do worry about how small a difference you're hoping
to guarentee in live-migration.

Dave

> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK