Re: x86: kvm: Revert "remove sched notifier for cross-cpu migrations"

[Radim Krcmar <rkrcmar@redhat.com>]

2015-03-26 11:51-0700, Andy Lutomirski:
> On Thu, Mar 26, 2015 at 4:29 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > On Wed, Mar 25, 2015 at 04:22:03PM -0700, Andy Lutomirski wrote:
> >> Suppose we start out with all vcpus agreeing on their pvti and perfect
> >> invariant TSCs.  Now the host updates its frequency (due to NTP or
> >> whatever).  KVM updates vcpu 0's pvti.  Before KVM updates vcpu 1's
> >> pvti, guest code on vcpus 0 and 1 see synced TSCs but different pvti.
> >> They'll disagree on the time, and one of them will be ahead until vcpu
> >> 1's pvti gets updated.
> >
> > The masterclock scheme enforces the same system_timestamp/tsc_timestamp pairs
> > to be visible at one time, for all vcpus.
> >
> >
> >  * That is, when timespec0 != timespec1, M < N. Unfortunately that is
> >  * not
> >  * always the case (the difference between two distinct xtime instances
> >  * might be smaller then the difference between corresponding TSC reads,
> >  * when updating guest vcpus pvclock areas).
> >  *
> >  * To avoid that problem, do not allow visibility of distinct
> >  * system_timestamp/tsc_timestamp values simultaneously: use a master
> >  * copy of host monotonic time values. Update that master copy
> >  * in lockstep.
> 
> Yuck.  So we have per cpu timing data, but the protocol is only usable
> for monotonic timing because we forcibly freeze all vcpus when we
> update the nominally per cpu data.
> 
> The obvious guest implementations are still unnecessarily slow,
> though.  It would be nice if the guest could get away without using
> any getcpu operation at all.
> 
> Even if we fixed the host to increment version as advertised, I think
> we can't avoid two getcpu ops.  We need one before rdtsc to figure out
> which pvti to look at,

Yes.

>                        and we need another to make sure that we were
> actually on that cpu at the time we did rdtsc.  (Rdtscp doesn't help
> -- we need to check version before rdtsc, and we don't know what
> version to check until we do a getcpu.).

Exactly, reading cpuid after rdtsc doesn't do that though, we could have
migrated back between those reads.
rtdscp would allow us to check that we read tsc of pvti's cpu.
(It doesn't get rid of that first read.)

>                                          The migration hook has the
> same issue -- we need to check the migration count, then confirm we're
> on that cpu, then check the migration count again, and we can't do
> that until we know what cpu we're on.

True;  the revert has a bug -- we need to check cpuid for the second
time before rdtsc.  (Migration hook is there just because we don't know
which cpu executed rdtsc.)

> If, on the other hand, we could rely on having all of these things in
> sync, then this complication goes away, and we go down from two getcpu
> ops to zero.

(Yeah, we should look what are the drawbacks of doing it differently.)