Re: What's kvmclock's custom sched_clock for?

[Marcelo Tosatti <mtosatti@redhat.com>]

On Tue, Jan 12, 2016 at 04:33:28PM +0100, Radim Krcmar wrote:
> 2016-01-11 19:00-0200, Marcelo Tosatti:
> > On Fri, Jan 08, 2016 at 03:13:16PM +0100, Radim Krcmar wrote:
> >> 2016-01-07 18:10-0200, Marcelo Tosatti:
> >>> On Thu, Jan 07, 2016 at 04:18:11PM +0100, Radim Krcmar wrote:
> >>>> Stable sched clock is quite unrelated to TSC features.  KVMs from last
> >>>> few years should always give good enough result to allow stable sched
> >>>> clock.  We wanted realtime guests and realtime linux needs no_hz=full
> >>>> that depends on stable sched clock.  The result is huge hack.
> >>>> 
> >>>> We'd need to say that migration creates powerful gravity fields to
> >>>> faithfully migrate constant/invariant TSC, but stable sched clock
> >>>> doesn't have that strict expectations about time.
> >>> 
> >>> Was that supposed to be a joke?
> >> 
> >> Yes, if you mean the first sentence of the second paragraph.
> >> (I think that we'll use a different disclaimer when we enable
> >>  best-effort migration with invariant TSC.)
> > 
> > About getting rid of kvmclock,
> 
> I never wanted to get rid of kvmclock.  In the first part of the email
> in question, I meant that the shift and scale can be accelerated by
> VMX-TSC hardware, leaving only a check that kvmclock in expected mode
> and rdtsc to get the result.

If host TSC can be used, then its not necessary to have the kvmclock
complication.

> >                                problem is steal time. Should
> > separate steal time reporting from rest of kvmclock, so that you
> > can use TSC clocksource and have steal time reporting.
> 
> We can already do that, steal time doesn't depend on guest sched clock.
> Steal time uses a MSR+memory based interface that is related to kvmclock
> only by shared notion of a second.

Err, i meant "guest stop notification" which is done via flags field.

> > Also, its very clear why migration was disabled, because 
> > invariant tsc man page says:
> > 
> > QEMU commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
> > 
> >     target-i386: block migration and savevm if invariant tsc is exposed
> > 
> >     Invariant TSC documentation mentions that "invariant TSC will run at a
> >     constant rate in all ACPI P-, C-. and T-states".
> > 
> >     This is not the case if migration to a host with different TSC frequency
> >     is allowed, or if savevm is performed. So block migration/savevm.
> > 
> > The issue is, even with migration to a host with 
> > proper frequency, TSC counting will stop for the duration of migration.
> 
> Stopping is the easiest solution.  We can also try to mitigate the
> difference by synchronizing time on source and destination hosts,
> sharing what UTC/TAI/... time there was at one TSC read on the source,
> and setting the appropriate TSC shift on the destination.  (And solve
> accumulation of the error, maybe by always using the initial pair.)
> 
> The result should be less off than when stopping and the guest couldn't
> tell that TSC rate varied as it can't have more reliable time source
> than the host.
> 
> The issue doesn't have a good solution and I think that some people will
> prefer drawbacks associated with invariant TSC migration.
> (They do so for other time sources and all have the issue + we already
>  migrate constant TSC, which can only match the spec if we make some
>  excuses, like "migration forces CPUs into a deep C-state".)
> 
> > But i suppose you can document the fact (that "invariant TSC" behaviour
> > as documented is different than what exposed by virtualization),
> 
> Yep, that generic explanation is quite likely, next to no documentation.
> 
> (There are some lawyerish explanations that don't need to violate the
>  spec, but I prefer the physics-based one.)
> 
> >                                                                  and 
> > go for it.
> 
> I definitely won't be proactive.