RE: [PATCH 0 of 5] PV on HVM Xen

[Dan Magenheimer <dan.magenheimer@oracle.com>]

Hi Sheng --

A few comments:
- with default tsc_mode, PV handles migration fine
  and I think HVM does also.  But that is because,
  in many situations in 4.0, rdtsc is trapped/emulated
- pvclock is not always the best way to handle timer
  (tsc as the clocksource is better in some cases...
  ask VMware)
- because KVM does something doesn't mean it is the
  best way to do it ;-)
- pvclock doesn't eliminate timer_mode cases...
  timer_mode is required for legacy guest OS's
- in case there is any confusion, tsc_mode and timer_mode
  are different options for very different problems

There are many many different cases to consider across
different processors, different guest OS's, different
migration scenarios.  Have you measured any of them
(using 4.0) or are you using old data or simply making
the assumption that the existing pvclock mechanism is
always the fastest?

I am very supportive of virtualization-aware guest OS's,
but let's get some measurements.

Dan

> -----Original Message-----
> From: Sheng Yang [mailto:sheng@linux.intel.com]
> Sent: Tuesday, March 16, 2010 12:09 AM
> To: Dan Magenheimer
> Cc: Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Eddie Dong;
> Frank Van Der Linden; Stefano Stabellini
> Subject: Re: [Xen-devel] [PATCH 0 of 5] PV on HVM Xen
> 
> On Tuesday 16 March 2010 08:32:22 Dan Magenheimer wrote:
> > Sorry I've fallen hopelessly behind on this thread (rope? :-)
> > so apologies if this has already been addressed:
> >
> > If the host and guest need to share the same tsc to work
> > properly, and there are apps in the guest that assume a
> > monotonically increasing TSC (or even "approximately monotonically
> > increasing", e.g. can filter out reasonably small differences),
> > and the VM migrates from a physical machine that's been running
> > a long time to a recently booted physical machine, what
> > happens to the app?  (Answer: Prior to 4.0, or with tsc_mode==2
> > in 4.0, the app fails.)
> 
> I haven't checked the live migration issue. But how does PV guest or
> kvmclock
> deal with it? The same should works here. Even we may got some trouble
> in some
> condition, we can still fall back to normal clocksource without any
> loss(though we didn't want this thing always happen).
> >
> > TSC emulation addresses this issue (which is why it is effectively
> > the default in Xen 4.0**) but has the side effect that the TSC reads
> > required for the pvclock algorithm are trapped/emulated.  In this
> > case, pvclock may not be much faster than alternative
> > clocksource implementations.
> 
> The hardware emulation is always not the best solution to address timer
> issue.
> For example, now we have 3 kinds of different timer mode, and we have
> to let
> user to determine which one is correct. And hope we won't need more
> timer mode
> in the future. PV is always the best way to handle timer. Take a look
> at KVM,
> they don't use much PV, but kvmclock is a must.
> 
> > > Sheng/Stefano, have you actually measured pvclock recently
> > and compared it to other alternatives?  I'm wondering how
> > much of this discussion is moot.
> 
> If live migration is a issue now, we can address it. But hardware
> emulation is
> even a issue for non-LM case - guest need to know the timer mode, which
> is
> impossible to address. It's surely not the preferred method for VM if a
> PV way
> is available(of course, and works well).
> 
> --
> regards
> Yang, Sheng
> 
> 
> > Dan
> >
> > P.S. Note that TSC emulation in many cases behaves differently
> > before and after migration, specifically on modern processors
> > depending on the clock frequency of the source/destination
> > machines, so please ensure post-migration is measured as this
> > will be the common case in many virtualized data centers.
> >
> > ** see xen-unstable.hg/docs/misc/tscmode.txt
> >
> > > -----Original Message-----
> > > From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> > > Sent: Monday, March 15, 2010 5:09 PM
> > > To: Stefano Stabellini
> > > Cc: xen-devel@lists.xensource.com; Sheng Yang
> > > Subject: Re: [Xen-devel] [PATCH 0 of 5] PV on HVM Xen
> > >
> > > On 03/15/2010 05:28 AM, Stefano Stabellini wrote:
> > > > I like your pv clocksource implementation.
> > > > The only reason why I would defer the patch is that I don't
> > >
> > > particularly
> > >
> > > > like the "enable_pv" hypercall, so I would try to get away
> without
> > >
> > > it,
> > >
> > > > resetting the tsc offset automatically when enabling the
> VIRQ_TIMER
> > >
> > > on
> > >
> > > > an HVM domain.
> > >
> > > Ah, so the issue is that if we're using the pvclock, the host and
> guest
> > > need to share the same tsc, so we can't deal with any kind of tsc
> > > offset?
> > >
> > > In that case, I'd prefer to have an explicit "set/remove tsc
> offset"
> > > vcpu op rather than making it the implicit side-effect of anything
> > > else.  In particular, since clock sources and event sources are
> > > completely distinct, making tsc offset (a clock source thing)
> affected
> > > VIRQ_TIMER (and event source thing) seems like a particularly poor
> > > idea.
> > >
> > > That, or make the pvclock structure the HVM vcpu sees have timing
> > > parameters which already incorporate the tsc offset.  We've already
> > > demonstrated that there's no need to have the time info in the real
> > > shared memory between Xen and the domain (it can be updated via
> copy
> > > when needed).
> > >
> > >      J
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel