pvclock (PV and HVM) and vsyscall

pvclock (PV and HVM) and vsyscall

[Dan Magenheimer]

The other discussion of RADclock reminded me:

IIRC the pvclock algorithm is still incompatible with
vsyscall/vdso (fast system calls) and there was no obvious
and upstreamable solution to resolve this.

This means that any userland call to the various gettimeofday
routines will always do a true system call on both (a) a PV
domain or (b) any PV on HVM domain with Stefanos' pvclock patch.

Since true syscalls are very expensive on a 64-bit
PV domain, I'm wondering if pvclock is still the right
default choice for upstream (at least for 64-bit).

Re: pvclock (PV and HVM) and vsyscall

[Jeremy Fitzhardinge]

 On 10/15/2010 08:48 AM, Dan Magenheimer wrote:
> The other discussion of RADclock reminded me:
>
> IIRC the pvclock algorithm is still incompatible with
> vsyscall/vdso (fast system calls) and there was no obvious
> and upstreamable solution to resolve this.
>
> This means that any userland call to the various gettimeofday
> routines will always do a true system call on both (a) a PV
> domain or (b) any PV on HVM domain with Stefanos' pvclock patch.
>
> Since true syscalls are very expensive on a 64-bit
> PV domain, I'm wondering if pvclock is still the right
> default choice for upstream (at least for 64-bit).

What other options are there?  If the tsc is globally stable, then using
pvclock in userspace will work fine; if it isn't, you'll need to do the
syscall anyway.

There's no basic problem with the vsyscall pvclock patch so long as we
can know under what circumstances it is safe to enable.

    J

RE: pvclock (PV and HVM) and vsyscall

[Dan Magenheimer]

> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> Sent: Friday, October 15, 2010 6:28 PM
> To: Dan Magenheimer
> Cc: Tim Deegan; Stefano Stabellini; xen-devel@lists.xensource.com
> Subject: Re: pvclock (PV and HVM) and vsyscall
> 
>  On 10/15/2010 08:48 AM, Dan Magenheimer wrote:
> > The other discussion of RADclock reminded me:
> >
> > IIRC the pvclock algorithm is still incompatible with
> > vsyscall/vdso (fast system calls) and there was no obvious
> > and upstreamable solution to resolve this.
> >
> > This means that any userland call to the various gettimeofday
> > routines will always do a true system call on both (a) a PV
> > domain or (b) any PV on HVM domain with Stefanos' pvclock patch.
> >
> > Since true syscalls are very expensive on a 64-bit
> > PV domain, I'm wondering if pvclock is still the right
> > default choice for upstream (at least for 64-bit).
> 
> What other options are there?  If the tsc is globally stable, then
> using
> pvclock in userspace will work fine; if it isn't, you'll need to do the
> syscall anyway.
> 
> There's no basic problem with the vsyscall pvclock patch so long as we
> can know under what circumstances it is safe to enable.

I think (but am not positive) that the circumstances under which
vsyscall pvclock can be enabled are exactly the same as those
for which tsc is globally stable.  And when tsc is globally stable,
upstream guest kernels can use tsc instead of pvclock.  And
when tsc is NOT globally stable, as of Xen 4.0, rdtsc emulation
is faster than non-vsyscall pvclock, at least in many/most
environments.

To quote the RADclock paper: "[the Xen Clocksource counter] works
well as intended, however note that it is a complex solution created
to solve a problem which will soon disappear as reliable TSC counters
again become ubiquitous."  (And, yes, we both know that there will
always be exceptions to "reliable" TSC, but that is in part why
rdtsc emulation is the default, and Xen and RADclock and pvclock
will all have the same problems as a native kernel when these exceptions
occur.)

I'm afraid my available time is too limited right now to fight
this battle, but I think we should be very cautious about building
even more infrastructure on top of pvclock.

Re: pvclock (PV and HVM) and vsyscall

[Jeremy Fitzhardinge]

 On 10/17/2010 06:29 PM, Dan Magenheimer wrote:
>> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
>> Sent: Friday, October 15, 2010 6:28 PM
>> To: Dan Magenheimer
>> Cc: Tim Deegan; Stefano Stabellini; xen-devel@lists.xensource.com
>> Subject: Re: pvclock (PV and HVM) and vsyscall
>>
>>  On 10/15/2010 08:48 AM, Dan Magenheimer wrote:
>>> The other discussion of RADclock reminded me:
>>>
>>> IIRC the pvclock algorithm is still incompatible with
>>> vsyscall/vdso (fast system calls) and there was no obvious
>>> and upstreamable solution to resolve this.
>>>
>>> This means that any userland call to the various gettimeofday
>>> routines will always do a true system call on both (a) a PV
>>> domain or (b) any PV on HVM domain with Stefanos' pvclock patch.
>>>
>>> Since true syscalls are very expensive on a 64-bit
>>> PV domain, I'm wondering if pvclock is still the right
>>> default choice for upstream (at least for 64-bit).
>> What other options are there?  If the tsc is globally stable, then
>> using
>> pvclock in userspace will work fine; if it isn't, you'll need to do the
>> syscall anyway.
>>
>> There's no basic problem with the vsyscall pvclock patch so long as we
>> can know under what circumstances it is safe to enable.
> I think (but am not positive) that the circumstances under which
> vsyscall pvclock can be enabled are exactly the same as those
> for which tsc is globally stable.  And when tsc is globally stable,
> upstream guest kernels can use tsc instead of pvclock.

Not wanting to go around in circles, but usermode can't reliably know
whether the tsc is stable enough to be usable directly.  Its most
reliable course is to always use the proper APIs and rely on the kernel
to implement the most efficient mechanism for those APIs.

    J

RE: pvclock (PV and HVM) and vsyscall

[Dan Magenheimer]

> >>  On 10/15/2010 08:48 AM, Dan Magenheimer wrote:
> >>> The other discussion of RADclock reminded me:
> >>>
> >>> IIRC the pvclock algorithm is still incompatible with
> >>> vsyscall/vdso (fast system calls) and there was no obvious
> >>> and upstreamable solution to resolve this.
> >>>
> >>> This means that any userland call to the various gettimeofday
> >>> routines will always do a true system call on both (a) a PV
> >>> domain or (b) any PV on HVM domain with Stefanos' pvclock patch.
> >>>
> >>> Since true syscalls are very expensive on a 64-bit
> >>> PV domain, I'm wondering if pvclock is still the right
> >>> default choice for upstream (at least for 64-bit).
> >> What other options are there?  If the tsc is globally stable, then
> >> using
> >> pvclock in userspace will work fine; if it isn't, you'll need to do
> the
> >> syscall anyway.
> >>
> >> There's no basic problem with the vsyscall pvclock patch so long as
> we
> >> can know under what circumstances it is safe to enable.
> > I think (but am not positive) that the circumstances under which
> > vsyscall pvclock can be enabled are exactly the same as those
> > for which tsc is globally stable.  And when tsc is globally stable,
> > upstream guest kernels can use tsc instead of pvclock.
> 
> Not wanting to go around in circles, but usermode can't reliably know
> whether the tsc is stable enough to be usable directly.  Its most
> reliable course is to always use the proper APIs and rely on the kernel
> to implement the most efficient mechanism for those APIs.

Understood.  In case of misunderstanding, I *am* talking about
usermode using the proper APIs and relying on the kernel to
implement the most efficient mechanism.  I'm saying that, at least
in many cases (and certainly in the case of 64-bit PV guests),
the most efficient mechanism will be tsc instead of pvclock,
but I think every pvops kernel running on Xen (as well as Stefano's 
PV-on-HVM patch) assumes pvclock is always best.  It is that
fiction (pvclock is always best) that I am concerned about
building further infrastructure on top of.