RE: [Xen-devel] Re: [PATCH 3/5] x86/pvclock: add vsyscall implementation

[Dan Magenheimer <dan.magenheimer@oracle.com>]

> From: Avi Kivity [mailto:avi@redhat.com]
> Sent: Thursday, October 29, 2009 9:07 AM
> To: Dan Magenheimer
> Cc: Jeremy Fitzhardinge; Glauber Costa; Jeremy Fitzhardinge; Kurt
> Hackel; the arch/x86 maintainers; Linux Kernel Mailing List; 
> Glauber de
> Oliveira Costa; Xen-devel; Keir Fraser; Zach Brown; Chris Mason; Ingo
> Molnar
> Subject: Re: [Xen-devel] Re: [PATCH 3/5] x86/pvclock: add vsyscall
> implementation
> 
> 
> On 10/29/2009 04:46 PM, Dan Magenheimer wrote:
> > No, the apps I'm familiar with (a DB and a JVM) need a timestamp
> > not a monotonic counter.  The timestamps must be relatively
> > accurate (e.g. we've been talking about gettimeofday generically,
> > but these apps would use clock_gettime for nsec resolution),
> > monotonically increasing, and work properly across a VM
> > migration.  The timestamps are taken up to a 100K/sec or
> > more so the apps need to ensure they are using the fastest
> > mechanism available that meets those requirements.
> 
> Out of interest, do you know (and can you relate) why those apps need 
> 100k/sec monotonically increasing timestamps?

I don't have any public data available for this DB usage, but basically
assume it is measuring transactions at a very high throughput, some
of which are to a memory-resident portion of the DB.  Anecdotally, 
I'm told the difference between non-vsyscall gettimeofday
and native rdtsc (on a machine with Invariant TSC support) can
affect overall DB performance by as much as 10-20%.

I did find the following public link for the JVM: 

http://download.oracle.com/docs/cd/E13188_01/jrockit/tools/intro/jmc3.html

Search for "flight recorder".  This feature is intended to
be enabled all the time, but with non-vsyscall gettimeofday
the performance impact is unacceptably high, so they are using
rdtscp instead (on those machines where it is available).  With
rdtscp, the performance impact is not measureable.

Though the processor/server vendors have finally fixed the
"unsynced TSC" problem on recent x86 platforms, thus allowing
enterprise software to obtain timestamps at rdtsc performance,
the problem comes back all over again with virtualization
because of migration.  Jeremy's vsyscall+pvclock is a great 
solution if the app can ensure that it is present; if not,
the apps will instead continue to use rdtsc as even emulated
rdtsc is 2-3x faster than non-vsyscall gettimeofday.

Does that help?