rdtsc in userspace

rdtsc in userspace

[Dan Magenheimer]

I'm continuing to investigate the usage of rdtsc in
userspace and whether there are programs "out there"
that use it "unsafely" that might randomly break
under Xen if rdtsc is not emulated, e.g. across a migration.
Some have argued that nobody should use rdtsc and any
programs that use rdtsc directly are "fundamentally broken"
so the default for rdtsc emulation should be off.

In measuring rdtsc usage in the kernel, both Jeremy
and I noticed that compiling the kernel causes a
large number of userland rdtsc's.  At first I thought
that this meant that gcc was using rdtsc, but gcc
sources do not show any use of rdtsc.  Next I suspected
bash, but it doesn't either.  Finally, it appears that
the calls are occurring from glibc, and searching for
uses of rdtsc, I found that glibc has its
own implementation of clock_gettime that uses rdtsc
directly!

The manpage for clock_gettime ("man 3 clock_gettime")
has a lengthy caveat about using clock_gettime on
an SMP system BUT provides a mechanism to test to
see if your SMP system is safe, clock_getcpuclockid(0).
See for example (near the end):

http://linux.die.net/man/3/clock_gettime

But testing this on a system I know is unsafe, the
test says my system is safe!  Looking at the source
code, I can see why... the test is a compile-time
test, not a run-time test, and on x86 it appears that
it will always pass.

So I wrote a simple test program that uses clock_gettime().
It does indeed do userland rdtsc's.

So... any program that uses clock_gettime to approximate
the passage of time is subject to break on Xen across
a migration.  Even if the programmer uses the prescribed
method for testing "safeness", it is subject to break.

True I still can't point to a specific example of breakage,
but this appears to be a much broader problem than code
where the programmer has explicitly used asm-like
constructs to directly utilize the rdtsc instruction,
in spite of dire warnings to not use it.

Dan

Re: rdtsc in userspace

[Jeremy Fitzhardinge]

On 10/23/09 15:51, Dan Magenheimer wrote:
> In measuring rdtsc usage in the kernel, both Jeremy
> and I noticed that compiling the kernel causes a
> large number of userland rdtsc's.  At first I thought
> that this meant that gcc was using rdtsc, but gcc
> sources do not show any use of rdtsc.  Next I suspected
> bash, but it doesn't either.

I think the dynamic linker may use rdtsc as a piece of randomness for
randomizing the addresses of libraries and other mappings.

>   Finally, it appears that
> the calls are occurring from glibc, and searching for
> uses of rdtsc, I found that glibc has its
> own implementation of clock_gettime that uses rdtsc
> directly!
>   

When I run clock_gettime on my system it uses the vsyscall version
(which is either a syscall or, with my recent patches, all in usermode).

Using strace on your test program should show whether its really
bypassing the syscall or not.

> The manpage for clock_gettime ("man 3 clock_gettime")
> has a lengthy caveat about using clock_gettime on
> an SMP system BUT provides a mechanism to test to
> see if your SMP system is safe, clock_getcpuclockid(0).
>   

My manpages don't have anything like that for clock_gettime().

> See for example (near the end):
>
> http://linux.die.net/man/3/clock_gettime
>   

That caveat is for CLOCK_PROCESS_CPUTIME_ID which doesn't seem
reasonable to count on for monotonicity (my manpages
CLOCK_PROCESS_CPUTIME_ID just refer to as "High resolution per-process
timer.").   CLOCK_REALTIME or CLOCK_MONOTONIC is a different matter.

> But testing this on a system I know is unsafe, the
> test says my system is safe!  Looking at the source
> code, I can see why... the test is a compile-time
> test, not a run-time test, and on x86 it appears that
> it will always pass.
>
> So I wrote a simple test program that uses clock_gettime().
> It does indeed do userland rdtsc's.
>   

Using what CLOCK_?

> So... any program that uses clock_gettime to approximate
> the passage of time is subject to break on Xen across
> a migration.  Even if the programmer uses the prescribed
> method for testing "safeness", it is subject to break.
>   

I always see clock_gettime(CLOCK_MONOTONIC or CLOCK_REALTIME, ...) make
a syscall or vsyscall.  If you're testing any of the thread-local times
then I think its less interesting.

    J

RE: rdtsc in userspace

[Dan Magenheimer]

> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> On 10/23/09 15:51, Dan Magenheimer wrote:
> > In measuring rdtsc usage in the kernel, both Jeremy
> > and I noticed that compiling the kernel causes a
> > large number of userland rdtsc's.  At first I thought
> > that this meant that gcc was using rdtsc, but gcc
> > sources do not show any use of rdtsc.  Next I suspected
> > bash, but it doesn't either.
> 
> I think the dynamic linker may use rdtsc as a piece of randomness for
> randomizing the addresses of libraries and other mappings.

Interesting.  I'll look into that.  I've collected
some other data that might appear somewhat contradictory:
I record the page offset of eip for each emulated rdtsc
and there are about 28 different values collected,
of which about 20 of them increment for each
exec'ed process.  One would think/hope that there
would only be one place where randomness is generated
for this kind of purpose, but I suppose if it is
inlined, it's hard to say.

Anyway, thanks for the tip, I will dig further.

> >   Finally, it appears that
> > the calls are occurring from glibc, and searching for
> > uses of rdtsc, I found that glibc has its
> > own implementation of clock_gettime that uses rdtsc
> > directly!
> >   
> 
> When I run clock_gettime on my system it uses the vsyscall version
> (which is either a syscall or, with my recent patches, all in 
> usermode).

I could be mistaken but I don't think my test guest,
which is EL5u2-32bit (2.6.18-based kernel) has vsyscall.
At least the detection mechanisms that I am aware
of ("sysctl -a | grep vsyscall" and "ls /proc/sys/kernel/vsys*"
don't work.  Is there another way on older kernels?
 
> Using strace on your test program should show whether its really
> bypassing the syscall or not.

You're right!  Strace does show syscalls for
each clock_gettime(), and I was mistaken... these
rdtsc's are in the kernel, not userland.

> > The manpage for clock_gettime ("man 3 clock_gettime")
> > has a lengthy caveat about using clock_gettime on
> > an SMP system BUT provides a mechanism to test to
> > see if your SMP system is safe, clock_getcpuclockid(0).
> 
> My manpages don't have anything like that for clock_gettime().

RHEL5 does.  Though now I am not sure how to use the
section 3 calls.

> That caveat is for CLOCK_PROCESS_CPUTIME_ID which doesn't seem
> reasonable to count on for monotonicity (my manpages
> CLOCK_PROCESS_CPUTIME_ID just refer to as "High resolution per-process
> timer.").   CLOCK_REALTIME or CLOCK_MONOTONIC is a different matter.
> 
> Using what CLOCK_?
> 
> I always see clock_gettime(CLOCK_MONOTONIC or CLOCK_REALTIME, 
> ...) make
> a syscall or vsyscall.  If you're testing any of the 
> thread-local times
> then I think its less interesting.
> 
>     J

I'll dig further next week.

Dan

RE: rdtsc in userspace

[Dan Magenheimer]

Just a followup on my attempt to track down existing
uses of rdtsc in userspace in booting an OS (for at
least one PV guest, 32-bit EL5u2).  See below for
previous status.

I removed all the uses of rdtsc in glibc, which were
mostly in the dynamic linker (used to gather stats
for LD_DEBUG=statistics output).  After that, there
remains 5 different "offsets" collected for userland
rdtsc (see below) and only one increments at every
process launch.  Three others have small totals
(102, 8, 14) that don't change after boot.  The
final/fifth is also small but seems to randomly
increment by one about once every 20-40 seconds.

While there remains a non-zero probability that any
one of these remaining rdtsc's could "break" on
an ill-timed migration, I don't think there's any
low-hanging fruit here to prove my point so am
abandoning this investigation.

> -----Original Message-----
> From: Dan Magenheimer 
> Sent: Saturday, October 24, 2009 7:21 PM
> To: Jeremy Fitzhardinge
> Cc: Xen-Devel (E-mail); Keir Fraser; Tim Deegan; Avi Kivity; 
> Kurt Hackel
> Subject: RE: rdtsc in userspace
> 
> 
> > From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> > On 10/23/09 15:51, Dan Magenheimer wrote:
> > > In measuring rdtsc usage in the kernel, both Jeremy
> > > and I noticed that compiling the kernel causes a
> > > large number of userland rdtsc's.  At first I thought
> > > that this meant that gcc was using rdtsc, but gcc
> > > sources do not show any use of rdtsc.  Next I suspected
> > > bash, but it doesn't either.
> > 
> > I think the dynamic linker may use rdtsc as a piece of 
> randomness for
> > randomizing the addresses of libraries and other mappings.
> 
> Interesting.  I'll look into that.  I've collected
> some other data that might appear somewhat contradictory:
> I record the page offset of eip for each emulated rdtsc
> and there are about 28 different values collected,
> of which about 20 of them increment for each
> exec'ed process.  One would think/hope that there
> would only be one place where randomness is generated
> for this kind of purpose, but I suppose if it is
> inlined, it's hard to say.
> 
> Anyway, thanks for the tip, I will dig further.
> 
> > >   Finally, it appears that
> > > the calls are occurring from glibc, and searching for
> > > uses of rdtsc, I found that glibc has its
> > > own implementation of clock_gettime that uses rdtsc
> > > directly!
> > >   
> > 
> > When I run clock_gettime on my system it uses the vsyscall version
> > (which is either a syscall or, with my recent patches, all in 
> > usermode).
> 
> I could be mistaken but I don't think my test guest,
> which is EL5u2-32bit (2.6.18-based kernel) has vsyscall.
> At least the detection mechanisms that I am aware
> of ("sysctl -a | grep vsyscall" and "ls /proc/sys/kernel/vsys*"
> don't work.  Is there another way on older kernels?
>  
> > Using strace on your test program should show whether its really
> > bypassing the syscall or not.
> 
> You're right!  Strace does show syscalls for
> each clock_gettime(), and I was mistaken... these
> rdtsc's are in the kernel, not userland.
> 
> > > The manpage for clock_gettime ("man 3 clock_gettime")
> > > has a lengthy caveat about using clock_gettime on
> > > an SMP system BUT provides a mechanism to test to
> > > see if your SMP system is safe, clock_getcpuclockid(0).
> > 
> > My manpages don't have anything like that for clock_gettime().
> 
> RHEL5 does.  Though now I am not sure how to use the
> section 3 calls.
> 
> > That caveat is for CLOCK_PROCESS_CPUTIME_ID which doesn't seem
> > reasonable to count on for monotonicity (my manpages
> > CLOCK_PROCESS_CPUTIME_ID just refer to as "High resolution 
> per-process
> > timer.").   CLOCK_REALTIME or CLOCK_MONOTONIC is a different matter.
> > 
> > Using what CLOCK_?
> > 
> > I always see clock_gettime(CLOCK_MONOTONIC or CLOCK_REALTIME, 
> > ...) make
> > a syscall or vsyscall.  If you're testing any of the 
> > thread-local times
> > then I think its less interesting.
> > 
> >     J
> 
> I'll dig further next week.
> 
> Dan
>  
>