RE: Guest TSC and Xen (Intel and AMD feedback please)

RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

Hi Andy --

Thanks for the reply.

No, I don't think the TSC offset capabilities in VT are sufficient.
If you are migrating from a TSC-synchronized SMP system and TSC
was selected as the clocksource by the guest at boot *because* TSC
is always synchronized, and then you migrate to a system where TSC
is not synchronized, Xen can synchronize it once at migrate time
but then the TSC's on the target system will immediately start to
diverge.  So TSC might be a reasonable clocksource on the first
system but not on the target system.  One could of course use
CPUID to disallow a TSC-unsynchronized host as a suitable target
for a TSC-synchronized-assumed guest, but that seems overly restrictive,
especially if TSC wasn't selected as the clocksource for the guest
and/or the guest (and its apps) isn't particularly time-sensitive.

> I don't know why you want to single out TSC here.

I'm singleing it out because it is a per-cpu clock rather
than a platform timer... a platform timer can be (and indeed
is) offset'ed on migration and that is sufficient if it is
selected as the clocksource.

> That is what Linux is testing for anyways. If it decides it is
> ok it is fine.

Not sure... if Linux thinks it is running on a uniprocessor,
but Xen reschedules this uniprocessor Linux guest on a different
processor on the same physical SMP system, does Xen adjust the
potential TSC difference?  I could be wrong, but I think not.

> The reason why it is an advantage to try to make TSC btw
> is that it is *much* faster than any other timer and there
> are definitely workloads that are very timer intensive.

Yes, understood, but if a timer-intensive application makes
the assumption that TSC is synchronized and thus will never
go backwards, but TSC is not synchronized and it DOES (apparently)
go backwards due to Xen scheduler or migration, a slower timer
might have been preferred.

Dan

> -----Original Message-----
> From: Andi Kleen [mailto:andi@firstfloor.org]
> Sent: Tuesday, July 01, 2008 2:42 PM
> To: dan.magenheimer@oracle.com
> Cc: Xen-Devel (E-mail)
> Subject: Re: Guest TSC and Xen (Intel and AMD feedback please)
> 
> 
> "Dan Magenheimer" <dan.magenheimer@oracle.com> writes:
> >
> > 1) Often this test is done once at guest boot; if the guest
> >    migrates to another machine without the bit set, time
> >    will be erratic.
> 
> First it doesn't have to be. Xen could use the TSC offset capabilities
> in VT to simulate a relatively [you would likely get a small hickup
> during the migration, but nothing really bad] stable TSC between
> machines per guest (assuming the TSC does not diverge by itself).
> That would require a TSC negotation phase during migration.
> 
> Then the tests are just a special instance of a CPUID bit test
> [although they sometimes check other things too, but from this
> perspective they are the same] and when your CPUID and other
> configuration values are not controlled between different systems you
> migrate to you cannot safely migrate anyways because everything else
> that depends on CPUID features will fail too.
> 
> I don't know why you want to single out TSC here.
> 
> > 2) I *think* that in some cases even within the same system,
> >    TSC values will skew somewhat.  Since a uniprocessor guest
> >    will often be rescheduled to a different pcpu by Xen,
> >    the underlying tsc may appear erratic.
> 
> That is what Linux is testing for anyways. If it decides it is
> ok it is fine.
> 
> The reason why it is an advantage to try to make TSC btw
> is that it is *much* faster than any other timer and there
> are definitely workloads that are very timer intensive.
> 
> -Andi
>

Re: RE: Guest TSC and Xen (Intel and AMD feedback please)

[Jeremy Fitzhardinge]

Dan Magenheimer wrote:
> Hi Andy --
>
> Thanks for the reply.
>
> No, I don't think the TSC offset capabilities in VT are sufficient.
> If you are migrating from a TSC-synchronized SMP system and TSC
> was selected as the clocksource by the guest at boot *because* TSC
> is always synchronized, and then you migrate to a system where TSC
> is not synchronized, Xen can synchronize it once at migrate time
> but then the TSC's on the target system will immediately start to
> diverge.  So TSC might be a reasonable clocksource on the first
> system but not on the target system.  One could of course use
> CPUID to disallow a TSC-unsynchronized host as a suitable target
> for a TSC-synchronized-assumed guest, but that seems overly restrictive,
> especially if TSC wasn't selected as the clocksource for the guest
> and/or the guest (and its apps) isn't particularly time-sensitive.
>   

Linux (as in, relatively modern kernels, at least) will downgrade the 
tsc if it starts misbehaving.  This can happen, for example, if the tsc 
seems fine on initial inspection, but then starts misbehaving when the 
cpu frequency changes.

On the other hand, if its a modern kernel you may as well use a pv linux 
kernel and get a proper pv clocksource.

This is an area in which kernel behaviour has been changing a lot over 
the last few years, so its hard to guess what any given kernel version 
(+hacked by vendor) will do in this case...

    J

RE: RE: Guest TSC and Xen (Intel and AMD feedback please)

[Tian, Kevin]

>From: Dan Magenheimer
>Sent: 2008年7月2日 5:43
>
>Hi Andy --
>
>Thanks for the reply.
>
>No, I don't think the TSC offset capabilities in VT are sufficient.
>If you are migrating from a TSC-synchronized SMP system and TSC
>was selected as the clocksource by the guest at boot *because* TSC
>is always synchronized, and then you migrate to a system where TSC
>is not synchronized, Xen can synchronize it once at migrate time
>but then the TSC's on the target system will immediately start to

If dest and src boxes are with same bits, saying same tsc freq, then 
the divergence is a fixed value and TSC offset adjustment is sufficient.
Or else TSC offset can't help.

>diverge.  So TSC might be a reasonable clocksource on the first
>system but not on the target system.  One could of course use
>CPUID to disallow a TSC-unsynchronized host as a suitable target
>for a TSC-synchronized-assumed guest, but that seems overly 
>restrictive,
>especially if TSC wasn't selected as the clocksource for the guest
>and/or the guest (and its apps) isn't particularly time-sensitive.

So if it's an option instead of a 'NEVER' clause, I agree as:

a) user may mark TSC unstable to migrate among boxes with 
mismatching TSC bits (bus crystal, cpu freq impact, etc.)

b) user may always use TSC as clocksource and then trap RDTSC
when migrating to a box with mismatching TSC bits

c) user may always use TSC as clocksource when migrating to a
box with same TSC bits, by adjusting TSC offset

d) migration may be prevented since no reliable methods to ensure
a)'s effect. Such prevention then falls into generic CPUID comparison
involved in migration

>
>> I don't know why you want to single out TSC here.
>
>I'm singleing it out because it is a per-cpu clock rather
>than a platform timer... a platform timer can be (and indeed
>is) offset'ed on migration and that is sufficient if it is
>selected as the clocksource.

The problem is not per-cpu vs platform, IMO. Instead, it's the 
problem that currently guest TSC is conveyed by host TSC plus 
an offset approach, without read trap. If you also virtualize a 
platform clocksource by a real one, like dedicating a HPET ch,
same concern also raises.

>
>> That is what Linux is testing for anyways. If it decides it is
>> ok it is fine.
>
>Not sure... if Linux thinks it is running on a uniprocessor,
>but Xen reschedules this uniprocessor Linux guest on a different
>processor on the same physical SMP system, does Xen adjust the
>potential TSC difference?  I could be wrong, but I think not.

Xen can do and should be, since SMP system is driven by same
crystal and thus host TSC is synced. But I guess by far Xen hasn't
do it, since the TSC drift (dozen of cycles) is smaller than the overhead
to migrate a vcpu. Thus guest won't observe a backward value in
theory.

>
>> The reason why it is an advantage to try to make TSC btw
>> is that it is *much* faster than any other timer and there
>> are definitely workloads that are very timer intensive.

Curiously, how much downgrade using a platform clock source may be,
for a time-intensive workload?

>
>Yes, understood, but if a timer-intensive application makes
>the assumption that TSC is synchronized and thus will never
>go backwards, but TSC is not synchronized and it DOES (apparently)
>go backwards due to Xen scheduler or migration, a slower timer
>might have been preferred.

Shouldn't this be a software bug instead?

Thanks,
Kevin

RE: RE: Guest TSC and Xen (Intel and AMD feedback please)

[Dan Magenheimer]

[-- Attachment #1: Type: text/plain, Size: 3193 bytes --]

Hi Kevin --

> a) user may mark TSC unstable to migrate among boxes with
> mismatching TSC bits (bus crystal, cpu freq impact, etc.)
>
> b) user may always use TSC as clocksource and then trap RDTSC
> when migrating to a box with mismatching TSC bits
>
> c) user may always use TSC as clocksource when migrating to a
> box with same TSC bits, by adjusting TSC offset
>
> d) migration may be prevented since no reliable methods to ensure
> a)'s effect. Such prevention then falls into generic CPUID comparison
> involved in migration

I think this is an excellent summary of the choices.

> >> I don't know why you want to single out TSC here.
> >
> >I'm singleing it out because it is a per-cpu clock rather
> >than a platform timer... a platform timer can be (and indeed
> >is) offset'ed on migration and that is sufficient if it is
> >selected as the clocksource.
>
> The problem is not per-cpu vs platform, IMO. Instead, it's the
> problem that currently guest TSC is conveyed by host TSC plus
> an offset approach, without read trap. If you also virtualize a
> platform clocksource by a real one, like dedicating a HPET ch,
> same concern also raises.

Yes, you are right.

> >> That is what Linux is testing for anyways. If it decides it is
> >> ok it is fine.
> >
> >Not sure... if Linux thinks it is running on a uniprocessor,
> >but Xen reschedules this uniprocessor Linux guest on a different
> >processor on the same physical SMP system, does Xen adjust the
> >potential TSC difference?  I could be wrong, but I think not.
>
> Xen can do and should be, since SMP system is driven by same
> crystal and thus host TSC is synced. But I guess by far Xen hasn't
> do it, since the TSC drift (dozen of cycles) is smaller than
> the overhead
> to migrate a vcpu. Thus guest won't observe a backward value in
> theory.

I'm not sure this is always the case (though the patch I posted
earlier today may indicate there is something else going on that
has led to my assumption that TSC was skewing worse than dozens
of cycles).

> >> The reason why it is an advantage to try to make TSC btw
> >> is that it is *much* faster than any other timer and there
> >> are definitely workloads that are very timer intensive.
>
> Curiously, how much downgrade using a platform clock source may be,
> for a time-intensive workload?

A good question.  We have a workload that spends >10% of its time
doing gettimeoffset_tsc()... not sure if that is realistic but
it would be interesting to measure that if it used an hpet instead
or if rdtsc was fully emulated.

> >Yes, understood, but if a timer-intensive application makes
> >the assumption that TSC is synchronized and thus will never
> >go backwards, but TSC is not synchronized and it DOES (apparently)
> >go backwards due to Xen scheduler or migration, a slower timer
> >might have been preferred.
>
> Shouldn't this be a software bug instead?

If the application is smart enough to check the TSC bits when
it launches, but stability changes later due to migration/scheduling,
I'm not sure htis is an application bug.  Or did I misunderstand
your question?

Thanks,
Dan

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel