[PATCH] Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER

[PATCH] Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER

[Tim Deegan]

[-- Attachment #1: Type: text/plain, Size: 230 bytes --]

because the default timesource (TSC) drifts under load.

The flag is only defined in ACPI 3.0, and we provide ACPI 2.0 tables,
but Windows seems happy enough to obey it anyway.

Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>



[-- Attachment #2: xen-unstable.hg.patch --]
[-- Type: text/x-patch, Size: 1522 bytes --]

# HG changeset patch
# User Tim Deegan <Tim.Deegan@citrix.com>
# Date 1275473757 -3600
# Node ID 39014be2745a52964fe7cdbb9bdc29e48b1b1fbc
# Parent  f86930e17ed9caf478b1498c855c82caaeaf23ff
Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER
because the default timesource (TSC) drifts under load.

The flag is only defined in ACPI 3.0, and we provide ACPI 2.0 tables,
but Windows seems happy enough to obey it anyway.

Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>

diff -r f86930e17ed9 -r 39014be2745a tools/firmware/hvmloader/acpi/acpi2_0.h
--- a/tools/firmware/hvmloader/acpi/acpi2_0.h	Wed Jun 02 11:15:52 2010 +0100
+++ b/tools/firmware/hvmloader/acpi/acpi2_0.h	Wed Jun 02 11:15:57 2010 +0100
@@ -255,6 +255,7 @@
 #define ACPI_SEALED_CASE    (1 << 11)
 #define ACPI_HEADLESS       (1 << 12)
 #define ACPI_CPU_SW_SLP     (1 << 13)
+#define ACPI_USE_PLATFORM_CLOCK (1 << 15)
 
 /*
  * Firmware ACPI Control Structure (FACS).
diff -r f86930e17ed9 -r 39014be2745a tools/firmware/hvmloader/acpi/static_tables.c
--- a/tools/firmware/hvmloader/acpi/static_tables.c	Wed Jun 02 11:15:52 2010 +0100
+++ b/tools/firmware/hvmloader/acpi/static_tables.c	Wed Jun 02 11:15:57 2010 +0100
@@ -70,7 +70,8 @@
     .iapc_boot_arch = ACPI_8042,
     .flags = (ACPI_PROC_C1 |
               ACPI_WBINVD |
-              ACPI_FIX_RTC | ACPI_TMR_VAL_EXT),
+              ACPI_FIX_RTC | ACPI_TMR_VAL_EXT |
+              ACPI_USE_PLATFORM_CLOCK),
 
     .reset_reg = {
         .address_space_id    = ACPI_SYSTEM_IO,

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER

[Dan Magenheimer]

Could you say more about this?  Which default timesource is
drifting?  Aren't all HVM platform timers emulated on top
of Xen system time?  Does w2k3 use rdtsc directly... under
what circumstances?

Thanks,
Dan

> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: Wednesday, June 02, 2010 4:19 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> FADT, to make w2k3 use the PMTIMER
> 
> because the default timesource (TSC) drifts under load.
> 
> The flag is only defined in ACPI 3.0, and we provide ACPI 2.0 tables,
> but Windows seems happy enough to obey it anyway.
> 
> Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> 
> 

Re: [PATCH] Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER

[Tim Deegan]

Hi, 

At 18:46 +0100 on 02 Jun (1275504378), Dan Magenheimer wrote:
> Could you say more about this?  Which default timesource is
> drifting?  Aren't all HVM platform timers emulated on top
> of Xen system time?  Does w2k3 use rdtsc directly... under
> what circumstances?

w2k3 will use RDTSC for its fine-grained time source (the one that's
exposed as a performance counter) even if the ACPI PMTIMER is available,
unless it's encouraged by this ACPI flag or the "/PMTIMER" boot.ini
switch.  W2k3sp2 and later Windowses use the PMTIMER by default,
presumably because of the let's-not-go-into-it-again difficulties with
using RDSTC as a time source.

http://blogs.technet.com/b/perfguru/archive/2008/02/18/explanation-for-the-usepmtimer-switch-in-the-boot-ini.aspx

Under stress testing, we found that this time source could drift (by
several percent) relative to real time, where the PMTIMER-based one did
not.

Yes, HVM timers are all based on the same source, except the RTC, which
is still implemented as a recurring timer event.  I suspect that the
drift is in the way Windows interprets the TSC values.  It could be, for
example, that it's warping forward to avoid cross-CPU
time-going-backward issues.

Cheers,

Tim.

> > -----Original Message-----
> > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > Sent: Wednesday, June 02, 2010 4:19 AM
> > To: xen-devel@lists.xensource.com
> > Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> > FADT, to make w2k3 use the PMTIMER
> > 
> > because the default timesource (TSC) drifts under load.
> > 
> > The flag is only defined in ACPI 3.0, and we provide ACPI 2.0 tables,
> > but Windows seems happy enough to obey it anyway.
> > 
> > Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> > 
> > 

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

RE: [PATCH] Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER

[Dan Magenheimer]

> presumably because of the let's-not-go-into-it-again difficulties with
> using RDSTC as a time source.

I promise, I'll try to be good :-)

I'm worried about this as a possible symptom of a bigger problem,
even if your patch provides an adequate workaround for one OS.

> w2k3 will use RDTSC for its fine-grained time source (the one that's
> exposed as a performance counter) even if the ACPI PMTIMER is
> available,

Wow, this is a surprise.  I thought Windows had eschewed all uses
of rdtsc.  I guess it is just too tempting ;-)

> Yes, HVM timers are all based on the same source, except the RTC, which
> is still implemented as a recurring timer event.  I suspect that the
> drift is in the way Windows interprets the TSC values.  It could be,
> for
> example, that it's warping forward to avoid cross-CPU
> time-going-backward issues.

So we're guessing that Windows is recognizing that raw TSC
is skewed and (like Xen and Solaris) providing some per-CPU
algorithm to correct for it.  Then, sadly, Xen changes
the vcpu-to-pcpu mapping and really confuses the correction
algorithm.

What tsc_mode is in use when this problem is encountered?
I'd think if tsc is emulated, there shouldn't be a problem
(and since pmtimer is emulated anyway, shouldn't be any slower).

> -----Original Message-----
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: Thursday, June 03, 2010 2:58 AM
> To: Dan Magenheimer
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> FADT, to make w2k3 use the PMTIMER
> 
> Hi,
> 
> At 18:46 +0100 on 02 Jun (1275504378), Dan Magenheimer wrote:
> > Could you say more about this?  Which default timesource is
> > drifting?  Aren't all HVM platform timers emulated on top
> > of Xen system time?  Does w2k3 use rdtsc directly... under
> > what circumstances?
> 
> w2k3 will use RDTSC for its fine-grained time source (the one that's
> exposed as a performance counter) even if the ACPI PMTIMER is
> available,
> unless it's encouraged by this ACPI flag or the "/PMTIMER" boot.ini
> switch.  W2k3sp2 and later Windowses use the PMTIMER by default,
> presumably because of the let's-not-go-into-it-again difficulties with
> using RDSTC as a time source.
> 
> http://blogs.technet.com/b/perfguru/archive/2008/02/18/explanation-for-
> the-usepmtimer-switch-in-the-boot-ini.aspx
> 
> Under stress testing, we found that this time source could drift (by
> several percent) relative to real time, where the PMTIMER-based one did
> not.
> 
> Yes, HVM timers are all based on the same source, except the RTC, which
> is still implemented as a recurring timer event.  I suspect that the
> drift is in the way Windows interprets the TSC values.  It could be,
> for
> example, that it's warping forward to avoid cross-CPU
> time-going-backward issues.
> 
> Cheers,
> 
> Tim.
> 
> > > -----Original Message-----
> > > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > > Sent: Wednesday, June 02, 2010 4:19 AM
> > > To: xen-devel@lists.xensource.com
> > > Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> > > FADT, to make w2k3 use the PMTIMER
> > >
> > > because the default timesource (TSC) drifts under load.
> > >
> > > The flag is only defined in ACPI 3.0, and we provide ACPI 2.0
> tables,
> > > but Windows seems happy enough to obey it anyway.
> > >
> > > Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> > >
> > >
> 
> --
> Tim Deegan <Tim.Deegan@citrix.com>
> Principal Software Engineer, XenServer Engineering
> Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: [PATCH] Add the USE_PLATFORM_TIMER flag to the FADT, to make w2k3 use the PMTIMER

[Tim Deegan]

At 17:41 +0100 on 03 Jun (1275586899), Dan Magenheimer wrote:
> So we're guessing that Windows is recognizing that raw TSC
> is skewed and (like Xen and Solaris) providing some per-CPU
> algorithm to correct for it.  Then, sadly, Xen changes
> the vcpu-to-pcpu mapping and really confuses the correction
> algorithm.

Something like that, yes, though the TSC offset code is supposed to mask
that at least to the TSC-sync precision that Xen can manage.  Or
possibly just the jitter added by scheduling and descheduling adds up
over time, because the guest only ever speeds up the stragglers -- I've
seen Xen's own system time do this on some older boxes, though not to
nearly the same extent.

> What tsc_mode is in use when this problem is encountered?

IIRC this was HVMPTM_no_delay_for_missed_ticks (on Xen 3.4 so no
emulated RDTSC).

> I'd think if tsc is emulated, there shouldn't be a problem
> (and since pmtimer is emulated anyway, shouldn't be any slower).

That sounds plausible.

Cheers,

Tim.

> > -----Original Message-----
> > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > Sent: Thursday, June 03, 2010 2:58 AM
> > To: Dan Magenheimer
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> > FADT, to make w2k3 use the PMTIMER
> > 
> > Hi,
> > 
> > At 18:46 +0100 on 02 Jun (1275504378), Dan Magenheimer wrote:
> > > Could you say more about this?  Which default timesource is
> > > drifting?  Aren't all HVM platform timers emulated on top
> > > of Xen system time?  Does w2k3 use rdtsc directly... under
> > > what circumstances?
> > 
> > w2k3 will use RDTSC for its fine-grained time source (the one that's
> > exposed as a performance counter) even if the ACPI PMTIMER is
> > available,
> > unless it's encouraged by this ACPI flag or the "/PMTIMER" boot.ini
> > switch.  W2k3sp2 and later Windowses use the PMTIMER by default,
> > presumably because of the let's-not-go-into-it-again difficulties with
> > using RDSTC as a time source.
> > 
> > http://blogs.technet.com/b/perfguru/archive/2008/02/18/explanation-for-
> > the-usepmtimer-switch-in-the-boot-ini.aspx
> > 
> > Under stress testing, we found that this time source could drift (by
> > several percent) relative to real time, where the PMTIMER-based one did
> > not.
> > 
> > Yes, HVM timers are all based on the same source, except the RTC, which
> > is still implemented as a recurring timer event.  I suspect that the
> > drift is in the way Windows interprets the TSC values.  It could be,
> > for
> > example, that it's warping forward to avoid cross-CPU
> > time-going-backward issues.
> > 
> > Cheers,
> > 
> > Tim.
> > 
> > > > -----Original Message-----
> > > > From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> > > > Sent: Wednesday, June 02, 2010 4:19 AM
> > > > To: xen-devel@lists.xensource.com
> > > > Subject: [Xen-devel] [PATCH] Add the USE_PLATFORM_TIMER flag to the
> > > > FADT, to make w2k3 use the PMTIMER
> > > >
> > > > because the default timesource (TSC) drifts under load.
> > > >
> > > > The flag is only defined in ACPI 3.0, and we provide ACPI 2.0
> > tables,
> > > > but Windows seems happy enough to obey it anyway.
> > > >
> > > > Signed-off-by: Tim Deegan <Tim.Deegan@citrix.com>
> > > >
> > > >
> > 
> > --
> > Tim Deegan <Tim.Deegan@citrix.com>
> > Principal Software Engineer, XenServer Engineering
> > Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)