xen PIT timer

xen PIT timer

[Ryan Harper]

Still looking at bug #195 which shows up occasionally  upon boot, and
now, I can recreate by generating a lot of traffic over the serial line
into Xen (holding down 'r' to print the run queues).  

I started comparing how Xen sets up and calibrates the timers with
Linux and have a few questions where they differ.

In Linux, the i386 and x86_64 set up the pit timer using 
binary, mode 2, LSB/MSB, ch 0, and then periodically read PIT_CH0 to
obtain a count.  Linux also uses PIT CH2 to do some calibration.

Xen, in xen/arch/x86/i8259.c sets up the PIT in the same manner (binary,
mode 2, LSB/MSB, ch 0)  but as far as I can see, PIT_CH0 is never used
again, specifically, PIT_CH0 is not used when handling a timer
interrupt.  

Instead, in xen/arch/x86/time.c, when Xen uses PIT as the platform timer,
it runs pit_read_counter() which gets a count from PIT_CH2, which was
used during calibrate_boot_tsc() but in mode 0.

I don't fully understand the different modes of the PIT, but looking at
the some intel [1]documentation I see:

    MODE 0: INTERRUPT ON TERMINAL COUNT
    Mode 0 is typically used for event counting After the Control Word
    is written OUT is initially low and will remain low until the
    Counter reaches zero OUT then goes high and remains high until a new
    count or a new Mode 0 Control Word is written into the Counter 

    MODE 2 RATE GENERATOR 
    This Mode functions like a divide-by-N counter It is typicially used
    to generate a Real Time Clock interrupt OUT will initially be high
    When the initial count has decremented to 1 OUT goes low for one CLK
    pulse OUT then goes high again the Counter reloads the initial count
    and the process is repeated Mode 2 is periodic the same sequence is
    repeated indefinitely For an initial count of N the sequence repeats
    every N CLK cycles 

Why does Xen chose to use mode 0, on PIT CH2 for calculating how much
time has passed rather than how Linux using PIT_CH0 in mode 2?  Are
there some trade offs?

1. http://www.cs.utexas.edu/users/dahlin/Classes/UGOS/reading/82C54.pdf

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: xen PIT timer

[Keir Fraser]

On 23 Sep 2005, at 20:49, Ryan Harper wrote:

> Why does Xen chose to use mode 0, on PIT CH2 for calculating how much
> time has passed rather than how Linux using PIT_CH0 in mode 2?  Are
> there some trade offs?

The way we track time in Xen is completely different to Linux. Linux 
tracks time off periodic interrupts from PIT_CH0: initially assuming 
each tick is 10ms but allowing that to be trimmed by adjtimex if ntpd 
is running.

Xen only uses PIT_CH0 as a fallback timer on uniprocessor systems with 
no local APIC: aprt from that it is basically unused. Xen doesn't rely 
on fixed periodic interrupts -- instead programming the local APIC 
timer to the next event of interest. We track time by assuming the PIT 
crystal is a precise 1.119380Mhz source: we can therefore measure the 
passage of time by reading how far PIT_CH2 has decremented and perform 
a multiplication. To more quickly estimate the passage of time, each 
CPU calibrates its local APIC oscillator to PIT_CH2 -- we can then 
estimate time-of-day with a RDTSC instruction + multiplication.

The main problem with using PIT as platform timer is that the counters 
are only 16 bits. We therefore periodically (every few 10s of ms) read 
the PIT_CH2 delta out into a wider 64-bit variable.

  -- Keir

Re: xen PIT timer

[Ryan Harper]

Keir,

Thanks for the explanation.  I'm still trying to reason out why we are
seeing the 'Time went backwards' every now and then, as well as being able
to forcibly create the issue via serial interrupt floods (holding 'r'
with serial input sent to Xen).

Is either assumption, PIT_CH0 ~= 10ms (Linux) or PIT_CH2 = 1.119380Mhz
source (Xen) more valid?  It sounds like either is valid.  Though Linux
can be adjusted via ntpd, is there any correcting factor for Xen?  I
know we can run ntpd in dom0 and it can update the wall clock timer, but
AFAICT, wall clock doesn't affect system_timestamp (which is where we
detect the Time went backwards in 
linux-2.6-xen-sparse/arch/x86/xen/i386/kernel/time.c).

Via some trace observation, I've noticed that the per-cpu time in Xen
(specifically stime_local_stamp) can vary widely between cpus.  Is this
the best source to be using for updating Linux system_timestamp since it
can vary significantly ( >1000000 ) between processors?

I haven't gotten around to doing it yet, but I was going to instrument
irq disable/enable to see how long we run with irq's disable with the
thought that we might be missing some events from which Xen derives time
calculations.  Is this a worthwhile investigation?

Do you have any other suggestions on where I should investigate?  I know
this isn't a problem for most folks, but we are still concerned that it
shows up every now and then on our platform under Xen, though we don't
see any of the Linux lost_tick stuff when running Linux.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: xen PIT timer

[Keir Fraser]

On 26 Sep 2005, at 16:22, Ryan Harper wrote:

> Keir,
>
> Thanks for the explanation.  I'm still trying to reason out why we are
> seeing the 'Time went backwards' every now and then, as well as being 
> able
> to forcibly create the issue via serial interrupt floods (holding 'r'
> with serial input sent to Xen).
>
> Is either assumption, PIT_CH0 ~= 10ms (Linux) or PIT_CH2 = 1.119380Mhz
> source (Xen) more valid?  It sounds like either is valid.  Though Linux
> can be adjusted via ntpd, is there any correcting factor for Xen?  I
> know we can run ntpd in dom0 and it can update the wall clock timer, 
> but
> AFAICT, wall clock doesn't affect system_timestamp (which is where we
> detect the Time went backwards in
> linux-2.6-xen-sparse/arch/x86/xen/i386/kernel/time.c).

NTP isn't plumbed thru into Xen yet. So ntpd will only affect the 
domain it is run in right now.

> Via some trace observation, I've noticed that the per-cpu time in Xen
> (specifically stime_local_stamp) can vary widely between cpus.  Is this
> the best source to be using for updating Linux system_timestamp since 
> it
> can vary significantly ( >1000000 ) between processors?

It is supposed to be a trustworthy source. Given we resync every CPU 
every second, being out by 1ms would indicate either a bug in the 
resync code or local oscillators jittering by 1000ppm, which is hard to 
believe!

> I haven't gotten around to doing it yet, but I was going to instrument
> irq disable/enable to see how long we run with irq's disable with the
> thought that we might be missing some events from which Xen derives 
> time
> calculations.  Is this a worthwhile investigation?

It would be interesting. Unless you are sync'ed to the PIT you should 
be able to go reasonably long periods with no timer interrupts with no 
ill effects (except the CPU time may get to wander off track a little 
more than it would otherwise have done). If you are sync'ed to the PIT 
(you have no cyclone, hpet or other chipset timer) then CPU0 needs to 
take a timer interrupt at least every 50ms or it will miss the 16-bit 
PIT counter wrapping.

> Do you have any other suggestions on where I should investigate?  I 
> know
> this isn't a problem for most folks, but we are still concerned that it
> shows up every now and then on our platform under Xen, though we don't
> see any of the Linux lost_tick stuff when running Linux.

Getting this stuff right seems a lot harder than it ought to be. There 
are clearly problems in the existing code -- if you've been able to 
ascertain that there are real sync issues between CPUs (ought by >1ms 
relative to each other) then that is a start. I would investigate how 
they manage to get so out of sync, when all oscillators in the system 
ought to be driven by crystals with stability much better than that.

  -- Keir

Re: xen PIT timer

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-09-26 10:45]:
> >I haven't gotten around to doing it yet, but I was going to instrument
> >irq disable/enable to see how long we run with irq's disable with the
> >thought that we might be missing some events from which Xen derives 
> >time
> >calculations.  Is this a worthwhile investigation?
> 
> It would be interesting. Unless you are sync'ed to the PIT you should 
> be able to go reasonably long periods with no timer interrupts with no 
> ill effects (except the CPU time may get to wander off track a little 
> more than it would otherwise have done). If you are sync'ed to the PIT 
> (you have no cyclone, hpet or other chipset timer) then CPU0 needs to 
> take a timer interrupt at least every 50ms or it will miss the 16-bit 
> PIT counter wrapping.

Interesting.  So, in the case of the dual-opteron box, we are slaved to
the PIT, and while there is an HPET (or at least Xen was happy when I
booted with hpet_force=1), it is not detectable via ACPI code
(i.e. ACPI tables don't include an ACPI_HPET table).

In the case where I can force the timer to miss via serial interrupts, I
believe we are preventing CPU0 from taking a timer interrupt within
50ms. 

The other case where I see 'Time went backwards' is during dom0 boot up.
I'll dig into tracking the frequency of timer interrupts.

Speaking of timer interrupts, why doesn't the xen timer_interrupt()
actually handle the platform timer read and overflow check there rather
than raising a softirq?

Thanks a lot for the help Keir.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com