* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-13 15:56 ` Dan Magenheimer
@ 2010-10-13 16:02 ` Jeremy Fitzhardinge
2010-10-13 16:09 ` Keir Fraser
` (2 subsequent siblings)
3 siblings, 0 replies; 24+ messages in thread
From: Jeremy Fitzhardinge @ 2010-10-13 16:02 UTC (permalink / raw)
To: Dan Magenheimer
Cc: xen-devel, Keir Fraser, Shunli Yi, Jan Beulich, Saipu Liu,
Hang Du
On 10/13/2010 08:56 AM, Dan Magenheimer wrote:
> (bringing the topic up to a more theoretical level and including
> Keir and Jeremy)
>
> I wonder if the "bug" here is that "dependent wall clock" is
> an incredibly poor replacement for NTP (or a Windows Time Server)
> and a hack and really shouldn't even exist? I suppose one could
> argue that it makes some sense in a XCP environment, and maybe
> in a server environment where a single physical server is extremely
> isolated, but does it ever make sense in a real world server
> environment?
>
> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.
Yes, that's always been my view. pvops kernels don't implement it.
J
>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@novell.com]
>> Sent: Wednesday, October 13, 2010 6:38 AM
>> To: Hang Du
>> Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
>> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
>> sync up time from dom0 to domU
>>
>>>>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
>>> Sorry for my brief description in previous mail and missing
>>> is_initial_xendomain check. The kernel I submit this patch is
>>> linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
>> xendomain
>>> check.
>> Actually, I didn't expect you to blindly insert that check, but rather
>> either explain why only DomU-s need the changed behavior (as your
>> original description suggested), or refine the description of the
>> problem you're trying to solve.
>>
>>> In this patch, we support the backward time changing sync to all
>> domUs which
>>> configured to use "dependent wall clock".
>>> Currently, without the backward time syncing, when we change the time
>>> backward in Dom0, the time in DomU would be froze.
>>> And this cause some commands hang and don't executed until the time
>> catch up
>>> with the domU time.
>>> For example:
>>> "rpm -q kernel-xen"
>>> "sleep 1"
>>> Monotonic time should be reset when sync up time from dom0 to domU to
>>> support domU backward time syncing.
>> I can see your point, however ...
>>
>>> diff -urN a/arch/i386/kernel/time-xen.c b/arch/i386/kernel/time-
>> xen.c
>>> --- a/arch/i386/kernel/time-xen.c 2010-10-11 10:41:06.000000000
>> +0800
>>> +++ b/arch/i386/kernel/time-xen.c 2010-10-11 10:43:32.000000000
>> +0800
>>> @@ -715,6 +715,8 @@
>>> }
>>>
>>> if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
>>> + if (!is_initial_xendomain() && !independent_wallclock)
>>> + monotonic_reset();
>>> update_wallclock();
>> ... you definitely need to call monotonic_reset() *after* the
>> update_wallclock() (or else another vCPU, preempted in
>> do_gettimeofday() between the end of the xtime read loop
>> and the acquiring of the monotonic lock, would be able to
>> overwrite monotonic.tv with values obtained before the wall
>> clock update - similar to the secondary problem addressed by
>> c/s 1021).
>>
>> Further, blindly running a reset here doesn't seem nice in
>> the (general) case of the time getting updated forwards.
>>
>>> schedule_clock_was_set_work = 1;
>>> }
>> You'll also want to address Dan's concerns, and you will want to
>> get your patch formatted so that it would actually apply (read:
>> no tab -> space conversion) if you expect it to be committed
>> by Keir at some point.
>>
>> Jan
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-13 15:56 ` Dan Magenheimer
2010-10-13 16:02 ` Jeremy Fitzhardinge
@ 2010-10-13 16:09 ` Keir Fraser
2010-10-13 16:16 ` Tim Deegan
2010-10-14 2:36 ` Yi, Shunli
3 siblings, 0 replies; 24+ messages in thread
From: Keir Fraser @ 2010-10-13 16:09 UTC (permalink / raw)
To: Dan Magenheimer, Jan Beulich, Hang Du, Jeremy Fitzhardinge
Cc: Saipu Liu, Shunli Yi, xen-devel
On 13/10/2010 16:56, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:
> (bringing the topic up to a more theoretical level and including
> Keir and Jeremy)
>
> I wonder if the "bug" here is that "dependent wall clock" is
> an incredibly poor replacement for NTP (or a Windows Time Server)
> and a hack and really shouldn't even exist? I suppose one could
> argue that it makes some sense in a XCP environment, and maybe
> in a server environment where a single physical server is extremely
> isolated, but does it ever make sense in a real world server
> environment?
Yes it's a bad replacement for ntp. Pv_ops doesn't bother to implement it,
syncs with Xen wallclock once during boot and thereafter expects ntp to be
used within the guest.
-- Keir
> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.
>
>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@novell.com]
>> Sent: Wednesday, October 13, 2010 6:38 AM
>> To: Hang Du
>> Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
>> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
>> sync up time from dom0 to domU
>>
>>>>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
>>> Sorry for my brief description in previous mail and missing
>>> is_initial_xendomain check. The kernel I submit this patch is
>>> linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
>> xendomain
>>> check.
>>
>> Actually, I didn't expect you to blindly insert that check, but rather
>> either explain why only DomU-s need the changed behavior (as your
>> original description suggested), or refine the description of the
>> problem you're trying to solve.
>>
>>> In this patch, we support the backward time changing sync to all
>> domUs which
>>> configured to use "dependent wall clock".
>>> Currently, without the backward time syncing, when we change the time
>>> backward in Dom0, the time in DomU would be froze.
>>> And this cause some commands hang and don't executed until the time
>> catch up
>>> with the domU time.
>>> For example:
>>> "rpm -q kernel-xen"
>>> "sleep 1"
>>> Monotonic time should be reset when sync up time from dom0 to domU to
>>> support domU backward time syncing.
>>
>> I can see your point, however ...
>>
>>> diff -urN a/arch/i386/kernel/time-xen.c b/arch/i386/kernel/time-
>> xen.c
>>>
>>> --- a/arch/i386/kernel/time-xen.c 2010-10-11 10:41:06.000000000
>> +0800
>>> +++ b/arch/i386/kernel/time-xen.c 2010-10-11 10:43:32.000000000
>> +0800
>>> @@ -715,6 +715,8 @@
>>> }
>>>
>>> if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
>>> + if (!is_initial_xendomain() && !independent_wallclock)
>>> + monotonic_reset();
>>> update_wallclock();
>>
>> ... you definitely need to call monotonic_reset() *after* the
>> update_wallclock() (or else another vCPU, preempted in
>> do_gettimeofday() between the end of the xtime read loop
>> and the acquiring of the monotonic lock, would be able to
>> overwrite monotonic.tv with values obtained before the wall
>> clock update - similar to the secondary problem addressed by
>> c/s 1021).
>>
>> Further, blindly running a reset here doesn't seem nice in
>> the (general) case of the time getting updated forwards.
>>
>>> schedule_clock_was_set_work = 1;
>>> }
>>
>> You'll also want to address Dan's concerns, and you will want to
>> get your patch formatted so that it would actually apply (read:
>> no tab -> space conversion) if you expect it to be committed
>> by Keir at some point.
>>
>> Jan
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-13 15:56 ` Dan Magenheimer
2010-10-13 16:02 ` Jeremy Fitzhardinge
2010-10-13 16:09 ` Keir Fraser
@ 2010-10-13 16:16 ` Tim Deegan
2010-10-13 16:48 ` Jeremy Fitzhardinge
2010-10-14 2:36 ` Yi, Shunli
3 siblings, 1 reply; 24+ messages in thread
From: Tim Deegan @ 2010-10-13 16:16 UTC (permalink / raw)
To: Dan Magenheimer
Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Keir Fraser,
Shunli Yi, Jan Beulich, Saipu Liu, Hang Du
At 16:56 +0100 on 13 Oct (1286988987), Dan Magenheimer wrote:
> I wonder if the "bug" here is that "dependent wall clock" is
> an incredibly poor replacement for NTP (or a Windows Time Server)
> and a hack and really shouldn't even exist?
I've seen Windows Time server do some pretty odd things, but leaving
that aside - the problem with dependent wallclock is that it distributes
only the wallclock time and not the drift parameters, so you get
sawtoothing.
Running NTP in guests has its own problems, particularly around live
migration, where NTP can lose stability quite badly.
There was a paper about this at OSDI last week:
http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
Tim.
> I suppose one could
> argue that it makes some sense in a XCP environment, and maybe
> in a server environment where a single physical server is extremely
> isolated, but does it ever make sense in a real world server
> environment?
>
> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.
>
> > -----Original Message-----
> > From: Jan Beulich [mailto:JBeulich@novell.com]
> > Sent: Wednesday, October 13, 2010 6:38 AM
> > To: Hang Du
> > Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
> > Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> > sync up time from dom0 to domU
> >
> > >>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
> > > Sorry for my brief description in previous mail and missing
> > > is_initial_xendomain check. The kernel I submit this patch is
> > > linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
> > xendomain
> > > check.
> >
> > Actually, I didn't expect you to blindly insert that check, but rather
> > either explain why only DomU-s need the changed behavior (as your
> > original description suggested), or refine the description of the
> > problem you're trying to solve.
> >
> > > In this patch, we support the backward time changing sync to all
> > domUs which
> > > configured to use "dependent wall clock".
> > > Currently, without the backward time syncing, when we change the time
> > > backward in Dom0, the time in DomU would be froze.
> > > And this cause some commands hang and don't executed until the time
> > catch up
> > > with the domU time.
> > > For example:
> > > "rpm -q kernel-xen"
> > > "sleep 1"
> > > Monotonic time should be reset when sync up time from dom0 to domU to
> > > support domU backward time syncing.
> >
> > I can see your point, however ...
> >
> > > diff -urN a/arch/i386/kernel/time-xen.c b/arch/i386/kernel/time-
> > xen.c
> > >
> > > --- a/arch/i386/kernel/time-xen.c 2010-10-11 10:41:06.000000000
> > +0800
> > > +++ b/arch/i386/kernel/time-xen.c 2010-10-11 10:43:32.000000000
> > +0800
> > > @@ -715,6 +715,8 @@
> > > }
> > >
> > > if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
> > > + if (!is_initial_xendomain() && !independent_wallclock)
> > > + monotonic_reset();
> > > update_wallclock();
> >
> > ... you definitely need to call monotonic_reset() *after* the
> > update_wallclock() (or else another vCPU, preempted in
> > do_gettimeofday() between the end of the xtime read loop
> > and the acquiring of the monotonic lock, would be able to
> > overwrite monotonic.tv with values obtained before the wall
> > clock update - similar to the secondary problem addressed by
> > c/s 1021).
> >
> > Further, blindly running a reset here doesn't seem nice in
> > the (general) case of the time getting updated forwards.
> >
> > > schedule_clock_was_set_work = 1;
> > > }
> >
> > You'll also want to address Dan's concerns, and you will want to
> > get your patch formatted so that it would actually apply (read:
> > no tab -> space conversion) if you expect it to be committed
> > by Keir at some point.
> >
> > Jan
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-13 16:16 ` Tim Deegan
@ 2010-10-13 16:48 ` Jeremy Fitzhardinge
2010-10-14 9:07 ` Tim Deegan
0 siblings, 1 reply; 24+ messages in thread
From: Jeremy Fitzhardinge @ 2010-10-13 16:48 UTC (permalink / raw)
To: Tim Deegan
Cc: Dan Magenheimer, xen-devel@lists.xensource.com, Keir Fraser,
Shunli Yi, Jan Beulich, Saipu Liu, Hang Du
On 10/13/2010 09:16 AM, Tim Deegan wrote:
> At 16:56 +0100 on 13 Oct (1286988987), Dan Magenheimer wrote:
>> I wonder if the "bug" here is that "dependent wall clock" is
>> an incredibly poor replacement for NTP (or a Windows Time Server)
>> and a hack and really shouldn't even exist?
> I've seen Windows Time server do some pretty odd things, but leaving
> that aside - the problem with dependent wallclock is that it distributes
> only the wallclock time and not the drift parameters, so you get
> sawtoothing.
>
> Running NTP in guests has its own problems, particularly around live
> migration, where NTP can lose stability quite badly.
There seems to be some work on replacements for NTP which converge
quicker and are shinier and better.
I've thought that having some kind of lightweight dom0<->domU sync
protocol would be a better replacement for dependent time than NTP
(unless full NTP is required).
> There was a paper about this at OSDI last week:
> http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
Ooh, look, RADclock, just what I was thinking about.
J
> Tim.
>
>> I suppose one could
>> argue that it makes some sense in a XCP environment, and maybe
>> in a server environment where a single physical server is extremely
>> isolated, but does it ever make sense in a real world server
>> environment?
>>
>> In other words, I'm arguing that the correct "fix" here is:
>> Don't use dependent wallclock.
>>
>>> -----Original Message-----
>>> From: Jan Beulich [mailto:JBeulich@novell.com]
>>> Sent: Wednesday, October 13, 2010 6:38 AM
>>> To: Hang Du
>>> Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
>>> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
>>> sync up time from dom0 to domU
>>>
>>>>>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
>>>> Sorry for my brief description in previous mail and missing
>>>> is_initial_xendomain check. The kernel I submit this patch is
>>>> linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
>>> xendomain
>>>> check.
>>> Actually, I didn't expect you to blindly insert that check, but rather
>>> either explain why only DomU-s need the changed behavior (as your
>>> original description suggested), or refine the description of the
>>> problem you're trying to solve.
>>>
>>>> In this patch, we support the backward time changing sync to all
>>> domUs which
>>>> configured to use "dependent wall clock".
>>>> Currently, without the backward time syncing, when we change the time
>>>> backward in Dom0, the time in DomU would be froze.
>>>> And this cause some commands hang and don't executed until the time
>>> catch up
>>>> with the domU time.
>>>> For example:
>>>> "rpm -q kernel-xen"
>>>> "sleep 1"
>>>> Monotonic time should be reset when sync up time from dom0 to domU to
>>>> support domU backward time syncing.
>>> I can see your point, however ...
>>>
>>>> diff -urN a/arch/i386/kernel/time-xen.c b/arch/i386/kernel/time-
>>> xen.c
>>>> --- a/arch/i386/kernel/time-xen.c 2010-10-11 10:41:06.000000000
>>> +0800
>>>> +++ b/arch/i386/kernel/time-xen.c 2010-10-11 10:43:32.000000000
>>> +0800
>>>> @@ -715,6 +715,8 @@
>>>> }
>>>>
>>>> if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
>>>> + if (!is_initial_xendomain() && !independent_wallclock)
>>>> + monotonic_reset();
>>>> update_wallclock();
>>> ... you definitely need to call monotonic_reset() *after* the
>>> update_wallclock() (or else another vCPU, preempted in
>>> do_gettimeofday() between the end of the xtime read loop
>>> and the acquiring of the monotonic lock, would be able to
>>> overwrite monotonic.tv with values obtained before the wall
>>> clock update - similar to the secondary problem addressed by
>>> c/s 1021).
>>>
>>> Further, blindly running a reset here doesn't seem nice in
>>> the (general) case of the time getting updated forwards.
>>>
>>>> schedule_clock_was_set_work = 1;
>>>> }
>>> You'll also want to address Dan's concerns, and you will want to
>>> get your patch formatted so that it would actually apply (read:
>>> no tab -> space conversion) if you expect it to be committed
>>> by Keir at some point.
>>>
>>> Jan
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-13 16:48 ` Jeremy Fitzhardinge
@ 2010-10-14 9:07 ` Tim Deegan
2010-10-14 23:35 ` Jeremy Fitzhardinge
2010-10-15 14:09 ` [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU Dan Magenheimer
0 siblings, 2 replies; 24+ messages in thread
From: Tim Deegan @ 2010-10-14 9:07 UTC (permalink / raw)
To: Jeremy Fitzhardinge
Cc: Dan Magenheimer, xen-devel@lists.xensource.com, Keir Fraser,
Shunli Yi, Jan Beulich, Saipu Liu, Hang Du
At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> > There was a paper about this at OSDI last week:
> > http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
>
> Ooh, look, RADclock, just what I was thinking about.
Yes, it looks pretty good. Also they can use Xen stime as the local
oscillator and distribute drift numbers from xenstore, so no hypervisor
patches (and no hypervisor-interface changes) required. :)
Tim.
--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-14 9:07 ` Tim Deegan
@ 2010-10-14 23:35 ` Jeremy Fitzhardinge
2010-10-15 8:39 ` RADclock on Xen (was Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU) Tim Deegan
2010-10-15 14:09 ` [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU Dan Magenheimer
1 sibling, 1 reply; 24+ messages in thread
From: Jeremy Fitzhardinge @ 2010-10-14 23:35 UTC (permalink / raw)
To: Tim Deegan
Cc: Dan Magenheimer, xen-devel@lists.xensource.com, Keir Fraser,
Shunli Yi, Jan Beulich, Saipu Liu, Hang Du
On 10/14/2010 02:07 AM, Tim Deegan wrote:
> At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
>>> There was a paper about this at OSDI last week:
>>> http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
>> Ooh, look, RADclock, just what I was thinking about.
> Yes, it looks pretty good. Also they can use Xen stime as the local
> oscillator and distribute drift numbers from xenstore, so no hypervisor
> patches (and no hypervisor-interface changes) required. :)
Neat! Is it something we should encourage them to submit for xen-unstable?
J
^ permalink raw reply [flat|nested] 24+ messages in thread* RADclock on Xen (was Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU)
2010-10-14 23:35 ` Jeremy Fitzhardinge
@ 2010-10-15 8:39 ` Tim Deegan
[not found] ` <5869AFE5-6E86-46D8-8817-98DCAF39F7FC@unimelb.edu.au>
0 siblings, 1 reply; 24+ messages in thread
From: Tim Deegan @ 2010-10-15 8:39 UTC (permalink / raw)
To: Jeremy Fitzhardinge
Cc: Dan Magenheimer, xen-devel@lists.xensource.com, Keir Fraser,
Shunli Yi, dveitch, Jan Beulich, Saipu Liu, Hang Du
At 00:35 +0100 on 15 Oct (1287102942), Jeremy Fitzhardinge wrote:
> On 10/14/2010 02:07 AM, Tim Deegan wrote:
> > At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> >>> There was a paper about this at OSDI last week:
> >>> http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
> >> Ooh, look, RADclock, just what I was thinking about.
> > Yes, it looks pretty good. Also they can use Xen stime as the local
> > oscillator and distribute drift numbers from xenstore, so no hypervisor
> > patches (and no hypervisor-interface changes) required. :)
>
> Neat! Is it something we should encourage them to submit for xen-unstable?
Well I did, but there are no hypervisor patches. :)
CC'ing the speaker: are there any dom0 tools or kernel patches we could
be looking at for xen-unstable?
Regards,
Tim.
--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
^ permalink raw reply [flat|nested] 24+ messages in thread
* RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-14 9:07 ` Tim Deegan
2010-10-14 23:35 ` Jeremy Fitzhardinge
@ 2010-10-15 14:09 ` Dan Magenheimer
2010-10-15 14:19 ` Tim Deegan
1 sibling, 1 reply; 24+ messages in thread
From: Dan Magenheimer @ 2010-10-15 14:09 UTC (permalink / raw)
To: Tim Deegan, Jeremy Fitzhardinge
Cc: xen-devel, Keir Fraser, Shunli Yi, Jan Beulich, Saipu Liu,
Hang Du
> From: Tim Deegan [mailto:Tim.Deegan@citrix.com]
> Sent: Thursday, October 14, 2010 3:08 AM
> To: Jeremy Fitzhardinge
> Cc: Dan Magenheimer; Jan Beulich; Hang Du; Keir Fraser; Saipu Liu;
> Shunli Yi; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> sync up time from dom0 to domU
>
> At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> > > There was a paper about this at OSDI last week:
> > > http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
> >
> > Ooh, look, RADclock, just what I was thinking about.
>
> Yes, it looks pretty good. Also they can use Xen stime as the local
> oscillator and distribute drift numbers from xenstore, so no hypervisor
> patches (and no hypervisor-interface changes) required. :)
Maybe I'm misunderstanding the paper, but isn't it required
that Xen stime be directly readable for every attempt to
sample time (e.g. requiring at least some small interface
change such as adding a hypercall to obtain Xen stime)?
The current pvclock mechanism still depends on TSC to
interpolate between "Xen stime samples periodically written
to memory" to get adequate precision.
Also, for those certain enterprise applications that want
to sample time 10000+ samples/second/processor, and need
to know "immediately" when a sample might be bad (due
to, for example, live migration), I think each sample would
need to check xenstore. Is xenstore up to that kind of
pounding (and is it fast enough)?
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-15 14:09 ` [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU Dan Magenheimer
@ 2010-10-15 14:19 ` Tim Deegan
2010-10-15 14:46 ` Dan Magenheimer
[not found] ` <c049c7f4-3db4-4552-9c70-80f0a7a440e5@default 43468AF5-4D81-47E9-A6F7-2D8A25A432FD@unimelb.edu.au>
0 siblings, 2 replies; 24+ messages in thread
From: Tim Deegan @ 2010-10-15 14:19 UTC (permalink / raw)
To: Dan Magenheimer
Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Keir Fraser,
Shunli Yi, Jan Beulich, Saipu Liu, Hang Du
At 15:09 +0100 on 15 Oct (1287155382), Dan Magenheimer wrote:
> > At 17:48 +0100 on 13 Oct (1286992083), Jeremy Fitzhardinge wrote:
> > > > There was a paper about this at OSDI last week:
> > > > http://www.usenix.org/events/osdi10/tech/full_papers/Broomhead.pdf
> > >
> > > Ooh, look, RADclock, just what I was thinking about.
> >
> > Yes, it looks pretty good. Also they can use Xen stime as the local
> > oscillator and distribute drift numbers from xenstore, so no hypervisor
> > patches (and no hypervisor-interface changes) required. :)
>
> Maybe I'm misunderstanding the paper, but isn't it required
> that Xen stime be directly readable for every attempt to
> sample time (e.g. requiring at least some small interface
> change such as adding a hypercall to obtain Xen stime)?
You can obtain xen stime directly from the shared-into page and RDTSC.
There might need to be _kernel_ changes to make that available to
userspace, though my impression is that they push packet-timestamping
into the kernel.
> Also, for those certain enterprise applications that want
> to sample time 10000+ samples/second/processor, and need
> to know "immediately" when a sample might be bad (due
> to, for example, live migration), I think each sample would
> need to check xenstore. Is xenstore up to that kind of
> pounding (and is it fast enough)?
No it's certainly not (either of those things), but:
- the problem of turning a distributed wallclock into something suitable
for timestamping that aggressively is orthogonal to the problem of
distributing that wallclock in the first place; and
- all the user really needs is a generation counter to know that the
clock correction values are stale, and the kernel can provide that
alongside the stime.
Cheers,
Tim.
--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
^ permalink raw reply [flat|nested] 24+ messages in thread* RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-15 14:19 ` Tim Deegan
@ 2010-10-15 14:46 ` Dan Magenheimer
2010-10-15 14:53 ` Dan Magenheimer
2010-10-15 14:58 ` Tim Deegan
[not found] ` <c049c7f4-3db4-4552-9c70-80f0a7a440e5@default 43468AF5-4D81-47E9-A6F7-2D8A25A432FD@unimelb.edu.au>
1 sibling, 2 replies; 24+ messages in thread
From: Dan Magenheimer @ 2010-10-15 14:46 UTC (permalink / raw)
To: Tim Deegan
Cc: Jeremy Fitzhardinge, xen-devel, Keir Fraser, Shunli Yi,
Jan Beulich, Saipu Liu, Hang Du
> > Also, for those certain enterprise applications that want
> > to sample time 10000+ samples/second/processor, and need
> > to know "immediately" when a sample might be bad (due
> > to, for example, live migration), I think each sample would
> > need to check xenstore. Is xenstore up to that kind of
> > pounding (and is it fast enough)?
>
> No it's certainly not (either of those things), but:
> - the problem of turning a distributed wallclock into something
> suitable
> for timestamping that aggressively is orthogonal to the problem of
> distributing that wallclock in the first place; and
Agreed that this is a better solution to a very important problem.
I'm just trying to determine if it helps solve another time-related
real world problem.
> - all the user really needs is a generation counter to know that the
> clock correction values are stale, and the kernel can provide that
> alongside the stime.
Agreed that a generation counter solves this. BUT...
(1) afaik there is no generation counter in any time-related
kernel API; and
(2) afaict this generation counter would need to be "pushed"
from the dom0 kernel so would either (a) require domU kernels
to read from xenstore, or (b) require some kind of privileged
hypercall from dom0 to the hypervisor telling the hypervisor
to change the generation counter (and scaling values?) for all
domUs.
^ permalink raw reply [flat|nested] 24+ messages in thread* RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-15 14:46 ` Dan Magenheimer
@ 2010-10-15 14:53 ` Dan Magenheimer
2010-10-15 14:58 ` Tim Deegan
1 sibling, 0 replies; 24+ messages in thread
From: Dan Magenheimer @ 2010-10-15 14:53 UTC (permalink / raw)
To: Tim Deegan
Cc: Jeremy Fitzhardinge, xen-devel, Keir Fraser, Julien, Shunli Yi,
Darryl Veitch, Jan Beulich, Saipu Liu, Hang Du, Timothy Broomhead,
Ridoux
(RADclock authors cc'ed in attempt to re-merge the two threads...
Tim, if it's not too late, please reply to this email rather
than the previous one without the full cc list)
> From: Dan Magenheimer
> Sent: Friday, October 15, 2010 8:47 AM
> To: Tim Deegan
> Cc: Jeremy Fitzhardinge; Jan Beulich; Hang Du; Keir Fraser; Saipu Liu;
> Shunli Yi; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> sync up time from dom0 to domU
>
> > > Also, for those certain enterprise applications that want
> > > to sample time 10000+ samples/second/processor, and need
> > > to know "immediately" when a sample might be bad (due
> > > to, for example, live migration), I think each sample would
> > > need to check xenstore. Is xenstore up to that kind of
> > > pounding (and is it fast enough)?
> >
> > No it's certainly not (either of those things), but:
> > - the problem of turning a distributed wallclock into something
> > suitable
> > for timestamping that aggressively is orthogonal to the problem of
> > distributing that wallclock in the first place; and
>
> Agreed that this is a better solution to a very important problem.
> I'm just trying to determine if it helps solve another time-related
> real world problem.
>
> > - all the user really needs is a generation counter to know that the
> > clock correction values are stale, and the kernel can provide that
> > alongside the stime.
>
> Agreed that a generation counter solves this. BUT...
>
> (1) afaik there is no generation counter in any time-related
> kernel API; and
>
> (2) afaict this generation counter would need to be "pushed"
> from the dom0 kernel so would either (a) require domU kernels
> to read from xenstore, or (b) require some kind of privileged
> hypercall from dom0 to the hypervisor telling the hypervisor
> to change the generation counter (and scaling values?) for all
> domUs.
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-15 14:46 ` Dan Magenheimer
2010-10-15 14:53 ` Dan Magenheimer
@ 2010-10-15 14:58 ` Tim Deegan
1 sibling, 0 replies; 24+ messages in thread
From: Tim Deegan @ 2010-10-15 14:58 UTC (permalink / raw)
To: Dan Magenheimer
Cc: Jeremy Fitzhardinge, xen-devel@lists.xensource.com, Keir Fraser,
Shunli Yi, Jan Beulich, Saipu Liu, Hang Du
At 15:46 +0100 on 15 Oct (1287157606), Dan Magenheimer wrote:
> Agreed that this is a better solution to a very important problem.
> I'm just trying to determine if it helps solve another time-related
> real world problem.
Yep. AFAICS it doesn't really affect that at all. :|
> > - all the user really needs is a generation counter to know that the
> > clock correction values are stale, and the kernel can provide that
> > alongside the stime.
>
> Agreed that a generation counter solves this. BUT...
>
> (1) afaik there is no generation counter in any time-related
> kernel API; and
OK, but one could be added wherever the stime is exposed. This line of
reasoning runs into the earlier discussions about PV time protocols fopr
userspace (inluding process migration &c), and I've nothing useful to
add to that now. :)
> (2) afaict this generation counter would need to be "pushed"
> from the dom0 kernel so would either (a) require domU kernels
> to read from xenstore, or (b) require some kind of privileged
> hypercall from dom0 to the hypervisor telling the hypervisor
> to change the generation counter (and scaling values?) for all
> domUs.
I was under the impression that the only time you'd really need to
invalidate the xenstore values was on live migration, which the domU
kernel can certainly know about. One reason they use xenstore is that
the correction numbers are quite stable and don't need to be polled
every time.
Tim.
--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
^ permalink raw reply [flat|nested] 24+ messages in thread
[parent not found: <c049c7f4-3db4-4552-9c70-80f0a7a440e5@default 43468AF5-4D81-47E9-A6F7-2D8A25A432FD@unimelb.edu.au>]
* RE: [PATCH] time-xen : Reset monotonic time when sync up time from dom0 to domU
2010-10-13 15:56 ` Dan Magenheimer
` (2 preceding siblings ...)
2010-10-13 16:16 ` Tim Deegan
@ 2010-10-14 2:36 ` Yi, Shunli
3 siblings, 0 replies; 24+ messages in thread
From: Yi, Shunli @ 2010-10-14 2:36 UTC (permalink / raw)
To: Dan Magenheimer, Jan Beulich, Du, Hang, Keir Fraser,
Jeremy Fitzhardinge
Cc: Liu, Saipu, xen-devel@lists.xensource.com
[-- Attachment #1: Type: text/plain, Size: 5246 bytes --]
> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com]
> Sent: 2010年10月13日 23:56
> To: Jan Beulich; Du, Hang; Keir Fraser; Jeremy Fitzhardinge
> Cc: Liu, Saipu; Yi, Shunli; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when sync up
> time from dom0 to domU
>
> (bringing the topic up to a more theoretical level and including Keir and Jeremy)
>
> I wonder if the "bug" here is that "dependent wall clock" is an incredibly poor
> replacement for NTP (or a Windows Time Server) and a hack and really shouldn't
> even exist? I suppose one could argue that it makes some sense in a XCP
> environment, and maybe in a server environment where a single physical server
> is extremely isolated, but does it ever make sense in a real world server
> environment?
>
[Shunli] We encounter the time-syncing issue when deploy enterprise applications.
We suppose the "dependent wall clock" feature is for some enterprise environment (or products) which have uniform administrator and manager tool.
So, they can maintain the time on Dom0 only, and Xen would maintain the time in DomainUs.
Why we think this defect should be fixed?
1. We implemented a forward time syncing from Dom0 to DomU, also the backward time syncing should be supported.
2. Currently, backward time setting in Dom0 would cause some DomU applications hang (when set "independent_wallclock=0").
For example : rpm
3. We don't think the backward time setting is risk for DomUs, this only impacts the DomUs who set "independent_wallclock=0", and the administrator should always know what they are doing.
> In other words, I'm arguing that the correct "fix" here is:
> Don't use dependent wallclock.
[Shunli] Of cause, we can, don't use the dependent wallclock.
But, we still want to do something for those peoples who want the "dependent wallclock".
It's easy to fix, and we didn't hear known risks about changing time backward in DomU.
>
> > -----Original Message-----
> > From: Jan Beulich [mailto:JBeulich@novell.com]
> > Sent: Wednesday, October 13, 2010 6:38 AM
> > To: Hang Du
> > Cc: Saipu Liu; Shunli Yi; xen-devel@lists.xensource.com
> > Subject: RE: [Xen-devel] [PATCH] time-xen : Reset monotonic time when
> > sync up time from dom0 to domU
> >
> > >>> On 13.10.10 at 05:24, "Du, Hang" <hdu@websense.com> wrote:
> > > Sorry for my brief description in previous mail and missing
> > > is_initial_xendomain check. The kernel I submit this patch is
> > > linux-2.6.18-xen-3.4.2, I submit the patch again with in_initial-
> > xendomain
> > > check.
> >
> > Actually, I didn't expect you to blindly insert that check, but rather
> > either explain why only DomU-s need the changed behavior (as your
> > original description suggested), or refine the description of the
> > problem you're trying to solve.
> >
> > > In this patch, we support the backward time changing sync to all
> > domUs which
> > > configured to use "dependent wall clock".
> > > Currently, without the backward time syncing, when we change the
> > > time backward in Dom0, the time in DomU would be froze.
> > > And this cause some commands hang and don't executed until the time
> > catch up
> > > with the domU time.
> > > For example:
> > > "rpm -q kernel-xen"
> > > "sleep 1"
> > > Monotonic time should be reset when sync up time from dom0 to domU
> > > to support domU backward time syncing.
> >
> > I can see your point, however ...
> >
> > > diff -urN a/arch/i386/kernel/time-xen.c b/arch/i386/kernel/time-
> > xen.c
> > >
> > > --- a/arch/i386/kernel/time-xen.c 2010-10-11 10:41:06.000000000
> > +0800
> > > +++ b/arch/i386/kernel/time-xen.c 2010-10-11 10:43:32.000000000
> > +0800
> > > @@ -715,6 +715,8 @@
> > > }
> > >
> > > if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
> > > + if (!is_initial_xendomain() && !independent_wallclock)
> > > + monotonic_reset();
> > > update_wallclock();
> >
> > ... you definitely need to call monotonic_reset() *after* the
> > update_wallclock() (or else another vCPU, preempted in
> > do_gettimeofday() between the end of the xtime read loop and the
> > acquiring of the monotonic lock, would be able to overwrite
> > monotonic.tv with values obtained before the wall clock update -
> > similar to the secondary problem addressed by c/s 1021).
> >
> > Further, blindly running a reset here doesn't seem nice in the
> > (general) case of the time getting updated forwards.
> >
> > > schedule_clock_was_set_work = 1;
> > > }
> >
> > You'll also want to address Dan's concerns, and you will want to get
> > your patch formatted so that it would actually apply (read:
> > no tab -> space conversion) if you expect it to be committed by Keir
> > at some point.
> >
> > Jan
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
>
> To report this as spam, please forward to spam@websense.com. Thank you.
Protected by Websense Hosted Email Security -- www.websense.com
[-- Attachment #2: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 24+ messages in thread