[RFC] timer_interrupt: Avoid device timeouts by freezing time if

[RFC] timer_interrupt: Avoid device timeouts by freezing time if

[Christoph Lameter]

In extraordinay circumstances (MCA init/ debugger invocation, hardware problems) the
system may not be able to process timer ticks for an extended period of time.

The timer interrupt will compensate as soon as the system becomes functional again by
calling do_timer for each missed tick. This will cause time to race forward in a very
fast way. Device drivers that wait for timeouts will find that the system times out
on everything and thus device drivers will conclude that the devices are not in
a functional state disabling them. The system then cannot continue from the frozen
state because the device drivers have given up.

This patch fixes that issue by checking if more than half a second has passed
since the last tick. If more than half a second has passed then we would need to do
around 500 calls to do_timer to compensate. So in order to avoid these timeouts
we act as if time has been frozen with the system and do not compensate for lost time.
Device drivers may still find that their outstanding requests have failed but they
will be able to reinitialize the device and the system can hopefully continue.

A consequence of this patch is that the wall clock will stand still if the no ticks
can be processed for more than half a second.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.13/arch/ia64/kernel/time.c
=================================--- linux-2.6.13.orig/arch/ia64/kernel/time.c	2005-08-28 16:41:01.000000000 -0700
+++ linux-2.6.13/arch/ia64/kernel/time.c	2005-09-09 14:45:37.000000000 -0700
@@ -55,6 +55,7 @@ static irqreturn_t
 timer_interrupt (int irq, void *dev_id, struct pt_regs *regs)
 {
 	unsigned long new_itm;
+	unsigned long itc;
 
 	if (unlikely(cpu_is_offline(smp_processor_id()))) {
 		return IRQ_HANDLED;
@@ -64,10 +65,25 @@ timer_interrupt (int irq, void *dev_id, 
 
 	new_itm = local_cpu_data->itm_next;
 
-	if (!time_after(ia64_get_itc(), new_itm))
+	itc = ia64_get_itc();
+	if (!time_after(itc, new_itm))
 		printk(KERN_ERR "Oops: timer tick before it's due (itc=%lx,itm=%lx)\n",
 		       ia64_get_itc(), new_itm);
 
+	/*
+	 * If more than half a second has passed since the last timer interrupt then
+	 * something significant froze the system. Skip the time adjustments
+	 * otherwise repeated calls to do_timer will trigger timeouts by devices.
+	 */
+	if (unlikely(time_after(itc, new_itm + HZ /2 * local_cpu_data->itm_delta))) {
+		new_itm = itc;
+		if (smp_processor_id() = TIME_KEEPER_ID) {
+			time_interpolator_reset();
+			printk(KERN_ERR "Oops: more than 0.5 seconds since last tick."
+				"Skipping time adjustments in order to avoid timeouts.\n");
+		}
+	}
+
 	profile_tick(CPU_PROFILING, regs);
 
 	while (1) {

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Magenheimer, Dan (HP Labs Fort Collins)]

I am aware of at least two ia64 virtualization systems
that rely on the existing behavior to compensate for
the fact that one guest linux may be inactive while another
is active.  This isn't to say that another solution
couldn't be found, but just turning off the existing
behavior doesn't seem like a good alternative. 

> -----Original Message-----
> From: linux-ia64-owner@vger.kernel.org 
> [mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of 
> Christoph Lameter
> Sent: Friday, September 09, 2005 4:02 PM
> To: linux-ia64@vger.kernel.org
> Subject: [RFC] timer_interrupt: Avoid device timeouts by 
> freezing time if system froze
> 
> In extraordinay circumstances (MCA init/ debugger invocation, 
> hardware problems) the
> system may not be able to process timer ticks for an extended 
> period of time.
> 
> The timer interrupt will compensate as soon as the system 
> becomes functional again by
> calling do_timer for each missed tick. This will cause time 
> to race forward in a very
> fast way. Device drivers that wait for timeouts will find 
> that the system times out
> on everything and thus device drivers will conclude that the 
> devices are not in
> a functional state disabling them. The system then cannot 
> continue from the frozen
> state because the device drivers have given up.
> 
> This patch fixes that issue by checking if more than half a 
> second has passed
> since the last tick. If more than half a second has passed 
> then we would need to do
> around 500 calls to do_timer to compensate. So in order to 
> avoid these timeouts
> we act as if time has been frozen with the system and do not 
> compensate for lost time.
> Device drivers may still find that their outstanding requests 
> have failed but they
> will be able to reinitialize the device and the system can 
> hopefully continue.
> 
> A consequence of this patch is that the wall clock will stand 
> still if the no ticks
> can be processed for more than half a second.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: linux-2.6.13/arch/ia64/kernel/time.c
> =================================> --- linux-2.6.13.orig/arch/ia64/kernel/time.c	2005-08-28 
> 16:41:01.000000000 -0700
> +++ linux-2.6.13/arch/ia64/kernel/time.c	2005-09-09 
> 14:45:37.000000000 -0700
> @@ -55,6 +55,7 @@ static irqreturn_t
>  timer_interrupt (int irq, void *dev_id, struct pt_regs *regs)
>  {
>  	unsigned long new_itm;
> +	unsigned long itc;
>  
>  	if (unlikely(cpu_is_offline(smp_processor_id()))) {
>  		return IRQ_HANDLED;
> @@ -64,10 +65,25 @@ timer_interrupt (int irq, void *dev_id, 
>  
>  	new_itm = local_cpu_data->itm_next;
>  
> -	if (!time_after(ia64_get_itc(), new_itm))
> +	itc = ia64_get_itc();
> +	if (!time_after(itc, new_itm))
>  		printk(KERN_ERR "Oops: timer tick before it's 
> due (itc=%lx,itm=%lx)\n",
>  		       ia64_get_itc(), new_itm);
>  
> +	/*
> +	 * If more than half a second has passed since the last 
> timer interrupt then
> +	 * something significant froze the system. Skip the 
> time adjustments
> +	 * otherwise repeated calls to do_timer will trigger 
> timeouts by devices.
> +	 */
> +	if (unlikely(time_after(itc, new_itm + HZ /2 * 
> local_cpu_data->itm_delta))) {
> +		new_itm = itc;
> +		if (smp_processor_id() = TIME_KEEPER_ID) {
> +			time_interpolator_reset();
> +			printk(KERN_ERR "Oops: more than 0.5 
> seconds since last tick."
> +				"Skipping time adjustments in 
> order to avoid timeouts.\n");
> +		}
> +	}
> +
>  	profile_tick(CPU_PROFILING, regs);
>  
>  	while (1) {
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[David Mosberger-Tang]

I also would be nervous about the proposed patch.

I'm wondering: could the problem be avoided perhaps by running all
other pending (lower-priority) interrupts first when you detect a
large jump in elapsed time?  In other words, when you detect a jump
from time T1 to T2 with (T2-T1) greater than some threshold, you make
sure you run all pending interrupts while still at time T1 and only
after that is done you let time catch up to T2.

  --david

On 9/9/05, Magenheimer, Dan (HP Labs Fort Collins)
<dan.magenheimer@hp.com> wrote:
> I am aware of at least two ia64 virtualization systems
> that rely on the existing behavior to compensate for
> the fact that one guest linux may be inactive while another
> is active.  This isn't to say that another solution
> couldn't be found, but just turning off the existing
> behavior doesn't seem like a good alternative.
> 
> > -----Original Message-----
> > From: linux-ia64-owner@vger.kernel.org
> > [mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of
> > Christoph Lameter
> > Sent: Friday, September 09, 2005 4:02 PM
> > To: linux-ia64@vger.kernel.org
> > Subject: [RFC] timer_interrupt: Avoid device timeouts by
> > freezing time if system froze
> >
> > In extraordinay circumstances (MCA init/ debugger invocation,
> > hardware problems) the
> > system may not be able to process timer ticks for an extended
> > period of time.
> >
> > The timer interrupt will compensate as soon as the system
> > becomes functional again by
> > calling do_timer for each missed tick. This will cause time
> > to race forward in a very
> > fast way. Device drivers that wait for timeouts will find
> > that the system times out
> > on everything and thus device drivers will conclude that the
> > devices are not in
> > a functional state disabling them. The system then cannot
> > continue from the frozen
> > state because the device drivers have given up.
> >
> > This patch fixes that issue by checking if more than half a
> > second has passed
> > since the last tick. If more than half a second has passed
> > then we would need to do
> > around 500 calls to do_timer to compensate. So in order to
> > avoid these timeouts
> > we act as if time has been frozen with the system and do not
> > compensate for lost time.
> > Device drivers may still find that their outstanding requests
> > have failed but they
> > will be able to reinitialize the device and the system can
> > hopefully continue.
> >
> > A consequence of this patch is that the wall clock will stand
> > still if the no ticks
> > can be processed for more than half a second.
> >
> > Signed-off-by: Christoph Lameter <clameter@sgi.com>
> >
> > Index: linux-2.6.13/arch/ia64/kernel/time.c
> > =================================> > --- linux-2.6.13.orig/arch/ia64/kernel/time.c 2005-08-28
> > 16:41:01.000000000 -0700
> > +++ linux-2.6.13/arch/ia64/kernel/time.c      2005-09-09
> > 14:45:37.000000000 -0700
> > @@ -55,6 +55,7 @@ static irqreturn_t
> >  timer_interrupt (int irq, void *dev_id, struct pt_regs *regs)
> >  {
> >       unsigned long new_itm;
> > +     unsigned long itc;
> >
> >       if (unlikely(cpu_is_offline(smp_processor_id()))) {
> >               return IRQ_HANDLED;
> > @@ -64,10 +65,25 @@ timer_interrupt (int irq, void *dev_id,
> >
> >       new_itm = local_cpu_data->itm_next;
> >
> > -     if (!time_after(ia64_get_itc(), new_itm))
> > +     itc = ia64_get_itc();
> > +     if (!time_after(itc, new_itm))
> >               printk(KERN_ERR "Oops: timer tick before it's
> > due (itc=%lx,itm=%lx)\n",
> >                      ia64_get_itc(), new_itm);
> >
> > +     /*
> > +      * If more than half a second has passed since the last
> > timer interrupt then
> > +      * something significant froze the system. Skip the
> > time adjustments
> > +      * otherwise repeated calls to do_timer will trigger
> > timeouts by devices.
> > +      */
> > +     if (unlikely(time_after(itc, new_itm + HZ /2 *
> > local_cpu_data->itm_delta))) {
> > +             new_itm = itc;
> > +             if (smp_processor_id() = TIME_KEEPER_ID) {
> > +                     time_interpolator_reset();
> > +                     printk(KERN_ERR "Oops: more than 0.5
> > seconds since last tick."
> > +                             "Skipping time adjustments in
> > order to avoid timeouts.\n");
> > +             }
> > +     }
> > +
> >       profile_tick(CPU_PROFILING, regs);
> >
> >       while (1) {
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-ia64" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Luck, Tony]

>I am aware of at least two ia64 virtualization systems
>that rely on the existing behavior to compensate for
>the fact that one guest linux may be inactive while another
>is active.  This isn't to say that another solution
>couldn't be found, but just turning off the existing
>behavior doesn't seem like a good alternative. 

There must be some minimum frequency at which a hypervisor
allows a guest to run in order for it to operate normally
[e.g. a guest that gets no cpu time for several seconds at
a stretch will experience network time-outs with external
systems that it is unable to supply with "keep-alive" packets].

I'm not sure what that minimum frequency is, but I expect that
it may be contrained by some of the shorter TCP timeouts.  I
think that there is one around the 200 milli-second mark.

So possibly a hypervisor that starves a guest for long enough
to trigger Christoph's patch has other problems than just
keeping time correct in the guest.

-Tony

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time

[Christoph Lameter]

On Fri, 9 Sep 2005, David Mosberger-Tang wrote:

> I also would be nervous about the proposed patch.

Yes, I am also a bit concerned but there seems to be no other 
good solution.
 
> I'm wondering: could the problem be avoided perhaps by running all
> other pending (lower-priority) interrupts first when you detect a
> large jump in elapsed time?  In other words, when you detect a jump
> from time T1 to T2 with (T2-T1) greater than some threshold, you make
> sure you run all pending interrupts while still at time T1 and only
> after that is done you let time catch up to T2.

I think we are not talking about hardware interupts. Those can happen 
anytime and given the system was frozen for an extended time period 
likely have already completed before the timer interrupt happens for 
the first time after the system begins to unfreeze.

I think the main issue are drivers events. These are dependent on jiffies 
(which may be checked by drivers in a varity of ways) or are timer events 
(which also may expect jiffies to have a certain value).

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[David Mosberger-Tang]

I see your point about non-interrupt sources but the idea can be
generalized: rather than letting time catch up in a single huge jump,
do it gradually.  Say for every timer tick, you advance by one extra
tick.  I _think_ that could work well in practice.

  --david

On 9/9/05, Christoph Lameter <clameter@engr.sgi.com> wrote:
> On Fri, 9 Sep 2005, David Mosberger-Tang wrote:
> 
> > I also would be nervous about the proposed patch.
> 
> Yes, I am also a bit concerned but there seems to be no other
> good solution.
> 
> > I'm wondering: could the problem be avoided perhaps by running all
> > other pending (lower-priority) interrupts first when you detect a
> > large jump in elapsed time?  In other words, when you detect a jump
> > from time T1 to T2 with (T2-T1) greater than some threshold, you make
> > sure you run all pending interrupts while still at time T1 and only
> > after that is done you let time catch up to T2.
> 
> I think we are not talking about hardware interupts. Those can happen
> anytime and given the system was frozen for an extended time period
> likely have already completed before the timer interrupt happens for
> the first time after the system begins to unfreeze.
> 
> I think the main issue are drivers events. These are dependent on jiffies
> (which may be checked by drivers in a varity of ways) or are timer events
> (which also may expect jiffies to have a certain value).
> 


-- 
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Magenheimer, Dan (HP Labs Fort Collins)]

We've found that guests are pretty resilient to
"timeouts", though its quite possible they've never
really exceeded 200 milliseconds.

> -----Original Message-----
> From: Luck, Tony [mailto:tony.luck@intel.com] 
> Sent: Friday, September 09, 2005 4:36 PM
> To: Magenheimer, Dan (HP Labs Fort Collins); Christoph 
> Lameter; linux-ia64@vger.kernel.org
> Subject: RE: [RFC] timer_interrupt: Avoid device timeouts by 
> freezing time if system froze
> 
> >I am aware of at least two ia64 virtualization systems
> >that rely on the existing behavior to compensate for
> >the fact that one guest linux may be inactive while another
> >is active.  This isn't to say that another solution
> >couldn't be found, but just turning off the existing
> >behavior doesn't seem like a good alternative. 
> 
> There must be some minimum frequency at which a hypervisor
> allows a guest to run in order for it to operate normally
> [e.g. a guest that gets no cpu time for several seconds at
> a stretch will experience network time-outs with external
> systems that it is unable to supply with "keep-alive" packets].
> 
> I'm not sure what that minimum frequency is, but I expect that
> it may be contrained by some of the shorter TCP timeouts.  I
> think that there is one around the 200 milli-second mark.
> 
> So possibly a hypervisor that starves a guest for long enough
> to trigger Christoph's patch has other problems than just
> keeping time correct in the guest.
> 
> -Tony
> 

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Magenheimer, Dan (HP Labs Fort Collins)]

It seems like both "test for too much time passed"
and "recover from too much time passed" at least ought
to be out-of-line and easily overridden, e.g. by adding
a level of abstraction along the lines of default_idle.

> -----Original Message-----
> From: dmosberger@gmail.com [mailto:dmosberger@gmail.com] On 
> Behalf Of David Mosberger-Tang
> Sent: Friday, September 09, 2005 5:18 PM
> To: Christoph Lameter
> Cc: Magenheimer, Dan (HP Labs Fort Collins); 
> linux-ia64@vger.kernel.org
> Subject: Re: [RFC] timer_interrupt: Avoid device timeouts by 
> freezing time if system froze
> 
> I see your point about non-interrupt sources but the idea can be
> generalized: rather than letting time catch up in a single huge jump,
> do it gradually.  Say for every timer tick, you advance by one extra
> tick.  I _think_ that could work well in practice.
> 
>   --david
> 
> On 9/9/05, Christoph Lameter <clameter@engr.sgi.com> wrote:
> > On Fri, 9 Sep 2005, David Mosberger-Tang wrote:
> > 
> > > I also would be nervous about the proposed patch.
> > 
> > Yes, I am also a bit concerned but there seems to be no other
> > good solution.
> > 
> > > I'm wondering: could the problem be avoided perhaps by running all
> > > other pending (lower-priority) interrupts first when you detect a
> > > large jump in elapsed time?  In other words, when you 
> detect a jump
> > > from time T1 to T2 with (T2-T1) greater than some 
> threshold, you make
> > > sure you run all pending interrupts while still at time 
> T1 and only
> > > after that is done you let time catch up to T2.
> > 
> > I think we are not talking about hardware interupts. Those 
> can happen
> > anytime and given the system was frozen for an extended time period
> > likely have already completed before the timer interrupt happens for
> > the first time after the system begins to unfreeze.
> > 
> > I think the main issue are drivers events. These are 
> dependent on jiffies
> > (which may be checked by drivers in a varity of ways) or 
> are timer events
> > (which also may expect jiffies to have a certain value).
> > 
> 
> 
> -- 
> Mosberger Consulting LLC, voice/fax: 510-744-9372,
> http://www.mosberger-consulting.com/
> 35706 Runckel Lane, Fremont, CA 94536
> 

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Tian, Kevin]

>From: Luck, Tony
>Sent: 2005Äê9ÔÂ10ÈÕ 6:36
>
>>I am aware of at least two ia64 virtualization systems
>>that rely on the existing behavior to compensate for
>>the fact that one guest linux may be inactive while another
>>is active.  This isn't to say that another solution
>>couldn't be found, but just turning off the existing
>>behavior doesn't seem like a good alternative.
>
>There must be some minimum frequency at which a hypervisor
>allows a guest to run in order for it to operate normally
>[e.g. a guest that gets no cpu time for several seconds at
>a stretch will experience network time-outs with external
>systems that it is unable to supply with "keep-alive" packets].
>
>I'm not sure what that minimum frequency is, but I expect that
>it may be contrained by some of the shorter TCP timeouts.  I
>think that there is one around the 200 milli-second mark.

Yes, this is one important facet in virtualization system. In one of our time virtualization approaches, we add max_jump for max suspension period when one guest is scheduled off. Once the guest is back alive, check is made upon max_jump whether exceeds. If exceeds, it means that we can't show a virtual itc to guest with actual drift which may make device request timeout. Instead we record the actual drift in another field, and only show guest with a max_jump increment on virtual itc. Next time the cached drift value will be checked again, and compensate virtual itc gradually. Max_jump is set as 500ms by default, which is tunable as Tony's example here.

Thanks,
Kevin

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Tian, Kevin]

Hi, Christoph,
	Whether gettimeofday will be influenced and can wall clock catch up later? Seems that time interpolator can compensate for lost jiffies, but I'm not sure here. ;-)

Regards,
Kevin
>-----Original Message-----
>From: linux-ia64-owner@vger.kernel.org [mailto:linux-ia64-owner@vger.kernel.org]
>On Behalf Of Christoph Lameter
>Sent: 2005Äê9ÔÂ10ÈÕ 6:02
>To: linux-ia64@vger.kernel.org
>Subject: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze
>
>In extraordinay circumstances (MCA init/ debugger invocation, hardware problems)
>the
>system may not be able to process timer ticks for an extended period of time.
>
>The timer interrupt will compensate as soon as the system becomes functional again
>by
>calling do_timer for each missed tick. This will cause time to race forward in a very
>fast way. Device drivers that wait for timeouts will find that the system times out
>on everything and thus device drivers will conclude that the devices are not in
>a functional state disabling them. The system then cannot continue from the frozen
>state because the device drivers have given up.
>
>This patch fixes that issue by checking if more than half a second has passed
>since the last tick. If more than half a second has passed then we would need to do
>around 500 calls to do_timer to compensate. So in order to avoid these timeouts
>we act as if time has been frozen with the system and do not compensate for lost
>time.
>Device drivers may still find that their outstanding requests have failed but they
>will be able to reinitialize the device and the system can hopefully continue.
>
>A consequence of this patch is that the wall clock will stand still if the no ticks
>can be processed for more than half a second.
>
>Signed-off-by: Christoph Lameter <clameter@sgi.com>
>
>Index: linux-2.6.13/arch/ia64/kernel/time.c
>================================
>=>--- linux-2.6.13.orig/arch/ia64/kernel/time.c	2005-08-28 16:41:01.000000000 -0700
>+++ linux-2.6.13/arch/ia64/kernel/time.c	2005-09-09 14:45:37.000000000 -0700
>@@ -55,6 +55,7 @@ static irqreturn_t
> timer_interrupt (int irq, void *dev_id, struct pt_regs *regs)
> {
> 	unsigned long new_itm;
>+	unsigned long itc;
>
> 	if (unlikely(cpu_is_offline(smp_processor_id()))) {
> 		return IRQ_HANDLED;
>@@ -64,10 +65,25 @@ timer_interrupt (int irq, void *dev_id,
>
> 	new_itm = local_cpu_data->itm_next;
>
>-	if (!time_after(ia64_get_itc(), new_itm))
>+	itc = ia64_get_itc();
>+	if (!time_after(itc, new_itm))
> 		printk(KERN_ERR "Oops: timer tick before it's due (itc=%lx,itm=%lx)\n",
> 		       ia64_get_itc(), new_itm);
>
>+	/*
>+	 * If more than half a second has passed since the last timer interrupt then
>+	 * something significant froze the system. Skip the time adjustments
>+	 * otherwise repeated calls to do_timer will trigger timeouts by devices.
>+	 */
>+	if (unlikely(time_after(itc, new_itm + HZ /2 * local_cpu_data->itm_delta))) {
>+		new_itm = itc;
>+		if (smp_processor_id() = TIME_KEEPER_ID) {
>+			time_interpolator_reset();
>+			printk(KERN_ERR "Oops: more than 0.5 seconds since last tick."
>+				"Skipping time adjustments in order to avoid timeouts.\n");
>+		}
>+	}
>+
> 	profile_tick(CPU_PROFILING, regs);
>
> 	while (1) {
>-
>To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time

[Christoph Lameter]

On Fri, 9 Sep 2005, David Mosberger-Tang wrote:

> I see your point about non-interrupt sources but the idea can be
> generalized: rather than letting time catch up in a single huge jump,
> do it gradually.  Say for every timer tick, you advance by one extra
> tick.  I _think_ that could work well in practice.

The time interpolator can do that too. If we remove the 
time_interpolator_reset() then we will have a huge difference between 
xtime and gettimeofday() that is slowly reduced.

However, any mechanism like that will have a time running strangely fast 
for awhile which may cause other subsystems to misbehave.

I still think that the proposed patch is the best compromise solution 
since it restored the normal flow of time as soon as possible. 

Maybe the time period for the time freeze to take effect should be 
configurable?

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[David Mosberger-Tang]

On 9/19/05, Christoph Lameter <clameter@engr.sgi.com> wrote:

> However, any mechanism like that will have a time running strangely fast
> for awhile which may cause other subsystems to misbehave.

That's already true due to NTP.

> I still think that the proposed patch is the best compromise solution
> since it restored the normal flow of time as soon as possible.

I disagree.  Time jumps are much worse than making time go slightly
faster/slower for a period of time.  What's even better, the timeouts
you're seeing when making time jump almost certainly will just
disappear, which is extremely nice.

  --david
-- 
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time

[Christoph Lameter]

On Mon, 19 Sep 2005, David Mosberger-Tang wrote:

> > I still think that the proposed patch is the best compromise solution
> > since it restored the normal flow of time as soon as possible.
> 
> I disagree.  Time jumps are much worse than making time go slightly
> faster/slower for a period of time.  What's even better, the timeouts
> you're seeing when making time jump almost certainly will just
> disappear, which is extremely nice.

Ok then we can simply remove the time_interpolator_reset() call from 
my patch. Then the self-tuning interpolator logic will compensate for the 
difference over time.

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Luck, Tony]

>Ok then we can simply remove the time_interpolator_reset() call from 
>my patch. Then the self-tuning interpolator logic will compensate for the 
>difference over time.

Can you provide a bit more detail in the form of a concrete example
or two.  Suppose I stop the system (e.g. by entering a debugger) and
after poking around for sixty seconds, I restart the OS.

How long does it take for the interpolator to compensate for the
lost minute?

What is the relative rate of time while this compensation happens?

Does it make a difference if NTP is running?

-Tony

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time

[Christoph Lameter]

On Mon, 19 Sep 2005, Luck, Tony wrote:

> >Ok then we can simply remove the time_interpolator_reset() call from 
> >my patch. Then the self-tuning interpolator logic will compensate for the 
> >difference over time.
> 
> Can you provide a bit more detail in the form of a concrete example
> or two.  Suppose I stop the system (e.g. by entering a debugger) and
> after poking around for sixty seconds, I restart the OS.
> 
> How long does it take for the interpolator to compensate for the
> lost minute?

The first thing that happens is that the interpolator will generate a huge 
offset of 60 seconds. This means that the result of gettimeofday is 60 
seconds offset from xtime.

The interpolator typically looses a few nanoseconds (around 10-100 
nanoseconds) on each timer tick. If the clock source is tuned properly 
then this will be less than 10 nanoseconds per tick.

The interpolator adjusts itself every minute or so. At the next check 
of the accuracy of the time interpolator clock the interpolator will find 
that the clock source is running too slow since there is a huge offset 
and slightly tune the clock rate. This will increase the decrease of the offset
by the factor 2 ^ time_interpolator->shift.

The interpolator will continue detuning itself until the offset is 
consumed and xtime =~ gettimeofday(). 

Then the interval will be suddenly too short so the interpolator will 
gradually increase the interpolator clock rate again in 2 ^ 
time_interpolator->shift steps until a minimal offset is produced.

So gettimeofday will continue running just fine, The time jumps forward 
in the time interval may increase to a couple of hundred nanoseconds per 
tick intermittendly and then return to the default of around 10 
nanoseconds per interval.

I think the main issue will be the deviation of xtime from gettimeofday(). 
Some kernel subsystem use do_gettimeofday() and some others may access 
xtime.

> What is the relative rate of time while this compensation happens?

That depends on the time source and the tuning factor. I guestimate 
it will take about an hour to compensate for 60 seconds. The compensation 
is not linear but should be almost logarithmic since the interpolator will 
detune further if the time difference is greater.

> Does it make a difference if NTP is running?

NTP Time adjustment wont work right until time is okay again. But this 
should only result in relatively small deviations of up to a milisecond.

Guess we should test this approach. But this is bad hack.

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[David Mosberger-Tang]

> That depends on the time source and the tuning factor. I guestimate
> it will take about an hour to compensate for 60 seconds. The compensation
> is not linear but should be almost logarithmic since the interpolator will
> detune further if the time difference is greater.

That seems rather slow to me.  Of course, you don't want to catch up
by more than the smallest timeout that you'd like to avoid triggering.
 By that logic, it should be almost guaranteed to be save to account
for 2 ticks per timer interrupt.  That way, it would only take 30 sec
to correct from a 60 sec stand-still.  My gut feeling is that you
could be much more aggressive (e.g., 8 ticks/interrupt) but it might
be a good idea to experiment with this a bit, to get a feel for how
sensitive things really are.  If you wanted to be super-safe, you
could account an extra tick only every other interrupt (i.e., 1.5
ticks/interrupt on average), since timeouts have a rounding-error of
1/2 tick anyhow.

  --david
-- 
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time

[Christoph Lameter]

On Mon, 19 Sep 2005, David Mosberger-Tang wrote:

> > That depends on the time source and the tuning factor. I guestimate
> > it will take about an hour to compensate for 60 seconds. The compensation
> > is not linear but should be almost logarithmic since the interpolator will
> > detune further if the time difference is greater.
> 
> That seems rather slow to me.  Of course, you don't want to catch up
> by more than the smallest timeout that you'd like to avoid triggering.
>  By that logic, it should be almost guaranteed to be save to account
> for 2 ticks per timer interrupt.  That way, it would only take 30 sec
> to correct from a 60 sec stand-still.  My gut feeling is that you
> could be much more aggressive (e.g., 8 ticks/interrupt) but it might
> be a good idea to experiment with this a bit, to get a feel for how
> sensitive things really are.  If you wanted to be super-safe, you
> could account an extra tick only every other interrupt (i.e., 1.5
> ticks/interrupt on average), since timeouts have a rounding-error of
> 1/2 tick anyhow.

Doing two ticks per interrupt may cause hardware to fail that relies on 
the interval accurately being observed. I.e. lets say we have a device 
that typically responds after 10 ticks but times out after 15.

If we use 2 ticks per interrupt then we will timeout after 15/2 = 7 ticks 
before the device had a chance to do anything.

I think all these additional tricks just make the situation worse. My 
patch is already a bad hack and adding more logic just makes follow-on 
failures difficult to analyze. I'd say the cleanest solution is still to 
stick with freezing time. I also do not feel too enthusiastic about 
abusing the time interpolator tuning logic for compenstation.

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[David Mosberger-Tang]

On 9/19/05, Christoph Lameter <clameter@engr.sgi.com> wrote:

> Doing two ticks per interrupt may cause hardware to fail that relies on
> the interval accurately being observed. I.e. lets say we have a device
> that typically responds after 10 ticks but times out after 15.
> 
> If we use 2 ticks per interrupt then we will timeout after 15/2 = 7 ticks
> before the device had a chance to do anything.

You're forgetting that the real time has already passed!

> I think all these additional tricks just make the situation worse. My
> patch is already a bad hack and adding more logic just makes follow-on
> failures difficult to analyze. I'd say the cleanest solution is still to
> stick with freezing time. I also do not feel too enthusiastic about
> abusing the time interpolator tuning logic for compenstation.

It was you who suggested to use the time interpolator logic.  My
approach would have been just to keep track of the "deficit" in the
timer interrupt handler.

  --david
-- 
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time

[Christoph Lameter]

On Mon, 19 Sep 2005, David Mosberger-Tang wrote:

> On 9/19/05, Christoph Lameter <clameter@engr.sgi.com> wrote:
> 
> > Doing two ticks per interrupt may cause hardware to fail that relies on
> > the interval accurately being observed. I.e. lets say we have a device
> > that typically responds after 10 ticks but times out after 15.
> > 
> > If we use 2 ticks per interrupt then we will timeout after 15/2 = 7 ticks
> > before the device had a chance to do anything.
> 
> You're forgetting that the real time has already passed!

I did not think about the freeze occurring in this interval.

Think about what happens if the timeout is scheduled after the system has 
been unfrozen while the 2 ticks per tick hack is active. 

> > I think all these additional tricks just make the situation worse. My
> > patch is already a bad hack and adding more logic just makes follow-on
> > failures difficult to analyze. I'd say the cleanest solution is still to
> > stick with freezing time. I also do not feel too enthusiastic about
> > abusing the time interpolator tuning logic for compenstation.
> 
> It was you who suggested to use the time interpolator logic.  My
> approach would have been just to keep track of the "deficit" in the
> timer interrupt handler.

I suggested it because it does just involve removing a statement. Your 
approach would involve adding even more logic to the timer interrupt.

RE: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[Luck, Tony]

I don't think that I've followed all the steps and proposed
solutions in this thread.  There are (at least) two issues to
try to solve when the system has been frozen for an extended
period of time (more than a few ticks):

1) The system clock (gettimeofday(2)) needs to catch up the
missed time.  We can either do this in one mighty bound, or
we can creep up on it.  Either is plausible (though each may
have its own issues for applications, neither should be
catastrophic as applications should already deal with jumping
and slewing time due to NTP and other agents setting the
system clock).

2) There are some pending timeouts from the interval of real
time that we skipped.  Dealing with these may be harder, as
some of them may be related to physical limitations of devices,
so almost any choice we make about these (call all the pending
timeouts immediately, call them at some accelerated rate, skip
them) may cause problems for some device driver.  I think to
make any progress on a solution here we need to restrict the
discussion to real device drivers that are currently in the
tree.  Otherwise we will rathole forever discussing theoretical
situations.

Any other issues?

-Tony

Re: [RFC] timer_interrupt: Avoid device timeouts by freezing time if system froze

[David Mosberger-Tang]

On 9/19/05, Luck, Tony <tony.luck@intel.com> wrote:
> I don't think that I've followed all the steps and proposed
> solutions in this thread.  There are (at least) two issues to
> try to solve when the system has been frozen for an extended
> period of time (more than a few ticks):
> 
> 1) The system clock (gettimeofday(2)) needs to catch up the
> missed time.  We can either do this in one mighty bound, or
> we can creep up on it.  Either is plausible (though each may
> have its own issues for applications, neither should be
> catastrophic as applications should already deal with jumping
> and slewing time due to NTP and other agents setting the
> system clock).
> 
> 2) There are some pending timeouts from the interval of real
> time that we skipped.  Dealing with these may be harder, as
> some of them may be related to physical limitations of devices,
> so almost any choice we make about these (call all the pending
> timeouts immediately, call them at some accelerated rate, skip
> them) may cause problems for some device driver.  I think to
> make any progress on a solution here we need to restrict the
> discussion to real device drivers that are currently in the
> tree.  Otherwise we will rathole forever discussing theoretical
> situations.
> 
> Any other issues?

Sounds about right to me.  The point I was making is that issue (2) is
largely due to the "priority-inversion" that you're getting when
letting time catch up in one big jump: even driver actions that
completed successfully will appear to have failed because the timeout
handler is run long before the completion action that would have
disarmed the timer.  Now, there are other failures that you cannot
hide that way (e.g., an input buffer might overrun on a device), but
there is really nothing you can do about that when you halt the system
for such long periods (and I don't think it's a point of contention
anyhow).

  --david

-- 
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536