Updated dynamic tick patches

Updated dynamic tick patches

[Srivatsa Vaddagiri]

I have cleaned up dynamic tick patch that Con last posted. One issue
that is still to be addressed precisely is recovering lost ticks.

This is supposedly much easier with something like ACPI PM timer, but
I found that the code which calculates lost ticks in timer_pm.c is
not accurate. I have attempted to fix it (in the patch that follows).
With the fix, time has remained stable for ~36 hrs on one machine,
but the same fix does not help in another machine (time speeds up
by couple of seconds after 4-6 hrs). Hence I consider that it needs 
some more rework. Suggestion on accurate lost-tick recovery are wellcome.

Using TSC to recover ticks also is more tricky and hence I have not
enabled TSC support in these patches.

This patch does not address those machines where all CPUs have to
be put to sleep simulaneously (otherwise they dont work well
or something like that), as pointed out by Tony. We could
add support for such machines in another release if they
are common enough to come by.


Following patches related to dynamic tick are posted in separate mails,
for convenience of review. The first patch probably applies w/o dynamic
tick consideration also.

Patch 1/3  -> Fixup lost tick calculation in timer_pm.c
Patch 2/3  -> Dyn-tick cleanups
Patch 3/3  -> Use lost tick information in dyn-tick time recovery 

These patches are against 2.6.13-rc6-mm2.

Con, would be great if you can upload a consolidated new version of
dyn-tick patch on your website!


-- 


Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Re: Updated dynamic tick patches

[Erik Andrén]

Great job Srivatsa,
Does these patches compile nicely against 2.6.13?
Otherwise would it be possible for you to repost patches made against 
2.6.13 instead for more public testing?

With best regards
Erik Andrén

Re: Updated dynamic tick patches

[Con Kolivas]

On Thu, 1 Sep 2005 02:58 am, Srivatsa Vaddagiri wrote:
> Following patches related to dynamic tick are posted in separate mails,
> for convenience of review. The first patch probably applies w/o dynamic
> tick consideration also.
>
> Patch 1/3  -> Fixup lost tick calculation in timer_pm.c
> Patch 2/3  -> Dyn-tick cleanups
> Patch 3/3  -> Use lost tick information in dyn-tick time recovery
>
> These patches are against 2.6.13-rc6-mm2.
>
> Con, would be great if you can upload a consolidated new version of
> dyn-tick patch on your website!

Great, thanks. I'll wait till 2.6.13-mm1 is out since that's due shortly and 
I'll resync everything with that and perhaps tweak along the way.

Cheers,
Con

Re: Updated dynamic tick patches

[Tony Lindgren]

[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]

* Con Kolivas <kernel@kolivas.org> [050901 08:22]:
> On Thu, 1 Sep 2005 02:58 am, Srivatsa Vaddagiri wrote:
> > Following patches related to dynamic tick are posted in separate mails,
> > for convenience of review. The first patch probably applies w/o dynamic
> > tick consideration also.
> >
> > Patch 1/3  -> Fixup lost tick calculation in timer_pm.c
> > Patch 2/3  -> Dyn-tick cleanups
> > Patch 3/3  -> Use lost tick information in dyn-tick time recovery
> >
> > These patches are against 2.6.13-rc6-mm2.
> >
> > Con, would be great if you can upload a consolidated new version of
> > dyn-tick patch on your website!
> 
> Great, thanks. I'll wait till 2.6.13-mm1 is out since that's due shortly and 
> I'll resync everything with that and perhaps tweak along the way.

I tried this quickly on a loaner ThinkPad T30, and needed the following
patch to compile. The patch does work with PIT, but with lapic the
system does not wake to timer interrupts :(

I also hacked together a little timer test utility that should go trough
on a completely idle system with no errors. Also posted it to:

http://www.muru.com/linux/dyntick/tools/dyntick-test.c

Srivatsa, could you try the dyntick-test.c on your system after booting
to init=/bin/sh to make the system as idle as possible?

Unfortunately I cannot debug the APIC issue right now, but I seem to
have an issue on ARM OMAP where the timer test occasionally fails on
some longer values, for example 3 second sleep can take 4 seconds.

I don't know yet if this is the problem George Anzinger mentioned with
next_timer_interrupt(), or if this is OMAP specific. But it only seems
to occur with very low idle HZ values. This may be related to the slow
boot time issue I mentioned yesterday.

Regards,

Tony

[-- Attachment #2: patch-fix-up-compile --]
[-- Type: text/plain, Size: 257 bytes --]

--- a/arch/i386/kernel/io_apic.c
+++ b/arch/i386/kernel/io_apic.c
@@ -2034,7 +2034,9 @@
 	.disable 	= mask_IO_APIC_irq,
 	.ack 		= ack_edge_ioapic,
 	.end 		= end_edge_ioapic,
+#ifdef CONFIG_SMP
 	.set_affinity 	= set_ioapic_affinity,
+#endif
 };
 #endif
 

[-- Attachment #3: dyntick-test.c --]
[-- Type: text/x-csrc, Size: 2927 bytes --]

/*
 * Tests timers to make sure dynamic tick works properly
 */

#include <stdio.h>
#include <unistd.h>
#include <sys/time.h>

#define MAX_SLEEP	(10)		/* seconds */
#define MAX_LATENCY	(100 * 1000)	/* usecs */

int test_sleep(unsigned int msec_len)
{
	sleep(msec_len / 1000);
	return 0;
}

int test_select(unsigned int msec_len)
{
	struct timeval tv_sel;

	tv_sel.tv_sec = msec_len / 1000;
	tv_sel.tv_usec = (msec_len % 1000) * 1000;

	return select(0, NULL, NULL, NULL, &tv_sel);
}

int test_usleep(unsigned int msec_len)
{
	usleep(msec_len * 1000);
}

/* This modified from some GNU exsample _not_ to hose y */
int timeval_subtract(struct timeval *result,
		     const struct timeval *x,
		     const struct timeval *y)
{
	struct timeval tmp;

	tmp.tv_sec = y->tv_sec;
	tmp.tv_usec = y->tv_usec;

	/* Perform the carry for the later subtraction */
	if (x->tv_usec < y->tv_usec) {
		int nsec = (y->tv_usec - x->tv_usec) / 1000000 + 1;
		tmp.tv_usec -= 1000000 * nsec;
		tmp.tv_sec += nsec;
	}
	if (x->tv_usec - y->tv_usec > 1000000) {
		int nsec = (x->tv_usec - y->tv_usec) / 1000000;
		tmp.tv_usec += 1000000 * nsec;
		tmp.tv_sec -= nsec;
	}
     
	/* Compute the time remaining to wait.
	   tv_usec is certainly positive. */
	result->tv_sec = x->tv_sec - tmp.tv_sec;
	result->tv_usec = x->tv_usec - tmp.tv_usec;
     
	/* Return 1 if result is negative. */
	return x->tv_sec < tmp.tv_sec;
}

int do_test(char * name, int (* test)(unsigned int len),
	    unsigned int len, int count)
{
	int i, ret;
	struct timeval tv_in;
	struct timeval tv_beg;
	struct timeval tv_end;
	struct timeval tv_len;
	struct timeval tv_lat;
	struct timezone tz;
	char * status = "OK";
	char * latency_type = "";

	tv_in.tv_sec = len / 1000;
	tv_in.tv_usec = (len % 1000) * 1000;

	gettimeofday(&tv_beg, &tz);
	for (i = 0; i < count; i++) {
		ret = test(len);
	}
	gettimeofday(&tv_end, &tz);

	ret = timeval_subtract(&tv_len, &tv_end, &tv_beg);
	if (ret)
		status = "ERROR";

	ret = timeval_subtract(&tv_lat, &tv_len, &tv_in);
	if (ret) {
		latency_type = "-";
		timeval_subtract(&tv_lat, &tv_in, &tv_len);
	}

	if (tv_lat.tv_sec > 0 || tv_lat.tv_usec > MAX_LATENCY)
		status = "ERROR";

	printf("  Test: %6s %4ums time: %2u.%06us "
	       "latency: %1s%u.%06us status: %s\n",
	       name, 
	       (len * count),
	       (unsigned int)tv_len.tv_sec,
	       (unsigned int)tv_len.tv_usec,
	       latency_type,
	       (unsigned int)tv_lat.tv_sec,
	       (unsigned int)tv_lat.tv_usec,
	       status);

	return ret;
}

int main(void)
{
	unsigned int i;
	int max_secs = MAX_SLEEP;

	printf("Testing sub-second select and usleep\n");
	for (i = 0; i < 1000; i += 100) {
		do_test("select", test_select, i, 1);
		do_test("usleep", test_usleep, i, 1);
	}

	printf("Testing multi-second select and sleep\n");
	for (i = 0; i < max_secs; i++) {
		do_test("select", test_select, i * 1000, 1);
		do_test("sleep", test_sleep, i * 1000, 1);
	}

	return 0;
}

Re: Updated dynamic tick patches

[David Weinehall]

On Thu, Sep 01, 2005 at 04:07:22PM +0300, Tony Lindgren wrote:
[snip]
> I tried this quickly on a loaner ThinkPad T30, and needed the following
> patch to compile. The patch does work with PIT, but with lapic the
> system does not wake to timer interrupts :(

That may be a thinkpad issue; I have to boot my Thinkpad with nolapic.

[snip]


Regards: David
-- 
 /) David Weinehall <tao@acc.umu.se> /) Northern lights wander      (\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\)  http://www.acc.umu.se/~tao/    (/   Full colour fire           (/

Re: Updated dynamic tick patches

[Tony Lindgren]

* David Weinehall <tao@acc.umu.se> [050901 16:19]:
> On Thu, Sep 01, 2005 at 04:07:22PM +0300, Tony Lindgren wrote:
> [snip]
> > I tried this quickly on a loaner ThinkPad T30, and needed the following
> > patch to compile. The patch does work with PIT, but with lapic the
> > system does not wake to timer interrupts :(
> 
> That may be a thinkpad issue; I have to boot my Thinkpad with nolapic.

Yeah, that could be. Or it could be the same old P4 does not wake to
APIC interrupt while P3 does bug :)

Tony

Re: Updated dynamic tick patches

[Srivatsa Vaddagiri]

On Thu, Sep 01, 2005 at 04:07:22PM +0300, Tony Lindgren wrote:
> I tried this quickly on a loaner ThinkPad T30, and needed the following
> patch to compile. The patch does work with PIT, but with lapic the
> system does not wake to timer interrupts :(

Even I have found that enabling lapic breaks it on my T30! I think
that is a T30 issue, as I dont see any other reason why it should not 
work (note that I have it tested on some other SMP P4 servers where
it works well).

> I also hacked together a little timer test utility that should go trough
> on a completely idle system with no errors. Also posted it to:
> 
> http://www.muru.com/linux/dyntick/tools/dyntick-test.c
> 
> Srivatsa, could you try the dyntick-test.c on your system after booting
> to init=/bin/sh to make the system as idle as possible?

Thanks for this test. Will test and let you know how it goes on x86.
ATM I am trying to corner the lost-tick-calculation problem with ACPI 
PM timer.

> Unfortunately I cannot debug the APIC issue right now, but I seem to
> have an issue on ARM OMAP where the timer test occasionally fails on
> some longer values, for example 3 second sleep can take 4 seconds.
> 
> I don't know yet if this is the problem George Anzinger mentioned with
> next_timer_interrupt(), or if this is OMAP specific. But it only seems

Will let you know if I see it on x86 too.



-- 


Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Re: Updated dynamic tick patches

[Srivatsa Vaddagiri]

[ Sorry didnt see this mail earlier ]

On Wed, Aug 31, 2005 at 06:53:12PM +0000, Erik Andrén wrote:
> Does these patches compile nicely against 2.6.13?
> Otherwise would it be possible for you to repost patches made against 
> 2.6.13 instead for more public testing?

Con should be posting a consolidated version on his site against 2.6.13-mm1 
pretty soon I hope.

-- 


Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Re: Updated dynamic tick patches

[Srivatsa Vaddagiri]

On Thu, Sep 01, 2005 at 04:07:22PM +0300, Tony Lindgren wrote:
> Srivatsa, could you try the dyntick-test.c on your system after booting
> to init=/bin/sh to make the system as idle as possible?

Tony,
	I get this o/p when I run your test on my SMP system with
2.6.13-mm1 + Con's latest patches (including the most recent
lost tick calculation patch that I posted after that).

Testing sub-second select and usleep
  Test: select    0ms time:  0.000012s latency:  0.000012s status: OK
  Test: usleep    0ms time:  0.000013s latency:  0.000013s status: OK
  Test: select  100ms time:  0.099386s latency: -0.000614s status: OK
  Test: usleep  100ms time:  0.104019s latency:  0.004019s status: OK
  Test: select  200ms time:  0.200013s latency:  0.000013s status: OK
  Test: usleep  200ms time:  0.204016s latency:  0.004016s status: OK
  Test: select  300ms time:  0.300043s latency:  0.000043s status: OK
  Test: usleep  300ms time:  0.304056s latency:  0.004056s status: OK
  Test: select  400ms time:  0.400010s latency:  0.000010s status: OK
  Test: usleep  400ms time:  0.404098s latency:  0.004098s status: OK
  Test: select  500ms time:  0.499992s latency: -0.000008s status: OK
  Test: usleep  500ms time:  0.504000s latency:  0.004000s status: OK
  Test: select  600ms time:  0.600050s latency:  0.000050s status: OK
  Test: usleep  600ms time:  0.603959s latency:  0.003959s status: OK
  Test: select  700ms time:  0.699969s latency: -0.000031s status: OK
  Test: usleep  700ms time:  0.704037s latency:  0.004037s status: OK
  Test: select  800ms time:  0.800026s latency:  0.000026s status: OK
  Test: usleep  800ms time:  0.803978s latency:  0.003978s status: OK
  Test: select  900ms time:  0.900046s latency:  0.000046s status: OK
  Test: usleep  900ms time:  0.904003s latency:  0.004003s status: OK
Testing multi-second select and sleep
  Test: select    0ms time:  0.000005s latency:  0.000005s status: OK
  Test:  sleep    0ms time:  0.000006s latency:  0.000006s status: OK
  Test: select 1000ms time:  1.000062s latency:  0.000062s status: OK
  Test:  sleep 1000ms time:  1.004069s latency:  0.004069s status: OK
  Test: select 2000ms time:  2.000727s latency:  0.000727s status: OK
  Test:  sleep 2000ms time:  2.004141s latency:  0.004141s status: OK
  Test: select 3000ms time:  3.000127s latency:  0.000127s status: OK
  Test:  sleep 3000ms time:  3.004048s latency:  0.004048s status: OK
  Test: select 4000ms time:  4.000032s latency:  0.000032s status: OK
  Test:  sleep 4000ms time:  4.004827s latency:  0.004827s status: OK
  Test: select 5000ms time:  5.000118s latency:  0.000118s status: OK
  Test:  sleep 5000ms time:  5.004131s latency:  0.004131s status: OK
  Test: select 6000ms time:  5.997241s latency: -0.002759s status: OK
  Test:  sleep 6000ms time:  6.008025s latency:  0.008025s status: OK
  Test: select 7000ms time:  6.997195s latency: -0.002805s status: OK
  Test:  sleep 7000ms time:  7.004180s latency:  0.004180s status: OK
  Test: select 8000ms time:  8.000512s latency:  0.000512s status: OK
  Test:  sleep 8000ms time:  8.008116s latency:  0.008116s status: OK
  Test: select 9000ms time:  8.996997s latency: -0.003003s status: OK
  Test:  sleep 9000ms time:  9.004279s latency:  0.004279s status: OK


Don't see any ERROR status. The negative latencies doesn't seem to sound
good. Do you see them too? I ran your test on my RH9 based T30 and
find several negative latencies there too.



-- 


Thanks and Regards,
Srivatsa Vaddagiri,
Linux Technology Center,
IBM Software Labs,
Bangalore, INDIA - 560017

Re: Updated dynamic tick patches

[Tony Lindgren]

On Fri, Sep 02, 2005 at 11:04:32PM +0530, Srivatsa Vaddagiri wrote:
> On Thu, Sep 01, 2005 at 04:07:22PM +0300, Tony Lindgren wrote:
> > Srivatsa, could you try the dyntick-test.c on your system after booting
> > to init=/bin/sh to make the system as idle as possible?
> 
> Tony,
> 	I get this o/p when I run your test on my SMP system with
> 2.6.13-mm1 + Con's latest patches (including the most recent
> lost tick calculation patch that I posted after that).
...
> 
> Don't see any ERROR status. The negative latencies doesn't seem to sound
> good. Do you see them too? I ran your test on my RH9 based T30 and
> find several negative latencies there too.

Good, thanks for testing.

>  Test: select 3000ms time:  3.000127s latency:  0.000127s status: OK

This is when I started seeing errors. Looks like if the next event from
next_timer_interrupt() is longer than HZ and idle HZ is very low, such
as 3 - 4 HZ, something gets confused.

I'll be looking into it more a bit later on, but until the problem
is solved, we should limit MAX_SKIP_TICKS to HZ/2.

Tony