[parisc-linux] sched_clock implementation

[parisc-linux] sched_clock implementation

[Matthew Wilcox]

Anyone want to do better than the lame implementation?  ;-)

----- Forwarded message from Andrew Morton <akpm@osdl.org> -----

I'll be merging Ingo & Con's CPOU scheduler changes into Linus's tree soon.

It does require that the architecture provides a new timing function:

A lame implementation is:

/*
 * Returns nanoseconds
 */

unsigned long long sched_clock(void)
{
	return (unsigned long long)jiffies * (1000000000 / HZ);
}

But for best CPU scheduler results the architecture should try to return a
higher-resolution number than this of course.

sched_clock() has no absolute time requirements: it just has to return some
number which goes up by 1,000,000,000 times per second.

I already have implementations for x86, ppc, sparc64 and ia64.

I have a completely stupid ppc64 implementation which is only accurate on
1GHz CPUs.  Anton please note!

As for the rest, it'll break the build, sorry.

----- End forwarded message -----

-- 
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk

Re: [parisc-linux] sched_clock implementation

[Joel Soete]

Matthew Wilcox wrote:

>Anyone want to do better than the lame implementation?  ;-)
>
>----- Forwarded message from Andrew Morton <akpm@osdl.org> -----
>
>I'll be merging Ingo & Con's CPOU scheduler changes into Linus's tree soon.
>
>It does require that the architecture provides a new timing function:
>
>A lame implementation is:
>
>/*
> * Returns nanoseconds
> */
>  
>
Hi Willy,

(Certainly yet another stupid question but) to reach such accuracy we 
would need to have access to some 'time device' with an accuracy better 
then the nanosec (iirc 10^-9) (because it doesn't seems to me possible 
to get enough accuracy with cpu clock < 10^9 ie 1Ghz: the most case for 
parisc systems). Does it exist such device and where to start to read 
some doc?

Tanks,
    Joel

>unsigned long long sched_clock(void)
>{
>	return (unsigned long long)jiffies * (1000000000 / HZ);
>}
>
>But for best CPU scheduler results the architecture should try to return a
>higher-resolution number than this of course.
>
>sched_clock() has no absolute time requirements: it just has to return some
>number which goes up by 1,000,000,000 times per second.
>
>I already have implementations for x86, ppc, sparc64 and ia64.
>
>I have a completely stupid ppc64 implementation which is only accurate on
>1GHz CPUs.  Anton please note!
>
>As for the rest, it'll break the build, sorry.
>
>----- End forwarded message -----
>
>  
>

Re: [parisc-linux] sched_clock implementation

[Carlos O'Donell]

On Fri, Sep 19, 2003 at 03:32:41PM +0000, Joel Soete wrote:
> Matthew Wilcox wrote:
> 
> >Anyone want to do better than the lame implementation?  ;-)
> >
> >----- Forwarded message from Andrew Morton <akpm@osdl.org> -----
> >
> >I'll be merging Ingo & Con's CPOU scheduler changes into Linus's tree soon.
> >
> >It does require that the architecture provides a new timing function:
> >
> >A lame implementation is:
> >
> >/*
> >* Returns nanoseconds
> >*/
> > 
> >
> Hi Willy,
> 
> (Certainly yet another stupid question but) to reach such accuracy we 
> would need to have access to some 'time device' with an accuracy better 
> then the nanosec (iirc 10^-9) (because it doesn't seems to me possible 
> to get enough accuracy with cpu clock < 10^9 ie 1Ghz: the most case for 
> parisc systems). Does it exist such device and where to start to read 
> some doc?

We could use cr16 to get better accuracy. See list discussions about
fast gettimeofday.

c.

Re: [parisc-linux] sched_clock implementation

[Joel Soete]

Carlos O'Donell wrote:

>On Fri, Sep 19, 2003 at 03:32:41PM +0000, Joel Soete wrote:
>  
>
>>Matthew Wilcox wrote:
>>
>>    
>>
>>>Anyone want to do better than the lame implementation?  ;-)
>>>
>>>----- Forwarded message from Andrew Morton <akpm@osdl.org> -----
>>>
>>>I'll be merging Ingo & Con's CPOU scheduler changes into Linus's tree soon.
>>>
>>>It does require that the architecture provides a new timing function:
>>>
>>>A lame implementation is:
>>>
>>>/*
>>>* Returns nanoseconds
>>>*/
>>>
>>>
>>>      
>>>
>>Hi Willy,
>>
>>(Certainly yet another stupid question but) to reach such accuracy we 
>>would need to have access to some 'time device' with an accuracy better 
>>then the nanosec (iirc 10^-9) (because it doesn't seems to me possible 
>>to get enough accuracy with cpu clock < 10^9 ie 1Ghz: the most case for 
>>parisc systems). Does it exist such device and where to start to read 
>>some doc?
>>    
>>
>
>We could use cr16 to get better accuracy. See list discussions about
>fast gettimeofday.
>  
>
ha ok I will have a look :)

Thanks a lot,
    Joel

Re: [parisc-linux] sched_clock implementation

[Joel Soete]

Carlos O'Donell wrote:

>On Fri, Sep 19, 2003 at 03:32:41PM +0000, Joel Soete wrote:
>  
>
>>Matthew Wilcox wrote:
>>
>>    
>>
>>>Anyone want to do better than the lame implementation?  ;-)
>>>
>>>----- Forwarded message from Andrew Morton <akpm@osdl.org> -----
>>>
>>>I'll be merging Ingo & Con's CPOU scheduler changes into Linus's tree soon.
>>>
>>>It does require that the architecture provides a new timing function:
>>>
>>>A lame implementation is:
>>>
>>>/*
>>>* Returns nanoseconds
>>>*/
>>>
>>>
>>>      
>>>
>>Hi Willy,
>>
>>(Certainly yet another stupid question but) to reach such accuracy we 
>>would need to have access to some 'time device' with an accuracy better 
>>then the nanosec (iirc 10^-9) (because it doesn't seems to me possible 
>>to get enough accuracy with cpu clock < 10^9 ie 1Ghz: the most case for 
>>parisc systems). Does it exist such device and where to start to read 
>>some doc?
>>    
>>
>
>We could use cr16 to get better accuracy. See list discussions about
>fast gettimeofday.
>
>c.
>
Hi all,

a quick look into paxx.pdf which about CR16 speak of "peak instruction 
rate" but do not define anywhere?
I presume that is the cpu clock but would like somebody confirm.

Thanks in advance,
    Joel

Re: [parisc-linux] sched_clock implementation

[Grant Grundler]

On Sat, Sep 20, 2003 at 06:02:10PM +0000, Joel Soete wrote:
> a quick look into paxx.pdf which about CR16 speak of "peak instruction 
> rate" but do not define anywhere?
> I presume that is the cpu clock but would like somebody confirm.

Read about "Interval Timer" (cr16) in the PA 2.0 Arch book.
PDC provides the exact rate that CR16 is changing.
Looks like PDC_TOD_ITIMER is the call but I'm not sure offhand.
In any case, I've only seen it used as CPU cycle counter.
(ie 1:1 with CPU clock).

hth,
grant

Re: [parisc-linux] sched_clock implementation

[Helge Deller]

On Sunday 21 September 2003 02:04, Grant Grundler wrote:
> On Sat, Sep 20, 2003 at 06:02:10PM +0000, Joel Soete wrote:
> > a quick look into paxx.pdf which about CR16 speak of "peak instruction
> > rate" but do not define anywhere?
> > I presume that is the cpu clock but would like somebody confirm.
>
> Read about "Interval Timer" (cr16) in the PA 2.0 Arch book.
> PDC provides the exact rate that CR16 is changing.
> Looks like PDC_TOD_ITIMER is the call but I'm not sure offhand.
> In any case, I've only seen it used as CPU cycle counter.
> (ie 1:1 with CPU clock).

Hi Joel,

I think we have the rate in the variable "boot_cpu_data.cpu_hz" (see 
arch/parisc/kernel/processor.c) already.

Helge

Re: [parisc-linux] sched_clock implementation

[Carlos O'Donell]

On Sun, Sep 21, 2003 at 01:20:36PM +0200, Helge Deller wrote:
> On Sunday 21 September 2003 02:04, Grant Grundler wrote:
> > On Sat, Sep 20, 2003 at 06:02:10PM +0000, Joel Soete wrote:
> > > a quick look into paxx.pdf which about CR16 speak of "peak instruction
> > > rate" but do not define anywhere?
> > > I presume that is the cpu clock but would like somebody confirm.
> >
> > Read about "Interval Timer" (cr16) in the PA 2.0 Arch book.
> > PDC provides the exact rate that CR16 is changing.
> > Looks like PDC_TOD_ITIMER is the call but I'm not sure offhand.
> > In any case, I've only seen it used as CPU cycle counter.
> > (ie 1:1 with CPU clock).
> 
> Hi Joel,
> 
> I think we have the rate in the variable "boot_cpu_data.cpu_hz" (see 
> arch/parisc/kernel/processor.c) already.

Technically that's our highest precision timer :)

c.

Re: [parisc-linux] sched_clock implementation

[Joel Soete]

Grant Grundler wrote:

>On Sat, Sep 20, 2003 at 06:02:10PM +0000, Joel Soete wrote:
>  
>
>>a quick look into paxx.pdf which about CR16 speak of "peak instruction 
>>rate" but do not define anywhere?
>>I presume that is the cpu clock but would like somebody confirm.
>>    
>>
>
>Read about "Interval Timer" (cr16) in the PA 2.0 Arch book.
>PDC provides the exact rate that CR16 is changing.
>Looks like PDC_TOD_ITIMER is the call but I'm not sure offhand.
>In any case, I've only seen it used as CPU cycle counter.
>(ie 1:1 with CPU clock).
>
>hth,
>grant
>_______________________________________________
>parisc-linux mailing list
>parisc-linux@lists.parisc-linux.org
>http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
>
>  
>
Much more for remind, here is a proposed implementation of a pdc_tod_itimer:
diff -Naur linux-2.6.0-test6-pa6.orig/arch/parisc/kernel/firmware.c 
linux-2.6.0-test6-pa6.test/arch/parisc/kernel/firmware.c
--- linux-2.6.0-test6-pa6.orig/arch/parisc/kernel/firmware.c    
2003-10-05 17:14:13.000000000 +0200
+++ linux-2.6.0-test6-pa6.test/arch/parisc/kernel/firmware.c    
2003-10-05 17:09:14.000000000 +0200
@@ -682,6 +682,26 @@
 }
 EXPORT_SYMBOL(pdc_tod_set);

+/**
+ * pdc_tod_Calibrate_timers - Read "Calibrate timers" data
+ * @tod_calib: The return buffer:
+ *
+ * Calibrate the Interval Timer (CR16).
+ */
+int pdc_tod_itimer(struct pdc_tod_calib *tod_calib)
+{
+        int retval;
+
+        spin_lock_irq(&pdc_lock);
+        retval = mem_pdc_call(PDC_TOD, PDC_TOD_ITIMER, 
__pa(pdc_result), 0);
+        convert_to_wide(pdc_result);
+        memcpy(tod_calib, pdc_result, sizeof(tod_calib));
+        spin_unlock_irq(&pdc_lock);
+
+        return retval;
+}
+EXPORT_SYMBOL(pdc_tod_itimer);
+
 #ifdef __LP64__
 int pdc_mem_mem_table(struct pdc_memory_table_raddr *r_addr,
                struct pdc_memory_table *tbl, unsigned long entries)
diff -Naur linux-2.6.0-test6-pa6.orig/include/asm-parisc/pdc.h 
linux-2.6.0-test6-pa6.test/include/asm-parisc/pdc.h
--- linux-2.6.0-test6-pa6.orig/include/asm-parisc/pdc.h 2003-10-05 
17:15:09.000000000 +0200
+++ linux-2.6.0-test6-pa6.test/include/asm-parisc/pdc.h 2003-10-05 
17:11:26.000000000 +0200
@@ -670,6 +670,13 @@
        unsigned long tod_usec;
 };

+struct pdc_tod_calib {
+       unsigned long calib_0;
+       unsigned long calib_1;
+       unsigned long TOD_acc;
+       unsigned long CR_acc;
+};
+
 #ifdef __LP64__
 struct pdc_pat_cell_num {
        unsigned long cell_num;
@@ -943,6 +950,7 @@
 int pdc_get_initiator(struct hardware_path *hwpath, unsigned char 
*scsi_id, unsigned long *period, char *width, char *mode);
 int pdc_tod_read(struct pdc_tod *tod);
 int pdc_tod_set(unsigned long sec, unsigned long usec);
+int pdc_tod_itimer(struct pdc_tod_calib *tod_calib);

 #ifdef __LP64__
 int pdc_mem_mem_table(struct pdc_memory_table_raddr *r_addr,

Well I don't know yet if it would help for a better sched_clock() because:
a) I already found some get_jiffies_64() which seems to be a good candidate?

b) I don't yet find how jiffies and jiffies_64 are continuously updated. 
I presume that we take advantage of an 'external interupt request' by 
CR16 (interval timer) but don't reach to find neither where this one is 
initialised (just to determine if get_jiffies_64 is enough (theoriticaly 
1/100s ?) or request more) nor where it is updated?

Thanks in advance for comments and idea,
    Joel

PS:
I added some test on my c110 and got some:
Oct  5 15:57:51 hpalin kernel: Calib_0 =  0x405e0000
Oct  5 15:57:51 hpalin kernel: Calib_1 =  0x00000000
Oct  5 15:57:51 hpalin kernel: TOD_acc =  0x00000000
Oct  5 15:57:51 hpalin kernel: CR_acc  =  0x00000000

merging Calib_1, Calib_0 in a union retrun me a frequency of 120Mhz 
(exactly) when "Oct  5 15:57:51 hpalin kernel: Calibrating delay loop... 
119.60 BogoMIPS" (but printk don't print float :( )

but it is curious that TOD_acc and CR_acc are 0 any idea?

Re: [parisc-linux] sched_clock implementation

[Carlos O'Donell]

On Sun, Oct 05, 2003 at 03:43:16PM +0000, Joel Soete wrote:
> +        spin_lock_irq(&pdc_lock);
> +        retval = mem_pdc_call(PDC_TOD, PDC_TOD_ITIMER, 
> __pa(pdc_result), 0);
> +        convert_to_wide(pdc_result);

Your mailer is folding lines.

Aside from that the patch looks interesting. It could be used to find
the rate at which cr16 is running, and then we can use that as an HP
timer into userspace. The next portion is solving the monotic
requirement on the clock (e.g. follow the idea of 3 cr16 reads to catch
a reschedule).

c.

Re: [parisc-linux] sched_clock implementation

[Carlos O'Donell]

On Mon, Oct 06, 2003 at 05:22:15PM +0200, Joel Soete wrote:
> >>
> >>On Sun, Oct 05, 2003 at 03:43:16PM +0000, Joel Soete wrote:
> >> +        spin_lock_irq(&pdc_lock);
> >> +        retval = mem_pdc_call(PDC_TOD, PDC_TOD_ITIMER, 
> >> __pa(pdc_result), 0);
> >> +        convert_to_wide(pdc_result);
> >
> >Your mailer is folding lines.
> 
> hmm mozilla? i think much more an error in cup and paste between a xterm
> and mozilla: my bad sorry (i just forgot to attach the file)
> 
> >Aside from that the patch looks interesting.
> 
> Thanks :)
> 
> >It could be used to find
> >the rate at which cr16 is running, and then we can use that as an HP
> >timer into userspace.
> 
> Yes, still have to find out how cr16 is init (mtctl(16,...) i presume but
> which  value?) and also I still have to see how to use data in userspace?

The issue is that cr16 is going to overrun at some point. We need to
track the overrun to track an accurate tick count. If we used another
32-bit quantity to tick on every cr16 overrun then we'd get overrun 
for this value in the range of ~800 years.

Pseudo 64-bit clock:

cr16 tick tick tick ... 2^32, 0, 1, 2 ...
crOV 0 ................ 0,    1, ........

So we add code to the jiffie update to see if cr16 has overrun, if it
has we update the overrun counter. You can then use the current cr16,
the overrun counter, and the bootup cr16 to calculate time accurate to
your CPU's MHz. On my 650MHz box that's somewhere near the 2 ns mark.

We need a way to calculate a difference between two times:

NOTES: No locks needed since I'm not updating anything, just reading.

==== Kernel view ====
Kernel Light weight syscall: get_current_diff(old_cr16, old_overrun,
                                              &delta_cr16, &delta_overrun)
input:
	old_cr16
	old_overrun

- Disable interrupts
- Get CPU #
 - Get Bootup cr16 for cpu (boot_cr16)

 /* Comprise a pseudo-64 bit clock */
 - Get Overrun cr16 for cpu (overrun_cr16)
 - Get Current cr16 for cpu (current_cr16)

 /* Does normalization go away if we can accurately sync all the
    cpu cr16's at bootup? Re: Grant's discussion about loop and
    sync */

 /* Normalize based on boot_cr16 (And drift?) */
 if ( boot_cr16 > current_cr16 ){
    new_overrun = overrun_cr16 - 1;
    new_cr16 = (2<<32) - (boot_cr16 - current_cr16); 
 } else {
    new_overrun = overrun_cr16;
    new_cr16 = current_cr16 - boot_cr16;
 }

 /* overrun offset (~800 year period on 650MHz with 32-bit clock) */
 if( new_overrun > old_overrun ){
   /* First overrun is calculated by -ve delta_cr16, so -1 */
   delta_overrun = new_overrun - old_overrun - 1;
 }

 /* cr16 underflow? (~6s period on 650MHz with 32-bit clock) */
 if( old_cr16 > new_cr16 ){
   /* Adjust for 1 overrun */
   delta_cr16 = (2<<32) - (old_cr16 - new_cr16);
 } else {
   delta_cr16 = new_cr16 - old_cr16;
 }
 /* WARNING: ADJUST FOR TIME TAKEN IN THIS ROUTINE? */

output:
	delta_cr16
	delta_overrun

- Enable interrupts 
====
Kernel Light Weight Syscall: get_current_time(&user_cr16, &user_overrun)

- Disable interrupts
- Get CPU #
 - Get Bootup cr16 for cpu (boot_cr16)

 /* Comprise a pseudo-64 bit clock */
 - Get Overrun cr16 for cpu (overrun_cr16)
 - Get Current cr16 for cpu (current_cr16)

 /* Does normalization go away if we can accurately sync all the
    cpu cr16's at bootup? Re: Grant's discussion about loop and
    sync */

 /* Normalize based on boot_cr16 (And drift?) */
 if ( boot_cr16 > current_cr16 ){
    *user_overrun = overrun_cr16 - 1;
    *user_cr16 = (2<<32) - (boot_cr16 - current_cr16); 
 } else {
    *user_overrun = overrun_cr16;
    *user_cr16 = current_cr16 - boot_cr16;
 }
 /* WARNING: ADJUST FOR TIME TAKEN IN THIS ROUTINE? */

- Enable interrupts
Output:
	user_cr16
	user_overrun
====

==== Userspace View ====

cr16_bitwidth=`/proc/cpuinfo | grep cr16_bitwidth | sed 's/cr16_bitwidth=//g'`
cr16_period=`/proc/cpuinfo | grep cr16_period | sed 's/cr16_period=//g'`

get_current_time(&user_cr16, &user_overrun);
... Code ...
get_current_diff(user_cr16, user_overrun, &delta_cr16, &delta_overrun);

nano_diff = delta_overrun * (2<<cr16_bitwidth) * cr16_period +
            delta_cr16 * cr16_period

====

With a 64-bit kernel and cpu these problem all go away. 

c.