Re: Continual reading from the PowerPc time base register is not stable

[Csdncannon <csdncannon@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4760 bytes --]

Ben
       Attached is the previous failing one.

Thanks
Gino

2010/3/26 Benjamin Herrenschmidt <benh@kernel.crashing.org>

> On Fri, 2010-03-26 at 09:11 +0800, Csdncannon wrote:
> > After trying the new code with "isync" and unsigned long long
> > convertion, this problem doesn't happen(I tested for several minutes).
> > But the previous block of codes(lacking of isync) is borrowed from
> > kernel. And if this is a bug of kernel?
>
> There's an outstanding question about that. Some processors make mftb
> context synchronizing but it appears that it may not be the case for all
> of them.
>
> Thus indeed, we -might- need some isync's in some places, it's not
> totally clear to me though.
>
> Can you send the code that fails (without the isync) ? The one you sent
> did have them everywhere.
>
> Cheers,
> Ben.
>
> > Thanks
> > Gino
> >
> > 2010/3/26 Benjamin Herrenschmidt <benh@kernel.crashing.org>
> >         On Thu, 2010-03-25 at 23:00 +0800, Csdncannon wrote:
> >         > I am really sorry that the previously attached code is
> >         wrong, this one
> >         > "timebase.c" is the right one, and the "log_timebase" file
> >         is the
> >         > right log.
> >         >
> >         > We are using FreeScale PowerPc 8378, kernel 2.6.28 and
> >         compiled as
> >         > 32-bit.
> >
> >
> >         And despite all those sync/isync you can still observe the
> >         timebase
> >         going backward ? That sounds scary. However, at this stage all
> >         I can
> >         suggest is getting freescale folks to have a look, as this
> >         should really
> >         not happen. Maybe there's some setting with that specific SoC
> >         that is
> >         missing or similar...
> >
> >         Cheers,
> >         Ben.
> >
> >
> >         >
> >         > Thanks
> >         > Gino
> >         >
> >         > 2010/3/25 Arnd Bergmann <arnd@arndb.de>
> >         >         On Thursday 25 March 2010, Benjamin Herrenschmidt
> >         wrote:
> >         >         > On Thu, 2010-03-25 at 10:41 +0800, Csdncannon
> >         wrote:
> >         >         > >          In my program, the value of the 64-bit
> >         time base
> >         >         register is
> >         >         > > read out, and you will find the later value is
> >         even
> >         >         smaller than the
> >         >         > > earlier value from the log “log_timebase”. While
> >         the
> >         >         kernel depends on
> >         >         > > the accuracy of the timebase for the
> >         compensation of the
> >         >         lost PIT
> >         >         > > interrupt, the negative value between two
> >         continual
> >         >         timebase reading
> >         >         > > will bring to the jump of the jiffies. And this
> >         timebase
> >         >         problem will
> >         >         > > bring to the instability of the gettimeofday
> >         system call.
> >         >         > >
> >         >         > >          Do you have any idea about this
> >         problem, thanks
> >         >         for your any
> >         >         > > advice. Attached is the code and log.
> >         >         >
> >         >         > This is a concern, it should definitely not
> >         happen. What
> >         >         machine is
> >         >         > that ? is the code compiled 32-bit or 64-bit ?
> >         What kernel
> >         >         version ?
> >         >         >
> >         >         > Arnd, any chance that could relate to the bug
> >         you've been
> >         >         chasing on
> >         >         > Cell ?
> >         >
> >         >
> >         >         We're still busy with the problem analysis on Cell,
> >         waiting
> >         >         for a time
> >         >         slot to run the next test kernel. So far it seems
> >         like the
> >         >         timebase
> >         >         is actually synchronized at a significant accuracy
> >         on QS22 to
> >         >         never
> >         >         cause this problem with correct code, however it is
> >         possible
> >         >         to
> >         >         observe incorrect timebase values on Cell whenever
> >         the mftb
> >         >         instruction
> >         >         is not serialized with memory accesses, e.g. by
> >         using an isync
> >         >         in front
> >         >         of the mftb. On Power6 and other CPUs, that problem
> >         will not
> >         >         happen.
> >         >
> >         >                Arnd
> >         >
> >
> >
> >
> >
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 6088 bytes --]

[-- Attachment #2: timebase.c --]
[-- Type: application/octet-stream, Size: 1576 bytes --]

/* TSC sync test
 *		by: john stultz (johnstul@us.ibm.com)
 *		(C) Copyright IBM 2003, 2005
 *		Licensed under the GPL
 */


#include <stdio.h>
#include <sys/time.h>
#include <stdlib.h>

#define CALLS_PER_LOOP 64

volatile unsigned long long getTimeBase()
{
	unsigned long upper,lower,upper2;
	do {
		asm volatile("sync":::"memory");
		asm volatile("mftbu %0" : "=r" (upper));
		asm volatile("sync":::"memory");
		asm volatile("mftbl %0" : "=r" (lower));
		asm volatile("sync":::"memory");
		asm volatile("mftbu %0" : "=r" (upper2));
		asm volatile("sync":::"memory");
	}while(upper2!=upper);

	return (upper<<32)|lower;
}

int main(int argc, char *argv[])
{
	volatile unsigned long long list[CALLS_PER_LOOP];
	bool bad[CALLS_PER_LOOP];
	int i, inconsistent;


	/* timestamp start of test */
	system("date");
	while(1){
		inconsistent = 0;

		/* Fill list */
		for(i=0; i < CALLS_PER_LOOP; i++)
			list[i] = getTimeBase();
		
		/* Check for inconsistencies */
		for(i=0; i < CALLS_PER_LOOP-1; i++)
		{
			if(list[i] > list[i+1])
			{
				inconsistent = i+1;
				bad[i] = true;
			}
			else{
				bad[i] = false;
			}
		}

		/* display inconsistency */
		if(inconsistent){
			inconsistent--;
			for(i=0; i < CALLS_PER_LOOP; i++){
				if(bad[i] == true)
					printf("--------------------\n");
				printf("0x%llx\n",list[i]);
				if(bad[i-1] == true && bad[i] == false )
					printf("--------------------\n");
			}
			fflush(0);
			/* timestamp inconsistency*/
			system("date");	
		}

	}
	return 0;
}