Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Andi Kleen]

On Mon, Sep 19, 2005 at 12:42:16PM -0700, john stultz wrote:
> On Mon, 2005-09-19 at 21:31 +0200, Andi Kleen wrote:
> > On Mon, Sep 19, 2005 at 12:16:43PM -0700, john stultz wrote:
> > > 	This patch should resolve the issue seen in bugme bug #5105, where it
> > > is assumed that dualcore x86_64 systems have synced TSCs. This is not
> > > the case, and alternate timesources should be used instead.
> > 
> > 
> > I asked AMD some time ago and they told me it was synchronized.
> > The TSC on K8 is C state invariant, but not P state invariant,
> > but P states always happen synchronized on dual cores.
> > 
> > So I'm not quite convinced of your explanation yet.
> 
> Would a litter userspace test checking the TSC synchronization maybe
> shed additional light on the issue?

Sure you can try it.

-Andi

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[john stultz]

On Mon, 2005-09-19 at 21:49 +0200, Andi Kleen wrote:
> On Mon, Sep 19, 2005 at 12:42:16PM -0700, john stultz wrote:
> > On Mon, 2005-09-19 at 21:31 +0200, Andi Kleen wrote:
> > > On Mon, Sep 19, 2005 at 12:16:43PM -0700, john stultz wrote:
> > > > 	This patch should resolve the issue seen in bugme bug #5105, where it
> > > > is assumed that dualcore x86_64 systems have synced TSCs. This is not
> > > > the case, and alternate timesources should be used instead.
> > > 
> > > 
> > > I asked AMD some time ago and they told me it was synchronized.
> > > The TSC on K8 is C state invariant, but not P state invariant,
> > > but P states always happen synchronized on dual cores.
> > > 
> > > So I'm not quite convinced of your explanation yet.
> > 
> > Would a litter userspace test checking the TSC synchronization maybe
> > shed additional light on the issue?
> 
> Sure you can try it.

So, bugzilla.kernel.org has (temporarily at least) lost the reports from
yesterday, but from the email i got, folks using my TSC consistency
check that I posted were seeing what appears to be unsynched TSCs on
dualcore AMD systems.

Personally I suspect that the powernow driver is putting the cores
independently into low power sleep and the TSCs are being independently
halted, causing them to become unsynchronized.

Do you still feel there is some other issue here? Any ideas for shaking
out whatever else might in play?

thanks
-john

RE: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Langsdorf, Mark]

> On Mon, 2005-09-19 at 21:49 +0200, Andi Kleen wrote:
> > On Mon, Sep 19, 2005 at 12:42:16PM -0700, john stultz wrote:
> > > On Mon, 2005-09-19 at 21:31 +0200, Andi Kleen wrote:
> > > > On Mon, Sep 19, 2005 at 12:16:43PM -0700, john stultz wrote:
> > > > > 	This patch should resolve the issue seen in 
> bugme bug #5105, 
> > > > > where it is assumed that dualcore x86_64 systems have synced 
> > > > > TSCs. This is not the case, and alternate timesources 
> should be 
> > > > > used instead.
> > > > 
> > > > 
> > > > I asked AMD some time ago and they told me it was synchronized. 
> > > > The TSC on K8 is C state invariant, but not P state 
> invariant, but 
> > > > P states always happen synchronized on dual cores.
> > > > 
> > > > So I'm not quite convinced of your explanation yet.
> > > 
> > > Would a litter userspace test checking the TSC 
> synchronization maybe 
> > > shed additional light on the issue?
> > 
> > Sure you can try it.
> 
> So, bugzilla.kernel.org has (temporarily at least) lost the 
> reports from yesterday, but from the email i got, folks using 
> my TSC consistency check that I posted were seeing what 
> appears to be unsynched TSCs on dualcore AMD systems.

My understanding was that each TSC on a dual-core processor
will advance individually and atomically.  They will not 
always be in synchronization.

> Personally I suspect that the powernow driver is putting the 
> cores independently into low power sleep and the TSCs are 
> being independently halted, causing them to become unsynchronized.

The powernow-k8 driver doesn't know what a low power sleep state
is, so I strongly doubt it is involved here.  It only handles
pstates.
 
-Mark Langsdorf
K8 PowerNow! Maintainer
AMD, Inc.

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Scott Lampert]

Langsdorf, Mark wrote:

>>On Mon, 2005-09-19 at 21:49 +0200, Andi Kleen wrote:
>>    
>>
>>>On Mon, Sep 19, 2005 at 12:42:16PM -0700, john stultz wrote:
>>>      
>>>
>>>>On Mon, 2005-09-19 at 21:31 +0200, Andi Kleen wrote:
>>>>        
>>>>
>>>>>On Mon, Sep 19, 2005 at 12:16:43PM -0700, john stultz wrote:
>>>>>          
>>>>>
>>>>>>	This patch should resolve the issue seen in 
>>>>>>            
>>>>>>
>>bugme bug #5105, 
>>    
>>
>>>>>>where it is assumed that dualcore x86_64 systems have synced 
>>>>>>TSCs. This is not the case, and alternate timesources 
>>>>>>            
>>>>>>
>>should be 
>>    
>>
>>>>>>used instead.
>>>>>>            
>>>>>>
>>>>>I asked AMD some time ago and they told me it was synchronized. 
>>>>>The TSC on K8 is C state invariant, but not P state 
>>>>>          
>>>>>
>>invariant, but 
>>    
>>
>>>>>P states always happen synchronized on dual cores.
>>>>>
>>>>>So I'm not quite convinced of your explanation yet.
>>>>>          
>>>>>
>>>>Would a litter userspace test checking the TSC 
>>>>        
>>>>
>>synchronization maybe 
>>    
>>
>>>>shed additional light on the issue?
>>>>        
>>>>
>>>Sure you can try it.
>>>      
>>>
>>So, bugzilla.kernel.org has (temporarily at least) lost the 
>>reports from yesterday, but from the email i got, folks using 
>>my TSC consistency check that I posted were seeing what 
>>appears to be unsynched TSCs on dualcore AMD systems.
>>    
>>
>
>My understanding was that each TSC on a dual-core processor
>will advance individually and atomically.  They will not 
>always be in synchronization.
>
>  
>
>>Personally I suspect that the powernow driver is putting the 
>>cores independently into low power sleep and the TSCs are 
>>being independently halted, causing them to become unsynchronized.
>>    
>>
>
>The powernow-k8 driver doesn't know what a low power sleep state
>is, so I strongly doubt it is involved here.  It only handles
>pstates.
> 
>-Mark Langsdorf
>K8 PowerNow! Maintainer
>AMD, Inc.
>
>  
>

Just to add some end-user input here, I see the same issues regardless 
of whether I'm running with the powernow-k8 or not.  The clock problems 
seem to be unrelated to that, at least on my system.
    -Scott

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[john stultz]

On Tue, 2005-09-20 at 12:24 -0700, Scott Lampert wrote:
> Langsdorf, Mark wrote:
> >>Personally I suspect that the powernow driver is putting the 
> >>cores independently into low power sleep and the TSCs are 
> >>being independently halted, causing them to become unsynchronized.
> >
> >The powernow-k8 driver doesn't know what a low power sleep state
> >is, so I strongly doubt it is involved here.  It only handles
> >pstates.
> > 
> Just to add some end-user input here, I see the same issues regardless 
> of whether I'm running with the powernow-k8 or not.  The clock problems 
> seem to be unrelated to that, at least on my system.

Hmmm. Ok, I don't know the cpufreq/power management code well enough. 

I know some Intel cpus halt the TSC in C3. Could the ACPI code be
causing this? 

Could anyone with better knowledge speak to why it looks like the TSCs
are unsynced? Is my test flawed?

thanks
-john

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Daniel Jacobowitz]

On Tue, Sep 20, 2005 at 11:59:45AM -0700, john stultz wrote:
> So, bugzilla.kernel.org has (temporarily at least) lost the reports from
> yesterday, but from the email i got, folks using my TSC consistency
> check that I posted were seeing what appears to be unsynched TSCs on
> dualcore AMD systems.
> 
> Personally I suspect that the powernow driver is putting the cores
> independently into low power sleep and the TSCs are being independently
> halted, causing them to become unsynchronized.
> 
> Do you still feel there is some other issue here? Any ideas for shaking
> out whatever else might in play?

FYI, at least I have reproduced this without powernow loaded.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Andi Kleen]

On Wed, Sep 21, 2005 at 10:15:08AM -0500, Ray Bryant wrote:
> On Tuesday 20 September 2005 23:03, Daniel Jacobowitz wrote:
> 
> >
> > FYI, at least I have reproduced this without powernow loaded.
> 
> There are cases that we are aware of where the TSC will count slower while the 
> processor is halted.    This can make TSC's get out of sync on dual cores.

Ok thanks for the confirmation. I guess John's patch is ok then.
Drawback is much slower to extremly slow gettimeofday  (depending
if the chipset/BIOS has usable HPET, most seem not to) 

> 
> I wonder if you can reproduce this problem while also running a pair of cpu 
> bound tasks on your dual core box.   If you can't, then this is the culprit.
> 
> In general, however, on multisocket systems, you can't depend on TSC's being 
> synchronized between sockets, so all of this is moot.   We just have to deal 
> with it. 

We handle this, but single socket dual core was special cased because
I was told previously it should be ok.

-Andi

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Ray Bryant]

On Tuesday 20 September 2005 23:03, Daniel Jacobowitz wrote:

>
> FYI, at least I have reproduced this without powernow loaded.

There are cases that we are aware of where the TSC will count slower while the 
processor is halted.    This can make TSC's get out of sync on dual cores.

I wonder if you can reproduce this problem while also running a pair of cpu 
bound tasks on your dual core box.   If you can't, then this is the culprit.

In general, however, on multisocket systems, you can't depend on TSC's being 
synchronized between sockets, so all of this is moot.   We just have to deal 
with it. 

-- 
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Ray Bryant]

On Wednesday 21 September 2005 10:04, Andi Kleen wrote:

>
> We handle this, but single socket dual core was special cased because
> I was told previously it should be ok.
>
> -Andi

AFAIK there is a processor state bit that enables/disables this behavior.
Apparently some BIOS's are setting this one way for desktop systems and the 
other way for servers.   If it is thought to be important I can track that 
down and see if it can be externally documented.  (It may actually be in the 
bios and kernel developer guide...)

-- 
Ray Bryant
AMD Performance Labs                   Austin, Tx
512-602-0038 (o)                 512-507-7807 (c)

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Andrew Morton]

Andi Kleen <ak@suse.de> wrote:
>
> On Wed, Sep 21, 2005 at 10:15:08AM -0500, Ray Bryant wrote:
> > On Tuesday 20 September 2005 23:03, Daniel Jacobowitz wrote:
> > 
> > >
> > > FYI, at least I have reproduced this without powernow loaded.
> > 
> > There are cases that we are aware of where the TSC will count slower while the 
> > processor is halted.    This can make TSC's get out of sync on dual cores.

You mean a single `hlt' instruction?   I guess that rules out resyncing them.

> Ok thanks for the confirmation. I guess John's patch is ok then.
> Drawback is much slower to extremly slow gettimeofday  (depending
> if the chipset/BIOS has usable HPET, most seem not to) 

That's a really big drawback.   Will this affect many CPU types?

If the user was prepared to use `idle=poll' then they could get their fast
gettimeofday() back, perhaps.

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Jonas Oreland]

Ray Bryant wrote:
> On Wednesday 21 September 2005 10:04, Andi Kleen wrote:
> 
> 
>>We handle this, but single socket dual core was special cased because
>>I was told previously it should be ok.
>>
>>-Andi
> 
> 
> AFAIK there is a processor state bit that enables/disables this behavior.
> Apparently some BIOS's are setting this one way for desktop systems and the 
> other way for servers.   If it is thought to be important I can track that 
> down and see if it can be externally documented.  (It may actually be in the 
> bios and kernel developer guide...)
> 

Hi,

This would be very good (for us single socket dual core users)
I tried a very small benchmark:

clock_gettime(CLOCK_REALTIME): elapsed 7336657 -> 733.665700ns/call
clock_gettime(CLOCK_PROCESS_CPUTIME_ID): elapsed 763247 -> 76.324700ns/call

It's a factor 10 faster if the TSC were to be in sync.

/Jonas

Re: [discuss] Re: [PATCH] x86-64: Fix bad assumption that dualcore cpus have synced TSCs

[Velu Erwan]

Vladimir B. Savkin a écrit :

>Well, I think not.
>Asus file download page is unavailable since yesterday.
>  
>
Agreed but ftp://ftp.asus.com:/pub/ASUS/mb/socket939/a8v-deluxe is still 
available ;)