cpufreq & dual core CPUs.

cpufreq & dual core CPUs.

[Dave Jones]

After being prompted by akpm to look into this (he noted his builds
were taking twice as long when cpuspeed was active on his core duo laptop),
I did some tests on a similar spec laptop myself, and it seems that
we have a number of shortcomings in cpufreq right now when presented
with a multi-core CPU.

This CPU has two cores in a single package, yet cpufreq doesn't
realise this, and presents them as two separately independant scalable
CPUs, as if they were regular SMP.

The problems this causes are twofold.

- We get into situations where strange things happen like
  CPU0 being at 1GHz whilst CPU1 is at 2GHz, and a foodfight
  ensues as both cores continually readjust themselves to
  match what the other is doing.
  (This is actually quite amusing to watch if you configure two
   of the gnome cpufreq applets)
  This seems to affect both userspace daemons like cpuspeed,
  and also (to a lesser extent) the ondemand/conservative governors.

  By adding dual-core awareness to the drivers, we can make sure
  that all cores in a single package stay in sync.

- We allow the possibility for different governors on each core.
  Ie, you can set core0 to ondemand and core1 to userspace.
  I'm not convinced this is a good idea even on SMP, but on
  multi-core it for sure makes no sense.

  One way to fix this would be to ensure that updating the
  governor on one core automatically updates the governor
  on all other cores.  It does mean that the CPU agnostic drivers/cpufreq/
  core will have to start caring about architecture specific things
  like the core maps however.

Before looking any further into why Andrews performance suffered
so much (it aparently never shifted up from 1GHz unless he ran
two intensive tasks, keeping both cores busy), I think getting
the above in order makes sense.

Comments?

		Dave

-- 
http://www.codemonkey.org.uk

RE: cpufreq & dual core CPUs.

[Pallipadi, Venkatesh]

I am assuming speedstep-centrino is being used here. This one core
reducing the frequency of the other core is not supposed to happen. The
way it is handled in Core Duo or later CPUs is:
1) We can either use software coordination when BIOS and OS supports it.
In which case, OS will know that these two CPUs are tied together and
controls both of them using one interface. However, the users of this
one common interface, cpuspeed or ondemand governor should be aware of
this fact that one interface controls two CPUs (by looking at the
affected_cpus in cpufreq directory). Ondemand handles this today. But,
cpuspeed. I am not sure.

2) More easier way is to do this coordination in hardware. Each Cpu will
write to their own MSR. Hardware will pick the maximum frequency of the
two MSR and controls the package at that frequency. This is the default
mode on most of Core Duo based platforms and we have found that this is
more effient in the sense that each CPU can indepently control the
frequency, with no set_cpu_allowed() and friends in the path.

Basically, with hardware coordination, you should not see this
particular issue. I feel, there is something broken in this particular
case and I will look into it closer tomorrow. My guess at this point is,
this is due to userlevel governor, which looks at utilization over a
period of time (few seconds) and then decides on the frequency. This can
be checked with ondemand governor, which should handle this case much
better.

Thanks,
Venki 

>-----Original Message-----
>From: Dave Jones [mailto:davej@redhat.com] 
>Sent: Tuesday, May 30, 2006 9:27 PM
>To: cpufreq@www.linux.org.uk
>Cc: Pallipadi, Venkatesh
>Subject: cpufreq & dual core CPUs.
>
>After being prompted by akpm to look into this (he noted his builds
>were taking twice as long when cpuspeed was active on his core 
>duo laptop),
>I did some tests on a similar spec laptop myself, and it seems that
>we have a number of shortcomings in cpufreq right now when presented
>with a multi-core CPU.
>
>This CPU has two cores in a single package, yet cpufreq doesn't
>realise this, and presents them as two separately independant scalable
>CPUs, as if they were regular SMP.
>
>The problems this causes are twofold.
>
>- We get into situations where strange things happen like
>  CPU0 being at 1GHz whilst CPU1 is at 2GHz, and a foodfight
>  ensues as both cores continually readjust themselves to
>  match what the other is doing.
>  (This is actually quite amusing to watch if you configure two
>   of the gnome cpufreq applets)
>  This seems to affect both userspace daemons like cpuspeed,
>  and also (to a lesser extent) the ondemand/conservative governors.
>
>  By adding dual-core awareness to the drivers, we can make sure
>  that all cores in a single package stay in sync.
>
>- We allow the possibility for different governors on each core.
>  Ie, you can set core0 to ondemand and core1 to userspace.
>  I'm not convinced this is a good idea even on SMP, but on
>  multi-core it for sure makes no sense.
>
>  One way to fix this would be to ensure that updating the
>  governor on one core automatically updates the governor
>  on all other cores.  It does mean that the CPU agnostic 
>drivers/cpufreq/
>  core will have to start caring about architecture specific things
>  like the core maps however.
>
>Before looking any further into why Andrews performance suffered
>so much (it aparently never shifted up from 1GHz unless he ran
>two intensive tasks, keeping both cores busy), I think getting
>the above in order makes sense.
>
>Comments?
>
>		Dave
>
>-- 
>http://www.codemonkey.org.uk
>

Re: cpufreq & dual core CPUs.

[Dave Jones]

On Tue, May 30, 2006 at 09:40:37PM -0700, Pallipadi, Venkatesh wrote:
 
 > I am assuming speedstep-centrino is being used here.

yes, sorry for omitting that, though I'm not sure this problem
is unique to Intel CPUs, though powernow-k8 does seem to have
some awareness of the cpu_core_map

 > This one core
 > reducing the frequency of the other core is not supposed to happen. The
 > way it is handled in Core Duo or later CPUs is:
 > 1) We can either use software coordination when BIOS and OS supports it.
 > In which case, OS will know that these two CPUs are tied together and
 > controls both of them using one interface. However, the users of this
 > one common interface, cpuspeed or ondemand governor should be aware of
 > this fact that one interface controls two CPUs (by looking at the
 > affected_cpus in cpufreq directory). Ondemand handles this today. But,
 > cpuspeed. I am not sure.

Looking at that file on my core duo laptop, I see ..

(00:47:47:davej@exile:~)$ cat /sys/devices/system/cpu/cpu0/cpufreq/affected_cpus
0
(00:47:54:davej@exile:~)$ cat /sys/devices/system/cpu/cpu1/cpufreq/affected_cpus
1

That implies to me that they're separate cores no?

 > 2) More easier way is to do this coordination in hardware. Each Cpu will
 > write to their own MSR. Hardware will pick the maximum frequency of the
 > two MSR and controls the package at that frequency. This is the default
 > mode on most of Core Duo based platforms and we have found that this is
 > more effient in the sense that each CPU can indepently control the
 > frequency, with no set_cpu_allowed() and friends in the path.
 > 
 > Basically, with hardware coordination, you should not see this
 > particular issue. I feel, there is something broken in this particular
 > case and I will look into it closer tomorrow. My guess at this point is,
 > this is due to userlevel governor, which looks at utilization over a
 > period of time (few seconds) and then decides on the frequency. This can
 > be checked with ondemand governor, which should handle this case much
 > better.

ondemand does behave better than userspace does, but something is
still amiss given that they can both still be set to individual
values.
This..

(00:52:15:davej@exile:~)$ grep MHz /proc/cpuinfo
cpu MHz         : 1000.000
cpu MHz         : 2167.000

Should never happen on a dual core system that has synchronised cores.
Either they're both running at 1GHz, or they're both running at 2.1GHz.

(That was with both cpu0 & cpu1 set to use ondemand governor).

		Dave

-- 
http://www.codemonkey.org.uk

Re: cpufreq & dual core CPUs.

[Jacob Shin]

Dave Jones wrote:

> After being prompted by akpm to look into this (he noted his builds
> were taking twice as long when cpuspeed was active on his core duo laptop),
> I did some tests on a similar spec laptop myself, and it seems that
> we have a number of shortcomings in cpufreq right now when presented
> with a multi-core CPU.
> 
> This CPU has two cores in a single package, yet cpufreq doesn't
> realise this, and presents them as two separately independant scalable
> CPUs, as if they were regular SMP.
> 
> The problems this causes are twofold.
> 
> - We get into situations where strange things happen like
>   CPU0 being at 1GHz whilst CPU1 is at 2GHz, and a foodfight
>   ensues as both cores continually readjust themselves to
>   match what the other is doing.
>   (This is actually quite amusing to watch if you configure two
>    of the gnome cpufreq applets)
>   This seems to affect both userspace daemons like cpuspeed,
>   and also (to a lesser extent) the ondemand/conservative governors.
> 
>   By adding dual-core awareness to the drivers, we can make sure
>   that all cores in a single package stay in sync.
> 
> - We allow the possibility for different governors on each core.
>   Ie, you can set core0 to ondemand and core1 to userspace.
>   I'm not convinced this is a good idea even on SMP, but on
>   multi-core it for sure makes no sense.
> 
>   One way to fix this would be to ensure that updating the
>   governor on one core automatically updates the governor
>   on all other cores.  It does mean that the CPU agnostic drivers/cpufreq/
>   core will have to start caring about architecture specific things
>   like the core maps however.
> 
> Before looking any further into why Andrews performance suffered
> so much (it aparently never shifted up from 1GHz unless he ran
> two intensive tasks, keeping both cores busy), I think getting
> the above in order makes sense.
> 
> Comments?

I believe the "affected CPUs" -- cpufreq_policy->cpus -- takes care of
this already.  When ->cpus is initialized with both of the cores set,
cpufreq driver will only manage 1 shared data structure and sysfs for
all of the cores.

The powernow-k8 driver will initialize ->cpus with cpu_core_map[],
since on current AMD hardware, Dual-Cores are tied together in freq
and voltage.  Carrying out P-State transition on one, will have affect
on the other core as well.

As far as the governors are concerned, they simply need to be aware
that changing freq on one of the affected CPUs will mirror on all
other CPUs in the ->cpus mask.

-Jacob Shin
AMD, Inc.

RE: cpufreq & dual core CPUs.

[Pallipadi, Venkatesh]

 

>-----Original Message-----
>From: Dave Jones [mailto:davej@redhat.com] 
>Sent: Tuesday, May 30, 2006 9:54 PM
>To: Pallipadi, Venkatesh
>Cc: cpufreq@www.linux.org.uk; Andrew Morton
>Subject: Re: cpufreq & dual core CPUs.
>
>
>Looking at that file on my core duo laptop, I see ..
>
>(00:47:47:davej@exile:~)$ cat 
>/sys/devices/system/cpu/cpu0/cpufreq/affected_cpus
>0
>(00:47:54:davej@exile:~)$ cat 
>/sys/devices/system/cpu/cpu1/cpufreq/affected_cpus
>1
>
>That implies to me that they're separate cores no?

This implies software coordination is not happening here.
But, that is OK, as hardware or BIOS coordination can be happening.

> > 2) More easier way is to do this coordination in hardware. 
>Each Cpu will
> > write to their own MSR. Hardware will pick the maximum 
>frequency of the
> > two MSR and controls the package at that frequency. This is 
>the default
> > mode on most of Core Duo based platforms and we have found 
>that this is
> > more effient in the sense that each CPU can indepently control the
> > frequency, with no set_cpu_allowed() and friends in the path.
> > 
> > Basically, with hardware coordination, you should not see this
> > particular issue. I feel, there is something broken in this 
>particular
> > case and I will look into it closer tomorrow. My guess at 
>this point is,
> > this is due to userlevel governor, which looks at utilization over a
> > period of time (few seconds) and then decides on the 
>frequency. This can
> > be checked with ondemand governor, which should handle this 
>case much
> > better.
>
>ondemand does behave better than userspace does, but something is
>still amiss given that they can both still be set to individual
>values.
>This..
>
>(00:52:15:davej@exile:~)$ grep MHz /proc/cpuinfo
>cpu MHz         : 1000.000
>cpu MHz         : 2167.000
>

Yes. Actually, both the cores will be running at 2GHz at this particular
instance.
It is just that OS does not know about this fact, as OS request to lower
frequency
has been recorded in hardware (through MSR) on CPU 0 and this change
will come 
into affect asynchronously in future, once CPU 1 also requests for lower
frequency.

>Should never happen on a dual core system that has synchronised cores.
>Either they're both running at 1GHz, or they're both running at 2.1GHz.
>
>(That was with both cpu0 & cpu1 set to use ondemand governor).
>

The problem is that currently there is no way to get the actual current
frequency 
and we depend on last requested frequency that went through without any
error.
Hence CPU 0 shows lower frequency even when we are actually at a higher
frequency.

Thanks,
Venki

Re: cpufreq & dual core CPUs.

[Jacob Shin]

Dave Jones wrote:

> On Tue, May 30, 2006 at 09:40:37PM -0700, Pallipadi, Venkatesh wrote:

> yes, sorry for omitting that, though I'm not sure this problem
> is unique to Intel CPUs, though powernow-k8 does seem to have
> some awareness of the cpu_core_map

Yes, powernow-k8 will populate cpufreq_policy->cpus with cpu_core_map[] in order
to communicate to cpufreq driver that the two cores are tied together in freq,
and voltage.

> Looking at that file on my core duo laptop, I see ..
> 
> (00:47:47:davej@exile:~)$ cat /sys/devices/system/cpu/cpu0/cpufreq/affected_cpus
> 0
> (00:47:54:davej@exile:~)$ cat /sys/devices/system/cpu/cpu1/cpufreq/affected_cpus
> 1
> 
> That implies to me that they're separate cores no?

Are both cores on core-duos tied together?  Then why not populate ->cpus with both
of the cores set?

-Jacob Shin
AMD, Inc.

Re: cpufreq & dual core CPUs.

[Dominik Brodowski]

Hi,

> >That implies to me that they're separate cores no?
> 
> This implies software coordination is not happening here.
> But, that is OK, as hardware or BIOS coordination can be happening.

However, I still fail to see why hardware/BIOS coordination is in any way
better to software coordination and why we don't use software coordination
for speedstep-centrino. Especially as it can so easily lead to confusion.

	Dominik

RE: cpufreq & dual core CPUs.

[Pallipadi, Venkatesh]

 

>-----Original Message-----
>From: Dominik Brodowski [mailto:linux@dominikbrodowski.net] 
>Sent: Tuesday, May 30, 2006 10:34 PM
>To: Pallipadi, Venkatesh
>Cc: Dave Jones; Andrew Morton; cpufreq@www.linux.org.uk
>Subject: Re: cpufreq & dual core CPUs.
>
>Hi,
>
>> >That implies to me that they're separate cores no?
>> 
>> This implies software coordination is not happening here.
>> But, that is OK, as hardware or BIOS coordination can be happening.
>
>However, I still fail to see why hardware/BIOS coordination is 
>in any way
>better to software coordination and why we don't use software 
>coordination
>for speedstep-centrino. Especially as it can so easily lead to 
>confusion.
>

I don't have the numbers comparing the two. But, this is how things go
with hardware and software coordination at ondemand and driver level
(Assuming single socket dual core CPU).

Software coordination
- There is a single instance of sampling happening in ondemand.
- That instance checks utilization of all CPUs (using kstat) and decides
whether to increase or decrease the frequency.
- Then it makes a call to the driver to change the frequency.
- Now driver has to do MSR write on both the cores to ensure the
frequency change.
- It uses set_cpus_allowed to go to first CPU and writes MSR
- It uses set_cpus_allowed to go to second CPU and writes MSR
- Note that set_cpus_allowed in turn kicks up migration thread and then
sets the affinity.

Hardware Coordination
- There can be two instances of sampling running totally independently
using percpu structures.
- Each of them will decide on a frequency based on utilization (kstat
and percpu ondemand structures).
- Then makes the call to the driver to write MSR. 
- With each instance of sampling happening on correct CPU, there will
not be any need to use set_cpus_allowed at the driver.
- It just sets the MSR on current CPU and returns.

So, avoiding set_cpus_allowed will make hardware coorindation better
than software coordination. This is assuming that ondemand sampling for
each CPU runs independently. Today it does not. But, I have patchsets
that I will be sending out soon that does this.

Thanks,
Venki

Re: cpufreq & dual core CPUs.

[Carl Thompson]

I wasn't aware of the "affected_cpus" thing; I will have a CPUSpeed
update out within the next week to fix this.

Thanks for the heads up,
Carl Thompson

PS: For various reasons, I believe that userspace programs like my
CPUSpeed can do a better job than the ondemand governor in situations
where maximizing battery life is the primary goal.


Pallipadi, Venkatesh wrote:
> I am assuming speedstep-centrino is being used here. This one core
> reducing the frequency of the other core is not supposed to happen. The
> way it is handled in Core Duo or later CPUs is:
> 1) We can either use software coordination when BIOS and OS supports it.
> In which case, OS will know that these two CPUs are tied together and
> controls both of them using one interface. However, the users of this
> one common interface, cpuspeed or ondemand governor should be aware of
> this fact that one interface controls two CPUs (by looking at the
> affected_cpus in cpufreq directory). Ondemand handles this today. But,
> cpuspeed. I am not sure.
>
> 2) More easier way is to do this coordination in hardware. Each Cpu will
> write to their own MSR. Hardware will pick the maximum frequency of the
> two MSR and controls the package at that frequency. This is the default
> mode on most of Core Duo based platforms and we have found that this is
> more effient in the sense that each CPU can indepently control the
> frequency, with no set_cpu_allowed() and friends in the path.
>
> Basically, with hardware coordination, you should not see this
> particular issue. I feel, there is something broken in this particular
> case and I will look into it closer tomorrow. My guess at this point is,
> this is due to userlevel governor, which looks at utilization over a
> period of time (few seconds) and then decides on the frequency. This can
> be checked with ondemand governor, which should handle this case much
> better.
>
> Thanks,
> Venki 
>
>   
>> -----Original Message-----
>> From: Dave Jones [mailto:davej@redhat.com] 
>> Sent: Tuesday, May 30, 2006 9:27 PM
>> To: cpufreq@www.linux.org.uk
>> Cc: Pallipadi, Venkatesh
>> Subject: cpufreq & dual core CPUs.
>>
>> After being prompted by akpm to look into this (he noted his builds
>> were taking twice as long when cpuspeed was active on his core 
>> duo laptop),
>> I did some tests on a similar spec laptop myself, and it seems that
>> we have a number of shortcomings in cpufreq right now when presented
>> with a multi-core CPU.
>>
>> This CPU has two cores in a single package, yet cpufreq doesn't
>> realise this, and presents them as two separately independant scalable
>> CPUs, as if they were regular SMP.
>>
>> The problems this causes are twofold.
>>
>> - We get into situations where strange things happen like
>>  CPU0 being at 1GHz whilst CPU1 is at 2GHz, and a foodfight
>>  ensues as both cores continually readjust themselves to
>>  match what the other is doing.
>>  (This is actually quite amusing to watch if you configure two
>>   of the gnome cpufreq applets)
>>  This seems to affect both userspace daemons like cpuspeed,
>>  and also (to a lesser extent) the ondemand/conservative governors.
>>
>>  By adding dual-core awareness to the drivers, we can make sure
>>  that all cores in a single package stay in sync.
>>
>> - We allow the possibility for different governors on each core.
>>  Ie, you can set core0 to ondemand and core1 to userspace.
>>  I'm not convinced this is a good idea even on SMP, but on
>>  multi-core it for sure makes no sense.
>>
>>  One way to fix this would be to ensure that updating the
>>  governor on one core automatically updates the governor
>>  on all other cores.  It does mean that the CPU agnostic 
>> drivers/cpufreq/
>>  core will have to start caring about architecture specific things
>>  like the core maps however.
>>
>> Before looking any further into why Andrews performance suffered
>> so much (it aparently never shifted up from 1GHz unless he ran
>> two intensive tasks, keeping both cores busy), I think getting
>> the above in order makes sense.
>>
>> Comments?
>>
>> 		Dave
>>
>> -- 
>> http://www.codemonkey.org.uk
>>
>>     
>
> _______________________________________________
> Cpufreq mailing list
> Cpufreq@lists.linux.org.uk
> http://lists.linux.org.uk/mailman/listinfo/cpufreq
>
>   

Re: cpufreq & dual core CPUs.

[Dominik Brodowski]

Hi,

ISTR that your major point against software cooridnation of P-States is the
cost associated with set_cpus_allowed() -- couldn't we get rid of this if
we'd start a per-CPU thread in speedstep-centrino which takes care of
switching frequencies? threads are cheap, AFAIK...

Thanks,
	Dominik