CPU hotplug vs. cpufreq on ppc64

CPU hotplug vs. cpufreq on ppc64

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 3449 bytes --]

I'm having an odd problem with ppc64 cpufreq and cpu hotplug. The cpu
hotplug I'm doing is just fake, but I need to unplug the CPUs for
suspend.

When the system boots up, all CPUs are added as such:
[   18.132830] cpufreq-core: trying to register driver powermac
[   18.132835] cpufreq-core: adding CPU 0
[   18.132976] cpufreq-core: CPU 1 already managed, adding link
[   18.132983] cpufreq-core: CPU 2 already managed, adding link
[   18.132989] cpufreq-core: CPU 3 already managed, adding link
[   18.132998] cpufreq-core: setting new policy for CPU 0: 1250000 - 2500000 kHz
[   18.133025] cpufreq-core: new min and max freqs are 1250000 - 2500000 kHz
[   18.133029] cpufreq-core: governor switch
[   18.133033] cpufreq-core: __cpufreq_governor for CPU 0, event 1
[   18.133043] cpufreq-core: target for CPU 0: 2500000 kHz, relation 1
[   18.133056] cpufreq-core: governor: change or update limits
[   18.133060] cpufreq-core: __cpufreq_governor for CPU 0, event 3
[   18.133069] cpufreq-core: target for CPU 0: 2500000 kHz, relation 1
[   18.133082] cpufreq-core: initialization complete
[   18.133086] cpufreq-core: adding CPU 1
[   18.133089] cpufreq-core: adding CPU 2
[   18.133092] cpufreq-core: adding CPU 3
[   18.133097] cpufreq-core: driver powermac up and running

This says that the 4 CPUs can actually only be switched all together
with CPU0. This is achieved by doing

        policy->cpus = cpu_possible_map;
in
g5_cpufreq_cpu_init (arch/powerpc/platforms/powermac/cpufreq_64.c)


When I suspend now, I get
[  167.563428] cpufreq-core: unregistering CPU 1
[  167.563440] cpufreq-core: removing link
[  167.569139] cpufreq-core: target for CPU 0: 1250000 kHz, relation 1
[  167.573690] cpufreq-core: unregistering CPU 2
[  167.573704] cpufreq-core: removing link
[  167.576175] cpufreq-core: target for CPU 0: 1250000 kHz, relation 1
[  167.582261] cpufreq-core: unregistering CPU 3
[  167.582270] cpufreq-core: removing link
[  168.415156] cpufreq-core: suspending cpu 0
[  168.415164] cpufreq-core: suspending cpu 1
[  168.415170] cpufreq-core: suspending cpu 2
[  168.415177] cpufreq-core: suspending cpu 3

which seems fine. However, when I resume, I get
[  168.446692] cpufreq-core: resuming cpu 0
[  168.446708] cpufreq-core: resuming cpu 1
[  168.446715] cpufreq-core: resuming cpu 2
[  168.446721] cpufreq-core: resuming cpu 3
[  168.624880] cpufreq-core: handle_update for cpu 0 called
[  168.624893] cpufreq-core: updating policy for CPU 0
[  168.624905] cpufreq-core: setting new policy for CPU 0: 1250000 - 2500000 kHz
[  168.624965] cpufreq-core: new min and max freqs are 1250000 - 2500000 kHz
[  168.624974] cpufreq-core: governor: change or update limits
[  168.624982] cpufreq-core: __cpufreq_governor for CPU 0, event 3
[  168.625009] cpufreq-core: target for CPU 0: 1250000 kHz, relation 0
[  169.232726] cpufreq-core: adding CPU 1
[  169.232741] cpufreq-core: initialization failed
[  169.239623] cpufreq-core: adding CPU 2
[  169.239636] cpufreq-core: initialization failed
[  169.247240] cpufreq-core: adding CPU 3
[  169.247255] cpufreq-core: initialization failed

The question now is where this needs to be handled, and how. The driver
can't really say that it initialised fine because it's still initialised
on CPU0. However, I suppose that for real CPU hotplug cpufreq can't try
to remember the pre-unplug groups either...

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: CPU hotplug vs. cpufreq on ppc64

[Benjamin Herrenschmidt]

> which seems fine. However, when I resume, I get
> [  168.446692] cpufreq-core: resuming cpu 0
> [  168.446708] cpufreq-core: resuming cpu 1
> [  168.446715] cpufreq-core: resuming cpu 2
> [  168.446721] cpufreq-core: resuming cpu 3
> [  168.624880] cpufreq-core: handle_update for cpu 0 called
> [  168.624893] cpufreq-core: updating policy for CPU 0
> [  168.624905] cpufreq-core: setting new policy for CPU 0: 1250000 - 2500000 kHz
> [  168.624965] cpufreq-core: new min and max freqs are 1250000 - 2500000 kHz
> [  168.624974] cpufreq-core: governor: change or update limits
> [  168.624982] cpufreq-core: __cpufreq_governor for CPU 0, event 3
> [  168.625009] cpufreq-core: target for CPU 0: 1250000 kHz, relation 0
> [  169.232726] cpufreq-core: adding CPU 1
> [  169.232741] cpufreq-core: initialization failed
> [  169.239623] cpufreq-core: adding CPU 2
> [  169.239636] cpufreq-core: initialization failed
> [  169.247240] cpufreq-core: adding CPU 3
> [  169.247255] cpufreq-core: initialization failed
> 
> The question now is where this needs to be handled, and how. The driver
> can't really say that it initialised fine because it's still initialised
> on CPU0. However, I suppose that for real CPU hotplug cpufreq can't try
> to remember the pre-unplug groups either...

So you mean that the policy->cpus mask is lost on unplug/replug ? Hrm...
I'm not sure what's the best way to restore it. Do we have a cpufreq
backend callback on hotplug to restore things ?

Ben.

Re: CPU hotplug vs. cpufreq on ppc64

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 890 bytes --]

On Mon, 2007-02-05 at 12:39 +1100, Benjamin Herrenschmidt wrote:

> So you mean that the policy->cpus mask is lost on unplug/replug ? Hrm...

Yes, but fixing that wouldn't fix the problem.

> I'm not sure what's the best way to restore it. Do we have a cpufreq
> backend callback on hotplug to restore things ?

Not afaict, but you can of course always use register_hotcpu_notifier.

This, however, is what cpufreq.c uses to be notified of a new CPU that
is coming online, and it then calls cpufreq_add_dev which even with a
correct policy->cpus mask wouldn't go back to adding the links back,
this is only done when CPU#0 comes online.

I think this is a more generic problem and code will need to be added to
cpufreq to
 (a) call into the cpufreq_driver asking it to update the mask on
     hotplug
 (b) after that, check the mask and re-link on hotplug.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: CPU hotplug vs. cpufreq on ppc64

[Jacob Shin]

Johannes Berg wrote:
> I'm having an odd problem with ppc64 cpufreq and cpu hotplug. The cpu
> hotplug I'm doing is just fake, but I need to unplug the CPUs for
> suspend.
> 
> When the system boots up, all CPUs are added as such:
> [   18.132830] cpufreq-core: trying to register driver powermac
> [   18.132835] cpufreq-core: adding CPU 0
> [   18.132976] cpufreq-core: CPU 1 already managed, adding link
> [   18.132983] cpufreq-core: CPU 2 already managed, adding link
> [   18.132989] cpufreq-core: CPU 3 already managed, adding link
> [   18.132998] cpufreq-core: setting new policy for CPU 0: 1250000 - 2500000 kHz
> [   18.133025] cpufreq-core: new min and max freqs are 1250000 - 2500000 kHz
> [   18.133029] cpufreq-core: governor switch
> [   18.133033] cpufreq-core: __cpufreq_governor for CPU 0, event 1
> [   18.133043] cpufreq-core: target for CPU 0: 2500000 kHz, relation 1
> [   18.133056] cpufreq-core: governor: change or update limits
> [   18.133060] cpufreq-core: __cpufreq_governor for CPU 0, event 3
> [   18.133069] cpufreq-core: target for CPU 0: 2500000 kHz, relation 1
> [   18.133082] cpufreq-core: initialization complete
> [   18.133086] cpufreq-core: adding CPU 1
> [   18.133089] cpufreq-core: adding CPU 2
> [   18.133092] cpufreq-core: adding CPU 3
> [   18.133097] cpufreq-core: driver powermac up and running
> 
> This says that the 4 CPUs can actually only be switched all together
> with CPU0. This is achieved by doing
> 
>         policy->cpus = cpu_possible_map;
> in
> g5_cpufreq_cpu_init (arch/powerpc/platforms/powermac/cpufreq_64.c)
> 
> 
> When I suspend now, I get
> [  167.563428] cpufreq-core: unregistering CPU 1
> [  167.563440] cpufreq-core: removing link
> [  167.569139] cpufreq-core: target for CPU 0: 1250000 kHz, relation 1
> [  167.573690] cpufreq-core: unregistering CPU 2
> [  167.573704] cpufreq-core: removing link
> [  167.576175] cpufreq-core: target for CPU 0: 1250000 kHz, relation 1
> [  167.582261] cpufreq-core: unregistering CPU 3
> [  167.582270] cpufreq-core: removing link
> [  168.415156] cpufreq-core: suspending cpu 0
> [  168.415164] cpufreq-core: suspending cpu 1
> [  168.415170] cpufreq-core: suspending cpu 2
> [  168.415177] cpufreq-core: suspending cpu 3
> 
> which seems fine. However, when I resume, I get
> [  168.446692] cpufreq-core: resuming cpu 0
> [  168.446708] cpufreq-core: resuming cpu 1
> [  168.446715] cpufreq-core: resuming cpu 2
> [  168.446721] cpufreq-core: resuming cpu 3
> [  168.624880] cpufreq-core: handle_update for cpu 0 called
> [  168.624893] cpufreq-core: updating policy for CPU 0
> [  168.624905] cpufreq-core: setting new policy for CPU 0: 1250000 - 2500000 kHz
> [  168.624965] cpufreq-core: new min and max freqs are 1250000 - 2500000 kHz
> [  168.624974] cpufreq-core: governor: change or update limits
> [  168.624982] cpufreq-core: __cpufreq_governor for CPU 0, event 3
> [  168.625009] cpufreq-core: target for CPU 0: 1250000 kHz, relation 0
> [  169.232726] cpufreq-core: adding CPU 1
> [  169.232741] cpufreq-core: initialization failed
> [  169.239623] cpufreq-core: adding CPU 2
> [  169.239636] cpufreq-core: initialization failed
> [  169.247240] cpufreq-core: adding CPU 3
> [  169.247255] cpufreq-core: initialization failed
> 
> The question now is where this needs to be handled, and how. The driver
> can't really say that it initialised fine because it's still initialised
> on CPU0. However, I suppose that for real CPU hotplug cpufreq can't try
> to remember the pre-unplug groups either...

Hi,

Consider the scenario where,

CPU0 comes online first, and no other CPUs are online yet.  CPU0 cannot 
advertise to cpufreq what his affected CPUs are because CPU1, CPU2, and 
CPU3 do not exist!  At this point in time, the kernel is only aware of 
CPU0, which is the only CPU that cpufreq manages for the moment.

Now, say CPU1 comes online.  CPU1 now knows that itself and CPU0 are 
tied together in freq scaling.  However, CPU0 is still left in the dark, 
and thinks that he only manages himself.  So CPU1 will register itself, 
and tell cpufreq that its affected cpus are itself and CPU0.  The 
cpufreq driver will see that CPU0 is already managed, update CPU0's 
affected cpus data structure, symlink CPU1 to CPU0, and finally call 
exit on CPU1 to clean up.

Same story for CPU2 and CPU3.

You can easily simulate this scenario using the maxcpus=1 kernel 
parameter, and bring other CPUs online one by one.

Hope this helps,

Jacob Shin
AMD, Inc.

Re: CPU hotplug vs. cpufreq on ppc64

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

Hi,

> CPU0 comes online first, and no other CPUs are online yet.  CPU0 cannot 
> advertise to cpufreq what his affected CPUs are because CPU1, CPU2, and 
> CPU3 do not exist!

Actually, the current powermac cpufreq code does (and can because it
knows that there are only 4 CPUs that you can't actually hotplug
physically.)

> At this point in time, the kernel is only aware of 
> CPU0, which is the only CPU that cpufreq manages for the moment.

Right, the current cpufreq code still sets policy->cpus = 1|2|4|8
though.

> Now, say CPU1 comes online.  CPU1 now knows that itself and CPU0 are 
> tied together in freq scaling.

Aha, so the knowledge should be the other way around.

> However, CPU0 is still left in the dark, 
> and thinks that he only manages himself.  So CPU1 will register itself, 
> and tell cpufreq that its affected cpus are itself and CPU0.  The 
> cpufreq driver will see that CPU0 is already managed, update CPU0's 
> affected cpus data structure, symlink CPU1 to CPU0, and finally call 
> exit on CPU1 to clean up.

Ok.

> Same story for CPU2 and CPU3.

Yeah, I understand now. Thanks for the explanation. I think the fix is
actually trivial too.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]