[PATCH] ARM: do not mark CPU 0 as hotpluggable

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Mike Turquette]

A quick poll of the ARM platforms that implement CPU Hotplug support
shows that every platform treats CPU 0 as a special case that cannot be
hotplugged.  In fact every platform has identical code for
platform_cpu_die which returns -EPERM in the case of CPU 0.

The user-facing sysfs interfaces should reflect this by not populating
an 'online' entry for CPU 0 at all.  This better reflects reality by
making it clear to users that CPU 0 cannot be hotplugged.

This patch prevents CPU 0 from being marked as hotpluggable on all ARM
platforms during CPU registration.  This in turn prevents the creation
of an 'online' sysfs interface for that CPU.

Signed-off-by: Mike Turquette <mturquette@ti.com>
---
 arch/arm/kernel/setup.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
index ed11fb0..a5fc969 100644
--- a/arch/arm/kernel/setup.c
+++ b/arch/arm/kernel/setup.c
@@ -940,7 +940,8 @@ static int __init topology_init(void)
 
 	for_each_possible_cpu(cpu) {
 		struct cpuinfo_arm *cpuinfo = &per_cpu(cpu_data, cpu);
-		cpuinfo->cpu.hotpluggable = 1;
+		if (cpu)
+			cpuinfo->cpu.hotpluggable = 1;
 		register_cpu(&cpuinfo->cpu, cpu);
 	}
 
-- 
1.7.4.1

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Rob Herring]

On 07/20/2011 06:32 PM, Mike Turquette wrote:
> A quick poll of the ARM platforms that implement CPU Hotplug support
> shows that every platform treats CPU 0 as a special case that cannot be
> hotplugged.  In fact every platform has identical code for
> platform_cpu_die which returns -EPERM in the case of CPU 0.
> 
> The user-facing sysfs interfaces should reflect this by not populating
> an 'online' entry for CPU 0 at all.  This better reflects reality by
> making it clear to users that CPU 0 cannot be hotplugged.
> 
> This patch prevents CPU 0 from being marked as hotpluggable on all ARM
> platforms during CPU registration.  This in turn prevents the creation
> of an 'online' sysfs interface for that CPU.
> 
Unless there is a kernel limitation why CPU0 can't be hot unplugged,
then this should remain a platform decision. This may be another case of
everybody just copying other platforms' code, not a platform limitation.

Rob

> Signed-off-by: Mike Turquette <mturquette@ti.com>
> ---
>  arch/arm/kernel/setup.c |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
> index ed11fb0..a5fc969 100644
> --- a/arch/arm/kernel/setup.c
> +++ b/arch/arm/kernel/setup.c
> @@ -940,7 +940,8 @@ static int __init topology_init(void)
>  
>  	for_each_possible_cpu(cpu) {
>  		struct cpuinfo_arm *cpuinfo = &per_cpu(cpu_data, cpu);
> -		cpuinfo->cpu.hotpluggable = 1;
> +		if (cpu)
> +			cpuinfo->cpu.hotpluggable = 1;
>  		register_cpu(&cpuinfo->cpu, cpu);
>  	}
>  

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Santosh Shilimkar]

On 7/21/2011 8:32 AM, Rob Herring wrote:
> On 07/20/2011 06:32 PM, Mike Turquette wrote:
>> A quick poll of the ARM platforms that implement CPU Hotplug support
>> shows that every platform treats CPU 0 as a special case that cannot be
>> hotplugged.  In fact every platform has identical code for
>> platform_cpu_die which returns -EPERM in the case of CPU 0.
>>
>> The user-facing sysfs interfaces should reflect this by not populating
>> an 'online' entry for CPU 0 at all.  This better reflects reality by
>> making it clear to users that CPU 0 cannot be hotplugged.
>>
>> This patch prevents CPU 0 from being marked as hotpluggable on all ARM
>> platforms during CPU registration.  This in turn prevents the creation
>> of an 'online' sysfs interface for that CPU.
>>
> Unless there is a kernel limitation why CPU0 can't be hot unplugged,
> then this should remain a platform decision. This may be another case of
> everybody just copying other platforms' code, not a platform limitation.
>
Just talking on behalf of OMAP, we can't offline CPU0 and limitation
would remain in future OMAPs too.

Regards
Santosh

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Russell King - ARM Linux]

On Wed, Jul 20, 2011 at 04:32:25PM -0700, Mike Turquette wrote:
> A quick poll of the ARM platforms that implement CPU Hotplug support
> shows that every platform treats CPU 0 as a special case that cannot be
> hotplugged.  In fact every platform has identical code for
> platform_cpu_die which returns -EPERM in the case of CPU 0.

Are you sure that's just not because everyone copied what Realview has
been doing (highly likely)?

I suspect that there's no reason that CPU0 can't be taken down, especially
on those platforms which don't take the CPU fully offline but just put it
into a WFI loop.

Those which restart the CPUs through the boot loader probably detect CPU0
as the boot CPU, so they probably can't take CPU0 down.

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Russell King - ARM Linux]

On Thu, Jul 21, 2011 at 11:03:04AM +0530, Santosh Shilimkar wrote:
> Just talking on behalf of OMAP, we can't offline CPU0 and limitation
> would remain in future OMAPs too.

Apart from the broken IRQ migration, and the way CPU0 immediately
reawakes if it is offlined on OMAP4 (even when IRQs are migrated off
CPU0) because omap_read_auxcoreboot0() returns 0, is there any other
reason?

With fixed IRQ migration and forcing CPU0 into an infinite WFI loop,
I can offline CPU0 and still have a running system.

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Woodruff, Richard]

> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
> kernel-bounces at lists.infradead.org] On Behalf Of Russell King - ARM Linux

> On Wed, Jul 20, 2011 at 04:32:25PM -0700, Mike Turquette wrote:
> > A quick poll of the ARM platforms that implement CPU Hotplug support
> > shows that every platform treats CPU 0 as a special case that cannot be
> > hotplugged.  In fact every platform has identical code for
> > platform_cpu_die which returns -EPERM in the case of CPU 0.
> 
> Are you sure that's just not because everyone copied what Realview has
> been doing (highly likely)?
> 
> I suspect that there's no reason that CPU0 can't be taken down, especially
> on those platforms which don't take the CPU fully offline but just put it
> into a WFI loop.
> 
> Those which restart the CPUs through the boot loader probably detect CPU0
> as the boot CPU, so they probably can't take CPU0 down.

This was a hot topic a couple years back for hw folks...

On 443x when you go to ARM-MPCore power states other than normal (dormant, powered-off) there are hardware and ti-trustzone-monitor constraints which dictate the CPU power state transition matrix (cpu1 can't make all calls cpu0 can and some hardware combinations were not validated).

For MPU-domain (mpcore-cluster + l2cache ++) and its sub-domains (cpu0, cpu1, ...) there exists ability to set multiple states which wfi is a gate into (on, inactive, retention, partial-off, off). A big matrix of possible states results. Things were coded and system tested for only a subset of these. The ARM code did line up with these constraints.

Maybe an offline to simple wfi loop is ok, but other lower states could not be entered from that state.

Some folks were talking about relaxing cpu0/cpu1 constraints moving forward.  Doing this could result in many more states-combinations to be validated at hardware level. Do you think Linux code would be simplified/enhanced such that it is worth the extra hardware validation effort?

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Turquette, Mike]

On Thu, Jul 21, 2011 at 12:45 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Jul 20, 2011 at 04:32:25PM -0700, Mike Turquette wrote:
>> A quick poll of the ARM platforms that implement CPU Hotplug support
>> shows that every platform treats CPU 0 as a special case that cannot be
>> hotplugged. ?In fact every platform has identical code for
>> platform_cpu_die which returns -EPERM in the case of CPU 0.
>
> Are you sure that's just not because everyone copied what Realview has
> been doing (highly likely)?

Copy/paste is always likely.  Would be nice for other platform folks
to weigh in on this.

> I suspect that there's no reason that CPU0 can't be taken down, especially
> on those platforms which don't take the CPU fully offline but just put it
> into a WFI loop.
>
> Those which restart the CPUs through the boot loader probably detect CPU0
> as the boot CPU, so they probably can't take CPU0 down.

OMAP does seem to have this limitation and Santosh/Richard can provide
much better details than me in the parallel thread.  Again, would be
nice to hear if other platforms have similar limitations from their
stakeholders.

The idea here is to not mark a CPU hotpluggable if it cannot be
hotplugged.  If the limitations turn out to be legitimate for some
platforms but not others, what's the best way to handle it?  Either
move the topology initialization to platform code or allow platforms
to set some config option.  Patches for the latter are below, but I
think the current discussion on whether or not other platforms can
actually hotplug CPU0 should run its course before considering below
patches too seriously.

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Santosh Shilimkar]

On 7/21/2011 7:00 PM, Russell King - ARM Linux wrote:
> On Thu, Jul 21, 2011 at 11:03:04AM +0530, Santosh Shilimkar wrote:
>> Just talking on behalf of OMAP, we can't offline CPU0 and limitation
>> would remain in future OMAPs too.
>
> Apart from the broken IRQ migration, and the way CPU0 immediately
> reawakes if it is offlined on OMAP4 (even when IRQs are migrated off
> CPU0) because omap_read_auxcoreboot0() returns 0, is there any other
> reason?
>
> With fixed IRQ migration and forcing CPU0 into an infinite WFI loop,
> I can offline CPU0 and still have a running system.
>
The secure software runs only on CPU0 and that's the biggest limitation.
Other issues like hand-shake as you pointed out, power sequencing etc
can be worked around.

Regards
Santosh

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Woodruff, Richard]

> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
> kernel-bounces at lists.infradead.org] On Behalf Of Shilimkar, Santosh

> > With fixed IRQ migration and forcing CPU0 into an infinite WFI loop,
> > I can offline CPU0 and still have a running system.
> >
> The secure software runs only on CPU0 and that's the biggest limitation.
> Other issues like hand-shake as you pointed out, power sequencing etc
> can be worked around.

As you know well some of the secure APIs work on CPU1 and others do not.

I notice in other thread Russell made assumption about CPU1 being able to use all because it could run some. This is not the case.

I'll re-add because of the many power states omap4 adds atop of standard 3 arm states there exists a lot of states and not all state transitions are valid. This will increase complexity of practical usage.

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Santosh Shilimkar]

On 7/22/2011 6:15 PM, Woodruff, Richard wrote:
>
>> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
>> kernel-bounces at lists.infradead.org] On Behalf Of Shilimkar, Santosh
>
>>> With fixed IRQ migration and forcing CPU0 into an infinite WFI loop,
>>> I can offline CPU0 and still have a running system.
>>>
>> The secure software runs only on CPU0 and that's the biggest limitation.
>> Other issues like hand-shake as you pointed out, power sequencing etc
>> can be worked around.
>
> As you know well some of the secure APIs work on CPU1 and others do not.
>
> I notice in other thread Russell made assumption about CPU1 being able to use all because it could run some. This is not the case.
>
I clarified that on the other thread.

Regards
Santosh

[PATCH] ARM: do not mark CPU 0 as hotpluggable

[Turquette, Mike]

On Fri, Jul 22, 2011 at 5:53 AM, Santosh Shilimkar
<santosh.shilimkar@ti.com> wrote:
> On 7/22/2011 6:15 PM, Woodruff, Richard wrote:
>>
>>> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-
>>> kernel-bounces at lists.infradead.org] On Behalf Of Shilimkar, Santosh
>>
>>>> With fixed IRQ migration and forcing CPU0 into an infinite WFI loop,
>>>> I can offline CPU0 and still have a running system.
>>>>
>>> The secure software runs only on CPU0 and that's the biggest limitation.
>>> Other issues like hand-shake as you pointed out, power sequencing etc
>>> can be worked around.
>>
>> As you know well some of the secure APIs work on CPU1 and others do not.
>>
>> I notice in other thread Russell made assumption about CPU1 being able to
>> use all because it could run some. This is not the case.
>>
> I clarified that on the other thread.

I've asked a few other ARM folks (out of band) to weigh in on this
thread to determine if their platform has similar limitations as OMAP.
 Unfortunately no one else has responded.

Still the limitation for OMAP remains.  Earlier in this thread I
provided an alternative to blacklisting CPU0 for all ARM platforms by
instead using a config option, but it received no comments.  What is
the best way to move forward?

Thanks,
Mike

> Regards
> Santosh
>