[PATCH] Help fix SMP boot issues with L2 cache

[PATCH] Help fix SMP boot issues with L2 cache

[Russell King - ARM Linux]

    ARM: Add L2 cache handling to smp boot support
    
    The page table and secondary data which we're asking the secondary CPU
    to make use of has to hit RAM to ensure that the secondary CPU can see
    it since it may not be taking part in coherency or cache searches at
    this point.
    
    Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

 arch/arm/kernel/smp.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 57162af..577543f 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -99,6 +99,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
 	*pmd = __pmd((PHYS_OFFSET & PGDIR_MASK) |
 		     PMD_TYPE_SECT | PMD_SECT_AP_WRITE);
 	flush_pmd_entry(pmd);
+	outer_clean_range(__pa(pmd), __pa(pmd + 1));
 
 	/*
 	 * We need to tell the secondary core where to find
@@ -106,7 +107,8 @@ int __cpuinit __cpu_up(unsigned int cpu)
 	 */
 	secondary_data.stack = task_stack_page(idle) + THREAD_START_SP;
 	secondary_data.pgdir = virt_to_phys(pgd);
-	wmb();
+	__cpuc_flush_dcache_area(&secondary_data, sizeof(secondary_data));
+	outer_clean_range(__pa(&secondary_data), __pa(&secondary_data + 1));
 
 	/*
 	 * Now bring the CPU into our world.

[PATCH] Help fix SMP boot issues with L2 cache

[Catalin Marinas]

Russell King - ARM Linux wrote:
>     ARM: Add L2 cache handling to smp boot support
>     
>     The page table and secondary data which we're asking the secondary CPU
>     to make use of has to hit RAM to ensure that the secondary CPU can see
>     it since it may not be taking part in coherency or cache searches at
>     this point.
>     
>     Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
> 
>  arch/arm/kernel/smp.c |    4 +++-
>  1 files changed, 3 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
> index 57162af..577543f 100644
> --- a/arch/arm/kernel/smp.c
> +++ b/arch/arm/kernel/smp.c
> @@ -99,6 +99,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
>  	*pmd = __pmd((PHYS_OFFSET & PGDIR_MASK) |
>  		     PMD_TYPE_SECT | PMD_SECT_AP_WRITE);
>  	flush_pmd_entry(pmd);
> +	outer_clean_range(__pa(pmd), __pa(pmd + 1));
>  
>  	/*
>  	 * We need to tell the secondary core where to find
> @@ -106,7 +107,8 @@ int __cpuinit __cpu_up(unsigned int cpu)
>  	 */
>  	secondary_data.stack = task_stack_page(idle) + THREAD_START_SP;
>  	secondary_data.pgdir = virt_to_phys(pgd);
> -	wmb();
> +	__cpuc_flush_dcache_area(&secondary_data, sizeof(secondary_data));
> +	outer_clean_range(__pa(&secondary_data), __pa(&secondary_data + 1));
>  
>  	/*
>  	 * Now bring the CPU into our world.

But is the outer_cache initialised at this point? I recall for RealView 
I had to change the l2x0_init to an early_initcall:

http://linux-arm.org/git?p=linux-2.6-stable.git;a=commitdiff;h=70e1b8d308f635eaeb7dad14fc5c79f082a147ef;hp=f034a4ab31b0663a5b68c65503122bed1eaca349

I think I have a patch for 2.6.33, I'll dig it out on Monday. But this 
was only needed when running the kernel in non-secure mode and the L2 
was pre-initialised by the secure monitor (it may be the case on some 
Omap boards).

-- 
Catalin

[PATCH] Help fix SMP boot issues with L2 cache

[Russell King - ARM Linux]

On Sat, Feb 13, 2010 at 11:47:03PM +0000, Catalin Marinas wrote:
> Russell King - ARM Linux wrote:
>>     ARM: Add L2 cache handling to smp boot support
>>         The page table and secondary data which we're asking the 
>> secondary CPU
>>     to make use of has to hit RAM to ensure that the secondary CPU can see
>>     it since it may not be taking part in coherency or cache searches at
>>     this point.
>>         Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
>>
>>  arch/arm/kernel/smp.c |    4 +++-
>>  1 files changed, 3 insertions(+), 1 deletions(-)
>>
>> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
>> index 57162af..577543f 100644
>> --- a/arch/arm/kernel/smp.c
>> +++ b/arch/arm/kernel/smp.c
>> @@ -99,6 +99,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
>>  	*pmd = __pmd((PHYS_OFFSET & PGDIR_MASK) |
>>  		     PMD_TYPE_SECT | PMD_SECT_AP_WRITE);
>>  	flush_pmd_entry(pmd);
>> +	outer_clean_range(__pa(pmd), __pa(pmd + 1));
>>   	/*
>>  	 * We need to tell the secondary core where to find
>> @@ -106,7 +107,8 @@ int __cpuinit __cpu_up(unsigned int cpu)
>>  	 */
>>  	secondary_data.stack = task_stack_page(idle) + THREAD_START_SP;
>>  	secondary_data.pgdir = virt_to_phys(pgd);
>> -	wmb();
>> +	__cpuc_flush_dcache_area(&secondary_data, sizeof(secondary_data));
>> +	outer_clean_range(__pa(&secondary_data), __pa(&secondary_data + 1));
>>   	/*
>>  	 * Now bring the CPU into our world.
>
> But is the outer_cache initialised at this point?

If it isn't, the calls are no-ops, so are harmless.  However, there are
platforms where the L2 is already initialized at this point, and therefore
this needs to be handled.

Whether platforms have initialized their L2 cache support while the
hardware is enabled is their own lookout at this point - that's outside
of the scope of this patch.

> I recall for RealView  
> I had to change the l2x0_init to an early_initcall:
>
> http://linux-arm.org/git?p=linux-2.6-stable.git;a=commitdiff;h=70e1b8d308f635eaeb7dad14fc5c79f082a147ef;hp=f034a4ab31b0663a5b68c65503122bed1eaca349

And that's what I'm addressing with this patch - the need to patch
every platsmp.c file with that kind of dirty hack.  With my patch,
the SMP generic data structures will be covered by the appropriate
code, rather than having every SMP platform having to do that itself.

Please update your patch for mine - the only thing you need to look
after is 'pen_release'.

[PATCH] Help fix SMP boot issues with L2 cache

[Russell King - ARM Linux]

Ping.  Acks would be nice.

On Sun, Feb 14, 2010 at 12:18:58AM +0000, Russell King - ARM Linux wrote:
> On Sat, Feb 13, 2010 at 11:47:03PM +0000, Catalin Marinas wrote:
> > Russell King - ARM Linux wrote:
> >>     ARM: Add L2 cache handling to smp boot support
> >>         The page table and secondary data which we're asking the 
> >> secondary CPU
> >>     to make use of has to hit RAM to ensure that the secondary CPU can see
> >>     it since it may not be taking part in coherency or cache searches at
> >>     this point.
> >>         Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
> >>
> >>  arch/arm/kernel/smp.c |    4 +++-
> >>  1 files changed, 3 insertions(+), 1 deletions(-)
> >>
> >> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
> >> index 57162af..577543f 100644
> >> --- a/arch/arm/kernel/smp.c
> >> +++ b/arch/arm/kernel/smp.c
> >> @@ -99,6 +99,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
> >>  	*pmd = __pmd((PHYS_OFFSET & PGDIR_MASK) |
> >>  		     PMD_TYPE_SECT | PMD_SECT_AP_WRITE);
> >>  	flush_pmd_entry(pmd);
> >> +	outer_clean_range(__pa(pmd), __pa(pmd + 1));
> >>   	/*
> >>  	 * We need to tell the secondary core where to find
> >> @@ -106,7 +107,8 @@ int __cpuinit __cpu_up(unsigned int cpu)
> >>  	 */
> >>  	secondary_data.stack = task_stack_page(idle) + THREAD_START_SP;
> >>  	secondary_data.pgdir = virt_to_phys(pgd);
> >> -	wmb();
> >> +	__cpuc_flush_dcache_area(&secondary_data, sizeof(secondary_data));
> >> +	outer_clean_range(__pa(&secondary_data), __pa(&secondary_data + 1));
> >>   	/*
> >>  	 * Now bring the CPU into our world.
> >
> > But is the outer_cache initialised at this point?
> 
> If it isn't, the calls are no-ops, so are harmless.  However, there are
> platforms where the L2 is already initialized at this point, and therefore
> this needs to be handled.
> 
> Whether platforms have initialized their L2 cache support while the
> hardware is enabled is their own lookout at this point - that's outside
> of the scope of this patch.
> 
> > I recall for RealView  
> > I had to change the l2x0_init to an early_initcall:
> >
> > http://linux-arm.org/git?p=linux-2.6-stable.git;a=commitdiff;h=70e1b8d308f635eaeb7dad14fc5c79f082a147ef;hp=f034a4ab31b0663a5b68c65503122bed1eaca349
> 
> And that's what I'm addressing with this patch - the need to patch
> every platsmp.c file with that kind of dirty hack.  With my patch,
> the SMP generic data structures will be covered by the appropriate
> code, rather than having every SMP platform having to do that itself.
> 
> Please update your patch for mine - the only thing you need to look
> after is 'pen_release'.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH] Help fix SMP boot issues with L2 cache

[Shilimkar, Santosh]

> -----Original Message-----
> From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
> Sent: Thursday, February 18, 2010 10:43 PM
> To: Catalin Marinas; Shilimkar, Santosh
> Cc: linux-arm-kernel at lists.infradead.org
> Subject: Re: [PATCH] Help fix SMP boot issues with L2 cache
> 
> Ping.  Acks would be nice.
> 
> On Sun, Feb 14, 2010 at 12:18:58AM +0000, Russell King - ARM Linux wrote:
> > On Sat, Feb 13, 2010 at 11:47:03PM +0000, Catalin Marinas wrote:
> > > Russell King - ARM Linux wrote:
> > >>     ARM: Add L2 cache handling to smp boot support
> > >>         The page table and secondary data which we're asking the
> > >> secondary CPU
> > >>     to make use of has to hit RAM to ensure that the secondary CPU can see
> > >>     it since it may not be taking part in coherency or cache searches at
> > >>     this point.
> > >>         Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
> > >>
> > >>  arch/arm/kernel/smp.c |    4 +++-
> > >>  1 files changed, 3 insertions(+), 1 deletions(-)
> > >>
> > >> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
> > >> index 57162af..577543f 100644
> > >> --- a/arch/arm/kernel/smp.c
> > >> +++ b/arch/arm/kernel/smp.c
> > >> @@ -99,6 +99,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
> > >>  	*pmd = __pmd((PHYS_OFFSET & PGDIR_MASK) |
> > >>  		     PMD_TYPE_SECT | PMD_SECT_AP_WRITE);
> > >>  	flush_pmd_entry(pmd);
> > >> +	outer_clean_range(__pa(pmd), __pa(pmd + 1));
> > >>   	/*
> > >>  	 * We need to tell the secondary core where to find
> > >> @@ -106,7 +107,8 @@ int __cpuinit __cpu_up(unsigned int cpu)
> > >>  	 */
> > >>  	secondary_data.stack = task_stack_page(idle) + THREAD_START_SP;
> > >>  	secondary_data.pgdir = virt_to_phys(pgd);
> > >> -	wmb();
> > >> +	__cpuc_flush_dcache_area(&secondary_data, sizeof(secondary_data));
> > >> +	outer_clean_range(__pa(&secondary_data), __pa(&secondary_data + 1));
> > >>   	/*
> > >>  	 * Now bring the CPU into our world.
> > >
> > > But is the outer_cache initialised at this point?
Outer cache is initialized by this time, so it's good. Infact this
is the right place to flush the secondary data and page tables instead of 
every arm-smp platform doing that as part of "boot_secondary".

Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>

Regards,
Santosh

[PATCH] Help fix SMP boot issues with L2 cache

[Catalin Marinas]

On Sun, 2010-02-14 at 00:18 +0000, Russell King - ARM Linux wrote:
> On Sat, Feb 13, 2010 at 11:47:03PM +0000, Catalin Marinas wrote:
> > Russell King - ARM Linux wrote:
> >>     ARM: Add L2 cache handling to smp boot support
> >>         The page table and secondary data which we're asking the
> >> secondary CPU
> >>     to make use of has to hit RAM to ensure that the secondary CPU can see
> >>     it since it may not be taking part in coherency or cache searches at
> >>     this point.
> >>         Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
> >>
> >>  arch/arm/kernel/smp.c |    4 +++-
> >>  1 files changed, 3 insertions(+), 1 deletions(-)
> >>
> >> diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
> >> index 57162af..577543f 100644
> >> --- a/arch/arm/kernel/smp.c
> >> +++ b/arch/arm/kernel/smp.c
> >> @@ -99,6 +99,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
> >>      *pmd = __pmd((PHYS_OFFSET & PGDIR_MASK) |
> >>                   PMD_TYPE_SECT | PMD_SECT_AP_WRITE);
> >>      flush_pmd_entry(pmd);
> >> +    outer_clean_range(__pa(pmd), __pa(pmd + 1));
> >>      /*
> >>       * We need to tell the secondary core where to find
> >> @@ -106,7 +107,8 @@ int __cpuinit __cpu_up(unsigned int cpu)
> >>       */
> >>      secondary_data.stack = task_stack_page(idle) + THREAD_START_SP;
> >>      secondary_data.pgdir = virt_to_phys(pgd);
> >> -    wmb();
> >> +    __cpuc_flush_dcache_area(&secondary_data, sizeof(secondary_data));
> >> +    outer_clean_range(__pa(&secondary_data), __pa(&secondary_data + 1));
> >>      /*
> >>       * Now bring the CPU into our world.
> >
> > But is the outer_cache initialised at this point?
> 
> If it isn't, the calls are no-ops, so are harmless.  However, there are
> platforms where the L2 is already initialized at this point, and therefore
> this needs to be handled.

Yes, for what it is intended the patch is fine:

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

-- 
Catalin