[RFC PATCH 0/6] ARM: augment cache flushing API

[RFC PATCH 0/6] ARM: augment cache flushing API

[Lorenzo Pieralisi]

This patch series provides an update of a previous posting:

http://www.spinics.net/lists/arm-kernel/msg169075.html

Main changes:
 - Changed the new API to Level of Unification Inner Shareable (LoUIS)
 - Fixed a pointer bug in __cpu_suspend_save code update
 - Added patches to update __cpu_disable and __v7_setup to LoUIS API
 - Changed the v7 dcache level function to make it clean/invalidate a specific
   level instead of up to a certain level

v7 ARM architecture introduced the concept of cache levels and relative
control registers to manage them. Cache operations that operate on set/way
require to define the cache level at which maintenance operations are carried
out by using coprocessor registers.

Processors like A7/A15 integrates a unified L2 that is part of the cache
level hierarchy; this implies that cache operations operating on all levels
also end up cleaning the L2 unified cache which is a very time consuming
operation and it is not needed for some power-down operations like single CPU
shutdown.

For v7, flush_kern_all() cleans all the cache levels up to the Level of
Coherency which includes L2 in it. This is suboptimal for code paths that end
up shutting-down a single processor like CPU hotplug and CPU idle, where only
per-CPU cache state (ie L1 integrated cache) has to be cleaned and invalidated.

To fix this performance issue this patchset introduces cache LoUIS (Level of
Unification Inner Shareable) maintenance operations in the kernel.

A new cache operations pointer is added to cpu_cache_fns

void (*flush_kern_cache_louis)(void);

that allows to clean and invalidate all data cache levels up to the LoUIS and
invalidate the instruction cache. This new API should provide a sufficiently
optimized API to be used in generic C code in the kernel for power management
operations on most v7 systems.

For architecture versions previous to v7, flush_kern_cache_louis() falls back
to flush_kern_all() leaving the current behaviour unchanged.

In order to allow finer grain operations on cache levels, this series also
defines an assembly stub for v7 that allows to clean and invalidate a specific
data cache level and it is provided for completeness.

For A9/A5 processors Level of Unification Inner Shareable and Level of
Coherency are equivalent hence this patch should not affect current kernel
behaviour in any way when run on A9/A5 based systems, but should nonetheless
be thoroughly tested on them.

Tested on:
  - OMAP4 (S2R, cpuidle and hotplug)
  - OMAP5 (out of tree code) (S2R, cpuidle and hotplug)
  - TC2 big.LITTLE testchip (out of tree code) (cpuidle, on both A7 and A15
    clusters)

Lorenzo Pieralisi (4):
  ARM: mm: define LoUIS API for cache maintenance ops
  ARM: mm: add v7 cache LoUIS API implementation
  ARM: kernel: update cpu_suspend code to use cache LoUIS operations
  ARM: kernel: update __cpu_disable to use cache LoUIS maintenance API

Santosh Shilimkar (2):
  ARM: mm: add v7 dcache level API
  ARM: mm: update __v7_setup() to the new LoUIS cache maintenance API

 arch/arm/include/asm/cacheflush.h | 17 +++++++++++
 arch/arm/kernel/smp.c             |  5 +++-
 arch/arm/kernel/suspend.c         | 17 ++++++++++-
 arch/arm/mm/cache-v7.S            | 62 +++++++++++++++++++++++++++++++++++++--
 arch/arm/mm/proc-macros.S         |  7 ++++-
 arch/arm/mm/proc-v7.S             |  2 +-
 6 files changed, 103 insertions(+), 7 deletions(-)

-- 
1.7.12

[RFC PATCH 1/6] ARM: mm: define LoUIS API for cache maintenance ops

[Lorenzo Pieralisi]

ARM v7 architecture introduced the concept of cache levels and related
coherency requirements. New processors like A7 and A15 embed an
L2 unified cache controller that becomes part of the cache level
hierarchy. Some operations in the kernel like cpu_suspend and __cpu_disable
does not require a flush of the entire cache hierarchy to DRAM but just the
cache levels belonging to the Level of Unification Inner Shareable (LoUIS),
which in most of ARM v7 systems correspond to L1.

The current cache flushing API used in cpu_suspend and __cpu_disable,
flush_cache_all(), ends up flushing the whole cache hierarchy since for
v7 it cleans and invalidates all cache levels up to Level of Coherency
(LoC) which cripples system performance when used in hot paths like hotplug
and cpuidle.

Therefore a new kernel cache maintenance API must be added to the
cpu_cache_fns structure of pointers to cope with latest ARM system requirements.
This patch adds flush_cache_louis() to the ARM kernel cache maintenance API.

This function cleans and invalidates all data cache levels up to the
level of unification inner shareable (LoUIS) and invalidates the instruction
cache.

The cpu_cache_fns struct reflects this change by adding a new function pointer
that is initialized by arch specific assembly files.

By default, all existing ARM archs do not instantiate any cache LoUIS function
pointer, and flush_dcache_louis just falls back to flush_kern_all.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
 arch/arm/include/asm/cacheflush.h | 17 +++++++++++++++++
 arch/arm/mm/proc-macros.S         |  7 ++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h
index c6e2ed9..7683316 100644
--- a/arch/arm/include/asm/cacheflush.h
+++ b/arch/arm/include/asm/cacheflush.h
@@ -50,6 +50,13 @@
  *
  *		Unconditionally clean and invalidate the entire cache.
  *
+ *     flush_kern_cache_louis()
+ *
+ *             Flush data cache levels up to the level of unification
+ *             inner shareable and invalidate the I-cache.
+ *             Only needed from v7 onwards, falls back to flush_cache_all()
+ *             for all other processor versions.
+ *
  *	flush_user_all()
  *
  *		Clean and invalidate all user space cache entries
@@ -98,6 +105,7 @@
 struct cpu_cache_fns {
 	void (*flush_icache_all)(void);
 	void (*flush_kern_all)(void);
+	void (*flush_kern_cache_louis)(void);
 	void (*flush_user_all)(void);
 	void (*flush_user_range)(unsigned long, unsigned long, unsigned int);
 
@@ -200,6 +208,15 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *,
 #define __flush_icache_preferred	__flush_icache_all_generic
 #endif
 
+/*
+ * Flush caches up to Level of Unification Inner Shareable
+ */
+#ifdef MULTI_CACHE
+#define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
+#else
+#define flush_cache_louis()   __cpuc_flush_kern_all()
+#endif
+
 static inline void __flush_icache_all(void)
 {
 	__flush_icache_preferred();
diff --git a/arch/arm/mm/proc-macros.S b/arch/arm/mm/proc-macros.S
index 2d8ff3a..28e91f8 100644
--- a/arch/arm/mm/proc-macros.S
+++ b/arch/arm/mm/proc-macros.S
@@ -293,12 +293,17 @@ ENTRY(\name\()_processor_functions)
 	.size	\name\()_processor_functions, . - \name\()_processor_functions
 .endm
 
-.macro define_cache_functions name:req
+.macro define_cache_functions name:req, cachelouis=0
 	.align 2
 	.type	\name\()_cache_fns, #object
 ENTRY(\name\()_cache_fns)
 	.long	\name\()_flush_icache_all
 	.long	\name\()_flush_kern_cache_all
+	.if \cachelouis
+	.long   \name\()_flush_kern_cache_louis
+	.else
+	.long   \name\()_flush_kern_cache_all
+	.endif
 	.long	\name\()_flush_user_cache_all
 	.long	\name\()_flush_user_cache_range
 	.long	\name\()_coherent_kern_range
-- 
1.7.12

[RFC PATCH 2/6] ARM: mm: add v7 cache LoUIS API implementation

[Lorenzo Pieralisi]

ARM v7 architecture introduces the concept of cache levels and registers to
probe and manage cache levels accordingly.

This patch adds v7 support for cache LoUIS (Level of Unification Inner
Shareable) operations and defines a function that allows to clean and
invalidate data caches up to LoUIS.

Power-down operations like hotplug and CPU idle require to clean/invalidate
only cache levels that are within the CPU power domain, and LoUIS reflects
this requirement properly in most of the systems.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
 arch/arm/mm/cache-v7.S | 42 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 3 deletions(-)

diff --git a/arch/arm/mm/cache-v7.S b/arch/arm/mm/cache-v7.S
index 39e3fb3..74aec79 100644
--- a/arch/arm/mm/cache-v7.S
+++ b/arch/arm/mm/cache-v7.S
@@ -33,6 +33,24 @@ ENTRY(v7_flush_icache_all)
 	mov	pc, lr
 ENDPROC(v7_flush_icache_all)
 
+ /*
+ *     v7_flush_dcache_louis()
+ *
+ *     Flush the D-cache up to the Level of Unification Inner Shareable
+ *
+ *     Corrupted registers: r0-r7, r9-r11 (r6 only in Thumb mode)
+ */
+
+ENTRY(v7_flush_dcache_louis)
+	dmb					@ ensure ordering with previous memory accesses
+	mrc	p15, 1, r0, c0, c0, 1		@ read clidr
+	ands	r3, r0, #0xe00000		@ extract louis from clidr
+	mov	r3, r3, lsr #20			@ left align louis bit field
+	moveq	pc, lr				@ return if level == 0
+	mov	r10, #0				@ starting level == 0
+	b	__flush_level
+ENDPROC(v7_flush_dcache_louis)
+
 /*
  *	v7_flush_dcache_all()
  *
@@ -49,7 +67,7 @@ ENTRY(v7_flush_dcache_all)
 	mov	r3, r3, lsr #23			@ left align loc bit field
 	beq	finished			@ if loc is 0, then no need to clean
 	mov	r10, #0				@ start clean at cache level 0
-loop1:
+__flush_level:
 	add	r2, r10, r10, lsr #1		@ work out 3x current cache level
 	mov	r1, r0, lsr r2			@ extract cache type bits from clidr
 	and	r1, r1, #7			@ mask of the bits for current cache only
@@ -88,7 +106,7 @@ loop3:
 skip:
 	add	r10, r10, #2			@ increment cache number
 	cmp	r3, r10
-	bgt	loop1
+	bgt	__flush_level
 finished:
 	mov	r10, #0				@ swith back to cache level 0
 	mcr	p15, 2, r10, c0, c0, 0		@ select current cache level in cssr
@@ -120,6 +138,24 @@ ENTRY(v7_flush_kern_cache_all)
 	mov	pc, lr
 ENDPROC(v7_flush_kern_cache_all)
 
+ /*
+ *     v7_flush_kern_cache_louis(void)
+ *
+ *     Flush the data cache up to Level of Unification Inner Shareable.
+ *     Invalidate the I-cache to the point of unification.
+ */
+ENTRY(v7_flush_kern_cache_louis)
+ ARM(	stmfd	sp!, {r4-r5, r7, r9-r11, lr}	)
+ THUMB(	stmfd	sp!, {r4-r7, r9-r11, lr}	)
+	bl	v7_flush_dcache_louis
+	mov	r0, #0
+	ALT_SMP(mcr	p15, 0, r0, c7, c1, 0)	@ invalidate I-cache inner shareable
+	ALT_UP(mcr	p15, 0, r0, c7, c5, 0)	@ I+BTB cache invalidate
+ ARM(	ldmfd	sp!, {r4-r5, r7, r9-r11, lr}	)
+ THUMB(	ldmfd	sp!, {r4-r7, r9-r11, lr}	)
+	mov	pc, lr
+ENDPROC(v7_flush_kern_cache_louis)
+
 /*
  *	v7_flush_cache_all()
  *
@@ -350,4 +386,4 @@ ENDPROC(v7_dma_unmap_area)
 	__INITDATA
 
 	@ define struct cpu_cache_fns (see <asm/cacheflush.h> and proc-macros.S)
-	define_cache_functions v7
+       define_cache_functions v7, cachelouis=1
-- 
1.7.12

[RFC PATCH 3/6] ARM: mm: add v7 dcache level API

[Lorenzo Pieralisi]

From: Santosh Shilimkar <santosh.shilimkar@ti.com>

On ARMv7 based SOC with an integrated L2 cache, there is a need
to have a flush API to operate on each cache level. In few low
power modes, L2 cache is retained whereas L1 is lost. The current
v7_flush_dcache_all(), flushes all the levels and it would be quite
expensive in cases where only one of the level needs to be flushed.

So this patch introduces v7_flush_dcache_level() API which takes a
parameter (cache level), and flush only that level.
This API is useful for the power management code where depending on CPU
and CPU cluster low power state, a specific cache level can be cleaned
instead of cleaning all the cache levels with existing flush_dcache_all().

Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
 arch/arm/mm/cache-v7.S | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/arm/mm/cache-v7.S b/arch/arm/mm/cache-v7.S
index 74aec79..d0fbe5c 100644
--- a/arch/arm/mm/cache-v7.S
+++ b/arch/arm/mm/cache-v7.S
@@ -51,6 +51,26 @@ ENTRY(v7_flush_dcache_louis)
 	b	__flush_level
 ENDPROC(v7_flush_dcache_louis)
 
+ /*
+ *     v7_flush_dcache_level(level)
+ *
+ *     Flush the D-cache the specified level passed as input parameter.
+ *
+ *     r0 - cache level
+ *
+ *     Corrupted registers: r0-r7, r9-r11 (r6 only in Thumb mode)
+ */
+
+ENTRY(v7_flush_dcache_level)
+	dmb					@ ensure ordering with previous memory accesses
+	sub	r10, r0, #1
+	mov	r10, r10, lsl #1
+	movs	r3, r0, lsl #1			@ level * 2
+	mrc	p15, 1, r0, c0, c0, 1		@ read clidr
+	moveq	pc, lr				@ return if level == 0
+	b	__flush_level
+ENDPROC(v7_flush_dcache_level)
+
 /*
  *	v7_flush_dcache_all()
  *
-- 
1.7.12

[RFC PATCH 4/6] ARM: kernel: update cpu_suspend code to use cache LoUIS operations

[Lorenzo Pieralisi]

In processors like A15/A7 L2 cache is unified and integrated within the
processor cache hierarchy, so that it is not considered an outer cache
anymore. For processors like A15/A7 flush_cache_all() ends up cleaning
all cache levels up to Level of Coherency (LoC) that includes
the L2 unified cache.

When a single CPU is suspended (CPU idle) a complete L2 clean is not
required, so generic cpu_suspend code must clean the data cache using the
newly introduced cache LoUIS function.

The context and stack pointer (context pointer) are cleaned to main memory
using cache area functions that operate on MVA and guarantee that the data
is written back to main memory (perform cache cleaning up to the Point of
Coherency - PoC) so that the processor can fetch the context when the MMU
is off in the cpu_resume code path.

outer_cache management remains unchanged.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
 arch/arm/kernel/suspend.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/arch/arm/kernel/suspend.c b/arch/arm/kernel/suspend.c
index 1794cc3..358bca3 100644
--- a/arch/arm/kernel/suspend.c
+++ b/arch/arm/kernel/suspend.c
@@ -17,6 +17,8 @@ extern void cpu_resume_mmu(void);
  */
 void __cpu_suspend_save(u32 *ptr, u32 ptrsz, u32 sp, u32 *save_ptr)
 {
+	u32 *ctx = ptr;
+
 	*save_ptr = virt_to_phys(ptr);
 
 	/* This must correspond to the LDM in cpu_resume() assembly */
@@ -26,7 +28,20 @@ void __cpu_suspend_save(u32 *ptr, u32 ptrsz, u32 sp, u32 *save_ptr)
 
 	cpu_do_suspend(ptr);
 
-	flush_cache_all();
+	flush_cache_louis();
+
+	/*
+	 * flush_cache_louis does not guarantee that
+	 * save_ptr and ptr are cleaned to main memory,
+	 * just up to the Level of Unification Inner Shareable.
+	 * Since the context pointer and context itself
+	 * are to be retrieved with the MMU off that
+	 * data must be cleaned from all cache levels
+	 * to main memory using "area" cache primitives.
+	*/
+	__cpuc_flush_dcache_area(ctx, ptrsz);
+	__cpuc_flush_dcache_area(save_ptr, sizeof(*save_ptr));
+
 	outer_clean_range(*save_ptr, *save_ptr + ptrsz);
 	outer_clean_range(virt_to_phys(save_ptr),
 			  virt_to_phys(save_ptr) + sizeof(*save_ptr));
-- 
1.7.12

[RFC PATCH 5/6] ARM: kernel: update __cpu_disable to use cache LoUIS maintenance API

[Lorenzo Pieralisi]

When a CPU is hotplugged out caches that reside in its power domain
lose their contents and so must be cleaned to the next memory level.

Currently, __cpu_disable calls flush_cache_all() that for new generation
processor like A15/A7 ends up cleaning and invalidating all cache levels
up to Level of Coherency, which includes the unified L2.

This ends up being a waste of cycles since the L2 cache contents are not
lost on power down.

This patch updates __cpu_disable to use the new LoUIS API cache operations.

Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
 arch/arm/kernel/smp.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index d3eb222..f44e9cd 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -136,8 +136,11 @@ int __cpu_disable(void)
 	/*
 	 * Flush user cache and TLB mappings, and then remove this CPU
 	 * from the vm mask set of all processes.
+	 *
+	 * Caches are flushed to the Level of Unification Inner Shareable
+	 * to write-back dirty lines to unified caches shared by all CPUs.
 	 */
-	flush_cache_all();
+	flush_cache_louis();
 	local_flush_tlb_all();
 
 	clear_tasks_mm_cpumask(cpu);
-- 
1.7.12

[RFC PATCH 6/6] ARM: mm: update __v7_setup() to the new LoUIS cache maintenance API

[Lorenzo Pieralisi]

From: Santosh Shilimkar <santosh.shilimkar@ti.com>

The ARMv7 processor setup function __v7_setup() cleans and invalidates the
CPU cache before enabling MMU to start the CPU with a clean CPU local cache.

But on ARMv7 architectures like Cortex-[A15/A8], this code will end
up flushing the L2 caches(up to level of Coherency) which is undesirable
and expensive. The setup functions are used in the CPU hotplug scenario too
and hence flushing all cache levels should be avoided.

This patch replaces the cache flushing call with the newly introduced
v7 dcache LoUIS API where only cache levels up to LoUIS are cleaned and
invalidated when a processors executes __v7_setup which is the expected
behavior.

For processors like A9 and A5 where the L2 cache is an outer one the
behavior should be unchanged.

Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
---
 arch/arm/mm/proc-v7.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index c2e2b66..846d279 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -172,7 +172,7 @@ __v7_ca15mp_setup:
 __v7_setup:
 	adr	r12, __v7_setup_stack		@ the local stack
 	stmia	r12, {r0-r5, r7, r9, r11, lr}
-	bl	v7_flush_dcache_all
+	bl      v7_flush_dcache_louis
 	ldmia	r12, {r0-r5, r7, r9, r11, lr}
 
 	mrc	p15, 0, r0, c0, c0, 0		@ read main ID register
-- 
1.7.12

[RFC PATCH 1/6] ARM: mm: define LoUIS API for cache maintenance ops

[Dave Martin]

On Thu, Sep 13, 2012 at 11:20:46AM +0100, Lorenzo Pieralisi wrote:
> ARM v7 architecture introduced the concept of cache levels and related
> coherency requirements. New processors like A7 and A15 embed an
> L2 unified cache controller that becomes part of the cache level
> hierarchy. Some operations in the kernel like cpu_suspend and __cpu_disable
> does not require a flush of the entire cache hierarchy to DRAM but just the
> cache levels belonging to the Level of Unification Inner Shareable (LoUIS),
> which in most of ARM v7 systems correspond to L1.
> 
> The current cache flushing API used in cpu_suspend and __cpu_disable,
> flush_cache_all(), ends up flushing the whole cache hierarchy since for
> v7 it cleans and invalidates all cache levels up to Level of Coherency
> (LoC) which cripples system performance when used in hot paths like hotplug
> and cpuidle.
> 
> Therefore a new kernel cache maintenance API must be added to the
> cpu_cache_fns structure of pointers to cope with latest ARM system requirements.
> This patch adds flush_cache_louis() to the ARM kernel cache maintenance API.
> 
> This function cleans and invalidates all data cache levels up to the
> level of unification inner shareable (LoUIS) and invalidates the instruction
> cache.
> 
> The cpu_cache_fns struct reflects this change by adding a new function pointer
> that is initialized by arch specific assembly files.
> 
> By default, all existing ARM archs do not instantiate any cache LoUIS function
> pointer, and flush_dcache_louis just falls back to flush_kern_all.
> 
> Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> ---
>  arch/arm/include/asm/cacheflush.h | 17 +++++++++++++++++
>  arch/arm/mm/proc-macros.S         |  7 ++++++-
>  2 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h
> index c6e2ed9..7683316 100644
> --- a/arch/arm/include/asm/cacheflush.h
> +++ b/arch/arm/include/asm/cacheflush.h
> @@ -50,6 +50,13 @@
>   *
>   *		Unconditionally clean and invalidate the entire cache.
>   *
> + *     flush_kern_cache_louis()
> + *
> + *             Flush data cache levels up to the level of unification
> + *             inner shareable and invalidate the I-cache.
> + *             Only needed from v7 onwards, falls back to flush_cache_all()
> + *             for all other processor versions.
> + *
>   *	flush_user_all()
>   *
>   *		Clean and invalidate all user space cache entries
> @@ -98,6 +105,7 @@
>  struct cpu_cache_fns {
>  	void (*flush_icache_all)(void);
>  	void (*flush_kern_all)(void);
> +	void (*flush_kern_cache_louis)(void);
>  	void (*flush_user_all)(void);
>  	void (*flush_user_range)(unsigned long, unsigned long, unsigned int);
>  
> @@ -200,6 +208,15 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *,
>  #define __flush_icache_preferred	__flush_icache_all_generic
>  #endif
>  
> +/*
> + * Flush caches up to Level of Unification Inner Shareable
> + */
> +#ifdef MULTI_CACHE
> +#define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
> +#else
> +#define flush_cache_louis()   __cpuc_flush_kern_all()
> +#endif

So, without MULTI_CACHE, we always fall back to flush_kern_all.

I'm guessing this is done because CPUs can't be relied on to provide
flush_kern_cache_louis.  Shouldn't this be handled directly? 

We could introduce something like CONFIG_ARM_HAVE_CACHEFLUSH_LOUIS, and
do:

<asm/glue-cache.h>
#ifndef MULTI_CACHE
#ifdef CONFIG_HAVE_ARM_CACHEFLUSH_LOUIS
#define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_cache_louis)
#else
#define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_all)
#endif
#endif

<asm/cacheflush.h>
#ifdef MULTI_CACHE
#define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
#else
#define flush_cache_louis()   __cpuc_flush_kern_cache_louis()
#endif


Any good?

Then the only question is whether this is worth the complexity.

This only works if the __cpuc_ aliases are not used from assembler.
That seems wrong anyway, since on a MULTI_CACHE kernel those would turn
into C struct member references which wouldn't be valid in assembler
anyway.

Cheers
---Dave

[RFC PATCH 1/6] ARM: mm: define LoUIS API for cache maintenance ops

[Russell King - ARM Linux]

On Thu, Sep 13, 2012 at 11:20:46AM +0100, Lorenzo Pieralisi wrote:
> +/*
> + * Flush caches up to Level of Unification Inner Shareable
> + */
> +#ifdef MULTI_CACHE
> +#define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
> +#else
> +#define flush_cache_louis()   __cpuc_flush_kern_all()
> +#endif

NAK.  This is broken as you don't seem to understand what MULTI_CACHE
actually means.  MULTI_CACHE means that we _may_ support more than one
type of cache, so it not being selected means nothing as far as whether
to use flush_kern_all() or not.

Follow the pattern in the rest of the file - that's the _only_ way to do
this.

Note that ARMv6 only and ARMv7 only kernels will not have MULTI_CACHE
defined (I mentioned this on Monday in our call, though not explicitly
by that name.)

> diff --git a/arch/arm/mm/proc-macros.S b/arch/arm/mm/proc-macros.S
> index 2d8ff3a..28e91f8 100644
> --- a/arch/arm/mm/proc-macros.S
> +++ b/arch/arm/mm/proc-macros.S
> @@ -293,12 +293,17 @@ ENTRY(\name\()_processor_functions)
>  	.size	\name\()_processor_functions, . - \name\()_processor_functions
>  .endm
>  
> -.macro define_cache_functions name:req
> +.macro define_cache_functions name:req, cachelouis=0
>  	.align 2
>  	.type	\name\()_cache_fns, #object
>  ENTRY(\name\()_cache_fns)
>  	.long	\name\()_flush_icache_all
>  	.long	\name\()_flush_kern_cache_all
> +	.if \cachelouis
> +	.long   \name\()_flush_kern_cache_louis
> +	.else
> +	.long   \name\()_flush_kern_cache_all
> +	.endif
>  	.long	\name\()_flush_user_cache_all
>  	.long	\name\()_flush_user_cache_range
>  	.long	\name\()_coherent_kern_range

And what the above means is that _every_ cache support file must supply a
XXX_flush_kern_cache_louis function name.  If there is no separate
implementation, then name it an alias for the XXX_flush_cache_all function.

[RFC PATCH 4/6] ARM: kernel: update cpu_suspend code to use cache LoUIS operations

[Dave Martin]

On Thu, Sep 13, 2012 at 11:20:49AM +0100, Lorenzo Pieralisi wrote:
> In processors like A15/A7 L2 cache is unified and integrated within the
> processor cache hierarchy, so that it is not considered an outer cache
> anymore. For processors like A15/A7 flush_cache_all() ends up cleaning
> all cache levels up to Level of Coherency (LoC) that includes
> the L2 unified cache.
> 
> When a single CPU is suspended (CPU idle) a complete L2 clean is not
> required, so generic cpu_suspend code must clean the data cache using the
> newly introduced cache LoUIS function.
> 
> The context and stack pointer (context pointer) are cleaned to main memory
> using cache area functions that operate on MVA and guarantee that the data
> is written back to main memory (perform cache cleaning up to the Point of
> Coherency - PoC) so that the processor can fetch the context when the MMU
> is off in the cpu_resume code path.
> 
> outer_cache management remains unchanged.

LoUIS matches the power domain affected by turning a single CPU off
on most (all?) current v7 SoCs where this matters, but only by coincidence.
There is no guarantee of that.

The _louis() API is useful, because this is exactly what you need to to
I-/D-/TLB synchronisation in an SMP OS.  Using it as preparation for
powering a CPU off feels like misuse, at least in theory.

For powerdown, we would logically need a separate function,
flush_cache_cpu() or something, whose job is precisely to flush those
levels which will be affected by the powerdown if that single CPU.

In a multi-cluster system, it's possible that the architectural cache
level this corresponds to is not even the same across all clusters (though
for the foreseeable future it probably will be -- at least for all clusters
participating in SMP).


I don't know how urgent it is to fix this if there are just a few call
sites for flush_cache_louis().  My worry would be that misuse of these
functions propagates before we find that this needs cleaning up...

Cheers
---Dave

> 
> Reviewed-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
> Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
> ---
>  arch/arm/kernel/suspend.c | 17 ++++++++++++++++-
>  1 file changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/kernel/suspend.c b/arch/arm/kernel/suspend.c
> index 1794cc3..358bca3 100644
> --- a/arch/arm/kernel/suspend.c
> +++ b/arch/arm/kernel/suspend.c
> @@ -17,6 +17,8 @@ extern void cpu_resume_mmu(void);
>   */
>  void __cpu_suspend_save(u32 *ptr, u32 ptrsz, u32 sp, u32 *save_ptr)
>  {
> +	u32 *ctx = ptr;
> +
>  	*save_ptr = virt_to_phys(ptr);
>  
>  	/* This must correspond to the LDM in cpu_resume() assembly */
> @@ -26,7 +28,20 @@ void __cpu_suspend_save(u32 *ptr, u32 ptrsz, u32 sp, u32 *save_ptr)
>  
>  	cpu_do_suspend(ptr);
>  
> -	flush_cache_all();
> +	flush_cache_louis();
> +
> +	/*
> +	 * flush_cache_louis does not guarantee that
> +	 * save_ptr and ptr are cleaned to main memory,
> +	 * just up to the Level of Unification Inner Shareable.
> +	 * Since the context pointer and context itself
> +	 * are to be retrieved with the MMU off that
> +	 * data must be cleaned from all cache levels
> +	 * to main memory using "area" cache primitives.
> +	*/
> +	__cpuc_flush_dcache_area(ctx, ptrsz);
> +	__cpuc_flush_dcache_area(save_ptr, sizeof(*save_ptr));
> +
>  	outer_clean_range(*save_ptr, *save_ptr + ptrsz);
>  	outer_clean_range(virt_to_phys(save_ptr),
>  			  virt_to_phys(save_ptr) + sizeof(*save_ptr));
> -- 
> 1.7.12
> 
> 

[RFC PATCH 1/6] ARM: mm: define LoUIS API for cache maintenance ops

[Lorenzo Pieralisi]

On Thu, Sep 13, 2012 at 01:36:04PM +0100, Russell King - ARM Linux wrote:
> On Thu, Sep 13, 2012 at 11:20:46AM +0100, Lorenzo Pieralisi wrote:
> > +/*
> > + * Flush caches up to Level of Unification Inner Shareable
> > + */
> > +#ifdef MULTI_CACHE
> > +#define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
> > +#else
> > +#define flush_cache_louis()   __cpuc_flush_kern_all()
> > +#endif
> 
> NAK.  This is broken as you don't seem to understand what MULTI_CACHE
> actually means.  MULTI_CACHE means that we _may_ support more than one
> type of cache, so it not being selected means nothing as far as whether
> to use flush_kern_all() or not.
> 
> Follow the pattern in the rest of the file - that's the _only_ way to do
> this.
> 
> Note that ARMv6 only and ARMv7 only kernels will not have MULTI_CACHE
> defined (I mentioned this on Monday in our call, though not explicitly
> by that name.)

Point taken, I will fix it.

> > diff --git a/arch/arm/mm/proc-macros.S b/arch/arm/mm/proc-macros.S
> > index 2d8ff3a..28e91f8 100644
> > --- a/arch/arm/mm/proc-macros.S
> > +++ b/arch/arm/mm/proc-macros.S
> > @@ -293,12 +293,17 @@ ENTRY(\name\()_processor_functions)
> >  	.size	\name\()_processor_functions, . - \name\()_processor_functions
> >  .endm
> >  
> > -.macro define_cache_functions name:req
> > +.macro define_cache_functions name:req, cachelouis=0
> >  	.align 2
> >  	.type	\name\()_cache_fns, #object
> >  ENTRY(\name\()_cache_fns)
> >  	.long	\name\()_flush_icache_all
> >  	.long	\name\()_flush_kern_cache_all
> > +	.if \cachelouis
> > +	.long   \name\()_flush_kern_cache_louis
> > +	.else
> > +	.long   \name\()_flush_kern_cache_all
> > +	.endif
> >  	.long	\name\()_flush_user_cache_all
> >  	.long	\name\()_flush_user_cache_range
> >  	.long	\name\()_coherent_kern_range
> 
> And what the above means is that _every_ cache support file must supply a
> XXX_flush_kern_cache_louis function name.  If there is no separate
> implementation, then name it an alias for the XXX_flush_cache_all function.

I will do that, thanks for the review.

Lorenzo

[RFC PATCH 4/6] ARM: kernel: update cpu_suspend code to use cache LoUIS operations

[Shilimkar, Santosh]

On Thu, Sep 13, 2012 at 6:23 PM, Dave Martin <dave.martin@linaro.org> wrote:
> On Thu, Sep 13, 2012 at 11:20:49AM +0100, Lorenzo Pieralisi wrote:
>> In processors like A15/A7 L2 cache is unified and integrated within the
>> processor cache hierarchy, so that it is not considered an outer cache
>> anymore. For processors like A15/A7 flush_cache_all() ends up cleaning
>> all cache levels up to Level of Coherency (LoC) that includes
>> the L2 unified cache.
>>
>> When a single CPU is suspended (CPU idle) a complete L2 clean is not
>> required, so generic cpu_suspend code must clean the data cache using the
>> newly introduced cache LoUIS function.
>>
>> The context and stack pointer (context pointer) are cleaned to main memory
>> using cache area functions that operate on MVA and guarantee that the data
>> is written back to main memory (perform cache cleaning up to the Point of
>> Coherency - PoC) so that the processor can fetch the context when the MMU
>> is off in the cpu_resume code path.
>>
>> outer_cache management remains unchanged.
>
> LoUIS matches the power domain affected by turning a single CPU off
> on most (all?) current v7 SoCs where this matters, but only by coincidence.
> There is no guarantee of that.
>
> The _louis() API is useful, because this is exactly what you need to to
> I-/D-/TLB synchronisation in an SMP OS.  Using it as preparation for
> powering a CPU off feels like misuse, at least in theory.
>
> For powerdown, we would logically need a separate function,
> flush_cache_cpu() or something, whose job is precisely to flush those
> levels which will be affected by the power-down if that single CPU.
>
In the series, there is patch "[PATCH 3/6]" which adds an
API which let you operate on a specific level.

Regards
Santosh

[RFC PATCH 1/6] ARM: mm: define LoUIS API for cache maintenance ops

[Russell King - ARM Linux]

On Thu, Sep 13, 2012 at 12:39:49PM +0100, Dave Martin wrote:
> We could introduce something like CONFIG_ARM_HAVE_CACHEFLUSH_LOUIS, and
> do:
> 
> <asm/glue-cache.h>
> #ifndef MULTI_CACHE
> #ifdef CONFIG_HAVE_ARM_CACHEFLUSH_LOUIS
> #define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_cache_louis)
> #else
> #define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_all)
> #endif
> #endif
> 
> <asm/cacheflush.h>
> #ifdef MULTI_CACHE
> #define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
> #else
> #define flush_cache_louis()   __cpuc_flush_kern_cache_louis()
> #endif

No, this is stupidly complicated and is fragile.  Just alias the
functions, like we do in cache-v4wt.S:

	.globl	v4wt_dma_flush_range
	.equ	v4wt_dma_flush_range, v4wt_dma_inv_range

except, you'll need:

	.globl	v4wt_flush_kern_cache_louis
	.equ	v4wt_flush_kern_cache_louis, v4wt_flush_kern_cache_all

You can do it automatically, using the attached sedscript and this bit
of shell:

$ for f in $(grep -l define_cache_functions arch/arm/mm/*.S ); do
    sed -if sedscript $f
    git add $f
done
$ git commit -s

Do that first, and then go over those which you need to add a real
flush_kern_cache_louis function to.

-------------- next part --------------
1,/__INITDATA\|define struct cpu_cache_fns/ {
	/ENTRY.*flush_kern_cache_all/ {
        	h
	        s/.*(\([^_]*\)_.*/\t.globl\t\1_flush_kern_cache_louis\n\t.equ\t\1_flush_kern_cache_louis, \1_flush_kern_cache_all\n/
        	x
	}
	/__INITDATA\|define struct cpu_cache_fns/ {
        	H
	        g
	}
}

[RFC PATCH 4/6] ARM: kernel: update cpu_suspend code to use cache LoUIS operations

[Russell King - ARM Linux]

On Thu, Sep 13, 2012 at 06:31:35PM +0530, Shilimkar, Santosh wrote:
> In the series, there is patch "[PATCH 3/6]" which adds an
> API which let you operate on a specific level.

Which is introduced but as far as I can see, is never used in the patch
set.  Therefore, it shouldn't be introduced.

We've been here before many many many times, where people introduce stuff
into the kernel, and then they never get around to using the damned stuff.
It's happened far too many times to permit on a "but I will use it in the
future" kind of arguments.

If you're going to introduce something new, include the users in the patch
set, or don't bother submitting the new function in the vague hope that
some day it will get used.

[RFC PATCH 4/6] ARM: kernel: update cpu_suspend code to use cache LoUIS operations

[Shilimkar, Santosh]

On Thu, Sep 13, 2012 at 6:38 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Sep 13, 2012 at 06:31:35PM +0530, Shilimkar, Santosh wrote:
>> In the series, there is patch "[PATCH 3/6]" which adds an
>> API which let you operate on a specific level.
>
> Which is introduced but as far as I can see, is never used in the patch
> set.  Therefore, it shouldn't be introduced.
>
> We've been here before many many many times, where people introduce stuff
> into the kernel, and then they never get around to using the damned stuff.
> It's happened far too many times to permit on a "but I will use it in the
> future" kind of arguments.
>
> If you're going to introduce something new, include the users in the patch
> set, or don't bother submitting the new function in the vague hope that
> some day it will get used.

Fair enough. We can postpone adding that API now in this series and
add it along with the user. For the record, it was added to use in the A15
low power code to operate on L1 and L2 levels based on the power domain
states.

Regards
santosh

[RFC PATCH 1/6] ARM: mm: define LoUIS API for cache maintenance ops

[Dave Martin]

On Thu, Sep 13, 2012 at 02:03:34PM +0100, Russell King - ARM Linux wrote:
> On Thu, Sep 13, 2012 at 12:39:49PM +0100, Dave Martin wrote:
> > We could introduce something like CONFIG_ARM_HAVE_CACHEFLUSH_LOUIS, and
> > do:
> > 
> > <asm/glue-cache.h>
> > #ifndef MULTI_CACHE
> > #ifdef CONFIG_HAVE_ARM_CACHEFLUSH_LOUIS
> > #define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_cache_louis)
> > #else
> > #define __cpuc_flush_kern_cache_louis __glue(_CACHE,_flush_kern_all)
> > #endif
> > #endif
> > 
> > <asm/cacheflush.h>
> > #ifdef MULTI_CACHE
> > #define flush_cache_louis()   cpu_cache.flush_kern_cache_louis()
> > #else
> > #define flush_cache_louis()   __cpuc_flush_kern_cache_louis()
> > #endif
> 
> No, this is stupidly complicated and is fragile.  Just alias the
> functions, like we do in cache-v4wt.S:
> 
> 	.globl	v4wt_dma_flush_range
> 	.equ	v4wt_dma_flush_range, v4wt_dma_inv_range
> 
> except, you'll need:
> 
> 	.globl	v4wt_flush_kern_cache_louis
> 	.equ	v4wt_flush_kern_cache_louis, v4wt_flush_kern_cache_all
> 
> You can do it automatically, using the attached sedscript and this bit
> of shell:
> 
> $ for f in $(grep -l define_cache_functions arch/arm/mm/*.S ); do
>     sed -if sedscript $f
>     git add $f
> done
> $ git commit -s
> 
> Do that first, and then go over those which you need to add a real
> flush_kern_cache_louis function to.

Sure, that works better.

I was trying to think of a more localised way to do it, but the result
was admittedly rather ugly (and not that localised once we select
HAVE_ARM_CACHEFLUSH_LOUIS all over the place).

Cheers
---Dave

> 1,/__INITDATA\|define struct cpu_cache_fns/ {
> 	/ENTRY.*flush_kern_cache_all/ {
>         	h
> 	        s/.*(\([^_]*\)_.*/\t.globl\t\1_flush_kern_cache_louis\n\t.equ\t\1_flush_kern_cache_louis, \1_flush_kern_cache_all\n/
>         	x
> 	}
> 	/__INITDATA\|define struct cpu_cache_fns/ {
>         	H
> 	        g
> 	}
> }

[RFC PATCH 4/6] ARM: kernel: update cpu_suspend code to use cache LoUIS operations

[Lorenzo Pieralisi]

On Thu, Sep 13, 2012 at 01:53:48PM +0100, Dave Martin wrote:
> On Thu, Sep 13, 2012 at 11:20:49AM +0100, Lorenzo Pieralisi wrote:
> > In processors like A15/A7 L2 cache is unified and integrated within the
> > processor cache hierarchy, so that it is not considered an outer cache
> > anymore. For processors like A15/A7 flush_cache_all() ends up cleaning
> > all cache levels up to Level of Coherency (LoC) that includes
> > the L2 unified cache.
> > 
> > When a single CPU is suspended (CPU idle) a complete L2 clean is not
> > required, so generic cpu_suspend code must clean the data cache using the
> > newly introduced cache LoUIS function.
> > 
> > The context and stack pointer (context pointer) are cleaned to main memory
> > using cache area functions that operate on MVA and guarantee that the data
> > is written back to main memory (perform cache cleaning up to the Point of
> > Coherency - PoC) so that the processor can fetch the context when the MMU
> > is off in the cpu_resume code path.
> > 
> > outer_cache management remains unchanged.
> 
> LoUIS matches the power domain affected by turning a single CPU off
> on most (all?) current v7 SoCs where this matters, but only by coincidence.
> There is no guarantee of that.
> 
> The _louis() API is useful, because this is exactly what you need to to
> I-/D-/TLB synchronisation in an SMP OS.  Using it as preparation for
> powering a CPU off feels like misuse, at least in theory.
> 
> For powerdown, we would logically need a separate function,
> flush_cache_cpu() or something, whose job is precisely to flush those
> levels which will be affected by the powerdown if that single CPU.

Yes, you are right, we discussed this at length and I think that
cleaning to LoUIS is a good trade-off for now. Detecting what cache levels are
affected by the power down in question implies linking caches to power
domains in some sensible (and ARM generic) way, let's not go that far for now.

Thanks,
Lorenzo

[RFC PATCH 4/6] ARM: kernel: update cpu_suspend code to use cache LoUIS operations

[Lorenzo Pieralisi]

On Thu, Sep 13, 2012 at 02:01:35PM +0100, Shilimkar, Santosh wrote:
> On Thu, Sep 13, 2012 at 6:23 PM, Dave Martin <dave.martin@linaro.org> wrote:

[...]

> > LoUIS matches the power domain affected by turning a single CPU off
> > on most (all?) current v7 SoCs where this matters, but only by coincidence.
> > There is no guarantee of that.
> >
> > The _louis() API is useful, because this is exactly what you need to to
> > I-/D-/TLB synchronisation in an SMP OS.  Using it as preparation for
> > powering a CPU off feels like misuse, at least in theory.
> >
> > For powerdown, we would logically need a separate function,
> > flush_cache_cpu() or something, whose job is precisely to flush those
> > levels which will be affected by the power-down if that single CPU.
> >
> In the series, there is patch "[PATCH 3/6]" which adds an
> API which let you operate on a specific level.

Yep, but that's not callable from __cpu_suspend_save in a generic way
(it is v7 specific and we are not going to add another API/macro for
that to be usable in generic code, at least for now).

Let's keep that patch in our trees and as Russell suggested we will repost it
with the PM BSP accordingly to raise the point and take it from there.

Thanks,
Lorenzo