[PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Stephen Boyd]

armv7's flush_cache_all() flushes caches via set/way. To
determine the cache attributes (line size, number of sets,
etc.) the assembly first writes the CSSELR register to select a
cache level and then reads the CCSIDR register. The CSSELR register
is banked per-cpu and is used to determine which cache level CCSIDR
reads. If the task is migrated between when the CSSELR is written and
the CCSIDR is read the CCSIDR value may be for an unexpected cache
level (for example L1 instead of L2) and incorrect cache flushing
could occur.

Disable preemption across the write and read so that the correct
cache attributes are read and used for the cache flushing
routine. This fixes a problem we see in scm_call() when
flush_cache_all() is called from preemptible context and
sometimes the L2 cache is not properly flushed out.

Signed-off-by: Stephen Boyd <sboyd@codeaurora.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
---

Should we move get_thread_info into assembler.h? It seems odd
to include entry-header.S but I saw that vfp was doing the same.

 arch/arm/mm/cache-v7.S |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/arch/arm/mm/cache-v7.S b/arch/arm/mm/cache-v7.S
index 07c4bc8..a033858 100644
--- a/arch/arm/mm/cache-v7.S
+++ b/arch/arm/mm/cache-v7.S
@@ -16,6 +16,7 @@
 #include <asm/unwind.h>
 
 #include "proc-macros.S"
+#include "../kernel/entry-header.S"
 
 /*
  *	v7_flush_icache_all()
@@ -54,9 +55,19 @@ loop1:
 	and	r1, r1, #7			@ mask of the bits for current cache only
 	cmp	r1, #2				@ see what cache we have at this level
 	blt	skip				@ skip if no cache, or just i-cache
+#ifdef CONFIG_PREEMPT
+	get_thread_info r9
+	ldr	r11, [r9, #TI_PREEMPT]		@ get preempt count
+	add	r11, r11, #1			@ increment it
+	str	r11, [r9, #TI_PREEMPT]
+#endif
 	mcr	p15, 2, r10, c0, c0, 0		@ select current cache level in cssr
 	isb					@ isb to sych the new cssr&csidr
 	mrc	p15, 1, r1, c0, c0, 0		@ read the new csidr
+#ifdef CONFIG_PREEMPT
+	sub	r11, r11, #1			@ decrement preempt count
+	str	r11, [r9, #TI_PREEMPT]
+#endif
 	and	r2, r1, #7			@ extract the length of the cache lines
 	add	r2, r2, #4			@ add 4 (line length offset)
 	ldr	r4, =0x3ff
-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Russell King - ARM Linux]

On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
> armv7's flush_cache_all() flushes caches via set/way. To
> determine the cache attributes (line size, number of sets,
> etc.) the assembly first writes the CSSELR register to select a
> cache level and then reads the CCSIDR register. The CSSELR register
> is banked per-cpu and is used to determine which cache level CCSIDR
> reads. If the task is migrated between when the CSSELR is written and
> the CCSIDR is read the CCSIDR value may be for an unexpected cache
> level (for example L1 instead of L2) and incorrect cache flushing
> could occur.
> 
> Disable preemption across the write and read so that the correct
> cache attributes are read and used for the cache flushing
> routine. This fixes a problem we see in scm_call() when
> flush_cache_all() is called from preemptible context and
> sometimes the L2 cache is not properly flushed out.

This isn't going to work for two reasons:

(1) (and the VFP code suffers from this) after we re-enable preemption,
    we really should check for a pending preemption event in every case.

(2) v7_flush_dcache_all() is called from __v7_setup() using a very small
    private stack.  This doesn't have a thread info structure at the
    bottom.

So, if we need to disable preemption here, we need to find a different
solution to it.

> Should we move get_thread_info into assembler.h? It seems odd
> to include entry-header.S but I saw that vfp was doing the same.

Probably yes, and probably also have preempt_disable and preempt_enable
assembler macros.  That's going to get rather icky if we have to
explicitly call the scheduler though (to solve (1)).

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Nicolas Pitre]

On Thu, 2 Feb 2012, Russell King - ARM Linux wrote:

> On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
> > Should we move get_thread_info into assembler.h? It seems odd
> > to include entry-header.S but I saw that vfp was doing the same.
> 
> Probably yes, and probably also have preempt_disable and preempt_enable
> assembler macros.  That's going to get rather icky if we have to
> explicitly call the scheduler though (to solve (1)).

What about a pair of helpers written in C instead?

v7_flush_dcache_all() could be renamed, and a wrapper function called 
v7_flush_dcache_all() would call the preemption disable helper, call the 
former v7_flush_dcache_all code, then call the preemption enable helper.

Then __v7_setup() could still call the core cache flush code without 
issues.


Nicolas

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Stephen Boyd]

On 02/02/12 13:38, Nicolas Pitre wrote:
> On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
>> On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
>>> Should we move get_thread_info into assembler.h? It seems odd
>>> to include entry-header.S but I saw that vfp was doing the same.
>> Probably yes, and probably also have preempt_disable and preempt_enable
>> assembler macros.  That's going to get rather icky if we have to
>> explicitly call the scheduler though (to solve (1)).
> What about a pair of helpers written in C instead?
>
> v7_flush_dcache_all() could be renamed, and a wrapper function called 
> v7_flush_dcache_all() would call the preemption disable helper, call the 
> former v7_flush_dcache_all code, then call the preemption enable helper.
>
> Then __v7_setup() could still call the core cache flush code without 
> issues.

I tried to put the preemption disable/enable right around the place
where it was needed. With this approach we would disable preemption
during the entire cache flush. I'm not sure if we want to make this
function worse for performance, do we? It certainly sounds easier than
writing all the preempt macros in assembly though.

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Russell King - ARM Linux]

On Thu, Feb 02, 2012 at 03:36:49PM -0800, Stephen Boyd wrote:
> On 02/02/12 13:38, Nicolas Pitre wrote:
> > On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
> >> On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
> >>> Should we move get_thread_info into assembler.h? It seems odd
> >>> to include entry-header.S but I saw that vfp was doing the same.
> >> Probably yes, and probably also have preempt_disable and preempt_enable
> >> assembler macros.  That's going to get rather icky if we have to
> >> explicitly call the scheduler though (to solve (1)).
> > What about a pair of helpers written in C instead?
> >
> > v7_flush_dcache_all() could be renamed, and a wrapper function called 
> > v7_flush_dcache_all() would call the preemption disable helper, call the 
> > former v7_flush_dcache_all code, then call the preemption enable helper.
> >
> > Then __v7_setup() could still call the core cache flush code without 
> > issues.
> 
> I tried to put the preemption disable/enable right around the place
> where it was needed. With this approach we would disable preemption
> during the entire cache flush. I'm not sure if we want to make this
> function worse for performance, do we? It certainly sounds easier than
> writing all the preempt macros in assembly though.

Err, why do you think it's a big task?

preempt disable is a case of incrementing the thread preempt count, while
preempt enable is a case of decrementing it, testing for zero, if zero,
then checking whether TIF_NEED_RESCHED is set and calling a function.

If that's too much, then the simple method in assembly to quickly disable
preemption over a very few set of instructions is using mrs/msr and cpsid i.
That'll be far cheaper than fiddling about with preempt counters or
messing about with veneers in C code.

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Stephen Boyd]

On 02/02/12 16:36, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2012 at 03:36:49PM -0800, Stephen Boyd wrote:
>> On 02/02/12 13:38, Nicolas Pitre wrote:
>>> On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
>>>> On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
>>>>> Should we move get_thread_info into assembler.h? It seems odd
>>>>> to include entry-header.S but I saw that vfp was doing the same.
>>>> Probably yes, and probably also have preempt_disable and preempt_enable
>>>> assembler macros.  That's going to get rather icky if we have to
>>>> explicitly call the scheduler though (to solve (1)).
>>> What about a pair of helpers written in C instead?
>>>
>>> v7_flush_dcache_all() could be renamed, and a wrapper function called 
>>> v7_flush_dcache_all() would call the preemption disable helper, call the 
>>> former v7_flush_dcache_all code, then call the preemption enable helper.
>>>
>>> Then __v7_setup() could still call the core cache flush code without 
>>> issues.
>> I tried to put the preemption disable/enable right around the place
>> where it was needed. With this approach we would disable preemption
>> during the entire cache flush. I'm not sure if we want to make this
>> function worse for performance, do we? It certainly sounds easier than
>> writing all the preempt macros in assembly though.
> Err, why do you think it's a big task?
>
> preempt disable is a case of incrementing the thread preempt count, while
> preempt enable is a case of decrementing it, testing for zero, if zero,
> then checking whether TIF_NEED_RESCHED is set and calling a function.
>
> If that's too much, then the simple method in assembly to quickly disable
> preemption over a very few set of instructions is using mrs/msr and cpsid i.
> That'll be far cheaper than fiddling about with preempt counters or
> messing about with veneers in C code.

I'll try the macros. So far it isn't bad, just the __v7_setup to resolve.

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Nicolas Pitre]

On Fri, 3 Feb 2012, Russell King - ARM Linux wrote:

> On Thu, Feb 02, 2012 at 03:36:49PM -0800, Stephen Boyd wrote:
> > On 02/02/12 13:38, Nicolas Pitre wrote:
> > > What about a pair of helpers written in C instead?
> > >
> > > v7_flush_dcache_all() could be renamed, and a wrapper function called 
> > > v7_flush_dcache_all() would call the preemption disable helper, call the 
> > > former v7_flush_dcache_all code, then call the preemption enable helper.
> > >
> > > Then __v7_setup() could still call the core cache flush code without 
> > > issues.
> > 
> > I tried to put the preemption disable/enable right around the place
> > where it was needed. With this approach we would disable preemption
> > during the entire cache flush. I'm not sure if we want to make this
> > function worse for performance, do we? It certainly sounds easier than
> > writing all the preempt macros in assembly though.
> 
> Err, why do you think it's a big task?
> 
> preempt disable is a case of incrementing the thread preempt count, while
> preempt enable is a case of decrementing it, testing for zero, if zero,
> then checking whether TIF_NEED_RESCHED is set and calling a function.

Oh certainly.  And we already do just that in a few places already.  I 
re-read your previous email to realize that I initially misread your 
remark about the ickness of explicitly calling the scheduler.

> If that's too much, then the simple method in assembly to quickly disable
> preemption over a very few set of instructions is using mrs/msr and cpsid i.
> That'll be far cheaper than fiddling about with preempt counters or
> messing about with veneers in C code.

Indeed.  And I think that would be plenty sufficient here as the 
protected region is really short.  I don't think that warrants any 
macros.


Nicolas

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Nicolas Pitre]

On Thu, 2 Feb 2012, Stephen Boyd wrote:

> On 02/02/12 16:36, Russell King - ARM Linux wrote:
> > On Thu, Feb 02, 2012 at 03:36:49PM -0800, Stephen Boyd wrote:
> >> On 02/02/12 13:38, Nicolas Pitre wrote:
> >>> On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
> >>>> On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
> >>>>> Should we move get_thread_info into assembler.h? It seems odd
> >>>>> to include entry-header.S but I saw that vfp was doing the same.
> >>>> Probably yes, and probably also have preempt_disable and preempt_enable
> >>>> assembler macros.  That's going to get rather icky if we have to
> >>>> explicitly call the scheduler though (to solve (1)).
> >>> What about a pair of helpers written in C instead?
> >>>
> >>> v7_flush_dcache_all() could be renamed, and a wrapper function called 
> >>> v7_flush_dcache_all() would call the preemption disable helper, call the 
> >>> former v7_flush_dcache_all code, then call the preemption enable helper.
> >>>
> >>> Then __v7_setup() could still call the core cache flush code without 
> >>> issues.
> >> I tried to put the preemption disable/enable right around the place
> >> where it was needed. With this approach we would disable preemption
> >> during the entire cache flush. I'm not sure if we want to make this
> >> function worse for performance, do we? It certainly sounds easier than
> >> writing all the preempt macros in assembly though.
> > Err, why do you think it's a big task?
> >
> > preempt disable is a case of incrementing the thread preempt count, while
> > preempt enable is a case of decrementing it, testing for zero, if zero,
> > then checking whether TIF_NEED_RESCHED is set and calling a function.
> >
> > If that's too much, then the simple method in assembly to quickly disable
> > preemption over a very few set of instructions is using mrs/msr and cpsid i.
> > That'll be far cheaper than fiddling about with preempt counters or
> > messing about with veneers in C code.
> 
> I'll try the macros. So far it isn't bad, just the __v7_setup to resolve.

If you simply disable/restore IRQs around the critical region then you 
don't have to worry about __v7_setup.  Plus this will allow for 
v7_flush_dcache_all to still be callable from atomic context.


Nicolas

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Saravana Kannan]

On 02/02/2012 04:36 PM, Russell King - ARM Linux wrote:
> On Thu, Feb 02, 2012 at 03:36:49PM -0800, Stephen Boyd wrote:
>> On 02/02/12 13:38, Nicolas Pitre wrote:
>>> On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
>>>> On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
>>>>> Should we move get_thread_info into assembler.h? It seems odd
>>>>> to include entry-header.S but I saw that vfp was doing the same.
>>>> Probably yes, and probably also have preempt_disable and preempt_enable
>>>> assembler macros.  That's going to get rather icky if we have to
>>>> explicitly call the scheduler though (to solve (1)).
>>> What about a pair of helpers written in C instead?
>>>
>>> v7_flush_dcache_all() could be renamed, and a wrapper function called
>>> v7_flush_dcache_all() would call the preemption disable helper, call the
>>> former v7_flush_dcache_all code, then call the preemption enable helper.
>>>
>>> Then __v7_setup() could still call the core cache flush code without
>>> issues.
>>
>> I tried to put the preemption disable/enable right around the place
>> where it was needed. With this approach we would disable preemption
>> during the entire cache flush. I'm not sure if we want to make this
>> function worse for performance, do we? It certainly sounds easier than
>> writing all the preempt macros in assembly though.
>
> Err, why do you think it's a big task?
>
> preempt disable is a case of incrementing the thread preempt count, while
> preempt enable is a case of decrementing it, testing for zero, if zero,
> then checking whether TIF_NEED_RESCHED is set and calling a function.
>
> If that's too much, then the simple method in assembly to quickly disable
> preemption over a very few set of instructions is using mrs/msr and cpsid i.
> That'll be far cheaper than fiddling about with preempt counters or
> messing about with veneers in C code.

Russell,

I think you misunderstood Stephen's point about the performance. He 
isn't referring to the performance difference between a C call to preemt 
disable/enable vs. a few assembly level instructions.

I believe he is referring to the performance hit of having preemption 
disabled during the entirety of the cache flush operation vs. having 
preemption disabled only for the duration of writing to CSSELR and 
reading back CCSIDR.

I would think a cache flush is a fairly long operation and to have 
preemption disable across it doesn't sound appealing to me.

Thoughts?

Thanks,
Saravana

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: [PATCH] ARM: cache-v7: Disable preemption when reading CCSIDR

[Stephen Boyd]

On 02/06/12 19:34, Saravana Kannan wrote:
> On 02/02/2012 04:36 PM, Russell King - ARM Linux wrote:
>> On Thu, Feb 02, 2012 at 03:36:49PM -0800, Stephen Boyd wrote:
>>> On 02/02/12 13:38, Nicolas Pitre wrote:
>>>> On Thu, 2 Feb 2012, Russell King - ARM Linux wrote
>>>>> On Thu, Feb 02, 2012 at 11:24:46AM -0800, Stephen Boyd wrote:
>>>>>> Should we move get_thread_info into assembler.h? It seems odd
>>>>>> to include entry-header.S but I saw that vfp was doing the same.
>>>>> Probably yes, and probably also have preempt_disable and
>>>>> preempt_enable
>>>>> assembler macros.  That's going to get rather icky if we have to
>>>>> explicitly call the scheduler though (to solve (1)).
>>>> What about a pair of helpers written in C instead?
>>>>
>>>> v7_flush_dcache_all() could be renamed, and a wrapper function called
>>>> v7_flush_dcache_all() would call the preemption disable helper,
>>>> call the
>>>> former v7_flush_dcache_all code, then call the preemption enable
>>>> helper.
>>>>
>>>> Then __v7_setup() could still call the core cache flush code without
>>>> issues.
>>>
>>> I tried to put the preemption disable/enable right around the place
>>> where it was needed. With this approach we would disable preemption
>>> during the entire cache flush. I'm not sure if we want to make this
>>> function worse for performance, do we? It certainly sounds easier than
>>> writing all the preempt macros in assembly though.
>>
>> Err, why do you think it's a big task?
>>
>> preempt disable is a case of incrementing the thread preempt count,
>> while
>> preempt enable is a case of decrementing it, testing for zero, if zero,
>> then checking whether TIF_NEED_RESCHED is set and calling a function.
>>
>> If that's too much, then the simple method in assembly to quickly
>> disable
>> preemption over a very few set of instructions is using mrs/msr and
>> cpsid i.
>> That'll be far cheaper than fiddling about with preempt counters or
>> messing about with veneers in C code.
>
> Russell,
>
> I think you misunderstood Stephen's point about the performance. He
> isn't referring to the performance difference between a C call to
> preemt disable/enable vs. a few assembly level instructions.
>
> I believe he is referring to the performance hit of having preemption
> disabled during the entirety of the cache flush operation vs. having
> preemption disabled only for the duration of writing to CSSELR and
> reading back CCSIDR.
>
> I would think a cache flush is a fairly long operation and to have
> preemption disable across it doesn't sound appealing to me.
>
> Thoughts?
>

Sorry I messed up the headers for v2 of the patch. It didn't get sent to
the msm list.

Anyway, disabling interrupts for those few instructions sounds like the
best approach and so I sent that out in v2.

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.