[BUG] v7_coherent_kern_range broken on big.LITTLE

[BUG] v7_coherent_kern_range broken on big.LITTLE

[Jon Medhurst (Tixy)]

The function v7_coherent_kern_range uses the macro icache_line_size to
read the current CPUs icache line size for the purpose of invalidating
all cache lines in the given range.

Unfortunately, on the TC2 big.LITTLE test chip, the A15 icache line size
is 64 bytes, but the A7 size is only 32 bytes. So when the function
executes on the A15 it will miss out every alternate cache line for the
A7.

Presumably all the functions which use dcache_line_size are also
potentially buggy?

This problem was discovered whilst investigating why function tracer was
crashing in an unpredictable fashion. This was discussed previously but
the root cause misdiagnosed, see http://lkml.org/lkml/2012/12/6/261

-- 
Tixy


 

[BUG] v7_coherent_kern_range broken on big.LITTLE

[Russell King - ARM Linux]

On Thu, Feb 14, 2013 at 05:07:43PM +0000, Jon Medhurst (Tixy) wrote:
> The function v7_coherent_kern_range uses the macro icache_line_size to
> read the current CPUs icache line size for the purpose of invalidating
> all cache lines in the given range.
> 
> Unfortunately, on the TC2 big.LITTLE test chip, the A15 icache line size
> is 64 bytes, but the A7 size is only 32 bytes. So when the function
> executes on the A15 it will miss out every alternate cache line for the
> A7.

This sounds... brain dead.  So, we have a MP system with A15 and A7.  On
A15, the cache line size reported by the cache ID registers is 64 bytes.
On the A7, the cache line size is reported to be 32 bytes.

When a 64-byte cache line is invalidated on an A15 core, only one of the
two overlapping A7 cache lines gets invalidated.

So, to find out what cache line granularity we have to operate on, we have
to read the cache size registers on both CPUs, and calculate the result
from that.  That means we must boot _all_ cores and read their cache ID
registers at kernel boot time, and then shut down the cores we don't want.
Hardly a good idea for low power platforms...

[BUG] v7_coherent_kern_range broken on big.LITTLE

[Will Deacon]

Hi Tixy,

On Thu, Feb 14, 2013 at 05:07:43PM +0000, Jon Medhurst (Tixy) wrote:
> The function v7_coherent_kern_range uses the macro icache_line_size to
> read the current CPUs icache line size for the purpose of invalidating
> all cache lines in the given range.
> 
> Unfortunately, on the TC2 big.LITTLE test chip, the A15 icache line size
> is 64 bytes, but the A7 size is only 32 bytes. So when the function
> executes on the A15 it will miss out every alternate cache line for the
> A7.

There is a signal (IMINLN) to the core which allows A15 to behave as though
it has a 32-byte line size and this should be driven correctly for big/little.

Can you confirm that on your TC2 the I-line size is advertised differently
depending on the cluster? This really shouldn't be the case.

Will

[BUG] v7_coherent_kern_range broken on big.LITTLE

[Jon Medhurst (Tixy)]

On Thu, 2013-02-14 at 17:16 +0000, Will Deacon wrote:
> Hi Tixy,
> 
> On Thu, Feb 14, 2013 at 05:07:43PM +0000, Jon Medhurst (Tixy) wrote:
> > The function v7_coherent_kern_range uses the macro icache_line_size to
> > read the current CPUs icache line size for the purpose of invalidating
> > all cache lines in the given range.
> > 
> > Unfortunately, on the TC2 big.LITTLE test chip, the A15 icache line size
> > is 64 bytes, but the A7 size is only 32 bytes. So when the function
> > executes on the A15 it will miss out every alternate cache line for the
> > A7.
> 
> There is a signal (IMINLN) to the core which allows A15 to behave as though
> it has a 32-byte line size and this should be driven correctly for big/little.

How do we set that signal? Is that something we have to set up in Linux
or is it something that we expect the Firmware to set up?

> Can you confirm that on your TC2 the I-line size is advertised differently
> depending on the cluster?

Yes, I had printed out the value returned by icache_line_size when I had
ftrace_replace_code running on an A7 and on an A15.

The reason I suspected this was because I had tried all four
combinations of A15 and A7 setup, and it was only when booting on A15
with an A7 second CPU that enable function tracing kills the system.
A15+A15, A7+A15 and A7+A7 are OK.

To eliminate the kind of boot CPU as a factor I tried the A7+A15 system
with ftrace forced to run on the A15, and the bug stil occurred. And the
final straw was when hard coding the icache line size to 32 fixed
things.

-- 
Tixy

[BUG] v7_coherent_kern_range broken on big.LITTLE

[Lorenzo Pieralisi]

On Fri, Feb 15, 2013 at 10:04:37AM +0000, Jon Medhurst (Tixy) wrote:
> On Thu, 2013-02-14 at 17:16 +0000, Will Deacon wrote:
> > Hi Tixy,
> > 
> > On Thu, Feb 14, 2013 at 05:07:43PM +0000, Jon Medhurst (Tixy) wrote:
> > > The function v7_coherent_kern_range uses the macro icache_line_size to
> > > read the current CPUs icache line size for the purpose of invalidating
> > > all cache lines in the given range.
> > > 
> > > Unfortunately, on the TC2 big.LITTLE test chip, the A15 icache line size
> > > is 64 bytes, but the A7 size is only 32 bytes. So when the function
> > > executes on the A15 it will miss out every alternate cache line for the
> > > A7.
> > 
> > There is a signal (IMINLN) to the core which allows A15 to behave as though
> > it has a 32-byte line size and this should be driven correctly for big/little.
> 
> How do we set that signal? Is that something we have to set up in Linux
> or is it something that we expect the Firmware to set up?

If I am not mistaken, SCC register at offset 0x400 (bit 7) allows IMINLN to
be forced to 0 (ie Instruction Cache minimum line size == 32 bytes).

This can be done through board.txt so that it is set up as we want.

HTH,
Lorenzo

[BUG] v7_coherent_kern_range broken on big.LITTLE

[Jon Medhurst (Tixy)]

On Fri, 2013-02-15 at 10:33 +0000, Lorenzo Pieralisi wrote:
> On Fri, Feb 15, 2013 at 10:04:37AM +0000, Jon Medhurst (Tixy) wrote:
> > On Thu, 2013-02-14 at 17:16 +0000, Will Deacon wrote:
> > > Hi Tixy,
> > > 
> > > On Thu, Feb 14, 2013 at 05:07:43PM +0000, Jon Medhurst (Tixy) wrote:
> > > > The function v7_coherent_kern_range uses the macro icache_line_size to
> > > > read the current CPUs icache line size for the purpose of invalidating
> > > > all cache lines in the given range.
> > > > 
> > > > Unfortunately, on the TC2 big.LITTLE test chip, the A15 icache line size
> > > > is 64 bytes, but the A7 size is only 32 bytes. So when the function
> > > > executes on the A15 it will miss out every alternate cache line for the
> > > > A7.
> > > 
> > > There is a signal (IMINLN) to the core which allows A15 to behave as though
> > > it has a 32-byte line size and this should be driven correctly for big/little.
> > 
> > How do we set that signal? Is that something we have to set up in Linux
> > or is it something that we expect the Firmware to set up?
> 
> If I am not mistaken, SCC register at offset 0x400 (bit 7) allows IMINLN to
> be forced to 0 (ie Instruction Cache minimum line size == 32 bytes).
> 
> This can be done through board.txt so that it is set up as we want.

According to the TRM for TC2 the default value for that register is
0x33330c80, so adding the line "SCC: 0x400 0x33330c00" and incrementing
TOTALSCCS does the trick, and the A15's now report an icache size of 32.

We'll have to get everyone with a TC2 to make that change then?

As this is a TC2 issue, and not Linux related, shall we drop
linux-arm-kernel from all future replies to avoid the noise? I've added
linaro-dev to the cc list so this continues to get public visibility.

-- 
Tixy