facing undefined inconsistent cache issues on cortex-a9

facing undefined inconsistent cache issues on cortex-a9

[Shiraz HASHIM]

Hi,
I am using linux-2.6.32 port on spear1300 platform, the port is largely based
on realview code.

SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in
SMP, but we are currently configuring Linux in non SMP mode, running Linux on
the first core only, while second core is spinning  forever in a tight loop.

Linux is running with default configuration w.r.t. cache and other things on
core1. Since SMP is not enabled this means that SCU and local timers are off, L2
cache is also disabled.  While on core2 only instruction cache is on and both
MMU and Data cache is off.

We observed a crash while booting. Investigating on the crash we found that this
was caused by a wrong value read by the core.  Strangely enough, the new value
was written few instruction before, but the core was still reading the old one.

What happens is someting like this:

r0 = X ; /* X is the address of a Linux variable */

str r5, [r0]; /* write new value */
...  
ldr r5, [r0]; /* read back value - we get the old one (???) */

The value read here, just few lines later the new value was written, is the old
one. Obviously we are expecting to find the new one.

Moreover, using the jtag debugger it looks clear that the new value is indeed in
the memory, but surprisingly NOT in the data-cache. 
If we put a read of X, before the store in order to avoid the write miss when
doing 'str r5, [r0]', we see that the problem doesn't appear.

We are completely lost, any pointers would really help, thanks

regards
Shiraz

facing undefined inconsistent cache issues on cortex-a9

[Shilimkar, Santosh]

> -----Original Message-----
> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
> bounces at lists.infradead.org] On Behalf Of Shiraz HASHIM
> Sent: Monday, May 24, 2010 9:14 AM
> To: linux-arm-kernel at lists.infradead.org
> Cc: 'Armando VISCONTI'; Vipin KUMAR; Viresh KUMAR
> Subject: facing undefined inconsistent cache issues on cortex-a9
> 
> Hi,
> I am using linux-2.6.32 port on spear1300 platform, the port is largely based
> on realview code.
> 
> SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in
> SMP, but we are currently configuring Linux in non SMP mode, running Linux on
> the first core only, while second core is spinning  forever in a tight loop.
> 
> Linux is running with default configuration w.r.t. cache and other things on
> core1. Since SMP is not enabled this means that SCU and local timers are off, L2
> cache is also disabled.  While on core2 only instruction cache is on and both
> MMU and Data cache is off.
> 
> We observed a crash while booting. Investigating on the crash we found that this
> was caused by a wrong value read by the core.  Strangely enough, the new value
> was written few instruction before, but the core was still reading the old one.
> 
> What happens is someting like this:
> 
> r0 = X ; /* X is the address of a Linux variable */
> 
> str r5, [r0]; /* write new value */
> ...
> ldr r5, [r0]; /* read back value - we get the old one (???) */
> 
> The value read here, just few lines later the new value was written, is the old
> one. Obviously we are expecting to find the new one.
> 
> Moreover, using the jtag debugger it looks clear that the new value is indeed in
> the memory, but surprisingly NOT in the data-cache.
> If we put a read of X, before the store in order to avoid the write miss when
> doing 'str r5, [r0]', we see that the problem doesn't appear.
> 
Does this work if disable L1D as well on the processor you are running Linux ??

Regards,
Santosh

facing undefined inconsistent cache issues on cortex-a9

[Armando VISCONTI]

Shilimkar, Santosh wrote:
>> -----Original Message-----
>> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
>> bounces at lists.infradead.org] On Behalf Of Shiraz HASHIM
>> Sent: Monday, May 24, 2010 9:14 AM
>> To: linux-arm-kernel at lists.infradead.org
>> Cc: 'Armando VISCONTI'; Vipin KUMAR; Viresh KUMAR
>> Subject: facing undefined inconsistent cache issues on cortex-a9
>>
>> Hi,
>> I am using linux-2.6.32 port on spear1300 platform, the port is largely based
>> on realview code.
>>
>> SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in
>> SMP, but we are currently configuring Linux in non SMP mode, running Linux on
>> the first core only, while second core is spinning  forever in a tight loop.
>>
>> Linux is running with default configuration w.r.t. cache and other things on
>> core1. Since SMP is not enabled this means that SCU and local timers are off, L2
>> cache is also disabled.  While on core2 only instruction cache is on and both
>> MMU and Data cache is off.
>>
>> We observed a crash while booting. Investigating on the crash we found that this
>> was caused by a wrong value read by the core.  Strangely enough, the new value
>> was written few instruction before, but the core was still reading the old one.
>>
>> What happens is someting like this:
>>
>> r0 = X ; /* X is the address of a Linux variable */
>>
>> str r5, [r0]; /* write new value */
>> ...
>> ldr r5, [r0]; /* read back value - we get the old one (???) */
>>
>> The value read here, just few lines later the new value was written, is the old
>> one. Obviously we are expecting to find the new one.
>>
>> Moreover, using the jtag debugger it looks clear that the new value is indeed in
>> the memory, but surprisingly NOT in the data-cache.
>> If we put a read of X, before the store in order to avoid the write miss when
>> doing 'str r5, [r0]', we see that the problem doesn't appear.
>>
>>     
> Does this work if disable L1D as well on the processor you are running Linux ??
>   
If we disable the L1 DCache we are never able to boot.
But this might be a different problem.

If we disable ICache and leave DCache on it is much more stable, and
I think it might be related to timing issue in the data path.

We are using Cortex A9 r2p1.

Any clue?

Arm



-- 
-- "Every step appears to be the unavoidable consequence of the
-- preceding one." (A. Einstein) 
-- 
Armando Visconti                  Mobile: (+39) 346 8879146
Senior SW Engineer                Fax:    (+39) 02 93519290
CPG                               Work:   (+39) 02 93519683
Computer System Division          e-mail: armando.visconti at st.com
ST Microelectronics               TINA:   051  4683
                                    
 

facing undefined inconsistent cache issues on cortex-a9

[Shilimkar, Santosh]

> -----Original Message-----
> From: Armando VISCONTI [mailto:armando.visconti at st.com]
> Sent: Tuesday, May 25, 2010 5:38 PM
> To: Shilimkar, Santosh
> Cc: Shiraz HASHIM; linux-arm-kernel at lists.infradead.org; Vipin KUMAR; Viresh KUMAR
> Subject: Re: facing undefined inconsistent cache issues on cortex-a9
> 
> Shilimkar, Santosh wrote:
> >> -----Original Message-----
> >> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
> >> bounces at lists.infradead.org] On Behalf Of Shiraz HASHIM
> >> Sent: Monday, May 24, 2010 9:14 AM
> >> To: linux-arm-kernel at lists.infradead.org
> >> Cc: 'Armando VISCONTI'; Vipin KUMAR; Viresh KUMAR
> >> Subject: facing undefined inconsistent cache issues on cortex-a9
> >>
> >> Hi,
> >> I am using linux-2.6.32 port on spear1300 platform, the port is largely based
> >> on realview code.
> >>
> >> SPEAr1300 has a Cortex A9 dual core, both cores configured as participating in
> >> SMP, but we are currently configuring Linux in non SMP mode, running Linux on
> >> the first core only, while second core is spinning  forever in a tight loop.
> >>
> >> Linux is running with default configuration w.r.t. cache and other things on
> >> core1. Since SMP is not enabled this means that SCU and local timers are off, L2
> >> cache is also disabled.  While on core2 only instruction cache is on and both
> >> MMU and Data cache is off.
> >>
> >> We observed a crash while booting. Investigating on the crash we found that this
> >> was caused by a wrong value read by the core.  Strangely enough, the new value
> >> was written few instruction before, but the core was still reading the old one.
> >>
> >> What happens is someting like this:
> >>
> >> r0 = X ; /* X is the address of a Linux variable */
> >>
> >> str r5, [r0]; /* write new value */
> >> ...
> >> ldr r5, [r0]; /* read back value - we get the old one (???) */
> >>
> >> The value read here, just few lines later the new value was written, is the old
> >> one. Obviously we are expecting to find the new one.
> >>
> >> Moreover, using the jtag debugger it looks clear that the new value is indeed in
> >> the memory, but surprisingly NOT in the data-cache.
> >> If we put a read of X, before the store in order to avoid the write miss when
> >> doing 'str r5, [r0]', we see that the problem doesn't appear.
> >>
> >>
> > Does this work if disable L1D as well on the processor you are running Linux ??
> >
> If we disable the L1 DCache we are never able to boot.
> But this might be a different problem.
> 
> If we disable ICache and leave DCache on it is much more stable, and
> I think it might be related to timing issue in the data path.
> 
For me, it seems that Cache isn't invalidated properly which leads to this
issue.
May be you can report the complete crash-log which can give more idea on the issue.

Regards,
Santosh

facing undefined inconsistent cache issues on cortex-a9

[Woodruff, Richard]

> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
> bounces at lists.infradead.org] On Behalf Of Shilimkar, Santosh

> > > Does this work if disable L1D as well on the processor you are running
> Linux ??
> > >
> > If we disable the L1 DCache we are never able to boot.
> > But this might be a different problem.
> >
> > If we disable ICache and leave DCache on it is much more stable, and
> > I think it might be related to timing issue in the data path.
> >
> For me, it seems that Cache isn't invalidated properly which leads to this
> issue.
> May be you can report the complete crash-log which can give more idea on the
> issue.

If you are unable to boot with dcache off then maybe your memory controller or memory part has timing issues.  Cache on will trigger burst accesses using different timings.  Such an early failure is more likely an issue outside of the arm unless you are doing something really odd.

Regards,
Richard W.

facing undefined inconsistent cache issues on cortex-a9

[Shiraz HASHIM]

Hello Woodruff,

On 5/26/2010 12:27 AM, Woodruff, Richard wrote:
> 
>> From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel-
>> bounces at lists.infradead.org] On Behalf Of Shilimkar, Santosh
> 
>>>> Does this work if disable L1D as well on the processor you are running
>> Linux ??
>>>>
>>> If we disable the L1 DCache we are never able to boot.
>>> But this might be a different problem.
>>>
>>> If we disable ICache and leave DCache on it is much more stable, and
>>> I think it might be related to timing issue in the data path.
>>>
>> For me, it seems that Cache isn't invalidated properly which leads to this
>> issue.
>> May be you can report the complete crash-log which can give more idea on the
>> issue.
> 
> If you are unable to boot with dcache off then maybe your memory controller or memory part has timing issues.  Cache on will trigger burst accesses using different timings.  Such an early failure is more likely an issue outside of the arm unless you are doing something really odd.

thanks for your inputs. We have some updates in this regard.
In our architecture we have 2 ports coming out of the A9SM (multi core cortex)
block to the interconnect matrix and the transfers are distributed between them.

When we used the address filtering option of SCU to divide and mutually exclude
the traffic between port0 and port1 of A9SM(multi core cortex) block. Like, 
assigning 1 half of the address space to port 1 and rest goes through port0.
This solved the problem with the same set of cache and other system
configurations.
We are not using L2 cache and it is kept disabled. The SCU also is only used for
address filtering and its synchronisation function is disabled (as we are not
bringing up the second core)
Now, we are not able to understand what it means. Do we need to divide the
traffic between ports as described and it is normal. Or there is some problem in
the way the transfers are handled in the system.

regards
Shiraz

facing undefined inconsistent cache issues on cortex-a9

[Woodruff, Richard]

> From: Shiraz HASHIM [mailto:shiraz.hashim at st.com]
> Sent: Friday, June 04, 2010 8:55 AM
> To: Woodruff, Richard

> thanks for your inputs. We have some updates in this regard.
> In our architecture we have 2 ports coming out of the A9SM (multi core cortex)
> block to the interconnect matrix and the transfers are distributed between
> them.
>
> When we used the address filtering option of SCU to divide and mutually
> exclude
> the traffic between port0 and port1 of A9SM(multi core cortex) block. Like,
> assigning 1 half of the address space to port 1 and rest goes through port0.
> This solved the problem with the same set of cache and other system
> configurations.
> We are not using L2 cache and it is kept disabled. The SCU also is only used
> for
> address filtering and its synchronisation function is disabled (as we are not
> bringing up the second core)
> Now, we are not able to understand what it means. Do we need to divide the
> traffic between ports as described and it is normal. Or there is some problem
> in
> the way the transfers are handled in the system.

Off hand I'm not sure of the right answer.  The design I'm working on does not require the filtering feature so I've not looked much at it outside of some errata.  I am aware of another design which accesses a bit larger address space which does use filtering.

I'd guess you need to understand your cortex-memmap + AXI-to-memory bus adapter characteristics.

Regards,
Richard W.