l2c: Kernel panic in l2c310_enable() in non-secure mode

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Marc Gonzalez]

Hello everyone,

Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM

On my platform, Linux v4.2 running in non-secure mode panics in
l2c310_enable() with the pc pointing at this instruction:

c03390cc: ee013f30 mcr 15, 0, r3, cr1, cr0, {1}

which corresponds to set_auxcr IIUC.

static inline void set_auxcr(unsigned int val)
{
  asm volatile("mcr p15, 0, %0, c1, c0, 1  @ set AUXCR" : : "r" (val));
  isb();
}

which comes from this part in l2c310_enable:

	if (aux & L310_AUX_CTRL_FULL_LINE_ZERO) {
		set_auxcr(get_auxcr() | BIT(3) | BIT(2) | BIT(1));
		cpu_notifier(l2c310_cpu_enable_flz, 0);
	}


The L2C-310 documentation states:
> When the L2C-310 AXI slave ports receive a write transaction with
> AWUSERSx[10], it indicates that the write actually targets a whole
> cache line and that all data of this cache line must be reset to
> zero. The Cortex-A9 processor is likely to use this feature when a
> CPU is executing a memset routine to initialise a particular memory
> area. When the L2C-310 receives such a write transaction it ignores
> the AXI attributes attached to the transaction, size, length, data,
> and strobes for example, because the whole cache line must be reset.
> This behavior is not compatible with the AXI protocol, it is disabled
> by default. You can enable it by setting the Full Line of Zero Enable
> bit of the Auxiliary Control Register, bit[0]. This behavior also
> relies on an enable bit in the Cortex-A9 processor. You must take
> care if you enable this feature because correct behavior relies on
> consistent enabling in both the Cortex-A9 processor and the L2C-310.


According to the Cortex A9 documentation,
ACTLR is RO in non-secure mode if NSACR[18]=0 and RW if NSACR[18]=1

I suppose writing to a RO register cause the exception I see?


Commit 8abd259f657d5 ("l2c: provide generic hook to intercept
writes to secure registers") introduced a mechanism for non-secure
platforms to define how to write to the L2CC AUXCTRL register.

>    When Linux is running in the non-secure world, any write to a secure
>    L2C register will generate an abort.  Platforms normally have to call
>    firmware to work around this.  Provide a hook for them to intercept
>    any L2C secure register write.

Is there a similar mechanism for asking the firmware to write
to the CP15 ACTRL?

I suppose a work-around might be to set NSACR[18]?

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Marc Gonzalez]

On 14/10/2015 16:17, Marc Gonzalez wrote:

> Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM
> 
> On my platform, Linux v4.2 running in non-secure mode panics in
> l2c310_enable() with the pc pointing at this instruction:
> 
> c03390cc: ee013f30 mcr 15, 0, r3, cr1, cr0, {1}
> 
> which corresponds to set_auxcr IIUC.
> 
> static inline void set_auxcr(unsigned int val)
> {
>   asm volatile("mcr p15, 0, %0, c1, c0, 1  @ set AUXCR" : : "r" (val));
>   isb();
> }
> 
> which comes from this part in l2c310_enable:
> 
> 	if (aux & L310_AUX_CTRL_FULL_LINE_ZERO) {
> 		set_auxcr(get_auxcr() | BIT(3) | BIT(2) | BIT(1));
> 		cpu_notifier(l2c310_cpu_enable_flz, 0);
> 	}

Just to be sure, I changed set_auxcr() to a NOP. The kernel does not
panic with that setup, and I can see the boot messages:

[    0.000000] l2x0_of_init: FOO
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310: enabling full line of zeros but not enabled in Cortex-A9
[    0.000000] reg=0x104 val=0x66460801
[    0.000000] reg=0x100 val=0x1
[    0.000000] L2C-310 I prefetch enabled, offset 1 lines
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 8 ways, 512 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x62460801

The "reg=0x%x val=0x%lx\n" lines are from tango_l2c_write_sec()

It seems the firmware forgot to enable FLOZ.


> The L2C-310 documentation states:
>> When the L2C-310 AXI slave ports receive a write transaction with
>> AWUSERSx[10], it indicates that the write actually targets a whole
>> cache line and that all data of this cache line must be reset to
>> zero. The Cortex-A9 processor is likely to use this feature when a
>> CPU is executing a memset routine to initialise a particular memory
>> area. When the L2C-310 receives such a write transaction it ignores
>> the AXI attributes attached to the transaction, size, length, data,
>> and strobes for example, because the whole cache line must be reset.
>> This behavior is not compatible with the AXI protocol, it is disabled
>> by default. You can enable it by setting the Full Line of Zero Enable
>> bit of the Auxiliary Control Register, bit[0]. This behavior also
>> relies on an enable bit in the Cortex-A9 processor. You must take
>> care if you enable this feature because correct behavior relies on
>> consistent enabling in both the Cortex-A9 processor and the L2C-310.
> 
> 
> According to the Cortex A9 documentation,
> ACTLR is RO in non-secure mode if NSACR[18]=0 and RW if NSACR[18]=1
> 
> I suppose writing to a RO register cause the exception I see?
> 
> 
> Commit 8abd259f657d5 ("l2c: provide generic hook to intercept
> writes to secure registers") introduced a mechanism for non-secure
> platforms to define how to write to the L2CC AUXCTRL register.
> 
>>    When Linux is running in the non-secure world, any write to a secure
>>    L2C register will generate an abort.  Platforms normally have to call
>>    firmware to work around this.  Provide a hook for them to intercept
>>    any L2C secure register write.
> 
> Is there a similar mechanism for asking the firmware to write
> to the CP15 ACTRL?
> 
> I suppose a work-around might be to set NSACR[18]?

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Rob Herring]

On Wed, Oct 14, 2015 at 9:47 AM, Marc Gonzalez
<marc_gonzalez@sigmadesigns.com> wrote:
> On 14/10/2015 16:17, Marc Gonzalez wrote:
>
>> Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM
>>
>> On my platform, Linux v4.2 running in non-secure mode panics in
>> l2c310_enable() with the pc pointing at this instruction:
>>
>> c03390cc: ee013f30 mcr 15, 0, r3, cr1, cr0, {1}
>>
>> which corresponds to set_auxcr IIUC.
>>
>> static inline void set_auxcr(unsigned int val)
>> {
>>   asm volatile("mcr p15, 0, %0, c1, c0, 1  @ set AUXCR" : : "r" (val));
>>   isb();
>> }
>>
>> which comes from this part in l2c310_enable:
>>
>>       if (aux & L310_AUX_CTRL_FULL_LINE_ZERO) {
>>               set_auxcr(get_auxcr() | BIT(3) | BIT(2) | BIT(1));
>>               cpu_notifier(l2c310_cpu_enable_flz, 0);
>>       }
>
> Just to be sure, I changed set_auxcr() to a NOP. The kernel does not
> panic with that setup, and I can see the boot messages:
>
> [    0.000000] l2x0_of_init: FOO
> [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
> [    0.000000] L2C-310: enabling full line of zeros but not enabled in Cortex-A9
> [    0.000000] reg=0x104 val=0x66460801
> [    0.000000] reg=0x100 val=0x1
> [    0.000000] L2C-310 I prefetch enabled, offset 1 lines
> [    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
> [    0.000000] L2C-310 cache controller enabled, 8 ways, 512 kB
> [    0.000000] L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x62460801
>
> The "reg=0x%x val=0x%lx\n" lines are from tango_l2c_write_sec()
>
> It seems the firmware forgot to enable FLOZ.

Yes, FLOZ has to be enabled after enabling the L2 and disabled before
disabling the L2.

>> The L2C-310 documentation states:
>>> When the L2C-310 AXI slave ports receive a write transaction with
>>> AWUSERSx[10], it indicates that the write actually targets a whole
>>> cache line and that all data of this cache line must be reset to
>>> zero. The Cortex-A9 processor is likely to use this feature when a
>>> CPU is executing a memset routine to initialise a particular memory
>>> area. When the L2C-310 receives such a write transaction it ignores
>>> the AXI attributes attached to the transaction, size, length, data,
>>> and strobes for example, because the whole cache line must be reset.
>>> This behavior is not compatible with the AXI protocol, it is disabled
>>> by default. You can enable it by setting the Full Line of Zero Enable
>>> bit of the Auxiliary Control Register, bit[0]. This behavior also
>>> relies on an enable bit in the Cortex-A9 processor. You must take
>>> care if you enable this feature because correct behavior relies on
>>> consistent enabling in both the Cortex-A9 processor and the L2C-310.
>>
>>
>> According to the Cortex A9 documentation,
>> ACTLR is RO in non-secure mode if NSACR[18]=0 and RW if NSACR[18]=1
>>
>> I suppose writing to a RO register cause the exception I see?

Yes.

>> Commit 8abd259f657d5 ("l2c: provide generic hook to intercept
>> writes to secure registers") introduced a mechanism for non-secure
>> platforms to define how to write to the L2CC AUXCTRL register.
>>
>>>    When Linux is running in the non-secure world, any write to a secure
>>>    L2C register will generate an abort.  Platforms normally have to call
>>>    firmware to work around this.  Provide a hook for them to intercept
>>>    any L2C secure register write.
>>
>> Is there a similar mechanism for asking the firmware to write
>> to the CP15 ACTRL?

No.

>> I suppose a work-around might be to set NSACR[18]?

You may find you need that anyway for control of the SMP bit if you
shut off cores.

Rob

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Russell King - ARM Linux]

On Wed, Oct 14, 2015 at 04:17:43PM +0200, Marc Gonzalez wrote:
> Hello everyone,
> 
> Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM
> 
> On my platform, Linux v4.2 running in non-secure mode panics in
> l2c310_enable() with the pc pointing at this instruction:
> 
> c03390cc: ee013f30 mcr 15, 0, r3, cr1, cr0, {1}
> 
> which corresponds to set_auxcr IIUC.
...
> According to the Cortex A9 documentation,
> ACTLR is RO in non-secure mode if NSACR[18]=0 and RW if NSACR[18]=1
> 
> I suppose writing to a RO register cause the exception I see?

It should not, the write should be ignored and no exception should be
raised.

> Commit 8abd259f657d5 ("l2c: provide generic hook to intercept
> writes to secure registers") introduced a mechanism for non-secure
> platforms to define how to write to the L2CC AUXCTRL register.

Wrong register.  This is the _CPU_ auxiliary control register, not the
L2CC auxiliary control register.

> Is there a similar mechanism for asking the firmware to write
> to the CP15 ACTRL?

That's entirely dependent on the secure monitor implementation.  ARM Ltd
didn't listen to me originally when I said there needs to be a spec for
that, so there's no standardised way to talk to it, sorry.

Now, you've quoted one line from the oops, and a load of information that
we already know (because we have access to the manuals).  You've omitted
the rest of the oops, which is information we don't know, and is information
that we, as kernel developers, have decided that the kernel should print
to allow _us_, on the receiving end of an oops, to be able to diagnose
what happened and why.

Please, if you get an oops, include the _full_ dump when reporting
problems, even if you've diagnosed it already.  Not only does it help to
confirm the diagnosis, but it also serves as a source of documentation
if/when we commit a change to solve it.

Thanks.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Russell King - ARM Linux]

On Wed, Oct 14, 2015 at 04:47:59PM +0200, Marc Gonzalez wrote:
> Just to be sure, I changed set_auxcr() to a NOP. The kernel does not
> panic with that setup, and I can see the boot messages:
> 
> [    0.000000] l2x0_of_init: FOO
> [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
> [    0.000000] L2C-310: enabling full line of zeros but not enabled in Cortex-A9
> [    0.000000] reg=0x104 val=0x66460801
> [    0.000000] reg=0x100 val=0x1
> [    0.000000] L2C-310 I prefetch enabled, offset 1 lines
> [    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
> [    0.000000] L2C-310 cache controller enabled, 8 ways, 512 kB
> [    0.000000] L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x62460801
> 
> The "reg=0x%x val=0x%lx\n" lines are from tango_l2c_write_sec()
> 
> It seems the firmware forgot to enable FLOZ.

Wrong.  Do _not_ enable FLZ in the Cortex-A9.  FLZ needs the L2 cache
enabled _before_ the Cortex A9.  This is not something you can do in
firmware/boot loader/etc.  It has to be done by the kernel when the L2
cache is initialised.

If your firmware prevents this, sorry, you can't ever use FLZ.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Mason]

On 14/10/2015 19:45, Russell King - ARM Linux wrote:

> On Wed, Oct 14, 2015 at 04:17:43PM +0200, Marc Gonzalez wrote:
>
>> Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM
>>
>> On my platform, Linux v4.2 running in non-secure mode panics in
>> l2c310_enable() with the pc pointing at this instruction:
>>
>> c03390cc: ee013f30 mcr 15, 0, r3, cr1, cr0, {1}
>>
>> which corresponds to set_auxcr IIUC.
> ...
>> According to the Cortex A9 documentation,
>> ACTLR is RO in non-secure mode if NSACR[18]=0 and RW if NSACR[18]=1
>>
>> I suppose writing to a RO register cause the exception I see?
> 
> It should not, the write should be ignored and no exception should be
> raised.

Interesting.

>> Commit 8abd259f657d5 ("l2c: provide generic hook to intercept
>> writes to secure registers") introduced a mechanism for non-secure
>> platforms to define how to write to the L2CC AUXCTRL register.
> 
> Wrong register.  This is the _CPU_ auxiliary control register, not the
> L2CC auxiliary control register.

Right.

>> Is there a similar mechanism for asking the firmware to write
>> to the CP15 ACTRL?
> 
> That's entirely dependent on the secure monitor implementation.  ARM Ltd
> didn't listen to me originally when I said there needs to be a spec for
> that, so there's no standardised way to talk to it, sorry.

My question was ambiguous. I didn't mean to ask how to interact
with the firmware, but rather if there existed some way to prevent
Linux from just writing the CP15 ACTRL, and instead go through
a platform-specific function (as you provided for L2CC AUXCTRL).

> Now, you've quoted one line from the oops, and a load of information that
> we already know (because we have access to the manuals).  You've omitted
> the rest of the oops, which is information we don't know, and is information
> that we, as kernel developers, have decided that the kernel should print
> to allow _us_, on the receiving end of an oops, to be able to diagnose
> what happened and why.
> 
> Please, if you get an oops, include the _full_ dump when reporting
> problems, even if you've diagnosed it already.  Not only does it help to
> confirm the diagnosis, but it also serves as a source of documentation
> if/when we commit a change to solve it.

I didn't provide the full oops because I don't know how to enable
early_printk on my platform. My ad-hoc method is attaching a debugger,
stop in panic() and examine the __log_buf array.

Can I just dump the relevant part of the unformatted log?

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Peter Maydell]

On 14 October 2015 at 18:45, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Oct 14, 2015 at 04:17:43PM +0200, Marc Gonzalez wrote:
>> Hello everyone,
>>
>> Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM
>>
>> On my platform, Linux v4.2 running in non-secure mode panics in
>> l2c310_enable() with the pc pointing at this instruction:
>>
>> c03390cc: ee013f30 mcr 15, 0, r3, cr1, cr0, {1}
>>
>> which corresponds to set_auxcr IIUC.
> ...
>> According to the Cortex A9 documentation,
>> ACTLR is RO in non-secure mode if NSACR[18]=0 and RW if NSACR[18]=1
>>
>> I suppose writing to a RO register cause the exception I see?
>
> It should not, the write should be ignored and no exception should be
> raised.

My reading of the v7 ARM ARM ('Read-only and write-only encodings'
under section B3.15.2) is that writes via MCR to a register defined as 'RO'
are UNPREDICTABLE. (In v8 this is tightened up, and such write
attempts are UNDEFINED.)

More specifically for this case, the Cortex-A9 TRM description of
NSACR.NS_SMP says that if it is zero then "A write to Auxiliary Control
Register in Non-secure state takes an Undefined Instruction exception".
So the behaviour Marc reports is expected if NS_SMP is 0.

thanks
-- PMM

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Mason]

On 14/10/2015 19:47, Russell King - ARM Linux wrote:
> On Wed, Oct 14, 2015 at 04:47:59PM +0200, Marc Gonzalez wrote:
>> Just to be sure, I changed set_auxcr() to a NOP. The kernel does not
>> panic with that setup, and I can see the boot messages:
>>
>> [    0.000000] l2x0_of_init: FOO
>> [    0.000000] L2C-310 enabling early BRESP for Cortex-A9
>> [    0.000000] L2C-310: enabling full line of zeros but not enabled in Cortex-A9
>> [    0.000000] reg=0x104 val=0x66460801
>> [    0.000000] reg=0x100 val=0x1
>> [    0.000000] L2C-310 I prefetch enabled, offset 1 lines
>> [    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
>> [    0.000000] L2C-310 cache controller enabled, 8 ways, 512 kB
>> [    0.000000] L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x62460801
>>
>> The "reg=0x%x val=0x%lx\n" lines are from tango_l2c_write_sec()
>>
>> It seems the firmware forgot to enable FLOZ.
> 
> Wrong.  Do _not_ enable FLZ in the Cortex-A9.  FLZ needs the L2 cache
> enabled _before_ the Cortex A9.  This is not something you can do in
> firmware/boot loader/etc.  It has to be done by the kernel when the L2
> cache is initialised.
> 
> If your firmware prevents this, sorry, you can't ever use FLZ.

Is FLZ just a performance optimization?

I can ask the firmware author not to enable bit 0 in L2CC AUXCTRL.

Then Linux won't try to write CP15 ACTRL, IIUC.

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Russell King - ARM Linux]

On Wed, Oct 14, 2015 at 09:19:56PM +0100, Peter Maydell wrote:
> On 14 October 2015 at 18:45, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Wed, Oct 14, 2015 at 04:17:43PM +0200, Marc Gonzalez wrote:
> >> Hello everyone,
> >>
> >> Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM
> >>
> >> On my platform, Linux v4.2 running in non-secure mode panics in
> >> l2c310_enable() with the pc pointing at this instruction:
> >>
> >> c03390cc: ee013f30 mcr 15, 0, r3, cr1, cr0, {1}
> >>
> >> which corresponds to set_auxcr IIUC.
> > ...
> >> According to the Cortex A9 documentation,
> >> ACTLR is RO in non-secure mode if NSACR[18]=0 and RW if NSACR[18]=1
> >>
> >> I suppose writing to a RO register cause the exception I see?
> >
> > It should not, the write should be ignored and no exception should be
> > raised.
> 
> My reading of the v7 ARM ARM ('Read-only and write-only encodings'
> under section B3.15.2) is that writes via MCR to a register defined as 'RO'
> are UNPREDICTABLE. (In v8 this is tightened up, and such write
> attempts are UNDEFINED.)

Sorry, but this isn't the case - the ARM ARM defines it otherwise:

"B4.1.1 ACTLR, IMPLEMENTATION DEFINED Auxiliary Control Register, VMSA
The ACTLR characteristics are:
The ACTLR provides IMPLEMENTATION DEFINED configuration and control options.

...
Configurations

If the implementation includes the Security Extensions, this register
is Banked. However, some bits might define global configuration settings,
and be common to the Secure and Non-secure copies of the register."

> More specifically for this case, the Cortex-A9 TRM description of
> NSACR.NS_SMP says that if it is zero then "A write to Auxiliary Control
> Register in Non-secure state takes an Undefined Instruction exception".
> So the behaviour Marc reports is expected if NS_SMP is 0.

Well, if that's what the CA9 TRM says, that's the behaviour that the
device has, which is contary to the ARM ARM.

If the ACTLR is not writable, then FLZ can't be enabled on the platform.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Marc Gonzalez]

On 14/10/2015 19:45, Russell King - ARM Linux wrote:

> Now, you've quoted one line from the oops, and a load of information that
> we already know (because we have access to the manuals).  You've omitted
> the rest of the oops, which is information we don't know, and is information
> that we, as kernel developers, have decided that the kernel should print
> to allow _us_, on the receiving end of an oops, to be able to diagnose
> what happened and why.
> 
> Please, if you get an oops, include the _full_ dump when reporting
> problems, even if you've diagnosed it already.  Not only does it help to
> confirm the diagnosis, but it also serves as a source of documentation
> if/when we commit a change to solve it.

Here are the relevant parts of the boot log.

old_aux = 0x62460801
L2C-310 enabling early BRESP for Cortex-A9
L2C-310: enabling full line of zeros but not enabled in Cortex-A9
reg=0x104 val=0x66460801
reg=0x100 val=0x1
L2C-310 I prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP ARM
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.2.0+ #259
Hardware name: Sigma Tango DT
task: c035ff28 ti: c035c000 task.ti: c035c000
PC is at l2c310_enable+0x1cc/0x21c
LR is at console_unlock+0x25c/0x554
pc : [<c0339258>]    lr : [<c00606b8>]    psr: 200000d3
sp : c035deb8  ip : c035dda8  fp : c035dee4
r10: c0371694  r9 : 00000000  r8 : 00000000
r7 : c0371694  r6 : e8806000  r5 : 00000008  r4 : 62460801
r3 : 0000004f  r2 : 00000001  r1 : 600000d3  r0 : 0000003a
Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel
Control: 10c5387d  Table: 8000404a  DAC: 00000015
Process swapper/0 (pid: 0, stack limit = 0xc035c218)
Stack: (0xc035deb8 to 0xc035e000)
dea0:                                                       00000000 00000008
dec0: 00000048 c0371694 c0355c28 410000c8 62460801 00000008 c035df4c c035dee8
dee0: c0339be8 c0339098 e7af5cf4 00000000 c033900c c035df1c c0339810 c0023578
df00: c00235ec c0023640 c00240b8 c0023870 c0023694 c0024038 c0024430 00000000
df20: 00000000 c0371694 e7af5cf4 62460801 c0355c28 c0371280 c03535e0 e7af5100
df40: c035df94 c035df50 c0339ee4 c0339a2c ffffffff 00000000 00000002 20100000
df60: 20100fff e7af5d4c 00000200 00000000 00000000 00000000 c03535d0 c0371280
df80: c035e400 ffffffff c035dfac c035df98 c0334910 c0339ce8 00000000 c0371280
dfa0: c035dff4 c035dfb0 c0333bd8 c03348ac ffffffff ffffffff 00000000 c03336d8
dfc0: 00000000 c03535e0 00000000 c0371514 c035e480 c03535dc c0360fec 8000406a
dfe0: 413fc090 00000000 00000000 c035dff8 8000807c c0333964 00000000 00000000
Backtrace: 
[<c033908c>] (l2c310_enable) from [<c0339be8>] (__l2c_init+0x1c8/0x248)
 r8:00000008 r7:62460801 r6:410000c8 r5:c0355c28 r4:c0371694
[<c0339a20>] (__l2c_init) from [<c0339ee4>] (l2x0_of_init+0x208/0x218)
 r10:e7af5100 r9:c03535e0 r8:c0371280 r7:c0355c28 r6:62460801 r5:e7af5cf4
 r4:c0371694
[<c0339cdc>] (l2x0_of_init) from [<c0334910>] (init_IRQ+0x70/0x88)
 r7:ffffffff r6:c035e400 r5:c0371280 r4:c03535d0
[<c03348a0>] (init_IRQ) from [<c0333bd8>] (start_kernel+0x280/0x368)
 r5:c0371280 r4:00000000
[<c0333958>] (start_kernel) from [<8000807c>] (0x8000807c)
 r10:00000000 r9:413fc090 r8:8000406a r7:c0360fec r6:c03535dc r5:c035e480
 r4:c0371514
Code: e3140001 0a000012 ee113f30 e383300e (ee013f30) 
---[ end trace cb88537fdc8fa200 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
---[ end Kernel panic - not syncing: Attempted to kill the idle task!


And here's the relevant objdump:

c0339220:       e31c0002        tst     ip, #2
c0339224:       e34c2031        movt    r2, #49201      ; 0xc031
c0339228:       e34c302f        movt    r3, #49199      ; 0xc02f
c033922c:       e30e00b8        movw    r0, #57528      ; 0xe0b8
c0339230:       11a01002        movne   r1, r2
c0339234:       01a01003        moveq   r1, r3
c0339238:       e31c0001        tst     ip, #1
c033923c:       e34c002f        movt    r0, #49199      ; 0xc02f
c0339240:       01a02003        moveq   r2, r3
c0339244:       ebfcf72d        bl      c0276f00 <printk>
c0339248:       e3140001        tst     r4, #1
c033924c:       0a000012        beq     c033929c <l2c310_enable+0x210>
c0339250:       ee113f30        mrc     15, 0, r3, cr1, cr0, {1}
c0339254:       e383300e        orr     r3, r3, #14
c0339258:       ee013f30        mcr     15, 0, r3, cr1, cr0, {1}
c033925c:       f57ff06f        isb     sy
c0339260:       e59f003c        ldr     r0, [pc, #60]   ; c03392a4 <l2c310_enable+0x218>
c0339264:       ebfcf241        bl      c0275b70 <register_cpu_notifier>
c0339268:       ea00000b        b       c033929c <l2c310_enable+0x210>

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Marc Gonzalez]

On 14/10/2015 19:06, Rob Herring wrote:

> Yes, FLOZ has to be enabled after enabling the L2 and disabled before
> disabling the L2.

Is FLOZ merely a performance optimization?

Have you (or someone else) measured the impact of having it enabled vs disabled?

>> I suppose a work-around might be to set NSACR[18]?
> 
> You may find you need that anyway for control of the SMP bit if you
> shut off cores.

NSACR[18] == NSACR.NS_SMP

Usage constraints
The ACTLR is: [...] RW in Non-secure state if NSACR.NS_SMP = 1.
In this case all bits are Write Ignore except for the SMP bit.

Thus, even if my firmware sets NSACR.NS_SMP, Linux won't be able to
set bits 1,2,3 in ACTLR.

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Russell King - ARM Linux]

On Thu, Oct 15, 2015 at 10:56:50AM +0200, Marc Gonzalez wrote:
> On 14/10/2015 19:06, Rob Herring wrote:
> 
> > Yes, FLOZ has to be enabled after enabling the L2 and disabled before
> > disabling the L2.
> 
> Is FLOZ merely a performance optimization?
> 
> Have you (or someone else) measured the impact of having it enabled vs
> disabled?

FLZ is a performance optimisation.  It allows the Cortex-A9 to indicate
to a L2 cache that an entire cache line should be zeroed, rather than
having the Cortex A9 write each word with zero.  As this uses non-AXI
compliant signalling, it needs to be enabled in two locations.

The enable bit in the L2 cache enables reception of this signalling.
This will only have an effect when both the L2 FLZ enable bit _and_
the L2 cache are both enabled, so the signalling can only be interpreted
by an appropriately configured and enabled L2 cache controller.

After that, the Cortex A9 FLZ enable bit can be set to allow the CA9 to
generate the non-standard signalling.  This enables the CA9 to generate
the non-standard signalling, and stops it generating the individual word
writes.

Enabling the CA9 first results in the non-standard signalling being used
but nothing on the bus can interpret it, which causes data corruption.

So, performance wise it's beneficial for zero initialising memory.

If you can't write to the Cortex A9 auxiliary control register, then you
can't use this feature, and discussing the performance impact of having
the feature disabled is irrelevant.

> >> I suppose a work-around might be to set NSACR[18]?
> > 
> > You may find you need that anyway for control of the SMP bit if you
> > shut off cores.
> 
> NSACR[18] == NSACR.NS_SMP
> 
> Usage constraints
> The ACTLR is: [...] RW in Non-secure state if NSACR.NS_SMP = 1.
> In this case all bits are Write Ignore except for the SMP bit.
> 
> Thus, even if my firmware sets NSACR.NS_SMP, Linux won't be able to
> set bits 1,2,3 in ACTLR.

However, at _least_ the ACTLR write won't fault.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Marc Gonzalez]

On 14/10/2015 19:47, Russell King - ARM Linux wrote:

> Wrong.  Do _not_ enable FLZ in the Cortex-A9.  FLZ needs the L2 cache
> enabled _before_ the Cortex A9.  This is not something you can do in
> firmware/boot loader/etc.  It has to be done by the kernel when the L2
> cache is initialised.

Can you tell me if the following setup is reasonable?

(The important thing to note is that I do have control over the firmware.)

l2c_enable and l2c_disable will hand control over to the firmware,
which will handle the details of FLOZ:

When enabling the L2, firmware will:
A1) set L2CC_reg1_aux_control[0] to 1
A2) enable the L2
A3) set CP15_ACTLR bits 1,2,3 to 1

When disabling the L2, firmware will:
B1) clear CP15_ACTLR bits 1,2,3 to 0
B2) disable the L2
B3) clear L2CC_reg1_aux_control[0] to 0

Thus, on my platform, Linux would no longer need to write CP15_ACTLR.

Then I just need a platform-specific bool e.g. "firmware_handles_cp15_actlr"
and check it where appropriate:

In enable:
	if (!firmware_handles_cp15_actlr && (aux & L310_AUX_CTRL_FULL_LINE_ZERO)) {
		set_auxcr(get_auxcr() | BIT(3) | BIT(2) | BIT(1));
		cpu_notifier(l2c310_cpu_enable_flz, 0);
	}

In disable:
	if (!firmware_handles_cp15_actlr && (l2x0_saved_regs.aux_ctrl & L310_AUX_CTRL_FULL_LINE_ZERO))
		set_auxcr(get_auxcr() & ~(BIT(3) | BIT(2) | BIT(1)));


What do you think?

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Marc Gonzalez]

On 15/10/2015 12:00, Marc Gonzalez wrote:

> On 14/10/2015 19:47, Russell King - ARM Linux wrote:
> 
>> Wrong.  Do _not_ enable FLZ in the Cortex-A9.  FLZ needs the L2 cache
>> enabled _before_ the Cortex A9.  This is not something you can do in
>> firmware/boot loader/etc.  It has to be done by the kernel when the L2
>> cache is initialised.
> 
> Can you tell me if the following setup is reasonable?
> 
> (The important thing to note is that I do have control over the firmware.)
> 
> l2c_enable and l2c_disable will hand control over to the firmware,
> which will handle the details of FLOZ:
> 
> When enabling the L2, firmware will:
> A1) set L2CC_reg1_aux_control[0] to 1
> A2) enable the L2
> A3) set CP15_ACTLR bits 1,2,3 to 1
> 
> When disabling the L2, firmware will:
> B1) clear CP15_ACTLR bits 1,2,3 to 0
> B2) disable the L2
> B3) clear L2CC_reg1_aux_control[0] to 0
> 
> Thus, on my platform, Linux would no longer need to write CP15_ACTLR.
> 
> Then I just need a platform-specific bool e.g. "firmware_handles_cp15_actlr"
> and check it where appropriate:
> 
> In enable:
> 	if (!firmware_handles_cp15_actlr && (aux & L310_AUX_CTRL_FULL_LINE_ZERO)) {
> 		set_auxcr(get_auxcr() | BIT(3) | BIT(2) | BIT(1));
> 		cpu_notifier(l2c310_cpu_enable_flz, 0);
> 	}
> 
> In disable:
> 	if (!firmware_handles_cp15_actlr && (l2x0_saved_regs.aux_ctrl & L310_AUX_CTRL_FULL_LINE_ZERO))
> 		set_auxcr(get_auxcr() & ~(BIT(3) | BIT(2) | BIT(1)));
> 
> 
> What do you think?

Come to think of it, instead of introducing this extra bool, I can
just ask the firmware author to set NSACR.NS_SMP, which effectively
turns the above set_auxcr() calls into NOPs. (And Rob noted that I'll
need that bit set anyway.)

That way, no changes are required in Linux, and everything magically
works as expected. I'll file an internal bug report.

One more question on this topic:

When Linux sees L310_AUX_CTRL_FULL_LINE_ZERO set, it then enables:

  bit3 = Write full line of zeros mode
  bit2 = L1 Prefetch enable
  bit1 = L2 Prefetch hint enable

Why are the "prefetch enable" features bundled in the operation?

Regards.

l2c: Kernel panic in l2c310_enable() in non-secure mode

[Mason]

On 15/10/2015 13:07, Marc Gonzalez wrote:

> One more question on this topic:
> 
> When Linux sees L310_AUX_CTRL_FULL_LINE_ZERO set, it then enables:
> 
>   bit3 = Write full line of zeros mode
>   bit2 = L1 Prefetch enable
>   bit1 = L2 Prefetch hint enable
> 
> Why are the "prefetch enable" features bundled in the operation?

Russell,

The prefetch enable bits were introduced in commit 8ef418c7178fa
("ARM: l2c: trial at enabling some Cortex-A9 optimisations")

I didn't find a ML thread discussing that patch. Whenever possible,
could you provide some insight into the rationale leading you to
set bits 1 and 2 along with bit 3? (The ARM integrator here tends
to think the two concepts are somewhat orthogonal.)

Regards.