OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

+ linux-arm, Russell and Catalin

On Mon, Jan 16, 2012 at 11:03 AM, Joe Woodward <jw@terrafix.co.uk> wrote:
> The latest uBoot release (2011.12) disables the L2/outer cache during boot on OMAP boards.
>
> uBoot commit: "armv7: disable L2 cache in cleanup_before_linux()" on 6th Dec 2011 by Aneesh V adds the following to
> uBootSources/arch/arm/cpu/armv7/cpu.c:cleanup_before_linux():
> ?...
> ?v7_out_cache_disable();
>
> The commit message implies this change was to make booting reliable on OMAP4 by disabling L2 cache before jumping to Linux. Sadly
> this commit also affects OMAP3-based boards...
>
> Since switching to uBoot 2011.12 I'm seeing about half the performance on my GUSMTIX Overo (OMAP3530).
>
> In the kernel sources there seem to be OMAP4 specific functions for re-enabling the L2 cache (omap4-common.c:omap_l2_cache_init()),
> but none for OMAP3.
>
> So my question is... do we expect that Linux should be re-enabling the L2/outer cache during boot after uBoot has disabled it on OMAP3?
>
Yes. Kernel already does enabled L2 on A8 CPU's but OMAP3 needs a
secure call and then the patching becomes
that much difficult.

We need some kind of generic hook in proc*_setup() function where in
such secure bits can be turned ON.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Mon, Jan 16, 2012 at 11:18:21AM +0100, Shilimkar, Santosh wrote:
> + linux-arm, Russell and Catalin
> 
> On Mon, Jan 16, 2012 at 11:03 AM, Joe Woodward <jw@terrafix.co.uk> wrote:
> > The latest uBoot release (2011.12) disables the L2/outer cache during boot on OMAP boards.
> >
> > uBoot commit: "armv7: disable L2 cache in cleanup_before_linux()" on 6th Dec 2011 by Aneesh V adds the following to
> > uBootSources/arch/arm/cpu/armv7/cpu.c:cleanup_before_linux():
> > ?...
> > ?v7_out_cache_disable();
> >
> > The commit message implies this change was to make booting reliable on OMAP4 by disabling L2 cache before jumping to Linux. Sadly
> > this commit also affects OMAP3-based boards...
> >
> > Since switching to uBoot 2011.12 I'm seeing about half the performance on my GUSMTIX Overo (OMAP3530).
> >
> > In the kernel sources there seem to be OMAP4 specific functions for re-enabling the L2 cache (omap4-common.c:omap_l2_cache_init()),
> > but none for OMAP3.
> >
> > So my question is... do we expect that Linux should be re-enabling the L2/outer cache during boot after uBoot has disabled it on OMAP3?
> >
> Yes. Kernel already does enabled L2 on A8 CPU's but OMAP3 needs a
> secure call and then the patching becomes
> that much difficult.
> 
> We need some kind of generic hook in proc*_setup() function where in
> such secure bits can be turned ON.

You can't have that kind of hook: there is no stack here to call other
functions, plus we aren't running with the MMU on at that point so
only carefully crafted assembly could be used.

Why is it soo difficult to issue the secure call?

This just proves my point that the lack of consistent secure monitor API
for these kinds of basic service _are_ a big problem, one which ARM Ltd
should have addressed when they invented this stuff.  My view continues
to be that the secure monitor API is a total abortion, one which should
never have been allowed to happen.

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Mon, Jan 16, 2012 at 11:59 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Mon, Jan 16, 2012 at 11:18:21AM +0100, Shilimkar, Santosh wrote:
>> + linux-arm, Russell and Catalin
>>
>> On Mon, Jan 16, 2012 at 11:03 AM, Joe Woodward <jw@terrafix.co.uk> wrote:
>> > The latest uBoot release (2011.12) disables the L2/outer cache during boot on OMAP boards.
>> >
>> > uBoot commit: "armv7: disable L2 cache in cleanup_before_linux()" on 6th Dec 2011 by Aneesh V adds the following to
>> > uBootSources/arch/arm/cpu/armv7/cpu.c:cleanup_before_linux():
>> > ?...
>> > ?v7_out_cache_disable();
>> >
>> > The commit message implies this change was to make booting reliable on OMAP4 by disabling L2 cache before jumping to Linux. Sadly
>> > this commit also affects OMAP3-based boards...
>> >
>> > Since switching to uBoot 2011.12 I'm seeing about half the performance on my GUSMTIX Overo (OMAP3530).
>> >
>> > In the kernel sources there seem to be OMAP4 specific functions for re-enabling the L2 cache (omap4-common.c:omap_l2_cache_init()),
>> > but none for OMAP3.
>> >
>> > So my question is... do we expect that Linux should be re-enabling the L2/outer cache during boot after uBoot has disabled it on OMAP3?
>> >
>> Yes. Kernel already does enabled L2 on A8 CPU's but OMAP3 needs a
>> secure call and then the patching becomes
>> that much difficult.
>>
>> We need some kind of generic hook in proc*_setup() function where in
>> such secure bits can be turned ON.
>
> You can't have that kind of hook: there is no stack here to call other
> functions, plus we aren't running with the MMU on at that point so
> only carefully crafted assembly could be used.
>
This code will be in assembly and that's what I have
been using. Not having stack shoudn't be a blocker
and can be work-around in this code. And this API
has to be anyway called before MMU is enabled.

> Why is it soo difficult to issue the secure call?
>
Nothing difficult.
It is just that there is no place today where platform
can call these secure API's Call where ?

> This just proves my point that the lack of consistent secure monitor API
> for these kinds of basic service _are_ a big problem, one which ARM Ltd
> should have addressed when they invented this stuff. ?My view continues
> to be that the secure monitor API is a total abortion, one which should
> never have been allowed to happen.

Am not sure what you mean because secure API
as such isn't a problem. If you mean one standard interface
for all the ARM SOC's then that's something won't be
easy to handled because it is tied up the security architecture
which can vary across SoCs.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Mon, Jan 16, 2012 at 01:43:03PM +0100, Shilimkar, Santosh wrote:
> This code will be in assembly and that's what I have
> been using. Not having stack shoudn't be a blocker
> and can be work-around in this code. And this API
> has to be anyway called before MMU is enabled.

What about SMC on OMAP potentially corrupting most of the integer
registers?  What if it corrupts an integer register which we depend
upon?  You don't have a stack here to save those registers.  We'd
need at least one additional register to save the old 'lr' value.

What about other secure monitors?  Some implementations require you to
give parameters via memory.  How do you obtain that memory that early
on in the kernel boot (you haven't parsed anything here.)

Not only that but you're asking to make the kernel boot a _lot_ more
fragile, when people start stuffing all kinds of utter shite into this
hook - and then we're burdened again with the old 'my kernel won't
boot and it remains silent, well your debug code is utter shite get rid
of it' crap that we used to have in the early 2000s.

I really don't want to go anywhere near that situation.

> Am not sure what you mean because secure API
> as such isn't a problem. If you mean one standard interface
> for all the ARM SOC's then that's something won't be
> easy to handled because it is tied up the security architecture
> which can vary across SoCs.

The latter.  This is exactly the kind of pain I forsaw with this security
shite, and when I heard about it I was utterly dismayed at ARM Ltd for
coming up with such a brain-dead lack of design here.

You're having to struggle with the results of that by having lots of
SoC specific hooks all around the place to fiddle with this that and the
other.  Your need to have something called from the early assembly code
is just yet more of that disease caused by ARM Ltd's lack of forsight
on this matter.

I have no solution for you on this.

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Mon, Jan 16, 2012 at 2:13 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Mon, Jan 16, 2012 at 01:43:03PM +0100, Shilimkar, Santosh wrote:
>> This code will be in assembly and that's what I have
>> been using. Not having stack shoudn't be a blocker
>> and can be work-around in this code. And this API
>> has to be anyway called before MMU is enabled.
>
> What about SMC on OMAP potentially corrupting most of the integer
> registers? ?What if it corrupts an integer register which we depend
> upon? ?You don't have a stack here to save those registers. ?We'd
> need at least one additional register to save the old 'lr' value.
>
That's right.

> What about other secure monitors? ?Some implementations require you to
> give parameters via memory. ?How do you obtain that memory that early
> on in the kernel boot (you haven't parsed anything here.)
>
> Not only that but you're asking to make the kernel boot a _lot_ more
> fragile, when people start stuffing all kinds of utter shite into this
> hook - and then we're burdened again with the old 'my kernel won't
> boot and it remains silent, well your debug code is utter shite get rid
> of it' crap that we used to have in the early 2000s.
>
> I really don't want to go anywhere near that situation.
>
Fair point. It will be harder to maintain and won't be consistent.

>> Am not sure what you mean because secure API
>> as such isn't a problem. If you mean one standard interface
>> for all the ARM SOC's then that's something won't be
>> easy to handled because it is tied up the security architecture
>> which can vary across SoCs.
>
> The latter. ?This is exactly the kind of pain I forsaw with this security
> shite, and when I heard about it I was utterly dismayed at ARM Ltd for
> coming up with such a brain-dead lack of design here.
>
> You're having to struggle with the results of that by having lots of
> SoC specific hooks all around the place to fiddle with this that and the
> other. ?Your need to have something called from the early assembly code
> is just yet more of that disease caused by ARM Ltd's lack of forsight
> on this matter.
>
> I have no solution for you on this

I managed use some secure macro kind of code but as you said it
will be really hard to maintain.

So we are stuck with this issue then.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Aneesh V]

Santosh, Russel,

On Monday 16 January 2012 06:52 PM, Shilimkar, Santosh wrote:
> On Mon, Jan 16, 2012 at 2:13 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk>  wrote:
>> On Mon, Jan 16, 2012 at 01:43:03PM +0100, Shilimkar, Santosh wrote:
>>> This code will be in assembly and that's what I have
>>> been using. Not having stack shoudn't be a blocker
>>> and can be work-around in this code. And this API
>>> has to be anyway called before MMU is enabled.
>>
>> What about SMC on OMAP potentially corrupting most of the integer
>> registers?  What if it corrupts an integer register which we depend
>> upon?  You don't have a stack here to save those registers.  We'd
>> need at least one additional register to save the old 'lr' value.
>>
> That's right.
>
>> What about other secure monitors?  Some implementations require you to
>> give parameters via memory.  How do you obtain that memory that early
>> on in the kernel boot (you haven't parsed anything here.)
>>
>> Not only that but you're asking to make the kernel boot a _lot_ more
>> fragile, when people start stuffing all kinds of utter shite into this
>> hook - and then we're burdened again with the old 'my kernel won't
>> boot and it remains silent, well your debug code is utter shite get rid
>> of it' crap that we used to have in the early 2000s.
>>
>> I really don't want to go anywhere near that situation.
>>
> Fair point. It will be harder to maintain and won't be consistent.
>
>>> Am not sure what you mean because secure API
>>> as such isn't a problem. If you mean one standard interface
>>> for all the ARM SOC's then that's something won't be
>>> easy to handled because it is tied up the security architecture
>>> which can vary across SoCs.
>>
>> The latter.  This is exactly the kind of pain I forsaw with this security
>> shite, and when I heard about it I was utterly dismayed at ARM Ltd for
>> coming up with such a brain-dead lack of design here.
>>
>> You're having to struggle with the results of that by having lots of
>> SoC specific hooks all around the place to fiddle with this that and the
>> other.  Your need to have something called from the early assembly code
>> is just yet more of that disease caused by ARM Ltd's lack of forsight
>> on this matter.
>>
>> I have no solution for you on this
>
> I managed use some secure macro kind of code but as you said it
> will be really hard to maintain.
>
> So we are stuck with this issue then.

Can't we delay the L2 initialization a bit and do it when we have a
stack, like we do in OMAP4 (early_initcall)? Am I missing something?

br,
Aneesh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On Tue, Jan 17, 2012 at 08:54:44AM +0000, Joe Woodward wrote:
> So, is the upshot of this that the kernel isn't going to be in a
> position to enable the L2/outer cache on OMAP3 (due to the need for
> hacky/unmaintainable code)?
> 
> Hence the bootloader/uBoot had better leave it enabled...

It could but the Linux decompressor needs to be aware and either flush
the L2 (more difficult as it doesn't have all the device information) or
set the page table attributes to outer non-cacheable (TEX[2:0] = 0b100).
The latter may still not work if there are stale L2 cache lines left by
U-Boot (and that's always possible unless U-Boot also uses outer
non-cacheable memory attributes).

But I would agree with Aneesh - can we not enabled the L2 at a later
point?

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Aneesh V]

Hi Catalin,

On Tuesday 17 January 2012 05:41 PM, Catalin Marinas wrote:
> On Tue, Jan 17, 2012 at 08:54:44AM +0000, Joe Woodward wrote:
>> So, is the upshot of this that the kernel isn't going to be in a
>> position to enable the L2/outer cache on OMAP3 (due to the need for
>> hacky/unmaintainable code)?
>>
>> Hence the bootloader/uBoot had better leave it enabled...
>
> It could but the Linux decompressor needs to be aware and either flush
> the L2 (more difficult as it doesn't have all the device information) or

Cortex-A8 is aware of L2$ and can flush it, isn't it?

> set the page table attributes to outer non-cacheable (TEX[2:0] = 0b100).

If the above is right, this is not needed right?

> The latter may still not work if there are stale L2 cache lines left by
> U-Boot (and that's always possible unless U-Boot also uses outer
> non-cacheable memory attributes).

U-Boot flushes the caches before disabling it. On top of it, it does an
invalidate after the disabling the caches to cover some corner case.
So, it's guaranteed that there won't be any stale entries in L2 after
U-Boot.

So, we could as well leave L2$ enabled (and invalidated) and caches
disabled globally after U-Boot. But I thought enabling L2$ again in
kernel is the cleaner solution.

br,
Aneesh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 17, 2012 at 1:27 PM, Aneesh V <aneesh@ti.com> wrote:
> Hi Catalin,
>
>
> On Tuesday 17 January 2012 05:41 PM, Catalin Marinas wrote:
>>
>> On Tue, Jan 17, 2012 at 08:54:44AM +0000, Joe Woodward wrote:
>>>
>>> So, is the upshot of this that the kernel isn't going to be in a
>>> position to enable the L2/outer cache on OMAP3 (due to the need for
>>> hacky/unmaintainable code)?
>>>
>>> Hence the bootloader/uBoot had better leave it enabled...
>>
>>
>> It could but the Linux decompressor needs to be aware and either flush
>> the L2 (more difficult as it doesn't have all the device information) or
>
>
> Cortex-A8 is aware of L2$ and can flush it, isn't it?
>
>
>> set the page table attributes to outer non-cacheable (TEX[2:0] = 0b100).
>
>
> If the above is right, this is not needed right?
>
Well the L2 can be configured as inner or outer, so above
alone won't work.

Boot-loader disabling L2 cache ( all caches)  is still right thing
and that's what kernel expect.

Since the early kernel code can't be patches for A8, may be
delaying L2 enabled would work.

But then on A15, we are back to square one since there is no
control to turn ON/OFF l2 cache. On A15 infact there are other
CP15 registers which needs to be set before MMU is enabled to
have best configuration.

So the need of this early hook is pretty much there. May be separate
CPU setup code for such cases is a possible way to go.

Basically the problem needs to be sorted out, because I already
see this is needed for A15 as well

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On Tue, Jan 17, 2012 at 12:40:54PM +0000, Shilimkar, Santosh wrote:
> On Tue, Jan 17, 2012 at 1:27 PM, Aneesh V <aneesh@ti.com> wrote:
> > Hi Catalin,
> >
> >
> > On Tuesday 17 January 2012 05:41 PM, Catalin Marinas wrote:
> >>
> >> On Tue, Jan 17, 2012 at 08:54:44AM +0000, Joe Woodward wrote:
> >>>
> >>> So, is the upshot of this that the kernel isn't going to be in a
> >>> position to enable the L2/outer cache on OMAP3 (due to the need for
> >>> hacky/unmaintainable code)?
> >>>
> >>> Hence the bootloader/uBoot had better leave it enabled...
> >>
> >> It could but the Linux decompressor needs to be aware and either flush
> >> the L2 (more difficult as it doesn't have all the device information) or
> >
> > Cortex-A8 is aware of L2$ and can flush it, isn't it?
> >
> >> set the page table attributes to outer non-cacheable (TEX[2:0] = 0b100).
> >
> > If the above is right, this is not needed right?
>
> Well the L2 can be configured as inner or outer, so above
> alone won't work.
> 
> Boot-loader disabling L2 cache ( all caches)  is still right thing
> and that's what kernel expect.
> 
> Since the early kernel code can't be patches for A8, may be
> delaying L2 enabled would work.

Ah, I missed the fact that the L2 is an inner cache on OMAP3+A8. In this
case, I would argue to just leave it enabled since as long as the MMU is
off it won't be accessed. Why does U-Boot need to disable the L2 (I
suspect via some auxiliary control register)?

> But then on A15, we are back to square one since there is no
> control to turn ON/OFF l2 cache.

Well, it's an inner cache, it gets disabled/enabled via the SCTLR.M bit
as is the L1 cache. It does not need to be disabled independently.

> On A15 infact there are other CP15 registers which needs to be set
> before MMU is enabled to have best configuration.

Yes, there are, and my view is that it is up to the platform firmware to
set things right.

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 17, 2012 at 2:39 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Tue, Jan 17, 2012 at 12:40:54PM +0000, Shilimkar, Santosh wrote:
>> On Tue, Jan 17, 2012 at 1:27 PM, Aneesh V <aneesh@ti.com> wrote:
>> > Hi Catalin,
>> >
>> >
>> > On Tuesday 17 January 2012 05:41 PM, Catalin Marinas wrote:
>> >>
>> >> On Tue, Jan 17, 2012 at 08:54:44AM +0000, Joe Woodward wrote:
>> >>>
>> >>> So, is the upshot of this that the kernel isn't going to be in a
>> >>> position to enable the L2/outer cache on OMAP3 (due to the need for
>> >>> hacky/unmaintainable code)?
>> >>>
>> >>> Hence the bootloader/uBoot had better leave it enabled...
>> >>
>> >> It could but the Linux decompressor needs to be aware and either flush
>> >> the L2 (more difficult as it doesn't have all the device information) or
>> >
>> > Cortex-A8 is aware of L2$ and can flush it, isn't it?
>> >
>> >> set the page table attributes to outer non-cacheable (TEX[2:0] = 0b100).
>> >
>> > If the above is right, this is not needed right?
>>
>> Well the L2 can be configured as inner or outer, so above
>> alone won't work.
>>
>> Boot-loader disabling L2 cache ( all caches) ?is still right thing
>> and that's what kernel expect.
>>
>> Since the early kernel code can't be patches for A8, may be
>> delaying L2 enabled would work.
>
> Ah, I missed the fact that the L2 is an inner cache on OMAP3+A8. In this
> case, I would argue to just leave it enabled since as long as the MMU is
> off it won't be accessed. Why does U-Boot need to disable the L2 (I
> suspect via some auxiliary control register)?
>
>> But then on A15, we are back to square one since there is no
>> control to turn ON/OFF l2 cache.
>
> Well, it's an inner cache, it gets disabled/enabled via the SCTLR.M bit
> as is the L1 cache. It does not need to be disabled independently.
>
>> On A15 infact there are other CP15 registers which needs to be set
>> before MMU is enabled to have best configuration.
>
> Yes, there are, and my view is that it is up to the platform firmware to
> set things right.
>
Firmware is ROM'ed in SOC's and can't be altered without a new silicon spin.
Then we get into situations where Errata A will say turn OFF/ON x bit and
so on. Most of these bits are secure write-only registers hence ROM
code provides few API's to alter these registers in future if needed.

BTW, look at the processor setup functions where we are altering these
bits in kernel today (Diagnostic, RR policy etc) and they work on
ARM SoC's which either don't use trustzone or kernel is in secure mode.

Now same thing if needs to be done for SoC's like OMAP where
these bits are under secure control, we need to use the ROM
secure API's and kernel don't seems to give any flexibility
to patch that.

I don't understand how __only__ these devices needs to be
patches outside the kernel. Firmware point is actually irrelevant for
OMAP since it can't be changed once it is ROMed.

Patching in boot-loaders isn't an option either since every customers
prefers to use there own boot-loader and then controlling
this vital bits is impossible.

So I re-iterate that we need to have solution to this problem.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On Tue, Jan 17, 2012 at 01:58:18PM +0000, Shilimkar, Santosh wrote:
> On Tue, Jan 17, 2012 at 2:39 PM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > On Tue, Jan 17, 2012 at 12:40:54PM +0000, Shilimkar, Santosh wrote:
> >> On A15 infact there are other CP15 registers which needs to be set
> >> before MMU is enabled to have best configuration.
> >
> > Yes, there are, and my view is that it is up to the platform firmware to
> > set things right.
>
> Firmware is ROM'ed in SOC's and can't be altered without a new silicon spin.
> Then we get into situations where Errata A will say turn OFF/ON x bit and
> so on. Most of these bits are secure write-only registers hence ROM
> code provides few API's to alter these registers in future if needed.

BTW, does the firmware make any checks on what bits it allows to be set?
Some of them may have security implications (and may not even be
documented).

Anyway, the first step is this API provided by the secure firmware.
Since such API may need to be called before the MMU is initialised,
Linux would need to have knowledge of the platform type early on. Having
some platform hook (asm macro) to call early on wouldn't work with the
single zImage configuration. Stack space is not an issue as we already
have one for ARMv7 for D-cache flushing (XIP kernels would work but they
aren't that many).

> BTW, look at the processor setup functions where we are altering these
> bits in kernel today (Diagnostic, RR policy etc) and they work on
> ARM SoC's which either don't use trustzone or kernel is in secure mode.

What we did for many of them was to check whether the bits are already
set to avoid a fault when running in NS mode.

> Now same thing if needs to be done for SoC's like OMAP where
> these bits are under secure control, we need to use the ROM
> secure API's and kernel don't seems to give any flexibility
> to patch that.
> 
> I don't understand how __only__ these devices needs to be
> patches outside the kernel.

Because there isn't a standard API to set those registers. And such
generic API would not even be easily possible given that you can't tell
what auxiliary (or even undocumented) registers a new CPU may have in
the future (unless the API is part of the TRM).

> Firmware point is actually irrelevant for OMAP since it can't be
> changed once it is ROMed.
> 
> Patching in boot-loaders isn't an option either since every customers
> prefers to use there own boot-loader and then controlling this vital
> bits is impossible.
> 
> So I re-iterate that we need to have solution to this problem.

I don't disagree with this, a solution is needed. In the past I was for
having platform hooks in the early kernel code path but in the light of
latest kernel developments (FDT, single zImage), I no longer see this as
acceptable.

If you can't change the boot loader, an alternative could be for the SoC
vendor to provide an additional binary that is executed by the boot
loader before Linux is started. Such binary could either live in some
flash or it could be wrapped around the Linux kernel image before the
kernel is stored into flash.

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 17, 2012 at 5:27 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Tue, Jan 17, 2012 at 01:58:18PM +0000, Shilimkar, Santosh wrote:
>> On Tue, Jan 17, 2012 at 2:39 PM, Catalin Marinas
>> <catalin.marinas@arm.com> wrote:
>> > On Tue, Jan 17, 2012 at 12:40:54PM +0000, Shilimkar, Santosh wrote:
>> >> On A15 infact there are other CP15 registers which needs to be set
>> >> before MMU is enabled to have best configuration.
>> >
>> > Yes, there are, and my view is that it is up to the platform firmware to
>> > set things right.
>>
>> Firmware is ROM'ed in SOC's and can't be altered without a new silicon spin.
>> Then we get into situations where Errata A will say turn OFF/ON x bit and
>> so on. Most of these bits are secure write-only registers hence ROM
>> code provides few API's to alter these registers in future if needed.
>
> BTW, does the firmware make any checks on what bits it allows to be set?
> Some of them may have security implications (and may not even be
> documented).
>
> Anyway, the first step is this API provided by the secure firmware.
> Since such API may need to be called before the MMU is initialised,
> Linux would need to have knowledge of the platform type early on. Having
> some platform hook (asm macro) to call early on wouldn't work with the
> single zImage configuration. Stack space is not an issue as we already
> have one for ARMv7 for D-cache flushing (XIP kernels would work but they
> aren't that many).
>
That's true.

>> BTW, look at the processor setup functions where we are altering these
>> bits in kernel today (Diagnostic, RR policy etc) and they work on
>> ARM SoC's which either don't use trustzone or kernel is in secure mode.
>
> What we did for many of them was to check whether the bits are already
> set to avoid a fault when running in NS mode.
>
Yes but then as I mentioned for the secure kernels, they can set from the
kernel code.

>> Now same thing if needs to be done for SoC's like OMAP where
>> these bits are under secure control, we need to use the ROM
>> secure API's and kernel don't seems to give any flexibility
>> to patch that.
>>
>> I don't understand how __only__ these devices needs to be
>> patches outside the kernel.
>
> Because there isn't a standard API to set those registers. And such
> generic API would not even be easily possible given that you can't tell
> what auxiliary (or even undocumented) registers a new CPU may have in
> the future (unless the API is part of the TRM).
>
>> Firmware point is actually irrelevant for OMAP since it can't be
>> changed once it is ROMed.
>>
>> Patching in boot-loaders isn't an option either since every customers
>> prefers to use there own boot-loader and then controlling this vital
>> bits is impossible.
>>
>> So I re-iterate that we need to have solution to this problem.
>
> I don't disagree with this, a solution is needed. In the past I was for
> having platform hooks in the early kernel code path but in the light of
> latest kernel developments (FDT, single zImage), I no longer see this as
> acceptable.
>
Thanks for acknowledging the problem. Not allowing such hooks for single
zImage doesn't seems to be right. Nobody ships kernel build for all socs,
and it's usage is really limited to few distro's. But that's different topic.

How about allowing platform hooks for single SOC builds. I mean having
this code under !single_zImage or something like that. That way we don't
impact the single zImage efforts and also allow socs to have all those
critical, vital bits enabled for the SOC specific builds.

> If you can't change the boot loader, an alternative could be for the SoC
> vendor to provide an additional binary that is executed by the boot
> loader before Linux is started. Such binary could either live in some
> flash or it could be wrapped around the Linux kernel image before the
> kernel is stored into flash.
>
We tried something similar with boot-loaders and failed badly to get that
right for all the product trees. So if the single zImage is the blocker
I would like to tackle that and have all this important information within
kernel code.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Nicolas Pitre]

On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:

> On Tue, Jan 17, 2012 at 5:27 PM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > BTW, does the firmware make any checks on what bits it allows to be set?
> > Some of them may have security implications (and may not even be
> > documented).
> >
> > Anyway, the first step is this API provided by the secure firmware.
> > Since such API may need to be called before the MMU is initialised,
> > Linux would need to have knowledge of the platform type early on. Having
> > some platform hook (asm macro) to call early on wouldn't work with the
> > single zImage configuration. Stack space is not an issue as we already
> > have one for ARMv7 for D-cache flushing (XIP kernels would work but they
> > aren't that many).
> >
> That's true.

That stack is ugly and _does_ break XIP (even if there aren't that many 
if at all on ARMv7).  Since we're already writing to RAM while setting 
up the initial page table and therefore we did the necessary work to 
locate it already, we could as well put a temporary stack for early 
usage right below the page table.

> >> Firmware point is actually irrelevant for OMAP since it can't be
> >> changed once it is ROMed.
> >>
> >> Patching in boot-loaders isn't an option either since every customers
> >> prefers to use there own boot-loader and then controlling this vital
> >> bits is impossible.
> >>
> >> So I re-iterate that we need to have solution to this problem.
> >
> > I don't disagree with this, a solution is needed. In the past I was for
> > having platform hooks in the early kernel code path but in the light of
> > latest kernel developments (FDT, single zImage), I no longer see this as
> > acceptable.
> >
> Thanks for acknowledging the problem. Not allowing such hooks for single
> zImage doesn't seems to be right. Nobody ships kernel build for all socs,
> and it's usage is really limited to few distro's. But that's different topic.

No, I think this is right on topic.  If nobody ships a kernel for 
multiple SOCs, this is because right now this can't be done.  But 
distros are craving for this ability.

> How about allowing platform hooks for single SOC builds. I mean having
> this code under !single_zImage or something like that. That way we don't
> impact the single zImage efforts and also allow socs to have all those
> critical, vital bits enabled for the SOC specific builds.

Absolutely not!  Because if we start doing that, people will get lazy 
and no platform will ever work in a multi-SOC kernel.

If your SOC require some fancy setup that is not shared by other 
platforms then please abstract that into the bootloader, or make sure it 
can be deferred later on.



Nicolas

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 17, 2012 at 8:39 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
>
>> On Tue, Jan 17, 2012 at 5:27 PM, Catalin Marinas
>> <catalin.marinas@arm.com> wrote:
>> > BTW, does the firmware make any checks on what bits it allows to be set?
>> > Some of them may have security implications (and may not even be
>> > documented).
>> >
>> > Anyway, the first step is this API provided by the secure firmware.
>> > Since such API may need to be called before the MMU is initialised,
>> > Linux would need to have knowledge of the platform type early on. Having
>> > some platform hook (asm macro) to call early on wouldn't work with the
>> > single zImage configuration. Stack space is not an issue as we already
>> > have one for ARMv7 for D-cache flushing (XIP kernels would work but they
>> > aren't that many).
>> >
>> That's true.
>
> That stack is ugly and _does_ break XIP (even if there aren't that many
> if at all on ARMv7). ?Since we're already writing to RAM while setting
> up the initial page table and therefore we did the necessary work to
> locate it already, we could as well put a temporary stack for early
> usage right below the page table.
>
>> >> Firmware point is actually irrelevant for OMAP since it can't be
>> >> changed once it is ROMed.
>> >>
>> >> Patching in boot-loaders isn't an option either since every customers
>> >> prefers to use there own boot-loader and then controlling this vital
>> >> bits is impossible.
>> >>
>> >> So I re-iterate that we need to have solution to this problem.
>> >
>> > I don't disagree with this, a solution is needed. In the past I was for
>> > having platform hooks in the early kernel code path but in the light of
>> > latest kernel developments (FDT, single zImage), I no longer see this as
>> > acceptable.
>> >
>> Thanks for acknowledging the problem. Not allowing such hooks for single
>> zImage doesn't seems to be right. Nobody ships kernel build for all socs,
>> and it's usage is really limited to few distro's. But that's different topic.
>
> No, I think this is right on topic. ?If nobody ships a kernel for
> multiple SOCs, this is because right now this can't be done. ?But
> distros are craving for this ability.
>
>> How about allowing platform hooks for single SOC builds. I mean having
>> this code under !single_zImage or something like that. That way we don't
>> impact the single zImage efforts and also allow socs to have all those
>> critical, vital bits enabled for the SOC specific builds.
>
> Absolutely not! ?Because if we start doing that, people will get lazy
> and no platform will ever work in a multi-SOC kernel.
>
> If your SOC require some fancy setup that is not shared by other
> platforms then please abstract that into the bootloader, or make sure it
> can be deferred later on.
>
There is nothing fancy here. It's an ARM security architecture feature which
OMAP implements. Have given enough reason about boot-loaders issues.

Is OMAP getting beaten up here just because it uses ARM security
feature and implements it's mechanics?

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Nicolas Pitre]

On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:

> On Tue, Jan 17, 2012 at 8:39 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
> >
> >> How about allowing platform hooks for single SOC builds. I mean having
> >> this code under !single_zImage or something like that. That way we don't
> >> impact the single zImage efforts and also allow socs to have all those
> >> critical, vital bits enabled for the SOC specific builds.
> >
> > Absolutely not! ?Because if we start doing that, people will get lazy
> > and no platform will ever work in a multi-SOC kernel.
> >
> > If your SOC require some fancy setup that is not shared by other
> > platforms then please abstract that into the bootloader, or make sure it
> > can be deferred later on.
> >
> There is nothing fancy here. It's an ARM security architecture feature which
> OMAP implements. Have given enough reason about boot-loaders issues.

I was not convinced by those reasons. Just push harder on the bootloader 
side.  There is _no_ reason for the bootloader not to take care of this 
very platform specific issue.  You can even do it into a standalone 
uImage that returns to u-Boot after it is done with its magic.

> Is OMAP getting beaten up here just because it uses ARM security
> feature and implements it's mechanics?

I don't care if this is OMAP, UX500 or i.MX.  There is a line to be 
drawn and this is just too bad if you are trying to cross it.


Nicolas

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Nicolas Pitre]

On Tue, 17 Jan 2012, Nicolas Pitre wrote:

> On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
> 
> > On Tue, Jan 17, 2012 at 8:39 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > > On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
> > >
> > >> How about allowing platform hooks for single SOC builds. I mean having
> > >> this code under !single_zImage or something like that. That way we don't
> > >> impact the single zImage efforts and also allow socs to have all those
> > >> critical, vital bits enabled for the SOC specific builds.
> > >
> > > Absolutely not! ?Because if we start doing that, people will get lazy
> > > and no platform will ever work in a multi-SOC kernel.
> > >
> > > If your SOC require some fancy setup that is not shared by other
> > > platforms then please abstract that into the bootloader, or make sure it
> > > can be deferred later on.
> > >
> > There is nothing fancy here. It's an ARM security architecture feature which
> > OMAP implements. Have given enough reason about boot-loaders issues.
> 
> I was not convinced by those reasons. Just push harder on the bootloader 
> side.  There is _no_ reason for the bootloader not to take care of this 
> very platform specific issue.  You can even do it into a standalone 
> uImage that returns to u-Boot after it is done with its magic.

Alternatively, you may leverage all the infrastructure that was 
implemented to support CPU offlining i.e. the creation of a 1:1 mapping 
needed to disable the MMU and so on.  That should be possible for you to 
just have the MMU turned off, then call the secure mode to turn on L2, 
then restore MMU and normal operation of the kernel.  This can be 
accomplished later during boot rather than right at the beginning.


Nicolas

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 17, 2012 at 9:45 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
>
>> On Tue, Jan 17, 2012 at 8:39 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> > On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
>> >
>> >> How about allowing platform hooks for single SOC builds. I mean having
>> >> this code under !single_zImage or something like that. That way we don't
>> >> impact the single zImage efforts and also allow socs to have all those
>> >> critical, vital bits enabled for the SOC specific builds.
>> >
>> > Absolutely not! ?Because if we start doing that, people will get lazy
>> > and no platform will ever work in a multi-SOC kernel.
>> >
>> > If your SOC require some fancy setup that is not shared by other
>> > platforms then please abstract that into the bootloader, or make sure it
>> > can be deferred later on.
>> >
>> There is nothing fancy here. It's an ARM security architecture feature which
>> OMAP implements. Have given enough reason about boot-loaders issues.
>
> I was not convinced by those reasons. Just push harder on the bootloader
> side. ?There is _no_ reason for the bootloader not to take care of this
> very platform specific issue. ?You can even do it into a standalone
> uImage that returns to u-Boot after it is done with its magic.
>
>> Is OMAP getting beaten up here just because it uses ARM security
>> feature and implements it's mechanics?
>
> I don't care if this is OMAP, UX500 or i.MX. ?There is a line to be
> drawn and this is just too bad if you are trying to cross it.
>
Well I was trying to get a solution for a genuine problem.
Let's take an example.

#ifdef CONFIG_ARM_ERRATA_742230
        cmp     r6, #0x22                       @ only present up to r2p2
        mrcle   p15, 0, r10, c15, c0, 1         @ read diagnostic register
        orrle   r10, r10, #1 << 4               @ set bit #4
        mcrle   p15, 0, r10, c15, c0, 1         @ write diagnostic register
#endif

The SoC's which doesn't have security restriction and the kernel
is in secure mode, above errata is applied in it's right place. Before
MMU is enabled. There are many bits like this which are patched
in kernel.

On OMAP, all these can't be set-up in kernel since kernel is
in non-secure mode. To set such bit's we need to call some
pre-defined security calls.

Regards
santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Nicolas Pitre]

On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:

> On Tue, Jan 17, 2012 at 9:45 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
> >
> >> On Tue, Jan 17, 2012 at 8:39 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> >> > On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
> >> >
> >> >> How about allowing platform hooks for single SOC builds. I mean having
> >> >> this code under !single_zImage or something like that. That way we don't
> >> >> impact the single zImage efforts and also allow socs to have all those
> >> >> critical, vital bits enabled for the SOC specific builds.
> >> >
> >> > Absolutely not! ?Because if we start doing that, people will get lazy
> >> > and no platform will ever work in a multi-SOC kernel.
> >> >
> >> > If your SOC require some fancy setup that is not shared by other
> >> > platforms then please abstract that into the bootloader, or make sure it
> >> > can be deferred later on.
> >> >
> >> There is nothing fancy here. It's an ARM security architecture feature which
> >> OMAP implements. Have given enough reason about boot-loaders issues.
> >
> > I was not convinced by those reasons. Just push harder on the bootloader
> > side. ?There is _no_ reason for the bootloader not to take care of this
> > very platform specific issue. ?You can even do it into a standalone
> > uImage that returns to u-Boot after it is done with its magic.
> >
> >> Is OMAP getting beaten up here just because it uses ARM security
> >> feature and implements it's mechanics?
> >
> > I don't care if this is OMAP, UX500 or i.MX. ?There is a line to be
> > drawn and this is just too bad if you are trying to cross it.
> >
> Well I was trying to get a solution for a genuine problem.
> Let's take an example.
> 
> #ifdef CONFIG_ARM_ERRATA_742230
>         cmp     r6, #0x22                       @ only present up to r2p2
>         mrcle   p15, 0, r10, c15, c0, 1         @ read diagnostic register
>         orrle   r10, r10, #1 << 4               @ set bit #4
>         mcrle   p15, 0, r10, c15, c0, 1         @ write diagnostic register
> #endif
> 
> The SoC's which doesn't have security restriction and the kernel
> is in secure mode, above errata is applied in it's right place. Before
> MMU is enabled. There are many bits like this which are patched
> in kernel.
> 
> On OMAP, all these can't be set-up in kernel since kernel is
> in non-secure mode. To set such bit's we need to call some
> pre-defined security calls.

Absolutely.  Just don't do it in the early kernel boot code where it is 
not convenient to do so.


Nicolas

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 17, 2012 at 10:02 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
>
>> On Tue, Jan 17, 2012 at 9:45 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> > On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
>> >
>> >> On Tue, Jan 17, 2012 at 8:39 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> >> > On Tue, 17 Jan 2012, Shilimkar, Santosh wrote:
>> >> >
>> >> >> How about allowing platform hooks for single SOC builds. I mean having
>> >> >> this code under !single_zImage or something like that. That way we don't
>> >> >> impact the single zImage efforts and also allow socs to have all those
>> >> >> critical, vital bits enabled for the SOC specific builds.
>> >> >
>> >> > Absolutely not! ?Because if we start doing that, people will get lazy
>> >> > and no platform will ever work in a multi-SOC kernel.
>> >> >
>> >> > If your SOC require some fancy setup that is not shared by other
>> >> > platforms then please abstract that into the bootloader, or make sure it
>> >> > can be deferred later on.
>> >> >
>> >> There is nothing fancy here. It's an ARM security architecture feature which
>> >> OMAP implements. Have given enough reason about boot-loaders issues.
>> >
>> > I was not convinced by those reasons. Just push harder on the bootloader
>> > side. ?There is _no_ reason for the bootloader not to take care of this
>> > very platform specific issue. ?You can even do it into a standalone
>> > uImage that returns to u-Boot after it is done with its magic.
>> >
>> >> Is OMAP getting beaten up here just because it uses ARM security
>> >> feature and implements it's mechanics?
>> >
>> > I don't care if this is OMAP, UX500 or i.MX. ?There is a line to be
>> > drawn and this is just too bad if you are trying to cross it.
>> >
>> Well I was trying to get a solution for a genuine problem.
>> Let's take an example.
>>
>> #ifdef CONFIG_ARM_ERRATA_742230
>> ? ? ? ? cmp ? ? r6, #0x22 ? ? ? ? ? ? ? ? ? ? ? @ only present up to r2p2
>> ? ? ? ? mrcle ? p15, 0, r10, c15, c0, 1 ? ? ? ? @ read diagnostic register
>> ? ? ? ? orrle ? r10, r10, #1 << 4 ? ? ? ? ? ? ? @ set bit #4
>> ? ? ? ? mcrle ? p15, 0, r10, c15, c0, 1 ? ? ? ? @ write diagnostic register
>> #endif
>>
>> The SoC's which doesn't have security restriction and the kernel
>> is in secure mode, above errata is applied in it's right place. Before
>> MMU is enabled. There are many bits like this which are patched
>> in kernel.
>>
>> On OMAP, all these can't be set-up in kernel since kernel is
>> in non-secure mode. To set such bit's we need to call some
>> pre-defined security calls.
>
> Absolutely. ?Just don't do it in the early kernel boot code where it is
> not convenient to do so.
>
Got it.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Tue, Jan 17, 2012 at 09:27:02PM +0100, Shilimkar, Santosh wrote:
> There is nothing fancy here. It's an ARM security architecture feature
> which OMAP implements. Have given enough reason about boot-loaders
> issues.

There is nothing fancy about not being permitted to access registers
due to security restrictions.  What is fancy is that every SoC vendor
out there implements their own private API to provide a method to access
these registers without any form of commonality.

> Is OMAP getting beaten up here just because it uses ARM security
> feature and implements it's mechanics?

To confirm what Nicolas said in reply to this - I really don't care
either what kind of SoC this is.  This is purely about the technical
problems with what is being proposed.

And, as I've said several times already, the root cause of this problem
is not the SoC vendors, but a lack of standardization about how these
services are provided.

Imagine this: what if every vendor of a PC out there provided their
ACPI data in totally different formats?

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Tue, Jan 17, 2012 at 04:27:21PM +0000, Catalin Marinas wrote:
> Anyway, the first step is this API provided by the secure firmware.
> Since such API may need to be called before the MMU is initialised,
> Linux would need to have knowledge of the platform type early on. Having
> some platform hook (asm macro) to call early on wouldn't work with the
> single zImage configuration. Stack space is not an issue as we already
> have one for ARMv7 for D-cache flushing (XIP kernels would work but they
> aren't that many).

It _is_ a problem.  How much stack space would these undefined platform
specific functions require?  16 words?  32 words?  A page?  Where does
this memory come from?  The .data section, or are we going to stuff it
into what is supposed to be the read-only text section?

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 17, 2012 at 8:47 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Tue, Jan 17, 2012 at 04:27:21PM +0000, Catalin Marinas wrote:
>> Anyway, the first step is this API provided by the secure firmware.
>> Since such API may need to be called before the MMU is initialised,
>> Linux would need to have knowledge of the platform type early on. Having
>> some platform hook (asm macro) to call early on wouldn't work with the
>> single zImage configuration. Stack space is not an issue as we already
>> have one for ARMv7 for D-cache flushing (XIP kernels would work but they
>> aren't that many).
>
> It _is_ a problem. ?How much stack space would these undefined platform
> specific functions require? ?16 words? ?32 words? ?A page? ?Where does
> this memory come from? ?The .data section, or are we going to stuff it
> into what is supposed to be the read-only text section?

Stack requirement is limited to saving and restoring the processor
registers. There is no additional need. If this is an issue, we can also use
SRAM as a stack for this operation.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Tue, Jan 17, 2012 at 09:11:37PM +0100, Shilimkar, Santosh wrote:
> On Tue, Jan 17, 2012 at 8:47 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Tue, Jan 17, 2012 at 04:27:21PM +0000, Catalin Marinas wrote:
> >> Anyway, the first step is this API provided by the secure firmware.
> >> Since such API may need to be called before the MMU is initialised,
> >> Linux would need to have knowledge of the platform type early on. Having
> >> some platform hook (asm macro) to call early on wouldn't work with the
> >> single zImage configuration. Stack space is not an issue as we already
> >> have one for ARMv7 for D-cache flushing (XIP kernels would work but they
> >> aren't that many).
> >
> > It _is_ a problem. ?How much stack space would these undefined platform
> > specific functions require? ?16 words? ?32 words? ?A page? ?Where does
> > this memory come from? ?The .data section, or are we going to stuff it
> > into what is supposed to be the read-only text section?
> 
> Stack requirement is limited to saving and restoring the processor
> registers. There is no additional need. If this is an issue, we can
> also use SRAM as a stack for this operation.

How do you know that's all that would be required?

Let's look at another secure monitor API implementation in the kernel.
MSM has one - arch/arm/mach-msm/scm.c.  This requires parameters passed
through memory - a command buffer consisting of at least four words in
memory, followed by a response buffer of at least three words.

So where would this be placed if not on the stack?  Pick some random
address and hope that there's some memory there?

This is what I'm saying: while we may know what the requirement is for
OMAP, that's not what it is for other SoCs.  Every SoC has different
requirements for this, and so we have _no_ idea what so ever how much
stack space would actually be required.  And if we don't know that, we
have a real chance of having some implementation overflow the stack,
thereby overwriting something (eg, maybe your DT data, maybe your
initramfs image) which would be extremely detrimental.

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Tue, Jan 17, 2012 at 02:58:18PM +0100, Shilimkar, Santosh wrote:
> Patching in boot-loaders isn't an option either since every customers
> prefers to use there own boot-loader and then controlling
> this vital bits is impossible.
> 
> So I re-iterate that we need to have solution to this problem.

And I'll re-iterate that it'll create an absolute abonimation of code
to do this in the kernel - not only will it be extremely fragile, it
will be hell to maintain in the longer term.  It will also be highly
platform specific.

Essentially, what you're asking is for us to find some way to call a
platform specific function depending on the platform (which we haven't
parsed yet), which uses a set of undefined registers, which may corrupt
an undefined set of registers.  And call that function from a place in
the kernel which requires a set of registers to be preserved, with no
stack available to save registers, and call the function via a PC
relative branch (as its virtual address is useless to us at this point.)

If anything goes wrong, there's no way to report that as your only
option is to stop dead (if you're lucky) or maybe end up with the
processor trying to execute instructions from a non-existent or
uninitialized vectors page, etc.

This is far too much.  This is far too fragile.  This is asking for
undebuggable problems.  This is asking for lots of people with unbootable
kernels.

No.  Thanks.

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On Fri, Jan 20, 2012 at 08:57:11AM +0000, Joe Woodward wrote:
> > So I re-iterate that we need to have solution to this problem.
> 
> ... I don't want to be a pain, but it seems to me that this dicussion
> didn't reach a full conclussion?

Probably not, because it depends on many variables. See below my take on
this.

> I think it was left with the open options being:
> 1) Leave the L2/outer cache enabled in the bootloader (not ideal and
> may cause problems with future devices)

This depends on whether the L2 is inner or outer:

L2 inner - leave it enabled in the boot loader
L2 outer - leave it disabled in the boot loader

> 2) Turn the L2/outer cache on for OMAP3 later in the kernel boot when
> the device is known

Same as above:

L2 inner - don't do anything, it gets used when SCTLR.M is enabled
L2 outer - enabled at boot time via the platform code (later, after MMU
	was enabled).

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Aneesh V]

Hi Catalin,

On Friday 27 January 2012 11:00 PM, Catalin Marinas wrote:
> On Fri, Jan 20, 2012 at 08:57:11AM +0000, Joe Woodward wrote:
>>> So I re-iterate that we need to have solution to this problem.
>>
>> ... I don't want to be a pain, but it seems to me that this dicussion
>> didn't reach a full conclussion?
>
> Probably not, because it depends on many variables. See below my take on
> this.
>
>> I think it was left with the open options being:
>> 1) Leave the L2/outer cache enabled in the bootloader (not ideal and
>> may cause problems with future devices)
>
> This depends on whether the L2 is inner or outer:
>
> L2 inner - leave it enabled in the boot loader
> L2 outer - leave it disabled in the boot loader
>
>> 2) Turn the L2/outer cache on for OMAP3 later in the kernel boot when
>> the device is known
>
> Same as above:
>
> L2 inner - don't do anything, it gets used when SCTLR.M is enabled
> L2 outer - enabled at boot time via the platform code (later, after MMU
> 	was enabled).
>

What is the reasoning behind this recommendation? Why the distinction
between L2 being inner or outer. I don't see anything to this effect in
the Cortex-A8 TRM? In fact the only recommendation I could find(section
8.3) is asking to set L2EN to 1 before setting C bit to 1 irrespective
of inner/outer?

br,
Aneesh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On 31 January 2012 05:21, Aneesh V <aneesh@ti.com> wrote:
> On Friday 27 January 2012 11:00 PM, Catalin Marinas wrote:
>> On Fri, Jan 20, 2012 at 08:57:11AM +0000, Joe Woodward wrote:
>>>> So I re-iterate that we need to have solution to this problem.
>>>
>>> ... I don't want to be a pain, but it seems to me that this dicussion
>>> didn't reach a full conclussion?
>>
>> Probably not, because it depends on many variables. See below my take on
>> this.
>>
>>> I think it was left with the open options being:
>>> 1) Leave the L2/outer cache enabled in the bootloader (not ideal and
>>> may cause problems with future devices)
>>
>> This depends on whether the L2 is inner or outer:
>>
>> L2 inner - leave it enabled in the boot loader
>> L2 outer - leave it disabled in the boot loader
>>
>>> 2) Turn the L2/outer cache on for OMAP3 later in the kernel boot when
>>> the device is known
>>
>> Same as above:
>>
>> L2 inner - don't do anything, it gets used when SCTLR.M is enabled
>> L2 outer - enabled at boot time via the platform code (later, after MMU
>> ? ? ? ?was enabled).
>
> What is the reasoning behind this recommendation? Why the distinction
> between L2 being inner or outer. I don't see anything to this effect in
> the Cortex-A8 TRM? In fact the only recommendation I could find(section
> 8.3) is asking to set L2EN to 1 before setting C bit to 1 irrespective
> of inner/outer?

Actually you have a good point. I was thinking about outer caches like
PL310 or L220 that Linux enables later during platform initialisation.
They could be left enabled or disabled at boot time but they must be
cleaned/invalidated (you would need to flush after disabling just to
make sure there are no further lines fetched; alternatively, there
shouldn't be any cacheable mappings).

So I would say in the context of A8, just leave the L2 enabled in the
boot loader. As long as the MMU is disabled, it won't be used. The
alternative is to enable L2 slightly later in Linux in the platform
code to avoid SoC-dependent code in proc-v7.S.

So in summary, either option is possible. You could take the approach
of the cache-l2x0.c - check whether it is already enabled and, if not,
issue an SMC from the platform code to enable the L2 cache.

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 31, 2012 at 1:01 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On 31 January 2012 05:21, Aneesh V <aneesh@ti.com> wrote:
>> On Friday 27 January 2012 11:00 PM, Catalin Marinas wrote:
>>> On Fri, Jan 20, 2012 at 08:57:11AM +0000, Joe Woodward wrote:
>>>>> So I re-iterate that we need to have solution to this problem.
>>>>
>>>> ... I don't want to be a pain, but it seems to me that this dicussion
>>>> didn't reach a full conclussion?
>>>
>>> Probably not, because it depends on many variables. See below my take on
>>> this.
>>>
>>>> I think it was left with the open options being:
>>>> 1) Leave the L2/outer cache enabled in the bootloader (not ideal and
>>>> may cause problems with future devices)
>>>
>>> This depends on whether the L2 is inner or outer:
>>>
>>> L2 inner - leave it enabled in the boot loader
>>> L2 outer - leave it disabled in the boot loader
>>>
>>>> 2) Turn the L2/outer cache on for OMAP3 later in the kernel boot when
>>>> the device is known
>>>
>>> Same as above:
>>>
>>> L2 inner - don't do anything, it gets used when SCTLR.M is enabled
>>> L2 outer - enabled at boot time via the platform code (later, after MMU
>>> ? ? ? ?was enabled).
>>
>> What is the reasoning behind this recommendation? Why the distinction
>> between L2 being inner or outer. I don't see anything to this effect in
>> the Cortex-A8 TRM? In fact the only recommendation I could find(section
>> 8.3) is asking to set L2EN to 1 before setting C bit to 1 irrespective
>> of inner/outer?
>
> Actually you have a good point. I was thinking about outer caches like
> PL310 or L220 that Linux enables later during platform initialisation.
> They could be left enabled or disabled at boot time but they must be
> cleaned/invalidated (you would need to flush after disabling just to
> make sure there are no further lines fetched; alternatively, there
> shouldn't be any cacheable mappings).
>
> So I would say in the context of A8, just leave the L2 enabled in the
> boot loader. As long as the MMU is disabled, it won't be used. The
> alternative is to enable L2 slightly later in Linux in the platform
> code to avoid SoC-dependent code in proc-v7.S.
>
> So in summary, either option is possible. You could take the approach
> of the cache-l2x0.c - check whether it is already enabled and, if not,
> issue an SMC from the platform code to enable the L2 cache.
>
if done late in kernel, wouldn't it violate "set L2EN to 1 before
setting C bit to 1"

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On 31 January 2012 07:38, Shilimkar, Santosh <santosh.shilimkar@ti.com> wrote:
> On Tue, Jan 31, 2012 at 1:01 PM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
>> On 31 January 2012 05:21, Aneesh V <aneesh@ti.com> wrote:
>>> On Friday 27 January 2012 11:00 PM, Catalin Marinas wrote:
>>>> On Fri, Jan 20, 2012 at 08:57:11AM +0000, Joe Woodward wrote:
>>>>>> So I re-iterate that we need to have solution to this problem.
>>>>>
>>>>> ... I don't want to be a pain, but it seems to me that this dicussion
>>>>> didn't reach a full conclussion?
>>>>
>>>> Probably not, because it depends on many variables. See below my take on
>>>> this.
>>>>
>>>>> I think it was left with the open options being:
>>>>> 1) Leave the L2/outer cache enabled in the bootloader (not ideal and
>>>>> may cause problems with future devices)
>>>>
>>>> This depends on whether the L2 is inner or outer:
>>>>
>>>> L2 inner - leave it enabled in the boot loader
>>>> L2 outer - leave it disabled in the boot loader
>>>>
>>>>> 2) Turn the L2/outer cache on for OMAP3 later in the kernel boot when
>>>>> the device is known
>>>>
>>>> Same as above:
>>>>
>>>> L2 inner - don't do anything, it gets used when SCTLR.M is enabled
>>>> L2 outer - enabled at boot time via the platform code (later, after MMU
>>>> ? ? ? ?was enabled).
>>>
>>> What is the reasoning behind this recommendation? Why the distinction
>>> between L2 being inner or outer. I don't see anything to this effect in
>>> the Cortex-A8 TRM? In fact the only recommendation I could find(section
>>> 8.3) is asking to set L2EN to 1 before setting C bit to 1 irrespective
>>> of inner/outer?
>>
>> Actually you have a good point. I was thinking about outer caches like
>> PL310 or L220 that Linux enables later during platform initialisation.
>> They could be left enabled or disabled at boot time but they must be
>> cleaned/invalidated (you would need to flush after disabling just to
>> make sure there are no further lines fetched; alternatively, there
>> shouldn't be any cacheable mappings).
>>
>> So I would say in the context of A8, just leave the L2 enabled in the
>> boot loader. As long as the MMU is disabled, it won't be used. The
>> alternative is to enable L2 slightly later in Linux in the platform
>> code to avoid SoC-dependent code in proc-v7.S.
>>
>> So in summary, either option is possible. You could take the approach
>> of the cache-l2x0.c - check whether it is already enabled and, if not,
>> issue an SMC from the platform code to enable the L2 cache.
>>
> if done late in kernel, wouldn't it violate "set L2EN to 1 before
> setting C bit to 1"

Not if you write the code sequence in such a way that it also switches
the C bit to 0 and back to 1. I think Nicolas suggested that the
platform code could go through the power management code. But I think
with some carefully coded asm code and cache flushing you could set
L2EN without a full MMU reset.

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 31, 2012 at 2:24 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On 31 January 2012 07:38, Shilimkar, Santosh <santosh.shilimkar@ti.com> wrote:
>> On Tue, Jan 31, 2012 at 1:01 PM, Catalin Marinas
>> <catalin.marinas@arm.com> wrote:
>>> On 31 January 2012 05:21, Aneesh V <aneesh@ti.com> wrote:
>>>> On Friday 27 January 2012 11:00 PM, Catalin Marinas wrote:
>>>>> On Fri, Jan 20, 2012 at 08:57:11AM +0000, Joe Woodward wrote:
>>>>>>> So I re-iterate that we need to have solution to this problem.
>>>>>>
>>>>>> ... I don't want to be a pain, but it seems to me that this dicussion
>>>>>> didn't reach a full conclussion?
>>>>>
>>>>> Probably not, because it depends on many variables. See below my take on
>>>>> this.
>>>>>
>>>>>> I think it was left with the open options being:
>>>>>> 1) Leave the L2/outer cache enabled in the bootloader (not ideal and
>>>>>> may cause problems with future devices)
>>>>>
>>>>> This depends on whether the L2 is inner or outer:
>>>>>
>>>>> L2 inner - leave it enabled in the boot loader
>>>>> L2 outer - leave it disabled in the boot loader
>>>>>
>>>>>> 2) Turn the L2/outer cache on for OMAP3 later in the kernel boot when
>>>>>> the device is known
>>>>>
>>>>> Same as above:
>>>>>
>>>>> L2 inner - don't do anything, it gets used when SCTLR.M is enabled
>>>>> L2 outer - enabled at boot time via the platform code (later, after MMU
>>>>> ? ? ? ?was enabled).
>>>>
>>>> What is the reasoning behind this recommendation? Why the distinction
>>>> between L2 being inner or outer. I don't see anything to this effect in
>>>> the Cortex-A8 TRM? In fact the only recommendation I could find(section
>>>> 8.3) is asking to set L2EN to 1 before setting C bit to 1 irrespective
>>>> of inner/outer?
>>>
>>> Actually you have a good point. I was thinking about outer caches like
>>> PL310 or L220 that Linux enables later during platform initialisation.
>>> They could be left enabled or disabled at boot time but they must be
>>> cleaned/invalidated (you would need to flush after disabling just to
>>> make sure there are no further lines fetched; alternatively, there
>>> shouldn't be any cacheable mappings).
>>>
>>> So I would say in the context of A8, just leave the L2 enabled in the
>>> boot loader. As long as the MMU is disabled, it won't be used. The
>>> alternative is to enable L2 slightly later in Linux in the platform
>>> code to avoid SoC-dependent code in proc-v7.S.
>>>
>>> So in summary, either option is possible. You could take the approach
>>> of the cache-l2x0.c - check whether it is already enabled and, if not,
>>> issue an SMC from the platform code to enable the L2 cache.
>>>
>> if done late in kernel, wouldn't it violate "set L2EN to 1 before
>> setting C bit to 1"
>
> Not if you write the code sequence in such a way that it also switches
> the C bit to 0 and back to 1. I think Nicolas suggested that the
> platform code could go through the power management code. But I think
> with some carefully coded asm code and cache flushing you could set
> L2EN without a full MMU reset.
>
That would need identity mapping and also correct cache flushing
as you said above. L2 enabled for A8 is fine but not sure about the
A15 where L2 is enabled with C bit. Then there are some settings for
ACTLR, L2ACCTRL and L2PFR which needs to be setup before
MMU is ebabled.

And then the errata's on A9, needing setting up dignostic register,
changing cache policy etc
Hopefully we don't get hit by those issues/erratas
in the window from initial MMU setup to this new sequence code.

Will this approach be ok for other cases I mentioned ?

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On 31 January 2012 09:05, Shilimkar, Santosh <santosh.shilimkar@ti.com> wrote:
> On Tue, Jan 31, 2012 at 2:24 PM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
>> On 31 January 2012 07:38, Shilimkar, Santosh <santosh.shilimkar@ti.com> wrote:
>>> On Tue, Jan 31, 2012 at 1:01 PM, Catalin Marinas
>>> <catalin.marinas@arm.com> wrote:
>>>> On 31 January 2012 05:21, Aneesh V <aneesh@ti.com> wrote:
>>>>> On Friday 27 January 2012 11:00 PM, Catalin Marinas wrote:
>>>>>> On Fri, Jan 20, 2012 at 08:57:11AM +0000, Joe Woodward wrote:
>>>>>>>> So I re-iterate that we need to have solution to this problem.
>>>>>>>
>>>>>>> ... I don't want to be a pain, but it seems to me that this dicussion
>>>>>>> didn't reach a full conclussion?
>>>>>>
>>>>>> Probably not, because it depends on many variables. See below my take on
>>>>>> this.
>>>>>>
>>>>>>> I think it was left with the open options being:
>>>>>>> 1) Leave the L2/outer cache enabled in the bootloader (not ideal and
>>>>>>> may cause problems with future devices)
>>>>>>
>>>>>> This depends on whether the L2 is inner or outer:
>>>>>>
>>>>>> L2 inner - leave it enabled in the boot loader
>>>>>> L2 outer - leave it disabled in the boot loader
>>>>>>
>>>>>>> 2) Turn the L2/outer cache on for OMAP3 later in the kernel boot when
>>>>>>> the device is known
>>>>>>
>>>>>> Same as above:
>>>>>>
>>>>>> L2 inner - don't do anything, it gets used when SCTLR.M is enabled
>>>>>> L2 outer - enabled at boot time via the platform code (later, after MMU
>>>>>> ? ? ? ?was enabled).
>>>>>
>>>>> What is the reasoning behind this recommendation? Why the distinction
>>>>> between L2 being inner or outer. I don't see anything to this effect in
>>>>> the Cortex-A8 TRM? In fact the only recommendation I could find(section
>>>>> 8.3) is asking to set L2EN to 1 before setting C bit to 1 irrespective
>>>>> of inner/outer?
>>>>
>>>> Actually you have a good point. I was thinking about outer caches like
>>>> PL310 or L220 that Linux enables later during platform initialisation.
>>>> They could be left enabled or disabled at boot time but they must be
>>>> cleaned/invalidated (you would need to flush after disabling just to
>>>> make sure there are no further lines fetched; alternatively, there
>>>> shouldn't be any cacheable mappings).
>>>>
>>>> So I would say in the context of A8, just leave the L2 enabled in the
>>>> boot loader. As long as the MMU is disabled, it won't be used. The
>>>> alternative is to enable L2 slightly later in Linux in the platform
>>>> code to avoid SoC-dependent code in proc-v7.S.
>>>>
>>>> So in summary, either option is possible. You could take the approach
>>>> of the cache-l2x0.c - check whether it is already enabled and, if not,
>>>> issue an SMC from the platform code to enable the L2 cache.
>>>>
>>> if done late in kernel, wouldn't it violate "set L2EN to 1 before
>>> setting C bit to 1"
>>
>> Not if you write the code sequence in such a way that it also switches
>> the C bit to 0 and back to 1. I think Nicolas suggested that the
>> platform code could go through the power management code. But I think
>> with some carefully coded asm code and cache flushing you could set
>> L2EN without a full MMU reset.
>>
> That would need identity mapping and also correct cache flushing
> as you said above. L2 enabled for A8 is fine but not sure about the
> A15 where L2 is enabled with C bit.

I don't see any problem here - why would you want to treat the L2
separately form the L1 cache? They are both inner caches, just treat
them as such - enabled when the C and M bits are set.

> Then there are some settings for
> ACTLR, L2ACCTRL and L2PFR which needs to be setup before
> MMU is ebabled.

These in general should go in the boot monitor/loader. There isn't
other easy clean solution if we want to keep the kernel booting
generic.

Maybe we could factor out the CPU-specific settings from proc-v*.S
into a separate arch/arm/boot/preload directory. We keep proc-v*.S
entirely generic and the implementation-defined bits setting in the
preload code. You could have an option to link the preload with the
kernel but we could recommend that people run this code from
boot-loader before invoking the kernel. This code would be dependent
on platform and chosen at build-time. But the actual kernel image
would be generic.

> And then the errata's on A9, needing setting up dignostic register,
> changing cache policy etc
> Hopefully we don't get hit by those issues/erratas
> in the window from initial MMU setup to this new sequence code.

We already enable the MMU in the decompressor and it doesn't have any
knowledge of CPU errata. But the preload code could solve this as
well.

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Tue, Jan 31, 2012 at 09:53:25AM +0000, Catalin Marinas wrote:
> Maybe we could factor out the CPU-specific settings from proc-v*.S
> into a separate arch/arm/boot/preload directory. We keep proc-v*.S
> entirely generic and the implementation-defined bits setting in the
> preload code. You could have an option to link the preload with the
> kernel but we could recommend that people run this code from
> boot-loader before invoking the kernel.

You've seen the resistance from uboot to stuff we do with the kernel,
this will be no different.  There's also quite a number of boot loaders
out there which aren't capable of running a binary, returning and then
loading and running the kernel.

Plus, of course, it's far from friendly towards TFTP setups.

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On 31 January 2012 10:10, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Tue, Jan 31, 2012 at 09:53:25AM +0000, Catalin Marinas wrote:
>> Maybe we could factor out the CPU-specific settings from proc-v*.S
>> into a separate arch/arm/boot/preload directory. We keep proc-v*.S
>> entirely generic and the implementation-defined bits setting in the
>> preload code. You could have an option to link the preload with the
>> kernel but we could recommend that people run this code from
>> boot-loader before invoking the kernel.
>
> You've seen the resistance from uboot to stuff we do with the kernel,
> this will be no different. ?There's also quite a number of boot loaders
> out there which aren't capable of running a binary, returning and then
> loading and running the kernel.
>
> Plus, of course, it's far from friendly towards TFTP setups.

For platforms that can't/don't want to modify the boot loader, they
could wrap (prepend) the kernel image with the preload code and they
just use that to kexec a distro generic kernel. I know it can go to
3-4 stages boot process on some platforms but you have the advantage
of a generic kernel image. If SoC vendors don't like the lengthier
boot process, they have the option of modifying the boot loader and
include the preload code that the kernel provides (or find a way to
load it together with the kernel at boot-time).

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Nicolas Pitre]

On Tue, 31 Jan 2012, Catalin Marinas wrote:

> Maybe we could factor out the CPU-specific settings from proc-v*.S
> into a separate arch/arm/boot/preload directory. We keep proc-v*.S
> entirely generic and the implementation-defined bits setting in the
> preload code. You could have an option to link the preload with the
> kernel but we could recommend that people run this code from
> boot-loader before invoking the kernel. This code would be dependent
> on platform and chosen at build-time. But the actual kernel image
> would be generic.

I'd recommend against that.  Again, the reason is not technical but 
rather has to do with the tendency to laziness of human beings.  So 
please let's not go there or it'll become the de facto standard.

Furthermore, if this is risky to leave a vulnerable window by activating 
some workarounds later during the boot process, then the risk won't go 
away entirely by moving those workarounds a bit earlier.  In which case 
the only real solution is to have them enabled by the bootloader before 
the cache is even turned on for the first time, and this is hardly Linux 
specific anyway.

The whole idea behind having a generic kernel is all about 
distributions.  If a generic kernel has to be distributed with a 
platform specific pre-kernel blob then we've gained nothing.  Same goes 
for the DTB.  Those platform specific blobs must be distributed _with_ 
the hardware and _not_ with the software distribution.  Having the dts 
files in the kernel right now is only a convenience for the transition 
to device tree.  Eventually they all will be removed from the tree when 
we get to a minimum of stability in the device tree implementation and 
the DTB will also have to come with the hardware along with the 
bootloader.


Nicolas

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

(sorry, I'm slow at replying, attending some internal ARM conference)

On 31 January 2012 18:09, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Tue, 31 Jan 2012, Catalin Marinas wrote:
>> Maybe we could factor out the CPU-specific settings from proc-v*.S
>> into a separate arch/arm/boot/preload directory. We keep proc-v*.S
>> entirely generic and the implementation-defined bits setting in the
>> preload code. You could have an option to link the preload with the
>> kernel but we could recommend that people run this code from
>> boot-loader before invoking the kernel. This code would be dependent
>> on platform and chosen at build-time. But the actual kernel image
>> would be generic.
>
> I'd recommend against that. ?Again, the reason is not technical but
> rather has to do with the tendency to laziness of human beings. ?So
> please let's not go there or it'll become the de facto standard.
...
> The whole idea behind having a generic kernel is all about
> distributions. ?If a generic kernel has to be distributed with a
> platform specific pre-kernel blob then we've gained nothing.

The problem we have with a generic kernel in this context (e.g. errata
bits) is that even though we use a read-check-write sequence for
setting various bits during __v*_setup, as long as the boot loader did
not set them the kernel will try a write access and fail. So with the
current scenario, there is no way for an OMAP platform to run a
generic kernel in non-secure mode (with all errata workarounds
enabled) without modifying the boot loader.

> Same goes
> for the DTB. ?Those platform specific blobs must be distributed _with_
> the hardware and _not_ with the software distribution. ?Having the dts
> files in the kernel right now is only a convenience for the transition
> to device tree.

We could do the same and move the bit enabling to separate repository :).

An alternative would be to also move those bits setting at a later
time in the kernel booting (and re-disabling/enabling the MMU) as we
don't have an easy way of figuring out whether the kernel is running
in secure or non-secure mode.

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Thu, Feb 02, 2012 at 02:32:03PM +0000, Catalin Marinas wrote:
> We could do the same and move the bit enabling to separate repository :).

We must certainly could do but that doesn't get around the errata
issues when you have things before the kernel (or before these blobs)
enabling things like the caches and MMU.

The only answer which assures complete system stability is for the
earliest reasonable point in the booting sequence to handle the errata,
before any of the potential errata scenarios occur.  For things like
working around L2 cache problems, that means before the L2 cache is
enabled for the first time.

If the boot loader enables the L2 cache, then _it_ has to take care of
the errata before it enables the L2 cache, and not some blob that it
loads.

If the boot loader enables the cache, and there are workarounds for buggy
cache behaviour, then the boot loader has to implement those errata
itself.

And so the list goes on.

I think the one issue which Santosh is justified about is: if that is
the case, why have the errata workarounds in the kernel in the first
place.  I agree - it makes no sense when we have things like the
decompressor enabling the caches.

So, I propose that we rip out all those work-arounds that are just
'set this bit in some register' at boot time from the kernel itself
right now, and reconsider how these are handled.

If that means boot loader people need to update their code to deal with
errata, because they want to use CPU features which have errata associated
with them, they then get to deal with the errata updates themselves, and
also have to carry the pain of dealing with running in non-secure mode.
Or, they will have to chose to avoid using those features.

But, having errata for things like DMB being faulty in the kernel after
things like the boot loader will _already_ have had to issue DMB
instructions, or for failing I-cache invalidate after we've already done
some I-cache invalidates (eg, boot loader or the decompressor) is quite
absurd.

So, I think we need to rip out a fair number of these errata from the
kernel itself; it's quite clear they're already being done far too late
in the system boot sequence.

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On 2 February 2012 14:49, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Feb 02, 2012 at 02:32:03PM +0000, Catalin Marinas wrote:
>> We could do the same and move the bit enabling to separate repository :).
>
> We must certainly could do but that doesn't get around the errata
> issues when you have things before the kernel (or before these blobs)
> enabling things like the caches and MMU.

That was my original point - allow such code to be built into a blob
that the boot loader can execute. But I got your point that not all
boot loaders are able to execute two binary images.

> The only answer which assures complete system stability is for the
> earliest reasonable point in the booting sequence to handle the errata,
> before any of the potential errata scenarios occur. ?For things like
> working around L2 cache problems, that means before the L2 cache is
> enabled for the first time.
>
> If the boot loader enables the L2 cache, then _it_ has to take care of
> the errata before it enables the L2 cache, and not some blob that it
> loads.
>
> If the boot loader enables the cache, and there are workarounds for buggy
> cache behaviour, then the boot loader has to implement those errata
> itself.

But as it was pointed out already, many of those are very rare
conditions that you can't actually trigger during the boot sequence.
There are other errata that require SMP for example and the boot
loader (or decompressor) doesn't care.

> So, I propose that we rip out all those work-arounds that are just
> 'set this bit in some register' at boot time from the kernel itself
> right now, and reconsider how these are handled.

Just don't remove them until we agree where we should *move* them.
They are still good documentation so far. I would say just disable the
workarounds in the kernel .config until we decide.

> But, having errata for things like DMB being faulty in the kernel after
> things like the boot loader will _already_ have had to issue DMB
> instructions, or for failing I-cache invalidate after we've already done
> some I-cache invalidates (eg, boot loader or the decompressor) is quite
> absurd.

As I said, most errata are really hard to trigger (but ARM cannot
guarantee that it is impossible, so they need to be published).

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Russell King - ARM Linux]

On Tue, Jan 31, 2012 at 02:35:51PM +0530, Shilimkar, Santosh wrote:
> On Tue, Jan 31, 2012 at 2:24 PM, Catalin Marinas
> <catalin.marinas@arm.com> wrote:
> > Not if you write the code sequence in such a way that it also switches
> > the C bit to 0 and back to 1. I think Nicolas suggested that the
> > platform code could go through the power management code. But I think
> > with some carefully coded asm code and cache flushing you could set
> > L2EN without a full MMU reset.
> 
> That would need identity mapping and also correct cache flushing
> as you said above. L2 enabled for A8 is fine but not sure about the
> A15 where L2 is enabled with C bit. Then there are some settings for
> ACTLR, L2ACCTRL and L2PFR which needs to be setup before
> MMU is ebabled.
> 
> And then the errata's on A9, needing setting up dignostic register,
> changing cache policy etc
> Hopefully we don't get hit by those issues/erratas
> in the window from initial MMU setup to this new sequence code.
> 
> Will this approach be ok for other cases I mentioned ?

Santosh,

It is very clear to me from the discussions, and the discussions I've
had with folk in ARM, that everyone thinks that the idea of putting
a platform specific hook in the early bring up code is insane.

Everyone seems to think that the place it is to be solved is either
in the boot loader or with a wrapper around the kernel.

One of my points that I brought up against it was the limited environment,
and the lack of stack at that point.  Let's look at what you're
proposing to do (thanks for the patch):

Your secure hook:
	stmia   r12, {r1-r11, r13-r14}

The stack you're writing to:

        .align  2
__v7_setup_stack:
        .space  4 * 11                          @ 11 registers

And it's layout in the kernel image:

c02ee75c <__v7_setup_stack>:
        ...
c02ee788:       00000c08        .word   0x00000c08
c02ee78c:       00000c09        .word   0x00000c09
c02ee790:       ff0a81a8        .word   0xff0a81a8
c02ee794:       40e040e0        .word   0x40e040e0

c02ee798 <gic_secondary_init>:
c02ee798:       e1a0c00d        mov     ip, sp
...

You're lucky that what you're _only_ overwriting is the two Cortex ID
values which have already been used, and not the following words, which
are the control register masks and the GIC initialization function.

Thank you for proving my point about why platform code can't be allowed
to get its grubby hands that early in the kernel boot - you are making
it more fragile by doing this stuff, because you haven't taken the time
to properly analyse the code before you've modified it.  As such, you
will see more silent boot failures.

So, as many people have said, we're not going to go down the route of
allowing platforms to hook into this code.  There's plenty of other
solutions to this problem in different parts of the system that would
be more than adequate, and would benefit everyone not just the kernel.

You're making bones about whether there's a window if you do it later
in the kernel.  You already have a window of opportunity for things to
have gone wrong and that's the zImage decompressor, which runs with
caches on.  So if you've managed to get to your existing code in the
v7 setup function, then you've already proven that it's fine.

Your continued resistance to every alternative suggestion but your
(broken) hook is getting extremely frustrating.  Please try some of
these other approaches which are being proposed.  Thanks.

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 31, 2012 at 3:26 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Tue, Jan 31, 2012 at 02:35:51PM +0530, Shilimkar, Santosh wrote:
>> On Tue, Jan 31, 2012 at 2:24 PM, Catalin Marinas
>> <catalin.marinas@arm.com> wrote:
>> > Not if you write the code sequence in such a way that it also switches
>> > the C bit to 0 and back to 1. I think Nicolas suggested that the
>> > platform code could go through the power management code. But I think
>> > with some carefully coded asm code and cache flushing you could set
>> > L2EN without a full MMU reset.
>>
>> That would need identity mapping and also correct cache flushing
>> as you said above. L2 enabled for A8 is fine but not sure about the
>> A15 where L2 is enabled with C bit. Then there are some settings for
>> ACTLR, L2ACCTRL and L2PFR which needs to be setup before
>> MMU is ebabled.
>>
>> And then the errata's on A9, needing setting up dignostic register,
>> changing cache policy etc
>> Hopefully we don't get hit by those issues/erratas
>> in the window from initial MMU setup to this new sequence code.
>>
>> Will this approach be ok for other cases I mentioned ?
>
> Santosh,
>
> It is very clear to me from the discussions, and the discussions I've
> had with folk in ARM, that everyone thinks that the idea of putting
> a platform specific hook in the early bring up code is insane.
>
> Everyone seems to think that the place it is to be solved is either
> in the boot loader or with a wrapper around the kernel.
>
> One of my points that I brought up against it was the limited environment,
> and the lack of stack at that point. ?Let's look at what you're
> proposing to do (thanks for the patch):
>
> Your secure hook:
> ? ? ? ?stmia ? r12, {r1-r11, r13-r14}
>
> The stack you're writing to:
>
> ? ? ? ?.align ?2
> __v7_setup_stack:
> ? ? ? ?.space ?4 * 11 ? ? ? ? ? ? ? ? ? ? ? ? ?@ 11 registers
>
> And it's layout in the kernel image:
>
> c02ee75c <__v7_setup_stack>:
> ? ? ? ?...
> c02ee788: ? ? ? 00000c08 ? ? ? ?.word ? 0x00000c08
> c02ee78c: ? ? ? 00000c09 ? ? ? ?.word ? 0x00000c09
> c02ee790: ? ? ? ff0a81a8 ? ? ? ?.word ? 0xff0a81a8
> c02ee794: ? ? ? 40e040e0 ? ? ? ?.word ? 0x40e040e0
>
> c02ee798 <gic_secondary_init>:
> c02ee798: ? ? ? e1a0c00d ? ? ? ?mov ? ? ip, sp
> ...
>
> You're lucky that what you're _only_ overwriting is the two Cortex ID
> values which have already been used, and not the following words, which
> are the control register masks and the GIC initialization function.
>
> Thank you for proving my point about why platform code can't be allowed
> to get its grubby hands that early in the kernel boot - you are making
> it more fragile by doing this stuff, because you haven't taken the time
> to properly analyse the code before you've modified it. ?As such, you
> will see more silent boot failures.
>
> So, as many people have said, we're not going to go down the route of
> allowing platforms to hook into this code. ?There's plenty of other
> solutions to this problem in different parts of the system that would
> be more than adequate, and would benefit everyone not just the kernel.
>
Sure. The marco has missed those details. Thanks for the
useful information. At least I can fix the stack size for internal
use.

> You're making bones about whether there's a window if you do it later
> in the kernel. ?You already have a window of opportunity for things to
> have gone wrong and that's the zImage decompressor, which runs with
> caches on. ?So if you've managed to get to your existing code in the
> v7 setup function, then you've already proven that it's fine.
>
I see your point.

> Your continued resistance to every alternative suggestion but your
> (broken) hook is getting extremely frustrating. ?Please try some of
> these other approaches which are being proposed. ?Thanks.

I was just trying to get more information about the alternatives and
trying to bring out possible issues. I know for sure that boot-loader is
not an option since we have had many instances where it has not worked.

I also understand that patching early common code is going to be tricky
and of-course against the single zImage.

So the option mentioned by Nicolas and Catalin about 1:1 mapping
and configuring the registers in platform specific code was looking
a way forward. So was asking more questions about whether this can
work in all cases. Specifically for A15. As per Catalin's email, it is
best to handle those cases before MMU is enabled with
boot-monitor or pre-load code.

I am not against trying option. Just trying to understand what is the
best option we have.

Regards
santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Nicolas Pitre]

On Tue, 31 Jan 2012, Shilimkar, Santosh wrote:

> On Tue, Jan 31, 2012 at 3:26 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > So, as many people have said, we're not going to go down the route of
> > allowing platforms to hook into this code. ?There's plenty of other
> > solutions to this problem in different parts of the system that would
> > be more than adequate, and would benefit everyone not just the kernel.
> >
> Sure. The marco has missed those details. Thanks for the
> useful information. At least I can fix the stack size for internal
> use.

If you intend to ignore our advises and continue with the quick and 
dirty solution for your internal tree, then at least don't brag on a 
public list about it.

> > Your continued resistance to every alternative suggestion but your
> > (broken) hook is getting extremely frustrating. ?Please try some of
> > these other approaches which are being proposed. ?Thanks.
> 
> I was just trying to get more information about the alternatives and
> trying to bring out possible issues. I know for sure that boot-loader is
> not an option since we have had many instances where it has not worked.
> 
> I also understand that patching early common code is going to be tricky
> and of-course against the single zImage.
> 
> So the option mentioned by Nicolas and Catalin about 1:1 mapping
> and configuring the registers in platform specific code was looking
> a way forward. So was asking more questions about whether this can
> work in all cases. Specifically for A15.

So far this is apparently your best option.  And luckily the code to 
create a 1:1 mapping, turn off MMU, then turn MMU back on and tear down 
the 1:1 mapping already exists.  All you need to do is to insert the 
cache workaround activation in the middle of that instead of shutting 
the CPU down.

> As per Catalin's email, it is best to handle those cases before MMU is 
> enabled with boot-monitor or pre-load code.

These days bootloaders are turning the MMU on for themselves, so you'll 
need pre-load code for the bootloader already.  Either that, or the 
issue is not necessarily _that_ critical in which case you can afford to 
enable your workaround a bit later during the kernel boot.


Nicolas

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Shilimkar, Santosh]

On Tue, Jan 31, 2012 at 11:57 PM, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Tue, 31 Jan 2012, Shilimkar, Santosh wrote:

[...]

>> I also understand that patching early common code is going to be tricky
>> and of-course against the single zImage.
>>
>> So the option mentioned by Nicolas and Catalin about 1:1 mapping
>> and configuring the registers in platform specific code was looking
>> a way forward. So was asking more questions about whether this can
>> work in all cases. Specifically for A15.
>
> So far this is apparently your best option. ?And luckily the code to
> create a 1:1 mapping, turn off MMU, then turn MMU back on and tear down
> the 1:1 mapping already exists. ?All you need to do is to insert the
> cache workaround activation in the middle of that instead of shutting
> the CPU down.
>
thanks.

>> As per Catalin's email, it is best to handle those cases before MMU is
>> enabled with boot-monitor or pre-load code.
>
> These days bootloaders are turning the MMU on for themselves, so you'll
> need pre-load code for the bootloader already. ?Either that, or the
> issue is not necessarily _that_ critical in which case you can afford to
> enable your workaround a bit later during the kernel boot.
>
This is indeed a good point.
The mainline u-boot(denx) now a days have MMU support so we need to have
the necessary WA before that code anyways.

We will attempt the 1:1 mapping option as well but to be 100 % safe for
the previous 3 statges of MMU ON ( 1. Bootloader, 2. De-compressor
and 3. Kernel setup), looks like moving all of this security settings
before all 3 stages is most appropriate.

Thanks for the good discussion.

Regards
Santosh

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Grazvydas Ignotas]

On Tue, Jan 17, 2012 at 3:39 PM, Catalin Marinas
<catalin.marinas@arm.com> wrote:
> On Tue, Jan 17, 2012 at 12:40:54PM +0000, Shilimkar, Santosh wrote:
>> Well the L2 can be configured as inner or outer, so above
>> alone won't work.
>>
>> Boot-loader disabling L2 cache ( all caches) ?is still right thing
>> and that's what kernel expect.
>>
>> Since the early kernel code can't be patches for A8, may be
>> delaying L2 enabled would work.
>
> Ah, I missed the fact that the L2 is an inner cache on OMAP3+A8.

At least on my OMAP3 board it's not. There is L2 AUX control register
"L2 inner" bit that's always cleared here. I know outer cacheablility
bits in page table descriptors have effect here, we have tested
performace before..


-- 
Gra?vydas

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Catalin Marinas]

On Tue, Jan 17, 2012 at 12:27:25PM +0000, Aneesh V wrote:
> Hi Catalin,
> 
> On Tuesday 17 January 2012 05:41 PM, Catalin Marinas wrote:
> > On Tue, Jan 17, 2012 at 08:54:44AM +0000, Joe Woodward wrote:
> >> So, is the upshot of this that the kernel isn't going to be in a
> >> position to enable the L2/outer cache on OMAP3 (due to the need for
> >> hacky/unmaintainable code)?
> >>
> >> Hence the bootloader/uBoot had better leave it enabled...
> >
> > It could but the Linux decompressor needs to be aware and either flush
> > the L2 (more difficult as it doesn't have all the device information) or
> 
> Cortex-A8 is aware of L2$ and can flush it, isn't it?

As I said to Santosh, I only had the outer cache in mind (e.g. PL310).
There is no extra configuration needed in the kernel decompressor in
this case.

> > set the page table attributes to outer non-cacheable (TEX[2:0] = 0b100).
> 
> If the above is right, this is not needed right?

Correct, since L2 is inner cache.

> > The latter may still not work if there are stale L2 cache lines left by
> > U-Boot (and that's always possible unless U-Boot also uses outer
> > non-cacheable memory attributes).
> 
> U-Boot flushes the caches before disabling it. On top of it, it does an
> invalidate after the disabling the caches to cover some corner case.
> So, it's guaranteed that there won't be any stale entries in L2 after
> U-Boot.
> 
> So, we could as well leave L2$ enabled (and invalidated) and caches
> disabled globally after U-Boot.

Yes.

> But I thought enabling L2$ again in kernel is the cleaner solution.

Why? It gets enabled via SCTLR.M together with L1.

-- 
Catalin

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Aneesh V]

On Tuesday 17 January 2012 07:11 PM, Catalin Marinas wrote:
> On Tue, Jan 17, 2012 at 12:27:25PM +0000, Aneesh V wrote:
>> Hi Catalin,
>>
>> On Tuesday 17 January 2012 05:41 PM, Catalin Marinas wrote:
>>> On Tue, Jan 17, 2012 at 08:54:44AM +0000, Joe Woodward wrote:
>>>> So, is the upshot of this that the kernel isn't going to be in a
>>>> position to enable the L2/outer cache on OMAP3 (due to the need for
>>>> hacky/unmaintainable code)?
>>>>
>>>> Hence the bootloader/uBoot had better leave it enabled...
>>>
>>> It could but the Linux decompressor needs to be aware and either flush
>>> the L2 (more difficult as it doesn't have all the device information) or
>>
>> Cortex-A8 is aware of L2$ and can flush it, isn't it?
>
> As I said to Santosh, I only had the outer cache in mind (e.g. PL310).
> There is no extra configuration needed in the kernel decompressor in
> this case.
>
>>> set the page table attributes to outer non-cacheable (TEX[2:0] = 0b100).
>>
>> If the above is right, this is not needed right?
>
> Correct, since L2 is inner cache.
>
>>> The latter may still not work if there are stale L2 cache lines left by
>>> U-Boot (and that's always possible unless U-Boot also uses outer
>>> non-cacheable memory attributes).
>>
>> U-Boot flushes the caches before disabling it. On top of it, it does an
>> invalidate after the disabling the caches to cover some corner case.
>> So, it's guaranteed that there won't be any stale entries in L2 after
>> U-Boot.
>>
>> So, we could as well leave L2$ enabled (and invalidated) and caches
>> disabled globally after U-Boot.
>
> Yes.
>
>> But I thought enabling L2$ again in kernel is the cleaner solution.
>
> Why? It gets enabled via SCTLR.M together with L1.

Well, technically I don't see any issue. But after all, it's a
bootloader dependency that could be avoided. We could not blame U-Boot
or any other bootloader for disabling L2$ given [1]

[1] http://www.arm.linux.org.uk/developer/booting.php#5

OMAP3 L2/outer cache enabled in kernel (after being disabled by uBoot)?

[Måns Rullgård]

Catalin Marinas <catalin.marinas@arm.com> writes:

> On Tue, Jan 17, 2012 at 12:27:25PM +0000, Aneesh V wrote:
> 
>> But I thought enabling L2$ again in kernel is the cleaner solution.
>
> Why? It gets enabled via SCTLR.M together with L1.

Not if the L2EN bit of the auxiliary control register was cleared, and
u-boot messes with this bit.

-- 
M?ns Rullg?rd
mans at mansr.com