[U-Boot] OMAP3 performance regression in 2011.12

[U-Boot] OMAP3 performance regression in 2011.12

[Joe Woodward]

Commit "armv7: disable L2 cache in cleanup_before_linux()" on 6th Dec 2011 by Aneesh V adds the following:

arch/arm/cpu/armv7/cpu.c:cleanup_before_linux()

...
v7_out_cache_disable();
...

The commit message implies this change was to make booting reliable on OMAP4 by disabling L2 cache before jumping to Linux.

However, when running with a stock 3.2 Linux kernel on an OMAP3 it has the effect of massively reducing system performance (when running using an OMAP3-
only 3.2 Linux Kernel on a GUSMTIX Overo OMAP3530).

Therefore, I assume this means that the kernel isn't turning the L2 cache back on for an OMAP3 (at least with my kernel build)!

So, my question is...

Are there any Kconfig options in Linux that will re-enable the L2 cache (something obvious that I've missed), or is this commit just bad-news for OMAP3?

Cheers,
Joe

[U-Boot] OMAP3 performance regression in 2011.12

[Tom Rini]

On Mon, Jan 9, 2012 at 3:27 AM, Joe Woodward <jw@terrafix.co.uk> wrote:
> Commit "armv7: disable L2 cache in cleanup_before_linux()" on 6th Dec 2011 by Aneesh V adds the following:
>
> arch/arm/cpu/armv7/cpu.c:cleanup_before_linux()
>
> ...
> v7_out_cache_disable();
> ...
>
> The commit message implies this change was to make booting reliable on OMAP4 by disabling L2 cache before jumping to Linux.
>
> However, when running with a stock 3.2 Linux kernel on an OMAP3 it has the effect of massively reducing system performance (when running using an OMAP3-
> only 3.2 Linux Kernel on a GUSMTIX Overo OMAP3530).
>
> Therefore, I assume this means that the kernel isn't turning the L2 cache back on for an OMAP3 (at least with my kernel build)!
>
> So, my question is...
>
> Are there any Kconfig options in Linux that will re-enable the L2 cache (something obvious that I've missed), or is this commit just bad-news for OMAP3?

Are you certain that this is the commit that's causing your problem?
The kernel is responsible for turning the cache back on and has for a
long time, iirc.

-- 
Tom

[U-Boot] OMAP3 performance regression in 2011.12

[Joe Woodward]

I'm fairly certain...

If I take the 2011.12 uBoot release the kernel takes about twice the time to boot (compared to 2011.09), and the device is noticably slower.

Then if I comment out the v7_out_cache_disable() line in cpu.c and rebuild uBoot then everything speeds up again.

I thought the kernel would turn on the cache again too...

Is there any easy way from userspace to determine if the cache is on?

I did a bit of Googling and found:
http://www.spinics.net/lists/arm-kernel/msg50083.html

It may be that the kernel is re-enabling the L1 cache, but expecting L2 to be on?
Or the way v7_out_cache_disable() disables L2 is not compatible with the way the kernel expects to re-enable it?

Cheers,
Joe

-----Original Message-----
From: Tom Rini <tom.rini@gmail.com>
To: Joe Woodward <jw@terrafix.co.uk>
Cc: u-boot at lists.denx.de
Date: Mon, 9 Jan 2012 08:11:07 -0700
Subject: Re: [U-Boot] OMAP3 performance regression in 2011.12

> On Mon, Jan 9, 2012 at 3:27 AM, Joe Woodward <jw@terrafix.co.uk> wrote:
> > Commit "armv7: disable L2 cache in cleanup_before_linux()" on 6th Dec
> 2011 by Aneesh V adds the following:
> >
> > arch/arm/cpu/armv7/cpu.c:cleanup_before_linux()
> >
> > ...
> > v7_out_cache_disable();
> > ...
> >
> > The commit message implies this change was to make booting reliable
> on OMAP4 by disabling L2 cache before jumping to Linux.
> >
> > However, when running with a stock 3.2 Linux kernel on an OMAP3 it
> has the effect of massively reducing system performance (when running
> using an OMAP3-
> > only 3.2 Linux Kernel on a GUSMTIX Overo OMAP3530).
> >
> > Therefore, I assume this means that the kernel isn't turning the L2
> cache back on for an OMAP3 (at least with my kernel build)!
> >
> > So, my question is...
> >
> > Are there any Kconfig options in Linux that will re-enable the L2
> cache (something obvious that I've missed), or is this commit just
> bad-news for OMAP3?
> 
> Are you certain that this is the commit that's causing your problem?
> The kernel is responsible for turning the cache back on and has for a
> long time, iirc.
> 
> -- 
> Tom

[U-Boot] OMAP3 performance regression in 2011.12

[Joe Woodward]

> -----Original Message-----
> From: Tom Rini <tom.rini@gmail.com>
> To: Joe Woodward <jw@terrafix.co.uk>
> Cc: u-boot at lists.denx.de
> Date: Mon, 9 Jan 2012 08:11:07 -0700
> Subject: Re: [U-Boot] OMAP3 performance regression in 2011.12
> 
> > On Mon, Jan 9, 2012 at 3:27 AM, Joe Woodward <jw@terrafix.co.uk>
> wrote:
> > > Commit "armv7: disable L2 cache in cleanup_before_linux()" on 6th
> Dec
> > 2011 by Aneesh V adds the following:
> > >
> > > arch/arm/cpu/armv7/cpu.c:cleanup_before_linux()
> > >
> > > ...
> > > v7_out_cache_disable();
> > > ...
> > >
> > > The commit message implies this change was to make booting reliable
> > on OMAP4 by disabling L2 cache before jumping to Linux.
> > >
> > > However, when running with a stock 3.2 Linux kernel on an OMAP3 it
> > has the effect of massively reducing system performance (when running
> > using an OMAP3-
> > > only 3.2 Linux Kernel on a GUSMTIX Overo OMAP3530).
> > >
> > > Therefore, I assume this means that the kernel isn't turning the L2
> > cache back on for an OMAP3 (at least with my kernel build)!
> > >
> > > So, my question is...
> > >
> > > Are there any Kconfig options in Linux that will re-enable the L2
> > cache (something obvious that I've missed), or is this commit just
> > bad-news for OMAP3?
> > 
> > Are you certain that this is the commit that's causing your problem?
> > The kernel is responsible for turning the cache back on and has for a
> > long time, iirc.
> > 
> > -- 
> > Tom

(apologies for previous top posting, wasn't paying attention to what I was doing!)

I'm fairly certain...

If I take the 2011.12 uBoot release the kernel takes about twice the time to boot (compared to 2011.09), and the device is noticably slower.

Then if I comment out the v7_out_cache_disable() line in cpu.c and rebuild uBoot then everything speeds up again.
 
I thought the kernel would turn on the cache again too...

Is there any easy way from userspace to determine if the cache is on?

I did a bit of Googling and found:
http://www.spinics.net/lists/arm-kernel/msg50064.html
http://www.spinics.net/lists/arm-kernel/msg50083.html

It may be that the kernel is re-enabling the L1 cache, but expecting L2 to be on?
Or the way v7_out_cache_disable() disables L2 is not compatible with the way the kernel expects to re-enable it?

Also, in the Linux there seem to be OMAP4 specific functions for re-enabling the L2 cache (omap4-common.c:omap_l2_cache_init()), but none for OMAP3. I'm 
assuming this is because up to now OMAP3 is assumed to have the L2 left enabled? Either that, or there is some generic Cortex-A8 method for enabling the L2 
cache in the kernel soures that I've not found...

Cheers,
Joe

> 
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] OMAP3 performance regression in 2011.12

[Joe Woodward]

-----Original Message-----
From: Philip Balister <philip@balister.org>
To: Joe Woodward <jw@terrafix.co.uk>
Date: Fri, 13 Jan 2012 15:53:37 -0500
Subject: Re: OMAP3 performance regression in 2011.12

> Did you find out anymore info on this? I may have a customer who is
> seeing this on a 3.o based overo.
> 

(CC'ing in the mailing list again).

Sadly not, no more replies to this.

I think either this change needs reverting (but that'll break OMAP4) or at least encasing in #ifdefs so that it isn't applied for OMAP3.

I was going to ask on the Linux OMAP mailing list if the L2/outer cache is enabled on OMAP3 in Linux but haven't got round to it yet!

Cheers,
Joe

> Philip
> 
> On 01/09/2012 10:48 AM, Joe Woodward wrote:
> >> -----Original Message-----
> >> From: Tom Rini <tom.rini@gmail.com>
> >> To: Joe Woodward <jw@terrafix.co.uk>
> >> Cc: u-boot at lists.denx.de
> >> Date: Mon, 9 Jan 2012 08:11:07 -0700
> >> Subject: Re: [U-Boot] OMAP3 performance regression in 2011.12
> >>
> >>> On Mon, Jan 9, 2012 at 3:27 AM, Joe Woodward <jw@terrafix.co.uk>
> >> wrote:
> >>>> Commit "armv7: disable L2 cache in cleanup_before_linux()" on 6th
> >> Dec
> >>> 2011 by Aneesh V adds the following:
> >>>>
> >>>> arch/arm/cpu/armv7/cpu.c:cleanup_before_linux()
> >>>>
> >>>> ...
> >>>> v7_out_cache_disable();
> >>>> ...
> >>>>
> >>>> The commit message implies this change was to make booting
> reliable
> >>> on OMAP4 by disabling L2 cache before jumping to Linux.
> >>>>
> >>>> However, when running with a stock 3.2 Linux kernel on an OMAP3 it
> >>> has the effect of massively reducing system performance (when
> running
> >>> using an OMAP3-
> >>>> only 3.2 Linux Kernel on a GUSMTIX Overo OMAP3530).
> >>>>
> >>>> Therefore, I assume this means that the kernel isn't turning the
> L2
> >>> cache back on for an OMAP3 (at least with my kernel build)!
> >>>>
> >>>> So, my question is...
> >>>>
> >>>> Are there any Kconfig options in Linux that will re-enable the L2
> >>> cache (something obvious that I've missed), or is this commit just
> >>> bad-news for OMAP3?
> >>>
> >>> Are you certain that this is the commit that's causing your
> problem?
> >>> The kernel is responsible for turning the cache back on and has for
> a
> >>> long time, iirc.
> >>>
> >>> -- 
> >>> Tom
> > 
> > (apologies for previous top posting, wasn't paying attention to what
> I was doing!)
> > 
> > I'm fairly certain...
> > 
> > If I take the 2011.12 uBoot release the kernel takes about twice the
> time to boot (compared to 2011.09), and the device is noticably slower.
> > 
> > Then if I comment out the v7_out_cache_disable() line in cpu.c and
> rebuild uBoot then everything speeds up again.
> >  
> > I thought the kernel would turn on the cache again too...
> > 
> > Is there any easy way from userspace to determine if the cache is on?
> > 
> > I did a bit of Googling and found:
> > http://www.spinics.net/lists/arm-kernel/msg50064.html
> > http://www.spinics.net/lists/arm-kernel/msg50083.html
> > 
> > It may be that the kernel is re-enabling the L1 cache, but expecting
> L2 to be on?
> > Or the way v7_out_cache_disable() disables L2 is not compatible with
> the way the kernel expects to re-enable it?
> > 
> > Also, in the Linux there seem to be OMAP4 specific functions for
> re-enabling the L2 cache (omap4-common.c:omap_l2_cache_init()), but
> none for OMAP3. I'm 
> > assuming this is because up to now OMAP3 is assumed to have the L2
> left enabled? Either that, or there is some generic Cortex-A8 method
> for enabling the L2 
> > cache in the kernel soures that I've not found...
> > 
> > Cheers,
> > Joe
> > 
> >>
> >> _______________________________________________
> >> U-Boot mailing list
> >> U-Boot at lists.denx.de
> >> http://lists.denx.de/mailman/listinfo/u-boot
> 

[U-Boot] OMAP3 performance regression in 2011.12

[Philip Balister]

On 01/16/2012 04:03 AM, Joe Woodward wrote:
> -----Original Message-----
> From: Philip Balister <philip@balister.org>
> To: Joe Woodward <jw@terrafix.co.uk>
> Date: Fri, 13 Jan 2012 15:53:37 -0500
> Subject: Re: OMAP3 performance regression in 2011.12
> 
>> Did you find out anymore info on this? I may have a customer who is
>> seeing this on a 3.o based overo.
>>
> 
> (CC'ing in the mailing list again).
> 
> Sadly not, no more replies to this.
> 
> I think either this change needs reverting (but that'll break OMAP4) or at least encasing in #ifdefs so that it isn't applied for OMAP3.
> 
> I was going to ask on the Linux OMAP mailing list if the L2/outer cache is enabled on OMAP3 in Linux but haven't got round to it yet!
> 
> Cheers,
> Joe
> 
>> Philip
>>
>> On 01/09/2012 10:48 AM, Joe Woodward wrote:
>>>> -----Original Message-----
>>>> From: Tom Rini <tom.rini@gmail.com>
>>>> To: Joe Woodward <jw@terrafix.co.uk>
>>>> Cc: u-boot at lists.denx.de
>>>> Date: Mon, 9 Jan 2012 08:11:07 -0700
>>>> Subject: Re: [U-Boot] OMAP3 performance regression in 2011.12
>>>>
>>>>> On Mon, Jan 9, 2012 at 3:27 AM, Joe Woodward <jw@terrafix.co.uk>
>>>> wrote:
>>>>>> Commit "armv7: disable L2 cache in cleanup_before_linux()" on 6th
>>>> Dec
>>>>> 2011 by Aneesh V adds the following:
>>>>>>
>>>>>> arch/arm/cpu/armv7/cpu.c:cleanup_before_linux()
>>>>>>
>>>>>> ...
>>>>>> v7_out_cache_disable();

I built u-boot with this change reverted and compared the amount of time
it took to build sip from source.

Reverting the change improved compile time by about a factor of four, so
it looks like the kernel does not properly re-enable the L2 cache.

See:

http://comments.gmane.org/gmane.linux.ports.arm.omap/69560

For discussion on the linux-omap list.

Philip

>>>>>> ...
>>>>>>
>>>>>> The commit message implies this change was to make booting
>> reliable
>>>>> on OMAP4 by disabling L2 cache before jumping to Linux.
>>>>>>
>>>>>> However, when running with a stock 3.2 Linux kernel on an OMAP3 it
>>>>> has the effect of massively reducing system performance (when
>> running
>>>>> using an OMAP3-
>>>>>> only 3.2 Linux Kernel on a GUSMTIX Overo OMAP3530).
>>>>>>
>>>>>> Therefore, I assume this means that the kernel isn't turning the
>> L2
>>>>> cache back on for an OMAP3 (at least with my kernel build)!
>>>>>>
>>>>>> So, my question is...
>>>>>>
>>>>>> Are there any Kconfig options in Linux that will re-enable the L2
>>>>> cache (something obvious that I've missed), or is this commit just
>>>>> bad-news for OMAP3?
>>>>>
>>>>> Are you certain that this is the commit that's causing your
>> problem?
>>>>> The kernel is responsible for turning the cache back on and has for
>> a
>>>>> long time, iirc.
>>>>>
>>>>> -- 
>>>>> Tom
>>>
>>> (apologies for previous top posting, wasn't paying attention to what
>> I was doing!)
>>>
>>> I'm fairly certain...
>>>
>>> If I take the 2011.12 uBoot release the kernel takes about twice the
>> time to boot (compared to 2011.09), and the device is noticably slower.
>>>
>>> Then if I comment out the v7_out_cache_disable() line in cpu.c and
>> rebuild uBoot then everything speeds up again.
>>>  
>>> I thought the kernel would turn on the cache again too...
>>>
>>> Is there any easy way from userspace to determine if the cache is on?
>>>
>>> I did a bit of Googling and found:
>>> http://www.spinics.net/lists/arm-kernel/msg50064.html
>>> http://www.spinics.net/lists/arm-kernel/msg50083.html
>>>
>>> It may be that the kernel is re-enabling the L1 cache, but expecting
>> L2 to be on?
>>> Or the way v7_out_cache_disable() disables L2 is not compatible with
>> the way the kernel expects to re-enable it?
>>>
>>> Also, in the Linux there seem to be OMAP4 specific functions for
>> re-enabling the L2 cache (omap4-common.c:omap_l2_cache_init()), but
>> none for OMAP3. I'm 
>>> assuming this is because up to now OMAP3 is assumed to have the L2
>> left enabled? Either that, or there is some generic Cortex-A8 method
>> for enabling the L2 
>>> cache in the kernel soures that I've not found...
>>>
>>> Cheers,
>>> Joe
>>>
>>>>
>>>> _______________________________________________
>>>> U-Boot mailing list
>>>> U-Boot at lists.denx.de
>>>> http://lists.denx.de/mailman/listinfo/u-boot
>>

[U-Boot] OMAP3 performance regression in 2011.12

[Andreas Müller]

On Monday, January 16, 2012 05:34:12 PM Philip Balister wrote:
> 
> I built u-boot with this change reverted and compared the amount of time
> it took to build sip from source.
> 
> Reverting the change improved compile time by about a factor of four, so
> it looks like the kernel does not properly re-enable the L2 cache.
> 
> See:
> 
> http://comments.gmane.org/gmane.linux.ports.arm.omap/69560
> 
> For discussion on the linux-omap list.
> 
> Philip
> 
FYI: included in meta-gumstix today

Andreas

[U-Boot] OMAP3 performance regression in 2011.12

[Aneesh V]

Hi Joe,

On Monday 09 January 2012 09:18 PM, Joe Woodward wrote:

<snip>

>
> (apologies for previous top posting, wasn't paying attention to what I was doing!)
>
> I'm fairly certain...
>
> If I take the 2011.12 uBoot release the kernel takes about twice the time to boot (compared to 2011.09), and the device is noticably slower.
>
> Then if I comment out the v7_out_cache_disable() line in cpu.c and rebuild uBoot then everything speeds up again.
>
> I thought the kernel would turn on the cache again too...
>
> Is there any easy way from userspace to determine if the cache is on?

It won't be easy I believe, unless you have a debugger that allows you
to see CP15 registers.

>
> I did a bit of Googling and found:
> http://www.spinics.net/lists/arm-kernel/msg50064.html
> http://www.spinics.net/lists/arm-kernel/msg50083.html
>
> It may be that the kernel is re-enabling the L1 cache, but expecting L2 to be on?

Ideally kernel should be enabling L2 too. But looks like L2 was enabled
by ROM code itself in OMAP3, but D-cache disabled globally. So, it gets
enabled as soon as the D-cache is enabled globally.

L2$ in OMAP3 is a bit tricky. The cache is known to ARM but
enabling/disabling it and affecting secure entries needs ROM
assistance. So, while ARMv7 generic code can flush L2, we need OMAP
specific code to enable/disable it.

So, before my patch and before cache-support came to U-boot L2$ always
remained enabled. Since de-compressor's flush indeed flushed L2$ we
didn't get into any coherency problems.

Let the discussion in l-o conclude. I think fixing it in kernel is the
right thing to do. But if that's not possible, we can have a patch in
U-boot to work around it.

br,
Aneesh

[U-Boot] OMAP3 performance regression in 2011.12

[Måns Rullgård]

Aneesh V <aneesh@ti.com> writes:

> Hi Joe,
>
> On Monday 09 January 2012 09:18 PM, Joe Woodward wrote:
>
>> If I take the 2011.12 uBoot release the kernel takes about twice the
>> time to boot (compared to 2011.09), and the device is noticably
>> slower.
>>
>> Then if I comment out the v7_out_cache_disable() line in cpu.c and
>> rebuild uBoot then everything speeds up again.
>>
>> I thought the kernel would turn on the cache again too...
>> [...]
>> I did a bit of Googling and found:
>> http://www.spinics.net/lists/arm-kernel/msg50064.html
>> http://www.spinics.net/lists/arm-kernel/msg50083.html
>>
>> It may be that the kernel is re-enabling the L1 cache, but expecting
>> L2 to be on?
>
> Ideally kernel should be enabling L2 too. But looks like L2 was enabled
> by ROM code itself in OMAP3, but D-cache disabled globally. So, it gets
> enabled as soon as the D-cache is enabled globally.
>
> L2$ in OMAP3 is a bit tricky. The cache is known to ARM but
> enabling/disabling it and affecting secure entries needs ROM
> assistance. So, while ARMv7 generic code can flush L2, we need OMAP
> specific code to enable/disable it.

On OMAP3 ES2 and later, the L2EN bit in the auxiliary control register
is banked between secure and non-secure modes, so there is no need for
ROM calls to enable/disable the L2$ on these chips.

-- 
M?ns Rullg?rd
mans at mansr.com

[U-Boot] OMAP3 performance regression in 2011.12

[Aneesh V]

On Tuesday 17 January 2012 08:21 PM, M?ns Rullg?rd wrote:
> Aneesh V<aneesh@ti.com>  writes:
>
>> Hi Joe,
>>
>> On Monday 09 January 2012 09:18 PM, Joe Woodward wrote:
>>
>>> If I take the 2011.12 uBoot release the kernel takes about twice the
>>> time to boot (compared to 2011.09), and the device is noticably
>>> slower.
>>>
>>> Then if I comment out the v7_out_cache_disable() line in cpu.c and
>>> rebuild uBoot then everything speeds up again.
>>>
>>> I thought the kernel would turn on the cache again too...
>>> [...]
>>> I did a bit of Googling and found:
>>> http://www.spinics.net/lists/arm-kernel/msg50064.html
>>> http://www.spinics.net/lists/arm-kernel/msg50083.html
>>>
>>> It may be that the kernel is re-enabling the L1 cache, but expecting
>>> L2 to be on?
>>
>> Ideally kernel should be enabling L2 too. But looks like L2 was enabled
>> by ROM code itself in OMAP3, but D-cache disabled globally. So, it gets
>> enabled as soon as the D-cache is enabled globally.
>>
>> L2$ in OMAP3 is a bit tricky. The cache is known to ARM but
>> enabling/disabling it and affecting secure entries needs ROM
>> assistance. So, while ARMv7 generic code can flush L2, we need OMAP
>> specific code to enable/disable it.
>
> On OMAP3 ES2 and later, the L2EN bit in the auxiliary control register
> is banked between secure and non-secure modes, so there is no need for
> ROM calls to enable/disable the L2$ on these chips.
>
Yes, but IIRC, there was an erratum around it in some ARM revisions (I
think 3430 ES2 was affected) and the workaround was to keep both the
banked bits at the same value always. So, to keep things simple and
working for all revisions, I try to change both together. In the U-Boot
code where this is done I have added a comment on this.