[U-Boot] AM335x: enabling datacache in SPL slows down system?

[U-Boot] AM335x: enabling datacache in SPL slows down system?

[Bas van den Berg]

I'm trying to optimize the boottime for an AM335x based board. Currently
we're loading the kernel from SPL directly (falcon mode). Loading the kernel
from flash to ram takes roughly 3 seconds.
When doing this from U-boot itself with 'nand read .., it only takes 1.8 seconds.
It seems that U-boot has data_cache enabled, while SPL did not.

When enabling the data cache with dcache_enable(), loading takes 3.5 seconds,
so even slower! The enabling itself does not take a significant amount of time.

Anyone have any idea why enabling the data cache slows SPL down?
Or does U-boot do anything else that I'm missing?

Bas

[U-Boot] AM335x: enabling datacache in SPL slows down system?

[Jeroen Hofstee]

Hallo Bas,

On 06/20/2013 10:05 AM, Bas van den Berg wrote:
> I'm trying to optimize the boottime for an AM335x based board. Currently
> we're loading the kernel from SPL directly (falcon mode). Loading the kernel
> from flash to ram takes roughly 3 seconds.
> When doing this from U-boot itself with 'nand read .., it only takes 1.8 seconds.
> It seems that U-boot has data_cache enabled, while SPL did not.
>
> When enabling the data cache with dcache_enable(), loading takes 3.5 seconds,
> so even slower! The enabling itself does not take a significant amount of time.
>
> Anyone have any idea why enabling the data cache slows SPL down?
> Or does U-boot do anything else that I'm missing?

For an am3517 I added the code below to spl_board_init to
enable caches in SPL. It adds the SRAM region to the mmu,
by an obvious hack, but it works. I don't remember by heart
why I change / update gd->tlb_addr. The used address is
somewhere in the end of DRAM.

No guarantees this is related to your issue (or to work at all).

---
Groet,
Jeroen

	if (!spl_start_uboot()) {
		dram_init_banksize();

		/* reserve TLB table (4k) */
		gd->tlb_addr = 0x8fff0000;
		debug("TLB table at: %08lx\n", gd->tlb_addr);

		/* add sram (64k actually, but the cp15 counts in MiBs) */
		gd->bd->bi_dram[1].start = CONFIG_SPL_TEXT_BASE;
		gd->bd->bi_dram[1].size =  (1 << 20);

		/* Enable caches */
		enable_caches();
	}

[U-Boot] AM335x: enabling datacache in SPL slows down system?

[Wolfgang Denk]

Dear "Bas van den Berg",

In message <20130620080523.aba38cb4@mail.altenpts.nl> you wrote:
> 
> I'm trying to optimize the boottime for an AM335x based board. Currently
> we're loading the kernel from SPL directly (falcon mode). Loading the kernel
> from flash to ram takes roughly 3 seconds.
> When doing this from U-boot itself with 'nand read .., it only takes 1.8 seconds.
> It seems that U-boot has data_cache enabled, while SPL did not.
> 
> When enabling the data cache with dcache_enable(), loading takes 3.5 seconds,
> so even slower! The enabling itself does not take a significant amount of time.

We observed the very same issue with the verry first prototype
implementation of Falcon mode on an AM3517 based board.  At this time
we did not have time nor resources to figure out what exactly was
causing this issue, but it's interesting to see that 1) it
still persists and 2) it also happens on AM335x

> Anyone have any idea why enabling the data cache slows SPL down?
> Or does U-boot do anything else that I'm missing?

@Tom: do you have any additional input what we should check?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
An Elephant is a mouse with an Operating System.              - Knuth

[U-Boot] AM335x: enabling datacache in SPL slows down system?

[Jeroen Hofstee]

Hello Wolfgang,

On 06/21/2013 07:33 AM, Wolfgang Denk wrote:
> We observed the very same issue with the verry first prototype
> implementation of Falcon mode on an AM3517 based board. [..]
For the am3517 the only thing needed is to add SRAM
to the known memory. See [1], that should change the
cache misses (which likely cause the extra delays) to
actual caches. U-boot only adds the detected memory
and since U-boot is running from DRAM and SPL is not,
the difference seems obvious.

Regards,
Jeroen

[1] http://lists.denx.de/pipermail/u-boot/2013-June/156949.html

[U-Boot] AM335x: enabling datacache in SPL slows down system?

[Tom Rini]

On Fri, Jun 21, 2013 at 08:46:47PM +0200, Jeroen Hofstee wrote:

> Hello Wolfgang,
> 
> On 06/21/2013 07:33 AM, Wolfgang Denk wrote:
> >We observed the very same issue with the verry first prototype
> >implementation of Falcon mode on an AM3517 based board. [..]
> For the am3517 the only thing needed is to add SRAM
> to the known memory. See [1], that should change the
> cache misses (which likely cause the extra delays) to
> actual caches. U-boot only adds the detected memory
> and since U-boot is running from DRAM and SPL is not,
> the difference seems obvious.

This would be my guess as folks have done functional but not
mainlineable changes in this direction, as well as DMA, before.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130621/f31f281f/attachment.pgp>

[U-Boot] AM335x: enabling datacache in SPL slows down system?

[Bas van den Berg]

Hi Jeroen,

After debugging a bit more, I found that in our code, we didn't set the RAM size
correctly (so no real RAM section was setup for caching). Fixing this fixed our
time to 2.4 seconds (U-boot still took 1.8 sec).

Adding your SRAM patch (modified, since AM335x has more SRAM),
The load time dropped from 2.4 -> 1.9 seconds. So adding this small area saved
half a second!
I write off the remaing difference (1.8 vs 1.9 seconds) to code differences between
our SPL and U-boot nand functions.

Thanks,
Bas

  _____  

From: Tom Rini [mailto:trini at ti.com]
To: Jeroen Hofstee [mailto:jeroen at myspectrum.nl]
Cc: Wolfgang Denk [mailto:wd at denx.de], bas.van.den.berg at alten.nl, u-boot at lists.denx.de
Sent: Fri, 21 Jun 2013 20:56:53 +0100
Subject: Re: [U-Boot] AM335x: enabling datacache in SPL slows down system?

On Fri, Jun 21, 2013 at 08:46:47PM +0200, Jeroen Hofstee wrote:
  
  > Hello Wolfgang,
  > 
  > On 06/21/2013 07:33 AM, Wolfgang Denk wrote:
  > >We observed the very same issue with the verry first prototype
  > >implementation of Falcon mode on an AM3517 based board. [..]
  > For the am3517 the only thing needed is to add SRAM
  > to the known memory. See [1], that should change the
  > cache misses (which likely cause the extra delays) to
  > actual caches. U-boot only adds the detected memory
  > and since U-boot is running from DRAM and SPL is not,
  > the difference seems obvious.
  
  This would be my guess as folks have done functional but not
  mainlineable changes in this direction, as well as DMA, before.
  
  -- 
  Tom
    

[U-Boot] AM335x: enabling datacache in SPL slows down system?

[Jeroen Hofstee]

Hello Bas,

On 06/24/2013 10:01 AM, Bas van den Berg wrote:
> After debugging a bit more, I found that in our code, we didn't set the RAM size
> correctly (so no real RAM section was setup for caching). Fixing this fixed our
> time to 2.4 seconds (U-boot still took 1.8 sec).
>
> Adding your SRAM patch (modified, since AM335x has more SRAM),
> The load time dropped from 2.4 -> 1.9 seconds. So adding this small area saved
> half a second!
> I write off the remaing difference (1.8 vs 1.9 seconds) to code differences between
> our SPL and U-boot nand functions.
Thanks for reporting and checking that adding the SRAM region is
needed as well (I suppose you already verified that -O2 cannot get
rid of the last difference).

Regards,
Jeroen

ps: the consensus on this mailinglist is not to top post..