RE: Memory performance / Cache problem

RE: Memory performance / Cache problem

[epsi]

Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-203) ) #1 Mon Oct 12 08:30:58 CEST 2009
CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: OMAP3 EVM
Memory policy: ECC disabled, Data cache writeback
OMAP3430 ES2.1
SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512                                                                                               

I see, you get the message about L2 Cache, which I don't have
Do you know how to enable this?
Shoudn't the kernel configure all this things - not rely on bootloader?
I am using the U-Boot 2008.10 (TIs PSP)
The old 22 kernel is independet from the uboot in this issue.

Thanks
Steffen
-- 
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser

RE: Memory performance / Cache problem

[Dasgupta, Romit]

Please update to the latest HEAD on the linux-omap pm branch. In the gitweb it shows
b7ecc865c5f0885fae4c4401fa78a24084f45c40

Thanks,
-Romit


>-----Original Message-----
>From: linux-omap-owner@vger.kernel.org [mailto:linux-omap-
>owner@vger.kernel.org] On Behalf Of epsi@gmx.de
>Sent: Monday, October 12, 2009 2:08 PM
>To: linux-omap@vger.kernel.org
>Subject: RE: Memory performance / Cache problem
>
>Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 (Sourcery G++ Lite 2009q1-
>203) ) #1 Mon Oct 12 08:30:58 CEST 2009
>CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
>CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
>Machine: OMAP3 EVM
>Memory policy: ECC disabled, Data cache writeback
>OMAP3430 ES2.1
>SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000
>Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
>
>I see, you get the message about L2 Cache, which I don't have
>Do you know how to enable this?
>Shoudn't the kernel configure all this things - not rely on bootloader?
>I am using the U-Boot 2008.10 (TIs PSP)
>The old 22 kernel is independet from the uboot in this issue.
>
>Thanks
>Steffen
>--
>Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
>sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser
>--
>To unsubscribe from this list: send the line "unsubscribe linux-omap" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Memory performance / Cache problem

[Dasgupta, Romit]

I am sorry, the code mentioned is from the Android Public Git tree not LO PM. Sorry for the confusion. 

>-----Original Message-----
>From: linux-omap-owner@vger.kernel.org [mailto:linux-omap-
>owner@vger.kernel.org] On Behalf Of Dasgupta, Romit
>Sent: Monday, October 12, 2009 2:38 PM
>To: epsi@gmx.de; linux-omap@vger.kernel.org
>Subject: RE: Memory performance / Cache problem
>
>Please update to the latest HEAD on the linux-omap pm branch. In the gitweb it
>shows
>b7ecc865c5f0885fae4c4401fa78a24084f45c40
>
>Thanks,
>-Romit
>
>
>>-----Original Message-----
>>From: linux-omap-owner@vger.kernel.org [mailto:linux-omap-
>>owner@vger.kernel.org] On Behalf Of epsi@gmx.de
>>Sent: Monday, October 12, 2009 2:08 PM
>>To: linux-omap@vger.kernel.org
>>Subject: RE: Memory performance / Cache problem
>>
>>Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 (Sourcery G++ Lite
>2009q1-
>>203) ) #1 Mon Oct 12 08:30:58 CEST 2009
>>CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
>>CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
>>Machine: OMAP3 EVM
>>Memory policy: ECC disabled, Data cache writeback
>>OMAP3430 ES2.1
>>SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000
>>Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
>>
>>I see, you get the message about L2 Cache, which I don't have
>>Do you know how to enable this?
>>Shoudn't the kernel configure all this things - not rely on bootloader?
>>I am using the U-Boot 2008.10 (TIs PSP)
>>The old 22 kernel is independet from the uboot in this issue.
>>
>>Thanks
>>Steffen
>>--
>>Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
>>sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser
>>--
>>To unsubscribe from this list: send the line "unsubscribe linux-omap" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-omap" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Memory performance / Cache problem

[Premi, Sanjeev]

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org 
> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of epsi@gmx.de
> Sent: Monday, October 12, 2009 2:08 PM
> To: linux-omap@vger.kernel.org
> Subject: RE: Memory performance / Cache problem
> 
> Linux version 2.6.31 (star@localhost) (gcc version 4.3.3 
> (Sourcery G++ Lite 2009q1-203) ) #1 Mon Oct 12 08:30:58 CEST 2009
> CPU: ARMv7 Processor [411fc082] revision 2 (ARMv7), cr=10c53c7f
> CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
> Machine: OMAP3 EVM
> Memory policy: ECC disabled, Data cache writeback
> OMAP3430 ES2.1
> SRAM: Mapped pa 0x40200000 to va 0xe3000000 size: 0x100000
> Built 1 zonelists in Zone order, mobility grouping on.  Total 
> pages: 32512                                                  
>                                              
> 
> I see, you get the message about L2 Cache, which I don't have
> Do you know how to enable this?
> Shoudn't the kernel configure all this things - not rely on 
> bootloader?
> I am using the U-Boot 2008.10 (TIs PSP)
> The old 22 kernel is independet from the uboot in this issue.

Can you upgrade to a newer u-boot? Either from the PSP release
OR u-boot tree hosted at git.denx.de (atleast 2009.03)?

Also, it will be good to see the sample program you are using.

~sanjeev

> 
> Thanks
> Steffen
> -- 
> Jetzt kostenlos herunterladen: Internet Explorer 8 und 
> Mozilla Firefox 3.5 -
> sicherer, schneller und einfacher! 
> http://portal.gmx.net/de/go/atbrowser
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: RE: Memory performance / Cache problem

[epsi]

> 
> Can you upgrade to a newer u-boot? Either from the PSP release
> OR u-boot tree hosted at git.denx.de (atleast 2009.03)?
> 
> Also, it will be good to see the sample program you are using.
> 
> ~sanjeev
> 

There is no newer u-boot from TI available. There is a SDK 02.01.03.11
but it contains the same uboot 2008.10 with the only addition of the second generation of EVM boards with another network chip.

So I checked the uboot from git, but this doesn't support Microns NAND Flash anymore. It is just working with ONENAND.

I found a patch which shows the L2 Cache status while kernel boot and implemented it : L2 Cache seems to be already enabled - so this is not the reason.

So any other ideas? 
-- 
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RE: Memory performance / Cache problem

[Woodruff, Richard]

> There is no newer u-boot from TI available. There is a SDK 02.01.03.11
> but it contains the same uboot 2008.10 with the only addition of the second
> generation of EVM boards with another network chip.
>
> So I checked the uboot from git, but this doesn't support Microns NAND Flash
> anymore. It is just working with ONENAND.
>
> I found a patch which shows the L2 Cache status while kernel boot and
> implemented it : L2 Cache seems to be already enabled - so this is not the
> reason.
>
> So any other ideas?

Are you confident your memory bus isn't running at 1/2 speed?

I recall there was a couple day window during wtbu kernel upgrades where memory bus speed with pm was running 1/2 speed after kernel started up.  This was somewhat a side effect of constraints frame work and a regression in forward porting. It seems unlikely psp kernel would have shipped with this bug but its something to check. This would match your results.

If your memcpy() is neon based then I might be worried about l1neon-caching effects along with factors of (exlcusive-l1-l2-read-allocate cache + pld not being effective on l1 only l2).

Which memcpy test are you using? Something in lmbench or just one you wrote.  Generally results are a little hard to interpret with exclusive cache behavior in 3430's r1px core.  3630's r3p2 core gives more traditional results as exclusive feature has been removed by arm.

If you have the ability using Lauterbach + per file will allow internal space dump which will show all critical parameters during test.  It's a 1 minute check for someone who has done it before to ensure the few parameters needed are in line.  I can send an example off line of how to do capture.  I won't have time to expand on all relevant parameters.

Regards,
Richard W.

Re: RE: RE: Memory performance / Cache problem

[epsi]

Mem clock is both times 166MHz. I don't know whether are differences in cycle access and timing, but memclock is fine.

Following Siarhei hints of initialize the buffers (around 1.2 MByte each)
I get different results in 22kernel for use of
malloc alone
memcpy =   473.764, loop4 =   448.430, loop1 =   102.770, rand =    29.641
calloc alone
memcpy =   405.947, loop4 =   361.550, loop1 =    95.441, rand =    21.853
malloc+memset:
memcpy =   239.294, loop4 =   188.617, loop1 =    80.871, rand =     4.726

In 31kernel all 3 measures are about the same (unfortunatly low) level of malloc+memset in 22.

First of all: What performance can be expected?
Does 22 make failures if it is so much faster?
Can the later kernels get a boost in memory handling?

I used a standard memcpy (think this is glib and hence not neonbased)? 
To be neonbased I guess it has to be recompiled?

How can I find out that neon and cache settings are ok?
Using a Omap3530 on EVM board

Unfortunatly I don't have a Lauterbach, just a Spectrum Digital which works only until Linux kernel is booting.

Best regards
Steffen


-------- Original-Nachricht --------
> Datum: Wed, 14 Oct 2009 08:59:05 -0500
> Von: "Woodruff, Richard" <r-woodruff2@ti.com>
> An: "epsi@gmx.de" <epsi@gmx.de>, "Premi, Sanjeev" <premi@ti.com>, "linux-omap@vger.kernel.org" <linux-omap@vger.kernel.org>
> Betreff: RE: RE: Memory performance / Cache problem

> > There is no newer u-boot from TI available. There is a SDK 02.01.03.11
> > but it contains the same uboot 2008.10 with the only addition of the
> second
> > generation of EVM boards with another network chip.
> >
> > So I checked the uboot from git, but this doesn't support Microns NAND
> Flash
> > anymore. It is just working with ONENAND.
> >
> > I found a patch which shows the L2 Cache status while kernel boot and
> > implemented it : L2 Cache seems to be already enabled - so this is not
> the
> > reason.
> >
> > So any other ideas?
> 
> Are you confident your memory bus isn't running at 1/2 speed?
> 
> I recall there was a couple day window during wtbu kernel upgrades where
> memory bus speed with pm was running 1/2 speed after kernel started up. 
> This was somewhat a side effect of constraints frame work and a regression in
> forward porting. It seems unlikely psp kernel would have shipped with this
> bug but its something to check. This would match your results.
> 
> If your memcpy() is neon based then I might be worried about
> l1neon-caching effects along with factors of (exlcusive-l1-l2-read-allocate cache + pld
> not being effective on l1 only l2).
> 
> Which memcpy test are you using? Something in lmbench or just one you
> wrote.  Generally results are a little hard to interpret with exclusive cache
> behavior in 3430's r1px core.  3630's r3p2 core gives more traditional
> results as exclusive feature has been removed by arm.
> 
> If you have the ability using Lauterbach + per file will allow internal
> space dump which will show all critical parameters during test.  It's a 1
> minute check for someone who has done it before to ensure the few parameters
> needed are in line.  I can send an example off line of how to do capture.  I
> won't have time to expand on all relevant parameters.
> 
> Regards,
> Richard W.

-- 
Neu: GMX DSL bis 50.000 kBit/s und 200,- Euro Startguthaben!
http://portal.gmx.net/de/go/dsl02

RE: RE: RE: Memory performance / Cache problem

[Woodruff, Richard]

> From: epsi@gmx.de [mailto:epsi@gmx.de]
> Sent: Wednesday, October 14, 2009 9:49 AM
> To: Woodruff, Richard; linux-omap@vger.kernel.org; Premi, Sanjeev
> Subject: Re: RE: RE: Memory performance / Cache problem
>
> Mem clock is both times 166MHz. I don't know whether are differences in cycle
> access and timing, but memclock is fine.

How did you physically verify this?

> Following Siarhei hints of initialize the buffers (around 1.2 MByte each)
> I get different results in 22kernel for use of
> malloc alone
> memcpy =   473.764, loop4 =   448.430, loop1 =   102.770, rand =    29.641
> calloc alone
> memcpy =   405.947, loop4 =   361.550, loop1 =    95.441, rand =    21.853
> malloc+memset:
> memcpy =   239.294, loop4 =   188.617, loop1 =    80.871, rand =     4.726
>
> In 31kernel all 3 measures are about the same (unfortunatly low) level of
> malloc+memset in 22.

Yes aligned buffers can make a difference.  But probably more so for small copies.  Of course you must touch the memory or mprotect() it so its faulted in, but indications are you have done this.

> I used a standard memcpy (think this is glib and hence not neonbased)?
> To be neonbased I guess it has to be recompiled?

The version of glibc in use can make a difference.  CodeSourcery in 2009 release added PLD's to mem operations.  This can give a good benefit.  It might be you have optimized library in one case and a non-optimized in another.

Regards,
Richard W.

Re: RE: RE: RE: Memory performance / Cache problem

[epsi]

> > Mem clock is both times 166MHz. I don't know whether are differences in
> cycle
> > access and timing, but memclock is fine.
> 
> How did you physically verify this?

Oszi show 166MHz, also the kernel message about freq are in both kernels the same.

> > Following Siarhei hints of initialize the buffers (around 1.2 MByte
> each)
> > I get different results in 22kernel for use of
> > malloc alone
> > memcpy =   473.764, loop4 =   448.430, loop1 =   102.770, rand =   
> 29.641
> > calloc alone
> > memcpy =   405.947, loop4 =   361.550, loop1 =    95.441, rand =   
> 21.853
> > malloc+memset:
> > memcpy =   239.294, loop4 =   188.617, loop1 =    80.871, rand =    
> 4.726
> >
> > In 31kernel all 3 measures are about the same (unfortunatly low) level
> of
> > malloc+memset in 22.
> 
> Yes aligned buffers can make a difference.  But probably more so for small
> copies.  Of course you must touch the memory or mprotect() it so its
> faulted in, but indications are you have done this.

Mh, alignment (to an address) is done with malloc already. Probably you mean something different. I don't understand the difference. For me is malloc+memset=calloc. 
I'll send you the benchmark code, if you like. 

> > I used a standard memcpy (think this is glib and hence not neonbased)?
> > To be neonbased I guess it has to be recompiled?
> 
> The version of glibc in use can make a difference.  CodeSourcery in 2009
> release added PLD's to mem operations.  This can give a good benefit.  It
> might be you have optimized library in one case and a non-optimized in
> another.

In both kernels I used the same rootfs (via NFS). Indeed I used CS2009q1 and its libs, but we are talking about factor 2..6. This must be something serious.

What is your feeling? Does the 22 something strange or are the newer kernels slower that they have to be.

Would be interesting to see results on other Omap3 boards with both old an new kernels.

Best regards
Steffen
-- 
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RE: RE: RE: Memory performance / Cache problem

[Woodruff, Richard]

> From: epsi@gmx.de [mailto:epsi@gmx.de]
> Sent: Wednesday, October 14, 2009 12:23 PM
> To: Woodruff, Richard; Premi, Sanjeev; linux-omap@vger.kernel.org
> Subject: Re: RE: RE: RE: Memory performance / Cache problem

> > Yes aligned buffers can make a difference.  But probably more so for small
> > copies.  Of course you must touch the memory or mprotect() it so its
> > faulted in, but indications are you have done this.
>
> Mh, alignment (to an address) is done with malloc already. Probably you mean
> something different. I don't understand the difference. For me is
> malloc+memset=calloc.
> I'll send you the benchmark code, if you like.

Ok, if it compiles I could try on sdp3430 or sdp3630.

Alignment to a cache line is always best.  This is 64 bytes in A8.  Better yet, being 4K aligned is a good thing to reduce MMU effects.

> In both kernels I used the same rootfs (via NFS). Indeed I used CS2009q1 and
> its libs, but we are talking about factor 2..6. This must be something serious.

2009 pay version has optimized thrumb2 and arm mode libraries you download separately.  I don't recall if free/lite version integrated this.

> What is your feeling? Does the 22 something strange or are the newer kernels
> slower that they have to be.

There are a lot of differences between 22 kernel and current ones.  First things I'd check would be around MMU settings, special ARM cp15 memory control regs, then omap memory and clock settings.  Also some bad device could undercut you.  Always good to cat /proc/interrupts and make sure something isn't spewing.

> Would be interesting to see results on other Omap3 boards with both old an new
> kernels.

I've not noticed anything on sdp on somewhat recent kernels.

Regards,
Richard W.

Re: Memory performance / Cache problem

[Siarhei Siamashka]

On Wednesday 14 October 2009 17:48:39 ext epsi@gmx.de wrote:
> Mem clock is both times 166MHz. I don't know whether are differences in
> cycle access and timing, but memclock is fine.
>
> Following Siarhei hints of initialize the buffers (around 1.2 MByte each)
> I get different results in 22kernel for use of
> malloc alone
> memcpy =   473.764, loop4 =   448.430, loop1 =   102.770, rand =    29.641
> calloc alone
> memcpy =   405.947, loop4 =   361.550, loop1 =    95.441, rand =    21.853
> malloc+memset:
> memcpy =   239.294, loop4 =   188.617, loop1 =    80.871, rand =     4.726
>
> In 31kernel all 3 measures are about the same (unfortunatly low) level of
> malloc+memset in 22.
>
> First of all: What performance can be expected?
> Does 22 make failures if it is so much faster?
> Can the later kernels get a boost in memory handling?

What you see is just a (fake) performance boost because you have a single
physical page shared between all the virtual pages in the source buffer. So
you get no cache misses on read operations and everything seems fast.

This is unlikely to happen on real use, and it does not reflect real memory
performance. So the benchmark is inadequate.

You can get some basic information here:
http://en.wikipedia.org/wiki/Copy-on-write

Regarding the difference in behavior between .22 and recent kernels. It may be
some regression in copy-on-write implementation, or just some change done on
purpose. That is assuming that the userspace stuff was identical in both
tests.

-- 
Best regards,
Siarhei Siamashka

RE: Memory performance / Cache problem

[Woodruff, Richard]

> From: Siarhei Siamashka [mailto:siarhei.siamashka@nokia.com]
> Sent: Wednesday, October 14, 2009 12:37 PM
> To: ext epsi@gmx.de

> What you see is just a (fake) performance boost because you have a single
> physical page shared between all the virtual pages in the source buffer. So
> you get no cache misses on read operations and everything seems fast.
>
> This is unlikely to happen on real use, and it does not reflect real memory
> performance. So the benchmark is inadequate.

Yep, benchmark is only useful so far.  If you control factors it can be useful but it's far from 1-1, to extrapolate it to something meaningful at system level.

You can actually get even 'better' numbers if you take the DDR part geometry into mind and SDRC (sdram-controller) scheduler.  Our silicon validation people report out of this world numbers for very specific test cases.  These are component tests for the memory controller to make sure it behaves.  If you alternate between open banks you can do really fast operations.

A good amount of that test is not practical to count on at HLOS level.  It can help explain some anomalies and help in designing a fair test.

Regards,
Richard W.

Re: Memory performance / Cache problem

[epsi]

> On Wednesday 14 October 2009 17:48:39 ext epsi@gmx.de wrote:
> > Mem clock is both times 166MHz. I don't know whether are differences in
> > cycle access and timing, but memclock is fine.
> >
> > Following Siarhei hints of initialize the buffers (around 1.2 MByte
> each)
> > I get different results in 22kernel for use of
> > malloc alone
> > memcpy =   473.764, loop4 =   448.430, loop1 =   102.770, rand =   
> 29.641
> > calloc alone
> > memcpy =   405.947, loop4 =   361.550, loop1 =    95.441, rand =   
> 21.853
> > malloc+memset:
> > memcpy =   239.294, loop4 =   188.617, loop1 =    80.871, rand =    
> 4.726
> >
> > In 31kernel all 3 measures are about the same (unfortunatly low) level
> of
> > malloc+memset in 22.
> >
> > First of all: What performance can be expected?
> > Does 22 make failures if it is so much faster?
> > Can the later kernels get a boost in memory handling?
> 
> What you see is just a (fake) performance boost because you have a single
> physical page shared between all the virtual pages in the source buffer.
> So
> you get no cache misses on read operations and everything seems fast.
> 
> This is unlikely to happen on real use, and it does not reflect real
> memory
> performance. So the benchmark is inadequate.

> 
> You can get some basic information here:
> http://en.wikipedia.org/wiki/Copy-on-write
> 
> Regarding the difference in behavior between .22 and recent kernels. It
> may be
> some regression in copy-on-write implementation, or just some change done
> on
> purpose. That is assuming that the userspace stuff was identical in both
> tests.
> 

Ok, understand the difference if the memory is uninitialised.
But why there is the difference in "malloc + memset" and "calloc"?
In both cases the memory will be cleared.


-- 
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser