Re: Beagle PM hangs

Re: Beagle PM hangs

[Kevin Hilman]

Peter Reid <ppeter.reid@gmail.com> writes:

> Hi Kevin,
>    I get hangs after boot on beagle with latest PM tree i have quite a
> few scripts that run on boot.
>
>    have you seen this before?

I have not, but I believe Koen has see something similar.  I've mainly
been testing just a minimal rootfs and a minimal set of drivers on
Beagle so I can hit full-chip off in idle.

It would help if you could desribe your setup in more detail.  What
kernel config are you using, what rootfs are you using?  Does it hang
in the same spot every time?  If so, what is going on when it hangs?
Have you written anything to any of the /sys/power/* files?

Kevin

Re: Beagle PM hangs

[Kevin Hilman]

Kevin Hilman <khilman@deeprootsystems.com> writes:

> Peter Reid <ppeter.reid@gmail.com> writes:
>
>> Hi Kevin,
>>    I get hangs after boot on beagle with latest PM tree i have quite a
>> few scripts that run on boot.
>>
>>    have you seen this before?
>
> I have not, but I believe Koen has see something similar.  I've mainly
> been testing just a minimal rootfs and a minimal set of drivers on
> Beagle so I can hit full-chip off in idle.
>
> It would help if you could desribe your setup in more detail.  What
> kernel config are you using, what rootfs are you using?  Does it hang
> in the same spot every time?  If so, what is going on when it hangs?
> Have you written anything to any of the /sys/power/* files?
>

Just a thought, do you still see these problems when using the CPUfreq
performance governor instead of the ondemand governor?

It could be pointing to some bugs in the switching of operating points.

Kevin

Re: Beagle PM hangs

[Sriram V]

Hi,
  I have done some testing on a custom board. it uses uart2
  I also see serial hang issues here. I have a minimum defconfig with
 only i2c enabled. no cpuidle/cpufreq.

  This is what i try to do -

  # echo -n mem > /sys/power/state

  I wait for a min and try to wakeup - It works.

  If i wait a little longer before waking it up say, 3 or 4 minutes.
It does not wakeup.

  I tried the same disabling clocks when idle The result is the same.

   # echo -n 1 > /sys/power/sleep_while_idle
   # echo -n 1 > /sys/power/clocks_off_while_idle
   # echo -n mem > /sys/power/state

   With the above,  i see all domains hitting retention.
   I have tried both keypads and uart activity as my wakeup
   sources.
   Am trying to debug this. why the serial is dead if
   there is no activity for a longer time.


Regards,
sriram









On Wed, Jan 28, 2009 at 11:16 PM, Kevin Hilman
<khilman@deeprootsystems.com> wrote:
> Kevin Hilman <khilman@deeprootsystems.com> writes:
>
>> Peter Reid <ppeter.reid@gmail.com> writes:
>>
>>> Hi Kevin,
>>>    I get hangs after boot on beagle with latest PM tree i have quite a
>>> few scripts that run on boot.
>>>
>>>    have you seen this before?
>>
>> I have not, but I believe Koen has see something similar.  I've mainly
>> been testing just a minimal rootfs and a minimal set of drivers on
>> Beagle so I can hit full-chip off in idle.
>>
>> It would help if you could desribe your setup in more detail.  What
>> kernel config are you using, what rootfs are you using?  Does it hang
>> in the same spot every time?  If so, what is going on when it hangs?
>> Have you written anything to any of the /sys/power/* files?
>>
>
> Just a thought, do you still see these problems when using the CPUfreq
> performance governor instead of the ondemand governor?
>
> It could be pointing to some bugs in the switching of operating points.
>
> Kevin
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: Beagle PM hangs

[Woodruff, Richard]

>   I wait for a min and try to wakeup - It works.
>
>   If i wait a little longer before waking it up say, 3 or 4 minutes.
> It does not wakeup.

That is consistent with your memory not being placed into self refresh properly.

5 general bugs I've seen over time:
        sw bug:
                - ctrl sequence issue before sleep
                - CKE balls not muxed correctly
                - crash on unexpected wake path
        chip bug:
                - (3430 >= es3 requires a errata sequence to work)
        board bug:
                - hooking ddr-chip-CKE wrongly or a way OMAP doesn't support like sharing between dies.

Regards,
Richard W.

Re: Beagle PM hangs

[Kevin Hilman]

Richard,

"Woodruff, Richard" <r-woodruff2@ti.com> writes:

>>   I wait for a min and try to wakeup - It works.
>>
>>   If i wait a little longer before waking it up say, 3 or 4 minutes.
>> It does not wakeup.
>
> That is consistent with your memory not being placed into self refresh properly.
>
> 5 general bugs I've seen over time:
>         sw bug:
>                 - ctrl sequence issue before sleep
>                 - CKE balls not muxed correctly
>                 - crash on unexpected wake path
>         chip bug:
>                 - (3430 >= es3 requires a errata sequence to work)
>         board bug:
>                 - hooking ddr-chip-CKE wrongly or a way OMAP doesn't support like sharing between dies.

Another possibility is that the the memory timings for the custom
board are not set correctly.

Richard, Do you have pointers to CDP/zoom code that has a workaround
for the 3430 > es3 errata as well as the 'ctrl sequence issue' you
mentioned?

Kevin

RE: Beagle PM hangs

[Pandita, Vikram]

>-----Original Message-----
>
>Richard,
>
>"Woodruff, Richard" <r-woodruff2@ti.com> writes:
>
>>>   I wait for a min and try to wakeup - It works.
>>>
>>>   If i wait a little longer before waking it up say, 3 or 4 minutes.
>>> It does not wakeup.
>>
>> That is consistent with your memory not being placed into self refresh properly.
>>
>> 5 general bugs I've seen over time:
>>         sw bug:
>>                 - ctrl sequence issue before sleep
>>                 - CKE balls not muxed correctly
>>                 - crash on unexpected wake path
>>         chip bug:
>>                 - (3430 >= es3 requires a errata sequence to work)
>>         board bug:
>>                 - hooking ddr-chip-CKE wrongly or a way OMAP doesn't support like sharing between
>dies.
>
>Another possibility is that the the memory timings for the custom
>board are not set correctly.
>
>Richard, Do you have pointers to CDP/zoom code that has a workaround
>for the 3430 > es3 errata as well as the 'ctrl sequence issue' you
>mentioned?

Richard, 
All the fixes should be available on [1] Time to Market Branch: ti-2.6.24-omap3
You could point patches from there if required.

[1] http://git.omapzoom.com/?p=omapkernel.git;a=shortlog;h=refs/heads/ti-2.6.24-omap3


>
>Kevin
>
>
>

Re: Beagle PM hangs

[Kevin Hilman]

Kevin Hilman <khilman@deeprootsystems.com> writes:

> Richard,
>
> "Woodruff, Richard" <r-woodruff2@ti.com> writes:
>
>>>   I wait for a min and try to wakeup - It works.
>>>
>>>   If i wait a little longer before waking it up say, 3 or 4 minutes.
>>> It does not wakeup.
>>
>> That is consistent with your memory not being placed into self refresh properly.
>>
>> 5 general bugs I've seen over time:
>>         sw bug:
>>                 - ctrl sequence issue before sleep
>>                 - CKE balls not muxed correctly
>>                 - crash on unexpected wake path
>>         chip bug:
>>                 - (3430 >= es3 requires a errata sequence to work)
>>         board bug:
>>                 - hooking ddr-chip-CKE wrongly or a way OMAP doesn't support like sharing between dies.
>
> Another possibility is that the the memory timings for the custom
> board are not set correctly.
>

Also, as another reference point:

I do not see this behavior on some other custom ES3-based hardware
using the PM branch.  So that suggests some board-specific issues are
involved.

Kevin

RE: Beagle PM hangs

[Woodruff, Richard]

> owner@vger.kernel.org] On Behalf Of Kevin Hilman
> Sent: Friday, January 30, 2009 11:12 AM

> Another possibility is that the the memory timings for the custom
> board are not set correctly.

Yes this is correct also.  For example if the XSR field is wrong in SDRC registers (exit from self refresh timing).

> Richard, Do you have pointers to CDP/zoom code that has a workaround
> for the 3430 > es3 errata as well as the 'ctrl sequence issue' you
> mentioned?

If you look for ES3 in code you'll find pointers.

Latest version is here (plus a bit of C code also not in link):
http://git.omapzoom.org/?p=omapkernel.git;a=blob;f=arch/arm/mach-omap2/sleep_34xx.S;h=1fcd9df16f5cbd5650c8c1c1775024e0ffd3bf7b;hb=b60074dc4c793cfc79273b178acc1eef364712f9

You'll find a bit better broken out version here:
http://git.omapzoom.com/?p=omapkernel.git;a=commitdiff;h=9543e96f5fdb32dbcd83437ac67e48edabdf9373

ROM code changed a bit and it resulted in self-refresh command not being sent.  Prior to sleep you need to set refresh on timeout and at wake up you need to hop into sram to reprogram ddr controller then hop back to complete context restore.

Regards,
Richard W.

RE: Beagle PM hangs

[Nayak, Rajendra]

> -----Original Message-----
> From: linux-omap-owner@vger.kernel.org 
> [mailto:linux-omap-owner@vger.kernel.org] On Behalf Of Kevin Hilman
> Sent: Friday, January 30, 2009 10:42 PM
> To: Woodruff, Richard
> Cc: Sriram V; Peter Reid; Koen Kooi; linux-omap@vger.kernel.org
> Subject: Re: Beagle PM hangs
> 
> Richard,
> 
> "Woodruff, Richard" <r-woodruff2@ti.com> writes:
> 
> >>   I wait for a min and try to wakeup - It works.
> >>
> >>   If i wait a little longer before waking it up say, 3 or 
> 4 minutes.
> >> It does not wakeup.
> >
> > That is consistent with your memory not being placed into 
> self refresh properly.
> >
> > 5 general bugs I've seen over time:
> >         sw bug:
> >                 - ctrl sequence issue before sleep
> >                 - CKE balls not muxed correctly
> >                 - crash on unexpected wake path
> >         chip bug:
> >                 - (3430 >= es3 requires a errata sequence to work)
> >         board bug:
> >                 - hooking ddr-chip-CKE wrongly or a way 
> OMAP doesn't support like sharing between dies.
> 
> Another possibility is that the the memory timings for the custom
> board are not set correctly.
> 
> Richard, Do you have pointers to CDP/zoom code that has a workaround
> for the 3430 > es3 errata

The es3 sdrc errata fix is already present in the pm branch.

 as well as the 'ctrl sequence issue' you
> mentioned?
> 
> Kevin
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Beagle PM hangs

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]


Op 28 jan 2009, om 18:46 heeft Kevin Hilman het volgende geschreven:

> Kevin Hilman <khilman@deeprootsystems.com> writes:
>
>> Peter Reid <ppeter.reid@gmail.com> writes:
>>
>>> Hi Kevin,
>>>   I get hangs after boot on beagle with latest PM tree i have  
>>> quite a
>>> few scripts that run on boot.
>>>
>>>   have you seen this before?
>>
>> I have not, but I believe Koen has see something similar.  I've  
>> mainly
>> been testing just a minimal rootfs and a minimal set of drivers on
>> Beagle so I can hit full-chip off in idle.
>>
>> It would help if you could desribe your setup in more detail.  What
>> kernel config are you using, what rootfs are you using?  Does it hang
>> in the same spot every time?  If so, what is going on when it hangs?
>> Have you written anything to any of the /sys/power/* files?
>>
>
> Just a thought, do you still see these problems when using the CPUfreq
> performance governor instead of the ondemand governor?
>
> It could be pointing to some bugs in the switching of operating  
> points.

Speaking of operating points, is there a reason why cpufreq only goes  
to 550MHz? It's disappointing that omap3 gets marketed with "600MHz",  
but that TI won't let it run at that speed with their cpufreq drivers.

regards,

Koen


[-- Attachment #2: Dit deel van het bericht is digitaal ondertekend --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

RE: Beagle PM hangs

[Woodruff, Richard]

> > Just a thought, do you still see these problems when using the CPUfreq
> > performance governor instead of the ondemand governor?
> >
> > It could be pointing to some bugs in the switching of operating
> > points.
>
> Speaking of operating points, is there a reason why cpufreq only goes
> to 550MHz? It's disappointing that omap3 gets marketed with "600MHz",
> but that TI won't let it run at that speed with their cpufreq drivers.

Run at that speed on your board if you like.  You can run even faster if you like but at some point you will start to have odd failures.

The 600MHz overdrive operating point is such that if you use it all the time your part life may be reduced.  If you use the stock ondemand governor with no kind of policy teak you'll be at that OPP a lot.

Percentage operation in overdrive figures into expected life of a given part. For reference code we just make the needed use cases explicitly ask for overdrive.  This way usage is predictable in well defined mobile product.

Depending on the chip you buy things are rated differently.  This rating requires part sorting which typically pushes the price up some.  Any step you add into fabrication of high volumes increases cost.

Regards,
Richard W.

Re: Beagle PM hangs

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1792 bytes --]


Op 30 jan 2009, om 23:03 heeft Woodruff, Richard het volgende  
geschreven:

>
>>> Just a thought, do you still see these problems when using the  
>>> CPUfreq
>>> performance governor instead of the ondemand governor?
>>>
>>> It could be pointing to some bugs in the switching of operating
>>> points.
>>
>> Speaking of operating points, is there a reason why cpufreq only goes
>> to 550MHz? It's disappointing that omap3 gets marketed with "600MHz",
>> but that TI won't let it run at that speed with their cpufreq  
>> drivers.
>
> Run at that speed on your board if you like.  You can run even  
> faster if you like but at some point you will start to have odd  
> failures.
>
> The 600MHz overdrive operating point is such that if you use it all  
> the time your part life may be reduced.  If you use the stock  
> ondemand governor with no kind of policy teak you'll be at that OPP  
> a lot.
>
> Percentage operation in overdrive figures into expected life of a  
> given part. For reference code we just make the needed use cases  
> explicitly ask for overdrive.  This way usage is predictable in well  
> defined mobile product.
>
> Depending on the chip you buy things are rated differently.  This  
> rating requires part sorting which typically pushes the price up  
> some.  Any step you add into fabrication of high volumes increases  
> cost.

If you look at http://beagleboard.org/hardware it says "600Mhz"  
multiple times, so I'd expect the board to be running at 600MHz if I  
were a customer. If I read the TRM right I can run the cpu at 600MHz  
continuously for a few years, which exceeds the lifespan of most  
mobile products I have owned.
So my real question is: why limit it in the kernel if all that's  
needed is a costum userspace governer?

regards,

Koen



[-- Attachment #2: Dit deel van het bericht is digitaal ondertekend --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: Beagle PM hangs

[Kevin Hilman]

Koen Kooi <koen@beagleboard.org> writes:

> Op 30 jan 2009, om 23:03 heeft Woodruff, Richard het volgende
> geschreven:
>
>>
>>>> Just a thought, do you still see these problems when using the
>>>> CPUfreq
>>>> performance governor instead of the ondemand governor?
>>>>
>>>> It could be pointing to some bugs in the switching of operating
>>>> points.
>>>
>>> Speaking of operating points, is there a reason why cpufreq only goes
>>> to 550MHz? It's disappointing that omap3 gets marketed with "600MHz",
>>> but that TI won't let it run at that speed with their cpufreq
>>> drivers.
>>
>> Run at that speed on your board if you like.  You can run even
>> faster if you like but at some point you will start to have odd
>> failures.
>>
>> The 600MHz overdrive operating point is such that if you use it all
>> the time your part life may be reduced.  If you use the stock
>> ondemand governor with no kind of policy teak you'll be at that OPP
>> a lot.
>>
>> Percentage operation in overdrive figures into expected life of a
>> given part. For reference code we just make the needed use cases
>> explicitly ask for overdrive.  This way usage is predictable in well
>> defined mobile product.
>>
>> Depending on the chip you buy things are rated differently.  This
>> rating requires part sorting which typically pushes the price up
>> some.  Any step you add into fabrication of high volumes increases
>> cost.
>
> If you look at http://beagleboard.org/hardware it says "600Mhz"
> multiple times, so I'd expect the board to be running at 600MHz if I
> were a customer. If I read the TRM right I can run the cpu at 600MHz
> continuously for a few years, which exceeds the lifespan of most
> mobile products I have owned.
> So my real question is: why limit it in the kernel if all that's
> needed is a costum userspace governer?

In fact, you wouldn't even need a custom governor.

You can just use performance or ondemand, and use CPUfreq policies set
the max available frequency at 550 most of the time, but set the max
to 600 for certain usecases.

Also, dropping the in-kernel restricion is very trivial.  See patch
below.  With that patch applied, just use performance or ondemand
governors and do this:

To avoid governor picking overdrive OPP:

# echo 550000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq  

To allow overdrive:

# echo 600000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq  

Kevin

diff --git a/arch/arm/mach-omap2/clock34xx.c b/arch/arm/mach-omap2/clock34xx.c
index 2389eb6..e9a75a3 100644
--- a/arch/arm/mach-omap2/clock34xx.c
+++ b/arch/arm/mach-omap2/clock34xx.c
@@ -698,8 +698,7 @@ void omap2_clk_init_cpufreq_table(struct cpufreq_frequency_table **table)
 	if (!mpu_opps)
 		return;
 
-	/* Avoid registering the 120% Overdrive with CPUFreq */
-	prcm = mpu_opps + MAX_VDD1_OPP - 1;
+	prcm = mpu_opps + MAX_VDD1_OPP;
 	for (; prcm->rate; prcm--) {
 		freq_table[i].index = i;
 		freq_table[i].frequency = prcm->rate / 1000;

RE: Beagle PM hangs

[Woodruff, Richard]

>In fact, you wouldn't even need a custom governor.
> To avoid governor picking overdrive OPP:
>
> # echo 550000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
>
> To allow overdrive:
>
> # echo 600000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq

Sure. Now is someone going to fix up every root file system which users have around.

It would be better to init the limit in the driver for this version of OMAP.

Even if you do that someone still needs to turn on and off the limit dynamically or create a custom governor.

Until that kind of smart control is pervasive it is safer to just put the interface a layer down for expert use.

But like I was saying, people are free to set the clock to what ever they want.  On a properly characterized part it might even be safe for many years.  A 3440 will sit well above 600MHz in nominal.

Regards,
Richard W.

Re: Beagle PM hangs

[Kevin Hilman]

"Woodruff, Richard" <r-woodruff2@ti.com> writes:

>>In fact, you wouldn't even need a custom governor.
>> To avoid governor picking overdrive OPP:
>>
>> # echo 550000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
>>
>> To allow overdrive:
>>
>> # echo 600000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
>
> Sure. Now is someone going to fix up every root file system which users have around.

I'm not suggesting this as the standard way.  The point was to say
that a custom governor is not the only way.

> It would be better to init the limit in the driver for this version of OMAP.

Agreed.

CPUfreq policy is just as easily set in the kernel at init time.  Then
the default would be set right, and the rootfs would be left to change
it if desired.

Kevin

> Even if you do that someone still needs to turn on and off the limit
> dynamically or create a custom governor.
>
> Until that kind of smart control is pervasive it is safer to just put the interface a layer down for expert use.
>
> But like I was saying, people are free to set the clock to what ever they want.  On a properly characterized part it might even be safe for many years.  A 3440 will sit well above 600MHz in nominal.
>
> Regards,
> Richard W.

Re: Beagle PM hangs

[Igor Stoppa]

On Sat, 2009-01-31 at 00:21 +0100, ext Kevin Hilman wrote:

> CPUfreq policy is just as easily set in the kernel at init time.  Then
> the default would be set right, and the rootfs would be left to change
> it if desired.

But in general the situation is more complex and knowledge about the
packaging and its thermal model is needed.

Note that with packaging i refer to both the omap package (does it use
stacked memory or not, is it a memory combo, etc) and the _device_
package.

Not getting errors is not a good indication that the system is really
operating in a safe mode; the device might be stable all the time but
its life be more than halved.

-- 

Cheers, Igor

---

Igor Stoppa
Maemo Software - Nokia Devices R&D - Helsinki

RE: Beagle PM hangs

[Woodruff, Richard]

> If you look at http://beagleboard.org/hardware it says "600Mhz"
> multiple times, so I'd expect the board to be running at 600MHz if I
> were a customer. If I read the TRM right I can run the cpu at 600MHz
> continuously for a few years, which exceeds the lifespan of most
> mobile products I have owned.
> So my real question is: why limit it in the kernel if all that's
> needed is a costum userspace governer?

In reference code all OPPs are modeled.  They all are just not exported to cpufreq.  Any driver or user space can peg the system to max opp.  There is nothing stopping anyone from hooking opp selection to cpufreq.  However, it does require a custom governor or a little policy script to set limits.  This is not the default in main line builds.  It could be pushed if someone wanted to do the work.

For the general testing and usage it is easy to just use ondemand.  It's a good general purpose governor.  There is no need to recreate it for general use.  Then treat special needs out of band.  If you want to create a new governor and can get a lot of people to use it then nothing is in the way but time to try and sell up stream people it is needed.

Initial design for customers required the ability to plug in custom managers which sit at the cpufreq level.  Not everyone in the world uses cpufreq for this kind of thing.  Because of this we purposefully kept separation.

Novice uses which just grab a kernel and take the root file system of the day will get it wrong.

Off hand I'm not sure which kernel level mechanism is blocking you in l-o.  I haven't had bandwidth to follow line by line.

Regards,
Richard W.

Re: Beagle PM hangs

[Peter Reid]

Hi Kevin,

On Wed, Jan 28, 2009 at 9:58 PM, Kevin Hilman
<khilman@deeprootsystems.com> wrote:
> Peter Reid <ppeter.reid@gmail.com> writes:
>
>> Hi Kevin,
>>    I get hangs after boot on beagle with latest PM tree i have quite a
>> few scripts that run on boot.
>>
>>    have you seen this before?
>
> I have not, but I believe Koen has see something similar.  I've mainly
> been testing just a minimal rootfs and a minimal set of drivers on
> Beagle so I can hit full-chip off in idle.

   I have a default omap3_beagle_defconfig with cpuidle, cpufreq
   enabled.   It hangs after running the /etc boot scripts. The behaviour is
   random. Have seen this working randomly before.
   yet to debug this.


  Regards,
   reid.


>
> It would help if you could desribe your setup in more detail.  What
> kernel config are you using, what rootfs are you using?  Does it hang
> in the same spot every time?  If so, what is going on when it hangs?
> Have you written anything to any of the /sys/power/* files?
>
> Kevin
>