CPU_IDLE causes random reboots on custom 4430

CPU_IDLE causes random reboots on custom 4430

[Chris Hoffmann]

Hi,

We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430 
ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly 
bean. The board works quite well, but we experience random hangs and the 
watchdog kicks the board to reboot.

The problem is that kmsg_last doesn't not show any useful information 
and the fiq debugger refuses to work when the device is locked up. We 
also enabled various kernel lock debugging configs to get more info ... 
in vain.

Experimenting with it for a while (removing non-standard drivers like 
accel, charger, battery, etc) we found that removing CONFIG_CPU_IDLE or 
setting disallow_smp_idle in cpuidle44xx.c seems to cure the random 
lockups and make the board work fine (apart from the maybe obvious 
consequences that it doesn't go to deep sleep anymore).

Further below a few sample infos from the board.

We'd very much appreciate any further help on how to proceed to find the 
root cause and happy to provide any more info from sys, debug or 
omapconf tool.

Rgds,
Chris

A dmesg from a normal boot:
https://dl.dropbox.com/u/21820416/CPU_IDLE_CRASH/dmesg.log

Kernel source:
https://github.com/Kuzma30/kernel3NookTablet/tree/jellybean

omapconf --cpuinfo
https://dl.dropbox.com/u/21820416/CPU_IDLE_CRASH/omapconf.cpuinfo

cat /sys/kernel/debug/clock/summary
https://dl.dropbox.com/u/21820416/CPU_IDLE_CRASH/clock.summary

... what else can we provide to track this down?

Re: CPU_IDLE causes random reboots on custom 4430

[Shilimkar, Santosh]

On Sat, Sep 22, 2012 at 4:19 AM, Chris Hoffmann <chrmhoffmann@gmail.com> wrote:
> Hi,
>
> We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430
> ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly
> bean. The board works quite well, but we experience random hangs and the
> watchdog kicks the board to reboot.
>
On the same kernel, you should have support for the persistent log. You might
want to check the output. That should give you pointers on what CPU was
doing before the freeze which resulted in reboot.

Regards
Santosh

Re: CPU_IDLE causes random reboots on custom 4430

[Chris Hoffmann]

On 09/22/2012 07:45 AM, Shilimkar, Santosh wrote:
> On Sat, Sep 22, 2012 at 4:19 AM, Chris Hoffmann <chrmhoffmann@gmail.com> wrote:
>> Hi,
>>
>> We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430
>> ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly
>> bean. The board works quite well, but we experience random hangs and the
>> watchdog kicks the board to reboot.
>>
> On the same kernel, you should have support for the persistent log. You might
> want to check the output. That should give you pointers on what CPU was
> doing before the freeze which resulted in reboot.

Hi,

I have some problems to provide logs. If I add -DDEBUG to cpuidle44xx.o 
the problem doesn't seem to occur. It could be that printk-ing 
alleviates the issue.

Also the watchdog seems to shutdown the device rather than rebooting it 
(or it hangs?) and then I can't provide /proc/last_kmsg.

How could I provide more info?

Rgds,
Chris

Re: CPU_IDLE causes random reboots on custom 4430

[Shilimkar, Santosh]

On Sat, Sep 22, 2012 at 10:41 PM, Chris Hoffmann <chrmhoffmann@gmail.com> wrote:
> On 09/22/2012 07:45 AM, Shilimkar, Santosh wrote:
>>
>> On Sat, Sep 22, 2012 at 4:19 AM, Chris Hoffmann <chrmhoffmann@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430
>>> ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly
>>> bean. The board works quite well, but we experience random hangs and the
>>> watchdog kicks the board to reboot.
>>>
>> On the same kernel, you should have support for the persistent log. You
>> might
>> want to check the output. That should give you pointers on what CPU was
>> doing before the freeze which resulted in reboot.
>
>
> Hi,
>
> I have some problems to provide logs. If I add -DDEBUG to cpuidle44xx.o the
> problem doesn't seem to occur. It could be that printk-ing alleviates the
> issue.
>
> Also the watchdog seems to shutdown the device rather than rebooting it (or
> it hangs?) and then I can't provide /proc/last_kmsg.
>
> How could I provide more info?
>
Check if you have "/sys/kernel/debug/persistent_trace" available on
your kernel. This generally helps whenever there are hangs, the last
call stack is stored on memory and on the reboot it can be cat'ed to
see if some useful information about hang is available.

Regards
Santosh

Re: CPU_IDLE causes random reboots on custom 4430

[Christian Hoffmann]

On 09/23/2012 06:11 PM, Shilimkar, Santosh wrote:
> On Sat, Sep 22, 2012 at 10:41 PM, Chris Hoffmann <chrmhoffmann@gmail.com> wrote:
>> On 09/22/2012 07:45 AM, Shilimkar, Santosh wrote:
>>>
>>> On Sat, Sep 22, 2012 at 4:19 AM, Chris Hoffmann <chrmhoffmann@gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> We're trying to get a custom 4430 board (aka. nook tablet with OMAP4430
>>>> ES2.3 HS TWL6030 ES2.1) working with p-android-omap-3.0 on android jelly
>>>> bean. The board works quite well, but we experience random hangs and the
>>>> watchdog kicks the board to reboot.
>>>>
>>> On the same kernel, you should have support for the persistent log. You
>>> might
>>> want to check the output. That should give you pointers on what CPU was
>>> doing before the freeze which resulted in reboot.
>>
>>
>> Hi,
>>
>> I have some problems to provide logs. If I add -DDEBUG to cpuidle44xx.o the
>> problem doesn't seem to occur. It could be that printk-ing alleviates the
>> issue.
>>
>> Also the watchdog seems to shutdown the device rather than rebooting it (or
>> it hangs?) and then I can't provide /proc/last_kmsg.
>>
>> How could I provide more info?
>>
> Check if you have "/sys/kernel/debug/persistent_trace" available on
> your kernel. This generally helps whenever there are hangs, the last
> call stack is stored on memory and on the reboot it can be cat'ed to
> see if some useful information about hang is available.

Hi Santosh, all,

the p-android-omap-3.0 doesn't have the persistent_trace but I was able 
to backport it from 3.4 without major issues (only tricky part is that 
in p-android-3.4 there's no apparent user of that device in omapzoom 
kernel).

The problem is now that the omap-watchdog doesn't kick the device to 
reboot but rather to shutdown, so I still have no trace. Soft-rebooting 
shows that I can get persistent_trace.

Switching the persistent tracer to ecc=true does not help as it seems to 
overload the device completely when activating the tracing.

Rgds,
Chris