Kernel hang on OMAP3 based Beagle board, RTC issue?

Kernel hang on OMAP3 based Beagle board, RTC issue?

[Dirk Behme]

At Beagle IRC there was some discussion [1] [2] today about kernel 
hang on OMAP3 based Beagle board. I'll try to summarize discussion and 
Khasim's findings, maybe somebody here has an idea.

Latest OMAP git kernel seems to hang on beagle board after some time. 
You don't see any crash, but after a while typing, anything doesn't 
work. SD stops as well, so you can only do stuff that's in ram, any IO 
will block. It can take >1 day to get there, but mostly it's ~30 minutes.

Khasim found that it seems disabling RTC will remove the hang. When 
booting the git kernel after some time there are no timer or PRCM 
interrupts and the control resides in cpu_idle. We don't know where 
exactly the problem in RTC is, but while browsing the history of files 
related to RTC Khasim found that some changes were done to the files 
to handle T2 interrupts in a centralized way for USB, Battery and RTC. 
It seems even Battery broke with this patch set

http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f

May be this is the change that is affecting RTC. But it's unclear why 
it is affecting GP timer and PRCM interrupts.

Any idea?

Thanks and please correct if something is wrong in the summary

Dirk

[1] http://www.beagleboard.org/irclogs/index.php?date=2008-07-02#T07:19:47

[2] http://www.beagleboard.org/irclogs/index.php?date=2008-07-02#T16:57:47

RE: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Gadiyar, Anand]

> At Beagle IRC there was some discussion [1] [2] today about kernel
> hang on OMAP3 based Beagle board. I'll try to summarize discussion and
> Khasim's findings, maybe somebody here has an idea.
>
> Latest OMAP git kernel seems to hang on beagle board after some time.
> You don't see any crash, but after a while typing, anything doesn't
> work. SD stops as well, so you can only do stuff that's in ram, any IO
> will block. It can take >1 day to get there, but mostly it's ~30 minutes.
>
> Khasim found that it seems disabling RTC will remove the hang. When
> booting the git kernel after some time there are no timer or PRCM
> interrupts and the control resides in cpu_idle. We don't know where
> exactly the problem in RTC is, but while browsing the history of files
> related to RTC Khasim found that some changes were done to the files
> to handle T2 interrupts in a centralized way for USB, Battery and RTC.
> It seems even Battery broke with this patch set
>
> http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f
>
> May be this is the change that is affecting RTC. But it's unclear why
> it is affecting GP timer and PRCM interrupts.
>
> Any idea?
>
> Thanks and please correct if something is wrong in the summary
>
> Dirk

Khasim also found that T2 RTC is not enabled by default in 3430sdp
defconfig, but it appears to be enabled in the beagleboard defconfig.
Maybe you can get by by simply disabling T2 RTC in menuconfig for now?

- Anand

Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]


Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:

>> At Beagle IRC there was some discussion [1] [2] today about kernel
>> hang on OMAP3 based Beagle board. I'll try to summarize discussion  
>> and
>> Khasim's findings, maybe somebody here has an idea.
>>
>> Latest OMAP git kernel seems to hang on beagle board after some time.
>> You don't see any crash, but after a while typing, anything doesn't
>> work. SD stops as well, so you can only do stuff that's in ram, any  
>> IO
>> will block. It can take >1 day to get there, but mostly it's ~30  
>> minutes.
>>
>> Khasim found that it seems disabling RTC will remove the hang. When
>> booting the git kernel after some time there are no timer or PRCM
>> interrupts and the control resides in cpu_idle. We don't know where
>> exactly the problem in RTC is, but while browsing the history of  
>> files
>> related to RTC Khasim found that some changes were done to the files
>> to handle T2 interrupts in a centralized way for USB, Battery and  
>> RTC.
>> It seems even Battery broke with this patch set
>>
>> http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f
>>
>> May be this is the change that is affecting RTC. But it's unclear why
>> it is affecting GP timer and PRCM interrupts.
>>
>> Any idea?
>>
>> Thanks and please correct if something is wrong in the summary
>>
>> Dirk
>
> Khasim also found that T2 RTC is not enabled by default in 3430sdp
> defconfig, but it appears to be enabled in the beagleboard defconfig.
> Maybe you can get by by simply disabling T2 RTC in menuconfig for now?

Tried that and I get the same hang, so I don't think the RTC is to  
blame.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Dirk Behme]

Koen Kooi wrote:
> 
> Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:
> 
>>> At Beagle IRC there was some discussion [1] [2] today about kernel
>>> hang on OMAP3 based Beagle board. I'll try to summarize discussion  and
>>> Khasim's findings, maybe somebody here has an idea.
>>>
>>> Latest OMAP git kernel seems to hang on beagle board after some time.
>>> You don't see any crash, but after a while typing, anything doesn't
>>> work. SD stops as well, so you can only do stuff that's in ram, any  IO
>>> will block. It can take >1 day to get there, but mostly it's ~30  
>>> minutes.
>>>
>>> Khasim found that it seems disabling RTC will remove the hang. When
>>> booting the git kernel after some time there are no timer or PRCM
>>> interrupts and the control resides in cpu_idle. We don't know where
>>> exactly the problem in RTC is, but while browsing the history of  files
>>> related to RTC Khasim found that some changes were done to the files
>>> to handle T2 interrupts in a centralized way for USB, Battery and  RTC.
>>> It seems even Battery broke with this patch set
>>>
>>> http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f 
>>>
>>>
>>> May be this is the change that is affecting RTC. But it's unclear why
>>> it is affecting GP timer and PRCM interrupts.
>>>
>>> Any idea?
>>>
>>> Thanks and please correct if something is wrong in the summary
>>>
>>> Dirk
>>
>>
>> Khasim also found that T2 RTC is not enabled by default in 3430sdp
>> defconfig, but it appears to be enabled in the beagleboard defconfig.
>> Maybe you can get by by simply disabling T2 RTC in menuconfig for now?
> 
> 
> Tried that and I get the same hang, so I don't think the RTC is to  blame.

Short update just in case anybody has an idea:

Gadiyar spent some time with git-bisect (thanks!):

It seems that the bad commit is probably between:
# good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c 
to pm24xx.c
and sleep.S to sleep24xx.S
and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP: 
Enable config
options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND

This is a 9-patch series by Jouni Hogander related to suspend and 
power-management.

Dirk

RE: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Gadiyar, Anand]

> -----Original Message-----
> From: Dirk Behme [mailto:dirk.behme@googlemail.com]
> Sent: Saturday, July 05, 2008 12:56 PM
> To: linux-omap@vger.kernel.org
> Cc: Gadiyar, Anand
> Subject: Re: Kernel hang on OMAP3 based Beagle board, RTC issue?
>
> Koen Kooi wrote:
> >
> > Op 2 jul 2008, om 22:03 heeft Gadiyar, Anand het volgende geschreven:
> >
> >>> At Beagle IRC there was some discussion [1] [2] today about kernel
> >>> hang on OMAP3 based Beagle board. I'll try to summarize discussion  and
> >>> Khasim's findings, maybe somebody here has an idea.
> >>>
> >>> Latest OMAP git kernel seems to hang on beagle board after some time.
> >>> You don't see any crash, but after a while typing, anything doesn't
> >>> work. SD stops as well, so you can only do stuff that's in ram, any  IO
> >>> will block. It can take >1 day to get there, but mostly it's ~30
> >>> minutes.
> >>>
> >>> Khasim found that it seems disabling RTC will remove the hang. When
> >>> booting the git kernel after some time there are no timer or PRCM
> >>> interrupts and the control resides in cpu_idle. We don't know where
> >>> exactly the problem in RTC is, but while browsing the history of  files
> >>> related to RTC Khasim found that some changes were done to the files
> >>> to handle T2 interrupts in a centralized way for USB, Battery and  RTC.
> >>> It seems even Battery broke with this patch set
> >>>
> >>> http://source.mvista.com/git/gitweb.cgi?p=linux-omap-2.6.git;a=commitdiff;h=68d7477caca19c0b52b5d4e85700cd3e6115577f
> >>>
> >>>
> >>> May be this is the change that is affecting RTC. But it's unclear why
> >>> it is affecting GP timer and PRCM interrupts.
> >>>
> >>> Any idea?
> >>>
> >>> Thanks and please correct if something is wrong in the summary
> >>>
> >>> Dirk
> >>
> >>
> >> Khasim also found that T2 RTC is not enabled by default in 3430sdp
> >> defconfig, but it appears to be enabled in the beagleboard defconfig.
> >> Maybe you can get by by simply disabling T2 RTC in menuconfig for now?
> >
> >
> > Tried that and I get the same hang, so I don't think the RTC is to  blame.
>
> Short update just in case anybody has an idea:
>
> Gadiyar spent some time with git-bisect (thanks!):
>
> It seems that the bad commit is probably between:
> # good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c
> to pm24xx.c
> and sleep.S to sleep24xx.S
> and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
> Enable config
> options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND
>
> This is a 9-patch series by Jouni Hogander related to suspend and
> power-management.
>
> Dirk

I'm still working on the git-bisect. I'm currently waiting for ten minutes
after boot before marking a commit good or bad. If the hang happens more
than ten minutes after boot, I might miss that and mark the commit good - so
I will re-check the good commits one more time and confirm.

- Anand

RE: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Gadiyar, Anand]

Hi All,

<snip>

> > > Tried that and I get the same hang, so I don't think the RTC is to  blame.
> >
> > Short update just in case anybody has an idea:
> >
> > Gadiyar spent some time with git-bisect (thanks!):
> >
> > It seems that the bad commit is probably between:
> > # good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c
> > and sleep.S to sleep24xx.S
> > and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
> > Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND
> >
> > This is a 9-patch series by Jouni Hogander related to suspend and
> > power-management.
> >
> > Dirk
>
> I'm still working on the git-bisect. I'm currently waiting
> for ten minutes
> after boot before marking a commit good or bad. If the hang
> happens more
> than ten minutes after boot, I might miss that and mark the
> commit good - so
> I will re-check the good commits one more time and confirm.
>
> - Anand

As far as I can tell, the beagle hang issue aries after this commit:

397f49cde4e75cf662a23814097db35f3e6c6b62 is first bad commit
commit 397f49cde4e75cf662a23814097db35f3e6c6b62
Author: Jouni Hogander <jouni.hogander@nokia.com>
Date:   Fri May 16 13:58:25 2008 +0300

    34XX: PM: Initial version of suspend and dynamic retention

    This is initial version of suspend and dynamic retention for
    34xx. Omap is tried to put to full retention on suspend and pm_idle.

    Signed-off-by: Jouni Hogander <jouni.hogander@nokia.com>
    Signed-off-by: Tony Lindgren <tony@atomide.com>


Could someone please test and confirm if the previous commit works without a hang?
I waited for around 20 minutes and did not observe a hang.

These were the last two good commits I got from git-bisect.
# good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel
git-bisect good 1c66e10c850950a84da68cafb07022eab5e7eec0
# good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx

Regards,
Anand

Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Högander Jouni]

Hi,

"ext Gadiyar, Anand" <gadiyar@ti.com> writes:

> Hi All,
>
> <snip>
>
>> > > Tried that and I get the same hang, so I don't think the RTC is to  blame.
>> >
>> > Short update just in case anybody has an idea:
>> >
>> > Gadiyar spent some time with git-bisect (thanks!):
>> >
>> > It seems that the bad commit is probably between:
>> > # good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c
>> > and sleep.S to sleep24xx.S
>> > and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
>> > Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND
>> >
>> > This is a 9-patch series by Jouni Hogander related to suspend and
>> > power-management.
>> >
>> > Dirk
>>
>> I'm still working on the git-bisect. I'm currently waiting
>> for ten minutes
>> after boot before marking a commit good or bad. If the hang
>> happens more
>> than ten minutes after boot, I might miss that and mark the
>> commit good - so
>> I will re-check the good commits one more time and confirm.
>>
>> - Anand
>
> As far as I can tell, the beagle hang issue aries after this commit:
>
> 397f49cde4e75cf662a23814097db35f3e6c6b62 is first bad commit
> commit 397f49cde4e75cf662a23814097db35f3e6c6b62
> Author: Jouni Hogander <jouni.hogander@nokia.com>
> Date:   Fri May 16 13:58:25 2008 +0300
>
>     34XX: PM: Initial version of suspend and dynamic retention
>
>     This is initial version of suspend and dynamic retention for
>     34xx. Omap is tried to put to full retention on suspend and pm_idle.
>
>     Signed-off-by: Jouni Hogander <jouni.hogander@nokia.com>
>     Signed-off-by: Tony Lindgren <tony@atomide.com>
>
>
> Could someone please test and confirm if the previous commit works without a hang?
> I waited for around 20 minutes and did not observe a hang.
>
> These were the last two good commits I got from git-bisect.
> # good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel
> git-bisect good 1c66e10c850950a84da68cafb07022eab5e7eec0
> # good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx

Unfortunately I have only sdp board here and I'm not able to reproduce
this with it. Are you using omap3_beagle_defconfig without modifications there?

You mentioned prcm interrupts in this thread. Are you normally seeing
them? I'm wondering this because you shouldn't see them with current
l-o tree and default beagle config. Are you enabling sleep_while_idle
(echo 1 > /sys/power/sleep_while_idle)?

You have end up to conclusion that "34XX: PM: Initial version of
suspend and dynamic retention" is causing this? Could you try to apply
this on top of mentioned commit and see if you still can see that
hang:

diff --git a/arch/arm/mach-omap2/pm34xx.c b/arch/arm/mach-omap2/pm34xx.c
index 202c269..308835b 100644
--- a/arch/arm/mach-omap2/pm34xx.c
+++ b/arch/arm/mach-omap2/pm34xx.c
@@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
                goto err1;
        }
 
-       ret = pwrdm_for_each(pwrdms_setup);
-       if (ret) {
-               printk(KERN_ERR "Failed to setup powerdomains\n");
-               goto err2;
-       }
+/*     ret = pwrdm_for_each(pwrdms_setup); */
+/*     if (ret) { */
+/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
+/*             goto err2; */
+/*     } */
 
        mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
        if (mpu_pwrdm == NULL) {

-- 
Jouni Högander

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Gadiyar, Anand]

Hi,

> Hi,
>
> "ext Gadiyar, Anand" <gadiyar@ti.com> writes:
>
> > Hi All,
> >
> > <snip>
> >
> >> > > Tried that and I get the same hang, so I don't think the RTC is to  blame.
> >> >
> >> > Short update just in case anybody has an idea:
> >> >
> >> > Gadiyar spent some time with git-bisect (thanks!):
> >> >
> >> > It seems that the bad commit is probably between:
> >> > # good: [3ffec4e18484c34838fa341de3848306c29ecd5d] 24XX: PM: Move pm.c to pm24xx.c
> >> > and sleep.S to sleep24xx.S
> >> > and # bad: [458776cfe389ff03bd6c56c47e059df0778cdfca] OMAP3430SDP:
> >> > Enable config options CONFIG_OMAP_RESET_CLOCKS and CONFIG_SUSPEND
> >> >
> >> > This is a 9-patch series by Jouni Hogander related to suspend and
> >> > power-management.
> >> >
> >> > Dirk
> >>
> >> I'm still working on the git-bisect. I'm currently waiting for ten minutes
> >> after boot before marking a commit good or bad. If the hang happens more
> >> than ten minutes after boot, I might miss that and mark the commit good - so
> >> I will re-check the good commits one more time and confirm.
> >>
> >> - Anand
> >
> > As far as I can tell, the beagle hang issue aries after this commit:
> >
> > 397f49cde4e75cf662a23814097db35f3e6c6b62 is first bad commit
> > commit 397f49cde4e75cf662a23814097db35f3e6c6b62
> > Author: Jouni Hogander <jouni.hogander@nokia.com>
> > Date:   Fri May 16 13:58:25 2008 +0300
> >
> >     34XX: PM: Initial version of suspend and dynamic retention
> >
> >     This is initial version of suspend and dynamic retention for
> >     34xx. Omap is tried to put to full retention on suspend and pm_idle.
> >
> >     Signed-off-by: Jouni Hogander <jouni.hogander@nokia.com>
> >     Signed-off-by: Tony Lindgren <tony@atomide.com>
> >
> >
> > Could someone please test and confirm if the previous commit works without a hang?
> > I waited for around 20 minutes and did not observe a hang.
> >
> > These were the last two good commits I got from git-bisect.
> > # good: [1c66e10c850950a84da68cafb07022eab5e7eec0] 34XX: Suspend: Take suspend sram code from ti cdp kernel
> > git-bisect good 1c66e10c850950a84da68cafb07022eab5e7eec0
> > # good: [131ce2d1dd5081f97885419678a8b3211c7c2dee] 34XX: Add miscellaneous definitions related to 34xx
>
> Unfortunately I have only sdp board here and I'm not able to reproduce
> this with it. Are you using omap3_beagle_defconfig without
> modifications there?
>

Yes.


> You mentioned prcm interrupts in this thread. Are you normally seeing
> them? I'm wondering this because you shouldn't see them with current
> l-o tree and default beagle config. Are you enabling sleep_while_idle
> (echo 1 > /sys/power/sleep_while_idle)?
>

Haven't tried this. All we did was boot the board from MMC and leave it running
for around ten minutes.


> You have end up to conclusion that "34XX: PM: Initial version of
> suspend and dynamic retention" is causing this? Could you try to apply
> this on top of mentioned commit and see if you still can see that
> hang:
>
> diff --git a/arch/arm/mach-omap2/pm34xx.c
> b/arch/arm/mach-omap2/pm34xx.c
> index 202c269..308835b 100644
> --- a/arch/arm/mach-omap2/pm34xx.c
> +++ b/arch/arm/mach-omap2/pm34xx.c
> @@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
>                 goto err1;
>         }
>
> -       ret = pwrdm_for_each(pwrdms_setup);
> -       if (ret) {
> -               printk(KERN_ERR "Failed to setup powerdomains\n");
> -               goto err2;
> -       }
> +/*     ret = pwrdm_for_each(pwrdms_setup); */
> +/*     if (ret) { */
> +/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
> +/*             goto err2; */
> +/*     } */
>
>         mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
>         if (mpu_pwrdm == NULL) {

With this change, the hang does not happene. Applied this patch on the current
linux-omap tree.

Thanks for the patch. It fixes the hang for now. I'm not sure why the hang
happens, but could it be because the SDP uses ttyS0 by default as the
console while the Beagle is on ttyS2?

Regards,
Anand

Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]


Op 14 jul 2008, om 15:27 heeft Gadiyar, Anand het volgende geschreven:

>> diff --git a/arch/arm/mach-omap2/pm34xx.c
>> b/arch/arm/mach-omap2/pm34xx.c
>> index 202c269..308835b 100644
>> --- a/arch/arm/mach-omap2/pm34xx.c
>> +++ b/arch/arm/mach-omap2/pm34xx.c
>> @@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
>>                goto err1;
>>        }
>>
>> -       ret = pwrdm_for_each(pwrdms_setup);
>> -       if (ret) {
>> -               printk(KERN_ERR "Failed to setup powerdomains\n");
>> -               goto err2;
>> -       }
>> +/*     ret = pwrdm_for_each(pwrdms_setup); */
>> +/*     if (ret) { */
>> +/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
>> +/*             goto err2; */
>> +/*     } */
>>
>>        mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
>>        if (mpu_pwrdm == NULL) {
>
> With this change, the hang does not happene. Applied this patch on  
> the current
> linux-omap tree.
>
> Thanks for the patch. It fixes the hang for now. I'm not sure why  
> the hang
> happens, but could it be because the SDP uses ttyS0 by default as the
> console while the Beagle is on ttyS2?

With that patch I still get the same behaviour:

serial console freezes, but responds to sysrq

I don't know if its related, but after the serial console freezes  
animated backgrounds in X stop animating, but will refresh if you  
press a key on the keyboard. The same effect can be seen with running  
'top' on the screen.
I'm using the CPUidle patches posted to the list a few weeks back +  
the timer suppression patch.

regards,

Koen


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Paul Walmsley]

On Tue, 15 Jul 2008, Koen Kooi wrote:

> > > --- a/arch/arm/mach-omap2/pm34xx.c
> > > +++ b/arch/arm/mach-omap2/pm34xx.c
> > > @@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
> > >               goto err1;
> > >       }
> > > 
> > > -       ret = pwrdm_for_each(pwrdms_setup);
> > > -       if (ret) {
> > > -               printk(KERN_ERR "Failed to setup powerdomains\n");
> > > -               goto err2;
> > > -       }
> > > +/*     ret = pwrdm_for_each(pwrdms_setup); */
> > > +/*     if (ret) { */
> > > +/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
> > > +/*             goto err2; */
> > > +/*     } */
> > > 
> > >       mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
> > >       if (mpu_pwrdm == NULL) {
> > 
> > With this change, the hang does not happene. Applied this patch on the
> > current
> > linux-omap tree.
> > 
> > Thanks for the patch. It fixes the hang for now. I'm not sure why the hang
> > happens, but could it be because the SDP uses ttyS0 by default as the
> > console while the Beagle is on ttyS2?
> 
> With that patch I still get the same behaviour:
> 
> serial console freezes, but responds to sysrq

Likewise on Beagle rev B4 here. Patching the PM code as above does not 
seem to make a difference; nor leaving out the RTC.

The failure mode that seems to recur here is that a userspace binary will 
exit, but no shell prompt is displayed afterwards.  Once wedged, Sysrq-T 
provided some interesting output, included below.  uart_wait_until_sent() ?
That doesn't seem right.

N.B., on the Beagle here, TWL4030 IRQ handling has to be commented out to 
accomplish anything useful.  Otherwise the console drowns in "TWL4030 
module irq 368 is disabled but can't be masked!" messages on at least 90% 
of reboots.  This is in drivers/i2c/chips/twl4030-core.c, commenting out 
the lines underneath:

   /* install an irq handler to demultiplex the TWL4030 interrupt */   

Have not yet had a chance to study the problem - could be a TWL4030 GPIO 
interrupt issue?


- Paul


<6>sh            Ssh            S c024cf70  c024cf70     0   362      1
    0   362      1
[<c024cd0c>] [<c024cd0c>] (schedule+0x0/0x2a0) (schedule+0x0/0x2a0) from 
[<c024d1d0>] from [<c024d1d0>] (schedule_timeout+0x90/0xb8)
(schedule_timeout+0x90/0xb8)
[<c024d140>] [<c024d140>] (schedule_timeout+0x0/0xb8) 
(schedule_timeout+0x0/0xb8) from [<c024d24c>] from [<c024d24c>] 
(schedule_timeout_interruptible+0x28/0x2c)
(schedule_timeout_interruptible+0x28/0x2c)
 r7:c03257b8 r7:c03257b8 r6:00000001 r6:00000001 r5:ffff888e r5:ffff888e 
r4:c7d9e000 r4:c7d9e000

[<c024d224>] [<c024d224>] (schedule_timeout_interruptible+0x0/0x2c) 
(schedule_timeout_interruptible+0x0/0x2c) from [<c0053fe4>] from 
[<c0053fe4>] (msleep_interruptible+0x28/0x58)
(msleep_interruptible+0x28/0x58)
[<c0053fbc>] [<c0053fbc>] (msleep_interruptible+0x0/0x58) 
(msleep_interruptible+0x0/0x58) from [<c017c56c>] from [<c017c56c>] 
(uart_wait_until_sent+0x94/0xf8)
(uart_wait_until_sent+0x94/0xf8)
 r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000

[<c017c4d8>] [<c017c4d8>] (uart_wait_until_sent+0x0/0xf8) 
(uart_wait_until_sent+0x0/0xf8) from [<c016e0cc>] from [<c016e0cc>] 
(tty_wait_until_sent+0xe4/0xf4)
(tty_wait_until_sent+0xe4/0xf4)
 r7:c7d9e000 r7:c7d9e000 r6:7fffffff r6:7fffffff r5:c7d6b640 r5:c7d6b640 
r4:c7d9fd30 r4:c7d9fd30

[<c016dfe8>] [<c016dfe8>] (tty_wait_until_sent+0x0/0xf4) 
(tty_wait_until_sent+0x0/0xf4) from [<c016e2b4>] from [<c016e2b4>] 
(set_termios+0x1d8/0x4cc)
(set_termios+0x1d8/0x4cc)
[<c016e0dc>] [<c016e0dc>] (set_termios+0x0/0x4cc) (set_termios+0x0/0x4cc) 
from [<c016e7f4>] from [<c016e7f4>] (tty_mode_ioctl+0x24c/0x3d8)
(tty_mode_ioctl+0x24c/0x3d8)
[<c016e5a8>] [<c016e5a8>] (tty_mode_ioctl+0x0/0x3d8) 
(tty_mode_ioctl+0x0/0x3d8) from [<c016ebd8>] from [<c016ebd8>] 
(n_tty_ioctl+0x258/0x268)
(n_tty_ioctl+0x258/0x268)
 r7:c7c6f740 r7:c7c6f740 r6:beaac3fc r6:beaac3fc r5:beaac3fc r5:beaac3fc 
r4:c7d80400 r4:c7d80400

[<c016e980>] [<c016e980>] (n_tty_ioctl+0x0/0x268) (n_tty_ioctl+0x0/0x268) 
from [<c016b4d4>] from [<c016b4d4>] (tty_ioctl+0xda8/0xe04)
(tty_ioctl+0xda8/0xe04)
 r7:c7c6f740 r7:c7c6f740 r6:c7d80400 r6:c7d80400 r5:beaac3fc r5:beaac3fc 
r4:00005403 r4:00005403

[<c016a72c>] [<c016a72c>] (tty_ioctl+0x0/0xe04) (tty_ioctl+0x0/0xe04) from 
[<c009fa80>] from [<c009fa80>] (vfs_ioctl+0x34/0x78)
(vfs_ioctl+0x34/0x78)
[<c009fa4c>] [<c009fa4c>] (vfs_ioctl+0x0/0x78) (vfs_ioctl+0x0/0x78) from 
[<c009fd34>] from [<c009fd34>] (do_vfs_ioctl+0x270/0x280)
(do_vfs_ioctl+0x270/0x280)
 r5:beaac3fc r5:beaac3fc r4:c7c6f740 r4:c7c6f740

[<c009fac4>] [<c009fac4>] (do_vfs_ioctl+0x0/0x280) 
(do_vfs_ioctl+0x0/0x280) from [<c009fd84>] from [<c009fd84>] 
(sys_ioctl+0x40/0x64)
(sys_ioctl+0x40/0x64)
 r7:c7c6f740 r7:c7c6f740 r6:00005403 r6:00005403 r5:beaac3fc r5:beaac3fc 
r4:00000000 r4:00000000

[<c009fd44>] [<c009fd44>] (sys_ioctl+0x0/0x64) (sys_ioctl+0x0/0x64) from 
[<c0027a40>] from [<c0027a40>] (ret_fast_syscall+0x0/0x2c)
(ret_fast_syscall+0x0/0x2c)
 r7:00000036 r7:00000036 r6:beaac488 r6:beaac488 r5:00000000 r5:00000000 
r4:00000a31 r4:00000a31

Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 5678 bytes --]


Op 15 jul 2008, om 09:49 heeft Paul Walmsley het volgende geschreven:

> On Tue, 15 Jul 2008, Koen Kooi wrote:
>
>>>> --- a/arch/arm/mach-omap2/pm34xx.c
>>>> +++ b/arch/arm/mach-omap2/pm34xx.c
>>>> @@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
>>>>              goto err1;
>>>>      }
>>>>
>>>> -       ret = pwrdm_for_each(pwrdms_setup);
>>>> -       if (ret) {
>>>> -               printk(KERN_ERR "Failed to setup powerdomains\n");
>>>> -               goto err2;
>>>> -       }
>>>> +/*     ret = pwrdm_for_each(pwrdms_setup); */
>>>> +/*     if (ret) { */
>>>> +/*             printk(KERN_ERR "Failed to setup powerdomains 
>>>> \n"); */
>>>> +/*             goto err2; */
>>>> +/*     } */
>>>>
>>>>      mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
>>>>      if (mpu_pwrdm == NULL) {
>>>
>>> With this change, the hang does not happene. Applied this patch on  
>>> the
>>> current
>>> linux-omap tree.
>>>
>>> Thanks for the patch. It fixes the hang for now. I'm not sure why  
>>> the hang
>>> happens, but could it be because the SDP uses ttyS0 by default as  
>>> the
>>> console while the Beagle is on ttyS2?
>>
>> With that patch I still get the same behaviour:
>>
>> serial console freezes, but responds to sysrq
>
> Likewise on Beagle rev B4 here. Patching the PM code as above does not
> seem to make a difference; nor leaving out the RTC.
>
> The failure mode that seems to recur here is that a userspace binary  
> will
> exit, but no shell prompt is displayed afterwards.  Once wedged,  
> Sysrq-T
> provided some interesting output, included below.   
> uart_wait_until_sent() ?
> That doesn't seem right.
>
> N.B., on the Beagle here, TWL4030 IRQ handling has to be commented  
> out to
> accomplish anything useful.  Otherwise the console drowns in "TWL4030
> module irq 368 is disabled but can't be masked!" messages on at  
> least 90%
> of reboots.

Cold boots seem to 'fix' that.

> This is in drivers/i2c/chips/twl4030-core.c, commenting out
> the lines underneath:
>
>   /* install an irq handler to demultiplex the TWL4030 interrupt */
>
> Have not yet had a chance to study the problem - could be a TWL4030  
> GPIO
> interrupt issue?

As discussed on #beagle yesterday: we suspect i2c bus issues that  
upset the TWL4030, dropping the bus speed to 400kHz (from 2.6MHz)  
seems to fix most issues. There was some disagreement if 400kHz is a  
sane speed when looking at the pull-up resistor values.

regards,

Koen


>
>
>
> - Paul
>
>
> <6>sh            Ssh            S c024cf70  c024cf70     0    
> 362      1
>    0   362      1
> [<c024cd0c>] [<c024cd0c>] (schedule+0x0/0x2a0) (schedule+0x0/0x2a0)  
> from
> [<c024d1d0>] from [<c024d1d0>] (schedule_timeout+0x90/0xb8)
> (schedule_timeout+0x90/0xb8)
> [<c024d140>] [<c024d140>] (schedule_timeout+0x0/0xb8)
> (schedule_timeout+0x0/0xb8) from [<c024d24c>] from [<c024d24c>]
> (schedule_timeout_interruptible+0x28/0x2c)
> (schedule_timeout_interruptible+0x28/0x2c)
> r7:c03257b8 r7:c03257b8 r6:00000001 r6:00000001 r5:ffff888e  
> r5:ffff888e
> r4:c7d9e000 r4:c7d9e000
>
> [<c024d224>] [<c024d224>] (schedule_timeout_interruptible+0x0/0x2c)
> (schedule_timeout_interruptible+0x0/0x2c) from [<c0053fe4>] from
> [<c0053fe4>] (msleep_interruptible+0x28/0x58)
> (msleep_interruptible+0x28/0x58)
> [<c0053fbc>] [<c0053fbc>] (msleep_interruptible+0x0/0x58)
> (msleep_interruptible+0x0/0x58) from [<c017c56c>] from [<c017c56c>]
> (uart_wait_until_sent+0x94/0xf8)
> (uart_wait_until_sent+0x94/0xf8)
> r5:ffff888e r5:ffff888e r4:c7d9e000 r4:c7d9e000
>
> [<c017c4d8>] [<c017c4d8>] (uart_wait_until_sent+0x0/0xf8)
> (uart_wait_until_sent+0x0/0xf8) from [<c016e0cc>] from [<c016e0cc>]
> (tty_wait_until_sent+0xe4/0xf4)
> (tty_wait_until_sent+0xe4/0xf4)
> r7:c7d9e000 r7:c7d9e000 r6:7fffffff r6:7fffffff r5:c7d6b640  
> r5:c7d6b640
> r4:c7d9fd30 r4:c7d9fd30
>
> [<c016dfe8>] [<c016dfe8>] (tty_wait_until_sent+0x0/0xf4)
> (tty_wait_until_sent+0x0/0xf4) from [<c016e2b4>] from [<c016e2b4>]
> (set_termios+0x1d8/0x4cc)
> (set_termios+0x1d8/0x4cc)
> [<c016e0dc>] [<c016e0dc>] (set_termios+0x0/0x4cc) (set_termios 
> +0x0/0x4cc)
> from [<c016e7f4>] from [<c016e7f4>] (tty_mode_ioctl+0x24c/0x3d8)
> (tty_mode_ioctl+0x24c/0x3d8)
> [<c016e5a8>] [<c016e5a8>] (tty_mode_ioctl+0x0/0x3d8)
> (tty_mode_ioctl+0x0/0x3d8) from [<c016ebd8>] from [<c016ebd8>]
> (n_tty_ioctl+0x258/0x268)
> (n_tty_ioctl+0x258/0x268)
> r7:c7c6f740 r7:c7c6f740 r6:beaac3fc r6:beaac3fc r5:beaac3fc  
> r5:beaac3fc
> r4:c7d80400 r4:c7d80400
>
> [<c016e980>] [<c016e980>] (n_tty_ioctl+0x0/0x268) (n_tty_ioctl 
> +0x0/0x268)
> from [<c016b4d4>] from [<c016b4d4>] (tty_ioctl+0xda8/0xe04)
> (tty_ioctl+0xda8/0xe04)
> r7:c7c6f740 r7:c7c6f740 r6:c7d80400 r6:c7d80400 r5:beaac3fc  
> r5:beaac3fc
> r4:00005403 r4:00005403
>
> [<c016a72c>] [<c016a72c>] (tty_ioctl+0x0/0xe04) (tty_ioctl 
> +0x0/0xe04) from
> [<c009fa80>] from [<c009fa80>] (vfs_ioctl+0x34/0x78)
> (vfs_ioctl+0x34/0x78)
> [<c009fa4c>] [<c009fa4c>] (vfs_ioctl+0x0/0x78) (vfs_ioctl+0x0/0x78)  
> from
> [<c009fd34>] from [<c009fd34>] (do_vfs_ioctl+0x270/0x280)
> (do_vfs_ioctl+0x270/0x280)
> r5:beaac3fc r5:beaac3fc r4:c7c6f740 r4:c7c6f740
>
> [<c009fac4>] [<c009fac4>] (do_vfs_ioctl+0x0/0x280)
> (do_vfs_ioctl+0x0/0x280) from [<c009fd84>] from [<c009fd84>]
> (sys_ioctl+0x40/0x64)
> (sys_ioctl+0x40/0x64)
> r7:c7c6f740 r7:c7c6f740 r6:00005403 r6:00005403 r5:beaac3fc  
> r5:beaac3fc
> r4:00000000 r4:00000000
>
> [<c009fd44>] [<c009fd44>] (sys_ioctl+0x0/0x64) (sys_ioctl+0x0/0x64)  
> from
> [<c0027a40>] from [<c0027a40>] (ret_fast_syscall+0x0/0x2c)
> (ret_fast_syscall+0x0/0x2c)
> r7:00000036 r7:00000036 r6:beaac488 r6:beaac488 r5:00000000  
> r5:00000000
> r4:00000a31 r4:00000a31
>
>


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

RE: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Gadiyar, Anand]

> >> diff --git a/arch/arm/mach-omap2/pm34xx.c
> >> b/arch/arm/mach-omap2/pm34xx.c
> >> index 202c269..308835b 100644
> >> --- a/arch/arm/mach-omap2/pm34xx.c
> >> +++ b/arch/arm/mach-omap2/pm34xx.c
> >> @@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
> >>                goto err1;
> >>        }
> >>
> >> -       ret = pwrdm_for_each(pwrdms_setup);
> >> -       if (ret) {
> >> -               printk(KERN_ERR "Failed to setup powerdomains\n");
> >> -               goto err2;
> >> -       }
> >> +/*     ret = pwrdm_for_each(pwrdms_setup); */
> >> +/*     if (ret) { */
> >> +/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
> >> +/*             goto err2; */
> >> +/*     } */
> >>
> >>        mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
> >>        if (mpu_pwrdm == NULL) {
> >
> > With this change, the hang does not happene. Applied this patch on
> > the current linux-omap tree.
> >
> > Thanks for the patch. It fixes the hang for now. I'm not sure why the hang
> > happens, but could it be because the SDP uses ttyS0 by default as the
> > console while the Beagle is on ttyS2?
>
> With that patch I still get the same behaviour:
>
> serial console freezes, but responds to sysrq
>
> I don't know if its related, but after the serial console freezes
> animated backgrounds in X stop animating, but will refresh if you
> press a key on the keyboard. The same effect can be seen with running
> 'top' on the screen.
> I'm using the CPUidle patches posted to the list a few weeks back +
> the timer suppression patch.
>
> regards,
>
> Koen

Okay. Guess I should have waited longer after applying the patch.

- Anand

RE: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Syed Mohammed, Khasim]

>
> > >> diff --git a/arch/arm/mach-omap2/pm34xx.c
> > >> b/arch/arm/mach-omap2/pm34xx.c
> > >> index 202c269..308835b 100644
> > >> --- a/arch/arm/mach-omap2/pm34xx.c
> > >> +++ b/arch/arm/mach-omap2/pm34xx.c
> > >> @@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
> > >>                goto err1;
> > >>        }
> > >>
> > >> -       ret = pwrdm_for_each(pwrdms_setup);
> > >> -       if (ret) {
> > >> -               printk(KERN_ERR "Failed to setup powerdomains\n");
> > >> -               goto err2;
> > >> -       }
> > >> +/*     ret = pwrdm_for_each(pwrdms_setup); */
> > >> +/*     if (ret) { */
> > >> +/*             printk(KERN_ERR "Failed to setup powerdomains\n"); */
> > >> +/*             goto err2; */
> > >> +/*     } */
> > >>
> > >>        mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
> > >>        if (mpu_pwrdm == NULL) {
> > >

Hi Jouni,

Commenting the lines did fix my UART 3 issue, but I am not clear as to what is happening by having it enabled. Before I dig in further can you give your thoughts as to why these lines could result in UART3 hang?

Thanks.

Regards,
Khasim

Re: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1488 bytes --]


Op 31 jul 2008, om 15:08 heeft Syed Mohammed, Khasim het volgende  
geschreven:

>
>>
>>>>> diff --git a/arch/arm/mach-omap2/pm34xx.c
>>>>> b/arch/arm/mach-omap2/pm34xx.c
>>>>> index 202c269..308835b 100644
>>>>> --- a/arch/arm/mach-omap2/pm34xx.c
>>>>> +++ b/arch/arm/mach-omap2/pm34xx.c
>>>>> @@ -380,11 +380,11 @@ int __init omap3_pm_init(void)
>>>>>               goto err1;
>>>>>       }
>>>>>
>>>>> -       ret = pwrdm_for_each(pwrdms_setup);
>>>>> -       if (ret) {
>>>>> -               printk(KERN_ERR "Failed to setup powerdomains\n");
>>>>> -               goto err2;
>>>>> -       }
>>>>> +/*     ret = pwrdm_for_each(pwrdms_setup); */
>>>>> +/*     if (ret) { */
>>>>> +/*             printk(KERN_ERR "Failed to setup powerdomains 
>>>>> \n"); */
>>>>> +/*             goto err2; */
>>>>> +/*     } */
>>>>>
>>>>>       mpu_pwrdm = pwrdm_lookup("mpu_pwrdm");
>>>>>       if (mpu_pwrdm == NULL) {
>>>>
>
> Hi Jouni,
>
> Commenting the lines did fix my UART 3 issue, but I am not clear as  
> to what is happening by having it enabled. Before I dig in further  
> can you give your thoughts as to why these lines could result in  
> UART3 hang?

I've had these lines commented out for some weeks now and still get  
the hang.


>
>
> Thanks.
>
> Regards,
> Khasim
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> omap" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

RE: Kernel hang on OMAP3 based Beagle board, RTC issue?

[Syed Mohammed, Khasim]

Hi Koen:

> -----Original Message-----
> From: Koen Kooi [mailto:k.kooi@student.utwente.nl]
> Sent: Thursday, July 31, 2008 7:03 PM
> To: Syed Mohammed, Khasim
> Cc: Gadiyar, Anand; H?gander Jouni; Dirk Behme; linux-omap@vger.kernel.org
> Subject: Re: Kernel hang on OMAP3 based Beagle board, RTC issue?
>
<snip>
> >>>>
> >
> > Hi Jouni,
> >
> > Commenting the lines did fix my UART 3 issue, but I am not clear as
> > to what is happening by having it enabled. Before I dig in further
> > can you give your thoughts as to why these lines could result in
> > UART3 hang?
>
> I've had these lines commented out for some weeks now and still get
> the hang.
>

I didn't apply any other patches to OMAP GIT, I just cloned a new version of OMAP GIT and built it for Beagle using default config, also commented the lines mentioned by Jouni.

If I don't comment these lines then UART hangs immediately after 30min approx, with lines commented I am running for 10 hrs now with out a hang.

Any way the issue could still exists, I would like to know what exactly this code does to help me in focusing on a specific area.

May be we miss porting some lines of code to Beagle.

Regards,
Khasim