Hang with isync

Hang with isync

[Manoj Sharma]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

Hi,

We use linux kernel 2.4.20 on ppc405 and the system hangs once in a while
when isync gets called in this function:

_GLOBAL(_nmask_and_or_msr)
    mfmsr   r0      /* Get current msr */
    andc    r0,r0,r3    /* And off the bits set in r3 (first parm) */
    or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
    sync            /* Some chip revs have problems here... */
    isync
    mtmsr   r0      /* Update machine state */
    isync
    blr         /* Done */

 2.5 onwards, I find that "sync; isync" has been replaced by a macro SYNC
(defined only for 601). I don't find it in any changelog and reason for the
change.

Can someone give some information on this change?

Appreciate any help.
Manoj

<linuxppc-dev@ozlabs.org>

[-- Attachment #2: Type: text/html, Size: 1235 bytes --]

Re: Hang with isync

[Manoj Sharma]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

>
> Hi,
>
> We use linux kernel 2.4.20 on ppc405 and the system hangs once in a while
> when isync gets called in this function:
>
> _GLOBAL(_nmask_and_or_msr)
>     mfmsr   r0      /* Get current msr */
>     andc    r0,r0,r3    /* And off the bits set in r3 (first parm) */
>     or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
>     sync            /* Some chip revs have problems here... */
>     isync
>     mtmsr   r0      /* Update machine state */
>     isync
>     blr         /* Done */
>
>  2.5 onwards, I find that "sync; isync" has been replaced by a macro SYNC
> (defined only for 601). I don't find it in any changelog and reason for the
> change.
>
> Can someone give some information on this change?
>
> Appreciate any help.
> Manoj
>
> <linuxppc-dev@ozlabs.org>
>

[-- Attachment #2: Type: text/html, Size: 1490 bytes --]

Re: Hang with isync

[Benjamin Herrenschmidt]

On Tue, 2006-09-19 at 18:16 -0700, Manoj Sharma wrote:
>         Hi,
>         
>         We use linux kernel 2.4.20 on ppc405 and the system hangs once
>         in a while when isync gets called in this function: 
>         
>         _GLOBAL(_nmask_and_or_msr)
>             mfmsr   r0      /* Get current msr */
>             andc    r0,r0,r3    /* And off the bits set in r3 (first
>         parm) */ 
>             or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
>             sync            /* Some chip revs have problems here... */
>             isync
>             mtmsr   r0      /* Update machine state */
>             isync
>             blr         /* Done */ 
>         
>          2.5 onwards, I find that "sync; isync" has been replaced by a
>         macro SYNC (defined only for 601). I don't find it in any
>         changelog and reason for the change.
>         
>         Can someone give some information on this change? 

Regardless of the change... on 2.4, _nmask_and_or_msr() was used for a
number of things. We would need to know where it was called from with
what values as arguments to have an idea of what's going wrong. It's
probably not dying on the isync, but rather on the following mtmsr due
to a problem with the values passed in....

Ben.

Re: Hang with isync

[Manoj Sharma]

[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]

This is the stack trace.

Registers:
GPR00: 00069030 C01F3000 C01F1080 00000000 00048000 C0639F48 C01F1080
FFFFFC18
GPR08: C02203FC 00000020 C0638000 C01F31B0 42FEE022 1056A7F8 00FE502A
00000000
GPR16: 00000000 FFC44232 00000000 00000000 FFC441EC 00080000 00010000
0000000A
GPR24: 00000000 0007CD80 00000CE0 00000000 00000000 C02B0000 00000000
C02B0000

NIP; c0005da4 _<_nmask_and_or_msr+0x18/0x20 [kernel]>
Trace; c0025328 _<check_pgt_cache+0x20/0x30 [kernel]>
Trace; c0004f4c _<idled+0x58/0x70 [kernel]>
Trace; c0004f74 _<cpu_idle+0x10/0x24 [kernel]>
Trace; c00012b0 _<rest_init+0x30/0x40 [kernel]>
Trace; c02a45a4 _<start_kernel+0x168/0x17c [kernel]>
Trace; c0000250 _<skpinv+0x1f8/0x234 [kernel]>


On 9/20/06, Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
>
> On Tue, 2006-09-19 at 18:16 -0700, Manoj Sharma wrote:
> >         Hi,
> >
> >         We use linux kernel 2.4.20 on ppc405 and the system hangs once
> >         in a while when isync gets called in this function:
> >
> >         _GLOBAL(_nmask_and_or_msr)
> >             mfmsr   r0      /* Get current msr */
> >             andc    r0,r0,r3    /* And off the bits set in r3 (first
> >         parm) */`
> >             or  r0,r0,r4    /* Or on the bits in r4 (second parm) */
> >             sync            /* Some chip revs have problems here... */
> >             isync
> >             mtmsr   r0      /* Update machine state */
> >             isync
> >             blr         /* Done */
> >
> >          2.5 onwards, I find that "sync; isync" has been replaced by a
> >         macro SYNC (defined only for 601). I don't find it in any
> >         changelog and reason for the change.
> >
> >         Can someone give some information on this change?
>
> Regardless of the change... on 2.4, _nmask_and_or_msr() was used for a
> number of things. We would need to know where it was called from with
> what values as arguments to have an idea of what's going wrong. It's
> probably not dying on the isync, but rather on the following mtmsr due
> to a problem with the values passed in....
>
> Ben.
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3673 bytes --]

Re: Hang with isync

[Benjamin Herrenschmidt]

On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
> This is the stack trace.
> 
> Registers:
> GPR00: 00069030 C01F3000 C01F1080 00000000 00048000 C0639F48 C01F1080
> FFFFFC18
> GPR08: C02203FC 00000020 C0638000 C01F31B0 42FEE022 1056A7F8 00FE502A
> 00000000
> GPR16: 00000000 FFC44232 00000000 00000000 FFC441EC 00080000 00010000
> 0000000A 
> GPR24: 00000000 0007CD80 00000CE0 00000000 00000000 C02B0000 00000000
> C02B0000
> 
> NIP; c0005da4 _<_nmask_and_or_msr+0x18/0x20 [kernel]>
> Trace; c0025328 _<check_pgt_cache+0x20/0x30 [kernel]>
> Trace; c0004f4c _<idled+0x58/0x70 [kernel]> 
> Trace; c0004f74 _<cpu_idle+0x10/0x24 [kernel]>
> Trace; c00012b0 _<rest_init+0x30/0x40 [kernel]>
> Trace; c02a45a4 _<start_kernel+0x168/0x17c [kernel]>
> Trace; c0000250 _<skpinv+0x1f8/0x234 [kernel]> 

Is this upstream 2.6.20 or do you have any additional patches ? (Like
Montavista stuff or RT linux or whatever ?)

It's unclear to me from just that backtrace what mask it is... it could
just be re-enabling interrupt and you have a stale IRQ line asserted....

Have you also checked the errata list for your 405 core, in case it has
a known issue ?

Ben.

> 
> On 9/20/06, Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
>         On Tue, 2006-09-19 at 18:16 -0700, Manoj Sharma wrote:
>         >         Hi,
>         >
>         >         We use linux kernel 2.4.20 on ppc405 and the system
>         hangs once
>         >         in a while when isync gets called in this function: 
>         >
>         >         _GLOBAL(_nmask_and_or_msr)
>         >             mfmsr   r0      /* Get current msr */
>         >             andc    r0,r0,r3    /* And off the bits set in
>         r3 (first
>         >         parm) */`
>         >             or  r0,r0,r4    /* Or on the bits in r4 (second
>         parm) */ 
>         >             sync            /* Some chip revs have problems
>         here... */
>         >             isync
>         >             mtmsr   r0      /* Update machine state */
>         >             isync
>         >             blr         /* Done */ 
>         >
>         >          2.5 onwards, I find that "sync; isync" has been
>         replaced by a
>         >         macro SYNC (defined only for 601). I don't find it
>         in any
>         >         changelog and reason for the change. 
>         >
>         >         Can someone give some information on this change?
>         
>         Regardless of the change... on 2.4, _nmask_and_or_msr() was
>         used for a
>         number of things. We would need to know where it was called
>         from with 
>         what values as arguments to have an idea of what's going
>         wrong. It's
>         probably not dying on the isync, but rather on the following
>         mtmsr due
>         to a problem with the values passed in....
>         
>         Ben.
>         
>         
>         
> 

Re: Hang with isync

[Linas Vepstas]

On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt wrote:
> On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
> > This is the stack trace.
> > 
> > Registers:
> > GPR00: 00069030 

This is the MSR and it has the user-mode bit set, which is surely wrong. 
This is not how one gets to user space.

00048000 

The MSR had this or'ed into it, which is setting the user-mode bit. 
Surely that's wrong.

--linas

Re: Hang with isync

[Manoj Sharma]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

No MSR is 00029030 and user mode bit is not set here.

I had missed it in the prev mail:

\x05NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000\x05 REGS: c01eff30 TRAP:
1020    Not tainted
MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c01f1080[0] 'swapper' Last syscall: 120
last math 00000000 last altivec 00000000
\x05PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000


On 9/20/06, Linas Vepstas <linas@austin.ibm.com> wrote:
>
> On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt wrote:
> > On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
> > > This is the stack trace.
> > >
> > > Registers:
> > > GPR00: 00069030
>
> This is the MSR and it has the user-mode bit set, which is surely wrong.
> This is not how one gets to user space.
>
> 00048000
>
> The MSR had this or'ed into it, which is setting the user-mode bit.
> Surely that's wrong.
>
> --linas
>

[-- Attachment #2: Type: text/html, Size: 1284 bytes --]

RE: Hang with isync

[Liu Dave-r63238]

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]


No MSR is 00029030 and user mode bit is not set here.

I had missed it in the prev mail:

\x05NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000\x05 REGS: c01eff30
TRAP: 1020    Not tainted
MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 
TASK = c01f1080[0] 'swapper' Last syscall: 120
last math 00000000 last altivec 00000000
\x05PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000
 
Dave>I notice that MSR and TRAP, MSR is 00029030- the critical interrupt
enable.
Dave>TRAP is 1020. --WatchDog timer exception is happening
Dave>You can disable the MSR[CE] bit to no critical exception or disable
the WD timer

 

	On 9/20/06, Linas Vepstas <linas@austin.ibm.com> wrote: 

		On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin
Herrenschmidt wrote:
		> On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
		> > This is the stack trace.
		> >
		> > Registers:
		> > GPR00: 00069030 
		
		This is the MSR and it has the user-mode bit set, which
is surely wrong.
		This is not how one gets to user space.
		
		00048000
		
		The MSR had this or'ed into it, which is setting the
user-mode bit.
		Surely that's wrong. 
		
		--linas
		



[-- Attachment #2: Type: text/html, Size: 2624 bytes --]

Re: Hang with isync

[Manoj Sharma]

[-- Attachment #1: Type: text/plain, Size: 1566 bytes --]

Dave, watchdog timeout is around one second and no cpu activity for that
long is something wrong. Is it ok to disable it to hide the problem lying
somewhere else? Do you think it can be because of sync-isync
instructions and moving to 2.6 might resolve it?


On 9/20/06, Liu Dave-r63238 <DaveLiu@freescale.com> wrote:
>
>
> No MSR is 00029030 and user mode bit is not set here.
>
> I had missed it in the prev mail:
>
> \x05NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000\x05 REGS: c01eff30
> TRAP: 1020    Not tainted
> MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
> TASK = c01f1080[0] 'swapper' Last syscall: 120
> last math 00000000 last altivec 00000000
> \x05PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000
>
> Dave>I notice that MSR and TRAP, MSR is 00029030- the critical interrupt
> enable.
> Dave>TRAP is 1020. --WatchDog timer exception is happening
> Dave>You can disable the MSR[CE] bit to no critical exception or disable
> the WD timer
>
>
>
> On 9/20/06, Linas Vepstas <linas@austin.ibm.com> wrote:
> >
> > On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt wrote:
> > > On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
> > > > This is the stack trace.
> > > >
> > > > Registers:
> > > > GPR00: 00069030
> >
> > This is the MSR and it has the user-mode bit set, which is surely wrong.
> > This is not how one gets to user space.
> >
> > 00048000
> >
> > The MSR had this or'ed into it, which is setting the user-mode bit.
> > Surely that's wrong.
> >
> > --linas
> >
>
>

[-- Attachment #2: Type: text/html, Size: 2975 bytes --]

RE: Hang with isync

[Liu Dave-r63238]

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

First, you must make sure if it really happen at watchdog timer
exception.
if it is, you need select one suitable way to fix it.
Second, I don't believe the sync-isync instructions make it happen.
you can try the 2.6, I don't know if 2.6 kernel can resolve your
problem.
 
-Dave



________________________________

	Dave, watchdog timeout is around one second and no cpu activity
for that long is something wrong. Is it ok to disable it to hide the
problem lying somewhere else? Do you think it can be because of
sync-isync instructions and moving to 2.6 might resolve it?

	 
	On 9/20/06, Liu Dave-r63238 <DaveLiu@freescale.com> wrote: 

		

		No MSR is 00029030 and user mode bit is not set here.
		
		I had missed it in the prev mail:
		
		\x05NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000\x05
REGS: c01eff30 TRAP: 1020    Not tainted 
		MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 
		TASK = c01f1080[0] 'swapper' Last syscall: 120
		last math 00000000 last altivec 00000000
		\x05PLB0: bear= 0x08000000 acr=   0xbb000000 besr=
0x00000000
		 
		Dave>I notice that MSR and TRAP, MSR is 00029030- the
critical interrupt enable.
		Dave>TRAP is 1020. --WatchDog timer exception is
happening
		Dave>You can disable the MSR[CE] bit to no critical
exception or disable the WD timer
		

		 

			On 9/20/06, Linas Vepstas <linas@austin.ibm.com
> wrote: 

				On Thu, Sep 21, 2006 at 08:38:13AM
+1000, Benjamin Herrenschmidt wrote:
				> On Wed, 2006-09-20 at 15:31 -0700,
Manoj Sharma wrote: 
				> > This is the stack trace.
				> >
				> > Registers:
				> > GPR00: 00069030 
				
				This is the MSR and it has the user-mode
bit set, which is surely wrong.
				This is not how one gets to user space. 
				
				00048000
				
				The MSR had this or'ed into it, which is
setting the user-mode bit.
				Surely that's wrong. 
				
				--linas
				




[-- Attachment #2: Type: text/html, Size: 4835 bytes --]

Re: Hang with isync

[Manoj Sharma]

[-- Attachment #1: Type: text/plain, Size: 2299 bytes --]

The hang has trigerred watchdog timer exception. It points that there is
problem somewhere but don't know what is it.
Even with 2.4, it does not happen regularly. It occurs once in a while and
not reproducable.

On 9/20/06, Liu Dave-r63238 <DaveLiu@freescale.com> wrote:
>
>  First, you must make sure if it really happen at watchdog timer
> exception.
> if it is, you need select one suitable way to fix it.
> Second, I don't believe the sync-isync instructions make it happen.
> you can try the 2.6, I don't know if 2.6 kernel can resolve your problem.
>
> -Dave
>
>  ------------------------------
>   Dave, watchdog timeout is around one second and no cpu activity for that
> long is something wrong. Is it ok to disable it to hide the problem lying
> somewhere else? Do you think it can be because of sync-isync
> instructions and moving to 2.6 might resolve it?
>
>
> On 9/20/06, Liu Dave-r63238 <DaveLiu@freescale.com> wrote:
> >
> >
> > No MSR is 00029030 and user mode bit is not set here.
> >
> > I had missed it in the prev mail:
> >
> > \x05NIP: C0005DA4 XER: 20000000 LR: C0004FE4 SP: C01F3000\x05 REGS: c01eff30
> > TRAP: 1020    Not tainted
> > MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
> > TASK = c01f1080[0] 'swapper' Last syscall: 120
> > last math 00000000 last altivec 00000000
> > \x05PLB0: bear= 0x08000000 acr=   0xbb000000 besr=  0x00000000
> >
> > Dave>I notice that MSR and TRAP, MSR is 00029030- the critical interrupt
> > enable.
> > Dave>TRAP is 1020. --WatchDog timer exception is happening
> > Dave>You can disable the MSR[CE] bit to no critical exception or disable
> > the WD timer
> >
> >
> >
> > On 9/20/06, Linas Vepstas <linas@austin.ibm.com > wrote:
> > >
> > > On Thu, Sep 21, 2006 at 08:38:13AM +1000, Benjamin Herrenschmidt
> > > wrote:
> > > > On Wed, 2006-09-20 at 15:31 -0700, Manoj Sharma wrote:
> > > > > This is the stack trace.
> > > > >
> > > > > Registers:
> > > > > GPR00: 00069030
> > >
> > > This is the MSR and it has the user-mode bit set, which is surely
> > > wrong.
> > > This is not how one gets to user space.
> > >
> > > 00048000
> > >
> > > The MSR had this or'ed into it, which is setting the user-mode bit.
> > > Surely that's wrong.
> > >
> > > --linas
> > >
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 4865 bytes --]