2.6.8 (or 7?) regression: sleep on older tibooks broken

2.6.8 (or 7?) regression: sleep on older tibooks broken

[David N. Welton]

I was previously using a stock 2.6.6 kernel without problems on my
tibook.  I upgraded to 2.6.8.1 and compiled using:

gcc (GCC) 3.3.4 (Debian 1:3.3.4-6sarge1)

and "Version: 2.14.90.0.7-8" of binutils, in Debian.

I saw there is a similar problem here:

http://lists.linuxppc.org/linuxppc-dev/200407/msg00092.html

but it's not the same problem... I removed the ohci_hcd module from
the kernel (it's present at boot), and sleep still doesn't happen.  I
don't even get the "breathing" light, and yet the computer still seems
warm after some time, seemingly indicative that it's not really asleep
or dead.  I can only restart it via the Ctrl-Command-Power
combination.

Logs don't say anything.

So - what got changed that could have caused this breakage?

Kernel configs and other info available on request.

@ashland [~] $ cat /proc/cpuinfo
processor       : 0
cpu             : 7410, altivec supported
temperature     : 1-76 C (uncalibrated)
clock           : 400MHz
revision        : 17.2 (pvr 800c 1102)
bogomips        : 796.67
machine         : PowerBook3,2
motherboard     : PowerBook3,2 MacRISC2 MacRISC Power Macintosh
detected as     : 71 (PowerBook Titanium)
pmac flags      : 0000000b
L2 cache        : 1024K unified
memory          : 256MB
pmac-generation : NewWorld

Thankyou,
-- 
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[Benjamin Herrenschmidt]

On Sun, 2004-08-15 at 18:45, David N. Welton wrote:

> but it's not the same problem... I removed the ohci_hcd module from
> the kernel (it's present at boot), and sleep still doesn't happen.  I
> don't even get the "breathing" light, and yet the computer still seems
> warm after some time, seemingly indicative that it's not really asleep
> or dead.  I can only restart it via the Ctrl-Command-Power
> combination.

Best thing at this point is to hack out the sleep code in the
video driver to see where it dies during the sleep process...

Ben.

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[David N. Welton]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

> On Sun, 2004-08-15 at 18:45, David N. Welton wrote:

> > but it's not the same problem... I removed the ohci_hcd module
> > from the kernel (it's present at boot), and sleep still doesn't
> > happen.  I don't even get the "breathing" light, and yet the
> > computer still seems warm after some time, seemingly indicative
> > that it's not really asleep or dead.  I can only restart it via
> > the Ctrl-Command-Power combination.

> Best thing at this point is to hack out the sleep code in the video
> driver to see where it dies during the sleep process...

I made the video driver's sleep routing return 0 immediately.

That was enough to at least get a couple of reports from xmon about a
vector 200 corresponding to an address in powerbook_sleep_Core99...
Still investigating, but this is new territory for me, and it's
certainly at a tricky moment in the life of the kernel.  Suggestions
appreciated as to what might have changed and what to look for.

Thankyou,
-- 
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[Benjamin Herrenschmidt]

> I made the video driver's sleep routing return 0 immediately.
> 
> That was enough to at least get a couple of reports from xmon about a
> vector 200 corresponding to an address in powerbook_sleep_Core99...
> Still investigating, but this is new territory for me, and it's
> certainly at a tricky moment in the life of the kernel.  Suggestions
> appreciated as to what might have changed and what to look for.

Ouch, that's pretty bad. 200 is a machine check, looks like the
HW shoked which shouldn't happen there. Can you get me the actual
xmon output and eventually backtrace ? (you can disable the adb
sleep code to get the kbd working in xmon)

Ben.

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[David N. Welton]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

> > I made the video driver's sleep routing return 0 immediately.

> > That was enough to at least get a couple of reports from xmon
> > about a vector 200 corresponding to an address in
> > powerbook_sleep_Core99...  Still investigating, but this is new
> > territory for me, and it's certainly at a tricky moment in the
> > life of the kernel.  Suggestions appreciated as to what might have
> > changed and what to look for.

> Ouch, that's pretty bad. 200 is a machine check, looks like the HW
> shoked which shouldn't happen there. 

Ok, I'll try that.  Piddling around myself, I was tracing the crash to
on or around this code (via-pmu.c:2550):

	/* Save & disable L2 and L3 caches*/
//	save_l3cr = _get_L3CR();	/* (returns -1 if not available) */
//	save_l2cr = _get_L2CR();	/* (returns -1 if not available) */
/* 	if (save_l3cr != 0xffffffff && (save_l3cr & L3CR_L3E) != 0)
		_set_L3CR(save_l3cr & 0x7fffffff);  */
/* 	if (save_l2cr != 0xffffffff && (save_l2cr & L2CR_L2E) != 0)
		_set_L2CR(save_l2cr & 0x7fffffff);  */

commenting it out lets things go to sleep, although they don't seem to
wake up again.

> Can you get me the actual xmon output and eventually backtrace ?
> (you can disable the adb sleep code to get the kbd working in xmon)

that would be adb_notify_sleep?  ... yes, that seems to work ...

Managed to catch this:

Oops: machine check, sig: 7 [#1]
NIP: C025C5D8 LR: C025C768 SP: CD121E50 REGS: cd121da0 TRAP: 0200    Not tainted
MSR: 02043420 EE: 0 PR: 0 FP: 1 ME: 1 IR/DR: 10
TASK = cf4b0660[886] 'pmud' THREAD: cd120000Last syscall: 54 
GPR00: FFFFFFFF CD121E50 CF4B0660 39200000 00400000 00000000 FFFFFF10 02003032 
GPR08: 8014C0AC C025C768 00000004 00000000 88044428 1001E214 10010000 10000000 
GPR16: 10000000 10000000 10000000 7FFFFD48 10000000 00000001 46FFFFFF 00000000 
GPR24: FFFFFFFF 00000000 CD121E60 00000000 B9000000 20004200 FFFFFFE7 C0260000 
NIP [c025c5d8] 0xc025c5d8
LR [c025c768] 0xc025c768
Call trace:
 [c025cf34] 0xc025cf34
 [c006e0e0] sys_ioctl+0xdc/0x2f4
 [c0005f40] ret_from_syscall+0x0/0x44

So it's just getting the ioctcl and going into sleep_Core99.

The instructions around C025C5D8 look like:


c025c584:       48 00 9c 8d     bl      c0266210 <pmu_request>
c025c588:       7f 43 d3 78     mr      r3,r26
c025c58c:       48 00 9e 51     bl      c02663dc <pmu_wait_complete>
c025c590:       4b db 0e e1     bl      c000d470 <_get_L3CR>
c025c594:       7c 7b 1b 78     mr      r27,r3
c025c598:       7c 78 18 f8     not     r24,r3
c025c59c:       57 77 0f fe     rlwinm  r23,r27,1,31,31
c025c5a0:       4b db 0d d1     bl      c000d370 <_get_L2CR>
c025c5a4:       31 38 ff ff     addic   r9,r24,-1
c025c5a8:       7c 09 c1 10     subfe   r0,r9,r24
c025c5ac:       7c 7c 1b 78     mr      r28,r3
c025c5b0:       7c 09 b8 39     and.    r9,r0,r23
c025c5b4:       40 82 01 b8     bne-    c025c76c <powerbook_sleep_Core99+0x298>
c025c5b8:       7f 96 e0 f8     not     r22,r28
c025c5bc:       57 95 0f fe     rlwinm  r21,r28,1,31,31
c025c5c0:       31 36 ff ff     addic   r9,r22,-1
c025c5c4:       7c 09 b1 10     subfe   r0,r9,r22
c025c5c8:       7c 09 a8 39     and.    r9,r0,r21
c025c5cc:       40 82 01 94     bne-    c025c760 <powerbook_sleep_Core99+0x28c>
c025c5d0:       3f e0 c0 26     lis     r31,-16346
c025c5d4:       80 1f 78 c4     lwz     r0,30916(r31)
c025c5d8:       2c 00 00 00     cmpwi   r0,0
c025c5dc:       41 82 01 54     beq-    c025c730 <powerbook_sleep_Core99+0x25c>
c025c5e0:       3f 20 c0 27     lis     r25,-16345
c025c5e4:       4b ff f8 91     bl      c025be74 <save_via_state>
c025c5e8:       39 39 93 54     addi    r9,r25,-27820
c025c5ec:       80 09 00 94     lwz     r0,148(r9)
c025c5f0:       2c 00 00 00     cmpwi   r0,0
c025c5f4:       40 82 01 1c     bne-    c025c710 <powerbook_sleep_Core99+0x23c>
c025c5f8:       80 1f 78 c4     lwz     r0,30916(r31)
c025c5fc:       2c 00 00 00     cmpwi   r0,0
c025c600:       41 82 01 08     beq-    c025c708 <powerbook_sleep_Core99+0x234>
c025c604:       3f c0 01 21     lis     r30,289
c025c608:       3b e0 13 87     li      r31,4999
c025c60c:       3f a0 c0 23     lis     r29,-16349
c025c610:       63 de ea c0     ori     r30,r30,60096

I don't have too many ideas.  If it's useful, I suppose I can try
backing of to 2.6.7 to see if it suffers from the same problem...

Thankyou,
-- 
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[David N. Welton]

davidw@dedasys.com (David N. Welton) writes:

> If it's useful, I suppose I can try backing of to 2.6.7 to see if it
> suffers from the same problem...

2.6.7 is also broken.  Fresh 2.6.7 compiled, and it doesn't work
either.

Kernel config at dedasys.com/kernelconfig

Thanks,
-- 
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[Benjamin Herrenschmidt]

On Tue, 2004-08-17 at 04:38, David N. Welton wrote:
> davidw@dedasys.com (David N. Welton) writes:
> 
> > If it's useful, I suppose I can try backing of to 2.6.7 to see if it
> > suffers from the same problem...
> 
> 2.6.7 is also broken.  Fresh 2.6.7 compiled, and it doesn't work
> either.

Can you try going backward in time find when it broke ?

Ben.

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[David N. Welton]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

> On Tue, 2004-08-17 at 04:38, David N. Welton wrote:
> > davidw@dedasys.com (David N. Welton) writes:

> > > If it's useful, I suppose I can try backing of to 2.6.7 to see
> > > if it suffers from the same problem...

> > 2.6.7 is also broken.  Fresh 2.6.7 compiled, and it doesn't work
> > either.

> Can you try going backward in time find when it broke ?

2.6.6 sleeps/wakes just fine.  I had a halfhearted look through the 7
2.6.patch, searching on things like 'ppc' and 'macintosh', but didn't
2.6.see much that jumped out at me.

Thanks,
-- 
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[Benjamin Herrenschmidt]

> 2.6.6 sleeps/wakes just fine.  I had a halfhearted look through the 7
> 2.6.patch, searching on things like 'ppc' and 'macintosh', but didn't
> 2.6.see much that jumped out at me.

Best would be if you could try the various 2.6.7-rc patches ...

Ben.

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[David N. Welton]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

> > 2.6.6 sleeps/wakes just fine.  I had a halfhearted look through
> > the 7 2.6.patch, searching on things like 'ppc' and 'macintosh',
> > but didn't 2.6.see much that jumped out at me.

> Best would be if you could try the various 2.6.7-rc patches ...

 .... patching, compiling ....

rc1 sleeps/wakes, rc2 does not.

-- 
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[David N. Welton]

davidw@dedasys.com (David N. Welton) writes:

> Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
> 
> > > 2.6.6 sleeps/wakes just fine.  I had a halfhearted look through
> > > the 7 2.6.patch, searching on things like 'ppc' and 'macintosh',
> > > but didn't 2.6.see much that jumped out at me.
> 
> > Best would be if you could try the various 2.6.7-rc patches ...
> 
>  .... patching, compiling ....
> 
> rc1 sleeps/wakes, rc2 does not.

The only thing that jumps out at me is that arch/ppc/mm/cachemap.c
'moved' to arch/ppc/kernel/dma-mapping.c and seems to have changed
some as well.  Other than that, I would need more suggestions for
debugging.

Thankyou,
-- 
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[Eugene Surovegin]

On Sat, Aug 21, 2004 at 10:42:51PM +0200, David N. Welton wrote:
> The only thing that jumps out at me is that arch/ppc/mm/cachemap.c
> 'moved' to arch/ppc/kernel/dma-mapping.c and seems to have changed
> some as well. 

dma-mapping.c is used only on non-coherent cache CPUs (like 4xx, 8xx). 
It's not relevant and is not even compiled for powermacs.

Eugene

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[Benjamin Herrenschmidt]

> The only thing that jumps out at me is that arch/ppc/mm/cachemap.c
> 'moved' to arch/ppc/kernel/dma-mapping.c and seems to have changed
> some as well.  Other than that, I would need more suggestions for
> debugging.

No, that's probably irrelevant. I'll have a look when I'm back, I've
been away for the week-end.

Ben.

Re: 2.6.8 (or 7?) regression: sleep on older tibooks broken

[Benjamin Herrenschmidt]

> Ok, I'll try that.  Piddling around myself, I was tracing the crash to
> on or around this code (via-pmu.c:2550):
> 
> 	/* Save & disable L2 and L3 caches*/
> //	save_l3cr = _get_L3CR();	/* (returns -1 if not available) */
> //	save_l2cr = _get_L2CR();	/* (returns -1 if not available) */
> /* 	if (save_l3cr != 0xffffffff && (save_l3cr & L3CR_L3E) != 0)
> 		_set_L3CR(save_l3cr & 0x7fffffff);  */
> /* 	if (save_l2cr != 0xffffffff && (save_l2cr & L2CR_L2E) != 0)
> 		_set_L2CR(save_l2cr & 0x7fffffff);  */
> 
> commenting it out lets things go to sleep, although they don't seem to
> wake up again.

Yah, you aren't disabling the cache ... I fail to see how flushing &
disabling the L2 would cause a machine check tho, it's really weird.

Yes, try 2.6.7 and let me know.

Ben.

> > Can you get me the actual xmon output and eventually backtrace ?
> > (you can disable the adb sleep code to get the kbd working in xmon)
> 
> that would be adb_notify_sleep?  ... yes, that seems to work ...
> 
> Managed to catch this:
> 
> Oops: machine check, sig: 7 [#1]
> NIP: C025C5D8 LR: C025C768 SP: CD121E50 REGS: cd121da0 TRAP: 0200    Not tainted
> MSR: 02043420 EE: 0 PR: 0 FP: 1 ME: 1 IR/DR: 10
> TASK = cf4b0660[886] 'pmud' THREAD: cd120000Last syscall: 54
> GPR00: FFFFFFFF CD121E50 CF4B0660 39200000 00400000 00000000 FFFFFF10 02003032
> GPR08: 8014C0AC C025C768 00000004 00000000 88044428 1001E214 10010000 10000000
> GPR16: 10000000 10000000 10000000 7FFFFD48 10000000 00000001 46FFFFFF 00000000
> GPR24: FFFFFFFF 00000000 CD121E60 00000000 B9000000 20004200 FFFFFFE7 C0260000
> NIP [c025c5d8] 0xc025c5d8
> LR [c025c768] 0xc025c768
> Call trace:
>  [c025cf34] 0xc025cf34
>  [c006e0e0] sys_ioctl+0xdc/0x2f4
>  [c0005f40] ret_from_syscall+0x0/0x44
> 
> So it's just getting the ioctcl and going into sleep_Core99.
> 
> The instructions around C025C5D8 look like:
> 
> 
> c025c584:       48 00 9c 8d     bl      c0266210 <pmu_request>
> c025c588:       7f 43 d3 78     mr      r3,r26
> c025c58c:       48 00 9e 51     bl      c02663dc <pmu_wait_complete>
> c025c590:       4b db 0e e1     bl      c000d470 <_get_L3CR>
> c025c594:       7c 7b 1b 78     mr      r27,r3
> c025c598:       7c 78 18 f8     not     r24,r3
> c025c59c:       57 77 0f fe     rlwinm  r23,r27,1,31,31
> c025c5a0:       4b db 0d d1     bl      c000d370 <_get_L2CR>
> c025c5a4:       31 38 ff ff     addic   r9,r24,-1
> c025c5a8:       7c 09 c1 10     subfe   r0,r9,r24
> c025c5ac:       7c 7c 1b 78     mr      r28,r3
> c025c5b0:       7c 09 b8 39     and.    r9,r0,r23
> c025c5b4:       40 82 01 b8     bne-    c025c76c <powerbook_sleep_Core99+0x298>
> c025c5b8:       7f 96 e0 f8     not     r22,r28
> c025c5bc:       57 95 0f fe     rlwinm  r21,r28,1,31,31
> c025c5c0:       31 36 ff ff     addic   r9,r22,-1
> c025c5c4:       7c 09 b1 10     subfe   r0,r9,r22
> c025c5c8:       7c 09 a8 39     and.    r9,r0,r21
> c025c5cc:       40 82 01 94     bne-    c025c760 <powerbook_sleep_Core99+0x28c>
> c025c5d0:       3f e0 c0 26     lis     r31,-16346
> c025c5d4:       80 1f 78 c4     lwz     r0,30916(r31)
> c025c5d8:       2c 00 00 00     cmpwi   r0,0
> c025c5dc:       41 82 01 54     beq-    c025c730 <powerbook_sleep_Core99+0x25c>
> c025c5e0:       3f 20 c0 27     lis     r25,-16345
> c025c5e4:       4b ff f8 91     bl      c025be74 <save_via_state>
> c025c5e8:       39 39 93 54     addi    r9,r25,-27820
> c025c5ec:       80 09 00 94     lwz     r0,148(r9)
> c025c5f0:       2c 00 00 00     cmpwi   r0,0
> c025c5f4:       40 82 01 1c     bne-    c025c710 <powerbook_sleep_Core99+0x23c>
> c025c5f8:       80 1f 78 c4     lwz     r0,30916(r31)
> c025c5fc:       2c 00 00 00     cmpwi   r0,0
> c025c600:       41 82 01 08     beq-    c025c708 <powerbook_sleep_Core99+0x234>
> c025c604:       3f c0 01 21     lis     r30,289
> c025c608:       3b e0 13 87     li      r31,4999
> c025c60c:       3f a0 c0 23     lis     r29,-16349
> c025c610:       63 de ea c0     ori     r30,r30,60096
> 
> I don't have too many ideas.  If it's useful, I suppose I can try
> backing of to 2.6.7 to see if it suffers from the same problem...
> 
> Thankyou,
> --
> David N. Welton
>      Personal: http://www.dedasys.com/davidw/
> Free Software: http://www.dedasys.com/freesoftware/
>    Apache Tcl: http://tcl.apache.org/
>        Photos: http://www.dedasys.com/photos/
> 
> ** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/
-- 
Benjamin Herrenschmidt <benh@kernel.crashing.org>