gdb hangs on Linux 2.6.11 on 8xx

gdb hangs on Linux 2.6.11 on 8xx

[Chris]

Hi,

I am running Linux 2.6.11 on a TQM823L development board. I find that 
gdb (version 6.3) hangs when I try to run an application - any 
application, even a simple helloworld. It's not just gdb hanging, the 
whole system is stone dead.

Before I start digging into the problem I would like to ask if anybody 
on this list has any experience, good or bad, of gdb on 2.6.xx on 8xx.

TIA,

Chris.

Re: gdb hangs on Linux 2.6.11 on 8xx

[Aristeu Sergio Rozanski Filho]

> I am running Linux 2.6.11 on a TQM823L development board. I find that 
> gdb (version 6.3) hangs when I try to run an application - any 
> application, even a simple helloworld. It's not just gdb hanging, the 
> whole system is stone dead.
"me too"
actually the system is not completely dead, you still can use sysrq keys.
Tried to use gdbserver too, but it also doesn't work (hangs after the first
breakpoint and timeouts).

gdb           S 30106778     0   691    687   692               (NOTLB)
Call trace:
 [c0005330] __switch_to+0x44/0x78
 [c015beb0] schedule+0x30c/0x730
 [c015c888] schedule_timeout+0xdc/0xe0
 [c0068918] sys_poll+0x27c/0x404
 [c0002810] syscall_dotrace_cont+0x24/0x38
flash_test    t 100009E4     0   692    691                     (NOTLB)
Call trace:
 [c0005330] __switch_to+0x44/0x78
 [c015beb0] schedule+0x30c/0x730
 [c0026248] ptrace_stop+0x68/0x88
 [c0026604] get_signal_to_deliver+0x130/0x2b4
 [c00063dc] do_signal+0x38/0x464
 [c0003034] do_user_signal+0x74/0xc4

-- 
Aristeu

Re: gdb hangs on Linux 2.6.11 on 8xx

[Chris]

Aristeu Sergio Rozanski Filho wrote:
>>I am running Linux 2.6.11 on a TQM823L development board. I find that 
>>gdb (version 6.3) hangs when I try to run an application - any 
>>application, even a simple helloworld. It's not just gdb hanging, the 
>>whole system is stone dead.
> 
> "me too"
> actually the system is not completely dead, you still can use sysrq keys.
> Tried to use gdbserver too, but it also doesn't work (hangs after the first
> breakpoint and timeouts).
> 
> gdb           S 30106778     0   691    687   692               (NOTLB)
> Call trace:
>  [c0005330] __switch_to+0x44/0x78
>  [c015beb0] schedule+0x30c/0x730
>  [c015c888] schedule_timeout+0xdc/0xe0
>  [c0068918] sys_poll+0x27c/0x404
>  [c0002810] syscall_dotrace_cont+0x24/0x38
> flash_test    t 100009E4     0   692    691                     (NOTLB)
> Call trace:
>  [c0005330] __switch_to+0x44/0x78
>  [c015beb0] schedule+0x30c/0x730
>  [c0026248] ptrace_stop+0x68/0x88
>  [c0026604] get_signal_to_deliver+0x130/0x2b4
>  [c00063dc] do_signal+0x38/0x464
>  [c0003034] do_user_signal+0x74/0xc4
> 

Good to know that it's not just me... As you point out it's not
completely dead. A register dump shows the following:

SysRq : Show Regs
NIP: C00047BC LR: C000A0C0 SP: C09FBDD0 REGS: c09fbd20 TRAP: 0501    Not
tainted
MSR: 00009022 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 10
TASK = c031f3e0[187] 'gdb' THREAD: c09fa000
Last syscall: 26
GPR00: 00009022 C09FBDD0 C031F3E0 00589000 00000100 C01F3000 00588000
808B0000
GPR08: 38600000 C01D0000 00009032 0000B100 33003035
NIP [c00047bc] __flush_dcache_icache_phys+0x38/0x54
LR [c000a0c0] flush_dcache_icache_page+0x20/0x30
Call trace:
  [c000a17c] update_mmu_cache+0x68/0x98
  [c003dbe8] do_wp_page+0x184/0x3a4
  [c003ebc8] handle_mm_fault+0x29c/0x42c
  [c003ee7c] get_user_pages+0x124/0x428
  [c001a030] access_process_vm+0xfc/0x1bc
  [c0006680] sys_ptrace+0x22c/0x4b4
  [c0002590] ret_from_syscall+0x0/0x44


Which looks like the call to ptrace caused a page fault, after which it
to stopped in __flush_dcache_icache_phys for some reason. Maybe some
serious digging is needed after all.

Chris.

Re: gdb hangs on Linux 2.6.11 on 8xx

[Marcelo Tosatti]

Folks, 

Can you please upgrade to a recent kernel? 

A problem with ptrace has been fixed sometime ago, it might be what you're hitting.

The fix is part of v2.6.13:

http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=bf85fa6c878aa3968df47d7f70a2b506c3e53b99

If that does not work, complain. 

On Tue, Sep 13, 2005 at 09:25:41AM -0300, Aristeu Sergio Rozanski Filho wrote:
> > I am running Linux 2.6.11 on a TQM823L development board. I find that 
> > gdb (version 6.3) hangs when I try to run an application - any 
> > application, even a simple helloworld. It's not just gdb hanging, the 
> > whole system is stone dead.
> "me too"
> actually the system is not completely dead, you still can use sysrq keys.
> Tried to use gdbserver too, but it also doesn't work (hangs after the first
> breakpoint and timeouts).
> 
> gdb           S 30106778     0   691    687   692               (NOTLB)
> Call trace:
>  [c0005330] __switch_to+0x44/0x78
>  [c015beb0] schedule+0x30c/0x730
>  [c015c888] schedule_timeout+0xdc/0xe0
>  [c0068918] sys_poll+0x27c/0x404
>  [c0002810] syscall_dotrace_cont+0x24/0x38
> flash_test    t 100009E4     0   692    691                     (NOTLB)
> Call trace:
>  [c0005330] __switch_to+0x44/0x78
>  [c015beb0] schedule+0x30c/0x730
>  [c0026248] ptrace_stop+0x68/0x88
>  [c0026604] get_signal_to_deliver+0x130/0x2b4
>  [c00063dc] do_signal+0x38/0x464
>  [c0003034] do_user_signal+0x74/0xc4
> 
> -- 
> Aristeu
> 
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

Re: gdb hangs on Linux 2.6.11 on 8xx

[Aristeu Sergio Rozanski Filho]

> Can you please upgrade to a recent kernel? 
> 
> A problem with ptrace has been fixed sometime ago, it might be what you're hitting.
> 
> The fix is part of v2.6.13:
> 
> http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=bf85fa6c878aa3968df47d7f70a2b506c3e53b99
> 
> If that does not work, complain. 
sorry, but it's a 2.6.10 with this patch applied and I did test with
2.6.13 kernel. I'll re-test with current Linus tree.

-- 
Aristeu

Re: gdb hangs on Linux 2.6.11 on 8xx

[Chris]

Marcelo Tosatti wrote:
> Folks, 
> 
> Can you please upgrade to a recent kernel? 
> 
> A problem with ptrace has been fixed sometime ago, it might be what you're hitting.
> 
> The fix is part of v2.6.13:
> 
> http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=bf85fa6c878aa3968df47d7f70a2b506c3e53b99
> 
> If that does not work, complain. 
> 

OK, that works. Gdb and gdbserver behave as they should.

Thanks,
Chris.

Re: gdb hangs on Linux 2.6.11 on 8xx

[Marcelo Tosatti]

On Tue, Sep 13, 2005 at 09:25:41AM -0300, Aristeu Sergio Rozanski Filho wrote:
> > I am running Linux 2.6.11 on a TQM823L development board. I find that 
> > gdb (version 6.3) hangs when I try to run an application - any 
> > application, even a simple helloworld. It's not just gdb hanging, the 
> > whole system is stone dead.
> "me too"
> actually the system is not completely dead, you still can use sysrq keys.
> Tried to use gdbserver too, but it also doesn't work (hangs after the first
> breakpoint and timeouts).
> 
> gdb           S 30106778     0   691    687   692               (NOTLB)
> Call trace:
>  [c0005330] __switch_to+0x44/0x78
>  [c015beb0] schedule+0x30c/0x730
>  [c015c888] schedule_timeout+0xdc/0xe0
>  [c0068918] sys_poll+0x27c/0x404
>  [c0002810] syscall_dotrace_cont+0x24/0x38
> flash_test    t 100009E4     0   692    691                     (NOTLB)
> Call trace:
>  [c0005330] __switch_to+0x44/0x78
>  [c015beb0] schedule+0x30c/0x730
>  [c0026248] ptrace_stop+0x68/0x88
>  [c0026604] get_signal_to_deliver+0x130/0x2b4
>  [c00063dc] do_signal+0x38/0x464
>  [c0003034] do_user_signal+0x74/0xc4

Aris,

How does the disassemble of __switch_to looks like? 

Chris oopsen is exactly what the 'icbi' patch fixes.

Maybe this is an additional problem then.

SysRq : Show Regs
NIP: C00047BC LR: C000A0C0 SP: C09FBDD0 REGS: c09fbd20 TRAP: 0501    Not
tainted
MSR: 00009022 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 10
TASK = c031f3e0[187] 'gdb' THREAD: c09fa000
Last syscall: 26
GPR00: 00009022 C09FBDD0 C031F3E0 00589000 00000100 C01F3000 00588000
808B0000
GPR08: 38600000 C01D0000 00009032 0000B100 33003035
NIP [c00047bc] __flush_dcache_icache_phys+0x38/0x54
LR [c000a0c0] flush_dcache_icache_page+0x20/0x30
Call trace:
 [c000a17c] update_mmu_cache+0x68/0x98
 [c003dbe8] do_wp_page+0x184/0x3a4
 [c003ebc8] handle_mm_fault+0x29c/0x42c
 [c003ee7c] get_user_pages+0x124/0x428
 [c001a030] access_process_vm+0xfc/0x1bc
 [c0006680] sys_ptrace+0x22c/0x4b4
 [c0002590] ret_from_syscall+0x0/0x44

Re: gdb hangs on Linux 2.6.11 on 8xx

[Aristeu Sergio Rozanski Filho]

Hi Marcelo,
	after the last Chris post, I decided to double check if I wasn't doing
something wrong. vanilla 2.6.13 hangs in the same way my patched 2.6.10 does.

> How does the disassemble of __switch_to looks like? 
from 2.6.13 kernel
Dump of assembler code for function __switch_to:
0xc00055a4 <__switch_to+0>:     mflr    r0
0xc00055a8 <__switch_to+4>:     stwu    r1,-16(r1)
0xc00055ac <__switch_to+8>:     stw     r31,12(r1)
0xc00055b0 <__switch_to+12>:    mr      r11,r4
0xc00055b4 <__switch_to+16>:    stw     r0,20(r1)
0xc00055b8 <__switch_to+20>:    mfmsr   r31
0xc00055bc <__switch_to+24>:    rlwinm  r0,r31,0,17,15
0xc00055c0 <__switch_to+28>:    mtmsr   r0
0xc00055c4 <__switch_to+32>:    lwz     r10,460(r4)
0xc00055c8 <__switch_to+36>:    addi    r3,r2,456
0xc00055cc <__switch_to+40>:    addi    r4,r4,456
0xc00055d0 <__switch_to+44>:    cmpwi   cr7,r10,0
0xc00055d4 <__switch_to+48>:    beq-    cr7,0xc00055e8 <__switch_to+68>
0xc00055d8 <__switch_to+52>:    lis     r9,-16354
0xc00055dc <__switch_to+56>:    lwz     r0,28880(r9)
0xc00055e0 <__switch_to+60>:    cmpw    cr7,r0,r11
0xc00055e4 <__switch_to+64>:    beq-    cr7,0xc0005604 <__switch_to+96>
0xc00055e8 <__switch_to+68>:    bl      0xc0003024 <_switch>
0xc00055ec <__switch_to+72>:    mtmsr   r31
0xc00055f0 <__switch_to+76>:    lwz     r0,20(r1)
0xc00055f4 <__switch_to+80>:    lwz     r31,12(r1)
0xc00055f8 <__switch_to+84>:    addi    r1,r1,16
0xc00055fc <__switch_to+88>:    mtlr    r0
0xc0005600 <__switch_to+92>:    blr
0xc0005604 <__switch_to+96>:    lwz     r0,132(r10)
0xc0005608 <__switch_to+100>:   oris    r0,r0,512
0xc000560c <__switch_to+104>:   stw     r0,132(r10)
0xc0005610 <__switch_to+108>:   b       0xc00055e8 <__switch_to+68>
End of assembler dump.

> Chris oopsen is exactly what the 'icbi' patch fixes.
notice that what I'm having here is not an oops

but if for Chris works, I must be doing something wrong, sorry for the noise

-- 
Aristeu

Re: gdb hangs on Linux 2.6.11 on 8xx

[Marcelo Tosatti]

On Tue, Sep 13, 2005 at 02:22:13PM -0300, Aristeu Sergio Rozanski Filho wrote:
> Hi Marcelo,
> 	after the last Chris post, I decided to double check if I wasn't doing
> something wrong. vanilla 2.6.13 hangs in the same way my patched 2.6.10 does.
> 
> > How does the disassemble of __switch_to looks like? 
> from 2.6.13 kernel
> Dump of assembler code for function __switch_to:
> 0xc00055a4 <__switch_to+0>:     mflr    r0
> 0xc00055a8 <__switch_to+4>:     stwu    r1,-16(r1)
> 0xc00055ac <__switch_to+8>:     stw     r31,12(r1)
> 0xc00055b0 <__switch_to+12>:    mr      r11,r4
> 0xc00055b4 <__switch_to+16>:    stw     r0,20(r1)
> 0xc00055b8 <__switch_to+20>:    mfmsr   r31
> 0xc00055bc <__switch_to+24>:    rlwinm  r0,r31,0,17,15
> 0xc00055c0 <__switch_to+28>:    mtmsr   r0
> 0xc00055c4 <__switch_to+32>:    lwz     r10,460(r4)
> 0xc00055c8 <__switch_to+36>:    addi    r3,r2,456
> 0xc00055cc <__switch_to+40>:    addi    r4,r4,456
> 0xc00055d0 <__switch_to+44>:    cmpwi   cr7,r10,0
> 0xc00055d4 <__switch_to+48>:    beq-    cr7,0xc00055e8 <__switch_to+68>
> 0xc00055d8 <__switch_to+52>:    lis     r9,-16354
> 0xc00055dc <__switch_to+56>:    lwz     r0,28880(r9)
> 0xc00055e0 <__switch_to+60>:    cmpw    cr7,r0,r11
> 0xc00055e4 <__switch_to+64>:    beq-    cr7,0xc0005604 <__switch_to+96>
> 0xc00055e8 <__switch_to+68>:    bl      0xc0003024 <_switch>
> 0xc00055ec <__switch_to+72>:    mtmsr   r31
> 0xc00055f0 <__switch_to+76>:    lwz     r0,20(r1)
> 0xc00055f4 <__switch_to+80>:    lwz     r31,12(r1)
> 0xc00055f8 <__switch_to+84>:    addi    r1,r1,16
> 0xc00055fc <__switch_to+88>:    mtlr    r0
> 0xc0005600 <__switch_to+92>:    blr
> 0xc0005604 <__switch_to+96>:    lwz     r0,132(r10)
> 0xc0005608 <__switch_to+100>:   oris    r0,r0,512
> 0xc000560c <__switch_to+104>:   stw     r0,132(r10)
> 0xc0005610 <__switch_to+108>:   b       0xc00055e8 <__switch_to+68>
> End of assembler dump.
> 
> > Chris oopsen is exactly what the 'icbi' patch fixes.
> notice that what I'm having here is not an oops
> 
> but if for Chris works, I must be doing something wrong, sorry for the noise

Hi Aris,

 gdb           S 30106778     0   691    687   692               (NOTLB)
 Call trace:
  [c0005330] __switch_to+0x44/0x78
  [c015beb0] schedule+0x30c/0x730
  [c015c888] schedule_timeout+0xdc/0xe0
  [c0068918] sys_poll+0x27c/0x404
  [c0002810] syscall_dotrace_cont+0x24/0x38
 flash_test    t 100009E4     0   692    691                     (NOTLB)
 Call trace:
  [c0005330] __switch_to+0x44/0x78
  [c015beb0] schedule+0x30c/0x730
  [c0026248] ptrace_stop+0x68/0x88
  [c0026604] get_signal_to_deliver+0x130/0x2b4
  [c00063dc] do_signal+0x38/0x464
  [c0003034] do_user_signal+0x74/0xc4

flash_test is trying to handle a signal, can you print some information
about it in get_signal_to_deliver (before ptrace_stop), such as si_signo:

typedef struct siginfo {
        int si_signo;
        int si_errno;
        int si_code;

ptrace_stop() calls do_notify_parent_cldstop() to wakeup gdb, maybe
there's something wrong during wakeup?

The box locks up completly or its just gdb that freezes? 

Re: gdb hangs on Linux 2.6.11 on 8xx

[Aristeu Sergio Rozanski Filho]

Hi Marcelo,

> flash_test is trying to handle a signal, can you print some information
> about it in get_signal_to_deliver (before ptrace_stop), such as si_signo:
> 
> typedef struct siginfo {
>         int si_signo;
>         int si_errno;
>         int si_code;
> 
> ptrace_stop() calls do_notify_parent_cldstop() to wakeup gdb, maybe
> there's something wrong during wakeup?
> 
> The box locks up completly or its just gdb that freezes?
seems it's more simple than I thought: seems to be a problem with serial
console as I'm able to use gdb using ssh
Is serial console (using ttyCPM) supposed to work with gdb?

-- 
Aristeu

Re: gdb hangs on Linux 2.6.11 on 8xx

[Marcelo Tosatti]

On Wed, Sep 14, 2005 at 12:42:18PM -0300, Aristeu Sergio Rozanski Filho wrote:
> Hi Marcelo,
> 
> > flash_test is trying to handle a signal, can you print some information
> > about it in get_signal_to_deliver (before ptrace_stop), such as si_signo:
> > 
> > typedef struct siginfo {
> >         int si_signo;
> >         int si_errno;
> >         int si_code;
> > 
> > ptrace_stop() calls do_notify_parent_cldstop() to wakeup gdb, maybe
> > there's something wrong during wakeup?
> > 
> > The box locks up completly or its just gdb that freezes?
> seems it's more simple than I thought: seems to be a problem with serial
> console as I'm able to use gdb using ssh
> Is serial console (using ttyCPM) supposed to work with gdb?

Using a different device than the one used by the serial console? Yes, 
think so..

Re: gdb hangs on Linux 2.6.11 on 8xx

[Aristeu Sergio Rozanski Filho]

> Using a different device than the one used by the serial console? Yes, 
> think so..
no, I'm not trying to use gdb over serial... :)

-- 
Aristeu