Gdbserver syscall clobber

Gdbserver syscall clobber

[Bill Gatliff]

Guys:


I'm trying to track down a problem where gdbserver is issuing a bogus 
syscall at the onset of a debugging session.  I'm using gdb-6.6, on an 
(ancient, I know) linux-2.4.16 kernel, pc603e machine.  Gcc-3.4.5, 
glibc-2.2.5 (both built with crosstools-0.43).  Gdbserver is statically 
linked.

When I run gdbserver under strace on the target, I see these in the log 
shortly after initiating the connection from my workstation:

...
ptrace(PTRACE_PEEKTEXT, 947, 0x10405394, [0x103e0cb0]) = 0
ptrace(PTRACE_PEEKTEXT, 947, 0x10405398, [0x103e0ce4]) = 0
ptrace(PTRACE_PEEKTEXT, 947, 0x1040539c, [0x103e0ce8]) = 0
send(4, "$103df2cc103df2e8103df2ec103df2f"..., 644, 0) = 644
recv(4, 0x7ffffd60, 1, 0)               = ? ERESTARTSYS (To be restarted)
--- SIGIO (I/O possible) @ 0 (0) ---
syscall_4294966784(0xa, 0x7ffffd34, 0x1, 0, 0x1008a3c7, 0x1008b5a3, 
0x1008b5a4, 0, 0x1, 0x80808080, 0x1008e778, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0x10080000, 0x10080000, 0x10080000, 0x1008b5a3, 0x10080000, 
0x1008b320, 0x100291a4, 0xd032, 0xa) = -1 (errno 38)
write(2, "putpkt(read): Function not imple"..., 39) = 39
...


Note the bogus syscall argument 4294966784.  It isn't *completely* 
bogus, interestingly, since if you google for that you come up with a 
few hits.  But no resolutions.

I'm stumped.  Does this problem sound familiar to anyone?  Had the same 
problem with gcc-2.95.3.


Kindest regards,


b.g.

-- 
Bill Gatliff
bgat@billgatliff.com

Re: Gdbserver syscall clobber

[Daniel Jacobowitz]

On Mon, Jul 16, 2007 at 10:43:41AM -0500, Bill Gatliff wrote:
> recv(4, 0x7ffffd60, 1, 0)               = ? ERESTARTSYS (To be restarted)
> --- SIGIO (I/O possible) @ 0 (0) ---
> syscall_4294966784(0xa, 0x7ffffd34, 0x1, 0, 0x1008a3c7, 0x1008b5a3, 0x1008b5a4, 

That's -512, a.k.a. the errno value used by syscall restarting.  I'd
say your glibc does not obey the restartable syscall convention used
by your kernel, and when it tries to restart the syscall the errno
value is not being replaced by the syscall number.  Check the assembly
for recv.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Gdbserver syscall clobber

[Bill Gatliff]

[-- Attachment #1: Type: text/plain, Size: 2843 bytes --]

Daniel Jacobowitz wrote:
> On Mon, Jul 16, 2007 at 10:43:41AM -0500, Bill Gatliff wrote:
>   
>> recv(4, 0x7ffffd60, 1, 0)               = ? ERESTARTSYS (To be restarted)
>> --- SIGIO (I/O possible) @ 0 (0) ---
>> syscall_4294966784(0xa, 0x7ffffd34, 0x1, 0, 0x1008a3c7, 0x1008b5a3, 0x1008b5a4, 
>>     
>
> That's -512, a.k.a. the errno value used by syscall restarting.  I'd
> say your glibc does not obey the restartable syscall convention used
> by your kernel, and when it tries to restart the syscall the errno
> value is not being replaced by the syscall number.  Check the assembly
> for recv.
>
>   

Very good catch!  Thanks soooo much.  Here's the code, from my libc.a:

00000000 <__libc_recv>:
   0:   94 21 ff d0     stwu    r1,-48(r1)
   4:   90 61 00 14     stw     r3,20(r1)
   8:   90 81 00 18     stw     r4,24(r1)
   c:   90 a1 00 1c     stw     r5,28(r1)
  10:   90 c1 00 20     stw     r6,32(r1)
  14:   81 42 00 0c     lwz     r10,12(r2)
  18:   2c 0a 00 00     cmpwi   r10,0
  1c:   40 82 00 20     bne-    3c <__libc_recv+0x3c>
  20:   38 60 00 0a     li      r3,10
  24:   38 81 00 14     addi    r4,r1,20
  28:   38 00 00 66     li      r0,102
  2c:   44 00 00 02     sc
  30:   38 21 00 30     addi    r1,r1,48
  34:   4c a3 00 20     bnslr+
  38:   48 00 00 00     b       38 <__libc_recv+0x38>

Again, this is 603e on linux-2.4.16 glibc-2.2.5 gcc-2.95.3.  (Odd, I 
can't seem to find this function in a statically-linked gdbserver, nor 
any reference to it in the gdbserver-6.5 source code).

On the kernel side:

_GLOBAL(DoSyscall)
...
        blrl                    /* Call handler */
        .globl  ret_from_syscall_1
ret_from_syscall_1:
20:     stw     r3,RESULT(r1)   /* Save result */
        li      r10,-_LAST_ERRNO
        cmpl    0,r3,r10
        blt     30f
        neg     r3,r3
        cmpi    0,r3,ERESTARTNOHAND
        bne     22f
        li      r3,EINTR
22:     lwz     r10,_CCR(r1)    /* Set SO bit in CR */
        oris    r10,r10,0x1000
        stw     r10,_CCR(r1)
30:     stw     r3,GPR3(r1)     /* Update return value */
        b       ret_from_except
...
ret_from_except:
...
        lwz     r3,_CCR(r1)
...
        mtcrf   0xFF,r3
...
        RFI


Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM 
lately), but it looks to me like the kernel is setting bit 0 in CR0 
(oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0 
(bnslr+) bit 3 a.k.a. SO.  Or maybe the other way around, I'm not sure 
after reading Sections 1.2 and 2.1 of the Programming Environments manual.

Or am I misinterpreting something?  I must be, this is well-trodden code 
I'm thinking...

The readchar() in gdbserver's remote-utils.c just calls read() on the 
file descriptor for the socket.  Still trying to track that code down...



b.g.

-- 
Bill Gatliff
bgat@billgatliff.com


[-- Attachment #2: Type: text/html, Size: 5471 bytes --]

Re: Gdbserver syscall clobber

[Bill Gatliff]

Daniel Jacobowitz wrote:
> On Mon, Jul 16, 2007 at 10:43:41AM -0500, Bill Gatliff wrote:
>   
>> recv(4, 0x7ffffd60, 1, 0)               = ? ERESTARTSYS (To be restarted)
>> --- SIGIO (I/O possible) @ 0 (0) ---
>> syscall_4294966784(0xa, 0x7ffffd34, 0x1, 0, 0x1008a3c7, 0x1008b5a3, 0x1008b5a4, 
>>     
>
> That's -512, a.k.a. the errno value used by syscall restarting.  I'd
> say your glibc does not obey the restartable syscall convention used
> by your kernel, and when it tries to restart the syscall the errno
> value is not being replaced by the syscall number.  Check the assembly
> for recv.
>
>   

Very good catch!  Thanks soooo much.  Here's the code, from my libc.a:

00000000 <__libc_recv>:
    0:   94 21 ff d0     stwu    r1,-48(r1)
    4:   90 61 00 14     stw     r3,20(r1)
    8:   90 81 00 18     stw     r4,24(r1)
    c:   90 a1 00 1c     stw     r5,28(r1)
   10:   90 c1 00 20     stw     r6,32(r1)
   14:   81 42 00 0c     lwz     r10,12(r2)
   18:   2c 0a 00 00     cmpwi   r10,0
   1c:   40 82 00 20     bne-    3c <__libc_recv+0x3c>
   20:   38 60 00 0a     li      r3,10
   24:   38 81 00 14     addi    r4,r1,20
   28:   38 00 00 66     li      r0,102
   2c:   44 00 00 02     sc
   30:   38 21 00 30     addi    r1,r1,48
   34:   4c a3 00 20     bnslr+
   38:   48 00 00 00     b       38 <__libc_recv+0x38>

Again, this is 603e on linux-2.4.16 glibc-2.2.5 gcc-2.95.3.  (Odd, I
can't seem to find this function in a statically-linked gdbserver, nor
any reference to it in the gdbserver-6.5 source code).

On the kernel side:

_GLOBAL(DoSyscall)
...
         blrl                    /* Call handler */
         .globl  ret_from_syscall_1
ret_from_syscall_1:
20:     stw     r3,RESULT(r1)   /* Save result */
         li      r10,-_LAST_ERRNO
         cmpl    0,r3,r10
         blt     30f
         neg     r3,r3
         cmpi    0,r3,ERESTARTNOHAND
         bne     22f
         li      r3,EINTR
22:     lwz     r10,_CCR(r1)    /* Set SO bit in CR */
         oris    r10,r10,0x1000
         stw     r10,_CCR(r1)
30:     stw     r3,GPR3(r1)     /* Update return value */
         b       ret_from_except
...
ret_from_except:
...
         lwz     r3,_CCR(r1)
...
         mtcrf   0xFF,r3
...
         RFI


Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM
lately), but it looks to me like the kernel is setting bit 0 in CR0
(oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0
(bnslr+) bit 3 a.k.a. SO.  Or maybe the other way around, I'm not sure
after reading Sections 1.2 and 2.1 of the Programming Environments manual.

Or am I misinterpreting something?  I must be, this is well-trodden code
I'm thinking...

The readchar() in gdbserver's remote-utils.c just calls read() on the
file descriptor for the socket.  Still trying to track that code down...



b.g.

-- 
Bill Gatliff
bgat@billgatliff.com

Re: Gdbserver syscall clobber

[Daniel Jacobowitz]

On Wed, Jul 18, 2007 at 12:59:42PM -0500, Bill Gatliff wrote:
> Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM
> lately), but it looks to me like the kernel is setting bit 0 in CR0
> (oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0
> (bnslr+) bit 3 a.k.a. SO.  Or maybe the other way around, I'm not sure
> after reading Sections 1.2 and 2.1 of the Programming Environments manual.

It's not checking for restart here - userspace isn't supposed to have to.
It's probably checking for error.  Check for the bit of kernel code
that's supposed to back you up two instructions.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Gdbserver syscall clobber

[Bill Gatliff]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Daniel Jacobowitz wrote:
> On Wed, Jul 18, 2007 at 12:59:42PM -0500, Bill Gatliff wrote:
>   
>> Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM
>> lately), but it looks to me like the kernel is setting bit 0 in CR0
>> (oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0
>> (bnslr+) bit 3 a.k.a. SO.  Or maybe the other way around, I'm not sure
>> after reading Sections 1.2 and 2.1 of the Programming Environments manual.
>>     
>
> It's not checking for restart here - userspace isn't supposed to have to.
> It's probably checking for error.  Check for the bit of kernel code
> that's supposed to back you up two instructions.
>
>   

I don't see it in this kernel.  What I see is this after the call to the 
syscall handler:

    li    r10,-_LAST_ERRNO
    cmpl    0,r3,r10
    blt    30f
    neg    r3,r3
    cmpi    0,r3,ERESTARTNOHAND
    bne    22f
    li    r3,EINTR
22:    lwz    r10,_CCR(r1)    /* Set SO bit in CR */
    oris    r10,r10,0x1000
    stw    r10,_CCR(r1)
30:    stw    r3,GPR3(r1)    /* Update return value */
    b    ret_from_except
66:    li    r3,ENOSYS
    b    22b


?

-- 
Bill Gatliff
bgat@billgatliff.com


[-- Attachment #2: Type: text/html, Size: 2229 bytes --]

Re: Gdbserver syscall clobber

[Bill Gatliff]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Daniel Jacobowitz wrote:
> On Wed, Jul 18, 2007 at 12:59:42PM -0500, Bill Gatliff wrote:
>   
>> Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM
>> lately), but it looks to me like the kernel is setting bit 0 in CR0
>> (oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0
>> (bnslr+) bit 3 a.k.a. SO.  Or maybe the other way around, I'm not sure
>> after reading Sections 1.2 and 2.1 of the Programming Environments manual.
>>     
>
> It's not checking for restart here - userspace isn't supposed to have to.
> It's probably checking for error.  Check for the bit of kernel code
> that's supposed to back you up two instructions.
>
>   

I don't see it in this kernel.  What I see is this after the call to the 
syscall handler:

    li    r10,-_LAST_ERRNO
    cmpl    0,r3,r10
    blt    30f
    neg    r3,r3
    cmpi    0,r3,ERESTARTNOHAND
    bne    22f
    li    r3,EINTR
22:    lwz    r10,_CCR(r1)    /* Set SO bit in CR */
    oris    r10,r10,0x1000
    stw    r10,_CCR(r1)
30:    stw    r3,GPR3(r1)    /* Update return value */
    b    ret_from_except
66:    li    r3,ENOSYS
    b    22b


?

-- 
Bill Gatliff
bgat@billgatliff.com


[-- Attachment #2: Type: text/html, Size: 2229 bytes --]

Re: Gdbserver syscall clobber

[Bill Gatliff]

Daniel Jacobowitz wrote:
> On Wed, Jul 18, 2007 at 12:59:42PM -0500, Bill Gatliff wrote:
>   
>> Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM
>> lately), but it looks to me like the kernel is setting bit 0 in CR0
>> (oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0
>> (bnslr+) bit 3 a.k.a. SO.  Or maybe the other way around, I'm not sure
>> after reading Sections 1.2 and 2.1 of the Programming Environments manual.
>>     
>
> It's not checking for restart here - userspace isn't supposed to have to.
> It's probably checking for error.  Check for the bit of kernel code
> that's supposed to back you up two instructions.
>
>   

I don't see it in this kernel.  What I see is this after the call to the 
syscall handler:

    li    r10,-_LAST_ERRNO
    cmpl    0,r3,r10
    blt    30f
    neg    r3,r3
    cmpi    0,r3,ERESTARTNOHAND
    bne    22f
    li    r3,EINTR
22:    lwz    r10,_CCR(r1)    /* Set SO bit in CR */
    oris    r10,r10,0x1000
    stw    r10,_CCR(r1)
30:    stw    r3,GPR3(r1)    /* Update return value */
    b    ret_from_except
66:    li    r3,ENOSYS
    b    22b


?

-- 
Bill Gatliff
bgat@billgatliff.com

Re: Gdbserver syscall clobber

[Daniel Jacobowitz]

On Mon, Jul 23, 2007 at 11:06:13AM -0500, Bill Gatliff wrote:
> Daniel Jacobowitz wrote:
> > On Wed, Jul 18, 2007 at 12:59:42PM -0500, Bill Gatliff wrote:
> >   
> >> Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM
> >> lately), but it looks to me like the kernel is setting bit 0 in CR0
> >> (oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0
> >> (bnslr+) bit 3 a.k.a. SO.  Or maybe the other way around, I'm not sure
> >> after reading Sections 1.2 and 2.1 of the Programming Environments manual.
> >>     
> >
> > It's not checking for restart here - userspace isn't supposed to have to.
> > It's probably checking for error.  Check for the bit of kernel code
> > that's supposed to back you up two instructions.
> >
> >   
> 
> I don't see it in this kernel.  What I see is this after the call to the 
> syscall handler:

Look around do_signal:

                        regs->nip -= 4; /* Back up & retry system call */

If your kernel has corrupted the register containing the syscall
number at this point, that would explain your problem.  It will then
do the wrong syscall.  I guess PPC only backs up one instruction.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Gdbserver syscall clobber

[Andreas Schwab]

Bill Gatliff <bgat@billgatliff.com> writes:

> Daniel Jacobowitz wrote:
>> On Wed, Jul 18, 2007 at 12:59:42PM -0500, Bill Gatliff wrote:
>>   
>>> Now, I'm a little rusty on PPC asm (I've been doing a lot of ARM
>>> lately), but it looks to me like the kernel is setting bit 0 in CR0
>>> (oris r10, r10, 0x1000) a.k.a LT, but the user side is looking at CR0
>>> (bnslr+) bit 3 a.k.a. SO.

Bits are numbered from left to right, thus 0x10000000 is bit 3 of CR0

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Gdbserver syscall clobber

[Bill Gatliff]

Daniel Jacobowitz wrote:
> Look around do_signal:
>
>                         regs->nip -= 4; /* Back up & retry system call */
>
> If your kernel has corrupted the register containing the syscall
> number at this point, that would explain your problem.  It will then
> do the wrong syscall.  I guess PPC only backs up one instruction.
>
>   

Found the code in do_signal, right where you said it would be.  I threw 
in a printk or two to see what's up when an ERESTARTSYS is detected, and 
started getting OOPSes all over the place.  Apparently that's not a good 
place for printk.  :)

Before I go on, can anyone confirm that gdbserver and/or strace _ever_ 
worked with circa 2.4.16 PPC kernels?



b.g.

-- 
Bill Gatliff
bgat@billgatliff.com