[parisc-linux] r19 (aka pic-register akak ltp) not restored on entry back to libc from libpthread?

[parisc-linux] r19 (aka pic-register akak ltp) not restored on entry back to libc from libpthread?

[Carlos O'Donell]

jda,

Perhaps you can help here with my problem, again it's an r19 related
issue.

make[2]: *** [/glibc-cvs/build-hppa/posix/tst-regex.out] Error 139

Breakpoint 1, fixup (l=0xfaf00d68, reloc_offset=1248) at dl-runtime.c:72
72        alloca (sizeof (int));
(gdb) c 22

I exit the loader, and I have ~2700 insn before I get to here...

(gdb) si 2713

I've traced it down to:

0000000040024000-000000004016d000 r-xp 0000000000000000 08:04 1308771
/mnt/flaire/src/glibc-cvs/build-hppa/libc.so

Looks like "public_mALLOc" from libc/malloc/malloc.c trying to unlock
the memory arena mutex.

00081ea0 <__libc_malloc>:
0x400a5f50:     copy r4,r19
0x400a5f54:     cmpib,= 0,ret0,0x400a5f8c
...

r19 = 40181d50 (All is good, all is quiet for 2700 insn)

(gdb) x/4 0x40181d50-0x1800+0x400
0x40180950: 0x400a6234 0x40181d50 0x40094320 0x40181d50

(Stub)
0x400abf0c:     b,l 0x400abf14,r1
0x400abf10:     addil 9f000,r1,%r1
0x400abf14:     be,n 71c(sr4,r1)
(Load r19 and target address from PLABEL)
0x4014b630:     bb,>=,n r22,1e,0x4014b640
0x4014b634:     depwi 0,31,2,r22
0x4014b638:     ldw 4(sr0,r22),r19
0x4014b63c:     ldw 0(sr0,r22),r22
0x4014b640:     bv r0(r22)
0x4014b644:     stw rp,-18(sr0,sp)

r19 = 401a57a8 (Good for libpthread ...)

(gdb) x /4 0x401a57a8-0x1800+0x400
0x401a43a8: 0x00000008 0x0000b6ac 0x0000b6e8 0x08000000
            ^^^^^^^^^^ Soon to be fatal return address.

000072e0 <__pthread_mutex_unlock>:
0x4018d2e0:     stw rp,-14(sr0,sp)
0x4018d2e4:     stw,ma r4,40(sr0,sp)
0x4018d2e8:     stw r19,-20(sr0,sp)		# 2719
...

0000000040186000-0000000040195000 r-xp 0000000000000000 08:04 655453
/mnt/flaire/src/glibc-cvs/build-hppa/linuxthreads/libpthread.so

0x4019063c:     stw r19,-20(sr0,sp)		# 2742
...
0x40190860:     stw r19,-20(sr0,sp)		
...
0x401908f8:     bv r0(rp)			# 2774
0x401908fc:     ldo -80(sp),sp
...
0x40190770:     bv r0(rp)
0x40190774:     ldo -80(sp),sp
...
000072e0 <__pthread_mutex_unlock>
0x4018d334:     bv r0(rp)			# 2806
0x4018d338:     ldw,mb -40(sr0,sp),r4
...
00081ea0 <__libc_malloc>
0x400a5f84:     b,l 0x400a5edc,r0
0x400a5f88:     copy r5,ret0
...

Hold your horses here, we made it back into libc but our ltp is still
that which we loaded upon entry to libpthread? :(

0x400a5eec:     bv r0(rp)
0x400a5ef0:     ldw,mb -40(sr0,sp),r6
...
0x401028e4:     cmpib,<> 0,r20,0x40102918
0x401028e8:     copy r3,r25
...
0x40102918:     b,l 0x40116658,rp		# 2839
0x4010291c:     copy r6,r26

First use of libpthread's r19 is fatal.

0x40116658:     addil -1800,r19,%r1		# 2840
0x4011665c:     ldw 400(sr0,r1),r21  <--- *BOOM* r21=0x8
0x40116660:     bv r0(r21)
0x40116664:     ldw 404(sr0,r1),r19 

Any thoughts? Did I miss something?

Cheers,
Carlos.

Re: [parisc-linux] r19 (aka pic-register akak ltp) not restored on entry back to libc from libpthread?

[John David Anglin]

> Hold your horses here, we made it back into libc but our ltp is still
> that which we loaded upon entry to libpthread? :(

That's ok.  It's the responsibility of the libc code to restore ltp
after a call or exception.  However, as discussed previously, there
is no restoration after a syscall.  That should be the system's job,
although I believe you were going to introduce a hack/workaround to
fix the syscalls that clobber r19.

You need to step through the libc code from the return point in
libc to see why ltp isn't being restored.  Possibly, libpthread is
being called by assembly code that doesn't restore ltp.  Normally,
r19 is restored quite soon after a call.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Re: [parisc-linux] r19 (aka pic-register akak ltp) not restored on entry back to libc from libpthread?

[Carlos O'Donell]

On Fri, Sep 19, 2003 at 12:56:46PM -0400, John David Anglin wrote:
> > Hold your horses here, we made it back into libc but our ltp is still
> > that which we loaded upon entry to libpthread? :(
> 
> That's ok.  It's the responsibility of the libc code to restore ltp
> after a call or exception.  However, as discussed previously, there
> is no restoration after a syscall.  That should be the system's job,
> although I believe you were going to introduce a hack/workaround to
> fix the syscalls that clobber r19.

For all syscalls whose wrappers are pure assembly or __asm(...) I have
placed restorations to follow ABI. This fixed all of the major failures.
It's a bit drastic, but it fixes the issue until I know exactly which
syscall numbers clobber r19. It looks like just fork and it's variants
that stick a return address into PT_REGS's r19. That's a side issue
about optimization though. 

> You need to step through the libc code from the return point in
> libc to see why ltp isn't being restored.  Possibly, libpthread is
> being called by assembly code that doesn't restore ltp.  Normally,
> r19 is restored quite soon after a call.

There should be no assembly code glue between the calls.
What about the gcc optimization where we don't restore r19 if it's not
used between the last call and the return?

mutex_unlock becomes:
__libc_maybe_call (__pthread_mutex_unlock, (m), (*(int *)(m) = 0))

Which is a big monstrosity that looks like it might be casting things
incorrectly for hppa...

===
#if defined _LIBC && defined IS_IN_libpthread
# define __libc_maybe_call(FUNC, ARGS, ELSE) FUNC ARGS
#else
# if defined __PIC__ || (defined _LIBC && defined SHARED)
#  define __libc_maybe_call(FUNC, ARGS, ELSE) \
  (__extension__ ({ __typeof (FUNC) *_fn = (FUNC); \
                    _fn != NULL ? (*_fn) ARGS : ELSE; }))
# else
#  define __libc_maybe_call(FUNC, ARGS, ELSE) \
  (FUNC != NULL ? FUNC ARGS : ELSE)
# endif
#endif
#if defined _LIBC && !defined NOT_IN_libc && defined SHARED
# define __libc_maybe_call2(FUNC, ARGS, ELSE) \
  ({__builtin_expect (__libc_pthread_functions.ptr_##FUNC != NULL, 0) \
    ? __libc_pthread_functions.ptr_##FUNC ARGS : ELSE; })
#else
# define __libc_maybe_call2(FUNC, ARGS, ELSE) __libc_maybe_call
# (__##FUNC, ARGS, ELSE)
#endif
===

I have class, then I'll be back to -E some of the malloc build to see
what this evaluates to and then try to determine if gcc did the right
thing.

c.

Re: [parisc-linux] r19 (aka pic-register akak ltp) not restored on entry back to libc from libpthread?

[James Bottomley]

On Fri, 2003-09-19 at 12:51, Carlos O'Donell wrote:
> mutex_unlock becomes:
> __libc_maybe_call (__pthread_mutex_unlock, (m), (*(int *)(m) = 0))

This seems all to work:

A test file

main() {
  int t1 = 4;
  __libc_maybe_call(test1, (t1), (t1=3));
}

compiles (PIC) to:

00000000 <main>:
[...]
 20:   2a 60 00 00     addil 0,r19,%r1
                        20: R_PARISC_DLTIND21L  .LC0
  24:   48 21 00 00     ldw 0(r1),r1
                        24: R_PARISC_DLTIND14R  .LC0
  28:   0c 20 10 94     ldw 0(,r1),r20
  2c:   0c 74 12 98     stw r20,c(,r3)
  30:   0c 78 10 94     ldw c(,r3),r20
  34:   86 80 20 3a     cmpib,=,n 0,r20,58 <main+0x58>
  38:   0c 78 10 94     ldw c(,r3),r20
  3c:   0c 70 10 9a     ldw 8(,r3),r26
  40:   08 14 02 56     copy r20,r22
  44:   08 13 02 44     copy r19,r4
  48:   eb e0 00 00     b,l 50 <main+0x50>,r31
                        48: R_PARISC_PCREL17F   $$dyncall
  4c:   08 1f 02 42     copy r31,rp
  50:   08 04 02 53     copy r4,r19
[...]
00000000 <.LC0>:
   0:   00 00 00 00     break 0,0
                        0: R_PARISC_PLABEL32    test1
[...]
00010578 <$$dyncall>:
   10578:	c7 d6 c0 12 	bb,>=,n r22,1e,10588 <$$dyncall+0x10>
   1057c:	d6 c0 1c 1e 	depwi 0,31,2,r22
   10580:	0e c8 10 93 	ldw 4(,r22),r19
   10584:	0e c0 10 96 	ldw 0(,r22),r22
   10588:	ea c0 c0 00 	bv r0(r22)
   1058c:	6b c2 3f d1 	stw rp,-18(sp)


The $$dyncall is where we indirect through r20 (which contains the
function pointer).  Note the copy restoring r19 around this.

James

Re: [parisc-linux] r19 (aka pic-register akak ltp) not restored on

[John David Anglin]

> On Fri, 2003-09-19 at 12:51, Carlos O'Donell wrote:
> > mutex_unlock becomes:
> > __libc_maybe_call (__pthread_mutex_unlock, (m), (*(int *)(m) = 0))
> 
> This seems all to work:

Yah, the normal code that gcc generates to restore the ltp is very
extensively tested.  Shared libraries would break almost instantly
if there were major problems.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Re: [parisc-linux] r19 (aka pic-register akak ltp) not restored on

[Carlos O'Donell]

On Fri, Sep 19, 2003 at 02:55:30PM -0400, John David Anglin wrote:
> > On Fri, 2003-09-19 at 12:51, Carlos O'Donell wrote:
> > > mutex_unlock becomes:
> > > __libc_maybe_call (__pthread_mutex_unlock, (m), (*(int *)(m) = 0))
> > 
> > This seems all to work:
> 
> Yah, the normal code that gcc generates to restore the ltp is very
> extensively tested.  Shared libraries would break almost instantly
> if there were major problems.
> 

If it were major we would have fixed it :)
I'm trying to find the "missed a restore" point.

c.

Re: [parisc-linux] r19 (aka pic-register akak ltp) not restored on

[John David Anglin]

> If it were major we would have fixed it :)
> I'm trying to find the "missed a restore" point.

0x4019063c:     stw r19,-20(sr0,sp)             # 2742

Just a note, GCC saves r19 in the frame marker in the prologue of
non-leaf functions but we never attempt to restore r19 from the frame
marker.  This is the mandated ABI behavior.  At the moment, GCC copies
r19 to r4 for the save.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

[parisc-linux] Found the r19 problem!

[Carlos O'Donell]

On Fri, Sep 19, 2003 at 04:10:47PM -0400, John David Anglin wrote:
> > If it were major we would have fixed it :)
> > I'm trying to find the "missed a restore" point.
> 
> 0x4019063c:     stw r19,-20(sr0,sp)             # 2742
> 
> Just a note, GCC saves r19 in the frame marker in the prologue of
> non-leaf functions but we never attempt to restore r19 from the frame
> marker.  This is the mandated ABI behavior.  At the moment, GCC copies
> r19 to r4 for the save.

I would like to start this off with:
"JDA said don't put r19 into clobber list"

I didn't listen. I thought it should all still work.
James reports that removing r19 from the clobber list works, but I still
haven't rebuilt my tree, so I'll see.

Example (Assembly trace provided at the end):

getcwd: (Start of function)
	stw r19,-20(sr0,sp)
	...
syscall:
	(Syscall with save/load r19 wrapper)
	stw r19,-20(sr0,sp)
	be,l 100(sr2,r0),%sr0,%r31
	ldi 6e,r20
	ldw -20(sr0,sp),r19
	...
	(Many insn later)
	(stub) (dyncall) -> libpthread.so
	-> libc.so
	(r19 not restored)
	(Jump to syscall:)
	...

o GCC is confused by the r19 asm(...) clobber?

Notes:

I generated insn traces using gdb scripts.

  __pthread_mutex_unlock: 0x7730 <0x4018c730>	(In libpthread)
 __libc_malloc: 0x7f4a0 <0x400a34a0>		(In libc)

(return stub)
0x400a34a0:     b,l 0x400a33f8,r0
0x400a34a4:     copy r5,ret0

(__libc_malloc returning)
0x400a33f8:     ldw -54(sr0,sp),rp
0x400a33fc:     ldw -3c(sr0,sp),r5		(No need to restore r19)
0x400a3400:     ldw -38(sr0,sp),r4
0x400a3404:     ldw -34(sr0,sp),r3
0x400a3408:     bv r0(rp)			(Back to getcwd)
0x400a340c:     ldw,mb -40(sr0,sp),r6

getcwd: 0xdb128 <0x400ff128>
0x400ff128:     ldi 0,r21
0x400ff12c:     cmpib,<> 0,ret0,0x400ff034	(Jump back to do syscall)
0x400ff130:     copy ret0,r6

0x400ff034:     copy r3,r25
0x400ff038:     copy r6,r26
0x400ff03c:     stw r19,-20(sr0,sp)
0x400ff040:     be,l 100(sr2,r0),%sr0,%r31
0x400ff044:     ldi 6e,r20
0x400ff048:     ldw -20(sr0,sp),r19 (si gdb artifact, lost insn inside syscall)
0x400ff048:     ldw -20(sr0,sp),r19 
0x400ff04c:     ldi -1000,r20
0x400ff050:     cmpb,>>= r20,ret0,0x400ff070
0x400ff054:     copy ret0,r3

0x400ff070:     cmpib,>,n 0,r3,0x400ff0cc
0x400ff074:     cmpiclr,<> 0,r7,r21
0x400ff078:     ldi 1,r21
0x400ff07c:     cmpiclr,<> 0,r5,r20
0x400ff080:     ldi 1,r20
0x400ff084:     and r20,r21,r20
0x400ff088:     cmpib,<> 0,r20,0x400ff0bc
0x400ff08c:     copy r3,r25			(No r19 restore yet!!!)

(call stub)
0x400ff0bc:     b,l 0x40114e2c,rp
0x400ff0c0:     copy r6,r26

(stub)
0x40114e2c:     addil -1800,r19,%r1
0x40114e30:     ldw 428(sr0,r1),r21
0x40114e34:     bv r0(r21)
0x40114e38:     ldw 42c(sr0,r1),r19


getcwd: 0xdb0c0 <0x400ff0c0>
(stub) *BOOM*

Is there any way we can make this work?

c.

[parisc-linux] Re: Found the r19 problem!

[John David Anglin]

> I would like to start this off with:
> "JDA said don't put r19 into clobber list"
> 
> I didn't listen. I thought it should all still work.
> James reports that removing r19 from the clobber list works, but I still
> haven't rebuilt my tree, so I'll see.
> 
> Example (Assembly trace provided at the end):
> 
> getcwd: (Start of function)
> 	stw r19,-20(sr0,sp)
> 	...
> syscall:
> 	(Syscall with save/load r19 wrapper)
> 	stw r19,-20(sr0,sp)
> 	be,l 100(sr2,r0),%sr0,%r31
> 	ldi 6e,r20
> 	ldw -20(sr0,sp),r19

It would be better to use a general register for the save/restore.
If you are going to use the slot in the frame marker, it probably
isn't necessary to save r19 before every syscall.  GCC saves r19
in the slot in the prologue of all pic functions.  However, we don't
currently copy the value when when the function does a dynamic stack
allocation.  That's easily fixed.

> 	...
> 	(Many insn later)
> 	(stub) (dyncall) -> libpthread.so
> 	-> libc.so

The above is an indirect call. r19 should be restored after the call
if it is used after the call.

> 	(r19 not restored)
> 	(Jump to syscall:)

It is likely that clobbering r19 in the syscall causes the restore
of r19 to be deleted.  Because of the clobber, GCC believes that
r19 is dead and the restore insn isn't needed.  I think you should
be able to see this by looking at the rtl for the routine.  The save
and restore of r19 in a call are split out after the GCC reload pass.

I'm still somewhat confused.  Isn't the syscall going to clobber
r19?  Don't you need to save and restore r19 here, rather than in the
wrapper?  There seems to be a call (i.e., you die in a linker call
stub) before you get back to the wrapper to restore r19.

> o GCC is confused by the r19 asm(...) clobber?

The management of r19 is very tricky.  It can't be exposed before
reload as all uses of r19 are not known until that time.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

[parisc-linux] Re: Found the r19 problem!

[Carlos O'Donell]

> > Example (Assembly trace provided at the end):
> > 
> > getcwd: (Start of function)
> > 	stw r19,-20(sr0,sp)
> > 	...
> > syscall:
> > 	(Syscall with save/load r19 wrapper)
> > 	stw r19,-20(sr0,sp)
> > 	be,l 100(sr2,r0),%sr0,%r31
> > 	ldi 6e,r20
> > 	ldw -20(sr0,sp),r19
> 
> It would be better to use a general register for the save/restore.
> If you are going to use the slot in the frame marker, it probably
> isn't necessary to save r19 before every syscall.  GCC saves r19
> in the slot in the prologue of all pic functions.  However, we don't
> currently copy the value when when the function does a dynamic stack
> allocation.  That's easily fixed.

So use a caller saves register, place it in the clobbers, and let gcc
work around the usage (e.g. r4).
 
> > 	...
> > 	(Many insn later)
> > 	(stub) (dyncall) -> libpthread.so
> > 	-> libc.so
> 
> The above is an indirect call. r19 should be restored after the call
> if it is used after the call.

It is, but only after the asm(...) that lists r19 in the clobber.

> > 	(r19 not restored)
> > 	(Jump to syscall:)
> 
> It is likely that clobbering r19 in the syscall causes the restore
> of r19 to be deleted.  Because of the clobber, GCC believes that
> r19 is dead and the restore insn isn't needed.  I think you should
> be able to see this by looking at the rtl for the routine.  The save
> and restore of r19 in a call are split out after the GCC reload pass.
> 
> I'm still somewhat confused.  Isn't the syscall going to clobber
> r19?  Don't you need to save and restore r19 here, rather than in the
> wrapper?  There seems to be a call (i.e., you die in a linker call
> stub) before you get back to the wrapper to restore r19.

The presence of r19 in an asm(...) clobber seems to confuse GCC into 
deleting an r19 restore after a return from an interlibrary call. We die
in the linker call stub because it uses libpthread's r19, because GCC
deleted the scheduled restore. Why it deleted the restore is unknown to
me. The syscall wrapper makes sure the syscall doesn't trash r19, it
doesn't assure that the incoming r19 isn't already wrong :)

> > o GCC is confused by the r19 asm(...) clobber?
> 
> The management of r19 is very tricky.  It can't be exposed before
> reload as all uses of r19 are not known until that time.

Agreed, tricky to track down too.

c.

Re: [parisc-linux] Re: Found the r19 problem!

[Carlos O'Donell]

> So use a caller saves register, place it in the clobbers, and let gcc
> work around the usage (e.g. r4).
>  
> > > 	...
> > > 	(Many insn later)
> > > 	(stub) (dyncall) -> libpthread.so
> > > 	-> libc.so
> > 
> > The above is an indirect call. r19 should be restored after the call
> > if it is used after the call.
> 
> It is, but only after the asm(...) that lists r19 in the clobber.
> 

Sorry "It is" should read "It is used" :)
Things get tricky in email with complex issues.

c.

Re: [parisc-linux] Re: Found the r19 problem!

[John David Anglin]

> > So use a caller saves register, place it in the clobbers, and let gcc
> > work around the usage (e.g. r4).
> >  
> > > > 	...
> > > > 	(Many insn later)
> > > > 	(stub) (dyncall) -> libpthread.so
> > > > 	-> libc.so
> > > 
> > > The above is an indirect call. r19 should be restored after the call
> > > if it is used after the call.
> > 
> > It is, but only after the asm(...) that lists r19 in the clobber.
> > 
> 
> Sorry "It is" should read "It is used" :)

Well that's why the restore is deleted.  So, the clobber must go
and r19 must be preserved across the asm(...).  As I said, you can't
clobber r19 when generating pic code.  It's treated as fixed register,
so clobbering it can lead to undefined behavior.

If the syscall used r19, it could include a use of r19 in the asm.
This would ensure the restore occurs before the syscall.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

Re: [parisc-linux] Re: Found the r19 problem!

[Carlos O'Donell]

On Sun, Sep 21, 2003 at 03:12:39PM -0400, John David Anglin wrote:
> Well that's why the restore is deleted.  So, the clobber must go
> and r19 must be preserved across the asm(...).  As I said, you can't
> clobber r19 when generating pic code.  It's treated as fixed register,
> so clobbering it can lead to undefined behavior.
> 
> If the syscall used r19, it could include a use of r19 in the asm.
> This would ensure the restore occurs before the syscall.

I'm going to stop writing r19 to the stack and tell gcc I'm using r4,
and then do a copy/copy to get r19 back.

c.

Re: [parisc-linux] Re: Found the r19 problem!

[John David Anglin]

> So use a caller saves register, place it in the clobbers, and let gcc
> work around the usage (e.g. r4).

Yes.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)