Re: [parisc-linux] userspace function pointers in the kernel

Re: [parisc-linux] userspace function pointers in the kernel

[Cary Coutant]

>It looks like the IA-64 runtime achieves this by generating official
>procedure descriptors for every defined function in an object and
>placing them in the .opd section, then using these descriptors
>whenever the address of a function is taken.  Presumably external
>functions are compared using their .IA_64.pltoff entries?  I'm not
>clear on how that works and don't have an IA-64 handy to experiment.

I can't say for sure how Linux/IA-64 uses the .opd section, but the 
runtime and the psABI recommend that the "official" function descriptors 
be created by the dynamic loader as needed in response to dynamic FPTR 
relocations. A function pointer is always loaded from the GOT or from a 
statically-initialized variable, either of which would have been tagged 
with a dynamic FPTR relocation. The dynamic loader will allocate one and 
only one OFD for each function whose address is taken, and resolve each 
function pointer to the address of that OFD. Function pointers can then 
be compared as simple 32- or 64-bit quantities, without worrying about 
the contents of the function descriptor that they point to.

One way of avoiding the dynamic allocation by the dynamic loader is for 
the linker to allocate OFDs statically for each exported function, and 
for each hidden or static function whose address is taken. It looks like 
this may be what Linux is doing on IA-64. The disadvantage is that you 
end up with far more OFDs than you'll ever need, since few functions ever 
really have their addresses taken.

The .pltoff section is for local copies of OFDs that are used by inlined 
import stubs. When the compiler decides that a function is unlikely to 
resolve within the load module, it generates an inlined external call 
sequence, rather than rely on the linker to resolve a direct call to an 
import stub. This inlined sequence needs a local copy of the function 
descriptor to avoid an extra indirection. Since these function 
descriptors never serve as an official function descriptor, it doesn't 
matter that they don't have the same address as an OFD.

Note that the same argument applies to C++ vtables -- we can put copies 
of the function descriptors there, too, since we don't need the pointers 
to vtable entries to be unique.

As an aside, the problem we had on HP-UX/PA was in statically allocating 
the official function descriptors -- the linker allocated OFD 
"candidates" whenever it saw the address of a function get taken. This 
produced multiple candidates for the OFD, and it was possible that none 
of those candidates were actually in the load module where the function 
was defined. Thus, the loader had to choose one arbitrarily, and 
sometimes could be tricked into choosing different ones at different 
times. Hence the problem.

-cary

Re: [parisc-linux] userspace function pointers in the kernel

[David Huggins-Daines]

Cary Coutant <cary@cup.hp.com> writes:

> I can't say for sure how Linux/IA-64 uses the .opd section, but the 
> runtime and the psABI recommend that the "official" function descriptors 
> be created by the dynamic loader as needed in response to dynamic FPTR 
> relocations.

Ah, okay, that makes sense.  This is what Linux/IA-64 does - dynamic
FPTR relocations result in OFDs being created dynamically (always, for
both local and global symbols).  And yes, the code in GNU libc does
ensure that they are mapped one-to-one with symbols in each process.

As mentioned earlier we have been trying to decide whether to do this
on PA-RISC as well.  Originally I threw it in so that (you guessed it)
comparisons of function pointers with NULL would work properly :-)

But as Alan says, it looks like we need it for other reasons as well.

> One way of avoiding the dynamic allocation by the dynamic loader is for 
> the linker to allocate OFDs statically for each exported function, and 
> for each hidden or static function whose address is taken.

Hm.  I'm not nearly as familiar with the linker code as I am with the
dynamic linker code, so I'm not sure what it does.  But, as noted
above, GNU libc does dynamic allocation, so it's probably not relevant.

-- 
dhd@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Re: [parisc-linux] userspace function pointers in the kernel

[Alan Modra]

On 12 Sep 2000, David Huggins-Daines wrote:

> Cary Coutant <cary@cup.hp.com> writes:
> 
> > One way of avoiding the dynamic allocation by the dynamic loader is for 
> > the linker to allocate OFDs statically for each exported function, and 
> > for each hidden or static function whose address is taken.
> 
> Hm.  I'm not nearly as familiar with the linker code as I am with the
> dynamic linker code, so I'm not sure what it does.  But, as noted
> above, GNU libc does dynamic allocation, so it's probably not relevant.

Well, I'm more familiar with the linker than the dynamic linker :)

The model I'm thinking of for elf32-hppa is along these lines:

o  The linker creates a plt entry for all plabel relocs.  plt entries for
   elf32-hppa are a function address, linkage table pointer pair, so
   there's no need for the dynamic linker to allocate fptrs.
o  A dynamic plabel reloc will have the function symbol, and an addend
   into the plt.  This is a rather unusual reloc because the function
   symbol value is ignored when calculating the final value.
o  The dynamic linker builds a list or hash table of function symbols
   versus plt offsets, and adjusts plabels so that only one plt entry is
   ever used per function.

Alan Modra
-- 
Linuxcare.  Support for the Revolution.

Function descriptors, etc. (was Re: [parisc-linux] userspace function pointers in the kernel)

[David Huggins-Daines]

[-- Attachment #1: Type: text/plain, Size: 4770 bytes --]

Alan Modra <alan@linuxcare.com.au> writes:

> The model I'm thinking of for elf32-hppa is along these lines:
> 
> o  The linker creates a plt entry for all plabel relocs.  plt entries for
>    elf32-hppa are a function address, linkage table pointer pair, so
>    there's no need for the dynamic linker to allocate fptrs.
> o  A dynamic plabel reloc will have the function symbol, and an addend
>    into the plt.  This is a rather unusual reloc because the function
>    symbol value is ignored when calculating the final value.
> o  The dynamic linker builds a list or hash table of function symbols
>    versus plt offsets, and adjusts plabels so that only one plt entry is
>    ever used per function.

Reasons why the IA-64 scheme is better:

        a) In the scheme above, we cannot index the list/table by the
        code address of the function in the PLT slot, because the PLT
        will usually not have been relocated at the point when PLABEL
        relocations are being processed.  Thus we have to:

                1) Relocate PLT slots as we encounter them in PLABEL
                   relocations, or
                2) Store a copy of the code address in the table apart
                   from the PLT slot, or
                3) Index the table by the address of the Elf32_Sym
                   structure, by the symbol name (I think this is
                   fraught with peril), or by something else.  This
                   also requires another field in the table apart from
                   the PLT slot.

        b) The IA-64 code uses a one-to-one mapping of function code
        addresses to function descriptors, not symbols to function
        descriptors, and indexes the list/table by the code address.
        I'm not sure whether this is important or not, but something
        tells me it is.

        c) We will still have to dynamically allocate function
        descriptors for symbols obtained with dlopen() and dlsym().
        First of all, we won't have a special PLABEL relocation, so we
        would have to search the PLT of the loaded object "manually".
        Also, think about the case where:

                - libfoo.so defines foo() strongly
                - libfoo2.so defines foo() weakly
                - libfoo2.so defines bar() which calls foo()
                - our program links against libfoo.so and libfoo2.so
                - our program calls foo()
                - our program calls bar()
                - our program dlopen()s libfoo2.so
                - our program calls dlsym(libfoo2, "foo")
                - oops!  the PLT slot for foo() in libfoo2.so points
                  to the strong definition in libfoo.so, so it's
                  useless as a function descriptor.
                - I've attached a tarball of this testcase.  Run it on
                  i386 and IA-64 and see what it does :-)

        d) If we have to dynamically allocate function descriptors as
        well as uniquify relocations to PLT slots, then we basically
        have to duplicate the code in dl-fptr.c, and we lose.
        
Can we please use the IA-64 scheme or something very close to it?
(i.e. closer than what you've proposed, which, in attempting to
implement it in ld.so, does not appear to be close enough) It is
already implemented and debugged, I had it successfully working last
week modulo problems with initializers and finalizers, and I don't see
there being a significant gain in performance or memory usage with the
scheme outlined above.

What I suggest is the following:

* The linker still creates a PLT entry for all PLABEL relocs.  This is
  undoubtedly useful for local symbols, and allows us to take
  advantage of lazy linking for them (potentially a big win in libc.so
  which uses lots and lots of vtables).

* We don't have to generate dynamic function descriptors for PLABEL
  relocations to local symbols because we know that their PLT entries
  are and will remain unique to their code addresses.  Again, a win.

* Dynamic PLABEL relocs will either be *ABS* or .plt relative and
  pointing to the PLT slot for local symbols, or contain the function
  symbol and no addend for global symbols.  For the latter we will
  generate unique function descriptors dynamically using the same code
  as on IA-64.  Yes, we lose 12 bytes of memory per global symbol this
  way.  I don't think that's a big deal given that it allows us to use
  the same code path for these symbols and for dlsym().

* dlsym() will always generate a unique function descriptor dynamically.

I've implemented this and checked it in (it kludges around the addends
on global PLABEL relocations, that is easy to fix :-).  It passes my
other tests but the one below causes the linker to segfault.


[-- Attachment #2: Interaction of dlsym() and weak dynamic symbols --]
[-- Type: application/x-gtar, Size: 1134 bytes --]

[-- Attachment #3: Type: text/plain, Size: 89 bytes --]


-- 
dhd@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Re: Function descriptors, etc. (was Re: [parisc-linux] userspace function pointers in the kernel)

[Alan Modra]

On 13 Sep 2000, David Huggins-Daines wrote:

> Reasons why the IA-64 scheme is better:
>[snip]

Hrmm, you've clearly had more time than I thinking about dynamic linker
issues.  :-)  My scheme of having the linker allocate space for the ld.so
structures necessary to generate unique function pointers can still
work;  It's just that plabels need a full struct hppa_fptr in addition
to our normal plt slot, rather than the measely single word I gave you.
This could have been accomplished simply by changing PLABEL_PLT_ENTRY_SIZE
to 20, and then ld.so wouldn't need those extra mmaps.

However, I guess we should go with what you have David - the time saved by
having the linker allocate space isn't huge, and in fact we might not even
need to mmap since ld.so has 256 fptr slots pre-allocated.

I just ran the following, and found libc.so only needs 63 global fptrs.

readelf -r libc.so | grep PLABEL \
| sed -e 's/.*R_PARISC/R_PARISC/' -e '/PLABEL      /d' -e 's/ *+.*//' \
| sort | uniq

Alan
-- 
Linuxcare.  Support for the Revolution.

Re: [parisc-linux] userspace function pointers in the kernel

[Cary Coutant]

>The model I'm thinking of for elf32-hppa is along these lines:
>
>o  The linker creates a plt entry for all plabel relocs.  plt entries for
>   elf32-hppa are a function address, linkage table pointer pair, so
>   there's no need for the dynamic linker to allocate fptrs.
>o  A dynamic plabel reloc will have the function symbol, and an addend
>   into the plt.  This is a rather unusual reloc because the function
>   symbol value is ignored when calculating the final value.
>o  The dynamic linker builds a list or hash table of function symbols
>   versus plt offsets, and adjusts plabels so that only one plt entry is
>   ever used per function.

This is close to how it works on HP-UX, but we didn't use the table in 
your third bullet, so it became possible for the dynamic loader to come 
up with one function descriptor (PLT entry) one time, and a different one 
the next time.

Another thing you need to watch out for is when a library is dlclosed. 
Because the PLT entries are allocated when you see the reference -- not 
the definition -- you may not have a candidate PLT entry in the load 
module that contains the definition of the function (unless you also 
allocate PLT entries for all exported symbols). This means that you might 
be asked to dlclose a library that contains the PLT entry that is 
currently in use as the official function pointer for a function in 
another load module. If you go ahead and deallocate the library, you're 
hosed.

-cary

Re: [parisc-linux] userspace function pointers in the kernel

[Alan Modra]

On Wed, 13 Sep 2000, Cary Coutant wrote:

> allocate PLT entries for all exported symbols). This means that you might 
> be asked to dlclose a library that contains the PLT entry that is 
> currently in use as the official function pointer for a function in 
> another load module. If you go ahead and deallocate the library, you're 
> hosed.

Yup, another reason, this one really solid, to use David's dynamically
allocated fptrs.

Note to David:  Make sure dl-fptr.c:_dl_unmap doesn't have this same
problem, if this function is called on a dlcose.

Alan
-- 
Linuxcare.  Support for the Revolution.

Re: [parisc-linux] userspace function pointers in the kernel

[Alan Modra]

On 12 Sep 2000, David Huggins-Daines wrote:

> Obviously that's wrong :-)
> 
> DYNAMIC RELOCATION RECORDS
> OFFSET   TYPE              VALUE 
> 00002bf8 R_PARISC_IPLT     *ABS*+0x00001270
> 00002bd8 R_PARISC_IPLT     __libc_start_main
> 00002be0 R_PARISC_IPLT     printf
> 00002c00 R_PARISC_IPLT     *ABS*+0x000019b4
> 00000000 R_PARISC_NONE     *ABS*
> 00000000 R_PARISC_NONE     *ABS*

Yeah, _any_ occurrence of a R_PARISC_NONE reloc indicates a linker bug.

Alan Modra
-- 
Linuxcare.  Support for the Revolution.

Re: [parisc-linux] userspace function pointers in the kernel

[Cary Coutant]

>Next, even if we *weren't* trapping we would still be doing the wrong
>thing, because $$sh_func_adrs is actually supposed to give us the
>address of the function's code, and it isn't going to do this since
>the function lives in a different space.
>
>In the case of sigaction() I think we actually want to compare the
>plabel value (rather than the code address) againsr SIG_IGN and
>SIG_DFL.  However there may be other cases where this will break.
>
>Yes this is yet another side effect of the broken PA-RISC run time
>architecture.

I'm deeply troubled to learn that Linux is using $$sh_func_adrs, as I 
will readily agree with you that it's seriously broken.

It was introduced as a way to compare function pointers reliably, but in 
fact accomplishes exactly the opposite. The problem is that our dynamic 
loader cannot guarantee that two function pointers that both refer to the 
same procedure will compare equal. Rather than fix the dynamic loader, we 
chose to modify the compilers to compare two function pointers by digging 
down into the function descriptor itself to see if they really refer to 
the same function. Unfortunately, not all data items typed as function 
pointers are really valid pointers, and they can't be dereferenced 
safely. Thus, the millicode routine uses the probe instruction to verify 
that the address can be dereferenced without causing a memory fault.

Of course, this isn't 100% reliable on HP-UX, either, since a probe 
failure doesn't mean that the address isn't mapped. It should also be a 
"probe,r" instead of "probe,w".

For Linux, I strongly recommend that the dynamic loader guarantee that 
there is a 1-1 mapping between function and function pointer. If you do 
this, you won't need this silly special case when comparing function 
pointers.

>What do the IA-64 people do about this problem?

On IA-64, the dynamic loader will maintain a 1-1 mapping.

-cary

Re: [parisc-linux] userspace function pointers in the kernel

[David Huggins-Daines]

Cary Coutant <cary@cup.hp.com> writes:

> I wrote:
> >Yes this is yet another side effect of the broken PA-RISC run time
> >architecture.

Hm.  As it turns out it is not the kernel that changed but rather the
linker.  We had previously been generating dynamic procedure
descriptors for PLABEL32 relocations only in shared libraries, but
switched to generating PLT entries for all functions (even in non-PIC
objects, which is a bug) and then pointing the PLABEL32 relocations at
these slots.  Hence I'm forwarding this to Alan.

Of course we would have had the same problem if a shared library had
tried to pass the address of a global function to signal(2), so I
guess it's a good thing we caught it now :-)

> I'm deeply troubled to learn that Linux is using $$sh_func_adrs, as I 
> will readily agree with you that it's seriously broken.
>
> For Linux, I strongly recommend that the dynamic loader guarantee that 
> there is a 1-1 mapping between function and function pointer. If you do 
> this, you won't need this silly special case when comparing function 
> pointers.

I agree that this is a valid goal.  Unfortunately I believe we are
falling short of it at the moment and we'll have to make some fairly
large changes to the toolchain.  If you don't mind I've got a few
questions about how this is done:

It looks like the IA-64 runtime achieves this by generating official
procedure descriptors for every defined function in an object and
placing them in the .opd section, then using these descriptors
whenever the address of a function is taken.  Presumably external
functions are compared using their .IA_64.pltoff entries?  I'm not
clear on how that works and don't have an IA-64 handy to experiment.

I'm also confused about how this interacts with dynamic linking in
another case - from what I've seen the descriptors in .opd are never
relocated, so if a function ends up being bound to another dynamic
object (in the case of weak symbols in shared libraries this can
happen, and we've had to make some adjustments to our linker and
assembler to handle this case since GNU libc relies on this behaviour
in some fundamental ways), it would appear that the .opd entries
become bogus and the pltoff entry (or whatever) should be used
instead.  How does the IA-64 runtime handle this case?

Also I've noticed that GNU libc doesn't actually use the .opd or
.pltoff entries to resolve dynamic FPTR64 relocations but rather
builds FPTRs dynamically when it encounters them.  So the 1-1 mapping
isn't really upheld at all in some cases.

But then HP/UX probably does something else entirely :-)

-- 
dhd@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Support for the revolution.

Re: [parisc-linux] userspace function pointers in the kernel

[Alan Modra]

On 12 Sep 2000, David Huggins-Daines wrote:

> Cary Coutant <cary@cup.hp.com> writes:
> > For Linux, I strongly recommend that the dynamic loader guarantee that 
> > there is a 1-1 mapping between function and function pointer. If you do 
> > this, you won't need this silly special case when comparing function 
> > pointers.
> 
> I agree that this is a valid goal.  Unfortunately I believe we are
> falling short of it at the moment and we'll have to make some fairly
> large changes to the toolchain.

David, I think this is the reason we do actually need your dynamic fptr
code, or at least something like it that can sort through plt entries.
Currently, plabels go through the plt whenever dynamic sections are
involved.  However, there is a problem in that we are duplicating plt
entries between the application and shared libs, and I can't think of a
way the linker can avoid this given that we have to deal with weak
symbols.

Alan Modra
-- 
Linuxcare.  Support for the Revolution.

[parisc-linux] userspace function pointers in the kernel

[David Huggins-Daines]

Hi,

This simple program:

#include <signal.h>

void foo(int signo)
{
	psignal(signo, "foo");
}

int main()
{
	signal(SIGFPE, foo);
	return 0;
}

Now causes the kernel (latest from CVS) to crash where it didn't
before:

Kernel Fault: Code=15 regs=c7c44600 (Addr=00002c04)

PSW  : 0006ff0a  GR 1 : c0233000  GR 2 : c011dcf0  GR 3 : c7c44518  
GR 4 : 20020350  GR 5 : 00000008  GR 6 : 00000007  GR 7 : 00000008  
GR 8 : c7c44600  GR 9 : 2002114c  GR10 : 20020670  GR11 : 00000040  
GR12 : 00000001  GR13 : 00000078  GR14 : 00057000  GR15 : 00057000  
GR16 : 00000063  GR17 : 00000020  GR18 : 2002058c  GR19 : 00000000  
GR20 : 00002c06  GR21 : c7f2387c  GR22 : fffffff8  GR23 : c7f23880  
GR24 : c7c44528  GR25 : c7c44518  GR26 : 00002c04  GR27 : c027a000  
GR28 : fffffff2  GR29 : 00002c06  GR30 : c7c44840  GR31 : c011d7c8  
SR0  : 00002002  SR1  : 00002002  SR2  : 00000000  SR3  : 00002002  
SR4  : 00000000  SR5  : 00000000  SR6  : 00000000  SR7  : 00000000  

IASQ : 00000000 00000000 IAOQ : c02336a0 c02336a4 ORIG_R28 : 00003fff
 IIR : 075f11d6 ISR : 00000000 IOR : 00002c04

Which is:

c0233698 <$$sh_func_adrs>:
c0233698:       37 5d 00 00     ldo 0(r26),ret1
c023369c:       d7 40 0c 3f     depw r0,30,1,r26
c02336a0:       07 5f 11 d6     probe,w (sr0,r26),r31,r22
                                ^^^^^^^^^^^^^^^^^^^^^^^^^  this insn
c02336a4:       d2 d6 3b ff     extrw,u,= r22,31,1,r22
c02336a8:       0f 40 10 9d     ldw  0(sr0,r26),ret1
c02336ac:       eb e0 c0 02     bv,n r0(r31)

Following the millicode return pointer takes us to do_signal() in
kernel/signal.c, somewhere around these lines:

		if (k->sa.sa_handler == SIG_IGN
		    || (k->sa.sa_handler == SIG_DFL

Note that the kernel is trying to compare a function pointer passed in
from userspace with an arbitrary value.

Now, first of all, this means our DTLB miss handler is broken with
respect to the PROBE instructions.  According to the architecture
manual,

        Notes: If this instruction causes a non-access data TLB miss
        fault/non-access data page fault, the operating system's
        handler is required to search its page tables for the given
        address. If found, it does the appropriate TLB insert and
        returns to the interrupting instruction. If not found, the
        handler must decode the target field of the instruction, set
        that GR to 0, set the IPSW[N] bit to 1, and return to the
        interrupting instruction.

Obviously that's not what's happening here.

Next, even if we *weren't* trapping we would still be doing the wrong
thing, because $$sh_func_adrs is actually supposed to give us the
address of the function's code, and it isn't going to do this since
the function lives in a different space.

In the case of sigaction() I think we actually want to compare the
plabel value (rather than the code address) againsr SIG_IGN and
SIG_DFL.  However there may be other cases where this will break.

Yes this is yet another side effect of the broken PA-RISC run time
architecture.  What do the IA-64 people do about this problem?

In the worst case scenario we may have to hack our millicode in the
kernel somewhat to deal with this.

-- 
dhd@linuxcare.com, http://www.linuxcare.com/
Linuxcare. Support for the revolution.