Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Jeremy Friesner]

Hi Linux Guys,

I have what I hope is an easy question for you... I have am writing an
user-space application that does some multithreading (using pthreads), and
does some thread-safe reference counting by using the functions in
asm/atomic.h.  Specifically, I'm using the atomic_inc(atomic_t *) and
atomic_dec_and_test(atomic_t *) functions.

Under Red Had Linux 7.1 on an Intel box, the program compiles and links fine.
Under LinuxPPC 2.2.17-0.6.1, however, the link fails with the error output
that is included below -- it fails whether I have optimization enabled (-O3),
or not.  Is there some special library I need to link in, or magic compiler
flag needed in order to access those functions?  Any advice?

Also, what does "relocation truncated to fit: R_PPC_REL24 atomic_inc(atomic_t
*)" mean?

Thanks,
Jeremy

Begin link output--------------

gcc  -o lxelcd Message.o AbstractMessageIOGateway.o MessageIOGateway.o
String.o StorageReflectSession.o AbstractReflectSession.o
DumbReflectSession.o ReflectServer.o StringMatcher.o PathMatcher.o
NetworkUtilityFunctions.o SysLog.o Thread.o muscled.o FileDescriptorDataIO.o
FilterSessionFactory.o PulseNode.o RateLimitSessionIOPolicy.o
MemoryAllocator.o GlobalMemoryAllocator.o CCSysexMessageIOGateway.o
CCModuleMessageIOGateway.o CCFrameMessageIOGateway.o CCSession.o
CCModuleChainSession.o CCControlSession.o CCServer.o CCFrameSession.o
CCComponentSession.o CCConfigFile.o CCRangeConvert.o CCModuleLogic.o
CCTransportModuleLogic.o CCEditorModuleLogic.o CCMeterModuleLogic.o
CCFaderModuleLogic.o CCBitPack.o CCFrameSubscription.o
CCDynamicHardwareSession.o CCSoftMeterSession.o CCSoftMeterModuleLogic.o
CCAbstractEditBoxModuleLogic.o CCEQEditBoxModuleLogic.o
CCBusAssignEditBoxModuleLogic.o CCSoftGenericSession.o
CCSoftGenericModuleLogic.o ControlPointSpec.o CCChannelInfo.o ccserverd.o
ccconfigd.o LXADTPSession.o LXFakeStorage.o LXFakeFileStorage.o
LXADTPIOGateway.o LXSCSIStorage.o lxaudiodtpd.o LXWTRXAudioIOGateway.o
LXWTRXAudioSession.o LXWTRXIOGateway.o LXWTRXServer.o LXWTRXSession.o
LXWTRXStatusIOGateway.o LXWTRXStatusSession.o tc.o lxwtrxd.o AsyncIORequest.o
LXWTRXAsyncDiskIOSession.o SafetyNetDataIO.o LXTCPComSession.o
LXTCPComIOGateway.o LXTCPHSBIOGateway.o LXTCPComServer.o lxtcpcomd.o lxelcd.o
webserve.o sock.o filewebserve.o getframesstatus.o sendsysex.o watchalerts.o
smallframestatus.o -lm -lpthread
Message.o: In function `muscle::AtomicCounter::AtomicIncrement(void)':
Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicIncrement(void)+0x24):
undefined reference to `atomic_inc(atomic_t *)'
Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicIncrement(void)+0x24):
relocation truncated to fit: R_PPC_REL24 atomic_inc(atomic_t *)
Message.o: In function `muscle::AtomicCounter::AtomicDecrement(void)':
Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicDecrement(void)+0x24):
undefined reference to `atomic_dec_and_test(atomic_t *)'
Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicDecrement(void)+0x24):
relocation truncated to fit: R_PPC_REL24 atomic_dec_and_test(atomic_t *)
collect2: ld returned 1 exit status
make: *** [lxelcd] Error 1
[jaf@jonathan lxelcd]$

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Jerry Van Baren]

At 07:24 PM 1/3/2002 -0800, Jeremy Friesner wrote:

>Hi Linux Guys,
>
>I have what I hope is an easy question for you... I have am writing an
>user-space application that does some multithreading (using pthreads), and
>does some thread-safe reference counting by using the functions in
>asm/atomic.h.  Specifically, I'm using the atomic_inc(atomic_t *) and
>atomic_dec_and_test(atomic_t *) functions.
>
>Under Red Had Linux 7.1 on an Intel box, the program compiles and links fine.
>Under LinuxPPC 2.2.17-0.6.1, however, the link fails with the error output
>that is included below -- it fails whether I have optimization enabled (-O3),
>or not.  Is there some special library I need to link in, or magic compiler
>flag needed in order to access those functions?  Any advice?
>
>Also, what does "relocation truncated to fit: R_PPC_REL24 atomic_inc(atomic_t
>*)" mean?

This means the function wasn't found by the linker (as noted on the line
previous).  You need to find or write atomic_inc() and atomic_dec_and_test().

The linker apparently uses a bogus number to fill in for unresolved
addresses and then, later, tries to fill out a jump target offset to get to
that bogus address.  On the PPC, normal branches and subroutine calls are
24 bit relative values, so you cannot branch directly to 0x80000000 (for
instance) from location 0x000xxxxx (you can get there, but it involves
loading a register with the destination and then branching using the register).

gvb



>Thanks,
>Jeremy
>
>Begin link output--------------
>
>gcc  -o lxelcd Message.o AbstractMessageIOGateway.o MessageIOGateway.o
>String.o StorageReflectSession.o AbstractReflectSession.o
>DumbReflectSession.o ReflectServer.o StringMatcher.o PathMatcher.o
>NetworkUtilityFunctions.o SysLog.o Thread.o muscled.o FileDescriptorDataIO.o
>FilterSessionFactory.o PulseNode.o RateLimitSessionIOPolicy.o
>MemoryAllocator.o GlobalMemoryAllocator.o CCSysexMessageIOGateway.o
>CCModuleMessageIOGateway.o CCFrameMessageIOGateway.o CCSession.o
>CCModuleChainSession.o CCControlSession.o CCServer.o CCFrameSession.o
>CCComponentSession.o CCConfigFile.o CCRangeConvert.o CCModuleLogic.o
>CCTransportModuleLogic.o CCEditorModuleLogic.o CCMeterModuleLogic.o
>CCFaderModuleLogic.o CCBitPack.o CCFrameSubscription.o
>CCDynamicHardwareSession.o CCSoftMeterSession.o CCSoftMeterModuleLogic.o
>CCAbstractEditBoxModuleLogic.o CCEQEditBoxModuleLogic.o
>CCBusAssignEditBoxModuleLogic.o CCSoftGenericSession.o
>CCSoftGenericModuleLogic.o ControlPointSpec.o CCChannelInfo.o ccserverd.o
>ccconfigd.o LXADTPSession.o LXFakeStorage.o LXFakeFileStorage.o
>LXADTPIOGateway.o LXSCSIStorage.o lxaudiodtpd.o LXWTRXAudioIOGateway.o
>LXWTRXAudioSession.o LXWTRXIOGateway.o LXWTRXServer.o LXWTRXSession.o
>LXWTRXStatusIOGateway.o LXWTRXStatusSession.o tc.o lxwtrxd.o AsyncIORequest.o
>LXWTRXAsyncDiskIOSession.o SafetyNetDataIO.o LXTCPComSession.o
>LXTCPComIOGateway.o LXTCPHSBIOGateway.o LXTCPComServer.o lxtcpcomd.o lxelcd.o
>webserve.o sock.o filewebserve.o getframesstatus.o sendsysex.o watchalerts.o
>smallframestatus.o -lm -lpthread
>Message.o: In function `muscle::AtomicCounter::AtomicIncrement(void)':
>Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicIncrement(void)+0x24):
>undefined reference to `atomic_inc(atomic_t *)'
>Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicIncrement(void)+0x24):
>relocation truncated to fit: R_PPC_REL24 atomic_inc(atomic_t *)
>Message.o: In function `muscle::AtomicCounter::AtomicDecrement(void)':
>Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicDecrement(void)+0x24):
>undefined reference to `atomic_dec_and_test(atomic_t *)'
>Message.o(.muscle::AtomicCounter::gnu.linkonce.t.AtomicDecrement(void)+0x24):
>relocation truncated to fit: R_PPC_REL24 atomic_dec_and_test(atomic_t *)
>collect2: ld returned 1 exit status
>make: *** [lxelcd] Error 1
>[jaf@jonathan lxelcd]$
>


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Benjamin Herrenschmidt]

>>I have what I hope is an easy question for you... I have am writing an
>>user-space application that does some multithreading (using pthreads), and
>>does some thread-safe reference counting by using the functions in
>>asm/atomic.h.  Specifically, I'm using the atomic_inc(atomic_t *) and
>>atomic_dec_and_test(atomic_t *) functions.

Those are kernel functions, not availble to userland. If your userland
application need them, you have to either re-implement them, or rely on
equivalent routines provided by glibc if any.

If you plan to re-implement them, beware that some CPU-specific errata
have to be taken into account, you probably want to add all the sync's
that are CONFIG-dependant in the kernel in your userland implementation
for it to work properly on all PPC CPUs and on SMP machines.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Michael Schmitz]

> Also, what does "relocation truncated to fit: R_PPC_REL24 atomic_inc(atomic_t
> *)" mean?

About as much as 'unresolved external reference'. atomic_inc isn't defined
in the scope of your code. Look at the kernel headers; it might be inside
#ifdef __KRENEL__ (actually it is).

Why do you think you need to use atomic_inc directly instead of some
pthreads wrapper?

	Michael

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[jaf]

Hi Michael,

>> Also, what does "relocation truncated to fit: R_PPC_REL24
atomic_inc(atomic_t
>> *)" mean?
>
>About as much as 'unresolved external reference'. atomic_inc isn't
defined
>in the scope of your code. Look at the kernel headers; it might be
inside
>#ifdef __KRENEL__ (actually it is).

I see... when I wrote the code using Red Hat, this was not the case.  I
assumed
things would be similar under all Linuxes... apparently a bad
assumption.

>Why do you think you need to use atomic_inc directly instead of some
>pthreads wrapper?

I wasn't able to find a decent equivalent to atomic_t in the pthreads
API.
The closest thing I could see would be to wrap my counter increments/
decrements in a pthread_mutex_t to serialize them, but creating a
separate mutex for each atomic counter seems a bit expensive/
inefficient, considering there may be thousands of such atomic counters
active at once.  Is there some other way to use pthreads to get an
atomic counter?  Or is a pthread_mutex_t really efficient enough to
make a decent atomic counter out of?  Or do I need to redesign how my
application works, because there is no good way to do a cheap, portable
user-land atomic counter under Linux?  :^(

Jeremy

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Michael Schmitz]

> >> Also, what does "relocation truncated to fit: R_PPC_REL24
> atomic_inc(atomic_t
> >> *)" mean?
> >
> >About as much as 'unresolved external reference'. atomic_inc isn't
> defined
> >in the scope of your code. Look at the kernel headers; it might be
> inside
> >#ifdef __KRENEL__ (actually it is).
>
> I see... when I wrote the code using Red Hat, this was not the case.  I
> assumed
> things would be similar under all Linuxes... apparently a bad
> assumption.

It's a kernel or libc header issue - what kernel/libc version was that
on? As far as I can see atomic operations are exported to user programs
except on ppc (not ppc64), mips*, arm and sparc. At least in one case it's
obvious why: MIPS needs to protect against interrupts to guarantee
atomicity.
Another corner case seems to be IBM 405 processors that use what I think
is a privileged instruction as well in order to work around a hardware
bug.

> >Why do you think you need to use atomic_inc directly instead of some
> >pthreads wrapper?
>
> I wasn't able to find a decent equivalent to atomic_t in the pthreads
> API.

pthreadss only came to mind first, there may be other packages. Point
being, on those architectures that cannot guarantee atomic operations
using nonprovileged instructions there has to be some sort of kernel
cooperation (system call?).

> The closest thing I could see would be to wrap my counter increments/
> decrements in a pthread_mutex_t to serialize them, but creating a
> separate mutex for each atomic counter seems a bit expensive/
> inefficient, considering there may be thousands of such atomic counters
> active at once.  Is there some other way to use pthreads to get an
> atomic counter?  Or is a pthread_mutex_t really efficient enough to

I have no experience with pthreads, sorry.

> make a decent atomic counter out of?  Or do I need to redesign how my
> application works, because there is no good way to do a cheap, portable
> user-land atomic counter under Linux?  :^(

Worst-case? The latter. At least for MIPS there doesn't seem to be such a
thing as atomic user land operations. How they would implement a mutex
without those, I wonder ...

You can try to copy the atomic ops from the kernel headers for those
architectures lacking them, hoping it won't break. If it doesn't, you have
a good argument in favor of removing the #ifdef __KERNEL__ :-)

	Michael

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Kevin B. Hendricks]

Hi,

There are a number of places you can get atomic increment and decrement
code:

1. borrow the implementations from the kernel

2. look at www.openOffice.org CVS sal/osl/unx/interlck.c
    for atomic increment and decrement code for both x86 linux and ppc
    linux, and the proper includes for Solaris.

3. adapt the code from
glibc-2.2.4/linuxthreads/sysdeps/powerpc/pt-machine.h for "compare
_and_swap" and "test_and_set" t do your own "dec_and_test"

4. simply use a pthread mutex to protect the counter (complete overkill)

5. look at the implementations of these synchronization functions in the
IBM/Motorola PowerPC Microprocessor Family: The Programming Environment"
manual (available as a pdf from the Motorola site).  They have
implementations for "Fetch and Add", "Test and Set" , "Compare and Swap"
"Lock Acquisition and Release" and "List Insertion"

I personally recommend using the code from interlck.c from OpenOffice as
your guide and falling back to using a mutex to protect the count for
unknown processors.  That way it will always work but if performance is an
issue, someone can add the support for other processors.  This is what
OpenOffice does for its refernce counted strings.

Hope this helps,

Kevin


On January 4, 2002 12:00, jaf wrote:
> Hi Michael,
>
> >> Also, what does "relocation truncated to fit: R_PPC_REL24
>
> atomic_inc(atomic_t
>
> >> *)" mean?
> >
> >About as much as 'unresolved external reference'. atomic_inc isn't
>
> defined
>
> >in the scope of your code. Look at the kernel headers; it might be
>
> inside
>
> >#ifdef __KRENEL__ (actually it is).
>
> I see... when I wrote the code using Red Hat, this was not the case.  I
> assumed
> things would be similar under all Linuxes... apparently a bad
> assumption.
>
> >Why do you think you need to use atomic_inc directly instead of some
> >pthreads wrapper?
>
> I wasn't able to find a decent equivalent to atomic_t in the pthreads
> API.
> The closest thing I could see would be to wrap my counter increments/
> decrements in a pthread_mutex_t to serialize them, but creating a
> separate mutex for each atomic counter seems a bit expensive/
> inefficient, considering there may be thousands of such atomic counters
> active at once.  Is there some other way to use pthreads to get an
> atomic counter?  Or is a pthread_mutex_t really efficient enough to
> make a decent atomic counter out of?  Or do I need to redesign how my
> application works, because there is no good way to do a cheap, portable
> user-land atomic counter under Linux?  :^(

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Kevin B. Hendricks]

Hi,

One other thing, PPC implements synchronization using a reservation bit
for specific address with a special instruction pair that sets and clears
the reservation.  But the address to set/clear the reservation on is only
unique down the the cache line size (in most cases 32 bytes for ppc) i.e.
the reservation granularity.

In English, what this means is that if two counters you want to atomically
increment or decrement are stored within 32 bytes of each other and
different threads are simulataneously trying to increment or decrement
them, the reservation can get spurious clears that prevents either one
from working without more looping (i.e performance starts to suffer).

In practive I have very very rarely ever seen this problem have an impact
on anything.  Others may know more.

Hope this helps,

Kevin



On January 4, 2002 01:33, Kevin B. Hendricks wrote:
> Hi,
>
> There are a number of places you can get atomic increment and decrement
> code:
>
> 1. borrow the implementations from the kernel
>
> 2. look at www.openOffice.org CVS sal/osl/unx/interlck.c
>     for atomic increment and decrement code for both x86 linux and ppc
>     linux, and the proper includes for Solaris.
>
> 3. adapt the code from
> glibc-2.2.4/linuxthreads/sysdeps/powerpc/pt-machine.h for "compare
> _and_swap" and "test_and_set" t do your own "dec_and_test"
>
> 4. simply use a pthread mutex to protect the counter (complete overkill)
>
> 5. look at the implementations of these synchronization functions in the
> IBM/Motorola PowerPC Microprocessor Family: The Programming Environment"
> manual (available as a pdf from the Motorola site).  They have
> implementations for "Fetch and Add", "Test and Set" , "Compare and Swap"
> "Lock Acquisition and Release" and "List Insertion"
>
> I personally recommend using the code from interlck.c from OpenOffice as
> your guide and falling back to using a mutex to protect the count for
> unknown processors.  That way it will always work but if performance is
> an issue, someone can add the support for other processors.  This is
> what OpenOffice does for its refernce counted strings.
>
> Hope this helps,
>
> Kevin
>
> On January 4, 2002 12:00, jaf wrote:
> > Hi Michael,
> >
> > >> Also, what does "relocation truncated to fit: R_PPC_REL24
> >
> > atomic_inc(atomic_t
> >
> > >> *)" mean?
> > >
> > >About as much as 'unresolved external reference'. atomic_inc isn't
> >
> > defined
> >
> > >in the scope of your code. Look at the kernel headers; it might be
> >
> > inside
> >
> > >#ifdef __KRENEL__ (actually it is).
> >
> > I see... when I wrote the code using Red Hat, this was not the case.
> > I assumed
> > things would be similar under all Linuxes... apparently a bad
> > assumption.
> >
> > >Why do you think you need to use atomic_inc directly instead of some
> > >pthreads wrapper?
> >
> > I wasn't able to find a decent equivalent to atomic_t in the pthreads
> > API.
> > The closest thing I could see would be to wrap my counter increments/
> > decrements in a pthread_mutex_t to serialize them, but creating a
> > separate mutex for each atomic counter seems a bit expensive/
> > inefficient, considering there may be thousands of such atomic
> > counters active at once.  Is there some other way to use pthreads to
> > get an atomic counter?  Or is a pthread_mutex_t really efficient
> > enough to make a decent atomic counter out of?  Or do I need to
> > redesign how my application works, because there is no good way to do
> > a cheap, portable user-land atomic counter under Linux?  :^(
>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test() in user app, help!

[Tom Rini]

On Fri, Jan 04, 2002 at 06:44:03PM +0100, Michael Schmitz wrote:
>
> > >> Also, what does "relocation truncated to fit: R_PPC_REL24
> > atomic_inc(atomic_t
> > >> *)" mean?
> > >
> > >About as much as 'unresolved external reference'. atomic_inc isn't
> > defined
> > >in the scope of your code. Look at the kernel headers; it might be
> > inside
> > >#ifdef __KRENEL__ (actually it is).
> >
> > I see... when I wrote the code using Red Hat, this was not the case.  I
> > assumed
> > things would be similar under all Linuxes... apparently a bad
> > assumption.

A userspace app using <asm/*.h> and <linux/*.h> is quite probably a bug.
If you need something from the kernel, make a local copy and never,
ever, ever, reference it directly, since it can change on you.

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test()in user app, help!

[Frank Rowand]

Michael Schmitz wrote:
>
> > >> Also, what does "relocation truncated to fit: R_PPC_REL24
> > atomic_inc(atomic_t
> > >> *)" mean?
> > >
> > >About as much as 'unresolved external reference'. atomic_inc isn't
> > defined
> > >in the scope of your code. Look at the kernel headers; it might be
> > inside
> > >#ifdef __KRENEL__ (actually it is).
> >
> > I see... when I wrote the code using Red Hat, this was not the case.  I
> > assumed
> > things would be similar under all Linuxes... apparently a bad
> > assumption.
>
> It's a kernel or libc header issue - what kernel/libc version was that
> on? As far as I can see atomic operations are exported to user programs
> except on ppc (not ppc64), mips*, arm and sparc. At least in one case it's
> obvious why: MIPS needs to protect against interrupts to guarantee
> atomicity.
> Another corner case seems to be IBM 405 processors that use what I think
> is a privileged instruction as well in order to work around a hardware
> bug.

The dcbt instruction (the workaround) should not be privileged.

< stuff deleted >

-Frank
--
Frank Rowand <frank_rowand@mvista.com>
MontaVista Software, Inc

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Having linking problems with atomic_inc(), atomic_dec_and_test()in user app, help!

[Michael Schmitz]

> > Another corner case seems to be IBM 405 processors that use what I think
> > is a privileged instruction as well in order to work around a hardware
> > bug.
>
> The dcbt instruction (the workaround) should not be privileged.

Thanks; anything not using syncs should be fine then?

	Michael


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/