Why does linking a.so with b.so require execs to be linked both -la and -lb?

Why does linking a.so with b.so require execs to be linked both -la and -lb?

[David Wuertele]

#  I'm creating a shared library like this:
libDaveA.so: $(LIB_OBJECTS)
	$(CXX) -shared -ldl -rdynamic -o $@ $^ -lDaveB -lDaveC -lDaveD -lDaveE

#  And I have some executables that use the shared lib:
daveprog: daveprog.cpp
	$(CXX) -o $@ $^ -lDaveA

The executable doesn't use any of the functions from libDaveB.so,
libDaveC.so, libDaveD.so, or libDaveE.so.  It only uses functions from
libDaveA.so.  But when I compile with the above makefile, gcc
complains:

  /usr/tools/lib/gcc-lib/mipsel-linux/2.96/../../../../mipsel-linux/bin/ld: warning: libDaveB.so.0, needed by /usr/local/lib/libDaveA.so, not found (try using -rpath or -rpath-link)

Why is that?  Why should daveprog, which never uses any functionality
AT ALL from libDaveB.so, have to link to it?

Dave

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[Mike Castle]

In article <m3k79jj4av.fsf@bfnet.com>,
David Wuertele  <dave-gnus@bfnet.com> wrote:
>Why is that?  Why should daveprog, which never uses any functionality
>AT ALL from libDaveB.so, have to link to it?

Consider what would happen if everything was using static libraries
instead.

In my opinion, it should be a consistent command line between static and
dynamic linking, with adding -static the only difference.

mrc
-- 
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[David Wuertele]

I asked my friend, who is a windows programmer, whether windows had
the same behavior.  He said absolutely not, and generated this test.
His test is even more interesting than mine.  It does the following:

1.  build two libraries libA and libB.  libA has one function that
    calls a function in libB.
2.  link libA and libB
3.  Erase libB
4.  link a program that uses A's function with libA --- WITHOUT HAVING
    ACCESS TO libB!

Why can't we do this on Linux?

Here's the windows demonstration:


C:\libtest>REM -- build.bat - build dynamic linking test 

C:\libtest>REM -- clean all output files and do a dir 

C:\libtest>erase libB.obj libB.dll libB.exp libB.lib  

C:\libtest>erase libA.obj libA.dll libA.exp libA.lib  

C:\libtest>erase prog.obj prog.exe 

C:\libtest>dir
 Volume in drive C is SilverBox
 Volume Serial Number is BCEE-2C23

 Directory of C:\libtest

08/11/2003  04:09 PM    <DIR>          .
08/11/2003  04:09 PM    <DIR>          ..
08/11/2003  04:07 PM             1,437 build.bat
08/11/2003  03:33 PM               159 libA.c
08/11/2003  03:33 PM                65 libA.h
08/11/2003  03:33 PM               112 libB.c
08/11/2003  03:33 PM                65 libB.h
08/11/2003  03:45 PM               202 prog.c
               6 File(s)          2,040 bytes
               2 Dir(s)  40,561,799,168 bytes free

C:\libtest>REM -- cat the contents of the files 

C:\libtest>type libA.h 
// FILE: libA.h

extern __declspec(dllimport) void libA_func();
C:\libtest>type libA.c 
// FILE: libA.c

#include "libB.h"

extern __declspec(dllexport) void libA_func()
{
    printf("libA_func() calling libB_func()\n");
    libB_func();
}
C:\libtest>type libB.h 
// FILE: libB.h

extern __declspec(dllimport) void libB_func();
C:\libtest>type libB.c 
// FILE: libB.c

extern __declspec(dllexport) void libB_func()
{
    printf("libB_func() executing()\n");
}
C:\libtest>type prog.c 
#include <stdio.h>
#include "libA.h"

int main(int argc, const char ** argv)
{
    printf("prog.exe calling libA_func()\n");
    libA_func();
    printf("prog.exe exiting\n");
    return 0;
}

C:\libtest>REM -- compile and link libB.c to libB.dll 

C:\libtest>cl -c libB.c 
libB.c

C:\libtest>link libB.obj /dll /out:"libB.dll" /implib:"libB.lib" 
Microsoft (R) Incremental Linker Version 6.00.8447
Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

   Creating library libB.lib and object libB.exp

C:\libtest>REM -- compile and link libA.c to libA.dll doing 

C:\libtest>REM -- the "-lB" equivalent 

C:\libtest>cl -c libA.c 
libA.c

C:\libtest>link libA.obj libB.lib /dll /out:"libA.dll" /implib:"libA.lib" 
Microsoft (R) Incremental Linker Version 6.00.8447
Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

   Creating library libA.lib and object libA.exp

C:\libtest>REM -- erase all of the libB output files to prove that linking 

C:\libtest>REM -- prog.c doesn't depend on libB. 

C:\libtest>erase libB.obj libB.dll libB.exp libB.lib  

C:\libtest>REM -- for good measure erase all of libA's output files except 

C:\libtest>REM -- the export library and the dll 

C:\libtest>erase  libA.obj libA.exp 

C:\libtest>REM -- do another dir so we can see what's in our directory now 

C:\libtest>dir
 Volume in drive C is SilverBox
 Volume Serial Number is BCEE-2C23

 Directory of C:\libtest

08/11/2003  04:09 PM    <DIR>          .
08/11/2003  04:09 PM    <DIR>          ..
08/11/2003  04:07 PM             1,437 build.bat
08/11/2003  03:33 PM               159 libA.c
08/11/2003  04:09 PM            45,056 libA.dll
08/11/2003  03:33 PM                65 libA.h
08/11/2003  04:09 PM             1,896 libA.lib
08/11/2003  03:33 PM               112 libB.c
08/11/2003  03:33 PM                65 libB.h
08/11/2003  03:45 PM               202 prog.c
               8 File(s)         48,992 bytes
               2 Dir(s)  40,561,750,016 bytes free

C:\libtest>REM -- compile prog.c into prog.exe 

C:\libtest>cl prog.c libA.lib 
prog.c
Microsoft (R) Incremental Linker Version 6.00.8447
Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

/out:prog.exe 
prog.obj 
libA.lib 

C:\libtest>REM -- now that prog.exe has built, re-compile and link 

C:\libtest>REM -- libB so that prog.exe will run 

C:\libtest>cl -c libB.c 
libB.c

C:\libtest>link libB.obj /dll /out:"libB.dll" /implib:"libB.lib" 
Microsoft (R) Incremental Linker Version 6.00.8447
Copyright (C) Microsoft Corp 1992-1998. All rights reserved.

   Creating library libB.lib and object libB.exp

C:\libtest>REM -- let's delete all output files again and do a dir 

C:\libtest>erase libB.obj libB.exp libB.lib  

C:\libtest>erase libA.obj libA.exp libA.lib  

C:\libtest>erase prog.obj 

C:\libtest>dir
 Volume in drive C is SilverBox
 Volume Serial Number is BCEE-2C23

 Directory of C:\libtest

08/11/2003  04:09 PM    <DIR>          .
08/11/2003  04:09 PM    <DIR>          ..
08/11/2003  04:07 PM             1,437 build.bat
08/11/2003  03:33 PM               159 libA.c
08/11/2003  04:09 PM            45,056 libA.dll
08/11/2003  03:33 PM                65 libA.h
08/11/2003  03:33 PM               112 libB.c
08/11/2003  04:09 PM            45,056 libB.dll
08/11/2003  03:33 PM                65 libB.h
08/11/2003  03:45 PM               202 prog.c
08/11/2003  04:09 PM            40,960 prog.exe
               9 File(s)        133,112 bytes
               2 Dir(s)  40,561,664,000 bytes free

C:\libtest>REM -- NOW FINALLY LET'S RUN THE PROGRAM! 

C:\libtest>prog.exe
prog.exe calling libA_func()
libA_func() calling libB_func()
libB_func() executing()
prog.exe exiting

C:\libtest>REM

C:\libtest>REM

C:\libtest>REM

C:\libtest>REM -- and let's type this file 

C:\libtest>type build.bat 
	REM -- build.bat - build dynamic linking test
	REM -- clean all output files and do a dir
erase libB.obj libB.dll libB.exp libB.lib 
erase libA.obj libA.dll libA.exp libA.lib 
erase prog.obj prog.exe
dir
	REM -- cat the contents of the files
type libA.h
type libA.c
type libB.h
type libB.c
type prog.c
	REM -- compile and link libB.c to libB.dll
cl -c libB.c
link libB.obj /dll /out:"libB.dll" /implib:"libB.lib"
	REM -- compile and link libA.c to libA.dll doing
	REM -- the "-lB" equivalent
cl -c libA.c
link libA.obj libB.lib /dll /out:"libA.dll" /implib:"libA.lib"
	REM -- erase all of the libB output files to prove that linking
	REM -- prog.c doesn't depend on libB.
erase libB.obj libB.dll libB.exp libB.lib 
	REM -- for good measure erase all of libA's output files except
	REM -- the export library and the dll
erase  libA.obj libA.exp
	REM -- do another dir so we can see what's in our directory now
dir
	REM -- compile prog.c into prog.exe
cl prog.c libA.lib
	REM -- now that prog.exe has built, re-compile and link
	REM -- libB so that prog.exe will run
cl -c libB.c
link libB.obj /dll /out:"libB.dll" /implib:"libB.lib"
	REM -- let's delete all output files again and do a dir
erase libB.obj libB.exp libB.lib 
erase libA.obj libA.exp libA.lib 
erase prog.obj
dir
	REM -- NOW FINALLY LET'S RUN THE PROGRAM!
prog.exe
	REM
	REM
	REM
	REM -- and let's type this file
type build.bat

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[Glynn Clements]

David Wuertele wrote:

> I asked my friend, who is a windows programmer, whether windows had
> the same behavior.  He said absolutely not, and generated this test.
> His test is even more interesting than mine.  It does the following:
> 
> 1.  build two libraries libA and libB.  libA has one function that
>     calls a function in libB.
> 2.  link libA and libB
> 3.  Erase libB
> 4.  link a program that uses A's function with libA --- WITHOUT HAVING
>     ACCESS TO libB!
> 
> Why can't we do this on Linux?
> 
> Here's the windows demonstration:

> C:\libtest>type libA.h 
> // FILE: libA.h
> 
> extern __declspec(dllimport) void libA_func();

> C:\libtest>type libA.c 
> // FILE: libA.c
> 
> #include "libB.h"
> 
> extern __declspec(dllexport) void libA_func()
> {
>     printf("libA_func() calling libB_func()\n");
>     libB_func();
> }

So libA_func() calls libB_func(). So any program which calls
libA_func() will only work if libB.dll is available.

> C:\libtest>type prog.c 
> #include <stdio.h>
> #include "libA.h"
> 
> int main(int argc, const char ** argv)
> {
>     printf("prog.exe calling libA_func()\n");
>     libA_func();
>     printf("prog.exe exiting\n");
>     return 0;
> }

So this program requires libB.dll.

> C:\libtest>REM -- erase all of the libB output files to prove that linking 
> 
> C:\libtest>REM -- prog.c doesn't depend on libB. 
> 
> C:\libtest>erase libB.obj libB.dll libB.exp libB.lib  

> C:\libtest>REM -- compile prog.c into prog.exe 
> 
> C:\libtest>cl prog.c libA.lib 
> prog.c
> Microsoft (R) Incremental Linker Version 6.00.8447
> Copyright (C) Microsoft Corp 1992-1998. All rights reserved.
> 
> /out:prog.exe 
> prog.obj 
> libA.lib 
> 
> C:\libtest>REM -- now that prog.exe has built, re-compile and link 
> 
> C:\libtest>REM -- libB so that prog.exe will run 

So, what you are asking is why Linux requires the indirect
dependencies to be present at link time, right?

One fundamental difference between the way in which Linux and Windows
handle shared libraries is that Windows associates external references
(either from a program or from a DLL) with a specific library. E.g. 
the program will have a dependency upon "libA_func in libA.dll", while
libA.dll will have a dependency upon "libB_func in libB.dll".

OTOH, on Linux, the dependency isn't associated with a specific
library. E.g. libA.so just needs libB_func; it doesn't matter where it
comes from. If the program exported that function, you could link
against libA.so without libB.so (unless libA.so listed it as an
explicit dependency).

One consequence of this difference is that Linux allows an executable
to export symbols to a library. Thus you can create a library with
unresolved references (i.e. symbols not provided by either the library
or any of it's explicit dependencies). So long as that symbol actually
gets provided at run-time (e.g. by the executable itself), everything
works.

The fact that Windows disallows this (every import has to be provided
by a specific library, not some yet-to-be-built executable) allows it
to perform the check for a specific library when that library is
built. OTOH, Linux can't perform the check until the final link.

BTW, In most cases, it doesn't actually *need* to do it until
run-time; the run-time linker just loads everything and, so long as
there are no unresolved symbols at that point, everything works. The
one exception is that, unless you use the -E switch, the executable
will only export those symbols which are required by any of the
libraries; consequently, this complete set of libraries must be
available to the linker.

In any case: if you need libB.so to actually run the executable, why
does it matter if you need it in order to link the program?

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[David Wuertele]

Thanks Glynn, for the excellent explanation!

Glynn> So, what you are asking is why Linux requires the indirect
Glynn> dependencies to be present at link time, right?

Exactly.

Glynn> One consequence of this difference is that Linux allows an
Glynn> executable to export symbols to a library. Thus you can create
Glynn> a library with unresolved references (i.e. symbols not provided
Glynn> by either the library or any of it's explicit dependencies). So
Glynn> long as that symbol actually gets provided at run-time (e.g. by
Glynn> the executable itself), everything works.

Does this "flexibility" have practical applications?

Glynn> In any case: if you need libB.so to actually run the
Glynn> executable, why does it matter if you need it in order to link
Glynn> the program?

Encapsulation and portability.  We're writing a toolkit that isolates
the programmer from the underlying implementation.  Requiring the
programmer to explicitly link to the underlying library requires that
the programmer know what that underlying library is.  Also, it
prevents us from changing the underlying library.  These goals don't
seem that unreasonable to us, and they are very important for our
application.

Dave

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[Glynn Clements]

David Wuertele wrote:

> Glynn> One consequence of this difference is that Linux allows an
> Glynn> executable to export symbols to a library. Thus you can create
> Glynn> a library with unresolved references (i.e. symbols not provided
> Glynn> by either the library or any of it's explicit dependencies). So
> Glynn> long as that symbol actually gets provided at run-time (e.g. by
> Glynn> the executable itself), everything works.
> 
> Does this "flexibility" have practical applications?

I have encountered uses of this in the past, but I can't recall any
specifics. Ultimately, you can always achieve the same effect by
having the application register callbacks with the library.

> Glynn> In any case: if you need libB.so to actually run the
> Glynn> executable, why does it matter if you need it in order to link
> Glynn> the program?
> 
> Encapsulation and portability.  We're writing a toolkit that isolates
> the programmer from the underlying implementation.  Requiring the
> programmer to explicitly link to the underlying library requires that
> the programmer know what that underlying library is.  Also, it
> prevents us from changing the underlying library.  These goals don't
> seem that unreasonable to us, and they are very important for our
> application.

The programmer doesn't *explicitly* link to the library; an
application will typically only link against the "topmost" library
(libA in your example), and the resulting executable will only have a
direct dependency (one which is actually stored in the executable)
upon that library.

In order for the link step to succeed, the library which is explicitly
referenced has to exist. If that library has any dependencies, they
must also exist at link time, but the information won't be encoded
into the executable (ldd will show the transitive dependencies, but
they aren't stored in the executable; if you hide the direct
dependencies, the transitive dependencies will disappear from the ldd
output).

E.g. if an application links with -lgtk, libgtk must exist; if the
implementation uses libX11, that must also exist, and so on. However,
the executable will only contain a dependency upon libgtk, not upon
libX11. You could transfer the executable to a system where the libgtk
implementation uses the framebuffer API instead of X11, and it should
still work.

Sometimes shared libraries are built without dependency information. 
In that case, the executable has to explicitly link against the
dependencies, otherwise you would end up with unresolved symbols at
run-time. However, omitting dependency information could be considered
a bug (for your purposes, it would definitely be a bug).

Finally, if you want to be able to build executables which use libA,
but are unable or unwilling to supply a specific libB, you could
supply a "stub" library (one which provides functions which do
nothing). Obviously, such a library would have no dependencies.

-- 
Glynn Clements <glynn.clements@virgin.net>

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[Steven]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

> Glynn> One consequence of this difference is that Linux allows an
> Glynn> executable to export symbols to a library. Thus you can create
> Glynn> a library with unresolved references (i.e. symbols not provided
> Glynn> by either the library or any of it's explicit dependencies). So
> Glynn> long as that symbol actually gets provided at run-time (e.g. by
> Glynn> the executable itself), everything works.
> Does this "flexibility" have practical applications?
If it's combined with weak symbols it's often handy.  Suppose library
A arranges to libA_error whenever it suffers an error, which prints an
error message and terminates the program.  If the libA_error is weak,
then an application can define its own libA_error as a strong symbol,
which will then override the weak variant and be called in preference
to the library's own version.

Systems with multiple, incompatible, threading libraries, such as
FreeBSD, often do something similar with symbols be provided by the
application's chosen threading library.  This means that libA should
be able to work reasonably well whether the application uses libkse or
libthr, without the author of libA having to do anything special.

Steven Smith,
sos22@cam.ac.uk.

[-- Attachment #2: Type: application/pgp-signature, Size: 187 bytes --]

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[David Wuertele]

Mike> Consider what would happen if everything was using static
Mike> libraries instead.

But they aren't.  One feature of shared libraries is runtime binding.

Mike> In my opinion, it should be a consistent command line between
Mike> static and dynamic linking, with adding -static the only
Mike> difference.

That seems like an unnecessarily strict requirement to me.  I don't
see the benefit.

Dave

Re: Why does linking a.so with b.so require execs to be linked both -la and -lb?

[Glynn Clements]

David Wuertele wrote:

> #  I'm creating a shared library like this:
> libDaveA.so: $(LIB_OBJECTS)
> 	$(CXX) -shared -ldl -rdynamic -o $@ $^ -lDaveB -lDaveC -lDaveD -lDaveE
> 
> #  And I have some executables that use the shared lib:
> daveprog: daveprog.cpp
> 	$(CXX) -o $@ $^ -lDaveA
> 
> The executable doesn't use any of the functions from libDaveB.so,
> libDaveC.so, libDaveD.so, or libDaveE.so.  It only uses functions from
> libDaveA.so.  But when I compile with the above makefile, gcc
> complains:
> 
>   /usr/tools/lib/gcc-lib/mipsel-linux/2.96/../../../../mipsel-linux/bin/ld: warning: libDaveB.so.0, needed by /usr/local/lib/libDaveA.so, not found (try using -rpath or -rpath-link)
> 
> Why is that?  Why should daveprog, which never uses any functionality
> AT ALL from libDaveB.so, have to link to it?

Because libDaveB is listed as a dependency of libDaveA.

Does libDaveA really require all of those other libraries? If not,
don't list them as dependencies. OTOH, if something in libDaveA
requires libDaveB, anything which requires libDaveA will also require
libDaveB.

Dependencies apply to the library as a whole, not to individual
functions. Even if the program only used functions which don't
themselves require libDaveB, the linker has no way of knowing that.

-- 
Glynn Clements <glynn.clements@virgin.net>