Re: LTTng unfriendly with mixed static/dynamic linking

[Joao Eduardo Luis <joao.luis@inktank.com>]

On 07/25/2014 11:12 PM, Sage Weil wrote:
> On Fri, 25 Jul 2014, Adam Crume wrote:
>> I tried all solutions, and it looks like only #1 works.  #2 gives the
>> error "/usr/bin/ld: main: hidden symbol `tracepoint_dlopen' in
>> common_tp.a(common_tp.o) is referenced by DSO" when linking.  #3 gives
>> the error "./liblibrary.so: undefined reference to
>> `tracepoint_dlopen'" when linking.  (Linking is complicated by the
>> fact that LTTng uses special symbol attributes, and tracepoint_dlopen
>> happens to be weak and hidden.)
>
> I think #1 is good for other reasons, too.  We already have issues (I
> think!) with binaries that use librados and also link libcommon
> statically.  Specifically, I think we've seen that having mismatched
> versions of librados and the binary installed lead to confusion about the
> contents/structure of mdconfig_t (g_conf).  This is one of the reasons
> why the libcommon and rgw packages require an identical version of
> librados or librbd or whatever--to avoid this inconsistency.
>
>> Unless I'm mistaken (and I very well
>> may be), we will have to ensure that all traced code is either 1)
>> placed in a shared library and never statically linked elsewhere, or
>> 2) never linked into any shared library.
>
> That sounds doable and sane to me:
>
>   - librados, librbd, libceph_common, etc. would have the tracepoints in
> the same .so
>   - ceph-osd could have its own tracepoints, as long as they are always
> static.  (For example, libos.la, which is linked statically by ceph-mon
> and ceph-osd but never dynamically.)
>
> One pain point in all of this, though, is that the libceph_common.so (or
> whatever) will need to go into a separate package that is required by
> librados.so and librbd and ceph-common and everything else.  'ceph-common'
> is what this ought to be called, but we've coopted it to mean 'ceph
> clients'.  I'm not sure it if it worthwhile to go through the hinjinx to
> rename ceph-common to ceph-clients and repurpose ceph-common for this?
>
> sage

I notice that ceph-common contains no libs whatsoever.  We may want to 
change ceph-common to ceph-client or something and have libcommon 
shipped as ceph-common, but I imagine that would be a pain as package 
management goes.  Or we could take the path of least resistance (and 
possibly open ourselves to confusion?) and ship libcommon in a 
'ceph-libs' package -- although it looks like it would be a 1-lib package :)

   -Joao

>
>
>
>>
>> Thoughts?
>>
>> Adam
>>
>> On Fri, Jul 25, 2014 at 11:48 AM, Adam Crume <adamcrume@gmail.com> wrote:
>>> LTTng requires tracepoints to be linked into a program only once.  If
>>> tracepoints are linked in multiple times, the program crashes at
>>> startup with: "LTTng-UST: Error (-17) while registering tracepoint
>>> probe. Duplicate registration of tracepoint probes having the same
>>> name is not allowed."
>>>
>>> This is problematic when mixing static and dynamic linking.  If the
>>> tracepoints are in a static library, that library can end up in an
>>> executable multiple times by being linked in directly, as well as
>>> being statically linked into a dynamic library.  Even if the
>>> tracepoints are not linked directly into the executable, they can be
>>> statically linked into multiple dynamic libraries that the executable
>>> loads.
>>>
>>> For us, this problem shows up with libcommon, and could show up with
>>> others such as libosd.  (In general, I think anything added to
>>> noinst_LTLIBRARIES is static, and anything added to lib_LTLIBRARIES is
>>> dynamic.)
>>>
>>> There are a few ways of solving the issue:
>>> 1. Change every library that has tracepoints, like libcommon, from
>>> static to dynamic.  This could be a big change, as at the very least
>>> we'd have to rename the library to something like libceph_common to
>>> avoid conflicts, since now it would be an installed file.  This has
>>> the advantage of keeping code and its tracepoints in the same library.
>>> 2. Keep tracepoints in a separate static library.  For example,
>>> libcommon and libcommon_tp.  Unfortunately, every executable (but not
>>> library!) that links in libcommon (directly or indirectly) would have
>>> to manually link in libcommon_tp, and I don't think Automake could
>>> automate that.
>>> 3. Keep tracepoints in a separate dynamic library.  In this case, I
>>> think libcommon could depend on libcommon_tp, so executables would not
>>> have to manually link in libcommon_tp.  (I'm not an Automake expert,
>>> so let me know if I'm wrong on that.)  Again, libcommon_tp would be an
>>> installed file, so we'd want to rename it to something like
>>> libceph_common_tp.
>>>
>>> I attached a minimal test case of the problem.
>>>
>>> Thoughts or suggestions?
>>>
>>> Thanks,
>>> Adam Crume
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com