turn libcommon into a shared library and package it

turn libcommon into a shared library and package it

[kefu chai]

hi cephers,

libcommon is a convenient library and is statically linked into
librados, librbd and libcephfs. and there are some static variables
living in libcommon, for instance, the ones defined in lockdep.cc. so,
we will have multiple copies of these global variables if an
application is linked against librados and (librbd | libcephfs).  the
"ceph" cli does link against librados and libcephfs via the their
python bindings.

this incurs some problems:

- cross reference each other's global variables: in my testbed, i ran
into https://github.com/ceph/ceph/pull/12249#issuecomment-264105597.
where libcephfs was referencing the lockdeps variables in librados
when relinquishing a lock. when libcephfs tries to acquire this lock
again, it found that this lock is *already* locked.
- more memory foot print and disk space: apparently, libcommon is
included by librados, librbd and libcephfs. so it's a waste to have
multiple copies of libcommon if librbd and/or libcephfs are loaded
into memory or installed onto disk.

after turning libcommon into a shared library, these problems are solved.

but yes, unlike librados and its friends, libcommon is not a user
facing library, so we will not install it right under /usr/lib/.
instead, it will be living in /usr/lib/ceph.  please note, the erasure
plugins are installed into /usr/lib/ceph/erasure-code/ and rados
classes are installed into /usr/lib/rados-classes/. this follows the
related section of FHS [1].

what do you think?

--
[1] http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA

-- 
Regards
Kefu Chai

Re: turn libcommon into a shared library and package it

[Ricardo Dias]

On 06-01-2017 11:38, kefu chai wrote:
> hi cephers,
>
> libcommon is a convenient library and is statically linked into
> librados, librbd and libcephfs. and there are some static variables
> living in libcommon, for instance, the ones defined in lockdep.cc. so,
> we will have multiple copies of these global variables if an
> application is linked against librados and (librbd | libcephfs).  the
> "ceph" cli does link against librados and libcephfs via the their
> python bindings.
>
> this incurs some problems:
>
> - cross reference each other's global variables: in my testbed, i ran
> into https://github.com/ceph/ceph/pull/12249#issuecomment-264105597.
> where libcephfs was referencing the lockdeps variables in librados
> when relinquishing a lock. when libcephfs tries to acquire this lock
> again, it found that this lock is *already* locked.
> - more memory foot print and disk space: apparently, libcommon is
> included by librados, librbd and libcephfs. so it's a waste to have
> multiple copies of libcommon if librbd and/or libcephfs are loaded
> into memory or installed onto disk.
>
> after turning libcommon into a shared library, these problems are solved.

I agree with this approach. In RBD we also had the same problems 
sometime ago, and we discussed the problem here
https://github.com/ceph/ceph/pull/8537

>
> but yes, unlike librados and its friends, libcommon is not a user
> facing library, so we will not install it right under /usr/lib/.
> instead, it will be living in /usr/lib/ceph.  please note, the erasure
> plugins are installed into /usr/lib/ceph/erasure-code/ and rados
> classes are installed into /usr/lib/rados-classes/. this follows the
> related section of FHS [1].
>
> what do you think?

+1 here. Please go forward with this :-)

>
> --
> [1] http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA
>

-- 
Ricardo Dias
Senior Software Engineer - Storage Team
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, 
HRB 21284 (AG Nürnberg)

Re: turn libcommon into a shared library and package it

[Jeff Layton]

On Fri, 2017-01-06 at 11:48 +0000, Ricardo Dias wrote:
> 
> On 06-01-2017 11:38, kefu chai wrote:
> > 
> > hi cephers,
> > 
> > libcommon is a convenient library and is statically linked into
> > librados, librbd and libcephfs. and there are some static variables
> > living in libcommon, for instance, the ones defined in lockdep.cc. so,
> > we will have multiple copies of these global variables if an
> > application is linked against librados and (librbd | libcephfs).  the
> > "ceph" cli does link against librados and libcephfs via the their
> > python bindings.
> > 
> > this incurs some problems:
> > 
> > - cross reference each other's global variables: in my testbed, i ran
> > into https://github.com/ceph/ceph/pull/12249#issuecomment-264105597.
> > where libcephfs was referencing the lockdeps variables in librados
> > when relinquishing a lock. when libcephfs tries to acquire this lock
> > again, it found that this lock is *already* locked.
> > - more memory foot print and disk space: apparently, libcommon is
> > included by librados, librbd and libcephfs. so it's a waste to have
> > multiple copies of libcommon if librbd and/or libcephfs are loaded
> > into memory or installed onto disk.
> > 
> > after turning libcommon into a shared library, these problems are solved.
> 
> I agree with this approach. In RBD we also had the same problems 
> sometime ago, and we discussed the problem here
> https://github.com/ceph/ceph/pull/8537
> 
> > 
> > 
> > but yes, unlike librados and its friends, libcommon is not a user
> > facing library, so we will not install it right under /usr/lib/.
> > instead, it will be living in /usr/lib/ceph.  please note, the erasure
> > plugins are installed into /usr/lib/ceph/erasure-code/ and rados
> > classes are installed into /usr/lib/rados-classes/. this follows the
> > related section of FHS [1].
> > 
> > what do you think?
> 
> +1 here. Please go forward with this :-)
> 
> > 
> > 
> > --
> > [1] http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA
> > 
> 

Sounds good to me.

It might also not hurt to rename it. libcommon as a name is awfully
generic and could run afoul of other libs on the machine that have
nothing to do with ceph (regardless of whether you install it in this
private location). Maybe libceph-common?

-- 
Jeff Layton <jlayton@redhat.com>

Re: turn libcommon into a shared library and package it

[kefu chai]

thanks guys. will rename it to libceph-common and package it in librados.

On Fri, Jan 6, 2017 at 8:20 PM, Jeff Layton <jlayton@redhat.com> wrote:
> On Fri, 2017-01-06 at 11:48 +0000, Ricardo Dias wrote:
>>
>> On 06-01-2017 11:38, kefu chai wrote:
>> >
>> > hi cephers,
>> >
>> > libcommon is a convenient library and is statically linked into
>> > librados, librbd and libcephfs. and there are some static variables
>> > living in libcommon, for instance, the ones defined in lockdep.cc. so,
>> > we will have multiple copies of these global variables if an
>> > application is linked against librados and (librbd | libcephfs).  the
>> > "ceph" cli does link against librados and libcephfs via the their
>> > python bindings.
>> >
>> > this incurs some problems:
>> >
>> > - cross reference each other's global variables: in my testbed, i ran
>> > into https://github.com/ceph/ceph/pull/12249#issuecomment-264105597.
>> > where libcephfs was referencing the lockdeps variables in librados
>> > when relinquishing a lock. when libcephfs tries to acquire this lock
>> > again, it found that this lock is *already* locked.
>> > - more memory foot print and disk space: apparently, libcommon is
>> > included by librados, librbd and libcephfs. so it's a waste to have
>> > multiple copies of libcommon if librbd and/or libcephfs are loaded
>> > into memory or installed onto disk.
>> >
>> > after turning libcommon into a shared library, these problems are solved.
>>
>> I agree with this approach. In RBD we also had the same problems
>> sometime ago, and we discussed the problem here
>> https://github.com/ceph/ceph/pull/8537
>>
>> >
>> >
>> > but yes, unlike librados and its friends, libcommon is not a user
>> > facing library, so we will not install it right under /usr/lib/.
>> > instead, it will be living in /usr/lib/ceph.  please note, the erasure
>> > plugins are installed into /usr/lib/ceph/erasure-code/ and rados
>> > classes are installed into /usr/lib/rados-classes/. this follows the
>> > related section of FHS [1].
>> >
>> > what do you think?
>>
>> +1 here. Please go forward with this :-)
>>
>> >
>> >
>> > --
>> > [1] http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA
>> >
>>
>
> Sounds good to me.
>
> It might also not hurt to rename it. libcommon as a name is awfully
> generic and could run afoul of other libs on the machine that have
> nothing to do with ceph (regardless of whether you install it in this
> private location). Maybe libceph-common?
>
> --
> Jeff Layton <jlayton@redhat.com>



-- 
Regards
Kefu Chai

Re: turn libcommon into a shared library and package it

[John Spray]

On Fri, Jan 6, 2017 at 11:38 AM, kefu chai <tchaikov@gmail.com> wrote:
> hi cephers,
>
> libcommon is a convenient library and is statically linked into
> librados, librbd and libcephfs. and there are some static variables
> living in libcommon, for instance, the ones defined in lockdep.cc. so,
> we will have multiple copies of these global variables if an
> application is linked against librados and (librbd | libcephfs).  the
> "ceph" cli does link against librados and libcephfs via the their
> python bindings.
>
> this incurs some problems:
>
> - cross reference each other's global variables: in my testbed, i ran
> into https://github.com/ceph/ceph/pull/12249#issuecomment-264105597.
> where libcephfs was referencing the lockdeps variables in librados
> when relinquishing a lock. when libcephfs tries to acquire this lock
> again, it found that this lock is *already* locked.
> - more memory foot print and disk space: apparently, libcommon is
> included by librados, librbd and libcephfs. so it's a waste to have
> multiple copies of libcommon if librbd and/or libcephfs are loaded
> into memory or installed onto disk.
>
> after turning libcommon into a shared library, these problems are solved.
>
> but yes, unlike librados and its friends, libcommon is not a user
> facing library, so we will not install it right under /usr/lib/.
> instead, it will be living in /usr/lib/ceph.  please note, the erasure
> plugins are installed into /usr/lib/ceph/erasure-code/ and rados
> classes are installed into /usr/lib/rados-classes/. this follows the
> related section of FHS [1].
>
> what do you think?

+1

John

>
> --
> [1] http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA
>
> --
> Regards
> Kefu Chai
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: turn libcommon into a shared library and package it

[Brett Niver]

+1

On Fri, Jan 6, 2017 at 7:36 AM, John Spray <jspray@redhat.com> wrote:
> On Fri, Jan 6, 2017 at 11:38 AM, kefu chai <tchaikov@gmail.com> wrote:
>> hi cephers,
>>
>> libcommon is a convenient library and is statically linked into
>> librados, librbd and libcephfs. and there are some static variables
>> living in libcommon, for instance, the ones defined in lockdep.cc. so,
>> we will have multiple copies of these global variables if an
>> application is linked against librados and (librbd | libcephfs).  the
>> "ceph" cli does link against librados and libcephfs via the their
>> python bindings.
>>
>> this incurs some problems:
>>
>> - cross reference each other's global variables: in my testbed, i ran
>> into https://github.com/ceph/ceph/pull/12249#issuecomment-264105597.
>> where libcephfs was referencing the lockdeps variables in librados
>> when relinquishing a lock. when libcephfs tries to acquire this lock
>> again, it found that this lock is *already* locked.
>> - more memory foot print and disk space: apparently, libcommon is
>> included by librados, librbd and libcephfs. so it's a waste to have
>> multiple copies of libcommon if librbd and/or libcephfs are loaded
>> into memory or installed onto disk.
>>
>> after turning libcommon into a shared library, these problems are solved.
>>
>> but yes, unlike librados and its friends, libcommon is not a user
>> facing library, so we will not install it right under /usr/lib/.
>> instead, it will be living in /usr/lib/ceph.  please note, the erasure
>> plugins are installed into /usr/lib/ceph/erasure-code/ and rados
>> classes are installed into /usr/lib/rados-classes/. this follows the
>> related section of FHS [1].
>>
>> what do you think?
>
> +1
>
> John
>
>>
>> --
>> [1] http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA
>>
>> --
>> Regards
>> Kefu Chai
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html