Storing cls and erasure code plugins in a pool

Storing cls and erasure code plugins in a pool

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

Hi Ceph,

There is a need for a cluster to share code such as cls https://github.com/ceph/ceph/tree/master/src/cls or erasure code plugins https://github.com/ceph/ceph/tree/master/src/erasure-code/.

These plugins could have a life cycle independent of Ceph, as long as they comply to the supported API ( https://github.com/ceph/ceph/blob/master/src/erasure-code/ErasureCodeInterface.h ). For erasure code plugins it currently works this way (or it will as soon as https://github.com/ceph/ceph/pull/2397 is merged):

a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
b) the MON will refuse to create an erasure coded pool using the new I* plugins, otherwise the Hammer nodes will find themselves unable to participate in the pool

Instead it could work this way:

a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
b) the new erasure code plugins are uploaded to a "plugins" pool
c) an erasure coded pool is created using a new plugin from I*
d) the Hammer OSD downloads the plugin from the pool and can participate in the pool

It is easier said than done and there are a lot of details to consider. However it not different from maintaining an Operating System that includes shared libraries and the path to do so properly is well known.

Thoughts ?

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Storing cls and erasure code plugins in a pool

[Milosz Tanski]

If you're planning on having plugins that are not shipped with the
host software you have to worry about both API and ABI stability.
Traditionally (and in my personal experience) keeping a C++ ABI
compatible is hard. For those reasons, I would strongly campaign for
having the plugins interface be in C (or et least their interface
would be C linkage).

Otherwise here's some things you can read read about C++ ABI stability
(from the KDE people):
https://techbase.kde.org/Policies/Binary_Compatibility_Issues_With_C++

This are my 2 cents based on my past experience.

On Sun, Sep 7, 2014 at 4:26 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi Ceph,
>
> There is a need for a cluster to share code such as cls https://github.com/ceph/ceph/tree/master/src/cls or erasure code plugins https://github.com/ceph/ceph/tree/master/src/erasure-code/.
>
> These plugins could have a life cycle independent of Ceph, as long as they comply to the supported API ( https://github.com/ceph/ceph/blob/master/src/erasure-code/ErasureCodeInterface.h ). For erasure code plugins it currently works this way (or it will as soon as https://github.com/ceph/ceph/pull/2397 is merged):
>
> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
> b) the MON will refuse to create an erasure coded pool using the new I* plugins, otherwise the Hammer nodes will find themselves unable to participate in the pool
>
> Instead it could work this way:
>
> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
> b) the new erasure code plugins are uploaded to a "plugins" pool
> c) an erasure coded pool is created using a new plugin from I*
> d) the Hammer OSD downloads the plugin from the pool and can participate in the pool
>
> It is easier said than done and there are a lot of details to consider. However it not different from maintaining an Operating System that includes shared libraries and the path to do so properly is well known.
>
> Thoughts ?
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>



-- 
Milosz Tanski
CTO
16 East 34th Street, 15th floor
New York, NY 10016

p: 646-253-9055
e: milosz@adfin.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Storing cls and erasure code plugins in a pool

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]



On 07/09/2014 15:38, Milosz Tanski wrote:
> If you're planning on having plugins that are not shipped with the
> host software you have to worry about both API and ABI stability.
> Traditionally (and in my personal experience) keeping a C++ ABI
> compatible is hard. For those reasons, I would strongly campaign for
> having the plugins interface be in C (or et least their interface
> would be C linkage).

Hi Milosz,

That's a good point indeed.

> Otherwise here's some things you can read read about C++ ABI stability
> (from the KDE people):
> https://techbase.kde.org/Policies/Binary_Compatibility_Issues_With_C++

Thanks, I did not even think C++ ABI compatibility was possible at all ;-) The web site is down at the moment but I'll take a look.

Cheers

> 
> This are my 2 cents based on my past experience.
> 
> On Sun, Sep 7, 2014 at 4:26 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Ceph,
>>
>> There is a need for a cluster to share code such as cls https://github.com/ceph/ceph/tree/master/src/cls or erasure code plugins https://github.com/ceph/ceph/tree/master/src/erasure-code/.
>>
>> These plugins could have a life cycle independent of Ceph, as long as they comply to the supported API ( https://github.com/ceph/ceph/blob/master/src/erasure-code/ErasureCodeInterface.h ). For erasure code plugins it currently works this way (or it will as soon as https://github.com/ceph/ceph/pull/2397 is merged):
>>
>> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
>> b) the MON will refuse to create an erasure coded pool using the new I* plugins, otherwise the Hammer nodes will find themselves unable to participate in the pool
>>
>> Instead it could work this way:
>>
>> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
>> b) the new erasure code plugins are uploaded to a "plugins" pool
>> c) an erasure coded pool is created using a new plugin from I*
>> d) the Hammer OSD downloads the plugin from the pool and can participate in the pool
>>
>> It is easier said than done and there are a lot of details to consider. However it not different from maintaining an Operating System that includes shared libraries and the path to do so properly is well known.
>>
>> Thoughts ?
>>
>> Cheers
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
>>
> 
> 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Storing cls and erasure code plugins in a pool

[Yehuda Sadeh]

On Sun, Sep 7, 2014 at 1:26 AM, Loic Dachary <loic@dachary.org> wrote:
> Hi Ceph,
>
> There is a need for a cluster to share code such as cls https://github.com/ceph/ceph/tree/master/src/cls or erasure code plugins https://github.com/ceph/ceph/tree/master/src/erasure-code/.
>
> These plugins could have a life cycle independent of Ceph, as long as they comply to the supported API ( https://github.com/ceph/ceph/blob/master/src/erasure-code/ErasureCodeInterface.h ). For erasure code plugins it currently works this way (or it will as soon as https://github.com/ceph/ceph/pull/2397 is merged):
>
> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
> b) the MON will refuse to create an erasure coded pool using the new I* plugins, otherwise the Hammer nodes will find themselves unable to participate in the pool
>
> Instead it could work this way:
>
> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
> b) the new erasure code plugins are uploaded to a "plugins" pool
> c) an erasure coded pool is created using a new plugin from I*
> d) the Hammer OSD downloads the plugin from the pool and can participate in the pool
>
> It is easier said than done and there are a lot of details to consider. However it not different from maintaining an Operating System that includes shared libraries and the path to do so properly is well known.
>
> Thoughts ?

And here we go (almost) full circle. Originally the objclass (cls)
mechanism worked somewhat similar. The code would be injected to the
monitors, and it was then distributed to the osds. When uploading the
objects we'd also specify the architecture, and there's an embedded
version in each so that it was possible to disable one version and
enable another.
The problem is that this specific method is very problematic when
dealing with heterogeneous environments, where each osd can run on a
different architecture, or different distribution. Also need to
maintain a very well contained api for the objclasses to use (which we
don't have), and be very careful about versioning. In other words, it
doesn't work, and the trouble doesn't worth the benefits.
I'm not too familiar with the erasure code plugins, but it seems to me
that they need to be versioned. Then you could have multiple plugins
with different versions installed, and then you could specify which
version to use. You could have a tool that would make sure that all
the osds have access to the appropriate plugin version. But the actual
installation of the plugin wouldn't be part of ceph's internal task.
It might be that the erasure code plugins are more contained than the
objclasses, and something like you suggested might actually work.
Though I'm having trouble seeing that happening having a compiled
shared object as the resource that needs to be distributed. The first
objclass implementation actually pushed python code to the nodes (it
really did!), maybe having something like that for erasure code could
work, given the appropriate environment and tools.

Yehuda

Re: Storing cls and erasure code plugins in a pool

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 3992 bytes --]

Hi Yehuda,

You are right : erasure code plugins must obey the same constraints as objclass plugins, only worse because once a plugin starts being used it must remain available for as long as the pool needs it to read/write chunks. 

Maintaining a repository of binary plugins is indeed not trivial. I just meant to write that it is not much different from apt-get installing shared libraries from architecture dependant repositories. We do not have to invent something, we can mimic and adapt existing best practices.

Maybe this idea is over engineering and there is a better / simpler solution to deal with erasure code plugin upgrades ? Or objclass upgrades for that matter.

Cheers

On 07/09/2014 21:42, Yehuda Sadeh wrote:
> On Sun, Sep 7, 2014 at 1:26 AM, Loic Dachary <loic@dachary.org> wrote:
>> Hi Ceph,
>>
>> There is a need for a cluster to share code such as cls https://github.com/ceph/ceph/tree/master/src/cls or erasure code plugins https://github.com/ceph/ceph/tree/master/src/erasure-code/.
>>
>> These plugins could have a life cycle independent of Ceph, as long as they comply to the supported API ( https://github.com/ceph/ceph/blob/master/src/erasure-code/ErasureCodeInterface.h ). For erasure code plugins it currently works this way (or it will as soon as https://github.com/ceph/ceph/pull/2397 is merged):
>>
>> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
>> b) the MON will refuse to create an erasure coded pool using the new I* plugins, otherwise the Hammer nodes will find themselves unable to participate in the pool
>>
>> Instead it could work this way:
>>
>> a) upgrade from Hammer to I* half the OSD nodes. The new I* have new erasure code plugins
>> b) the new erasure code plugins are uploaded to a "plugins" pool
>> c) an erasure coded pool is created using a new plugin from I*
>> d) the Hammer OSD downloads the plugin from the pool and can participate in the pool
>>
>> It is easier said than done and there are a lot of details to consider. However it not different from maintaining an Operating System that includes shared libraries and the path to do so properly is well known.
>>
>> Thoughts ?
> 
> And here we go (almost) full circle. Originally the objclass (cls)
> mechanism worked somewhat similar. The code would be injected to the
> monitors, and it was then distributed to the osds. When uploading the
> objects we'd also specify the architecture, and there's an embedded
> version in each so that it was possible to disable one version and
> enable another.
> The problem is that this specific method is very problematic when
> dealing with heterogeneous environments, where each osd can run on a
> different architecture, or different distribution. Also need to
> maintain a very well contained api for the objclasses to use (which we
> don't have), and be very careful about versioning. In other words, it
> doesn't work, and the trouble doesn't worth the benefits.
> I'm not too familiar with the erasure code plugins, but it seems to me
> that they need to be versioned. Then you could have multiple plugins
> with different versions installed, and then you could specify which
> version to use. You could have a tool that would make sure that all
> the osds have access to the appropriate plugin version. But the actual
> installation of the plugin wouldn't be part of ceph's internal task.
> It might be that the erasure code plugins are more contained than the
> objclasses, and something like you suggested might actually work.
> Though I'm having trouble seeing that happening having a compiled
> shared object as the resource that needs to be distributed. The first
> objclass implementation actually pushed python code to the nodes (it
> really did!), maybe having something like that for erasure code could
> work, given the appropriate environment and tools.
> 
> Yehuda
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]