jerasure packaging and Ceph

jerasure packaging and Ceph

[Ken Dreyer]

Hi Loic,

I was looking through Ceph's bundled libraries recently and I was
wondering why Ceph bundles its own copy of jerasure.

Could you give some background on that? Why don't we link to an separate
system package?

- Ken

Re: jerasure packaging and Ceph

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1910 bytes --]

Hi Ken,

On 07/08/2015 19:25, Ken Dreyer wrote:
> Hi Loic,
> 
> I was looking through Ceph's bundled libraries recently and I was
> wondering why Ceph bundles its own copy of jerasure.
> 
> Could you give some background on that? Why don't we link to an separate
> system package?

Mainly because there is no proper non regression testing of the packages found in the distributions. It is absolutely critical for Ceph to ensure there is no regression because it would mean data loss. The packagers do not have that concern in mind right now, nor do they have the infrastructure to run non regression tests, to the best of my knowledge.

Even if they had non regression tests, whenever a new package is published, we would need to run Ceph integration tests before it lands in the distribution repositories to ensure that everything is fine from the Ceph perspective. The recent work with teuthology and OpenStack simplified this quite a lot and anyone can run teuthology now. However the level of coordination it would require between the jerasure packager and the ceph packager is far from what is going on currently.

I offered to package jerasure for Debian to solve that problem in the Debian / Ubuntu realm. I thought a first step to decouple ceph from jerasure could be that I care for jerasure because I have access to the test infrastructure and I understand what Ceph needs. And I could gradually make it possible for any packager to do the same, somehow (I have no idea how to do that, honestly). Unfortunately the person responsible for packaging jerasure did not respond favorably to my offer. Nor does he plan to implement integration or non regression tests.

Hopefully that will change in the future, but for now I think bundling jerasure with Ceph is the best way to preserve the data of our users.

Cheers

> 
> - Ken
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: jerasure packaging and Ceph

[Ken Dreyer]

On 08/07/2015 01:27 PM, Loic Dachary wrote:
> Hi Ken,
> 
> On 07/08/2015 19:25, Ken Dreyer wrote:
>> Hi Loic,
>>
>> I was looking through Ceph's bundled libraries recently and I was
>> wondering why Ceph bundles its own copy of jerasure.
>>
>> Could you give some background on that? Why don't we link to an separate
>> system package?
> 
> Mainly because there is no proper non regression testing of the packages found in the distributions. It is absolutely critical for Ceph to ensure there is no regression because it would mean data loss. The packagers do not have that concern in mind right now, nor do they have the infrastructure to run non regression tests, to the best of my knowledge.
> 
> Even if they had non regression tests, whenever a new package is published, we would need to run Ceph integration tests before it lands in the distribution repositories to ensure that everything is fine from the Ceph perspective. The recent work with teuthology and OpenStack simplified this quite a lot and anyone can run teuthology now. However the level of coordination it would require between the jerasure packager and the ceph packager is far from what is going on currently.
> 
> I offered to package jerasure for Debian to solve that problem in the Debian / Ubuntu realm. I thought a first step to decouple ceph from jerasure could be that I care for jerasure because I have access to the test infrastructure and I understand what Ceph needs. And I could gradually make it possible for any packager to do the same, somehow (I have no idea how to do that, honestly). Unfortunately the person responsible for packaging jerasure did not respond favorably to my offer. Nor does he plan to implement integration or non regression tests.
> 
> Hopefully that will change in the future, but for now I think bundling jerasure with Ceph is the best way to preserve the data of our users.
> 

Hi Loic,

Thank you, that helps me understand the context a lot more.

It sounds like you're concerned that Debian might change their package
behind Ceph's back, and it could have implications for the way Ceph
stores its' data.

I guess at some level it's true that all of Ceph's dependencies could
impact its behavior.

I would like it if Teuthology ran with the "epel-testing" repository
enabled, for example, so we could catch these sort of problems before
they went into the main "epel" repo that most users consume. That
certainly would have caught some issues in the past with the conflicts
between EPEL and Firefly.

Is there some equivalent in Debian?

- Ken

Re: jerasure packaging and Ceph

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 4200 bytes --]



On 07/08/2015 22:09, Ken Dreyer wrote:
> On 08/07/2015 01:27 PM, Loic Dachary wrote:
>> Hi Ken,
>>
>> On 07/08/2015 19:25, Ken Dreyer wrote:
>>> Hi Loic,
>>>
>>> I was looking through Ceph's bundled libraries recently and I was
>>> wondering why Ceph bundles its own copy of jerasure.
>>>
>>> Could you give some background on that? Why don't we link to an separate
>>> system package?
>>
>> Mainly because there is no proper non regression testing of the packages found in the distributions. It is absolutely critical for Ceph to ensure there is no regression because it would mean data loss. The packagers do not have that concern in mind right now, nor do they have the infrastructure to run non regression tests, to the best of my knowledge.
>>
>> Even if they had non regression tests, whenever a new package is published, we would need to run Ceph integration tests before it lands in the distribution repositories to ensure that everything is fine from the Ceph perspective. The recent work with teuthology and OpenStack simplified this quite a lot and anyone can run teuthology now. However the level of coordination it would require between the jerasure packager and the ceph packager is far from what is going on currently.
>>
>> I offered to package jerasure for Debian to solve that problem in the Debian / Ubuntu realm. I thought a first step to decouple ceph from jerasure could be that I care for jerasure because I have access to the test infrastructure and I understand what Ceph needs. And I could gradually make it possible for any packager to do the same, somehow (I have no idea how to do that, honestly). Unfortunately the person responsible for packaging jerasure did not respond favorably to my offer. Nor does he plan to implement integration or non regression tests.
>>
>> Hopefully that will change in the future, but for now I think bundling jerasure with Ceph is the best way to preserve the data of our users.
>>
> 
> Hi Loic,
> 
> Thank you, that helps me understand the context a lot more.
> 
> It sounds like you're concerned that Debian might change their package
> behind Ceph's back, and it could have implications for the way Ceph
> stores its' data.

Yes. Although "behind Ceph's back" suggest malicious behavior but I don't think it's going to be malicious. It's just the way packaging is done currently: integration testing with dependencies of a library that are official debian packages is not mandatory, it is left at the discretion of the packager.

> I guess at some level it's true that all of Ceph's dependencies could
> impact its behavior.

Yes. I guess I would be less concerned if it was not as critical as data corruption and if there were many users and packages depending on jerasure. In my opinion the benefit of using a jerasure package instead of a version bundled in Ceph is not worth the risk. A lot of work must be done to introduce integration testing in distributions and lower that risk. I'm hopeful this will happen eventually.

> I would like it if Teuthology ran with the "epel-testing" repository
> enabled, for example, so we could catch these sort of problems before
> they went into the main "epel" repo that most users consume. That
> certainly would have caught some issues in the past with the conflicts
> between EPEL and Firefly.
> 
> Is there some equivalent in Debian?

There is Debian experimental and it could be used for that purpose. The main problem being to find the resources to analyze and fix the problems found in epel-testing and debian experimental. Does that work involve fall on the Ceph developers at large ? If so does that mean we collectively agree to add these declared "unstable" repositories in the list of officially supported repositories ?

It's a discussion I would love to have and, fortunately, we now have the technical means to claim "you can run teuthology easily and cheaply". There is no technical blockers and we can dream about building a community to expand the supported distributions. And make it so packaging is always associated with integration tests.

Cheers


> 
> - Ken
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]