civetweb upstream/downstream divergence

civetweb upstream/downstream divergence

[Nathan Cutler]

Hi Ceph:

The civetweb code in RGW is taken from https://github.com/ceph/civetweb/
which is a fork of https://github.com/civetweb/civetweb. The last commit
to our fork took place on March 18.

Upstream civetweb development has progressed ("This branch is 19 commits
ahead, 972 commits behind civetweb:master.")

Are there plans to rebase to a newer upstream version or should we think
more in terms of backporting (to ceph/civetweb.git) from upstream
(civetweb/civetweb.git) when we need to fix bugs or add features?

Thanks and regards

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037

Re: civetweb upstream/downstream divergence

[Wido den Hollander]

On 29-10-15 10:19, Nathan Cutler wrote:
> Hi Ceph:
> 
> The civetweb code in RGW is taken from https://github.com/ceph/civetweb/
> which is a fork of https://github.com/civetweb/civetweb. The last commit
> to our fork took place on March 18.
> 
> Upstream civetweb development has progressed ("This branch is 19 commits
> ahead, 972 commits behind civetweb:master.")
> 
> Are there plans to rebase to a newer upstream version or should we think
> more in terms of backporting (to ceph/civetweb.git) from upstream
> (civetweb/civetweb.git) when we need to fix bugs or add features?
> 

I think it would be smart to keep tracking civetweb from upstream
otherwise we forked Civetweb.

We might run into some issues with Civetweb which we need to fix
upstream, that's a lot easier if we are close to where upstream is.

Wido

> Thanks and regards
> 

Re: civetweb upstream/downstream divergence

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

Hi Nathan,

It would be a good thing to get a report on a regular basis that shows how different the git submodules are from their respective upstream.

Cheers

On 29/10/2015 18:19, Nathan Cutler wrote:
> Hi Ceph:
> 
> The civetweb code in RGW is taken from https://github.com/ceph/civetweb/
> which is a fork of https://github.com/civetweb/civetweb. The last commit
> to our fork took place on March 18.
> 
> Upstream civetweb development has progressed ("This branch is 19 commits
> ahead, 972 commits behind civetweb:master.")
> 
> Are there plans to rebase to a newer upstream version or should we think
> more in terms of backporting (to ceph/civetweb.git) from upstream
> (civetweb/civetweb.git) when we need to fix bugs or add features?
> 
> Thanks and regards
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: civetweb upstream/downstream divergence

[Yehuda Sadeh-Weinraub]

On Thu, Oct 29, 2015 at 2:19 AM, Nathan Cutler <ncutler@suse.cz> wrote:
> Hi Ceph:
>
> The civetweb code in RGW is taken from https://github.com/ceph/civetweb/
> which is a fork of https://github.com/civetweb/civetweb. The last commit
> to our fork took place on March 18.
>
> Upstream civetweb development has progressed ("This branch is 19 commits
> ahead, 972 commits behind civetweb:master.")
>
> Are there plans to rebase to a newer upstream version or should we think
> more in terms of backporting (to ceph/civetweb.git) from upstream
> (civetweb/civetweb.git) when we need to fix bugs or add features?
>
> Thanks and regards
>

We should definitely do it. We're based off civetweb 1.6, and there
was no official civetweb version for quite a while, but 1.7 was tagged
a few months ago. I made some effort and got most of our material
changes upstream, however, there are some changes that might need some
more work before we can get them merged, or might not make complete
sense at all. iirc, the biggest change that would be challenging to
get upstream is the chunked encoding handling that we have. Other than
that there are a few minor changes that we did to make rgw with
civetweb behave more like rgw over mod_fastcgi, build related changes,
and some more trivial stuff.

Yehuda

Re: civetweb upstream/downstream divergence

[Pete Zaitcev]

On Thu, 29 Oct 2015 10:58:07 -0700
Yehuda Sadeh-Weinraub <yehuda@redhat.com> wrote:

> We should definitely do it. We're based off civetweb 1.6, and there
> was no official civetweb version for quite a while, but 1.7 was tagged
> a few months ago. I made some effort and got most of our material
> changes upstream, however, there are some changes that might need some
> more work before we can get them merged, or might not make complete
> sense at all.

I take it Nathan is volunteering to parse the delta into logical pieces
and identify what upstream is willing to accept, right?

Dunno about SuSE, but as a Fedora packager I would prefer if we (Ceph)
talked upstream into making regular releases and then for us to stop
carrying it entirely. One less git submodule if nothing else.

-- Pete

Re: civetweb upstream/downstream divergence

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]

Hi Pete,

On 30/10/2015 13:57, Pete Zaitcev wrote:
> On Thu, 29 Oct 2015 10:58:07 -0700
> Yehuda Sadeh-Weinraub <yehuda@redhat.com> wrote:
> 
>> We should definitely do it. We're based off civetweb 1.6, and there
>> was no official civetweb version for quite a while, but 1.7 was tagged
>> a few months ago. I made some effort and got most of our material
>> changes upstream, however, there are some changes that might need some
>> more work before we can get them merged, or might not make complete
>> sense at all.
> 
> I take it Nathan is volunteering to parse the delta into logical pieces
> and identify what upstream is willing to accept, right?

I've discussed with Nathan about this general problem a few times. The issue is much less about volunteering and much more about how to track the progress of the delta over time.

> Dunno about SuSE, but as a Fedora packager I would prefer if we (Ceph)
> talked upstream into making regular releases and then for us to stop
> carrying it entirely. One less git submodule if nothing else.

Right now we have no method. For the jerasure / gf-complete sub-modules, I'm watching the delta and do the right thing but it's mostly an unwritten process: someone else would do it completely differently. For other Ceph sub-modules I suppose each developer has his own way of dealing with the delta.I remember Sage recently proposed patches upstream for rocksdb but I'm unaware of where or how. I would not be able to help him in any way. And I don't think anyone could figure out exactly how to deal with the jerasure / gf-complete sub-modules either.

Do you happen to know a project that is using submodules (or copies of projects instead of dependencies) and that is also well organized to track the delta ?

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: civetweb upstream/downstream divergence

[Ken Dreyer]

On Thu, Oct 29, 2015 at 10:57 PM, Pete Zaitcev <zaitcev@redhat.com> wrote:
> On Thu, 29 Oct 2015 10:58:07 -0700
> Yehuda Sadeh-Weinraub <yehuda@redhat.com> wrote:
>
>> We should definitely do it. We're based off civetweb 1.6, and there
>> was no official civetweb version for quite a while, but 1.7 was tagged
>> a few months ago. I made some effort and got most of our material
>> changes upstream, however, there are some changes that might need some
>> more work before we can get them merged, or might not make complete
>> sense at all.
>
> I take it Nathan is volunteering to parse the delta into logical pieces
> and identify what upstream is willing to accept, right?
>
> Dunno about SuSE, but as a Fedora packager I would prefer if we (Ceph)
> talked upstream into making regular releases and then for us to stop
> carrying it entirely. One less git submodule if nothing else.

I would heartily support the effort to get civetweb into the distros,
too, and make this a proper package with a shared library that RGW can
link against. This can be done in parallel to the "reconciling the
code content with civetweb upstream" effort of course :)

- Ken

Re: civetweb upstream/downstream divergence

[Martin Millnert]

On Fri, 2015-10-30 at 15:38 -0600, Ken Dreyer wrote:
> On Thu, Oct 29, 2015 at 10:57 PM, Pete Zaitcev <zaitcev@redhat.com> wrote:
> > Dunno about SuSE, but as a Fedora packager I would prefer if we (Ceph)
> > talked upstream into making regular releases and then for us to stop
> > carrying it entirely. One less git submodule if nothing else.
> 
> I would heartily support the effort to get civetweb into the distros,
> too, and make this a proper package with a shared library that RGW can
> link against. This can be done in parallel to the "reconciling the
> code content with civetweb upstream" effort of course :)

From a "devops" perspective with local CI infra, this (proper package
with shared libs) is a preferred packaging model.

/M

Re: civetweb upstream/downstream divergence

[Nathan Cutler]

IMHO the first step should be to get rid of the evil submodule. Arguably
the most direct path leading to this goal is to simply package up the
downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
the supported distros. The resulting package would be Ceph-specific,
obviously, so it could be called "civetweb-ceph".

Like Ken says, the upstreaming effort can continue in parallel.

After we get Ceph/RGW working fine with civetweb-ceph 1.6, we can rebase
the package to upstream civetweb 1.7.

I am not volunteering to do all the work, but we at SUSE are certainly
prepared to shoulder our share of it.

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037

Re: civetweb upstream/downstream divergence

[Sage Weil]

On Tue, 3 Nov 2015, Nathan Cutler wrote:
> IMHO the first step should be to get rid of the evil submodule. Arguably
> the most direct path leading to this goal is to simply package up the
> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
> the supported distros. The resulting package would be Ceph-specific,
> obviously, so it could be called "civetweb-ceph".
> 
> Like Ken says, the upstreaming effort can continue in parallel.

I'm not sure I agree.  As long as everything is not upstream and we are 
running a fork, what is the value of having it in a separate package?  
That just means all of the effort of managing the package dependency and 
making sure it is in all of the appropriate distros (and similar pain for 
those building manually) without any of the benefits (upstream bug fixes, 
etc.).

sage

Re: civetweb upstream/downstream divergence

[Ken Dreyer]

On Tue, Nov 3, 2015 at 4:22 AM, Sage Weil <sweil@redhat.com> wrote:
> On Tue, 3 Nov 2015, Nathan Cutler wrote:
>> IMHO the first step should be to get rid of the evil submodule. Arguably
>> the most direct path leading to this goal is to simply package up the
>> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
>> the supported distros. The resulting package would be Ceph-specific,
>> obviously, so it could be called "civetweb-ceph".
>>
>> Like Ken says, the upstreaming effort can continue in parallel.
>
> I'm not sure I agree.  As long as everything is not upstream and we are
> running a fork, what is the value of having it in a separate package?
> That just means all of the effort of managing the package dependency and
> making sure it is in all of the appropriate distros (and similar pain for
> those building manually) without any of the benefits (upstream bug fixes,
> etc.).

I think there's value in getting the packaging bits ready ahead of
time and letting those "bake in" in Fedora/Ubuntu/Debian/SUSE while we
continue to merge Ceph's civetweb changes to Civetweb upstream.

Now that Civetweb with RGW is mainstream, I'm looking forward to
eventually using a pre-built civetweb package that can shave time off
our Ceph Gitbuilder/Jenkins runs :)

- Ken

Re: civetweb upstream/downstream divergence

[Ken Dreyer]

On Wed, Nov 4, 2015 at 1:25 PM, Ken Dreyer <kdreyer@redhat.com> wrote:
> On Tue, Nov 3, 2015 at 4:22 AM, Sage Weil <sweil@redhat.com> wrote:
>> On Tue, 3 Nov 2015, Nathan Cutler wrote:
>>> IMHO the first step should be to get rid of the evil submodule. Arguably
>>> the most direct path leading to this goal is to simply package up the
>>> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
>>> the supported distros. The resulting package would be Ceph-specific,
>>> obviously, so it could be called "civetweb-ceph".
>>>
>>> Like Ken says, the upstreaming effort can continue in parallel.
>>
>> I'm not sure I agree.  As long as everything is not upstream and we are
>> running a fork, what is the value of having it in a separate package?
>> That just means all of the effort of managing the package dependency and
>> making sure it is in all of the appropriate distros (and similar pain for
>> those building manually) without any of the benefits (upstream bug fixes,
>> etc.).
>
> I think there's value in getting the packaging bits ready ahead of
> time and letting those "bake in" in Fedora/Ubuntu/Debian/SUSE while we
> continue to merge Ceph's civetweb changes to Civetweb upstream.
>
> Now that Civetweb with RGW is mainstream, I'm looking forward to
> eventually using a pre-built civetweb package that can shave time off
> our Ceph Gitbuilder/Jenkins runs :)

Oh, I just re-read this, and Nathan's proposing to package up
"civetweb-ceph" as a fork... I'm not sure that's worth it (at least,
speaking for packaging in Fedora).

When I was talking about a "parallel effort", what I meant is that
we'd get vanilla civetweb upstream into the distros, and we'd also
continue to bundle civetweb in Ceph, until we can reliably use the
upstream Civetweb package.

- Ken

Re: civetweb upstream/downstream divergence

[Sage Weil]

On Wed, 4 Nov 2015, Ken Dreyer wrote:
> On Wed, Nov 4, 2015 at 1:25 PM, Ken Dreyer <kdreyer@redhat.com> wrote:
> > On Tue, Nov 3, 2015 at 4:22 AM, Sage Weil <sweil@redhat.com> wrote:
> >> On Tue, 3 Nov 2015, Nathan Cutler wrote:
> >>> IMHO the first step should be to get rid of the evil submodule. Arguably
> >>> the most direct path leading to this goal is to simply package up the
> >>> downstream civetweb (i.e. 1.6 plus all the downstream patches) for all
> >>> the supported distros. The resulting package would be Ceph-specific,
> >>> obviously, so it could be called "civetweb-ceph".
> >>>
> >>> Like Ken says, the upstreaming effort can continue in parallel.
> >>
> >> I'm not sure I agree.  As long as everything is not upstream and we are
> >> running a fork, what is the value of having it in a separate package?
> >> That just means all of the effort of managing the package dependency and
> >> making sure it is in all of the appropriate distros (and similar pain for
> >> those building manually) without any of the benefits (upstream bug fixes,
> >> etc.).
> >
> > I think there's value in getting the packaging bits ready ahead of
> > time and letting those "bake in" in Fedora/Ubuntu/Debian/SUSE while we
> > continue to merge Ceph's civetweb changes to Civetweb upstream.
> >
> > Now that Civetweb with RGW is mainstream, I'm looking forward to
> > eventually using a pre-built civetweb package that can shave time off
> > our Ceph Gitbuilder/Jenkins runs :)
> 
> Oh, I just re-read this, and Nathan's proposing to package up
> "civetweb-ceph" as a fork... I'm not sure that's worth it (at least,
> speaking for packaging in Fedora).
> 
> When I was talking about a "parallel effort", what I meant is that
> we'd get vanilla civetweb upstream into the distros, and we'd also
> continue to bundle civetweb in Ceph, until we can reliably use the
> upstream Civetweb package.

Ah, this sounds better to me.  There may be some work to build civetweb as 
a shared library (currently it's just a statically linked module) but 
probably not too bad.

sage

Re: civetweb upstream/downstream divergence

[Martin Millnert]

On Wed, 2015-11-04 at 16:43 -0700, Ken Dreyer wrote:
> When I was talking about a "parallel effort", what I meant is that
> we'd get vanilla civetweb upstream into the distros, and we'd also
> continue to bundle civetweb in Ceph, until we can reliably use the
> upstream Civetweb package.

That's what sounds logical to me too.
So that aside, [Sage's] concern to this could probably be reduced to:
 1) getting new code out to clients via distro packages is slow,
 2) getting new code into upstream projects is tricky

2) as I understand it, has shown tricky for, for example for Sage, with
some FS projects, at least. And there's probably a risk to fall back
into the urge to fork again in the future if.
1) is probably a minor concern, at least simply considering how we're
deploying packages currently (as ceph operator). If you want the latest
and greatest, you solve your dependencies... If you run distro packages,
well, you're on distro packages already and they are packaged for
compatibility as-is. Maintainers of Ceph distro packages would need to
assure they work with civetweb packages.

Ultimately it's a question of whether or not the added labor of keeping
a fork is worth the benefit it brings in reduction of external
dependencies when deploying. The downside is, as in this case, risk of
lagging behind if fork isn't kept up-to-date.

Not obvious which is better.
/M