starting with jewel v10.2.5

starting with jewel v10.2.5

[Loic Dachary]

Hi Abhishek & Nathan & Abhishek,

Hopefully jewel will start testing with QE tomorrow or early next week. 

In the past we kept backporting but we were very careful not to merge anything because the release process is not able to work with a SHA1, it will create the next jewel release with whatever is at the tip of that branch. This limitation has been there for years and there is little chance that it will be resolved any time soon. It routinely creates frustration when something is merged by accident and tests have to be run again (or worse: they are not run and the release is done anyway). It also creates extra work for the backporters who need to collect backports that could have been merged because they passed the tests.

Instead, I propose that we backport, test and merge to wip-jewel, as if it was the jewel branch, until the release is done. And when it's done we rebase wip-jewel on top of jewel (or we verify that there is no need to rebase), we push all commits from wip-jewel to jewel, we delete the wip-jewel branch and we resume backporting to the jewel branch.

In a nutshell it means we have a little extra work:

a) re-targeting backports to wip-jewel in each PR (which is now possible with github)
b) pushing wip-jewel to jewel after the release and re-targeting backports to jewel

And the benefits are that 

a) we don't have to stop merging while QE runs tests and the release is prepared, which takes a few weeks
b) there is no risk of an accidental merge that would break the release process

What do you think ?

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: starting with jewel v10.2.5

[Nathan Cutler]

On 10/27/2016 12:02 PM, Loic Dachary wrote:
> Hi Abhishek & Nathan & Abhishek,
>
> Hopefully jewel will start testing with QE tomorrow or early next week.
>
> In the past we kept backporting but we were very careful not to merge anything because the release process is not able to work with a SHA1, it will create the next jewel release with whatever is at the tip of that branch. This limitation has been there for years and there is little chance that it will be resolved any time soon. It routinely creates frustration when something is merged by accident and tests have to be run again (or worse: they are not run and the release is done anyway). It also creates extra work for the backporters who need to collect backports that could have been merged because they passed the tests.
>
> Instead, I propose that we backport, test and merge to wip-jewel, as if it was the jewel branch, until the release is done. And when it's done we rebase wip-jewel on top of jewel (or we verify that there is no need to rebase), we push all commits from wip-jewel to jewel, we delete the wip-jewel branch and we resume backporting to the jewel branch.
>
> In a nutshell it means we have a little extra work:
>
> a) re-targeting backports to wip-jewel in each PR (which is now possible with github)
> b) pushing wip-jewel to jewel after the release and re-targeting backports to jewel
>
> And the benefits are that
>
> a) we don't have to stop merging while QE runs tests and the release is prepared, which takes a few weeks
> b) there is no risk of an accidental merge that would break the release process
>
> What do you think ?

I wouldn't say it completely removes the risk of an accidental merge to 
jewel, but it certainly reduces it :-)

Anyway, it sounds good to me. The only awkwardness is that it will 
require existing backport PRs to be closed and reopened (AFAIK there is 
no other way to "retarget" a PR to a different branch).

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037

Re: starting with jewel v10.2.5

[Loic Dachary]

On 27/10/2016 12:19, Nathan Cutler wrote:
> On 10/27/2016 12:02 PM, Loic Dachary wrote:
>> Hi Abhishek & Nathan & Abhishek,
>>
>> Hopefully jewel will start testing with QE tomorrow or early next week.
>>
>> In the past we kept backporting but we were very careful not to merge anything because the release process is not able to work with a SHA1, it will create the next jewel release with whatever is at the tip of that branch. This limitation has been there for years and there is little chance that it will be resolved any time soon. It routinely creates frustration when something is merged by accident and tests have to be run again (or worse: they are not run and the release is done anyway). It also creates extra work for the backporters who need to collect backports that could have been merged because they passed the tests.
>>
>> Instead, I propose that we backport, test and merge to wip-jewel, as if it was the jewel branch, until the release is done. And when it's done we rebase wip-jewel on top of jewel (or we verify that there is no need to rebase), we push all commits from wip-jewel to jewel, we delete the wip-jewel branch and we resume backporting to the jewel branch.
>>
>> In a nutshell it means we have a little extra work:
>>
>> a) re-targeting backports to wip-jewel in each PR (which is now possible with github)
>> b) pushing wip-jewel to jewel after the release and re-targeting backports to jewel
>>
>> And the benefits are that
>>
>> a) we don't have to stop merging while QE runs tests and the release is prepared, which takes a few weeks
>> b) there is no risk of an accidental merge that would break the release process
>>
>> What do you think ?
> 
> I wouldn't say it completely removes the risk of an accidental merge to jewel, but it certainly reduces it :-)

:-) One would have to a) open a PR against jewel, b) not be noticed by someone from the backport team, c) be merged by someone who is not aware of the new convention. It's indeed not impossible.

> Anyway, it sounds good to me. The only awkwardness is that it will require existing backport PRs to be closed and reopened (AFAIK there is no other way to "retarget" a PR to a different branch).

It is now possible, when you click "Edit" on the PR title you also have the option to change the target branch of the PR.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: starting with jewel v10.2.5

[Loic Dachary]

On 27/10/2016 13:19, Abhishek L wrote:
> 
> Loic Dachary writes:
> 
>> On 27/10/2016 12:19, Nathan Cutler wrote:
>>> On 10/27/2016 12:02 PM, Loic Dachary wrote:
>>>> Hi Abhishek & Nathan & Abhishek,
>>>>
>>>> Hopefully jewel will start testing with QE tomorrow or early next week.
>>>>
>>>> In the past we kept backporting but we were very careful not to merge anything because the release process is not able to work with a SHA1, it will create the next jewel release with whatever is at the tip of that branch. This limitation has been there for years and there is little chance that it will be resolved any time soon. It routinely creates frustration when something is merged by accident and tests have to be run again (or worse: they are not run and the release is done anyway). It also creates extra work for the backporters who need to collect backports that could have been merged because they passed the tests.
>>>>
>>>> Instead, I propose that we backport, test and merge to wip-jewel, as if it was the jewel branch, until the release is done. And when it's done we rebase wip-jewel on top of jewel (or we verify that there is no need to rebase), we push all commits from wip-jewel to jewel, we delete the wip-jewel branch and we resume backporting to the jewel branch.
>>>>
>>>> In a nutshell it means we have a little extra work:
>>>>
>>>> a) re-targeting backports to wip-jewel in each PR (which is now possible with github)
>>>> b) pushing wip-jewel to jewel after the release and re-targeting backports to jewel
>>>>
>>>> And the benefits are that
>>>>
>>>> a) we don't have to stop merging while QE runs tests and the release is prepared, which takes a few weeks
>>>> b) there is no risk of an accidental merge that would break the release process
>>>>
>>>> What do you think ?
>>>
>>> I wouldn't say it completely removes the risk of an accidental merge to jewel, but it certainly reduces it :-)
> 
> +1
> Let's do it.

Yeah !

>>
>> :-) One would have to a) open a PR against jewel, b) not be noticed by someone from the backport team, c) be merged by someone who is not aware of the new convention. It's indeed not impossible.
>>
>>> Anyway, it sounds good to me. The only awkwardness is that it will require existing backport PRs to be closed and reopened (AFAIK there is no other way to "retarget" a PR to a different branch).
>>
>> It is now possible, when you click "Edit" on the PR title you also have the option to change the target branch of the PR.
> 
> Nice, other note is for others who may send in PRs at this
> time for jewel branch, we may need to do this until 10.2.4 is released

True. It's low overhead though and we review every PR to check/add cross reference anyway so the burden is not high :-)

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: starting with jewel v10.2.5

[Abhishek Varshney]

If I understand the concerns right, how about doing it the other way round?

a) Create a new branch (say qe-jewel) from jewel whenever the release
is ready for QE testing.
b) The tests can now run on this branch and the release can be
prepared against this.
c) The PRs can continue to be merged to jewel as usual, while the
tests are being done.
d) Remove this branch once release is done, or push all commits from
jewel to qe-jewel whenever next release is ready.

This would require no change to the existing workflow atleast for
developers, and accidental merges to the branch under test can be
completely avoided.

~ Abhishek

On Thu, Oct 27, 2016 at 5:59 PM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 27/10/2016 13:19, Abhishek L wrote:
>>
>> Loic Dachary writes:
>>
>>> On 27/10/2016 12:19, Nathan Cutler wrote:
>>>> On 10/27/2016 12:02 PM, Loic Dachary wrote:
>>>>> Hi Abhishek & Nathan & Abhishek,
>>>>>
>>>>> Hopefully jewel will start testing with QE tomorrow or early next week.
>>>>>
>>>>> In the past we kept backporting but we were very careful not to merge anything because the release process is not able to work with a SHA1, it will create the next jewel release with whatever is at the tip of that branch. This limitation has been there for years and there is little chance that it will be resolved any time soon. It routinely creates frustration when something is merged by accident and tests have to be run again (or worse: they are not run and the release is done anyway). It also creates extra work for the backporters who need to collect backports that could have been merged because they passed the tests.
>>>>>
>>>>> Instead, I propose that we backport, test and merge to wip-jewel, as if it was the jewel branch, until the release is done. And when it's done we rebase wip-jewel on top of jewel (or we verify that there is no need to rebase), we push all commits from wip-jewel to jewel, we delete the wip-jewel branch and we resume backporting to the jewel branch.
>>>>>
>>>>> In a nutshell it means we have a little extra work:
>>>>>
>>>>> a) re-targeting backports to wip-jewel in each PR (which is now possible with github)
>>>>> b) pushing wip-jewel to jewel after the release and re-targeting backports to jewel
>>>>>
>>>>> And the benefits are that
>>>>>
>>>>> a) we don't have to stop merging while QE runs tests and the release is prepared, which takes a few weeks
>>>>> b) there is no risk of an accidental merge that would break the release process
>>>>>
>>>>> What do you think ?
>>>>
>>>> I wouldn't say it completely removes the risk of an accidental merge to jewel, but it certainly reduces it :-)
>>
>> +1
>> Let's do it.
>
> Yeah !
>
>>>
>>> :-) One would have to a) open a PR against jewel, b) not be noticed by someone from the backport team, c) be merged by someone who is not aware of the new convention. It's indeed not impossible.
>>>
>>>> Anyway, it sounds good to me. The only awkwardness is that it will require existing backport PRs to be closed and reopened (AFAIK there is no other way to "retarget" a PR to a different branch).
>>>
>>> It is now possible, when you click "Edit" on the PR title you also have the option to change the target branch of the PR.
>>
>> Nice, other note is for others who may send in PRs at this
>> time for jewel branch, we may need to do this until 10.2.4 is released
>
> True. It's low overhead though and we review every PR to check/add cross reference anyway so the burden is not high :-)
>
> Cheers
>
> --
> Loïc Dachary, Artisan Logiciel Libre

Re: starting with jewel v10.2.5

[Loic Dachary]

On 27/10/2016 17:04, Abhishek Varshney wrote:
> If I understand the concerns right, how about doing it the other way round?

That would make a lot of sense. It would however require changing the scripts that are used for the release and I suspect it won't be possible for the same reasons changing them to use the SHA1 did not happen.

> a) Create a new branch (say qe-jewel) from jewel whenever the release
> is ready for QE testing.
> b) The tests can now run on this branch and the release can be
> prepared against this.
> c) The PRs can continue to be merged to jewel as usual, while the
> tests are being done.
> d) Remove this branch once release is done, or push all commits from
> jewel to qe-jewel whenever next release is ready.
> 
> This would require no change to the existing workflow atleast for
> developers, and accidental merges to the branch under test can be
> completely avoided.
> 
> ~ Abhishek
> 
> On Thu, Oct 27, 2016 at 5:59 PM, Loic Dachary <loic@dachary.org> wrote:
>>
>>
>> On 27/10/2016 13:19, Abhishek L wrote:
>>>
>>> Loic Dachary writes:
>>>
>>>> On 27/10/2016 12:19, Nathan Cutler wrote:
>>>>> On 10/27/2016 12:02 PM, Loic Dachary wrote:
>>>>>> Hi Abhishek & Nathan & Abhishek,
>>>>>>
>>>>>> Hopefully jewel will start testing with QE tomorrow or early next week.
>>>>>>
>>>>>> In the past we kept backporting but we were very careful not to merge anything because the release process is not able to work with a SHA1, it will create the next jewel release with whatever is at the tip of that branch. This limitation has been there for years and there is little chance that it will be resolved any time soon. It routinely creates frustration when something is merged by accident and tests have to be run again (or worse: they are not run and the release is done anyway). It also creates extra work for the backporters who need to collect backports that could have been merged because they passed the tests.
>>>>>>
>>>>>> Instead, I propose that we backport, test and merge to wip-jewel, as if it was the jewel branch, until the release is done. And when it's done we rebase wip-jewel on top of jewel (or we verify that there is no need to rebase), we push all commits from wip-jewel to jewel, we delete the wip-jewel branch and we resume backporting to the jewel branch.
>>>>>>
>>>>>> In a nutshell it means we have a little extra work:
>>>>>>
>>>>>> a) re-targeting backports to wip-jewel in each PR (which is now possible with github)
>>>>>> b) pushing wip-jewel to jewel after the release and re-targeting backports to jewel
>>>>>>
>>>>>> And the benefits are that
>>>>>>
>>>>>> a) we don't have to stop merging while QE runs tests and the release is prepared, which takes a few weeks
>>>>>> b) there is no risk of an accidental merge that would break the release process
>>>>>>
>>>>>> What do you think ?
>>>>>
>>>>> I wouldn't say it completely removes the risk of an accidental merge to jewel, but it certainly reduces it :-)
>>>
>>> +1
>>> Let's do it.
>>
>> Yeah !
>>
>>>>
>>>> :-) One would have to a) open a PR against jewel, b) not be noticed by someone from the backport team, c) be merged by someone who is not aware of the new convention. It's indeed not impossible.
>>>>
>>>>> Anyway, it sounds good to me. The only awkwardness is that it will require existing backport PRs to be closed and reopened (AFAIK there is no other way to "retarget" a PR to a different branch).
>>>>
>>>> It is now possible, when you click "Edit" on the PR title you also have the option to change the target branch of the PR.
>>>
>>> Nice, other note is for others who may send in PRs at this
>>> time for jewel branch, we may need to do this until 10.2.4 is released
>>
>> True. It's low overhead though and we review every PR to check/add cross reference anyway so the burden is not high :-)
>>
>> Cheers
>>
>> --
>> Loïc Dachary, Artisan Logiciel Libre
> 

-- 
Loïc Dachary, Artisan Logiciel Libre

Re: starting with jewel v10.2.5

[Ken Dreyer]

On Thu, Oct 27, 2016 at 10:11 AM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 27/10/2016 17:04, Abhishek Varshney wrote:
>> If I understand the concerns right, how about doing it the other way round?
>
> That would make a lot of sense. It would however require changing the scripts that are used for the release and I suspect it won't be possible for the same reasons changing them to use the SHA1 did not happen.

Right, I think we would need to alter the build workflow to avoid
hard-coding any version at all within /debian/changelog, so that we no
longer need to do the "bump" commit, similar to what we do for the
RPMs with ceph.spec.in.

In Jenkins/Gitbuilder, we already have Jenkins run "dch" on every
build, to further distinguish trusty vs xenial builds that would
otherwise have the same NVR. I think we'd need to expand that concept
to also get the entire version from "git describe".

It would be cool to have this work so we could tag/release arbitrary
SHA1s on jewel, rather than having to bump /debian/changelog on the
tip of the branch and then tag that "new version" bump commit.

- Ken

Re: starting with jewel v10.2.5

[Alfredo Deza]

On Thu, Oct 27, 2016 at 1:53 PM, Ken Dreyer <kdreyer@redhat.com> wrote:
> On Thu, Oct 27, 2016 at 10:11 AM, Loic Dachary <loic@dachary.org> wrote:
>>
>>
>> On 27/10/2016 17:04, Abhishek Varshney wrote:
>>> If I understand the concerns right, how about doing it the other way round?
>>
>> That would make a lot of sense. It would however require changing the scripts that are used for the release and I suspect it won't be possible for the same reasons changing them to use the SHA1 did not happen.
>
> Right, I think we would need to alter the build workflow to avoid
> hard-coding any version at all within /debian/changelog, so that we no
> longer need to do the "bump" commit, similar to what we do for the
> RPMs with ceph.spec.in.
>
> In Jenkins/Gitbuilder, we already have Jenkins run "dch" on every
> build, to further distinguish trusty vs xenial builds that would
> otherwise have the same NVR. I think we'd need to expand that concept
> to also get the entire version from "git describe".
>
> It would be cool to have this work so we could tag/release arbitrary
> SHA1s on jewel, rather than having to bump /debian/changelog on the
> tip of the branch and then tag that "new version" bump commit.'

Exactly. That is the actual reason why we can't really build from a
sha1. However the changelog
is not the only file that is now changing, the CMakeLists.txt is now
affected too:

https://github.com/ceph/ceph/commit/697fe64f9f106252c49a2c4fe4d79aea29363be7

If we can find a middle ground where we don't need to keep modifying
these things on the fly for the build that would be easier
for us as well.


>
> - Ken
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: starting with jewel v10.2.5

[Loic Dachary]

Hi,

For the record the plan was implemented (i.e. using jewel-next until 10.2.4 is released):

* Documentation http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO_keep_backporting_while_the_release_branch_is_frozen
* jewel 10.2.5 backports http://tracker.ceph.com/issues/17851
* all PR re-targeted to jewel-next https://github.com/ceph/ceph/milestone/8
* tests running on jewel-backports (integration branch with ~40 PRs) http://tracker.ceph.com/issues/17851#note-2

The only unforseen inconvenience is that we must not use the ceph-workbench script to set the release version after merging because it relies on the existing tags to figure out which is next. This is however self discipline that only is a concern for backporters and will be transparent to everyone else. Ideally ceph-workbench is patched to address that but I'd rather wait until we are sure the plan to keep backporting while testing actually is helpful.

Cheers

On 27/10/2016 12:02, Loic Dachary wrote:
> Hi Abhishek & Nathan & Abhishek,
> 
> Hopefully jewel will start testing with QE tomorrow or early next week. 
> 
> In the past we kept backporting but we were very careful not to merge anything because the release process is not able to work with a SHA1, it will create the next jewel release with whatever is at the tip of that branch. This limitation has been there for years and there is little chance that it will be resolved any time soon. It routinely creates frustration when something is merged by accident and tests have to be run again (or worse: they are not run and the release is done anyway). It also creates extra work for the backporters who need to collect backports that could have been merged because they passed the tests.
> 
> Instead, I propose that we backport, test and merge to wip-jewel, as if it was the jewel branch, until the release is done. And when it's done we rebase wip-jewel on top of jewel (or we verify that there is no need to rebase), we push all commits from wip-jewel to jewel, we delete the wip-jewel branch and we resume backporting to the jewel branch.
> 
> In a nutshell it means we have a little extra work:
> 
> a) re-targeting backports to wip-jewel in each PR (which is now possible with github)
> b) pushing wip-jewel to jewel after the release and re-targeting backports to jewel
> 
> And the benefits are that 
> 
> a) we don't have to stop merging while QE runs tests and the release is prepared, which takes a few weeks
> b) there is no risk of an accidental merge that would break the release process
> 
> What do you think ?
> 

-- 
Loïc Dachary, Artisan Logiciel Libre